How-To Tutorials

Having effective executive sponsorship is a key success factor for any business solution implementation. There are countless books and presentations that have communicated this best practice to the marketplace. Executive sponsorship may guarantee financial support but it does not guarantee organizational acceptance and adoption. For example, Panorama Consulting conducted a study of 2008 Enterprise resource planning (ERP) implementations from over one thousand organizations across the globe. The study focused on the top ERP challenges. Respondents indicated that lack of employee adoption as the biggest challenge (33%). Surprisingly, not a single respondent identified lack of executive support as the biggest problem. Trickle down acceptance falls short A key assumption made on several packaged software implementations is that if the executive management selects Commercial Off The Shelf (COTS) software, then the organization would adopt the executive's intentions. When executive management selects packaged software, they are making the following statement: A custom software solution is not strategic for the organization. It is interesting to note that the above statement is not publicly reiterated throughout the life of the implementation project. It's like a little secret that is implied but hardly repeated. Unfortunately, the key business process owners and other stakeholders do not always respond to "trickle down" acceptance. Their expectation is based on their past software experience, which is usually getting what they want without any questions asked. Packaged software makes for an expensive custom solution. If the project team does not reset existing business software expectations, then it will be extremely difficult to be successful. If the project team makes no software changes and use packaged software as delivered then end user adoption and satisfaction will be harder to develop. If the project team focuses on building a custom solution then the Total Cost of Ownership (TCO) will be significant and will result in less executive satisfaction due to the increased cost. Successful COTS software implementations are those that are able to balance (negotiate) tradeoffs that result in minimizing TCO and maximizing organizational acceptance. The following section will discuss the key areas that the project team should address as part of their negotiation strategy. Developing an effective negotiation strategy An effective negotiation strategy for packaged software implementations is where all parties are aligned in supporting the success of the implementation. It is also importantfor the project team to perform certain activities to promote alignment and adoption. The first step in developing an effective negotiation strategy is to establish realistic expectations to close the gap between custom software and packaged software. Paradigm shift in business software expectations One of the key mindsets that the project team have to make with the stakeholders and end users are expectations of business software. Typically, stakeholders and end users see business software as a simple tool that can easily be adapted to support their activities. It is rare for stakeholders and end users to understand the complete ramifications of their requests. There are times when stakeholders and end users do not care to know what efforts are required to make the business software do exactly what they want. Regardless of the underlying drivers, the project team must make the effort to educate and influence stakeholders and end users to develop areas of compromise. Case Study I was the functional consulting lead for a packaged software application to replace a customer's legacy time reporting system. The customer's process consisted of electronic spreadsheets coupled together with a data load process to a staging table for payroll processing. When I started working with the customer to define their business requirements, it was apparent that there was a difference between executive and business stakeholder expectations. The customer insisted on having the exact functionality that they had in their existing systems. I knew that no matter how good the developers were, we would not be able to cost-effectively build the existing flexibility into the packaged software. It would totally invalidate the justification for going with a packaged software solution in the first place. The customer was used to getting exactly what they wanted from software to the point where software replaced the need for training (i.e., dummy-proofing). I did an initial gap analysis and identified over forty (40) gaps that required changes to the packaged software. I came to the realization that if I proceeded with a traditional approach to gathering requirements that: (a) I would spend a lot of wasted effort gathering non-value-added business requirements and (b) the fit/gap session would be much longer than anticipated. It was a no-win situation. The best approach was to reset software expectations with the customer up-front. I met with the customer to discuss the reasons for selecting packaged software and the inherent advantages and disadvantages with packaged software. This was not a one-time discussion and required several additional discussions. In the end, the customer's expectations adapted, which enabled us to accelerate our fit/gap session. We still identified ten gaps. Of the ten gaps, we negotiated to have only five addressed via a software change and the other five gaps were handled manually due to the frequency of occurrence. Our negotiations resulted in an 87% reduction from the initial estimate and were only made possible by establishing a suitable environment for effective negotiating. If either party has unrealistic expectations then effective negotiations would have been extremely difficult. Paradigm shift in organizational acceptance The second paradigm shift that the project team has to make is in how to gain organizational acceptance. This area is addressed in organizational change management. Organizational Change Management is good at defining the tactical steps that you need to perform in preparing the organization for a new business solution. However, a more aggressive approach needs to be taken. The entire project team needs to sell the solution to the organization. I'm talking about a concerted effort—no soft selling here. The project team needs to have real influence with the stakeholders and end users. The project team needs to lead discussions with stakeholders and key end users; negotiating a business solution that balances their needs (not wants) with the strategy of using packaged software. The project team needs to persuade their customers that they have the customer's best interests at heart and will produce a competent business solution. Understand when and where to negotiate Negotiating on business requirements is a certainty that must be planned for by the project team. The majority of projects do not have a proactive strategy in place to negotiate with stakeholders on business requirements that are not satisfied by the delivered packaged software. Many projects take a reactive approach to negotiating business requirements. This approach typically results in confusion, analysis paralysis, and decisions that undermine the objectives of selecting packaged business software. Following are the key points that the project team needs to address in understanding potential areas for negotiation: Identify areas within the packaged software where software changes are typically made by customers. With the correct implementation partner, the project team may have the opportunity to leverage their software change library to streamline the modification efforts. With the correct packaged software provider, you may have the opportunity to partner together and share the development effort. Identify areas within the packaged software where software changes will have minimal Total Cost of Ownership impact. For example, making a software changes to generate an exception report has less of an impact than building a new application page or process. Determine where the business software provider provides the most support for software changes within their application and lead with these options in the negotiations. For example, reporting is an area that most packaged software providers anticipate their customers modifying, and provide tools and templates for streamlined development and upgrades in this area. Identify areas within the packaged software where software changes will have an adverse impact to the Total Cost of Ownership. For example, software modifications to compiled executables can have several downstream impacts to other components within the packaged software. These software changes can not only negatively impact the Total Cost of Ownership but can also result in unstable business software, and invalidate software support and quality agreements. It is also important for the project team to understand the boundaries of the technical constraints inherent with the packaged software. Determine what to challenge and what to accept regarding business requirements not supported by the packaged software. In general, the project team should focus on maximizing enhancements and minimizing customizations. In the case where customizations must be addressed for project success, implement the customizations with the least impact to the Total Cost of Ownership. As the project team engages the key stakeholders and end users, do not lead with the functional areas that the team plans to address via software changes. Allow the stakeholders and end users to justify the reasons for negotiating and building software changes. What I learned from these software negotiations is that every party needs to perceive that they have won something. In theory, one can look at this as an inefficient process; however, it reflects human nature and a certain reality that the project team must address. The project team, should as well deal with these types of negotiations in a practical manner. Too often, I have witnessed project teams get stuck in theoretical absolutes that add no business value and slow down the project.

(For more resources related to this topic, see here.) Logging and reporting Reporting is the most important part of any test execution, reason being it helps the user to understand the result of the test execution, point of failure, and reasons for the failure. Logging, on the other hand, is important to keep an eye on the execution flow or for debugging in case of any failures. TestNG by default generates a different type of report for its test execution. This includes an HTML and an XML report output. TestNG also allows its users to write their own reporter and use it with TestNG. There is also an option to write your own loggers, which are notified at runtime by TestNG. There are two main ways to generate a report with TestNG: Listeners : For implementing a listener class, the class has to implement the org.testng.ITestListener interface. These classes are notified at runtime by TestNG when the test starts, finishes, fails, skips, or passes. Reporters : For implementing a reporting class, the class has to implement an org.testng.IReporter interface. These classes are called when the whole suite run ends. The object containing the information of the whole test run is passed to this class when called. Each of them should be used depending upon the condition of how and when the reports have to be generated. For example, if you want to generate a custom HTML report at the end of execution then you should implement IReporter interface while writing extension. But in case you want to update a file at runtime and have to print a message as and when the tests are getting executed, then we should use the ITestListener interface. Writing your own logger We had earlier read about the different options that TestNG provides for logging and reporting. Now let's learn how to start using them. To start with, we will write a sample program in which we will use the ITestListener interface for logging purposes. Time for action – writing a custom logger Open Eclipse and create a Java project with the name CustomListener and with the following structure. Please make sure that the TestNG library is added to the build path of the project. Create a new class named SampleTest under the test.sample package and replace the following code in it: package test.sample; import org.testng.Assert; import org.testng.annotations.Test; public class SampleTest { @Test public void testMethodOne(){ Assert.assertTrue(true); } @Test public void testMethodTwo(){ Assert.assertTrue(false); } @Test(dependsOnMethods={"testMethodTwo"}) public void testMethodThree(){ Assert.assertTrue(true); } } The preceding test class contains three test methods out of which testMethodOne and testMethodThree will pass when executed, whereas testMethodTwo is made to fail by passing a false Boolean value to the Assert.assertTrue method which is used for truth conditions in the tests. In the preceding class test method testMethodThree depends on testMethodTwo. Create another new class named CustomLogging under the test.logger package and replace its code with the following code: package test.logger; import java.text.DateFormat; import java.text.SimpleDateFormat; import java.util.Date; import org.testng.ITestContext; import org.testng.ITestListener; import org.testng.ITestResult; public class CustomLogging implements ITestListener{ //Called when the test-method execution starts @Override public void onTestStart(ITestResult result) { System.out.println("Test method started: "+ result.getName()+ " and time is: "+getCurrentTime()); } //Called when the test-method execution is a success @Override public void onTestSuccess(ITestResult result) { System.out.println("Test method success: "+ result.getName()+ " and time is: "+getCurrentTime()); } //Called when the test-method execution fails @Override public void onTestFailure(ITestResult result) { System.out.println("Test method failed: "+ result.getName()+ " and time is: "+getCurrentTime()); } //Called when the test-method is skipped @Override public void onTestSkipped(ITestResult result) { System.out.println("Test method skipped: "+ result.getName()+ " and time is: "+getCurrentTime()); } //Called when the test-method fails within success percentage @Override public void onTestFailedButWithinSuccessPercentage(ITestResult result) { // Leaving blank } //Called when the test in xml suite starts @Override public void onStart(ITestContext context) { System.out.println("Test in a suite started: "+ context.getName()+ " and time is: "+getCurrentTime()); } //Called when the test in xml suite finishes @Override public void onFinish(ITestContext context) { System.out.println("Test in a suite finished: "+ context.getName()+ " and time is: "+getCurrentTime()); } //Returns the current time when the method is called public String getCurrentTime(){ DateFormat dateFormat = new SimpleDateFormat("HH:mm:ss:SSS"); Date dt = new Date(); return dateFormat.format(dt); } } The above test class extends the ITestListener interface and defines the overriding methods of the interface. Details of each of the methods are provided as comments inside the previous code. Each method when executed prints the respective test method name or the suite name and the time when it was called. The getCurrentTime method returns the current time in HH:mm:ss:SSS format using the Date and DateFormat class. Create a new testing XML file under the project with name simple-logger-testng.xml and copy the following contents onto it: <suite name="Simple Logger Suite"> <listeners> <listener class-name="test.logger.CustomLogging" /> </listeners> <test name="Simple Logger test"> <classes> <class name="test.sample.SampleTest" /> </classes> </test> </suite> The preceding XML defines a simple test which considers the class test.sample.SampleTest for test execution. The CustomLogging class which implements the ITestListener is added as a listener to the test suite using the listeners tag as shown in the preceding XML. Select the previous testing XML file in Eclipse and run it as TestNG suite. You will see the following test result in the Console window of Eclipse: The following screenshot shows the test methods that were executed, failed, and skipped in the test run: What just happened? We created a custom logger class which implements the ITestListener interface and attached itself to the TestNG test suite as a listener. Methods of this listener class are invoked by TestNG as and when certain conditions are met in the execution, for example, test started, test failure, test success, and so on. Multiple listeners can be implemented and added to the test suite execution, TestNG will invoke all the listeners that are attached to the test suite. Logging listeners are mainly used when we need to see the continuous status of the test execution when the tests are getting executed. Writing your own reporter In the earlier section we had seen an example of writing your custom logger and attaching it to TestNG. In this section we will cover, with an example, the method of writing your custom reporter and attaching it to TestNG. To write a custom reporter class, our extension class should implement the IReporter interface. Let's go ahead and create an example with the custom reporter. Time for action – writing a custom reporter Open the previously created project named CustomListener and create a package named reporter under the test package. Create a new class named CustomReporter under the test.reporter package and add the following code to it: package test.reporter; import java.util.List; import java.util.Map; import org.testng.IReporter; import org.testng.ISuite; import org.testng.ISuiteResult; import org.testng.ITestContext; import org.testng.xml.XmlSuite; public class CustomReporter implements IReporter { @Override public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) { //Iterating over each suite included in the test for (ISuite suite : suites) { //Following code gets the suite name String suiteName = suite.getName(); //Getting the results for the said suite Map<String, ISuiteResult> suiteResults = suite.getResults(); for (ISuiteResult sr : suiteResults.values()) { ITestContext tc = sr.getTestContext(); System.out.println("Passed tests for suite '" + suiteName + "' is:" + tc.getPassedTests().getAllResults().size()); System.out.println("Failed tests for suite '" + suiteName + "' is:" + tc.getFailedTests().getAllResults().size()); System.out.println("Skipped tests for suite '" + suiteName + "' is:" + tc.getSkippedTests().getAllResults().size()); } } } } The preceding class implements the org.testng.IReporter interface. It implements the definition for the method generateReport of the IReporter interface. The method takes three arguments , the first being xmlSuite, which is the list suites mentioned in the testng XML being executed. The second one being suites which contains the suite information after the test execution; this object contains all the information about the packages, classes, test methods, and their test execution results. The third being the outputDirectory, which contains the information of the output folder path where the reports will be generated. The custom report prints the total number of tests passed, failed, and skipped for each suite included in the particular test execution when added to TestNG as a listener. Create a new file named simple-reporter-testng.xml to the project and add the following code to it: <suite name="Simple Reporter Suite"> <listeners> <listener class-name="test.reporter.CustomReporter" /> </listeners> <test name="Simple Reporter test"> <classes> <class name="test.sample.SampleTest" /> </classes> </test> </suite> The preceding XML is a testng XML configuration file. It contains a single test with the class test.sample.SampleTest to be considered for test execution. The CustomReporter class is added as a listener to the test suite using the listeners and listener tag as defined in the previous file. Select the preceding XML file and run it as TestNG test suite in Eclipse. You will see the following test results under the Console window of Eclipse: What just happened? We successfully created an example of writing custom reporter and attaching it to TestNG as a listener. The preceding example shows a simple custom reporter which prints the number of failed, passed, and skipped tests on the console for each suite included the said test execution. Reporter is mainly used to generate the final report for the test execution. The extension can be used to generate XML, HTML, XLS, CSV, or text format files depending upon the report requirement. TestNG HTML and XML report TestNG comes with certain predefined listeners as part of the library. These listeners are by default added to any test execution and generate different HTML and XML reports for any test execution. The report is generated by default under the folder named testoutput and can be changed to any other folder by configuring it. These reports consist of certain HTML and XML reports that are TestNG specific. Let's create a sample project to see how the TestNG report is generated. Time for action – generating TestNG HTML and XML reports Open Eclipse and create a Java project with the name SampleReport having the following structure. Please make sure that the TestNG library is added to the build path of the project. Create a new class named SampleTest under the test package and replace the following code in it: package test; import org.testng.Assert; import org.testng.annotations.Test; public class SampleTest { @Test public void testMethodOne(){ Assert.assertTrue(true); } @Test public void testMethodTwo(){ Assert.assertTrue(false); } @Test(dependsOnMethods={"testMethodTwo"}) public void testMethodThree(){ Assert.assertTrue(true); } } The preceding test class contains three test methods out of which testMethodOne and testMethodThree will pass when executed, whereas testMethodTwo is made to fail by passing a false Boolean value to Assert.assertTrue method. In the preceding class test method testMethodThree depends on testMethodTwo. Select the previously created test class and run it as a TestNG test through Eclipse. Now refresh the Java project in Eclipse by selecting the project and pressing the F5 button or right-clicking and selecting Refresh , and this will refresh the project. You will see a new folder named test-output under the project. Expand the folder in Eclipse and you will see the following files as shown in the screenshot: Open index.html as shown in the preceding screenshot on your default web browser. You will see the following HTML report: Now open the file testing-results.xml in the default XML editor on your system, and you will see the following results in the XML file: What just happened? We successfully created a test project and generated a TestNG HTML and XML report for the test project. TestNG by default generates multiple reports as part of its test execution. These reports mainly include TestNG HTML report, TestNG emailable report, TestNG report XML, and JUnit report XML files. These files can be found under the output report folder (in this case test-output). These default report generation can be disabled while running the tests by setting the value of the property useDefaultListeners to false. This property can be set while using the build tools like Ant or Maven as explained in the previous chapter. Generating a JUnit HTML report JUnit is one of those unit frameworks which were initially used by many Java applications as a Unit test framework. By default, JUnit tests generate a simple report XML files for its test execution. These XML files can then be used to generate any custom reports as per the testing requirement. We can also generate HTML reports using the XML files. Ant has such a utility task which takes these JUnit XML files as input and generates an HTML report from it. We had earlier learnt that TestNG by default generates the JUnit XML reports for any test execution. We can use these XML report files as input for generation of a JUnit HTML report. Assuming we already have JUnit XML reports available from the earlier execution let's create a simple Ant build configuration XML file to generate an HTML report for the test execution. Time for action – generating a JUnit report Go to the previously created Java project in Eclipse. Create a new file named junit-report-build.xml by selecting the project. Add the following code to the newly created file and save it: <project name="Sample Report" default="junit-report" basedir="."> <!-- Sets the property variables to point to respective directories --> <property name="junit-xml-dir" value="${basedir}/test-output/junitreports"/> <property name="report-dir" value="${basedir}/html-report" /> <!-- Ant target to generate html report --> <target name="junit-report"> <!-- Delete and recreate the html report directories --> <delete dir="${report-dir}" failonerror="false"/> <mkdir dir="${report-dir}" /> <mkdir dir="${report-dir}/Junit" /> <!-- Ant task to generate the html report. todir - Directory to generate the output reports fileset - Directory to look for the junit xml reports. report - defines the type of format to be generated. Here we are using "noframes" which generates a single html report. --> <junitreport todir="${report-dir}/Junit"> <fileset dir="${junit-xml-dir}"> <include name="**/*.xml" /> </fileset> <report format="noframes" todir="${report-dir}/Junit" /> </junitreport> </target> </project> The preceding XML defines a simple Ant build.xml file having a specific Ant target named junit-report that generates a JUnit report when executed. The target looks for the JUnit report XML files under the directory test-output/junitreports. For the Ant configuration file the default target to execute is configured as junit-report. Open the terminal window and go to the Java project directory in the terminal. Run the command ant –buildfile junit-report-build.xml and press Enter . Once executed a JUnit HTML report will be generated in the configured directory /html-report/Junit. Open the file named junit-noframes.html on your default web browser. You will see the following HTML report: What just happened? In this section we have seen how to use the JUnit XML report generated by TestNG and generate HTML report using Ant. There are two kinds of reports that can be generated using this method: frames and no-frames. If the report generation is configured with frames there will multiple files generated for each class and the main report will connect to them through links. A no-frames report consists of a single file with all the results of the test execution. This can be configured by providing the respective value to the format attribute of the report task in Ant. Generating a ReportNG report We had earlier seen that TestNG provides options to add custom listeners for logging and reporting. These listeners can easily be added to the TestNG execution and will be called during the execution or at the end of the execution depending upon the type of listener. ReportNG is a reporter add-on for TestNG that implements the report listener of TestNG. ReportNG reports are better looking reports compared to the original HTML reports. To generate a ReportNG report we have to add the reporting class to the list of listeners of TestNG while executing the tests. Let's see how to add ReportNG listener to TestNG and generate a ReportNG HTML report. In the following example we will use an Ant build XML file used to run our tests. Time for action – generating a ReportNG report Go to the earlier created Java project SampleProject in Eclipse. Download ReportNG from http://reportng.uncommons.org/ download site. Unzip and copy the reportng-<version>.jar and velocity-dep-<version>.jar from the unzipped folder to the lib folder under the project. Download guice from guice site https://code.google.com/p/google-guice/downloads/list. Unzip the downloaded guice zip file and copy the guice-<version>.jar to the lib folder under the project. Create a new file named testng.xml under the folder and add the following content to it: <suite name="Sample Suite"> <test name="Sample test"> <classes> <class name="test.SampleTest" /> </classes> </test> </suite> Create a new file named reportng-build.xml by selecting the project. Add the following code to the newly created file and save it: <project name="Testng Ant build" basedir="."> <!-- Sets the property varaibles to point to respective directories --> <property name="report-dir" value="${basedir}/html-report" /> <property name="testng-report-dir" value="${report-dir}/TestNG-report" /> <property name="lib-dir" value="${basedir}/lib" /> <property name="bin-dir" value="${basedir}/bin-dir" /> <property name="src-dir" value="${basedir}/src" /> <!-- Sets the classpath including the bin directory and all thejars under the lib folder --> <path id="test.classpath"> <pathelement location="${bin-dir}" /> <fileset dir="${lib-dir}"> <include name="*.jar" /> </fileset> </path> <!-- Deletes and recreate the bin and report directory --> <target name="init"> <delete dir="${bin-dir}" /> <mkdir dir="${bin-dir}" /> <delete dir="${report-dir}" /> <mkdir dir="${report-dir}" /> </target> <!-- Compiles the source code present under the "srcdir" and place class files under bin-dir --> <target name="compile" depends="init"> <javac srcdir="${src-dir}" classpathref="test.classpath" includeAntRuntime="No" destdir="${bin-dir}" /> </target> <!-- Defines a TestNG task with name "testng" --> <taskdef name="testng" classname="org.testng.TestNGAntTask" classpathref="test.classpath" /> <!--Executes the testng tests configured in the testng.xml file--> <target name="testng-execution" depends="compile"> <mkdir dir="${testng-report-dir}" /> <testng outputdir="${testng-report-dir}" classpathref="test.classpath" useDefaultListeners="false" listeners="org.uncommons.reportng.HTMLReporter"> <!-- Configures the testng xml file to use as test-suite --> <xmlfileset dir="${basedir}" includes="testng.xml" /> <sysproperty key="org.uncommons.reportng.title" value="ReportNG Report" /> </testng> </target> </project> The preceding XML defines a simple Ant build XML file that generates a ReportNG report when executed. The said XML compiles and runs the TestNG tests. ReportNG is added as a listener and the default listener of TestNG is disabled by setting a false value to the useDefaultListeners attribute while using the testng Ant task. Open the terminal window and go to the Java project directory in the terminal. Run the command ant -buildfile reportng-build.xml testing execution and then press Enter . Once executed a ReportNG HTML report will be generated in the configured directory html-reportTestNG-reporthtml under the said project directory. Go to the said directory and open the index.html file on your default web browser. You will see the following HTML report: By clicking on the Sample test link, you will see details of the test report as shown in the following screenshot: What just happened? In the previous section we learned how to generate a ReportNG HTML report for our test execution. We disabled the default TestNG reports in the previous Ant XML file but, if required, we can generate both the default as well as ReportNG reports by enabling the default report listeners. In the previous example the title of the report is configured by setting the property org.uncommons.reportng.title. There are other configuration options that we can use while generating the report, and we will cover these in the next section. ReportNG configuration options ReportNG provides different configuration options based on which the respective HTML report is generated. Following is a list of configurations that are supported: org.uncommons.reportng.coverage-report: This is configured as the link to the test coverage report. org.uncommons.reportng.escape-output: This property is used to turn off the log output in reports. By default it's turned off and is not recommended to be switched on as enabling this may require certain hacks to be implemented for proper report generation. org.uncommons.reportng.frames: This property is used to generate an HTML report with frameset and without frameset. The default value is set to true and hence it generates HTML reports with frameset by default. org.uncommons.reportng.locale: Used to override the localized messages in the generated HTML report. org.uncommons.reportng.stylesheet: This property can be used to customize the CSS property of the generated HTML report. org.uncommons.reportng.title: Used to define a report title for the generated HTML report. Have a go hero Write an Ant file to configure ReportNG to generate an HTML report without any frames for the TestNG execution. Generating a Reporty-ng (former TestNG-xslt) report While looking at the test report, senior managers might like to see the report in a graphical representation to know the status of the execution with just a glance. Reporty-ng (formerly called TestNG-xslt) is one such add-on report that generates a pie chart for your test execution with all the passed, failed, and skipped tests. This plugin uses the XSL file to convert the TestNG XML report into the custom HTML report with a pie chart. To use this plugin we will write an Ant target which will use the TestNG results XML file to generate the report. Let's go ahead and write an Ant target to generate the report. Time for action – generating a Reporty-ng report Open the previously created SampleReport in Eclipse. Download the Reporty-ng from the URL: https://github.com/cosminaru/reporty-ng At the time of writing this book the latest version available was Reporty-ng 1.2. You can download a newer version if available. Changes in the installation process should be minor if there are any at all. Unzip the downloaded zip and copy a file named testng-results.xsl from srcmainresources onto the resources folder under the said project. Copy the JARs saxon-8.7.jar and SaxonLiason.jar from the unzipped Reporty-ng lib folder to the project lib folder. Create a new Ant XML configuration file named reporty-ng-report.xml and paste the following code onto it: <project name="Reporty-ng Report" default="reporty-ng-report" basedir="."> <!-- Sets the property variables to point to respective directories --> <property name="xslt-report-dir" value="${basedir}/reporty-ng/" /> <property name="report-dir" value="${basedir}/html-report" /> <property name="lib-dir" value="${basedir}/lib" /> <path id="test.classpath"> <fileset dir="${lib-dir}"> <include name="**/*.jar" /> </fileset> </path> <target name="reporty-ng-report"> <delete dir="${xslt-report-dir}" /> <mkdir dir="${xslt-report-dir}" /> <xslt in="${basedir}/test-output/testng-results.xml" style="${basedir}/resources/testng-results.xsl" out="${xslt-report-dir}/index.html"> <param name="testNgXslt.outputDir" expression="${xslt-report-dir}" /> <param name="testNgXslt.sortTestCaseLinks" expression="true" /> <param name="testNgXslt.testDetailsFilter" expression="FAIL,SKIP,PASS,CONF,BY_CLASS" /> <param name="testNgXslt.showRuntimeTotals" expression="true" /> <classpath refid="test.classpath" /> </xslt> </target> </project> The preceding XML defines Ant build XML configuration, it contains an Ant target which generates the Reporty-ng report. The input path for testng results XML is configured through the in attribute of the xslt task in Ant. The transformation from XML to HTML is done using the testng-results.xsl of Reporty-ng, the location of which is configured by using the style attribute of the xslt task. The output HTML name is configured using the out attribute. Different configuration parameters for Reporty-ng are configured using the param task of Ant as shown in the preceding code. Now go to the said project folder through the command terminal and type the command ant –buildfile reporty-ng-build.xml and press Enter . You will see the following console output on the terminal: Now go to the configured report output folder reporty-ng (in this case) and open the file index.html in your default browser. You will see the following test report: On clicking the Default suite link on the left-hand side, a detailed report of the executed test cases will be displayed as shown in the following screenshot: What just happened? In the previous example we learned how to generate a Reporty-ng report using Ant. The report is very good from a report point of view as it gives a clear picture of the test execution through the pie chart. The output report can be configured using different configurations, which we will cover in the next section. Configuration options for Reporty-ng report As said earlier there are different configuration options that the Reporty-ng report supports while generating the report. Following is the list of supported configuration options and how they affect the report generation: testNgXslt.outputDir: Sets the target output directory for the HTML content. This is mandatory and must be an absolute path. If you are using the Maven plugin, this is set automatically so you don't have to provide it. testNgXslt.cssFile: Specifies an alternative style sheet file overriding the default settings. This parameter is not required. testNgXslt.showRuntimeTotals: Boolean flag indicating if the report should display the aggregated information about the method durations. The information is displayed for each test case and aggregated for the whole suite. Non-mandatory parameter, defaults to false. testNgXslt.reportTitle: Use this setting to specify a title for your HTML reports. This is not a mandatory parameter and defaults to TestNG Results. testNgXslt.sortTestCaseLinks: Indicates whether the test case links (buttons) in the left frame should be sorted alphabetically. By default they are rendered in the order they are generated by TestNG so you should set this to true to change this behavior. testNgXslt.chartScaleFactor: A scale factor for the SVG pie chart in case you want it larger or smaller. Defaults to 1. testNgXslt.testDetailsFilter: Specifies the default settings for the checkbox filters at the top of the test details page. Can be any combination (comma-separated) of: FAIL,PASS,SKIP,CONF,BY_CLASS. Have a go hero Write an Ant target in Ant to generate a Reporty-ng report with only Fail and Pass filter options. Pop quiz – logging and reports Q1. Which interface should the custom class implement for tracking the execution status as and when the test is executed? org.testng.ITestListener org.testng.IReporter Q2. Can we disable the default reports generated by the TestNG? Yes No Summary In this chapter we have covered different sections related to logging and reporting with TestNG. We have learned about different logging and reporting options provided by TestNG, writing our custom loggers and reporters and methods to generate different reports for our TestNG execution. Each of the reports have certain characteristics and any or all of the reports can be generated and used for test execution depending upon the requirement. Till now we have been using the XML configuration methodology of defining and writing our TestNG test suites. In the next chapter we will learn how to define/configure the TestNG test suite through code. This is helpful for defining the test suite at runtime. Resources for Article : Further resources on this subject: Developing Seam Applications [Article] The Aliens Have Landed! [Article] Visual Studio 2010 Test Types [Article]

  (For more resources on BackTrack, see here.) Man-in-the-Middle attack MITM attacks are probably one of most potent attacks on a WLAN system. There are different configurations that can be used to conduct the attack. We will use the most common one—the attacker is connected to the Internet using a wired LAN and is creating a fake access point on his client card. This access point broadcasts an SSID similar to a local hotspot in the vicinity. A user may accidently get connected to this fake access point and may continue to believe that he is connected to the legitimate access point. The attacker can now transparently forward all the user's traffic over the Internet using the bridge he has created between the wired and wireless interfaces. In the following lab exercise, we will simulate this attack. Time for action – Man-in-the-Middle attack Follow these instructions to get started: To create the Man-in-the-Middle attack setup, we will first c create a soft access point called mitm on the hacker laptop using airbase-ng. We run the command airbase-ng --essid mitm –c 11 mon0: It is important to note that airbase-ng when run, creates an interface at0 (tap interface). Think of this as the wired-side interface of our software-based access point mitm. Let us now create a bridge on the hacker laptop, consisting of the wired (eth0) and wireless interface (at0). The succession of commands used for this are—brctl addbr mitm-bridge, brctl addif mitm-bridge eth0, brctl addif mitmbridge at0, ifconfig eth0 0.0.0.0 up, ifconfig at0 0.0.0.0 up: We can assign an IP address to this bridge and check the connectivity with the gateway. Please note that we could do the same using DHCP as well. We can assign an IP address to the bridge interface with the command—ifconfig mitm-bridge 192.168.0.199 up. We can then try pinging the gateway 192.168.0.1 to ensure we are connected to the rest of the network: Let us now turn on IP Forwarding in the kernel so that routing and packet forwarding can happen correctly using echo > 1 /proc/sys/net/ipv4/ip_forward: Now let us connect a wireless client to our access point mitm. It would automatically get an IP address over DHCP (server running on the wired-side gateway). The client machine in this case receives the IP address 192.168.0.197. We can ping the wired side gateway 192.168.0.1 to verify connectivity: We see that the host responds to the ping requests as seen: We can also verify that the client is connected by looking at the airbase-ng terminal on the hacker machine: It is interesting to note here that because all the traffic is being relayed from the wireless interface to the wired-side, we have full control over the traffic. We can verify this by starting Wireshark and start sniffing on the at0 interface: (Move the mouse over the image to enlarge it.) Let us now ping the gateway 192.168.0.1 from the client machine. We can now see the packets in Wireshark (apply a display filter for ICMP), even though the packets are not destined for us. This is the power of Man-in-the-Middle attacks! (Move the mouse over the image to enlarge it.) What just happened? We have successfully created the setup for a wireless Man-In-The-Middle attack. We did this by creating a fake access point and bridging it with our Ethernet interface. This ensured that any wireless client connecting to the fake access point would "perceive" that it is connected to the Internet via the wired LAN. Have a go hero – Man-in-the-Middle over pure wireless In the previous exercise, we bridged the wireless interface with a wired one. As we noted earlier, this is one of the possible connection architectures for an MITM. There are other combinations possible as well. An interesting one would be to have two wireless interfaces, one creates the fake access point and the other interface is connected to the authorized access point. Both these interfaces are bridged. So, when a wireless client connects to our fake access point, it gets connected to the authorized access point through the attacker machine. Please note that this configuration would require the use of two wireless cards on the attacker laptop. Check if you can conduct this attack using the in-built card on your laptop along with the external one. This should be a good challenge!

(For more resources related to this topic, see here.) The following diagram provides a high-level view of the basic Provisioning Services infrastructure and clarifies how Provisioning Services components might appear within the datacenter post installation and implementation: Provisioning Service License server The License Server either should be installed within the shared infrastructure or an existing Citrix license server can be selected. However, we have to ensure the Provisioning Service license is configured in your existing Citrix Enterprise License servers. A License Server can be selected when the Provisioning Service Configuration Wizard is run on a planned server. All Provisioning Servers within the farm must be able to communicate with the License Server. Provisioning Service Database server The database stores all system configuration settings that exist within a farm. Only one database can exist within a provisioning service farm. We can choose an existing SQL Server database or install an SQL Server in cluster for High Availability from a redundancy business continuities perspective. The Database server can be selected when the Provisioning Service Configuration Wizard runs on a planned server. All Provisioning Servers within the farm must be able to communicate with the Database server, and only one database can exist within a Provisioning Service farm Provisioning Service Admin Console Citrix Provisioning Service Admin Console is a tool that is used to control your Provisioning Services implementation. After logging on to the console, we can select the farm that we want to connect to. Our role determines what we can look at in the console and operate in the Provisioning Service farm. Shared storage service Citrix Provisioning Service requires shared storage for vDisks that are accessible by all of the users in a network. They are intended for file storage and allowing simultaneous access by multiple users without the need to replicate files to their machines' vDisk. The supported shared storages are SAN, NAS, iSCSI, and CIFS. Active Directory Server Citrix Provisioning service requires Microsoft's Active Directory. It provides authentication and authorization mechanisms as well as a framework, within which other related services can be deployed. Microsoft Active Directory is an LDAP-compliant database that contains objects. The most commonly used objects are users, computers, and groups Network services Dynamic Host Control Protocol (DHCP) is used for the purpose of getting IP addresses for servers and systems. Trivial File Transfer Protocol (TFTP) is used for automated transfer of boot configuration files between servers and a system in a network. Preboot Execution Environment (PXE) is a standard used for client/server interface that allows networked computers that boot remotely to boot locally instead. System requirements Citrix Provisioning Service can be installed with following requirements: Citrix Provisioning Server Requirement Description Operation system Windows 2012: Standard, Essential, and Datacenter editions; Windows 2008 R2; Windows 2008 R2 SP1: Standard, Enterprise, and DataCenter editions; and all editions of Windows 2008 (32 or 64-bit) Processor Intel or AMD x86 or x64 compatible 2 GHz / 3 GHz (preferred) / 3.5 GHz Dual Core / HT or an equal one for growing capacity fulfiller Memory 2 GB RAM; 4 GB (greater than 250 vDisks) Hard disk To determine IOPS needed along RAID Level, please plan your sizing based on the following formula: Total Raw IOPS = Disk Speed IOPS x # of Disks Functional IOPS = ((Total Raw IOPS * Write %)/RAID Penalty ) + (Total Raw IOPS*Read %) For more, please refer to http://support.citrix.com/servlet/KbServlet/download/24559-102-647931/ Network adapter IP assignment to servers should be static. 1 GB is recommended for less than 250 target devices. If you are planning for more than 250 devices, Dual 1 GB is recommended. For High Availability, please have two NICs for redundancy purposes. Pre-requisite software components Microsoft .NET 4.0 and Microsoft Powershell 3.0 loaded on a fresh OS The Infrastructure components required are described as follows: Requirement Description Supported database Microsoft SQL 2008, Microsoft SQL 2008 R2, and Microsoft SQL 2012 Server (32-bit or 64-bit editions) databases can be used for the Provisioning ServicesDB sizing. Please refer to http://msdn.microsoft.com/en-us/library/ms187445.aspx. For HA Planning, please refer to http://support.citrix.com/proddocs/topic/provisioning-7/pvs-installtask1-plan-6-0.html. Supported hypervisor XenServer 6.0, Microsoft SCVMM 2012 SP1 with Hyper-V 3.0; SCVMM 2012 with Hyper-V 2.0, VMware ESX 4.1, ESX 5, or ESX 5 Update 1; vSphere 5.0, 5.1, 5.1 Update 1; along with Physical Devices for 3D Pro Graphics (Blade Servers, Windows Server OS machines, and Windows Desktop OS machines with XenDesktop VDA installed). Provisioning Console Hardware requirement: Processor 2 GHz, Memory 2 GB ,Hard Disk 500 MB Supported Operating Systems: all editions of Windows Server 2008 (32-bit or 64- bit); Windows Server 2008 R2: Standard, DataCenter, and Enterprise editions; Windows Server 2012: Standard, Essential, and Datacenter editions; Windows 7 (32-bit or 64-bit); Windows XP Professional (32-bit or 64-bit); Windows Vista (32-bit or 64-bit): Business, Enterprise, and Ultimate (retail licensing); and all editions of Windows 8 (32-bit or 64-bit). Pre-Requisite Software: MMC 3.0, Microsoft .NET 4.0, and Windows PowerShell 2.0 In case we are using Provisioning Services, we would require XenDesktop and, NET 3.5 SP1, and in the event that we are using Provisioning Services then we would require SCVMM 2012 SP1 and PowerShell 3.0. Supported ESD Apply only in case VDisk Update Management is used; ESD supports WSUS Server-3.0 SP2 and Microsoft System Center Configuration Management 2007 SP2, 2012, and 2012 SP1 Supported target device Supported Operating Systems: all editions of Windows 8 (32 or 64-bit); Windows 7 SP1 (32 bits or 64 bits): Enterprise, Professional, and Ultimate (Support alone in Private Mode); Windows XP Professional SP3 32-bit and Windows XP Professional SP2 64-bit; Windows Server 2008 R2 SP1: Standard, DataCenter, and Enterprise editions; Windows Server 2012: Standard, Essential, and Datacenter editions. Summary This article has thus covered the several components that make up a Citrix Provisioning Services farm and the system requirements that need to be met to run the software. Resources for Article: Further resources on this subject: Introduction to XenConvert [article] Citrix XenApp Performance Essentials [article] Getting Started with XenApp 6 [article]

This article is written by Yu-Wei, Chiu (David Chiu), the author of Machine Learning with R Cookbook. In this article, we will cover the following topics: Preparing the RHadoop environment Installing rmr2 Installing rhdfs Operating HDFS with rhdfs Implementing a word count problem with RHadoop Comparing the performance between an R MapReduce program and a standard R program Testing and debugging the rmr2 program Installing plyrmr Manipulating data with plyrmr Conducting machine learning with RHadoop Configuring RHadoop clusters on Amazon EMR (For more resources related to this topic, see here.) RHadoop is a collection of R packages that enables users to process and analyze big data with Hadoop. Before understanding how to set up RHadoop and put it in to practice, we have to know why we need to use machine learning to big-data scale. The emergence of Cloud technology has made real-time interaction between customers and businesses much more frequent; therefore, the focus of machine learning has now shifted to the development of accurate predictions for various customers. For example, businesses can provide real-time personal recommendations or online advertisements based on personal behavior via the use of a real-time prediction model. However, if the data (for example, behaviors of all online users) is too large to fit in the memory of a single machine, you have no choice but to use a supercomputer or some other scalable solution. The most popular scalable big-data solution is Hadoop, which is an open source framework able to store and perform parallel computations across clusters. As a result, you can use RHadoop, which allows R to leverage the scalability of Hadoop, helping to process and analyze big data. In RHadoop, there are five main packages, which are: rmr: This is an interface between R and Hadoop MapReduce, which calls the Hadoop streaming MapReduce API to perform MapReduce jobs across Hadoop clusters. To develop an R MapReduce program, you only need to focus on the design of the map and reduce functions, and the remaining scalability issues will be taken care of by Hadoop itself. rhdfs: This is an interface between R and HDFS, which calls the HDFS API to access the data stored in HDFS. The use of rhdfs is very similar to the use of the Hadoop shell, which allows users to manipulate HDFS easily from the R console. rhbase: This is an interface between R and HBase, which accesses Hbase and is distributed in clusters through a Thrift server. You can use rhbase to read/write data and manipulate tables stored within HBase. plyrmr: This is a higher-level abstraction of MapReduce, which allows users to perform common data manipulation in a plyr-like syntax. This package greatly lowers the learning curve of big-data manipulation. ravro: This allows users to read avro files in R, or write avro files. It allows R to exchange data with HDFS. In this article, we will start by preparing the Hadoop environment, so that you can install RHadoop. We then cover the installation of three main packages: rmr, rhdfs, and plyrmr. Next, we will introduce how to use rmr to perform MapReduce from R, operate an HDFS file through rhdfs, and perform a common data operation using plyrmr. Further, we will explore how to perform machine learning using RHadoop. Lastly, we will introduce how to deploy multiple RHadoop clusters on Amazon EC2. Preparing the RHadoop environment As RHadoop requires an R and Hadoop integrated environment, we must first prepare an environment with both R and Hadoop installed. Instead of building a new Hadoop system, we can use the Cloudera QuickStart VM (the VM is free), which contains a single node Apache Hadoop Cluster and R. In this recipe, we will demonstrate how to download the Cloudera QuickStart VM. Getting ready To use the Cloudera QuickStart VM, it is suggested that you should prepare a 64-bit guest OS with either VMWare or VirtualBox, or the KVM installed. If you choose to use VMWare, you should prepare a player compatible with WorkStation 8.x or higher: Player 4.x or higher, ESXi 5.x or higher, or Fusion 4.x or higher. Note, 4 GB of RAM is required to start VM, with an available disk space of at least 3 GB. How to do it... Perform the following steps to set up a Hadoop environment using the Cloudera QuickStart VM: Visit the Cloudera QuickStart VM download site (you may need to update the link as Cloudera upgrades its VMs , the current version of CDH is 5.3) at http://www.cloudera.com/content/cloudera/en/downloads/quickstart_vms/cdh-5-2-x.html. A screenshot of the Cloudera QuickStart VM download site Depending on the virtual machine platform installed on your OS, choose the appropriate link (you may need to update the link as Cloudera upgrades its VMs) to download the VM file: To download VMWare: You can visit https://downloads.cloudera.com/demo_vm/vmware/cloudera-quickstart-vm-5.2.0-0-vmware.7z To download KVM: You can visit https://downloads.cloudera.com/demo_vm/kvm/cloudera-quickstart-vm-5.2.0-0-kvm.7z To download VirtualBox: You can visit https://downloads.cloudera.com/demo_vm/virtualbox/cloudera-quickstart-vm-5.2.0-0-virtualbox.7z Next, you can start the QuickStart VM using the virtual machine platform installed on your OS. You should see the desktop of Centos 6.2 in a few minutes. The screenshot of Cloudera QuickStart VM. You can then open a terminal and type hadoop, which will display a list of functions that can operate a Hadoop cluster. The terminal screenshot after typing hadoop Open a terminal and type R. Access an R session and check whether version 3.1.1 is already installed in the Cloudera QuickStart VM. If you cannot find R installed in the VM, please use the following command to install R: $ yum install R R-core R-core-devel R-devel How it works... Instead of building a Hadoop system on your own, you can use the Hadoop VM application provided by Cloudera (the VM is free). The QuickStart VM runs on CentOS 6.2 with a single node Apache Hadoop cluster, Hadoop Ecosystem module, and R installed. This helps you to save time, instead of requiring you to learn how to install and use Hadoop. The QuickStart VM requires you to have a computer with a 64-bit guest OS, at least 4 GB of RAM, 3 GB of disk space, and either VMWare, VirtualBox, or KVM installed. As a result, you may not be able to use this version of VM on some computers. As an alternative, you could consider using Amazon's Elastic MapReduce instead. We will illustrate how to prepare a RHadoop environment in EMR in the last recipe of this article. Setting up the Cloudera QuickStart VM is simple. Download the VM from the download site and then open the built image with either VMWare, VirtualBox, or KVM. Once you can see the desktop of CentOS, you can then access the terminal and type hadoop to see whether Hadoop is working; then, type R to see whether R works in the QuickStart VM. See also Besides using the Cloudera QuickStart VM, you may consider using a Sandbox VM provided by Hontonworks or MapR. You can find Hontonworks Sandbox at http://hortonworks.com/products/hortonworks-sandbox/#install and mapR Sandbox at https://www.mapr.com/products/mapr-sandbox-hadoop/download. Installing rmr2 The rmr2 package allows you to perform big data processing and analysis via MapReduce on a Hadoop cluster. To perform MapReduce on a Hadoop cluster, you have to install R and rmr2 on every task node. In this recipe, we will illustrate how to install rmr2 on a single node of a Hadoop cluster. Getting ready Ensure that you have completed the previous recipe by starting the Cloudera QuickStart VM and connecting the VM to the Internet, so that you can proceed with downloading and installing the rmr2 package. How to do it... Perform the following steps to install rmr2 on the QuickStart VM: First, open the terminal within the Cloudera QuickStart VM. Use the permission of the root to enter an R session: $ sudo R You can then install dependent packages before installing rmr2: > install.packages(c("codetools", "Rcpp", "RJSONIO", "bitops", "digest", "functional", "stringr", "plyr", "reshape2", "rJava", "caTools")) Quit the R session: > q() Next, you can download rmr-3.3.0 to the QuickStart VM. You may need to update the link if Revolution Analytics upgrades the version of rmr2: $ wget --no-check-certificate https://raw.githubusercontent.com/RevolutionAnalytics/rmr2/3.3.0/build/rmr2_3.3.0.tar.gz You can then install rmr-3.3.0 to the QuickStart VM: $ sudo R CMD INSTALL rmr2_3.3.0.tar.gz Lastly, you can enter an R session and use the library function to test whether the library has been successfully installed: $ R > library(rmr2) How it works... In order to perform MapReduce on a Hadoop cluster, you have to install R and RHadoop on every task node. Here, we illustrate how to install rmr2 on a single node of a Hadoop cluster. First, open the terminal of the Cloudera QuickStart VM. Before installing rmr2, we first access an R session with root privileges and install dependent R packages. Next, after all the dependent packages are installed, quit the R session and use the wget command in the Linux shell to download rmr-3.3.0 from GitHub to the local filesystem. You can then begin the installation of rmr2. Lastly, you can access an R session and use the library function to validate whether the package has been installed. See also To see more information and read updates about RHadoop, you can refer to the RHadoop wiki page hosted on GitHub: https://github.com/RevolutionAnalytics/RHadoop/wiki Installing rhdfs The rhdfs package is the interface between R and HDFS, which allows users to access HDFS from an R console. Similar to rmr2, one should install rhdfs on every task node, so that one can access HDFS resources through R. In this recipe, we will introduce how to install rhdfs on the Cloudera QuickStart VM. Getting ready Ensure that you have completed the previous recipe by starting the Cloudera QuickStart VM and connecting the VM to the Internet, so that you can proceed with downloading and installing the rhdfs package. How to do it... Perform the following steps to install rhdfs: First, you can download rhdfs 1.0.8 from GitHub. You may need to update the link if Revolution Analytics upgrades the version of rhdfs: $wget --no-check-certificate https://raw.github.com/ RevolutionAnalytics/rhdfs/master/build/rhdfs_1.0.8.tar.gz Next, you can install rhdfs under the command-line mode: $ sudo HADOOP_CMD=/usr/bin/hadoop R CMD INSTALL rhdfs_1.0.8.tar.gz You can then set up JAVA_HOME. The configuration of JAVA_HOME depends on the installed Java version within the VM: $ sudo JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera R CMD javareconf Last, you can set up the system environment and initialize rhdfs. You may need to update the environment setup if you use a different version of QuickStart VM: $ R > Sys.setenv(HADOOP_CMD="/usr/bin/hadoop") > Sys.setenv(HADOOP_STREAMING="/usr/lib/hadoop-mapreduce/hadoop-streaming-2.5.0-cdh5.2.0.jar") > library(rhdfs) > hdfs.init() How it works... The package, rhdfs, provides functions so that users can manage HDFS using R. Similar to rmr2, you should install rhdfs on every task node, so that one can access HDFS through the R console. To install rhdfs, you should first download rhdfs from GitHub. You can then install rhdfs in R by specifying where the HADOOP_CMD is located. You must configure R with Java support through the command, javareconf. Next, you can access R and configure where HADOOP_CMD and HADOOP_STREAMING are located. Lastly, you can initialize rhdfs via the rhdfs.init function, which allows you to begin operating HDFS through rhdfs. See also To find where HADOOP_CMD is located, you can use the which hadoop command in the Linux shell. In most Hadoop systems, HADOOP_CMD is located at /usr/bin/hadoop. As for the location of HADOOP_STREAMING, the streaming JAR file is often located in /usr/lib/hadoop-mapreduce/. However, if you cannot find the directory, /usr/lib/Hadoop-mapreduce, in your Linux system, you can search the streaming JAR by using the locate command. For example: $ sudo updatedb $ locate streaming | grep jar | more Operating HDFS with rhdfs The rhdfs package is an interface between Hadoop and R, which can call an HDFS API in the backend to operate HDFS. As a result, you can easily operate HDFS from the R console through the use of the rhdfs package. In the following recipe, we will demonstrate how to use the rhdfs function to manipulate HDFS. Getting ready To proceed with this recipe, you need to have completed the previous recipe by installing rhdfs into R, and validate that you can initial HDFS via the hdfs.init function. How to do it... Perform the following steps to operate files stored on HDFS: Initialize the rhdfs package: > Sys.setenv(HADOOP_CMD="/usr/bin/hadoop") > Sys.setenv(HADOOP_STREAMING="/usr/lib/hadoop-mapreduce/hadoop-streaming-2.5.0-cdh5.2.0.jar") > library(rhdfs) > hdfs.init () You can then manipulate files stored on HDFS, as follows:     hdfs.put: Copy a file from the local filesystem to HDFS: > hdfs.put('word.txt', './')     hdfs.ls: Read the list of directory from HDFS: > hdfs.ls('./')     hdfs.copy: Copy a file from one HDFS directory to another: > hdfs.copy('word.txt', 'wordcnt.txt')     hdfs.move : Move a file from one HDFS directory to another: > hdfs.move('wordcnt.txt', './data/wordcnt.txt')     hdfs.delete: Delete an HDFS directory from R: > hdfs.delete('./data/')     hdfs.rm: Delete an HDFS directory from R: > hdfs.rm('./data/')     hdfs.get: Download a file from HDFS to a local filesystem: > hdfs.get(word.txt', '/home/cloudera/word.txt')     hdfs.rename: Rename a file stored on HDFS: hdfs.rename('./test/q1.txt','./test/test.txt')     hdfs.chmod: Change the permissions of a file or directory: > hdfs.chmod('test', permissions= '777')     hdfs.file.info: Read the meta information of the HDFS file: > hdfs.file.info('./') Also, you can write stream to the HDFS file: > f = hdfs.file("iris.txt","w") > data(iris) > hdfs.write(iris,f) > hdfs.close(f) Lastly, you can read stream from the HDFS file: > f = hdfs.file("iris.txt", "r") > dfserialized = hdfs.read(f) > df = unserialize(dfserialized) > df > hdfs.close(f) How it works... In this recipe, we demonstrate how to manipulate HDFS using the rhdfs package. Normally, you can use the Hadoop shell to manipulate HDFS, but if you would like to access HDFS from R, you can use the rhdfs package. Before you start using rhdfs, you have to initialize rhdfs with hdfs.init(). After initialization, you can operate HDFS through the functions provided in the rhdfs package. Besides manipulating HDFS files, you can exchange streams to HDFS through hdfs.read and hdfs.write. We, therefore, demonstrate how to write a data frame in R to an HDFS file, iris.txt, using hdfs.write. Lastly, you can recover the written file back to the data frame using the hdfs.read function and the unserialize function. See also To initialize rhdfs, you have to set HADOOP_CMD and HADOOP_STREAMING in the system environment. Instead of setting the configuration each time you're using rhdfs, you can put the configurations in the .rprofile file. Therefore, every time you start an R session, the configuration will be automatically loaded. Implementing a word count problem with RHadoop To demonstrate how MapReduce works, we illustrate the example of a word count, which counts the number of occurrences of each word in a given input set. In this recipe, we will demonstrate how to use rmr2 to implement a word count problem. Getting ready In this recipe, we will need an input file as our word count program input. You can download the example input from https://github.com/ywchiu/ml_R_cookbook/tree/master/CH12. How to do it... Perform the following steps to implement the word count program: First, you need to configure the system environment, and then load rmr2 and rhdfs into an R session. You may need to update the use of the JAR file if you use a different version of QuickStart VM: > Sys.setenv(HADOOP_CMD="/usr/bin/hadoop") > Sys.setenv(HADOOP_STREAMING="/usr/lib/hadoop-mapreduce/hadoop-streaming-2.5.0-cdh5.2.0.jar ") > library(rmr2) > library(rhdfs) > hdfs.init() You can then create a directory on HDFS and put the input file into the newly created directory: > hdfs.mkdir("/user/cloudera/wordcount/data") > hdfs.put("wc_input.txt", "/user/cloudera/wordcount/data") Next, you can create a map function: > map = function(.,lines) { keyval( +   unlist( +     strsplit( +       x = lines, +       split = " +")), +   1)} Create a reduce function: > reduce = function(word, counts) +   keyval(word, sum(counts)) + } Call the MapReduce program to count the words within a document: > hdfs.root = 'wordcount' > hdfs.data = file.path(hdfs.root, 'data') > hdfs.out = file.path(hdfs.root, 'out') > wordcount = function (input, output=NULL) { + mapreduce(input=input, output=output, input.format="text", map=map, + reduce=reduce) + } > out = wordcount(hdfs.data, hdfs.out) Lastly, you can retrieve the top 10 occurring words within the document: > results = from.dfs(out) > results$key[order(results$val, decreasing = TRUE)][1:10] How it works... In this recipe, we demonstrate how to implement a word count using the rmr2 package. First, we need to configure the system environment and load rhdfs and rmr2 into R. Then, we specify the input of our word count program from the local filesystem into the HDFS directory, /user/cloudera/wordcount/data, via the hdfs.put function. Next, we begin implementing the MapReduce program. Normally, we can divide the MapReduce program into the map and reduce functions. In the map function, we first use the strsplit function to split each line into words. Then, as the strsplit function returns a list of words, we can use the unlist function to character vectors. Lastly, we can return key-value pairs with each word as a key and the value as one. As the reduce function receives the key-value pair generated from the map function, the reduce function sums the count and returns the number of occurrences of each word (or key). After we have implemented the map and reduce functions, we can submit our job via the mapreduce function. Normally, the mapreduce function requires four inputs, which are the HDFS input path, the HDFS output path, the map function, and the reduce function. In this case, we specify the input as wordcount/data, output as wordcount/out, mapfunction as map, reduce function as reduce, and wrap the mapreduce call in function, wordcount. Lastly, we call the function, wordcount and store the output path in the variable, out. We can use the from.dfs function to load the HDFS data into the results variable, which contains the mapping of words and number of occurrences. We can then generate the top 10 occurring words from the results variable. See also In this recipe, we demonstrate how to write an R MapReduce program to solve a word count problem. However, if you are interested in how to write a native Java MapReduce program, you can refer to http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html. Comparing the performance between an R MapReduce program and a standard R program Those not familiar with how Hadoop works may often see Hadoop as a remedy for big data processing. Some might believe that Hadoop can return the processed results for any size of data within a few milliseconds. In this recipe, we will compare the performance between an R MapReduce program and a standard R program to demonstrate that Hadoop does not perform as quickly as some may believe. Getting ready In this recipe, you should have completed the previous recipe by installing rmr2 into the R environment. How to do it... Perform the following steps to compare the performance of a standard R program and an R MapReduce program: First, you can implement a standard R program to have all numbers squared: > a.time = proc.time() > small.ints2=1:100000 > result.normal = sapply(small.ints2, function(x) x^2) > proc.time() - a.time To compare the performance, you can implement an R MapReduce program to have all numbers squared: > b.time = proc.time() > small.ints= to.dfs(1:100000) > result = mapreduce(input = small.ints, map = function(k,v)       cbind(v,v^2)) > proc.time() - b.time How it works... In this recipe, we implement two programs to square all the numbers. In the first program, we use a standard R function, sapply, to square the sequence from 1 to 100,000. To record the program execution time, we first record the processing time before the execution in a.time, and then subtract a.time from the current processing time after the execution. Normally, the execution takes no more than 10 seconds. In the second program, we use the rmr2 package to implement a program in the R MapReduce version. In this program, we also record the execution time. Normally, this program takes a few minutes to complete a task. The performance comparison shows that a standard R program outperforms the MapReduce program when processing small amounts of data. This is because a Hadoop system often requires time to spawn daemons, job coordination between daemons, and fetching data from data nodes. Therefore, a MapReduce program often takes a few minutes to a couple of hours to finish the execution. As a result, if you can fit your data in the memory, you should write a standard R program to solve the problem. Otherwise, if the data is too large to fit in the memory, you can implement a MapReduce solution. See also In order to check whether a job will run smoothly and efficiently in Hadoop, you can run a MapReduce benchmark, MRBench, to evaluate the performance of the job: $ hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-test.jar mrbench -numRuns 50 Testing and debugging the rmr2 program Since running a MapReduce program will require a considerable amount of time, varying from a few minutes to several hours, testing and debugging become very important. In this recipe, we will illustrate some techniques you can use to troubleshoot an R MapReduce program. Getting ready In this recipe, you should have completed the previous recipe by installing rmr2 into an R environment. How to do it... Perform the following steps to test and debug an R MapReduce program: First, you can configure the backend as local in rmr.options: > rmr.options(backend = 'local') Again, you can execute the number squared MapReduce program mentioned in the previous recipe: > b.time = proc.time() > small.ints= to.dfs(1:100000) > result = mapreduce(input = small.ints, map = function(k,v)       cbind(v,v^2)) > proc.time() - b.time In addition to this, if you want to print the structure information of any variable in the MapReduce program, you can use the rmr.str function: > out = mapreduce(to.dfs(1), map = function(k, v) rmr.str(v)) Dotted pair list of 14 $ : language mapreduce(to.dfs(1), map = function(k, v) rmr.str(v)) $ : language mr(map = map, reduce = reduce, combine = combine, vectorized.reduce, in.folder = if (is.list(input)) {     lapply(input, to.dfs.path) ...< $ : language c.keyval(do.call(c, lapply(in.folder, function(fname) {     kv = get.data(fname) ... $ : language do.call(c, lapply(in.folder, function(fname) {     kv = get.data(fname) ... $ : language lapply(in.folder, function(fname) {     kv = get.data(fname) ... $ : language FUN("/tmp/Rtmp813BFJ/file25af6e85cfde"[[1L]], ...) $ : language unname(tapply(1:lkv, ceiling((1:lkv)/(lkv/(object.size(kv)/10^6))), function(r) {     kvr = slice.keyval(kv, r) ... $ : language tapply(1:lkv, ceiling((1:lkv)/(lkv/(object.size(kv)/10^6))), function(r) {     kvr = slice.keyval(kv, r) ... $ : language lapply(X = split(X, group), FUN = FUN, ...) $ : language FUN(X[[1L]], ...) $ : language as.keyval(map(keys(kvr), values(kvr))) $ : language is.keyval(x) $ : language map(keys(kvr), values(kvr)) $ :length 2 rmr.str(v) ..- attr(*, "srcref")=Class 'srcref' atomic [1:8] 1 34 1 58 34 58 1 1 .. .. ..- attr(*, "srcfile")=Classes 'srcfilecopy', 'srcfile' <environment: 0x3f984f0> v num 1 How it works... In this recipe, we introduced some debugging and testing techniques you can use while implementing the MapReduce program. First, we introduced the technique to test a MapReduce program in a local mode. If you would like to run the MapReduce program in a pseudo distributed or fully distributed mode, it would take you a few minutes to several hours to complete the task, which would involve a lot of wastage of time while troubleshooting your MapReduce program. Therefore, you can set the backend to the local mode in rmr.options so that the program will be executed in the local mode, which takes lesser time to execute. Another debugging technique is to list the content of the variable within the map or reduce function. In an R program, you can use the str function to display the compact structure of a single variable. In rmr2, the package also provides a function named rmr.str, which allows you to print out the content of a single variable onto the console. In this example, we use rmr.str to print the content of variables within a MapReduce program. See also For those who are interested in the option settings for the rmr2 package, you can refer to the help document of rmr.options: > help(rmr.options) Installing plyrmr The plyrmr package provides common operations (as found in plyr or reshape2) for users to easily perform data manipulation through the MapReduce framework. In this recipe, we will introduce how to install plyrmr on the Hadoop system. Getting ready Ensure that you have completed the previous recipe by starting the Cloudera QuickStart VM and connecting the VM to the Internet. Also, you need to have the rmr2 package installed beforehand. How to do it... Perform the following steps to install plyrmr on the Hadoop system: First, you should install libxml2-devel and curl-devel in the Linux shell: $ yum install libxml2-devel $ sudo yum install curl-devel You can then access R and install the dependent packages: $ sudo R > Install.packages(c(" Rcurl", "httr"), dependencies = TRUE > Install.packages("devtools", dependencies = TRUE) > library(devtools) > install_github("pryr", "hadley") > install.packages(c(" R.methodsS3", "hydroPSO"), dependencies = TRUE > q() Next, you can download plyrmr 0.5.0 and install it on Hadoop VM. You may need to update the link if Revolution Analytics upgrades the version of plyrmr: $ wget -no-check-certificate https://raw.github.com/RevolutionAnalytics/plyrmr/master/build/plyrmr_0.5.0.tar.gz $ sudo R CMD INSTALL plyrmr_0.5.0.tar.gz Lastly, validate the installation: $ R > library(plyrmr) How it works... Besides writing an R MapReduce program using the rmr2 package, you can use the plyrmr to manipulate data. The plyrmr package is similar to hive and pig in the Hadoop ecosystem, which is the abstraction of the MapReduce program. Therefore, we can implement an R MapReduce program in plyr style instead of implementing the map f and reduce functions. To install plyrmr, first install the package of libxml2-devel and curl-devel, using the yum install command. Then, access R and install the dependent packages. Lastly, download the file from GitHub and install plyrmr in R. See also To read more information about plyrmr, you can use the help function to refer to the following document: > help(package=plyrmr) Manipulating data with plyrmr While writing a MapReduce program with rmr2 is much easier than writing a native Java version, it is still hard for nondevelopers to write a MapReduce program. Therefore, you can use plyrmr, a high-level abstraction of the MapReduce program, so that you can use plyr-like operations to manipulate big data. In this recipe, we will introduce some operations you can use to manipulate data. Getting ready In this recipe, you should have completed the previous recipes by installing plyrmr and rmr2 in R. How to do it... Perform the following steps to manipulate data with plyrmr: First, you need to load both plyrmr and rmr2 into R: > library(rmr2) > library(plyrmr) You can then set the execution mode to the local mode: > plyrmr.options(backend="local") Next, load the Titanic dataset into R: > data(Titanic) > titanic = data.frame(Titanic) Begin the operation by filtering the data: > where( +   Titanic, + Freq >=100) You can also use a pipe operator to filter the data: > titanic %|% where(Freq >=100) Put the Titanic data into HDFS and load the path of the data to the variable, tidata: > tidata = to.dfs(data.frame(Titanic), output = '/tmp/titanic') > tidata Next, you can generate a summation of the frequency from the Titanic data: > input(tidata) %|% transmute(sum(Freq)) You can also group the frequency by sex: > input(tidata) %|% group(Sex) %|% transmute(sum(Freq)) You can then sample 10 records out of the population: > sample(input(tidata), n=10) In addition to this, you can use plyrmr to join two datasets: > convert_tb = data.frame(Label=c("No","Yes"), Symbol=c(0,1)) ctb = to.dfs(convert_tb, output = 'convert') > as.data.frame(plyrmr::merge(input(tidata), input(ctb), by.x="Survived", by.y="Label")) > file.remove('convert') How it works... In this recipe, we introduce how to use plyrmr to manipulate data. First, we need to load the plyrmr package into R. Then, similar to rmr2, you have to set the backend option of plyrmr as the local mode. Otherwise, you will have to wait anywhere between a few minutes to several hours if plyrmr is running on Hadoop mode (the default setting). Next, we can begin the data manipulation with data filtering. You can choose to call the function nested inside the other function call in step 4. On the other hand, you can use the pipe operator, %|%, to chain multiple operations. Therefore, we can filter data similar to step 4, using pipe operators in step 5. Next, you can input the dataset into either the HDFS or local filesystem, using to.dfs in accordance with the current running mode. The function will generate the path of the dataset and save it in the variable, tidata. By knowing the path, you can access the data using the input function. Next, we illustrate how to generate a summation of the frequency from the Titanic dataset with the transmute and sum functions. Also, plyrmr allows users to sum up the frequency by gender. Additionally, in order to sample data from a population, you can also use the sample function to select 10 records out of the Titanic dataset. Lastly, we demonstrate how to join two datasets using the merge function from plyrmr. See also Here we list some functions that can be used to manipulate data with plyrmr. You may refer to the help function for further details on their usage and functionalities: Data manipulation: bind.cols: This adds new columns select: This is used to select columns where: This is used to select rows transmute: This uses all of the above plus their summaries From reshape2: melt and dcast: It converts long and wide data frames Summary: count quantile sample Extract: top.k bottom.k Conducting machine learning with RHadoop At this point, some may believe that the use of RHadoop can easily solve machine learning problems of big data via numerous existing machine learning packages. However, you cannot use most of these to solve machine learning problems as they cannot be executed in the MapReduce mode. In the following recipe, we will demonstrate how to implement a MapReduce version of linear regression and compare this version with the one using the lm function. Getting ready In this recipe, you should have completed the previous recipe by installing rmr2 into the R environment. How to do it... Perform the following steps to implement a linear regression in MapReduce: First, load the cats dataset from the MASS package: > library(MASS) > data(cats) > X = matrix(cats$Bwt) > y = matrix(cats$Hwt) You can then generate a linear regression model by calling the lm function: > model = lm(y~X) > summary(model)   Call: lm(formula = y ~ X)   Residuals:    Min    1Q Median     3Q     Max -3.5694 -0.9634 -0.0921 1.0426 5.1238   Coefficients:            Estimate Std. Error t value Pr(>|t|)   (Intercept) -0.3567     0.6923 -0.515   0.607   X             4.0341     0.2503 16.119   <2e-16 *** --- Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1   Residual standard error: 1.452 on 142 degrees of freedom Multiple R-squared: 0.6466, Adjusted R-squared: 0.6441 F-statistic: 259.8 on 1 and 142 DF, p-value: < 2.2e-16 You can now make a regression plot with the given data points and model: > plot(y~X) > abline(model, col="red") Linear regression plot of cats dataset Load rmr2 into R: > Sys.setenv(HADOOP_CMD="/usr/bin/hadoop") > Sys.setenv(HADOOP_STREAMING="/usr/lib/hadoop-mapreduce/hadoop-> streaming-2.5.0-cdh5.2.0.jar") > library(rmr2) > rmr.options(backend="local") You can then set up X and y values: > X = matrix(cats$Bwt) > X.index = to.dfs(cbind(1:nrow(X), X)) > y = as.matrix(cats$Hwt) Make a Sum function to sum up the values: > Sum = +   function(., YY) +     keyval(1, list(Reduce('+', YY))) Compute Xtx in MapReduce, Job1: > XtX = +   values( +     from.dfs( +       mapreduce( +         input = X.index, +         map = +           function(., Xi) { +             Xi = Xi[,-1] +              keyval(1, list(t(Xi) %*% Xi))}, +         reduce = Sum, +         combine = TRUE)))[[1]] You can then compute Xty in MapReduce, Job2: Xty = +   values( +     from.dfs( +       mapreduce( +         input = X.index, +         map = function(., Xi) { +           yi = y[Xi[,1],] +           Xi = Xi[,-1] +           keyval(1, list(t(Xi) %*% yi))}, +         reduce = Sum, +         combine = TRUE)))[[1]] Lastly, you can derive the coefficient from XtX and Xty: > solve(XtX, Xty)          [,1] [1,] 3.907113 How it works... In this recipe, we demonstrate how to implement linear logistic regression in a MapReduce fashion in R. Before we start the implementation, we review how traditional linear models work. We first retrieve the cats dataset from the MASS package. We then load X as the body weight (Bwt) and y as the heart weight (Hwt). Next, we begin to fit the data into a linear regression model using the lm function. We can then compute the fitted model and obtain the summary of the model. The summary shows that the coefficient is 4.0341 and the intercept is -0.3567. Furthermore, we draw a scatter plot in accordance with the given data points and then draw a regression line on the plot. As we cannot perform linear regression using the lm function in the MapReduce form, we have to rewrite the regression model in a MapReduce fashion. Here, we would like to implement a MapReduce version of linear regression in three steps, which are: calculate the Xtx value with the MapReduce, job1, calculate the Xty value with MapReduce, job2, and then derive the coefficient value: In the first step, we pass the matrix, X, as the input to the map function. The map function then calculates the cross product of the transposed matrix, X, and, X. The reduce function then performs the sum operation defined in the previous section. In the second step, the procedure of calculating Xty is similar to calculating XtX. The procedure calculates the cross product of the transposed matrix, X, and, y. The reduce function then performs the sum operation. Lastly, we use the solve function to derive the coefficient, which is 3.907113. As the results show, the coefficients computed by lm and MapReduce differ slightly. Generally speaking, the coefficient computed by the lm model is more accurate than the one calculated by MapReduce. However, if your data is too large to fit in the memory, you have no choice but to implement linear regression in the MapReduce version. See also You can access more information on machine learning algorithms at: https://github.com/RevolutionAnalytics/rmr2/tree/master/pkg/tests Configuring RHadoop clusters on Amazon EMR Until now, we have only demonstrated how to run a RHadoop program in a single Hadoop node. In order to test our RHadoop program on a multi-node cluster, the only thing you need to do is to install RHadoop on all the task nodes (nodes with either task tracker for mapreduce version 1 or node manager for map reduce version 2) of Hadoop clusters. However, the deployment and installation is time consuming. On the other hand, you can choose to deploy your RHadoop program on Amazon EMR, so that you can deploy multi-node clusters and RHadoop on every task node in only a few minutes. In the following recipe, we will demonstrate how to configure RHadoop cluster on an Amazon EMR service. Getting ready In this recipe, you must register and create an account on AWS, and you also must know how to generate a EC2 key-pair before using Amazon EMR. For those who seek more information on how to start using AWS, please refer to the tutorial provided by Amazon at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html. How to do it... Perform the following steps to configure RHadoop on Amazon EMR: First, you can access the console of the Amazon Web Service (refer to https://us-west-2.console.aws.amazon.com/console/) and find EMR in the analytics section. Then, click on EMR. Access EMR service from AWS console. You should find yourself in the cluster list of the EMR dashboard (refer to https://us-west-2.console.aws.amazon.com/elasticmapreduce/home?region=us-west-2#cluster-list::); click on Create cluster. Cluster list of EMR Then, you should find yourself on the Create Cluster page (refer to https://us-west-2.console.aws.amazon.com/elasticmapreduce/home?region=us-west-2#create-cluster:). Next, you should specify Cluster name and Log folder S3 location in the cluster configuration. Cluster configuration in the create cluster page You can then configure the Hadoop distribution on Software Configuration. Configure the software and applications Next, you can configure the number of nodes within the Hadoop cluster. Configure the hardware within Hadoop cluster You can then specify the EC2 key-pair for the master node login. Security and access to the master node of the EMR cluster To set up RHadoop, one has to perform bootstrap actions to install RHadoop on every task node. Please write a file named bootstrapRHadoop.sh, and insert the following lines within the file: echo 'install.packages(c("codetools", "Rcpp", "RJSONIO", "bitops", "digest", "functional", "stringr", "plyr", "reshape2", "rJava", "caTools"), repos="http://cran.us.r-project.org")' > /home/hadoop/installPackage.R sudo Rscript /home/hadoop/installPackage.R wget --no-check-certificate https://raw.githubusercontent.com/RevolutionAnalytics/rmr2/master/build/rmr2_3.3.0.tar.gz sudo R CMD INSTALL rmr2_3.3.0.tar.gz wget --no-check-certificate https://raw.github.com/RevolutionAnalytics/rhdfs/master/build/rhdfs_1.0.8.tar.gz sudo HADOOP_CMD=/home/hadoop/bin/hadoop R CMD INSTALL rhdfs_1.0.8.tar.gz You should upload bootstrapRHadoop.sh to S3. You now need to add the bootstrap action with Custom action, and add s3://<location>/bootstrapRHadoop.sh within the S3 location. Set up the bootstrap action Next, you can click on Create cluster to launch the Hadoop cluster. Create the cluster Lastly, you should see the master public DNS when the cluster is ready. You can now access the terminal of the master node with your EC2-key pair: A screenshot of the created cluster How it works... In this recipe, we demonstrate how to set up RHadoop on Amazon EMR. The benefit of this is that you can quickly create a scalable, on demand Hadoop with just a few clicks within a few minutes. This helps save you time from building and deploying a Hadoop application. However, you have to pay for the number of running hours for each instance. Before using Amazon EMR, you should create an AWS account and know how to set up the EC2 key-pair and the S3. You can then start installing RHadoop on Amazon EMR. In the first step, access the EMR cluster list and click on Create cluster. You can see a list of configurations on the Create cluster page. You should then set up the cluster name and log folder in the S3 location in the cluster configuration. Next, you can set up the software configuration and choose the Hadoop distribution you would like to install. Amazon provides both its own distribution and the MapR distribution. Normally, you would skip this section unless you have concerns about the default Hadoop distribution. You can then configure the hardware by specifying the master, core, and task node. By default, there is only one master node, and two core nodes. You can add more core and task nodes if you like. You should then set up the key-pair to login to the master node. You should next make a file containing all the start scripts named bootstrapRHadoop.sh. After the file is created, you should save the file in the S3 storage. You can then specify custom action in Bootstrap Action with bootstrapRHadoop.sh as the Bootstrap script. Lastly, you can click on Create cluster and wait until the cluster is ready. Once the cluster is ready, one can see the master public DNS and can use the EC2 key-pair to access the terminal of the master node. Beware! Terminate the running instance if you do not want to continue using the EMR service. Otherwise, you will be charged per instance for every hour you use. See also Google also provides its own cloud solution, the Google compute engine. For those who would like to know more, please refer to https://cloud.google.com/compute/. Summary In this article, we started by preparing the Hadoop environment, so that you can install RHadoop. We then covered the installation of three main packages: rmr, rhdfs, and plyrmr. Next, we introduced how to use rmr to perform MapReduce from R, operate an HDFS file through rhdfs, and perform a common data operation using plyrmr. Further, we explored how to perform machine learning using RHadoop. Lastly, we introduced how to deploy multiple RHadoop clusters on Amazon EC2. Resources for Article: Further resources on this subject: Warming Up [article] Derivatives Pricing [article] Using R for Statistics, Research, and Graphics [article]

Configuring Open MQ using standalone tools Open MQ is shipped with a collection of utilities that can be used to configure and administer its runtime environment and resources. In this section, we provide a brief introduction to these utilities. For a detailed description of these utilities, you can refer to the Open MQ document set, located at http://docs.sun.com/app/docs/coll/1307.6. Starting and stopping the Open MQ broker The imqbrokerd utility is used to start a message broker. To start the default broker configured for GlassFish, open a command-line terminal, and enter the following commands: # cd $AS_INSTALL/imq/bin# ./imqbrokerd -tty You should see an output describing the information about the broker, and the confirmation of the successful start of the broker. If we do not specify a broker name, the imqbrokerd command will try to start the default broker named imqbroker. We can also supply a name <broker_name>. If the broker named <brokername> is already created, then the imqbrokerd command starts it; otherwise, a new broker of this name is created and then started. To shutdown the message broker, enter Ctrl+C in the terminal where imqbrokerd is running. Refer to the Open MQ documentation for information on how to run Open MQ brokers as a service. Administering the Open MQ Broker using imqcmd The main CLI utility we use to manage the Open MQ is imqcmd. It supports a long list of commands that can be used to create, destroy, modify, and display the information of a resource, such as a broker or a physical destination, and so on. To show the commands and options supported, enter the following commands in a terminal: # cd $AS_INSTALL/imq/bin# ./imqcmd -h As the first example, we can use the following command to display the status information of the currently running broker: # cd $AS_INSTALL/imq/bin# ./imqcmd query bkr -u admin Once you are prompted to enter the password, enter the default password admin. You should see an output describing the information regarding the currently running broker. We can store the password in a password file and use the -passfile option to point to this file. As another example, the following command queries the physical queue destination named PSQueueDest, and displays its information in the command-line terminal. # cd $AS_INSTALL/imq/bin# ./imqcmd query dst -t q -n PSQueueDest -u admin In this example, the option –t specifies the type of the destination as a topic, and the option –n PSQueueDest specifies the name of the physical destination. Refer to the help information and the MQ Administration Guide for a detailed description of the imqcmd utility. Using the imqadmin administration console The imqadmin utility is the primary GUI-based administration console of the Open MQ. It supports more of the functions of the imqcmd utility. Besides, it can also manage and confi gure the administration objects used by applications, such as specifying a naming service provider, and registering connection factories and destination resources in the naming service. To start the administration console, open a command-line terminal, and type the following command: # cd $AS_INSTALL/imq/bin# ./imqadmin You should see the administrator console window displayed. The administration console's navigation pane contains two top-level nodes, Brokers and Object Stores. The Brokers node organizes all the registered Open MQ brokers being managed by the administration console. The Object Stores node organizes the naming service providers to which the JMS administration objects (connection factories and destination resources) are registered. The following steps walk you through the typical process of adding a broker, and managing the broker resources. From the navigation pane of the administrator console, right click the Brokers node, and choose Add Broker. You should see a dialog box displayed. Fill out the Add Broker dialog according to the following screenshot, enter admin in the password field, and click OK. Now you should see the broker PSBroker is listed under the Brokers node with a red X on its icon, which indicates that the broker is not currently connected to the administrator console yet. Adding a broker does not create or start a new physical broker. It merely registers an existing broker to the administration console, and once we connect the administration console, we will be able to manage the resources of the broker. From the navigation pane, right click PSBroker under the Brokers node, and choose Connect to Broker. Once you connect to the broker, you should see the red X disappeared on the Brokers node. From the navigation pane, click Services under PSBroker. In the result pane, you should see all the available connection services listed for the broker, as shown in the following screenshot. From the navigation pane, right click Destinations under PSBroker, and select Add Broker Destination. You should see a dialog displayed. Enter PSQueueDest in the Destination Name field, select Queue radio button if not already selected. Keep all the other settings unchanged, and click OK to add the physical destination. From the navigation pane, click Destinations under PSBroker, the newly added PSQueueDest should appear. With physical destinations created, we can go through the following steps to configure the connection factory and destination resources in the Object Stores: Right click the Object Stores node in the navigation pane, and choose Add Object Store. Enter the following information in the dialog, and click OK to add the Object Store. Enter PSStore in the Object Store Label field. From the Name pull-down menu, select java.naming.factory.initial, and enter com.sun.jndi.fscontext.RefFSContextFactory in the value field. Click the Add button. These steps set the JNDI service provider you will use, to a filesystem based object store. From the Name pull-down menu, select java.naming.provider.url, and enter file:///tmp in the value field. Click the Add button. These steps set the exact location of the object store to the directory /tmp. In the navigation pane, you will see the object store PSStore you just added is listed under the Object Stores node with a red X on its icon, which indicates that the object store is not currently connected to the administrator console yet. From the navigation pane, right click PSStore under the Object Stores node, and choose Connect to Object Store. Once you connect to the object store, you should see that the red X has disappeared on the Object Stores node. From the navigation pane of the Open MQ administrator console, right click Destinations under PSStore node, and select Add Destination Object. You should see a dialog box displayed. Enter PSQueue in the Lookup Name field, select Queue from the Destination Type radio button. Enter PSQueueDest in the Destination Name field, and click OK to add the queue destination, as shown in the following screenshot. Now that we have seen how Open MQ is integrated with GlassFish, let's look at another very popular open source JMS implementation, ActiveMQ. Configuring ActiveMQ for GlassFish ActiveMQ is a very popular open source provider that is fully JMS 1.1 compliant. Besides, Active MQ has several very appealing features, such as cross language support for client development, high availability support, and support for REST API, and so on. For a more detailed description of the ActiveMQ product, visit the ActiveMQ project website hosted by Apache: http://activemq.apache.org. GlassFish is shipped with a generic JMS JCA resource adapter. In this section, we use ActiveMQ as an example to demonstrate how this generic resource adapter allows GlassFish to integrate with third-party JMS providers. Many JMS providers, including ActiveMQ includes its own JCA resource adapters for integrating with a Java EE application server. In this book, we do not cover this topic. You can refer to the JMS provider's documentation to enable this. For example, integrating ActiveMQ into GlassFish using ActiveMQ's resource adapter is actually quite similar to using the GlassFish resource adapter. Installing and configuring ActiveMQ The first step of integrating with ActiveMQ is to install it. Installing ActiveMQ is very straight forward. Simply download the latest ActiveMQ binary distribution (ZIP format for Window, and GZIP for UNIX/Linux/Mac OS X) from http://activemq.apache.org, and extract it to a target directory. For example, /opt. Next, let's start the ActiveMQ broker with the following commands: # cd /opt/apache-activemq-5.3.0/bin# ./activemq-admin start There are a variety of ways to configure ActiveMQ. The most comprehensive mechanism is to work with XML-based configuration files. This approach can be used to define all aspects of a message broker and its components, such as message destination. For a detailed discussion, please refer to the ActiveMQ documentation for more information on this. ActiveMQ also provides a simple web-based user interface for basic administrative tasks, such as creating/deleting message destinations (both topics and queues), sending messages to a destination for testing, and browsing messages in a destination, and so on. Now let's use the administrative user interface to configure a queue. To do this, complete the following steps: Open a browser, and access the ActiveMQ admin URL http://localhost:8161/admin. The ActiveMQ administration page appears. Click Queues. The browser lists the message queues configured for the ActiveMQ broker. By default, a queue named example.A is configured, as shown in the following screenshot. In the next section, we show you how to integrate this sample queue into GlassFish.  

(For more resources related to this topic, see here.) Setting up the OpenGL v3.3 core profile on Visual Studio 2010 using the GLEW and freeglut libraries We will start with a very basic example in which we will set up the modern OpenGL v3.3 core profile. This example will simply create a blank window and clear the window with red color. OpenGL or any other graphics API for that matter requires a window to display graphics in. This is carried out through platform specific codes. Previously, the GLUT library was invented to provide windowing functionality in a platform independent manner. However, this library was not maintained with each new OpenGL release. Fortunately, another independent project, freeglut, followed in the GLUT footsteps by providing similar (and in some cases better) windowing support in a platform independent way. In addition, it also helps with the creation of the OpenGL core/compatibility profile contexts. The latest version of freeglut may be downloaded from http://freeglut.sourceforge.net. The version used in the source code accompanying this book is v2.8.0. After downloading the freeglut library, you will have to compile it to generate the libs/dlls. The extension mechanism provided by OpenGL still exists. To aid with getting the appropriate function pointers, the GLEW library is used. The latest version can be downloaded from http://glew.sourceforge.net. The version of GLEW used in the source code accompanying this book is v1.9.0. If the source release is downloaded, you will have to build GLEW first to generate the libs and dlls on your platform. You may also download the pre-built binaries. Prior to OpenGL v3.0, the OpenGL API provided support for matrices by providing specific matrix stacks such as the modelview, projection, and texture matrix stacks. In addition, transformation functions such as translate, rotate, and scale, as well as projection functions were also provided. Moreover, immediate mode rendering was supported, allowing application programmers to directly push the vertex information to the hardware. In OpenGL v3.0 and above, all of these functionalities are removed from the core profile, whereas for backward compatibility they are retained in the compatibility profile. If we use the core profile (which is the recommended approach), it is our responsibility to implement all of these functionalities including all matrix handling and transformations. Fortunately, a library called glm exists that provides math related classes such as vectors and matrices. It also provides additional convenience functions and classes. For all of the demos in this book, we will use the glm library. Since this is a headers only library, there are no linker libraries for glm. The latest version of glm can be downloaded from http://glm.g-truc.net. The version used for the source code in this book is v0.9.4.0. There are several image formats available. It is not a trivial task to write an image loader for such a large number of image formats. Fortunately, there are several image loading libraries that make image loading a trivial task. In addition, they provide support for both loading as well as saving of images into various formats. One such library is the SOIL image loading library. The latest version of SOIL can be downloaded from http://www.lonesock.net/soil.html. Once we have downloaded the SOIL library, we extract the file to a location on the hard disk. Next, we set up the include and library paths in the Visual Studio environment. The include path on my development machine is D:LibrariessoilSimple OpenGL Image Librarysrc whereas, the library path is set to D:LibrariessoilSimple OpenGL Image LibrarylibVC10_Debug. Of course, the path for your system will be different than mine but these are the folders that the directories should point to. These steps will help us to set up our development environment. For all of the recipes in this book, Visual Studio 2010 Professional version is used. Readers may also use the free express edition or any other version of Visual Studio (for example, Ultimate/Enterprise). Since there are a myriad of development environments, to make it easier for users on other platforms, we have provided premake script files as well. The code for this recipe is in the Chapter1/GettingStarted directory. How to do it... Let us setup the development environment using the following steps: After downloading the required libraries, we set up the Visual Studio 2010 environment settings. We first create a new Win32 Console Application project as shown in the preceding screenshot. We set up an empty Win32 project as shown in the following screenshot: Next, we set up the include and library paths for the project by going into the Project menu and selecting project Properties . This opens a new dialog box. In the left pane, click on the Configuration Properties option and then on VC++ Directories . In the right pane, in the Include Directories field, add the GLEW and freeglut subfolder paths. Similarly, in the Library Directories , add the path to the lib subfolder of GLEW and freeglut libraries as shown in the following screenshot: Next, we add a new .cpp file to the project and name it main.cpp. This is the main source file of our project. You may also browse through Chapter1/ GettingStarted/GettingStarted/main.cpp which does all this setup already. Let us skim through the Chapter1/ GettingStarted/GettingStarted/main.cpp file piece by piece. #include <GL/glew.h> #include <GL/freeglut.h> #include <iostream> These lines are the include files that we will add to all of our projects. The first is the GLEW header, the second is the freeglut header, and the final include is the standard input/output header. In Visual Studio, we can add the required linker libraries in two ways. The first way is through the Visual Studio environment (by going to the Properties menu item in the Project menu). This opens the project's property pages. In the configuration properties tree, we collapse the Linker subtree and click on the Input item. The first field in the right pane is Additional Dependencies. We can add the linker library in this field as shown in the following screenshot: The second way is to add the glew32.lib file to the linker settings programmatically. This can be achieved by adding the following pragma: #pragma comment(lib, "glew32.lib") The next line is the using directive to enable access to the functions in the std namespace. This is not mandatory but we include this here so that we do not have to prefix std:: to any standard library function from the iostream header file. using namespace std; The next lines define the width and height constants which will be the screen resolution for the window. After these declarations, there are five function definitions . The OnInit() function is used for initializing any OpenGL state or object, OnShutdown() is used to delete an OpenGL object, OnResize() is used to handle the resize event, OnRender() helps to handle the paint event, and main() is the entry point of the application. We start with the definition of the main() function. const int WIDTH = 1280; const int HEIGHT = 960; int main(int argc, char** argv) { glutInit(&argc, argv); glutInitDisplayMode(GLUT_DEPTH | GLUT_DOUBLE | GLUT_RGBA); glutInitContextVersion (3, 3); glutInitContextFlags (GLUT_CORE_PROFILE | GLUT_DEBUG); glutInitContextProfile(GLUT_FORWARD_COMPATIBLE); glutInitWindowSize(WIDTH, HEIGHT); The first line glutInit initializes the GLUT environment. We pass the command line arguments to this function from our entry point. Next, we set up the display mode for our application. In this case, we request the GLUT framework to provide support for a depth buffer, double buffering (that is a front and a back buffer for smooth, flicker-free rendering), and the format of the frame buffer to be RGBA (that is with red, green, blue, and alpha channels). Next, we set the required OpenGL context version we desire by using the glutInitContextVersion. The first parameter is the major version of OpenGL and the second parameter is the minor version of OpenGL. For example, if we want to create an OpenGL v4.3 context, we will call glutInitContextVersion (4, 3). Next, the context flags are specified: glutInitContextFlags (GLUT_CORE_PROFILE | GLUT_DEBUG); glutInitContextProfile(GLUT_FORWARD_COMPATIBLE); In OpenGL v4.3, we can register a callback when any OpenGL related error occurs. Passing GLUT_DEBUG to the glutInitContextFlags functions creates the OpenGL context in debug mode which is needed for the debug message callback. For any version of OpenGL including OpenGL v3.3 and above, there are two profiles available: the core profile (which is a pure shader based profile without support for OpenGL fixed functionality) and the compatibility profile (which supports the OpenGL fixed functionality). All of the matrix stack functionality glMatrixMode(*), glTranslate*, glRotate*, glScale*, and so on, and immediate mode calls such as glVertex*, glTexCoord*, and glNormal* of legacy OpenGL, are retained in the compatibility profile. However, they are removed from the core profile. In our case, we will request a forward compatible core profile which means that we will not have any fixed function OpenGL functionality available. Next, we set the screen size and create the window: glutInitWindowSize(WIDTH, HEIGHT); glutCreateWindow("Getting started with OpenGL 3.3"); Next, we initialize the GLEW library. It is important to initialize the GLEW library after the OpenGL context has been created. If the function returns GLEW_OK the function succeeds, otherwise the GLEW initialization fails. glewExperimental = GL_TRUE; GLenum err = glewInit(); if (GLEW_OK != err){ cerr<<"Error: "<<glewGetErrorString(err)<<endl; } else { if (GLEW_VERSION_3_3) { cout<<"Driver supports OpenGL 3.3nDetails:"<<endl; } } cout<<"tUsing glew "<<glewGetString(GLEW_VERSION)<<endl; cout<<"tVendor: "<<glGetString (GL_VENDOR)<<endl; cout<<"tRenderer: "<<glGetString (GL_RENDERER)<<endl; cout<<"tVersion: "<<glGetString (GL_VERSION)<<endl; cout<<"tGLSL: "<<glGetString(GL_SHADING_LANGUAGE_VERSION)<<endl; The glewExperimental global switch allows the GLEW library to report an extension if it is supported by the hardware but is unsupported by the experimental or pre-release drivers. After the function is initialized, the GLEW diagnostic information such as the GLEW version, the graphics vendor, the OpenGL renderer, and the shader language version are printed to the standard output. Finally, we call our initialization function OnInit() and then attach our uninitialization function OnShutdown() as the> glutCloseFunc method—the close callback function which will be called when the window is about to close. Next, we attach our display and reshape function to their corresponding callbacks. The main function is terminated with a call to the glutMainLoop() function> which starts the application's main loop. OnInit(); glutCloseFunc(OnShutdown); glutDisplayFunc(OnRender); glutReshapeFunc(OnResize); glutMainLoop(); return 0; } There's more… The remaining functions are defined as follows: void OnInit() { glClearColor(1,0,0,0); cout<<"Initialization successfull"<<endl; } void OnShutdown() { cout<<"Shutdown successfull"<<endl; } void OnResize(int nw, int nh) { } void OnRender() { glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT); glutSwapBuffers(); } For this simple example, we set the clear color to red (R:1, G:0, B:0, A:0). The first three are the red, green, and blue channels and the last is the alpha channel which is used in alpha blending. The only other function defined in this simple example is the OnRender() function, which is our display callback function that is called on the paint event. This function first clears the color and depth buffers to the clear color and clear depth values respectively. Similar to the color buffer, there is another buffer called the depth buffer. Its clear value can be set using the glClearDepth function. It is used for hardware based hidden surface removal. It simply stores the depth of the nearest fragment encountered so far. The incoming fragment's depth value overwrites the depth buffer value based on the depth clear function specified for the depth test using the> glDepthFunc function. By default the depth value gets overwritten if the current fragment's depth is lower than the existing depth in the depth buffer. The glutSwapBuffers function is then called to set the current back buffer as the current front buffer that is shown on screen. This call is required in a double buffered OpenGL application. Running the code gives us the output shown in the following screenshot. Designing a GLSL shader class We will now have a look at how to set up shaders. Shaders are special programs that are run on the GPU. There are different shaders for controlling different stages of the programmable graphics pipeline. In the modern GPU, these include the vertex shader (which is responsible for calculating the clip-space position of a vertex), the tessellation control shader (which is responsible for determining the amount of tessellation of a given patch), the tessellation evaluation shader (which computes the interpolated positions and other attributes on the tessellation result), the geometry shader (which processes primitives and can add additional primitives and vertices if needed), and the fragment shader (which converts a rasterized fragment into a colored pixel and a depth). The modern GPU pipeline highlighting the different shader stages is shown in the following figure. Note that the tessellation control/evaluation shaders are only available in the hardware supporting OpenGL v4.0 and above. Since the steps involved in shader handling as well as compiling and attaching shaders for use in OpenGL applications are similar, we wrap these steps in a simple class we call GLSLShader. Getting ready The GLSLShader class is defined in the GLSLShader.[h/cpp] files. We first declare the constructor and destructor which initialize the member variables. The next three functions, LoadFromString, LoadFromFile, and CreateAndLinkProgram handle the shader compilation, linking, and program creation. The next two functions, Use and UnUse functions bind and unbind the program. Two std::map datastructures are used. They store the attribute's/uniform's name as the key and its location as the value. This is done to remove the redundant call to get the attribute's/uniform's location each frame or when the location is required to access the attribute/uniform. The next two functions, AddAttribute and AddUniform add the locations of the attribute and uniforms into their respective std::map (_attributeList and _uniformLocationList). class GLSLShader { public: GLSLShader(void); ~GLSLShader(void); void LoadFromString(GLenum whichShader, const string& source); void LoadFromFile(GLenum whichShader, const string& filename); void CreateAndLinkProgram(); void Use(); void UnUse(); void AddAttribute(const string& attribute); void AddUniform(const string& uniform); GLuint operator[](const string& attribute); GLuint operator()(const string& uniform); void DeleteShaderProgram(); private: enum ShaderType{VERTEX_SHADER,FRAGMENT_SHADER,GEOMETRY_SHADER}; GLuint _program; int _totalShaders; GLuint _shaders[3]; map<string,GLuint> _attributeList; map<string,GLuint> _uniformLocationList; }; To make it convenient to access the attribute and uniform locations from their maps , we declare the two indexers. For attributes, we overload the square brackets ([]) whereas for uniforms, we overload the parenthesis operation (). Finally, we define a function DeleteShaderProgram for deletion of the shader program object. Following the function declarations are the member fields. How to do it In a typical shader application, the >usage of the GLSLShader object is as follows: Create the GLSLShader object either on stack (for example, GLSLShader shader;) or on the heap (for example, GLSLShader* shader=new GLSLShader();) Call LoadFromFile on the GLSLShader object reference Call CreateAndLinkProgram on the GLSLShader object reference Call Use on the GLSLShader object reference to bind the shader object Call AddAttribute/AddUniform to store locations of all of the shader's attributes and uniforms respectively Call UnUse on the GLSLShader object reference to unbind the shader object Note that the above steps are required at initialization only. We can set the values of the uniforms that remain constant throughout the execution of the application in the Use/UnUse block given above. At the rendering step, we access uniform(s), if we have uniforms that change each frame (for example, the modelview matrices). We first bind the shader by calling the GLSLShader::Use function. We then set the uniform by calling the glUniform{*} function, invoke the rendering by calling the glDraw{*} function, and then unbind the shader (GLSLShader::UnUse). Note that the glDraw{*} call passes the attributes to the GPU. How it works In a typical OpenGL shader application, the shader specific functions and their sequence of execution are as follows: glCreateShader glShaderSource glCompileShader glGetShaderInfoLog Execution of the above four functions creates a shader object. After the shader object is created, a shader program object is >created using the following set of functions in the following sequence: glCreateProgram glAttachShader glLinkProgram glGetProgramInfoLog Note that after the shader program has been linked, we can safely delete the shader object. There's more In the GLSLShader class, the first four steps are handled in the LoadFromString function and the later four steps are handled by the CreateAndLinkProgram member function. After the shader program object has been created, we can set the program for execution on the GPU. This process is called shader binding This is carried out by the glUseProgram function which is called through the Use/UnUse functions in the GLSLShader class. To enable communication between the application and the shader, there are two different kinds of fields available in the shader. The first are the attributes which may change during shader execution across different shader stages. All per-vertex attributes fall in this category. The second are the uniforms which remain constant throughout the shader execution. Typical examples include the modelview matrix and the texture samplers. In order to communicate with the shader program, the application must obtain the location of an attribute/uniform after the shader program is bound. The location identifies the attribute/uniform. In the GLSLShader class, for convenience, we store the locations of attributes and uniforms in two separate std::map objects. For accessing any attribute/uniform location, we provide an indexer in the GLSLShader class. In cases where there is an error in the compilation or linking stage, the shader log is printed to the console. Say for example, our GLSLshader object is called shader and our shader contains a uniform called MVP. We can first add it to the map of GLSLShader by calling shader.AddUniform("MVP"). This function adds the uniform's location to the map. Then when we want to access the uniform, we directly call shader("MVP") and it returns the location of our uniform.

In just 6 quick steps, we're going to make our own Space Invaders game using Psykick2D. All of the code for this can be found here. What is Psykick2D? Psykick2D is a 2D game engine built with [Pixi.js] and designed with modularity in mind. Game objects are entities which are made up of components, systems contain and act on entities, and layers run systems. Getting Started After you download the latest Psykick2D build create an HTML page referencing Psykick2D. Your page should look something like this. Psykick2D will be looking for a container with a psykick id to place the game in. Now create a main.js and initialize the game world. var World = Psykick2D.World; World.init({ width: 400, height: 500, backgroundColor: '#000' }); Reload the page and you should see something like this. Blank screens aren't very exciting so let's add some sprites. Drawing Sprites You can find the graphics used here (special thanks to Jacob Zinman-Jeanes for providing these). Before we can use them though, we need to preload them. Add a preload option to the world initialization like so: World.init({ ... preload: [ 'sprites/player.json', 'sprites/enemy1.json', 'sprites/enemy2.json', 'sprites/enemy3.json', 'sprites/enemy4.json' ] }); Accessing sprites is as easy as referencing their frame name (given in the .json files) in a sprite component. To make an animated player, we just have to assemble the right parts in an entity. (I suggest placing these in a factory.js file) var Sprite = Psykick2D.Components.GFX.Sprite, Animation = Psykick2D.Components.GFX.Animation; function createPlayer(x, y) { var player = World.createEntity(), // Generate a new entity sprite = new Sprite({ frameName: 'player-1', x: x, y: y, width: 64, height: 29, pivot: { // The original image is sideways x: 64, y: 29 }, rotation: (270 * Math.PI) / 180 // 270 degrees in radians }), animation = new Animation({ maxFrames: 3, // zero-indexed frames: [ 'player-1', 'player-2', 'player-3', 'player-4' ] }); player.addComponent(sprite); player.addComponent(animation); return player; } Entities are comprised of components. All an entity needs is the right components and they'll work with any system. Because of thise, creating enemies looks almost exactly the same just using the enemy sprites. Once those components are attached, we just add the entities to a system then the system to a layer. The rest is taken care of automatically. var mainLayer = World.createLayer(), spriteSystem = new Psykick2D.Systems.Render.Sprite(), animationSystem = new Psykick2D.Systems.Behavior.Animate(); var player = createPlayer(210, 430); spriteSystem.addEntity(player); animationSystem.addEntity(player); mainLayer.addSystem(spriteSystem); mainLayer.addSystem(animationSystem); World.pushLayer(mainLayer); If you repeat the process for the enemies then you'll end up with a result like what you see here. With your ship on screen, let's add some controls. source Ship Controls To control the ship, we'll want to extend the BehaviorSystem to update the player's position on every update. This is made easier using the Keyboard module. var BehaviorSystem = Psykick2D.BehaviorSystem, Keyboard = Psykick2D.Input.Keyboard, Keys = Psykick2D.Keys, SPEED = 100; var PlayerControl = function() { this.player = null; BehaviorSystem.call(this); }; Psykick2D.Helper.inherit(PlayerControl, BehaviorSystem); PlayerControl.prototype.update = function(delta) { var velocity = 0; // Give smooth movement by using the change in time if (Keyboard.isKeyDown(Keys.Left)) { velocity = -SPEED * delta; } else if (Keyboard.isKeyDown(Keys.Right)) { velocity = SPEED * delta; } var player = this.player.getComponent('Sprite'); player.x += velocity; // Don't leave the screen if (player.x < 15) { player.x = 15; } else if (player.x > 340) { player.x = 340; } }; Since there's only one player we'll just set them directly instead of using the addEntity method. // main.js ... var controls = new PlayerControl(); controls.player = player; mainLayer.addSystem(controls); ... Now that the player can move we should level the playing field a little bit and give the enemies some brains. source Enemy AI In the original Space Invaders, the group of aliens would move left to right and then move closer to the player whenever they hit the edge. Since systems only accept entities with the right components, let's tag the enemies as enemies. function createEnemy(x, y) { var enemy = World.createEntity(); ... enemy.addComponentAs(true, 'Enemy'); return enemy; } Creating the enemy AI itself is pretty easy. var EnemyAI = function() { BehaviorSystem.call(this); this.requiredComponents = ['Enemy']; this.speed = 30; this.direction = 1; }; Psykick2D.Helper.inherit(EnemyAI, BehaviorSystem); EnemyAI.prototype.update = function(delta) { var minX = 1000, maxX = -1000, velocity = this.speed * this.direction * delta; for (var i = 0; i < this.actionOrder.length; i++) { var enemy = this.actionOrder[i].getComponent('Sprite'); enemy.x += velocity; // Prevent it from going outside the bounds if (enemy.x < 15) { enemy.x = 15; } else if (enemy.x > 340) { enemy.x = 340; } // Track the min/max minX = Math.min(minX, enemy.x); maxX = Math.max(maxX, enemy.x); } // If they hit the boundary if (minX <= 15 || maxX >= 340) { // Flip around and speed up this.direction = this.direction * -1; this.speed += 1; // Move the row down for (var i = 0; i < this.actionOrder.length; i++) { var enemy = this.actionOrder[i].getComponent('Sprite'); enemy.y += enemy.height / 2; } } }; Like before, we just add the correct entities to the system and add the system to the layer. var enemyAI = new EnemyAI(); enemyAI.addEntity(enemy1); enemyAI.addEntity(enemy2); ... mainLayer.addSystem(enemyAI); Incoming! Aliens are now raining down from the sky. We need a way to stop these invaders from space. source Set phasers to kill To start shooting alien scum, add the bullet sprite to the preload list preload: [ ... 'sprites/bullet.json' ] then generate a bullet just like you did the player. Since the original only let one bullet exist on screen at once, we're going to do the same. So in your PlayerControl system give it a new property for bullet and we'll add some shooting ability. var PlayerControl = function() { BehaviorSystem.call(this); this.player = null; this.bullet = null; }; ... PlayerControl.prototype.update = function(delta) { ... var bullet = this.bullet.getComponent('Sprite'); // If the bullet is off-screen and the player pressed the spacebar if (bullet.y <= -bullet.height && Keyboard.isKeyDown(Keys.Space)) { // Fire! bullet.y = player.x - 18; bullet.y = player.y; } else if (bullet.y > -bullet.height) { // Move the bullet up bullet.y -= 250 * delta; } }; Now we just need to draw the bullet and attach it to the PlayerControl system. var bullet = createBullet(0, -100); spriteSystem.addEntity(bullet); controls.bullet = bullet; And just like that you've got yourself a working gun. But you can't quite destroy those aliens yet. We need a way of making the bullet collide with the aliens. source Final Step Psykick2D has a couple of different collision structures built in. For our purposes we're going to use a grid. But in order to keep everything in sync, we want a dedicated physics system. So we're going to give our sprite components new properties, newX and newY, and set the new position there. Example: player.newX += velocity; To create a physics system, simply extend the BehaviorSystem and give it a collision structure. var Physics = function() { BehaviorSystem.call(this); this.requiredComponents = ['Sprite']; this.grid = new Psykick2D.DataStructures.CollisionGrid({ width: 400, height: 500, cellSize: 100, componentType: 'Sprite' // Do all collision checks using the sprite }); }; Psykick2D.Helper.inherit(Physics, BehaviorSystem); There's a little bit of work involved so you can view the full source here. What's important is that we check what kind of entity we're colliding with (entity.hasComponent('Bullet')) then we can destroy it by removing it from the layer. Here's the final product of all of your hard work. A fully functional space invaders-like game! Psykick2D has a lot more built in. Go ahead and really polish it up! final source About the Author Mike Cluck is a software developer interested in game development. He can be found on Github at MCluck90.

In this article by Russ McKendrick, the author of the book Docker Bootcamp, we will cover following topics: Creating a Swarm manually Launching a service (For more resources related to this topic, see here.) Creating a Swarm manually To start off with we need to launch the hosts and to do this, run the following commands, remembering to replace the digital ocean API access token with your own: docker-machine create --driver digitalocean --digitalocean-access-token 57e4aeaff8d7d1a8a8e46132969c2149117081536d50741191c79d8bc083ae73 swarm01 docker-machine create --driver digitalocean --digitalocean-access-token 57e4aeaff8d7d1a8a8e46132969c2149117081536d50741191c79d8bc083ae73 swarm02 docker-machine create --driver digitalocean --digitalocean-access-token 57e4aeaff8d7d1a8a8e46132969c2149117081536d50741191c79d8bc083ae73 swarm03 Once launched, running docker-machine ls should show you a list of your images. Also, this should be reflected in your digital ocean control panel: Now we have our Docker hosts and we need to assign a role to each of the nodes within the cluster. Docker Swarm has two node roles: Manager: A manager is a node which dispatches tasks to the workers, all your interaction with the Swarm cluster will be targeted against a manager node.You can have more than one manger node, however in this example we will be using just one. Worker: Worker nodes accept the tasks dispatched by the manager node, these are where all your services are launched. We will go in to services in more detail once we have our cluster configured. In our cluster, swarm01 will be the manager node with swarm02 and swarm03 being our two worker nodes. We are going to use the docker-machine ssh command to execute commands directly on our three nodes, starting with configuring our manager node. The commands in the walk through will only work with Mac and Linux, commands to run on Windows will be covered at the end of this section. Before we initialize the manager node, we need to capture the IP address of swarm01 as a command-line variable: managerIP=$(docker-machine ip swarm01) Now that we have the IP address, run the following command to check if it is correct: echo $managerIP And then to configure the manager node to run: docker-machine ssh swarm01 docker swarm init --advertise-addr $managerIP You will then receive confirmation that swarm01 is now a manager along with instructions on what to run to add a worker to the cluster: You don’t have to a make a note of the instructions as we will be running the command in a slightly different way. To add our two workers, we need to capture the join token in a similar way we captured the IP address of our manager node using the $managerIP variable, to do this run: joinToken=$(docker-machine ssh swarm01 docker swarm join-token -q worker) Again, you echo the variable out to check that it is valid: echo $joinToken Now it’s time to add our two worker nodes into the cluster by running: docker-machine ssh swarm02 docker swarm join --token $joinToken $managerIP:2377 docker-machine ssh swarm03 docker swarm join --token $joinToken $managerIP:2377 You should see something same as the following terminal output: Connecting your local Docker client to the manager node using: eval $(docker-machine env swarm01) and then running a docker-machine ls again shows. As you can see from the list of hosts,swarm01 is now active but there is nothing in the SWARM column, why is that? Confusingly, there are two different types of Docker Swarm cluster, there is the Legacy Docker Swarm which was managed by Docker machine, and then there is the new Docker Swarm mode which is managed by the Docker engine itself. We have a launched a Docker Swarm mode cluster, this is now the preferred way of launching Swarm, the legacy Docker Swarm is slowly being retired. To get a list of the nodes within our Swarm cluster we need to run the following command: For information on each node you can run the following command (the --pretty flag renders the JSON output from the Docker API): docker node inspect swarm01--pretty You are given a wealth of information about the host, including the fact that it is a manager and it has been launched in digital ocean. Running the same command, but for a worker node shows using similar information: docker node inspect swarm02 --pretty However, as the node is not a manager that section is missing. Before we look at launching services into our cluster we should look at how to launch our cluster using Docker machine on Windows as there are a few differences in the commands used due differences between powershell and bash. First, we need to launch the three hosts: docker-machine.exe create --driver digitalocean --digitalocean-access-token 57e4aeaff8d7d1a8a8e46132969c2149117081536d50741191c79d8bc083ae73 swarm01 docker-machine.exe create --driver digitalocean --digitalocean-access-token 57e4aeaff8d7d1a8a8e46132969c2149117081536d50741191c79d8bc083ae73 swarm02 docker-machine.exe create --driver digitalocean --digitalocean-access-token 57e4aeaff8d7d1a8a8e46132969c2149117081536d50741191c79d8bc083ae73 swarm03 Once the three hosts are up and running: You can create the manager node by running: $managerIP = $(docker-machine.exe ip swarm01) echo $managerIP docker-machine.exe ssh swarm01 docker swarm init --advertise-addr $managerIP Once you have your manager you can add the two worker nodes: $joinIP = “$(docker-machine.exe ip swarm01):2377” echo $joinIP $joinToken = $(docker-machine.exe ssh swarm01 docker swarm join-token -q worker) echo $joinToken docker-machine.exe ssh swarm02 docker swarm join --token $joinToken $joinIP docker-machine.exe ssh swarm03 docker swarm join --token $joinToken $joinIP and then configure your local Docker client to use your manager node and check the cluster status: docker-machine.exe env --shell powershell swarm01 | Invoke-Expression docker-machine.exe ls docker node ls At this stage, no matter which operating system you are using, you should have a three node Docker Swarm cluster in digital ocean, we can now look at a launching service into our cluster. Launching a service Rather than launching containers using the docker container run command you need to create a service A service defines a task which the manager then passes to one of the workers and then a container is launched: docker service create --name cluster -p:80:80/tcp russmckendrick/cluster That’s it, we should now have a single container running on one of our three nodes. To check that the service is running and get a little more information about the service, run the following commands: docker service ls docker service inspect cluster --pretty Now that we have confirmed that our service is running, you will be able to open your browser and enter the IP address of one of your three nodes (which you can get by running docker-machine ls).One of the features of Docker Swarm is it’s routing mesh: A routing mesh? When we exposed the port using the -p:80:80/tcp flag, we did a little more than map port 80 on the host to port 80 on the container, we actually created a Swarm load balancer on port 80 across all of the hosts within the cluster. The Swarm load balancer then directs requests to containers within our cluster. Running the commands as shown following, should show you which tasks are running on which nodes, remember tasks are containers which have been launched by the service: docker node ps swarm01 docker node ps swarm02 docker node ps swarm03 Like me, you probably have your single task running on swarm01: We can make things more interesting by scaling our service to add more tasks, to do this simply run the following commands to scale and check our service: docker service scale cluster=6 docker service ls docker service inspect cluster --pretty As you should see, we now have 6 tasks running within our cluster service. Checking the nodes should show that the tasks are evenly distributed between our three nodes: docker node ps swarm01 docker node ps swarm02 docker node ps swarm03 Hitting refresh in your browser should also update the hostname under the Docker image change, another way of seeing this on Mac and Linux is to run the following command: curl -s http://$(docker-machine ip swarm01)/ | grep class= As you can see from the terminal in following output, our requests are being load balanced between the running tasks: Before we terminate our Docker Swarm cluster let’s look at another way we can launch services, before we do we need to remove the currently running service, to do this simply run: docker service rm cluster Summary In this article we have learned how to create a Swarm manually, and how to launch a service. Resources for Article: Further resources on this subject: Orchestration with Docker Swarm [article] Hands On with Docker Swarm [article] Introduction to Docker [article]

In the course of this three part series, you will learn how to write a simple 3D space shooter game with Three.js. The game will introduce the basic concepts of a Three.js application, how to write modular code and the core principles of a game, such as camera, player motion and collision detection. In Part 1 we set up our package and created the world of our game. In Part 2, we added the spaceship and the asteroids for our game. In this final Part 3 of the series, we will set the collision detection, add weapons to our craft and add a way to score and manage our game health as well. Collisions make things go boom Okay, now we'll need to set up collision detection and shooting. Let's start with collision detection! We will be using a technique called hitbox, where we'll create bounding boxes for the asteroids and the spaceship and check for intersections. Luckily, Three.js has a THREE.Box3 class to help us with this. The additions to the Player module: var Player = function(parent) { var loader = newObjMtlLoader(), self = this this.loaded = false this.hitbox = newTHREE.Box3() this.update = function() { if(!spaceship) return this.hitbox.setFromObject(spaceship) } This adds the hitbox and an update method that updates the hitbox by using the spaceship object to get dimensions and position for the box. Now we'll adjust the Asteroid module to do the same: var Asteroid = function(rockType) { var mesh = newTHREE.Object3D(), self = this // Speed of motion and rotation mesh.velocity = Math.random() * 2 + 2 mesh.vRotation = newTHREE.Vector3(Math.random(), Math.random(), Math.random()) this.hitbox = newTHREE.Box3() and tweak the update method: this.update = function(z) { mesh.position.z += mesh.velocity mesh.rotation.x += mesh.vRotation.x * 0.02; mesh.rotation.y += mesh.vRotation.y * 0.02; mesh.rotation.z += mesh.vRotation.z * 0.02; if(mesh.children.length > 0) this.hitbox.setFromObject(mesh.children[0]) if(mesh.position.z > z) { this.reset(z) } } You may have noticed the reset method that isn't implemented yet. It'll come in handy later - so let's make that method: this.reset = function(z) { mesh.velocity = Math.random() * 2 + 2 mesh.position.set(-50 + Math.random() * 100, -50 + Math.random() * 100, z - 1500 - Math.random() * 1500) } This method allows us to quickly push an asteroid back into action whenever we need to. On to the render loop: function render() { cam.position.z -= 1 tunnel.update(cam.position.z) player.update() for(var i=0;i<NUM_ASTEROIDS;i++) { if(!asteroids[i].loaded) continue asteroids[i].update(cam.position.z) if(player.loaded && player.hitbox.isIntersectionBox(asteroids[i].hitbox)) { asteroids[i].reset(cam.position.z) } } } So for each asteroid that is loaded, we're checking if the hitbox of our player is intersecting (i.e. colliding) with the hitbox of the asteroid. If so, we'll reset (i.e. push into the vortex ahead of us) the asteroid, based on the camera offset. Pew! Pew! Pew! Now on to get us some weaponry! Let's create a Shot module: var THREE = require('three') var shotMtl = newTHREE.MeshBasicMaterial({ color: 0xff0000, transparent: true, opacity: 0.5 }) var Shot = function(initialPos) { this.mesh = newTHREE.Mesh( newTHREE.SphereGeometry(3, 16, 16), shotMtl ) this.mesh.position.copy(initialPos) this.getMesh = function() { returnthis.mesh } this.update = function(z) { this.mesh.position.z -= 5 if(Math.abs(this.mesh.position.z - z) > 1000) { returnfalse deletethis.mesh } returntrue } returnthis } module.exports = Shot In this module we're creating a translucent, red sphere, spawned at the initial position given to the constructor function. The update method is a bit different from those we've seen so far as it returns either true (still alive) or false (dead, remove now) based on the position. Once the shot is too far from the camera, it gets cleaned up. Now back to our main.js: var shots = [] functionrender() { cam.position.z -= 1 tunnel.update(cam.position.z) player.update() for(var i=0; i<shots.length; i++) { if(!shots[i].update(cam.position.z)) { World.getScene().remove(shots[i].getMesh()) shots.splice(i, 1) } } This snippet is adding in a loop over all the shots and updates them, removing them if needed. But we also have to check for collisions with the asteroids: for(var i=0;i<NUM_ASTEROIDS;i++) { if(!asteroids[i].loaded) continue asteroids[i].update(cam.position.z) if(player.loaded && player.hitbox.isIntersectionBox(asteroids[i].hitbox)) { asteroids[i].reset(cam.position.z) } for(var j=0; j<shots.length; j++) { if(asteroids[i].hitbox.isIntersectionBox(shots[j].hitbox)) { asteroids[i].reset(cam.position.z) World.getScene().remove(shots[j].getMesh()) shots.splice(j, 1) break } } } Last but not least we need some code to take keyboard input to fire the shots: window.addEventListener('keyup', function(e) { switch(e.keyCode) { case32: // Space var shipPosition = cam.position.clone() shipPosition.sub(newTHREE.Vector3(0, 25, 100)) var shot = newShot(shipPosition) shots.push(shot) World.add(shot.getMesh()) break } }) This code - when the spacebar key is pressed - is adding a new shot to the array, which will then be updated in the render loop. Move it, move it! Cool, but while we're at the keyboard handler, let's make things moving a bit more! window.addEventListener('keydown', function(e) { if(e.keyCode == 37) { cam.position.x -= 5 } elseif(e.keyCode == 39) { cam.position.x += 5 } if(e.keyCode == 38) { cam.position.y += 5 } elseif(e.keyCode == 40) { cam.position.y -= 5 } }) This code uses the arrow keys to move the camera around. Finishing touches Now the last bits come into play: Score and health management as well. Start with defining the two variables in main.js: var score = 0, health = 100 and change these values where appropriate: if(player.loaded && player.hitbox.isIntersectionBox(asteroids[i].hitbox)) { asteroids[i].reset(cam.position.z) health -= 20 document.getElementById('health').textContent = health if(health < 1) { World.pause() alert('Game over! You scored ' + score + ' points') window.location.reload() } } This decreases the health by 20 points whenever the spaceship hits an asteroid and shows a "Game over" box and reloads the game afterwards. for(var j=0; j<shots.length; j++) { if(asteroids[i].hitbox.isIntersectionBox(shots[j].hitbox)) { score += 10 document.getElementById("score").textContent = score asteroids[i].reset(cam.position.z) World.getScene().remove(shots[j].getMesh()) shots.splice(j, 1) break } } This increases the score by 10 whenever a shot hits an asteroid. You may have noticed the two document.getElementById calls that will not work just yet. Those are for two UI elements that we'll add to the index.html to show the player the current health and score situation: <body> <div id="bar"> Health: <span id="health">100</span>% &nbsp;&nbsp; Score: <span id="score">0</span> </div> <script src="app.js"></script> </body> And throw in some CSS, too: @import url(http://fonts.googleapis.com/css?family=Orbitron); #bar{ font-family: Orbitron, sans-serif; position:absolute; left:0; right:0; height:1.5em; background:black; color:white; line-height:1.5em; } Wrap up With all 3 Parts of this series, we now with the help of Three.js have a basic 3D game running in the browser. There's a bunch of improvements to be made - the controls, mobile input compatibility and performance, but the basic concepts are in place. Now have fun playing! About the author Martin Naumann is an open source contributor and web evangelist by heart from Zurich with a decade of experience from the trenches of software engineering in multiple fields. He works as a software engineer at Archilogic in front and backend. He devotes his time to moving the web forward, fixing problems, building applications and systems and breaking things for fun & profit. Martin believes in the web platform and is working with bleeding edge technologies that will allow the web to prosper.

Doug Paterson- Open Source Publisher Who are you and what do you do? My name is Douglas Paterson, and I’m the Publisher of the Open Source brand. I have overall responsibility for the books that we publish on Open Source projects – working with the Acquisition Editors to identify new projects where we can see a desire from the community for information to help them get the most from the project. What do you enjoy about your role? Providing new information for an Open Source project that we haven’t dealt with before to make life easier for users of that project. We are often dealing with projects that no other publisher has felt the need to engage with before, and there is great satisfaction from the community enjoying our book and responding. How important is the Open Source community to Packt and to your role? My role is only about Open Source, and of course it is very important. Packt’s first titles were about Open Source technologies, and we continue to contribute to the funding of Open Source projects through our Open Source Project Royalty Scheme and the Open Source Awards, a unique position among publishers. What are your hobbies outside of your role? Right now I’m currently enjoying have my hair pulled by my 4 month old daughter, that takes up most of my time these days. It even gets in the way of reading a new zombie novel.     Julian Copes- Open Source Marketing Executive Who are you and what do you do? My name is Julian Copes and I am the Marketing Executive who works on Open Source books and organizes the annual Open Source Awards. My role involves building relationships with projects and experienced figures within Open Source. A huge part of my role involves ensuring that the various Open Source communities are aware of relevant books when published. What do you enjoy about your role? The thing I really enjoy about my role is the opportunity to meet lots of different people from different walks of life over the internet or face to face through conferences. I am a real people person and my role allows me to build relationships with people on a worldwide scale. I also love being part of the Open Source community as I get to work with and meet the readers of Packt books, finding out ways to provide a greater service to the Open Source community in the future. How important is the Open Source community to Packt and to your role? The Open Source community is vital, not only the publication of Packt books but the level of quality books available in the Open Source book market. I work with various individuals within Open Source from project leaders to respected community members. All members of the Open Source community are important and we find we are provided with great assistance during our annual Open Source Awards from authorities in the industry who offer to be judges. What are your hobbies outside of your role? In my spare time I enjoy reading, listening to music, playing sports (especially soccer) and reading/writing poetry. I love travelling and spending time with my family. Thanks for your time Julian, lastly what projects, if any, are you working on at the moment? At present, I’m working on the new range of Packt mini books and completing some early preparation for the 2011 Open Source Awards.     Sarah Cullington- Open Source Acquisition Editor Who are you and what do you do? My name is Sarah and I am one of Packt’s Open Source Acquisition Editors. My role is to research Open Source technologies to see if there are any potential opportunities for new books to be created. If I find an interesting topic area, I will put together a title proposal to get a better idea of what the book will be about and who the audience will be. From there, I find authors to write the books, help them to develop an outline for the book, then work with them on perfecting their first few chapters. What do you enjoy about your role? One of my favourite things to do is to help new authors to develop a book idea and outline, and to really make it their own. It’s exciting to see a book idea turn into an actual outline, it makes the book seem much more real. How important is the Open Source community to Packt and to your role? Everyone in the Open Source community is incredibly friendly and helpful. I have approached many people over the years about writing a book for us, or to discuss books ideas that I have. Even if people don’t have time to write a book, they will always have time to discuss my ideas with me, to suggest book ideas of their own, or to introduce me to friends and colleagues who are also very friendly and great to chat to about Open Source topics. What are your hobbies outside of your role? I try to go to the gym three times a week, although it often turns out to be a lot less frequent! I also enjoy reading, listening to music (and often singing along), and spending time with my friends and family. Thanks for your time Sarah, lastly what projects, if any, are you working on at the moment? I am helping to put together the Packt mini-books and am working on updating a number of our Drupal, Moodle, and jQuery books.     Usha Iyer- Open Source Acquisition Editor Who are you and what do you do? I am Usha Iyer, an Acquisition Editor working with Packt and I work on Packt Open Source books. My job is to find book ideas with good potential and write book proposals on them. Once the publisher approves the book proposal, I find suitable authors for the book and contract them. I work on the initial chapters of the book, the structure of the book and I also aid the author to ensure the content is at the required standard before I pass the book on to other editors. What do you enjoy about your role? I like to research for new book ideas which have a good market and write title proposals on them. I love to see books getting published after a long process and definitely enjoy the success when a book sells well. How important is the Open Source community to Packt and to your role? Packt is the biggest publisher of Open Source books, so Open Source books are vital to Packt and how I perform in my role. As I work only on Open Source books, the growth of the Open Source community is very important to me. It feels good to work on books published on Open Source topics in areas where there is a real need for accesible books. What are your hobbies outside of your role? My hobbies are cooking, listening to music, and watching TV. I like to read fiction when I get time for it. Thanks for your time Usha, lastly what projects, if any, are you working on at the moment? Nothing special, I am basically spending time on researching for more good Open Source book ideas!     Robin de Jongh - Open Source Acquisition Editor Who are you and what do you do? I’m Robin, one of the Open Source Acquisition Editors at Packt. I started off as one of their authors, and found writing two books with them an excellent route into editing. I now investigate current and future trends in the software landscape so that I can match our books to future demand of our readers. Books take a long time to write and edit before they appear on the bookshelves, so I have to try and line up the titles you want, for when you want them, before you know you do! What do you enjoy about your role? I enjoy being part of a vibrant company, a supportive and talented team, and being brought cakes and biscuits by the CEO every time I press the big red cake button under my desk. Also it’s great to be in a role where thinking outside the box is encouraged and valued. How important is the Open Source community to Packt and to your role? Open Source has completely changed the software landscape for the better, allowing users to decide on which projects win, and which to quietly forget. Software is now driven by need, rather than by what makes money in the short term. And that’s very good for books too. It means that we, an Open Source focused publisher, directly address needs and wants, rather than just where the money is. Other publishers are catching on slowly to what that means – but Packt is blazing the trial. I see Open Source as no-tie-in software. It costs you nothing to install it, so there’s zero disincentive to trying it out. And because people are happy to experiment with Open Source software they learn to do things they never thought they could. Lack of user documentation is often the one factor stopping people benefitting from the software, stunting the growth of a project, and so Packt books can often be the catalyst to success. What are your hobbies outside of your role? I’m a committed Bible student. That’s probably not a popular thing to be right now, but I find that having a higher wisdom to draw on is vital to living a contented life. I also continue to write when I have the time. Thanks for your time Robin, lastly what projects, if any, are you working on at the moment? I’m working hard on building a list of talented, enthusiastic, hard working, Dutch authors to write books for me. If you fit at least three of those criteria, you should get in touch ;)     Aaron Nash- Editorial Quality Executive   Who are you and what do you do? My name is Aaron and I work as an Editorial Quality Executive at Packt, trying to maintain a high standard of quality across our range of books. This mainly consists of checking over the books and making sure that they meet our in-house style. What do you enjoy about your role? I enjoy getting to see a finished book, knowing that I’m usually the last person who has had any input on its content. Also, my role lets me exercise my unhealthy obsession with grammar in a way that isn’t just annoying. How important is the Open Source community to Packt and to your role? As part of my job to ensure good editorial quality, I’ve hired several freelance proofreaders. Quite a few of these have come from within the Open Source community – members of the community have been really helpful in this role, since they already have a familiarity with the material and technical terms used in our books. What are your hobbies outside of your role? I like to travel as much as possible, but when I’m stuck in the UK, I generally like reading (trashy crime novels in particular) and listening to music in my free time. I’m interested in current affairs, history, politics and other such exciting topics. This year I also finally discovered the Xbox 360 – it’s changed my life. Thanks for your time Aaron, lastly what projects, if any, are you working on at the moment? Currently I’m working with our copy editors on new ways to maintain the high quality of our books.     The Online PR Team Who are you and what do you do? We are Packt’s Online PR Team, and work closely with the people and communities surrounding the technologies on which Packt publishes. What do you enjoy about your role? We enjoy discussing our books with experts in different fields, and working with them to have our books reviewed. This helps to give us an understanding on the quality of our books, as well as promoting them to potential customers. How important is the Open Source community within Packt to you and your role? The Open Source community plays a very vital role within Packt since we publish a significant amount of Open Source titles. As the communities surrounding Open Source projects are so welcoming, it’s usually a pleasure to work with them, and we are always happy to support them with books or content. What are your hobbies outside of your role? Hanging out with friends, watching movies, and traveling. Lastly what projects, if any, are you/the MREs working on at the moment? We have recently worked on promoting the Microsoft Monday and IBM Tuesday campaigns, and are also helping to promote Packt’s online digital book library, PacktLib. Most of our time and focus revolves around the core of Packt’s business though, the books themselves.

(For more resources on JBoss see here.) CEP and ESP CEP and ESP are styles of processing in an Event Driven Architecture (General introduction to Event Driven Architecture can be found at: http://elementallinks.typepad.com/bmichelson/2006/02/eventdriven_arc.html). One of the core benefits of such an architecture is that it provides loose coupling of its components. A component simply publishes events about actions that it is executing and other components can subscribe/listen to these events. The producer and the subscriber are completely unaware of each other. A subscriber listens for events and doesn't care where they come from. Similarly, a publisher generates events and doesn't know anything about who is listening to those events. Some orchestration layer then deals with the actual wiring of subscribers to publishers. An event represents a significant change of state. It usually consists of a header and a body. The header contains meta information such as its name, time of occurrence, duration, and so on. The body describes what happened. For example, if a transaction has been processed, the event body would contain the transaction ID, the amount transferred, source account number, destination account number, and so on. CEP deals with complex events. A complex event is a set of simple events. For example, a sequence of large withdrawals may raise a suspicious transaction event. The simple events are considered to infer that a complex event has occurred. ESP is more about real-time processing of huge volume of events. For example, calculating the real-time average transaction volume over time. More information about CEP and ESP can be found on the web site, http://complexevents.com/ or in a book written by Prof. David Luckham, The Power of Events. This book is considered the milestone for the modern research and development of CEP. There are many existing pure CEP/ESP engines, both commercial and open source. Drools Fusion enhances the rule based programming with event support. It makes use of its Rete algorithm and provides an alternative to existing engines. Drools Fusion Drools Fusion is a Drools module that is a part of the Business Logic Integration Platform. It is the Drools event processing engine covering both CEP and ESP. Each event has a type, a time of occurrence, and possibly, a duration. Both point in time (zero duration) and interval-based events are supported. Events can also contain other data like any other facts—properties with a name and type. All events are facts but not all facts are events. An event's state should not be changed. However, it is valid to populate the unpopulated values. Events have clear life cycle windows and may be transparently garbage collected after the life cycle window expires (for example, we may be interested only in transactions that happened in the last 24 hours). Rules can deal with time relationships between events. Fraud detection It will be easier to explain these concepts by using an example—a fraud detection system. Fraud in banking systems is becoming a major concern. The amount of online transactions is increasing every day. An automatic system for fraud detection is needed. The system should analyze various events happening in a bank and, based on a set of rules, raise an appropriate alarm. This problem cannot be solved by the standard Drools rule engine. The volume of events is huge and it happens asynchronously. If we simply inserted them into the knowledge session, we would soon run out of memory. While the Rete algorithm behind Drools doesn't have any theoretical limitation on number of objects in the session, we could use the processing power more wisely. Drools Fusion is the right candidate for this kind of task. Problem description Let's consider the following set of business requirements for the fraud detection system: If a notification is received from a customer about a stolen card, block this account and any withdrawals from this account. Check each transaction against a blacklist of account numbers. If the transaction is transferring money from/to such an account, then flag this transaction as suspicious with the maximum severity. If there are two large debit transactions from the same account within a ninety second period and each transaction is withdrawing more than 300% of the average monthly (30 days) withdrawal amount, flag these transactions as suspicious with minor severity. If there is a sequence of three consecutive, increasing, debit transactions originating from a same account within a three minute period and these transactions are together withdrawing more than 90% of the account's average balance over 30 days, then flag those transactions as suspicious with major severity and suspend the account. If the number of withdrawals over a day is 500% higher than the average number of withdrawals over a 30 day period and the account is left with less than 10% of the average balance over a month (30 days), then flag the account as suspicious with minor severity. Duplicate transactions check—if two transactions occur in a time window of 15 seconds that have the same source/destination account number, are of the same amount, and just differ in the transaction ID, then flag those transactions as duplicates. Monitoring: Monitor the average withdrawal amount over all of the accounts for 30 days. Monitor the average balance across all of the accounts. Design and modeling Looking at the requirements, we'll need a way of flagging a transaction as suspicious. This state can be added to an existing Transaction type, or we can externalize this state to a new event type. We'll do the latter. The following new events will be defined: TransactionCreatedEvent—An event that is triggered when a new transaction is created. It contains a transaction identifier, source account number, destination account number, and the actual amount transferred. TransactionCompletedEvent—An event that is triggered when an existing transaction has been processed. It contains the same fields as the TransactionCreatedEvent class. AccountUpdatedEvent—An event that is triggered when an account has been updated. It contains the account number, current balance, and the transaction identifier of a transaction that initiated this update. SuspiciousAccount—An event triggered when there is some sort of a suspicion around the account. It contains the account number and severity of the suspicion. It is an enumeration that can have two values MINOR and MAJOR. This event's implementation is shown in the following code. SuspiciousTransaction—Similar to SuspiciousAccount, this is an event that flags a transaction as suspicious. It contains a transaction identifier and severity level. LostCardEvent—An event indicating that a card was lost. It contains an account number. One of events described—SuspiciousAccount—is shown in the following code. It also defines SuspiciousAccountSeverity enumeration that encapsulates various severity levels that the event can represent. The event will define two properties. One of them is already mentioned severity and the other one will identify the account—accountNumber. /*** marks an account as suspicious*/public class SuspiciousAccount implements Serializable { public enum SuspiciousAccountSeverity { MINOR, MAJOR}private final Long accountNumber;private final SuspiciousAccountSeverity severity;public SuspiciousAccount(Long accountNumber, SuspiciousAccountSeverity severity) { this.accountNumber = accountNumber; this.severity = severity; } private transient String toString; @Override public String toString() { if (toString == null) { toString = new ToStringBuilder(this).appendSuper( super.toString()).append("accountNumber", accountNumber).append("severity", severity) .toString(); } return toString;} Code listing 1: Implementation of SuspiciousAccount event. Please note that the equals and hashCode methods in SuspiciousAccount from the preceding code listing are not overridden. This is because an event represents an active entity, which means that each instance is unique. The toString method is implemented using org.apache.commons.lang.builder.ToStringBuilder. All of these event classes are lightweight, and they have no references to other domain classes (no object reference; only a number—accountNumber—in this case). They are also implementing the Serializable interface. This makes them easier to transfer between JVMs. As best practice, this event is immutable. The two properties above (accountNumber and severity) are marked as final. They can be set only through a constructor (there are no set methods—only two get methods). The get methods were excluded from this code listing. The events themselves don't carry a time of occurrence—a time stamp (they easily could, if we needed it; we'll see how in the next set of code listings). When the event is inserted into the knowledge session, the rule engine assigns such a time stamp to FactHandle that is returned. (Remember? session.insert(..) returns a FactHandle). In fact, there is a special implementation of FactHandle called EventFactHandle. It extends the DefaultFactHandle (which is used for normal facts) and adds few additional fields, for example—startTimestamp and duration. Both contain millisecond values and are of type long. Ok, we now have the event classes and we know that there is a special FactHandle for events. However, we still don't know how to tell Drools that our class represents an event. Drools type declarations provide this missing link. It can define new types and enhance existing types. For example, to specify that the class TransactionCreatedEvent is an event, we have to write: declare TransactionCreatedEvent @role( event )end Code listing 2: Event role declaration (cep.drl file). This code can reside inside a normal .drl file. If our event had a time stamp property or a duration property, we could map it into startTimestamp or duration properties of EventFactHandle by using the following mapping: @duration( durationProperty ) Code listing 3: Duration property mapping. The name in brackets is the actual name of the property of our event that will be mapped to the duration property of EventFactHandle. This can be done similarly for startTimestamp property. As an event's state should not be changed (only unpopulated values can be populated), think twice before declaring existing beans as events. Modification to a property may result in an unpredictable behavior. In the following examples, we'll also see how to define a new type declaration Fraud detection rules Let's imagine that the system processes thousands of transactions at any given time. It is clear that this is challenging in terms of time and memory consumption. It is simply not possible to keep all of the data (transactions, accounts, and so on) in memory. A possible solution would be to keep all of the accounts in memory as there won't be that many of them (in comparison to transactions) and keep only transactions for a certain period. With Drools Fusion, we can do this very easily by declaring that a Transaction is an event. The transaction will then be inserted into the knowledge session through an entry-point. Each entry point defines a partition in the input data storage, reducing the match space and allowing patterns to target specific partitions. Matching data from a partition requires explicit reference at the pattern declaration. This makes sense, especially if there are large quantities of data and only some rules are interested in them. We'll look at entry points in the following example. If you are still concerned about the volume of objects in memory, this solution can be easily partitioned, for example, by account number. There might be more servers, each processing only a subset of accounts (simple routing strategy might be: accountNumber module totalNumberOfServersInCluster). Then each server would receive only appropriate events. Notification The requirement we're going to implement here is essentially to block an account whenever a LostCardEvent is received. This rule will match two facts: one of type Account and one of type LostCardEvent. The rule will then set the the status of this account to blocked. The implementation of the rule is as follows: rule notification when $account : Account( status != Account.Status.BLOCKED ) LostCardEvent( accountNumber == $account.number ) from entry-point LostCardStream then modify($account) { setStatus(Account.Status.BLOCKED) };end Code listing 4: Notification rule that blocks an account (cep.drl file). As we already know, Account is an ordinary fact from the knowledge session. The second fact—LostCardEvent—is an event from an entry point called LostCardStream. Whenever a new event is created and goes through the entry point, LostCardStream, this rule tries to match (checks if its conditions can be satisfied). If there is an Account in the knowledge session that didn't match with this event yet, and all conditions are met, the rule is activated. The consequence sets the status of the account to blocked in a modify block. As we're updating the account in the consequence and also matching on it in the condition, we have to add a constraint that matches only the non-blocked accounts to prevent looping (see above: status != Account.Status.BLOCKED). Test configuration setup Following the best practice that every code/rule needs to be tested, we'll now set up a class for writing unit tests. All of the rules will be written in a file called cep.drl. When creating this file, just make sure it is on the classpath. The creation of knowledgeBase won't be shown. It is similar to the previous tests that we've written. We just need to change the default knowledge base configuration slightly: KnowledgeBaseConfiguration config = KnowledgeBaseFactory .newKnowledgeBaseConfiguration();config.setOption( EventProcessingOption.STREAM ); Code listing 5: Enabling STREAM event processing mode on knowledge base configuration. This will enable the STREAM event processing mode. KnowledgeBaseConfiguration from the preceding code is then used when creating the knowledge base—KnowledgeBaseFactory.newKnowledgeBase(config). Part of the setup is also clock initialization. We already know that every event has a time stamp. This time stamp comes from a clock which is inside the knowledge session. Drools supports several clock types, for example, a real-time clock or a pseudo clock. The real-time clock is the default and should be used in normal circumstances. The pseudo clock is especially useful for testing as we have complete control over the time. The following initialize method sets up a pseudo clock. This is done by setting the clock type on KnowledgeSessionConfiguration and passing this object to the newStatefulKnowledgeSession method of knowledgeBase. The initialize method then makes this clock available as a test instance variable called clock when calling session.getSessionClock(), as we can see in the following code: public class CepTest { static KnowledgeBase knowledgeBase; StatefulKnowledgeSession session; Account account; FactHandle accountHandle; SessionPseudoClock clock; TrackingAgendaEventListener trackingAgendaEventListener; WorkingMemoryEntryPoint entry; @Before public void initialize() throws Exception { KnowledgeSessionConfiguration conf = KnowledgeBaseFactory.newKnowledgeSessionConfiguration(); conf.setOption( ClockTypeOption.get( "pseudo" ) ); session = knowledgeBase.newStatefulKnowledgeSession(conf, null); clock = (SessionPseudoClock) session.getSessionClock(); trackingAgendaEventListener = new TrackingAgendaEventListener(); session.addEventListener(trackingAgendaEventListener); account = new Account(); account.setNumber(123456l); account.setBalance(BigDecimal.valueOf(1000.00)); accountHandle = session.insert(account); Code listing 6: Unit tests setup (CepTest.java file). The preceding initialize method also creates an event listener and passes it into the session. The event listener is called TrackingAgendaEventListener. It simply tracks all of the rule executions. It is useful for unit testing to verify whether a rule fired or not. Its implementation is as follows: public class TrackingAgendaEventListener extends DefaultAgendaEventListener { List<String> rulesFiredList = new ArrayList<String>(); @Override public void afterActivationFired( AfterActivationFiredEvent event) { rulesFiredList.add(event.getActivation().getRule() .getName()); } public boolean isRuleFired(String ruleName) { for (String firedRuleName : rulesFiredList) { if (firedRuleName.equals(ruleName)) { return true; } } return false; } public void reset() { rulesFiredList.clear(); } } Code listing 7: Agenda event listener that tracks all rules that have been fired. DefaultAgendaEventListener comes from the org.drools.event.rule package that is part of drools-api.jar file as opposed to the org.drools.event package that is part of the old API in drools-core.jar. All of the Drools agenda event listeners must implement the AgendaEventListener interface. TrackingAgendaEventListener in the preceding code extends DefaultAgendaEventListener so that we don't have to implement all of the methods defined in the AgendaEventListener interface. Our listener just overrides the afterActivationFired method that will be called by Drools every time a rule's consequence has been executed. Our implementation of this method adds the fired rule name into a list of fired rules—rulesFiredList. Then the convenience method isRuleFired takes a ruleName as a parameter and checks if this rule has been executed/fired. The reset method is useful for clearing out the state of this listener, for example, after session.fireAllRules is called. Now, back to the test configuration setup. The last part of the initialize method from code listing 6 is account object creation (account = new Account(); ...). This is for convenience purposes so that every test does not have to create one. The account balance is set to 1000. The account is inserted into the knowledge session and its FactHandle is stored so that the account object can be easily updated. Testing the notification rule The test infrastructure is now fully set up and we can write a test for the notification rule from code listing 4 as follows: @Test public void notification() throws Exception { session.fireAllRules(); assertNotSame(Account.Status.BLOCKED,account.getStatus()); entry = session .getWorkingMemoryEntryPoint("LostCardStream"); entry.insert(new LostCardEvent(account.getNumber())); session.fireAllRules(); assertSame(Account.Status.BLOCKED, account.getStatus()); } Code listing 8: Notification rule's unit test (CepTest.java file). The test verifies that the account is not blocked by accident first. Then it gets the LostCardStream entry point from the session by calling: session.getWorkingMemoryEntryPoint("LostCardStream"). Then the code listing demonstrates how an event can be inserted into the knowledge session through an entry-point—entry.insert(new LostCardEvent(...)). In a real application, you'll probably want to use Drools Pipeline for inserting events into the knowledge session. It can be easily connected to an existing ESB (Enterprise Service Bus) or a JMS topic or queue. Drools entry points are thread-safe. Each entry point can be used by a different thread; however, no two threads can concurrently use the same entry point. In this case, it makes sense to start the engine in fireUntilHalt mode in a separate thread like this: new Thread(new Runnable() { public void run() { session.fireUntilHalt(); } }).start(); Code listing 9: Continuous execution of rules. The engine will then continuously execute activations until the session.halt() method is called. The test then verifies that the status of the account is blocked. If we simply executed session.insert(new LostCardEvent(..)); the test would fail because the rule wouldn't see the event.

PHP 5 Social Networking We obviously need to keep private messages separate from the rest of the site, and ensure that they are only accessible to the sender and the receiver. While we could alter the public messages feature developed earlier, this would raise a few issues, such as being more difficult to tell whether the message being sent or read was private, and when using the Internet in a public area, the message would be shown on the area of the social network the user would most likely be visiting, which isn't ideal for private information. Because private messages will be separate from statuses, and won't need to make use of other media types to make them more interesting (though, we could set them up to make use of other media if we wanted), it makes sense for us to also use separate database tables and models for this feature. Database Our database needs provisions for the sender of the message, the recipient of the message, the subject of the message, and of course the message itself. We should also provide for if the message has been read, when the message was sent, and an ID for the message. The following illustrates a suitable structure for a messages table in our database:   Field Type Description ID Integer, Autoincrement, Primary Key Reference ID for the message Sender Integer The sender of the message Recipient Integer The recipient of the message Subject Varchar The subject the message relates to Sent Timestamp When the message was sent Message Longtext The contents of the message itself Read Boolean Indicates whether the message has been read or not More than one recipient? This database structure, and the code that follows, only supports one recipient per message. Our users might want to send to more than one recipient—feel free to add this functionality if you wish. Message model As with the majority of our database access, we require a model (models/message. php) to create, update, and retrieve message-related data from the database and encapsulate it within itself. It would also be helpful if the model pulled in a little more information from the database, including: A more user friendly representation of the date (we can get this via the MySQL DATE_FORMAT function) The name of the sender, by joining the messages table to the profile table The name of the recipient, by joining the messages table to the profile table again The first part of our model simply defines the class variables: <?php /** * Private message class */ class Message { /** * The registry object */ private $registry; /** * ID of the message */ private $id=0; /** * ID of the sender */ private $sender; /** * Name of the sender */ private $senderName; /** * ID of the recipient */ private $recipient; /** * Name of the recipient */ private $recipientName; /** * Subject of the message */ private $subject; /** * When the message was sent (TIMESTAMP) */ private $sent; /** * User readable, friendly format of the time the message was sent */ private $sentFriendlyTime; /** * Has the message been read */ private $read=0; /** * The message content itself */ private $message; The constructor takes the registry and ID of the message as parameters, if the ID has been defined, then it queries the database and sets the class variables. The database query here also formats a copy of the date into a friendlier format, and looks up the names of the sender and recipient of the message: /** * Message constructor * @param Registry $registry the registry object * @param int $id the ID of the message * @return void */ public function __construct( Registry $registry, $id=0 ) { $this->registry = $registry; $this->id = $id; if( $this->id > 0 ) { $sql = "SELECT m.*, DATE_FORMAT(m.sent, '%D %M %Y') as sent_friendly, psender.name as sender_name, precipient.name as recipient_name FROM messages m, profile psender, profile precipient WHERE precipient.user_id=m.recipient AND psender.user_id=m.sender AND m.ID=" . $this->id; $this->registry->getObject('db')->executeQuery( $sql ); if( $this->registry->getObject('db')->numRows() > 0 ) { $data = $this->registry->getObject('db')->getRows(); $this->sender = $data['sender']; $this->recipient = $data['recipient']; $this->sent = $data['sent']; $this->read = $data['read']; $this->subject = $data['subject']; $this->message = $data['message']; $this->sentFriendlyTime = $data['sent_friendly']; $this->senderName = $data['sender_name']; $this->recipientName = $data['recipient_name']; } else { $this->id = 0; } } } Next, we have setter methods for most of the class variables: /** * Set the sender of the message * @param int $sender * @return void */ public function setSender( $sender ) { $this->sender = $sender; } /** * Set the recipient of the message * @param int $recipient * @return void */ public function setRecipient( $recipient ) { $this->recipient = $recipient; } /** * Set the subject of the message * @param String $subject * @return void */ public function setSubject( $subject ) { $this->subject = $subject; } /** * Set if the message has been read * @param boolean $read * @return void */ public function setRead( $read ) { $this->read = $read; } /** * Set the message itself * @param String $message * @return void */ public function setMessage( $message ) { $this->message = $message; } The save method takes the class variables that directly relate to the messages table in the database and either inserts them as a new record, or updates the existing record: /** * Save the message into the database * @return void */ public function save() { if( $this->id > 0 ) { $update = array(); $update['sender'] = $this->sender; $update['recipient'] = $this->recipient; $update['read'] = $this->read; $update['subject'] = $this->subject; $update['message'] = $this->message; $this->registry->getObject('db')->updateRecords( 'messages', $update, 'ID=' . $this->id ); } else { $insert = array(); $insert['sender'] = $this->sender; $insert['recipient'] = $this->recipient; $insert['read'] = $this->read; $insert['subject'] = $this->subject; $insert['message'] = $this->message; $this->registry->getObject('db')->insertRecords( 'messages', $insert ); $this->id = $this->registry->getObject('db')->lastInsertID(); } } One getter method that we need, is to return the user ID of the recipient, so we can check that the currently logged in user has permission to read the message: /** * Get the recipient of the message * @return int */ public function getRecipient() { return $this->recipient; } We should also provide a method to delete the message from the database, should the user wish to delete a message: /** * Delete the current message * @return boolean */ public function delete() { $sql = "DELETE FROM messages WHERE ID=" . $this->id; $this->registry->getObject('db')->executeQuery( $sql ); if( $this->registry->getObject('db')->affectedRows() > 0 ) { $this->id =0; return true; } else { return false; } } Finally, we have a toTags method, which converts all of the non-object and non-array variables into template tags, so when we create a view message method in the controller, we simply need to construct the message object and call the toTags method: /** * Convert the message data to template tags * @param String $prefix prefix for the template tags * @return void */ public function toTags( $prefix='' ) { foreach( $this as $field => $data ) { if( ! is_object( $data ) && ! is_array( $data ) ) { $this->registry->getObject('template')->getPage()->addTag( $prefix.$field, $data ); } } } } ?> Messages model Similar to how we have a model for representing a single relationship and another for representing a number of relationships, we also need a model to represent a number of messages within the site. This is to handle the lookup of a user's private message inbox. <?php /** * Messages model */ class Messages { /** * Messages constructor * @param Registry $registry * @return void */ public function __construct( Registry $registry ) { $this->registry = $registry; } /** * Get a users inbox * @param int $user the user * @return int the cache of messages */ public function getInbox( $user ) { $sql = "SELECT IF(m.read=0,'unread','read') as read_style, m.subject, m.ID, m.sender, m.recipient, DATE_FORMAT(m.sent, '%D %M %Y') as sent_friendly, psender.name as sender_name FROM messages m, profile psender WHERE psender.user_id=m.sender AND m.recipient=" . $user . " ORDER BY m.ID DESC"; $cache = $this->registry->getObject('db')->cacheQuery( $sql ); return $cache; } } ?>

(For more resources related to this topic, see here.) Styling a site with Sass and Compass Personally, while abstract examples are fine, I find it difficult to fully understand concepts until I put them into practice. With that in mind, at this point I'm going to amend the markup in our project's index.html file, so we have some markup to flex our growing Sass and Compass muscles on. We're going to build up a basic home page for this book. Hopefully, the basic layout and structure will be common enough to be of help. If you want to see how things turn out, you can point your browser at http://sassandcompass.com, as that's what we'll be building. The markup for the home page is fairly simple. It consists of a header with links, navigation made up of list items, images, and content and a footer area. Pasting the home page markup here will span a few pages (and be extremely dull to read. Don't get too hung up on the specifics of the markup. Let me be clear. The actual code, selectors used, and the finished webpage that we'll create are not important. The Sass and Compass techniques and tools we use to create them are. You can download the code from the book's page at http://packtpub.com and also from http://sassandcompass.com You can download the code from the book's page at http:// packtpub.com and also from http://sassandcompass.com At this point, here's how the page looks in the browser: Image Wow, what a looker! Thankfully, with Sass and Compass we're going to knock this into shape in no time at all. Notice that in the source code there are semantically named classes in the markup. Some people dislike this practice, but I have found that it makes it easier to create more modular and maintainable code. A full discussion on the reasoning behind using classes against styling elements themselves is a little beyond the scope of this book. However, a good book on this topic is SMACSS by Jonathan Snook (http:// smacss.com). It's not essential to adhere slavishly to the conventions he describes, but it's a great start in thinking about how to structure and write style sheets in general. First, let's open the _normalize.scss partial file and amend the default styles for links (changing the default text-underline to a dotted bottom border) and remove the padding and margin for the ul tags. Now I think about this, before we get knee-deep in code, a little more organization is called for. Separating the layout from visuals Before getting into nesting, @extend, placeholders, and mixins, it makes sense to create some partial files for organizing the styles. Rather than create a partial Sass file for each structural area (the header, footer, and navigation), and lump all the visual styles relevant inside, we can structure the code in a slightly more abstract manner. There is no right or wrong way to split up Sass files. However, it's worth looking at mature and respected projects such as Twitter Bootstrap (https://github.com/twitter/ bootstrap) and Foundation from Zurb (https://github.com/zurb/foundation) to see how they organize their code. We'll create a partial file called _base.scss that will contain some base styles: code Then a partial file called _layout.scss. That will only contain rules pertaining to visual layout and positioning of the main page areas (the previously mentioned header, footer, aside, section, and navigation). Here are the initial styles being added into the _layout.scss partial file: code There is nothing particular to Sass there, it's all standard CSS. Debug help in the browser Thanks to the popularity of Sass there are now experimental features in browsers to make debugging Sass even easier. When inspecting an element with developer tools, the source file and line number is provided, making it easier to find the offending selector. For Chrome, here's a step-by-step explanation: http://benfra.in/1z1 Alternatively, if using Firefox, check out the FireSass extension: https://addons.mozilla.org/en-us/firefox/addon/ firesass-for-firebug/ Let's create another partial file called _modules.scss. This will contain all the modular pieces of code. This means, should we need to move the .testimonial section (which should be a modular section) in the source code, it's not necessary to move it from one partial file to another (which would be the case if the partial files were named according to their existing layout). The _module.scss file will probably end up being the largest file in the Sass project, as it will contain all the aesthetics for the items on the page. However, the hope is that these modular pieces of code will be flexible enough to look good, regardless of the structural constraints defined in the _layout.scss partial file. If working on a very large project, it may be worth splitting the modules into sub-folders. Just create a sub-folder and import the file. For example, if there was a partial called _callouts.scss in a sub-folder called modules, it could be imported using the following code: code Here is the contents of the amended styles.scss file: code A quick look at the HTML in the browser shows the basic layout styles applied: Image With that done, let's look at how Sass's ability to nest styles can help us make modules of code. What nesting is and how it facilitates modules of code Nesting provides the ability to nest styles within one another. This provides a handy way to write mini blocks of modular code. Nesting syntax With a normal CSS style declaration, there is a selector, then an opening curly brace, and then any number of property and value pairs, followed by a closing curly brace. For example: code Where Sass differs is that before the closing curly brace, you can nest another rule within. What do I mean by that? Consider this example: code In the previous code, a color has been set for the link tag using a variable. Then the hover and focus state have been nested within the main anchor style with a different color set. Furthermore, styles for the visited and active state have been nested with an alternate color defined. To reference a variable, simply write the dollar sign ( $) and then the name of the variable, for example, $variable. When compiled, this produces the following CSS: code Notice that Sass is also smart enough to know that the hex value #7FFF00 is actually the color called chartreuse, and when it has generated the CSS it has converted it to the CSS color name. How can this be used practically? Let's look at the markup for the list of navigation links on the home page. Each currently looks like this: code I've omitted the additional list items and closing tags in the previous code for the sake of brevity. From a structure point of view, each list item contains an anchor link with a <b> tag and a <span> tag within. Here's the initial Sass nested styles that have been added into the _modules.scss partial to control that section: code Notice that the rules are being nested inside the class .chapter-summary. That's because it then limits the scope of the nested styles to only apply to elements within a list item with a class of .chapter-summary. However, remember that if the styles can be re-used elsewhere, it isn't necessarily the best practice to nest them, as they may become too specific. The nesting of the selectors in this manner partially mirrors the structure of the HTML, so it's easy to see that the styles nested within only apply to those that are similarly structured in the markup. We're starting with the outermost element that needs to be styled (the <li class=”chapter-summary”>) and then nesting the first element within that. In this instance, it's the anchor tag. Then further styles have been nested for the hover and visited states along with the <b> and <span> tags. I tend to add focus and active states at the end of a project, but that's a personal thing. If you're likely to forget them, by all means add them at the same time you add hover. Here's the generated CSS from that block: code The nested styles are generated with a degree of specificity that limits their scope; they will only apply to elements within a <li class=”chapter-summary”>. It's possible to nest all manner of styles within one another. Classes, IDs, pseudo classes; Sass doesn't much care. For example, let's suppose the <li> elements need to have a dotted border, except for the last one. Let's take advantage of the last-child CSS pseudo class in combination with nesting and add this section: code The parent selector Notice that any pseudo selector that needs to be nested in Sass is prefixed with the ampersand symbol (&), then a colon (:). The ampersand symbol has a special meaning when nesting. It always references the parent selector. It's perfect for defining pseudo classes as it effectively says 'the parent plus this pseudo element'. However, there are other ways it can work too, for example, nesting a few related selectors and expressing different relationships between them: code This actually results in the following CSS: code We have effectively reversed the selectors while nesting by using the & parent selector. Chaining selectors It's also possible to create chained selectors using the parent selector. Consider this: code That will compile to this: code That will only select an element that has both HTML classes, namely selector-one and selector-two. The parent selector is perfect for pseudo selectors and at odd times its necessary to chain selectors, but beyond that I couldn't find much practical use for it, until I saw this set of slides by Sass guru Brandon Mathis http://speakerdeck.com/u/imathis/p/sass-compass-the-future-of-stylesheets-now. In this, he illustrates how to use the parent selector for nesting a Modernizr relevant rule.

Using CiviCRM This article speaks to people who have the responsibility for initiating, scoping, and managing development and implementation for the CRM strategy. If you already have CiviCRM or another CRM operating in-house without a CRM strategy, it's not too late to be more methodical in your approach. The steps below can be used to plan to re-invigorate and re-focus the use of CRM within your organization as much as to plan the first implementation. Barriers to success Constituent Relationship Management initiatives can be difficult. At their best, they involve changing external relationships and internal processes and tools. Externally, the experiences that the constituents have of your organization need to change, so that they provide more value and fewer barriers to involvement. Internally, business processes and supporting technological systems need to change in order to break down departmental operations' silos, increase efficiencies, and enable more effective targeting, improved responsiveness, and new initiatives. The success of the CRM projects often depends on changing the behavior and attitudes of individuals across the organization and replacing, changing, and/or integrating many IT systems used across the organization. Succeeding with the management of organizational culture change may involve getting staff members to take on tasks and responsibilities that may not directly benefit them or the department managed by their supervisor, but only provide value to the organization by what it enables others in the organization to accomplish or avoid. As a result, it is more challenging to align the interests of the staff and organizational units with the organization's interest in improved constituent relations, as promised by the CRM strategy. This is why an Executive Sponsor, such as the Executive Director of a small or a medium-sized non-profit organization, is so important. On the technical side, CRM projects for reasonably-sized organizations typically involve replacing or integrating many systems. Configuring and customizing a single new software system, migrating data to it, testing and deploying it, and training the staff members can be a challenge at the best of times. Doing it for multiple systems and more users multiplies the challenge. Since a CRM initiative involves integrating separate systems, the complexity of such endeavors must be faced, such as disparate data schemas requiring transformations for interoperability, and keeping middleware in sync with changes in multiple independent software packages. Unfortunately, these challenges to the CRM implementation initiative may lead to a project failure if they are not realized and addressed. The common causes for failure are as follows: Lack of executive-level sponsorship resulting in improperly resolved turf wars. IT-led initiatives have a greater tendency to focus on cost efficiency. This focus will generally not result in better constituent relations that are oriented toward achieving the organization's mission. An IT approach, particularly where users and usability experts are not on the project team, may also lead to poor user adoption if the system is not adapted to their needs, or even if the users are poorly trained. No customer data integration approach resulting in not overcoming the data silos problem, no consolidated view of constituents, poorer targeting, and an inability to realize enterprise-level benefits. Lack of buy-in, leading to a lack of use of the new CRM system and continued use of the old processes and systems it was meant to supplant. Lack of training and follow-up training causing staff anxiety and opposition. This may cause non-use or misuse of the system, resulting in poor data handling and mix-ups in the way in which constituents are treated. Not customizing enough to actually meet the requirements of the organization in the areas of: Data integration Business processes User experiences Over-customizing, causing: The costs of the system to escalate The best practices incorporated in the base functionality to be overridden in some cases User forms to become overly complex User experiences to become off-putting No strategy for dealing with the technical challenges associated with developing, extending, and/or integrating the CRM software system, leading to: Cost overruns Poorly designed and built software Poor user experiences Incomplete or invalid data However, this does not mean that project failure is inevitable or common. These clearly identifiable causes of failure can be overcome through effective project planning. Perfection is the enemy of the good CRM systems and their functional components such as fundraising, ticket sales, communication with subscribers and other stakeholders, membership management, and case management are essential for the core operations of most non-profits. This can lead to a legitimate fear of project failure when changing them. However, this fear can easily create a perfectionist mentality, where the project team attempts to overcompensate by creating too much oversight, too much contingency planning, and too much project discovery time in an effort to avoid missing any potentially useful feature that could be integrated into the project. While planning is good, perfection may not be good, since perfection is often the enemy of the good. CRM implementations risk erring on the side of what is known, somewhat tongue-in- cheek, as the MIT Approach. The MIT approach believes in, and attempts to design, construct, and deploy, the "Right Thing" right from the start. Its big-brain approach to problem-solving leads to correctness, completeness, and consistency in the design. It values simplicity in the user interface over simplicity in the implementation design. The other end of the spectrum is captured with aphorisms like "Less is More," "KISS" (Keep it simple, stupid), and "Worse is Better". This alternate view willingly accepts deviations from correctness, completeness, and consistency in design in favor of general simplicity, or simplicity of implementation over the simplicity of user interface. The reason that such counter-intuitive approaches to developing solutions have become respected and popular is the problems and failures that can result from trying to do it all perfectly from the start. Neither end of the spectrum is healthier. Handcuffing the project to an unattainable standard of perfection, or over-simplifying in order to artificially reduce complexity will both lead to project failure. There is no perfect antidote to these two extremes. As a project manager, it will be your responsibility to set the tone, determine priorities, and plan the implementation and development process. Although it is not a perspective on project management, one rule that will help achieve balance and move the project forward is "Release early, release often." This is commonly embraced in the open source community where collaboration is essential to success. This motto: Captures the intent of catching errors earlier Allows users to capture value from the system sooner Allows users to better imagine and articulate what the software should do through ongoing use and interaction with a working system early in the process Development methodologies Whatever approach your organization decides to take for developing and implementing its CRM strategy, it's usually good to have an agreed upon process and methodology. Your processes define the steps to be taken as you implement the project. Your methodology defines the rules for the process, that is, the methods to be used throughout the course of the project. The spirit of the problem-solving approaches just reviewed can be seen in the Traditional Waterfall Development model and in the contrasting Iterative and Incremental Development model. Projects naturally change and evolve over time. You may find that you embrace one of these methodologies for initial implementation, and then migrate to a different method or mixed-method for maintenance and future development work. By no means should you feel restricted by the definitions provided, but rather adjust the principles to meet your changing needs throughout the course of the project. That being said, it's important that your team understands the project rules at a given point in time, so that the project management principles are respected. The conventional Waterfall Development methodology The traditional Waterfall method of software development is sometimes thought of as "big design upfront". It employs a sequential approach to development, moving from needs analysis and requirements, to architectural and user experience, detailed design, implementation, integration, testing, deployment, and maintenance. The output of each step or phase flows downward, like water, to the next step in the process, as illustrated by the arrows in the following figure: The Waterfall model tends to be more formal, more planned, includes more documentation, and often has a stronger division of labor. This methodology benefits from having clear, linear, and progressive development steps in which each phase builds upon the previous one. However, it can suffer from inflexibility if used too rigidly. For example, if during the verification and quality assurance phase, you realize a significant functionality gap resulting from incorrect (or changing) specification requirements, then it may be difficult to interject those new needs into the process. The "release early, release often" iterative principle mentioned earlier can help overcome that inflexibility. If the overall process is kept tight and the development window short, you can justify delaying the new functionality or corrective specifications for the next release. Iterative development methodology Iterative development models depart from this structure by breaking the work up into chunks that can be developed and delivered separately. The Waterfall process is used in each phase or segment of the project, but the overall project structure is not necessarily held to the same rigid process. As one moves farther away from the Waterfall approach, there is a greater emphasis on evaluating incrementally-delivered pieces of the solution, and incorporating feedback on what has already been developed into the planning of future work, as illustrated in the loop in the following figure: This methodology seeks to take what is good in the traditional Waterfall approach— structure, clearly-defined linear steps, a strong development/quality assurance/roll out process—and improve it through shorter development cycles that are centered on smaller segments of the overall project. Perhaps the biggest challenge in this model is the project management role, as it may result in many moving pieces that must be tightly coordinated in order to release the final working product. Agile development methodology Agile development methodologies are an effective derivative of the iterative development model that moves one step further away from the Waterfall model. They are characterized by requirements and solutions evolving together, requiring work teams to be drawn from all the relevant parts of the organization. They organize themselves to work in rapid one to four-week iteration cycles. Agile centers on time-based release cycles, and in this way, differs from the other methodologies discussed, which are oriented more toward functionality-based releases. The following figure illustrates the implementation of an Agile methodology that highlights short daily Scrum status meetings, a product backlog containing features or user stories for each iteration, and a sprint backlog containing revisable and reprioritizable work items for the team during the current iteration. A deliberate effort is usually made in order to ensure that the sprint backlog is long enough to ensure that the lowest priority items will not be dealt with before the end of the iteration. Although they can be put onto the list of work items that may or may not be selected for the next iteration, the idea is that the client or the product owner should, at some point, decide that it not worth investing more resources in the "nice to have, but not really necessary" items. As one might expect, this methodology relies heavily on effective prioritization. Since software releases and development cycles adhere to rigid timeframes, only high priority issues or features are actively addressed at a given point in time; the remainder issues falling lower on the list are subject to reassignment for the next cycle.