How-To Tutorials

article-image-building-responsive-image-sliders

20 Nov 2013

7 min read

Building Responsive Image Sliders

20 Nov 2013

(For more resources related to this topic, see here.) Responsive image sliders Opening a website and seeing an image slider in the header area is common nowadays. Image sliders display highlighted content, which are really useful, within a limited space. Although the free space is more limited when a site is viewed through mobile devices, the slider element still catches the client's attention. The difference between how much area can be used to display a highlighted content and the resource available to render it is really big if compared with desktop, where we generally do not have problems with script performance, and the interaction of each transition is performed through the use of arrow signs to switch images. When the responsive era started, the way that people normally interacted with image sliders was observed, and changes, such as the way to change each slide, were identiﬁed, based on the progressive enhancement concept. The solution was to provide a similar experience to the users of mobile devices: the adoption of gestures and touches on image slider elements for devices that accept them instead of displaying fallbacks. With the constant evolution of browsers and technologies, there are many image slider plugins with responsive characteristics. My personal favorite plugins are Elastislide, FlexSlider2, ResponsiveSlides, Slicebox, and Swiper. There are plenty available, and the only way to ﬁnd one you truly like is to try them! Let's look in detail at how each of them works. Elastislide plugin Elastislide is a responsive image slider that will adapt its size and behavior in order to work on any screen size based on jQuery. This jQuery plugin handles the slider's structure, including images with percentage-based width inside, displaying it horizontally or vertically with a predeﬁned minimum number of shown images. Elastislide is licensed under the MIT license and can be downloaded from https://github.com/codrops/Elastislide. When we are implementing an image slider, simply decreasing the container size and displaying a horizontal scrollbar will not solve the problem for small devices gracefully. The recommendation is to resize the internal items too. Elastislide ﬁxes this resizing issue very well and deﬁnes the minimum elements we want to show instead of simply hiding those using CSS. Also, Elastislide uses a complementary and customized version of jQuery library named jQuery++. jQuery++ is another JavaScript library very useful to deal with DOM and special events. In this case, Elastislide has a custom version of jQuery++, which enables the plugin working with swipe events on touch devices. How to do it As we will see four different applications of this plugin for the same carousel, we will use the same HTML carousel's structure and may modify only the JavaScript before executing the plugin, specifying the parameters: <ul id="carousel" class="elastislide-list"> <li><a href="#"><img src = "image-photo.jpg" /></a></li> <li><a href="#"><img src = "image-sky.jpg" /></a></li> <li><a href="#"><img src = "image-gardem.jpg" /></a></li> <li><a href="#"><img src = "image-flower.jpg" /></a></li> <li><a href="#"><img src = "image-belt.jpg" /></a></li> <li><a href="#"><img src = "image-wall.jpg" /></a></li> <li><a href="#"><img src = "image-street.jpg" /></a></li> </ul> At the bottom of the DOM (before the </body> closing tag), we will need to include the jQuery and jQuery++ libraries (required for this solution), and then the ElastiSlide script: <script src = "http://code.jquery.com/jquery-1.9.1.min.js"></script> <script src = "js/jquerypp.custom.js"></script> <script src = "js/modernizr.custom.17475.js"></script> <script src = "js/jquery.elastislide.js"></script> Then, include the CSS stylesheet inside the <head> tag: <link rel="stylesheet" type="text/css" href="css/elastislide.css" /> Alright, now we already have the basis to show four different examples. For each example, you must add different parameters when executing the plugin script, in order to get different rendering, depending on the project need. Example 1 – minimum of three visible images (default) In this ﬁrst example, we will see the default visual and behavior, and whether we will put the following code right after it, including the ElastiSlide plugin: <script type="text/javascript"> $('#carousel').elastislide(); </script> The default options that come with this solution are: Minimum three items will be shown Speed of scroll effect is 0.5 seconds Horizontal orientation Easing effect is deﬁned as ease-in-out The carousel will start to show the ﬁrst image on the list The following screenshot represents what the implementation of this code will look like. Look at the difference between its versions shown on tablets and smartphones: Example 2 – vertical with a minimum of three visible images There is an option to render the carousel vertically, just by changing one parameter. Furthermore, we may speed up the scrolling effect. Remember to include the same ﬁles used in Example 1, and then insert the following code into the DOM: <script type="text/javascript"> $('#carousel').elastislide({ orientation: 'vertical', speed: 250 }); </script> By default, three images are displayed as a minimum. But this minimum value can be modiﬁed as we will see in our next example: Example 3 – ﬁxed wrapper with a minimum of two visible images In this example, we will deﬁne the minimum visible items in the carousel, the difference may be noticed when the carousel is viewed on small screens and the images will not reduce too much. Also, we may deﬁne the image to be shown starting from the third one. Remember to include the same ﬁ les that were used in Example 1, and then execute the scripts informing the following parameters and positioning them after including the ElastiSlide plugin: <script> $('#carousel').elastislide({ minItems: 2, start: 2 }); </script> Example 4 – minimum of four images visible in an image gallery In the fourth example, we can see many JavaScript implementations. However, the main objective of this example is to show the possibility which this plugin provides to us. Through the use of plugin callback functions and private functions we may track the click and the current image, and then handle this image change on demand by creating an image gallery: <script> var current = 0; var $preview = $('#preview'); var $carouselEl = $('#carousel'); var $carouselItems = $carouselEl.children(); var carousel = $carouselEl.elastislide({ current: current, minItems: 4, onClick: function(el, pos, evt){ changeImage(el, pos); evt.preventDefault(); }, onReady: function(){ changeImage($carouselItems.eq(current), current); } }); function changeImage(el, pos) { $preview.attr('src', el.data('preview')); $carouselItems.removeClass('current-img'); el.addClass('current-img'); carousel.setCurrent(pos); } </script> For this purpose, ElastiSlide may not have big advantages if compared with other plugins because it depends on our extra development to ﬁnalize this gallery. So, let's see what the next plugin offers to solve this problem. Summary This article, explains four different image-slider plugins and their implementation. These examples are elaborative for the use of the plugins. Resources for Article: Further resources on this subject: Top Features You Need to Know About – Responsive Web Design [Article] Different strategies to make responsive websites [Article] So, what is KineticJS? [Article]

0
0
13684

How-To Tutorials

Packt

20 Nov 2013

6 min read

Securing the Hadoop Ecosystem

Packt

20 Nov 2013

6 min read

(For more resources related to this topic, see here.) Each ecosystem component has its own security challenges and needs to be configured uniquely based on its architecture to secure them. Each of these ecosystem components has end users directly accessing the component or a backend service accessing the Hadoop core components (HDFS and MapReduce). The following are the topics that we'll be covering in this article: Configuring authentication and authorization for the following Hadoop ecosystem components: Hive Oozie Flume HBase Sqoop Pig Best practices in configuring secured Hadoop components Configuring Kerberos for Hadoop ecosystem components The Hadoop ecosystem is growing continuously and maturing with increasing enterprise adoption. In this section, we look at some of the most important Hadoop ecosystem components, their architecture, and how they can be secured. Securing Hive Hive provides the ability to run SQL queries over the data stored in the HDFS. Hive provides the Hive query engine that converts Hive queries provided by the user to a pipeline of MapReduce jobs that are submitted to Hadoop (JobTracker or ResourceManager) for execution. The results of the MapReduce executions are then presented back to the user or stored in HDFS. The following figure shows a high-level interaction of a business user working with Hive to run Hive queries on Hadoop: There are multiple ways a Hadoop user can interact with Hive and run Hive queries; these are as follows: The user can directly run the Hive queries using Command Line Interface (CLI). The CLI connects to the Hive metastore using the metastore server and invokes Hive query engine directly to execute Hive query on the cluster. Custom applications written in Java and other languages interacts with Hive using the HiveServer. HiveServer, internally, uses the metastore server and the Hive Query Engine to execute the Hive query on the cluster. To secure Hive in the Hadoop ecosystem, the following interactions should be secured: User interaction with Hive CLI or HiveServer User roles and privileges needs to be enforced to ensure users have access to only authorized data The interaction between Hive and Hadoop (JobTracker or ResourceManager) has to be secured and the user roles and privileges should be propagated to Hadoop jobs To ensure secure Hive user interaction, there is a need to ensure that the user is authenticated by HiveServer or CLI before running any jobs on the cluster. The user has to first use the kinit command to fetch the Kerberos ticket. This ticket is stored in the credential cache and used to authenticate with Kerberos-enabled systems. Once the user is authenticated, Hive submits the job to Hadoop (JobTracker or ResourceManager). Hive needs to impersonate the user to execute MapReduce on the cluster. From Hive Version 0.11 onwards, HiveServer2 was introduced. The earlier HiveServer had serious security limitations related to user authentication. HiveServer2 supports Kerberos and LDAP authentication for the user authentication. When HiveServer2 is configured to have LDAP authentication, Hive users are managed using the LDAP store. Hive asks the users to submit the MapReduce jobs to Hadoop. Thus, if we configure HiveServer2 to use LDAP, only the user authentication between the client and HiveServer2 is addressed. The interaction of Hive with Hadoop is insecure, and Hive MapReduce will be able to access other users' data in the Hadoop cluster. On the other hand, when we use Kerberos authentication for Hive users with HiveServer2, the same user is impersonated to execute MapReduce on the Hadoop cluster. So it is recommended that in a production environment, we configure HiveServer2 with Kerberos to have a seamless authentication and access control for the users submitting Hive queries. The credential store for Kerberos KDC can be configured to be LDAP so that we can centrally manage the user credentials of the end users. To set up a secured Hive interactions, we need to do the following steps: One of the key steps in securing Hive interaction is to ensure that the Hive user is impersonated in Hadoop, as Hive executes a MapReduce job on the Hadoop cluster. To achieve this goal, we need to add the hive.server2.enable.impersonation configuration in hive-site.xml, and hadoop.proxyuser.hive.hosts and hadoop. proxyuser.hive.groups in core-site.xml. <property> <name>hive.server2.authentication</name> <value>KERBEROS</value> </property> <property> <name>hive.server2.authentication.kerberos.principal</name> <value>hive/_HOST@YOUR-REALM.COM</value> </property> <property> <name>hive.server2.authentication.kerberos.keytab</name> <value>/etc/hive/conf/hive.keytab</value> </property> <property> <name>hive.server2.enable.impersonation</name> <description>Enable user impersonation for HiveServer2</description> <value>true</value> </property> Securing Hive using Sentry In the previous section, we saw how Hive authentication can be enforced using Kerberos and the user privileges that are enforced by using user impersonation in Hadoop by the superuser. Sentryis the one of the latest entrant in the Hadoop ecosystem that provides finegrained user authorization for the data that is stored in Hive. Sentry provides finegrained, role-based authorization to Hive and Impala. Sentry uses HiveServer2 and metastore server to execute the queries on the Hadoop platform. However, the user impersonation is turned off in HiveServer2 when Sentry is used. Sentry enforces user privileges on the Hadoop data using the Hive metastore. Sentry supports authorization policies per database/schema. This could be leveraged to enforce user management policies. More details on Sentry are available at the following URL: http://www.cloudera.com/content/cloudera/en/products/cdh/sentry.html Summary In this article we learned how to configure Kerberos for Hadoop ecosystem components. We also looked at how to secure Hive using Sentry. Resources for Article: Further resources on this subject: Advanced Hadoop MapReduce Administration [Article] Managing a Hadoop Cluster [Article] Making Big Data Work for Hadoop and Solr [Article]

0
0
3877

Packt

20 Nov 2013

6 min read

Understanding outside-in

Packt

20 Nov 2013

6 min read

0
0
2166

article-image-understanding-web-based-applications-and-other-multimedia-forms

Packt

20 Nov 2013

5 min read

Understanding Web-based Applications and Other Multimedia Forms

Packt

20 Nov 2013

5 min read

(For more resources related to this topic, see here.) However, we will not look at blogs, wikis, or social networking sites that are usually referred to as web-based reference tools. Moodle already has these, so instead we will take a look at web applications that allows the easy creation, collaboration, and sharing of multimedia elements, such as interactive floor planners, online maps, timelines, and many others applications that are very easy to use, and that support different learning styles. Usually, I use Moodle as a school operating system and web apps as its social applications, to illustrate what I believe can be a very powerful way of using Moodle and the web for learning. Designing meaningful activities in Moodle gives students the opportunity to express their creativity by using these tools, and reflecting on the produced multimedia artifacts with both peers and teacher. However, we have to keep in mind some issues of e-safety, backups, and licensing when using these online tools, usually associated with online communities. After all, we will have our students using them, and they will therefore be exposed to some risks. Creating dynamic charts using Google Drive (Spreadsheets) Assigning students in our Moodle course tasks will require them to use a tool like Google Spreadsheets to present their plans to colleagues in a visual way. Google Drive (http://drive.google.com) provides a set of online productivity tools that work on web standards and recreates a typical Office suite. We can make documents, spreadsheets, presentations, drawings, or forms. To use Google Drive, we will need a Google account. After creating our account and logging in to Google Drive, we can organize the files displayed on the right side of the screen, add them to folders, tag them, search (of course, it's Google!), collaborate (imagine a wiki spreadsheet), export to several formats (including the usual formats for Office documents from Microsoft, Open Office, or Adobe PDF), and publish these documents online. We will start by creating a new Spreadsheet to make a budget for a music studio which will be built during the music course, by navigating to CREATE | Spreadsheet. Insert a chart As in any spreadsheet application, we can add a title by double-clicking on Untitled spreadsheet and then we add some equipment and cost to the cells: After populating our table with values and selecting all of them, we should click on the Insert chart button. The Start tab will show up in the Chart Editor pop up, as shown in the following screenshot: If we click on the Charts tab, we can pick from a list of available charts. Let's pick one of the pie charts. In the Customize tab, we can add a title to the chart, and change its appearance: When everything is done, we can click on the Insert button, and the chart previewed in the Customize tab will be added to the spreadsheet. Publish If we click on the chart, a square will be displayed on the upper-right corner, and if we click on the drop-down arrow, we see a Publish chart... option, which can be used to publish the chart. When we click on this option, we will be presented with two ways of embedding the chart, the first, as an interactive chart, and the second, as an image. Both change dynamically if we change the values or the chart in Google Drive. We should use the image code to put the chart on a Moodle forum. Share, comment, and collaborate Google Drive has the options of sharing and allowing comments and changes in our spreadsheet by other people. On the upper-right corner of each opened document, there are two buttons for that, Comments and Share. To add collaborators to our spreadsheet, we have to click on the Share button and then add their contacts (for example, e-mail) in the Invite people: field, then click on the Share & save button, and hit Done. If a collaborator is working on the same spreadsheet, at the same time we are, we can see it below the Comments and Share buttons as shown in the following screenshot: If we click on the arrow next to 1 other viewer we can chat directly with the collaborator as we edit it collaboratively: Remember that, this can be quite useful in distance courses that have collaborative tasks assigned to groups. Creating a shared folder using Google Drive We can also use the sharing functionality to share documents with the collaborators (15 GB of space for that). In the main Google Drive page, we can create a folder by navigating to Create | Folder. We are then required to give it a name: The folder will be shown in the files and folder explorer in Google Drive: To share it with someone, we need to right-click the folder and choose the Share... option. Then, just like the process of sharing a spreadsheet we saw previously, we just need to add our collaborators' contacts (for example, e-mail) in the Invite people: field, then click on Share & save and hit Done. The invited people will receive an e-mail to add the shared folder to their Google Drive (they need a Google account for this) and it is done. Everything we add to this folder is automatically synced with everyone. This includes all the Google Drive documents, PDFs, and all the files uploaded to this folder. And it's an easy way to share multimedia projects between a group of people working on the same project.

0
0
4082

How-To Tutorials

article-image-learning-data-analytics-r-and-hadoop

Packt

20 Nov 2013

6 min read

Learning Data Analytics with R and Hadoop

Packt

20 Nov 2013

6 min read

(For more resources related to this topic, see here.) Understanding the data analytics project life cycle While dealing with the data analytics projects, there are some fixed tasks that should be followed to get the expected output. So here we are going to build a data analytics project cycle, which will be a set of standard data-driven processes to lead data to insights effectively. The defined data analytics processes of a project life cycle should be followed by sequences for effectively achieving the goal using input datasets. This data analytics process may include identifying the data analytics problems, designing, and collecting datasets, data analytics, and data visualization. The data analytics project life cycle stages are seen in the following diagram: Let's get some perspective on these stages for performing data analytics. Identifying the problem Today, business analytics trends change by performing data analytics over web datasets for growing business. Since their data size is increasing gradually day by day, their analytical application needs to be scalable for collecting insights from their datasets. With the help of web analytics; we can solve the business analytics problems. Let's assume that we have a large e-commerce website, and we want to know how to increase the business. We can identify the important pages of our website by categorizing them as per popularity into high, medium, and low. Based on these popular pages, their types, their traffic sources, and their content, we will be able to decide the roadmap to improve business by improving web traffic, as well as content. Designing data requirement To perform the data analytics for a specific problem, it needs datasets from related domains. Based on the domain and problem specification, the data source can be decided and based on the problem definition; the data attributes of these datasets can be defined. For example, if we are going to perform social media analytics (problem specification), we use the data source as Facebook or Twitter. For identifying the user characteristics, we need user profile information, likes, and posts as data attributes. Preprocessing data In data analytics, we do not use the same data sources, data attributes, data tools, and algorithms all the time as all of them will not use data in the same format. This leads to the performance of data operations, such as data cleansing, data aggregation, data augmentation, data sorting, and data formatting, to provide the data in a supported format to all the data tools as well as algorithms that will be used in the data analytics. In simple terms, preprocessing is used to perform data operation to translate data into a fixed data format before providing data to algorithms or tools. The data analytics process will then be initiated with this formatted data as the input. In case of Big Data, the datasets need to be formatted and uploaded to Hadoop Distributed File System (HDFS) and used further by various nodes with Mappers and Reducers in Hadoop clusters. Performing analytics over data After data is available in the required format for data analytics algorithms, data analytics operations will be performed. The data analytics operations are performed for discovering meaningful information from data to take better decisions towards business with data mining concepts. It may either use descriptive or predictive analytics for business intelligence. Analytics can be performed with various machine learning as well as custom algorithmic concepts, such as regression, classification, clustering, and model-based recommendation. For Big Data, the same algorithms can be translated to MapReduce algorithms for running them on Hadoop clusters by translating their data analytics logic to theMapReduce job which is to be run over Hadoop clusters. These models need to be further evaluated as well as improved by various evaluation stages of machine learning concepts. Improved or optimized algorithms can provide better insights. Visualizing data Data visualization is used for displaying the output of data analytics. Visualization is an interactive way to represent the data insights. This can be done with various data visualization softwares as well as R packages. R has a variety of packages for the visualization of datasets. They are as follows: ggplot2: This is an implementation of the Grammar of Graphics by Dr. Hadley Wickham (http://had.co.nz/). For more information refer http://cran.r-project.org/web/packages/ggplot2/. rCharts: This is an R package to create, customize, and publish interactive JavaScript visualizations from R by using a familiar lattice-style plotting interface by Markus Gesmann and Diego de Castillo. For more information refer http://ramnathv.github.io/rCharts/. Some popular examples of visualization with R are as follows: Plots for facet scales (ggplot): The following figure shows the comparison of males and females with different measures; namely, education, income, life expectancy, and literacy, using ggplot: Dashboard charts: This is an rCharts type. Using this we can build interactive animated dashboards with R. Understanding data analytics problems In this section, we have included three practical data analytics problems with various stages of data-driven activity with R and Hadoop technologies. These data analytics problem definitions are designed such that readers can understand how Big Data analytics can be done with the analytical power of functions, packages of R, and the computational powers of Hadoop. The data analytics problem definitions are as follows: Exploring the categorization of web pages Computing the frequency of changes in the stock market Predicting the sale price of a blue book for bulldozers (case study) Exploring web pages categorization This data analytics problem is designed to identify the category of a web page of a website, which may categorized popularity wise as high, medium, or low (regular), based on the visit count of the pages. While designing the data requirement stage of the data analytics life cycle, we will see how to collect these types of data from Google Analytics. Identifying the problem As this is a web analytics problem, the goal of the problem is to identify the importance of web pages designed for websites. Based on this information, the content, design, or visits of the lower popular pages can be improved or increased. Designing data requirement In this section, we will be working with data requirement as well as data collection for this data analytics problem. First let's see how the requirement for data can be achieved for this problem. Since this is a web analytics problem, we will use Google Analytics data source. To retrieve this data from Google Analytics, we need to have an existent Google Analytics account with web traffic data stored on it. To increase the popularity, we will require the visits information of all of the web pages. Also, there are many other attributes available in Google Analytics with respect to dimensions and metrics.

0
0
3311

Packt

20 Nov 2013

5 min read

Using memcached with Python

Packt

20 Nov 2013

5 min read

(For more resources related to this topic, see here.) If you want to make such a connection, there are several clients available for you. The most popular ones are: python-memcached: This is a pure-python implementation of the memcached client (implemented 100 percent in Python). It offers good performance and is extremely simple to install and use. pylibmc: This is a Python wrapper on the libmemcached C/C++ library, it offers excellent performance, thread safety, and light memory usage, yet it's not as simple as python-memcached to install, since you will need to have the libmemcached library compiled and installed on your system. Twisted memcache: This client is part of the Python twisted event-driven networking engine for Python. It offers a reactive code structure and excellent performance as well, but it is not as simple to use as pylibmc or python-memcached but it fits perfectly if your entire application is built on twisted. In this recipe, we will be using python-memcached for the sake of simplicity and since other clients have almost the same API, it does not make much difference from a developer's perspective. Getting ready It's always a good idea to create virtualenv for your experiments to keep your experiments contained and not to pollute the global system with the packages you install. You can create virtualenv easily: virtualenv memcache_experiments source memcache_experiments/bin/activate We will need to install python-memcached first, using the pip package manager on our system: sudo pip install python-memcached How to do it... Let's start with a simple set and get script: import memcache client = memcache.Client([('127.0.0.1', 11211)]) sample_obj = {"name": "Soliman", "lang": "Python"} client.set("sample_user", sample_obj, time=15) print "Stored to memcached, will auto-expire after 15 seconds" print client.get("sample_user") Save the script into a file called memcache_test1.py and run it using python memcache_test1.py. On running the script you should see something like the following: Stored to memcached, will auto-expire after 15 seconds {'lang': 'Python', 'name': 'Soliman'} Let's now try other memcached features: import memcache client = memcache.Client([('127.0.0.1', 11211)]) client.set("counter", "10") client.incr("counter") print "Counter was incremented on the server by 1, now it's %s" % client.get("counter") client.incr("counter", 9) print "Counter was incremented on the server by 9, now it's %s" % client.get("counter") client.decr("counter") print "Counter was decremented on the server by 1, now it's %s" % client.get("counter") The output of the script looks like the following: Counter was incremented on the server by 1, now it's 11 Counter was incremented on the server by 9, now it's 20 Counter was decremented on the server by 1, now it's 19 The incr and decr methods allow you to specify a delta value or to by default increment/decrement by 1. Alright, now let's sync a Python dict to memcached with a certain prefix: import memcache client = memcache.Client([('127.0.0.1', 11211)]) data = {"some_key1": "value1", "some_key2": "value2"} client.set_multi(data, time=15, key_prefix="pfx_") print "saved the dict with prefix pfx_" print "getting one key: %s" % client.get("pfx_some_key1") print "Getting all values: %s" % client.get_multi(["some_key1", "some_ key2"], key_prefix="pfx_") How it works... In this script, we are connecting to the memcached server(s) using the Client constructor, and then we are using the set method to store a standard Python dict as the value of the "sample_user" key. After that we use the get method to retrieve the value. The client automatically serialized the python dict to memcached and deserialized the object after getting it from memcached server. In the second script, we are playing with some of the features we never tried in the memcached server. The incr and decr are methods that allow you to increment and decrement integer values directly on the server automatically. Then, we are using an awesome feature that we also didn't play with before, that is get/set_multi that allows us to set or get multiple key/values at a single request. Also it allows us to add a certain prefix to all the keys during the set or get operations. The output of the last script should look like the following: saved the dict with prefix pfx_ getting one key: value1 Getting all values: {'some_key1': 'value1', 'some_key2': 'value2'} There's more... In the Client constructor, we specified the server hostname and port in a tuple (host, port) and passed that in a list of servers. This allows you to connect to a cluster of memcached servers by adding more servers to this list. For example: client = memcache.Client([('host1', 1121), ('host2', 1121), ('host3', 1122)]) Also, you can also specify custom picklers/unpicklers to tell the memcached client how to serialize or de-serialize the Python types using your custom algorithm. Summary Thus we learned how to connect to memcached servers from your python application. Resources for Article: Further resources on this subject: Working with Databases [Article] Testing and Debugging in Grok 1.0: Part 2 [Article] Debugging AJAX using Microsoft AJAX Library, Internet Explorer and Mozilla Firefox [Article]

0
0
22234

article-image-getting-started-with-fortigate-troubleshooting

Packt

20 Nov 2013

6 min read

Getting Started with Fortigate: Troubleshooting

Packt

20 Nov 2013

6 min read

0
0
32921

Packt

20 Nov 2013

7 min read

Limits of Game Data Analysis

Packt

20 Nov 2013

7 min read

(For more resources related to this topic, see here.) Which game analytics should be used This section will focus on the role data that should take in your production process. As a studio, the first step is to identify your needs and to choose the goals you will attribute to game analytics. Game analytics as a tool Firstly, it is important to understand that game analytics are a tool, which means they can serve several purposes. You can use them for marketing, science, sociological studies, and so on. Following this statement, you will need different tools and different approaches to reach your goal. As this article has tried to highlight it, tools are chosen according to problems, regardless if the choice is technique or analysis. You must not choose a tool because it is said to be the best performing tool ever made, or because it is fashionable. Instead, you must choose a tool because it is said to be the most efficient tool for your needs. Try to answer the following questions: What are the long-term uses I plan to do with game analytics? Is it simply reporting the Key Performance Indicators or is it the building a user-centric framework for deep analysis? What are the types and the level of skills of the people who will work on it?Do I have all of the skills, from data scientists to game analysts, or do I need to choose a solution which will offset some lacks in a particular field? How much data will be collected? How do I plan to deal with possible peaks of frequentation? How do I adapt temporalities of reporting and analysis with the rhythmof production I have on my project? Do I split them weekly or monthly? What are the main goals of my process? Do I want to build a predictive model (for example, based on correlations) in order to define the next acquisition campaign I will run? Do I want to increase the monetization rate on the current player base? Do I want to perform A/B testing? And the list goes on. Game analytics must serve your team Secondly, it is important to ensure that the use of game analytics must serve your team as a whole. They should not have any disagreements about the long-term objectives that you have chosen. They must accompany it and especially improve it, but the general objective should remain the same. Given the current state of the field, withdrawing the "human touch" from the design process entirely and listening only to data would be a mistake. That's why the game analytics process should be thought through the prism of your own team; and therefore, should be presented as a new tool. This will help them to make good decisions for the game. The best example for the democratization of "game analytics way of thinking" inside your team is certainly the A/B testing aspect. If you experience debates about particular features in the game, instead of taking part you can propose to use A/B tests for some of those features. Following this, there are no particular limits to the use of the tool. A game designer can test different balancing on the virtual economy of a game and an artist can experience different graphic styles. When starting, focus your attention on simple practices If you are new to the field, the following list may help you to start defining your first objectives. It contains most of the typical use for online games, especially free-to-play games: Producing KPIs on a weekly or monthly basis, according to your needs. These KPIs will help you to orient the upcoming development of your game and to anticipate the return on investment of your acquisition campaigns. Identifying if some of the steps of your tutorial phase are poorly designed; for example, if you have a sudden player loss at a particular step of your tutorial. On the same idea, having the loss of players at each level is also very useful to improve the general balancing of your game, especially the progress curve and the difficulty. This topic is more important if you have a part of your business model based on purchasable goods, which can increase the progression rate of the player. You can evaluate which area and which purchasable goods of your game are generating the best income. You can perform A/B testing on particular key features of your game in order to see which ones are the most efficient. What game analytics should not be used for On the other hand, there are a few limits that you need to know before using methods and processes from game analytics. Keep away from numbers You must always be careful about the fact that numbers are used to represent a given situation during a "T" instant. From this statement, the predictive models must always be revised and improved. they should never be considered as the perfect truth. In order for the process to be efficient, it is quite important to keep research on the data inside the structure defined by the initial goals. Otherwise, you might split your efforts and no actionable insights would be identified. In other words, numbers must remain at their place. They are a tool in the hands of a human subject, and they should not become an obsession. Try to reason if they make any sense and if you are asking the right question. Practices that need to be avoided As mentioned in the the previous section, if you are new to this field, be aware of the following situations: Data cannot dictate the full content of your next update. If it is the case, you may first re-evaluate the general intention behind your product and talk with the game designer. When starting, try to avoid complex questions that involve external factors in the game, even if they seem crucial for you. For example, trying to understand why people stopped playing your game over a long period of time is usually impossible. Old players might stop playing because another game came out or they just got bored. Data cannot make miracles at this point of the engagement. Data must not take too much ampleness in the creative process. There are some human intentions and ideas, and only then the data comes in order to verify and improve the potential success of those intentions. Data must not slow down the performances of the game. One of the common methods to avoid this is to send the data when the player logs in or logs out and not at each click or each action. Summary This is the end of this article, and the most important thing you need to remember about game analytics in general is the importance of the definition of your objectives. The reason why you choose this tool instead of another (and this article has tried to list a maximum of them, from data mining to pure analysis) is because it fits your needs as much as possible. This statement is true at every stage of the refiection process which surrounds game analytics, from the choice of the storage solution to the type of analysis you want to perform. The rising of a fully-connected state in the video game industry offers developers the opportunity to change the way they create games, but there is no doubt that the level of maturation related to this tool has not reached its maximum yet. Therefore, even if the benefits of game analytics are great, be prepared to make mistakes as well; and keep your own process open to various criticisms from your team. Resources for Article: Further resources on this subject: Flash 10 Multiplayer Game: Game Interface Design [Article] GNU Octave: Data Analysis Examples [Article] HTML5 Games Development: Using Local Storage to Store Game Data [Article]

0
0
25291

Packt

20 Nov 2013

5 min read

Reducing Data Size

Packt

20 Nov 2013

5 min read

(For more resources related to this topic, see here.) Selecting attributes using models Weighting by the PCA approach, mentioned previously, is an example where the combination of attributes within an example drives the generation of the principal components, and the correlation of an attribute with these generates the attribute's weight. When building classifiers, it is logical to take this a stage further and use the potential model itself as the determinant of whether the addition or removal of an attribute makes for better predictions. RapidMiner provides a number of operators to facilitate this, and the following sections go into detail for one of these operators with the intention of showing how applicable the techniques are to other similar operations. The operator that will be explained in detail is Forward Selection. This is similar to a number of others in the Optimization group within the Attribute selection and Data transformation section of the RapidMiner GUI operator tree. These operators include Backward Elimination and a number of Optimize Selection operators. The techniques illustrated are transferrable to these other operators. A process that uses Forward Selection is shown in the next screenshot: The Retrieve operator (labeled 1) simply retrieves the sonar data from the local sample repository. This data has 208 examples and 60 regular attributes named attribute_1 to attribute_60. The label is named class and has two values, Rock and Mine. Forward Selection operator (labeled 2) tests the performance of a model on The examples containing more and more attributes. The inner operators within this operator perform this testing. The Log to Data operator (labeled 3) creates an example set from the log entries that were written inside the Forward selection operator. Example sets are easier to process and store in the repository. The Guess Types operator (labeled 4) changes the types of attributes based on their The contents. This is simply a cosmetic step to change real numbers into integers to make plotting them look better. Now, let's return to the Forward Selection operator, which starts by invoking its inner operators to check the model performance using each of the 60 regular attributes individually. This means it runs 60 times. The attribute that gives the best performance is then retained, and the process is repeated with two attributes using the remaining 59 attributes along with the best from the first run. The best pair of attributes is then retained, and the process is repeated with three attributes using each of the remaining 58. This is repeated until the stopping conditions are met. For illustrative purposes, the parameters shown in the following screenshot are chosen to allow it to continue for 60 iterations and use all the 60 attributes. The inner operator to the Forward Selection operator is a simple cross validation with the number of folds set to three. Using cross validation ensures that the performance is an estimate of what the performance would be on unseen data. Some overfitting will inevitably occur, and it is likely that setting the number of validations to three will increase this. However, this process is for illustrative purposes and needs to run reasonably quickly, and a low cross-validation count facilitates this. Inside the Validation operator itself, there are operators to generate a model, calculate performance, and log data. These are shown in the following screenshot: The Naïve Bayes operator is a simple model that does not require a large runtime to complete. Within the Validation operator, it runs on different training partitions of the data. The Apply Model and Performance operators check the performance of the operator using test partitions. The Log operator outputs information each time it is called, and the following screenshot shows the details of what it logs. Running the process gives the log output as shown in the following screenshot: It is worth understanding this output because it gives a good overview of how the operators work and fit together in a process. For example, the attributes applyCountPerformance, applyCountValidation, and applyCountForwardSelection increment by one each time the respective operator is executed. The expected behavior is that applyCountPerformance will increment with each new row in the result, applyCountValidation will increment every three rows, which corresponds to the number of cross validation folds, and applyCountForwardSelection will remain at 1 throughout the process. Note that validationPerformance is missing for the first three rows. This is because the validation operator has not calculated a performance yet. The first occurrence of the logging operator is called validationPerformance; it is the average of innerPerformance within the validation operator. So, for example, the values for innerPerformance are 0.652, 0.514, and 0.580 for the first three rows; these values average out to 0.582, which is the value for validationPerformance in the fourth row. The featureNames attribute shows the attributes that were used to create the various performance measurements. The results are plotted as a graph as shown: This shows that as the number of attributes increases, the validation performance increases and reaches a maximum when the number of attributes is 23. From there, it steadily decreases as the number of attributes reaches 60. The best performance is given by the attributes immediately before the maximum validationPerformance attribute value. In this case, the attributes are: attribute_12, attribute_40, attribute_16, attribute_11, attribute_6, attribute_28, attribute_19, attribute_17, attribute_44, attribute_37, attribute_30, attribute_53, attribute_47, attribute_22, attribute_41, attribute_54, attribute_34, attribute_23, attribute_27, attribute_39, attribute_57, attribute_36, attribute_10. The point is that the number of attributes has reduced and indeed the model accuracy has increased. In real-world situations with large datasets and a reduction in the attribute count, an increase in performance is very valuable. Summary This article has covered the important topic of reducing data size by both the removal of examples and attributes. This is important to speed up processing time, and in some cases can even improve classification accuracy. Generally though, classification accuracy reduces as data reduces. Resources for Article: Further resources on this subject: Data Analytics [Article] Using the Spark Shell [Article] Getting started with Haskell [Article]

0
0
2059

Packt

20 Nov 2013

6 min read

Foundations

Packt

20 Nov 2013

6 min read

(For more resources related to this topic, see here.) Installation If you do not have node installed, visit: http://nodejs.org/download/. There is also an installation guide on the node GitHub repository wiki if you prefer not to or cannot use an installer: https://github.com/joyent/node/wiki/Installation. Let's install Express globally: npm install -g express If you have downloaded the source code, install its dependencies by running this command: npm install Testing Express with Mocha and SuperTest Now that we have Express installed and our package.json file in place, we can begin to drive out our application with a test-first approach. We will now install two modules to assist us: mocha and supertest. Mocha is a testing framework for node; it's flexible, has good async support, and allows you to run tests in both a TDD and BDD style. It can also be used on both the client and server side. Let's install Mocha with the following command: npm install -g mocha –-save-dev SuperTest is an integration testing framework that will allow us to easily write tests against a RESTful HTTP server. Let's install SuperTest: npm install supertest –-save-dev Continuous testing with Mocha One of the great things about working with a dynamic language and one of the things that has drawn me to node is the ability to easily do Test-Driven Development and continuous testing. Simply run Mocha with the -w watch switch and Mocha will respond when changes to our codebase are made, and will automatically rerun the tests: mocha -w Extracting routes Express supports multiple options for application structure. Extracting elements of an Express application into separate files is one option; a good candidate for this is routes. Let's extract our route heartbeat into ./lib/routes/heartbeat.js; the following listing simply exports the route as a function called index: exports.index = function(req, res){ res.json(200, 'OK'); }; Let's make a change to our Express server and remove the anonymous function we pass to app.get for our route and replace it with a call to the function in the following listing. We import the route heartbeat and pass in a callback function, heartbeat.index: var express = require('express') , http = require('http') , config = require('../configuration') , heartbeat = require('../routes/heartbeat') , app = express(); app.set('port', config.get('express:port')); app.get('/heartbeat', heartbeat.index); http.createServer(app).listen(app.get('port')); module.exports = app; 404 handling middleware In order to handle a 404 Not Found response, let's add a 404 not found middleware. Let's write a test, ./test/heartbeat.js; the content type returned should be JSON and the status code expected should be 404 Not Found: describe('vision heartbeat api', function(){ describe('when requesting resource /missing', function(){ it('should respond with 404', function(done){ request(app) .get('/missing') .expect('Content-Type', /json/) .expect(404, done); }) }); }); Now, add the following middleware to ./lib/middleware/notFound.js. Here we export a function called index and call res.json, which returns a 404 status code and the message Not Found. The next parameter is not called as our 404 middleware ends the request by returning a response. Calling next would call the next middleware in our Express stack; we do not have any more middleware due to this, it's customary to add error middleware and 404 middleware as the last middleware in your server: exports.index = function(req, res, next){ res.json(404, 'Not Found.'); }; Now add the 404 not found middleware to ./lib/express/index.js: var express = require('express') , http = require('http') , config = require('../configuration') , heartbeat = require('../routes/heartbeat') , notFound = require('../middleware/notFound') , app = express(); app.set('port', config.get('express:port')); app.get('/heartbeat', heartbeat.index); app.use(notFound.index); http.createServer(app).listen(app.get('port')); module.exports = app; Logging middleware Express comes with a logger middleware via Connect; it's very useful for debugging an Express application. Let's add it to our Express server ./lib/express/index.js: var express = require('express') , http = require('http') , config = require('../configuration') , heartbeat = require('../routes/heartbeat') , notFound = require('../middleware/notFound') , app = express(); app.set('port', config.get('express:port')); app.use(express.logger({ immediate: true, format: 'dev' })); app.get('/heartbeat', heartbeat.index); app.use(notFound.index); http.createServer(app).listen(app.get('port')); module.exports = app; The immediateoption will write a log line on request instead of on response. The devoption provides concise output colored by the response status. The logger middleware is placed high in the Express stack in order to log all requests. Logging with Winston We will now add logging to our application using Winston; let's install Winston: npm install winston --save The 404 middleware will need to log 404 not found, so let's create a simple logger module, ./lib/logger/index.js; the details of our logger will be configured with Nconf. We import Winston and the configuration modules. We define our Logger function, which constructs and returns a file logger—winston.transports. File—that we configure using values from our config. We default the loggers maximum size to 1 MB, with a maximum of three rotating files. We instantiate the Logger function, returning it as a singleton. var winston = require('winston') , config = require('../configuration'); function Logger(){ return winston.add(winston.transports.File, { filename: config.get('logger:filename'), maxsize: 1048576, maxFiles: 3, level: config.get('logger:level') }); } module.exports = new Logger(); Let's add the Loggerconfiguration details to our config files ./config/ development.jsonand ./config/test.json: { "express": { "port": 3000 }, "logger" : { "filename": "logs/run.log", "level": "silly", } } Let's alter the ./lib/middleware/notFound.js middleware to log errors. We import our logger and log an error message via logger when a 404 Not Found response is thrown: var logger = require("../logger"); exports.index = function(req, res, next){ logger.error('Not Found'); res.json(404, 'Not Found'); }; Summary This article has shown in detail with all the commands how Node.js is installed along with Express. The testing of our Express with Mocha and SuperTest is shown in detail. The logging in into our application is shown with middleware and Winston. Resources for Article: Further resources on this subject: Spring Roo 1.1: Working with Roo-generated Web Applications [Article] Building tiny Web-applications in Ruby using Sinatra [Article] Develop PHP Web Applications with NetBeans, VirtualBox and Turnkey LAMP Appliance [Article]

0
0
2499

How-To Tutorials

Packt

20 Nov 2013

6 min read

Issues and Wikis in GitLab

Packt

20 Nov 2013

6 min read

(For more resources related to this topic, see here.) Issues The built-in features for issue tracking and documentation will be very beneficial to you, especially if you're working on extensive software projects, the ones with many components, or those that need to be supported in multiple versions at once, for example, stable, testing, and unstable. In this article, we will have a closer look at the formats that are supported for issues and wiki pages (in particular, Markdown); also the elements that can be referenced from within these and how issues can be organized. Furthermore, we will go through the process of assigning issues to team members, and keeping documentation in wiki pages, which can also be edited locally. Lastly, we will see how the RSS feeds generated by GitLab can keep your team in a closer loop around the projects they work on. The metadata covered in this article may seem trivial, but many famous software projects have gained traction due to their extensive and well-written documentation, which initially was done by core developers. It enables your users to do the same with their projects, even if only internally; it opens up for a much more efficient collaboration. GitLab-flavored Markdown GitLab comes with a Markdown formatting parser that is fairly similar to GitHubs, which makes it very easy to adapt and migrate. Many standalone editors also support this format, such as Mou (http://mouapp.com/) for Mac or MarkdownPad (http://markdownpad.com/) for Windows. On Linux, editors with a split view, such as ReText (http://sourceforge.net/projects/retext/) or the more Zen-writing UberWriter (http://uberwriter.wolfvollprecht.de/) are available. For the popular Vim editor , multiple Markdown plugins too are up for grabs on a number of GitHub repositories; one of them is Vim Markdown (https://github.com/tpope/vim-markdown) by Tim Pope. Lastly, I'd like to mention that you don't need a dedicated editor for Markdown because they are plain text files. The mentioned editors simply enhance the view through syntax highlighting and preview modes. About Markdown Markdown was originally written by John Gruber, and has since evolved into various flavors. The intention of this very lightweight markup language is to have a source that is easy to edit and can be transformed into meaningful HTML to be displayed on the Web. Different variations of Markdown have made it to a majority of very successful software projects as the default language; readme files, documentation, and even blogging engines adopt it. In Markdown, text styles can be applied, links placed, and images can be inserted. If ever Markdown, by default, does not support what you are currently trying to do, you can insert plain HTML, which will not be altered by the Markdown parser. Referring to elements inside GitLab When working with source code, it can be of importance to refer to a line of code, a file, or other things, when discussing something. Because many development teams are nowadays spread throughout the world, GitLab adapts to that and makes it easy to refer and reference many things directly from comments, wiki pages, or issues. Some things like files or lines can be referenced via links, because GitLab has unique links to the branches of a repository; others are more directly accessible. The following items (basically, prefixed strings or IDs) can be referenced through shortcodes: commit messages comments wall posts issues merge requests milestones wiki pages To reference items, use the following shortcodes inside any field that supports Markdown or RDoc on the web interface: @foofor team members #123for issues !123for merge requests $123for snippets 1234567for commits Issues, knowing what needs to be done An issue is a text message of variable length, describing a bug in the code, an improvement to be made, or something else that should be done or discussed. By commenting on the issue, developers or project leaders can respond to this request or statement. The meta information attached to an issue can be very valuable to the team, because developers can be assigned to an issue, and it can be tagged or labeled with keywords that describe the content or area to which it belongs. Furthermore, you can also set a goal for the milestone to be included in this fix or feature. In the following screenshot, you can see the interface for issues: Creating issues By navigating to the Issues tab of a repository in the web interface, you can easily create new issues. Their title should be brief and precise, because a more elaborate description area is available. The description area supports the GitLab-flavored Markdown, as mentioned previously. Upon creation, you can choose a milestone and a user to assign an issue to, but you can also leave these fields unset, possibly to let your developers themselves choose with what they want to work and at what time. Before they begin their work, they can assign the issues to themselves. In the following screenshot, you can see what the issue creation form looks like: Working with labels or tags Labels are tags used to organize issues by the topic and severity. Creating labels is as easy as inserting them, separated by a comma, into the respective field while creating an issue. Currently in Version 5.2, certain keywords trigger a certain background color on the label. Labels like critical or bug turn red, feature turns green, and other labels are blue by default. The following screenshot shows what a list of labeled features looks like: After the creation of a label, it will be listed under the Labels tab within the Issues page, with a link that lists all the issues that have been labeled the same. Filtering by the label, assigned user, or milestone is also possible from the list of issues within each projects overview. Summary In this article, we have had a look at the project management side of things. You can now make use of the built-in possibilities to distribute tasks across team members through issues, keep track of things that still have to do with the issues, or enable observers to point out bugs. Resources for Article : Further resources on this subject: Using Gerrit with GitHub [Article] The architecture of JavaScriptMVC [Article] Using the OSGi Bundle Repository in OSGi and Apache Felix 3.0 [Article]

0
1
17589

Packt

20 Nov 2013

8 min read

Styling the Forms

Packt

20 Nov 2013

8 min read

0
0
10789

How-To Tutorials

article-image-overview-process-management-microsoft-visio-2013

Packt

20 Nov 2013

6 min read

Overview of Process Management in Microsoft Visio 2013

Packt

20 Nov 2013

6 min read

(For more resources related to this topic, see here.) When Visio was first conceived of over 20 years ago, its first stated marketing aim was to outsell ABC Flowcharter, the best-selling process diagramming tool at the time. Therefore, Visio had to have all of the features from the start that are core in the creation of flowcharts, namely the ability to connect one shape to another and to have the lines route themselves around shapes. Visio soon achieved its aim, and looked for other targets to reach. So, process flow diagrams have long been a cornerstone of Visio's popularity and appeal and, although there have been some usability improvements over the years, there have been few enhancements to turn the diagrams into models that can be managed efficiently. Microsoft Visio 2010 saw the introduction of two features, structured diagrams and validation rules, that make process management achievable and customizable, and Microsoft Visio 2013 sees these features enhanced. In this article, you will be introduced to the new features that have been added to Microsoft Visio to support structured diagrams and validation. You will see where Visio fits in the Process Management stack, and explore the relevant out of the box content. Exploring the new process management features in Visio 2013 Firstly, Microsoft Visio 2010 introduced a new Validation API for structured diagrams and provided several examples of this in use, for example with the BPMN (Business Process Modeling Notation) Diagram and Microsoft SharePoint Workflow templates and the improvements to the Basic Flowchart and Cross-Functional Flowchart templates, all of which are found in the Flowchart category. Microsoft Visio 2013 has updated the version of BPMN from 1.1 to 2.0, and has introduced a new SharePoint 2013 Workflow template, in addition to the 2010 one. Templates in Visio consist of a predefined Visio document that has one or more pages, and may have a series of docked stencils (usually positioned on the left-hand side of workspace area). The template document may have an associated list of add-ons that are active while it is in use, and, with Visio 2013 Professional edition, an associated list of structured diagram validation rulesets as well. Most of the templates that contain validation rules in Visio 2013 are in the Flowchart category, as seen in the following screenshot, with the exception being the Six Sigma template in the Business category. Secondly, the concept of a Subprocess was introduced in Visio 2010. This enables processes to hyperlink to other pages describing the subprocesses in the same document, or even across documents. This latter point is necessary if subprocesses are stored in a document library, such as Microsoft SharePoint. The following screenshot illustrates how an existing subprocess can be associated with a shape in a larger process, selecting an existing shape in the diagram, before selecting the existing page that it links to from the drop-down menu on the Link to Existing button. In addition, a subprocess page can be created from an existing shape, or a selection of shapes, in which case they will be moved to the newly-created page. There were also a number of ease-of-use features introduced in Microsoft Visio 2010 to assist in the creation and revision of process flow diagrams. These include: Easy auto-connection of shapes Aligning and spacing of shapes Insertion and deletion of connected shapes Improved cross-functional flowcharts Subprocesses An infinite page option, so you need not go over the edge of the paper ever again Microsoft Visio 2013 has added two more notable features: Commenting (a replacement for the old reviewer's comments) Co-authoring However, this book is not about teaching the user how to use these features, since there will be many other authors willing to show you how to perform tasks that only need to be explained once. This book is about understanding the Validation API in particular, so that you can create, or amend, the rules to match the business logic that your business requires. Reviewing Visio Process Management capabilities Microsoft Visio now sits at the top of the Microsoft Process Management Product Stack, providing a Business Process Analysis (BPA) or Business Process Modeling (BPM) tool for business analysts, process owners/participants, and line of business software architects/developers. Understanding the Visio BMP Maturity Model If we look at the Visio BPM Maturity Model that Microsoft has previously presented to its partners, then we can see that Visio 2013 has filled some of the gaps that were still there after Visio 2010. However, we can also see that there are plenty of opportunities for partners to provide solutions on top of the Visio platform. The maturity model shows how Visio initially provided the means to capture paper-drawn business processes into electronic format, and included the ability to encapsulate data into each shape and infer the relationship and order between elements through connectors. Visio 2007 Professional added the ability to easily link shapes, which represent processes, tasks, decisions, gateways, and so on with a data source. Along with that, data graphics were provided to enable shape data to be displayed simply as icons, data bars, text, or to be colored by value. This enriched the user experience and provided quicker visual representation of data, thus increasing the comprehension of the data in the diagrams. Generic templates for specific types of business modeling were provided. Visio had a built-in report writer for many versions, which provided the ability to export to Excel or XML, but Visio 2010 Premium introduced the concept of validation and structured diagrams, which meant that the information could be verified before exporting. Some templates for specific types of business modeling were provided. Visio 2010 Premium also saw the introduction of Visio Services on SharePoint that provided the automatic (without involving the Visio client) refreshing of data graphics that were linked to specific types of data sources. Throughout this book we will be going into detail about Level 5 (Validation) in Visio 2013, because it is important to understand the core capabilities provided in Visio 2013. We will then be able to take the opportunity to provide custom Business Rule Modeling and Visualization. Reviewing the foundations of structured diagramming A structured diagram is a set of logical relationships between items, where these relationships provide visual organization or describe special interaction behaviors between them. The Microsoft Visio team analyzed the requirements for adding structure to diagrams and came up with a number of features that needed to be added to the Visio product to achieve this: Container Management: The ability to add labeled boxes around shapes to visually organize them Callout Management: The ability to associate callouts with shapes to display notes List Management: To provide order to shapes within a container Validation API: The ability to test the business logic of a diagram Connectivity API: The ability to create, remove, or traverse connections easily The following diagram demonstrates the use of Containers and Callouts in the construction of a basic flowchart, that has been validated using the Validation API, which in turn uses the Connectivity API.

0
0
9330

Packt

20 Nov 2013

11 min read

Setting Up NameNode HA

Packt

20 Nov 2013

11 min read

(For more resources related to this topic, see here.) We will configure our NameNode HA setup by adding several options to the core-site.xml file. The following is the structure of the file for this particular step. It will give you an idea of the XML structure, if you are not familiar with it. The header comments are stripped out: <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>fs.default.name</name> <value>hdfs://sample-cluster/</value> </property> <property> <name>ha.zookeeper.quorum</name> <value>nn1.hadoop.test.com:2181,nn2.hadoop.test.com:2181,jt1.hadoop. test.com:2181 </value> </property> </configuration> The configuration file format is pretty much self-explanatory; variables are surrounded by the <property> tag, and each variable has a name and a value. There are only two variables that we need to add at this stage. fs.default.name is the logical name of the NameNode cluster. The value hdfs://sample-cluster/ is specific to the HA setup. This is the logical name of the NameNode cluster. We will define the servers that comprise of it in the hdfs-site.xml file. In a non-HA setup, this variable is assigned a host and a port of the NameNode, since there is only one NameNode in the cluster. The ha.zookeeper.quorum variable specifies locations and ports of the ZooKeeper servers. The ZooKeeper cluster can be used by other services, such as HBase, that is why it is defined in core-site.xml. The next step is to configure the hdfs-site.xml file and add all HDFS-specific parameters there. I will omit the <property> tag and only include <name> and <value> to make the list less verbose. <name>dfs.name.dir</name> <value>/dfs/nn/</value> NameNode will use the location specified by the dfs.name.dir variable to store the persistent snapshot of HDFS metadata. This is where the fsimage file will be stored. As discussed previously, the volume on which this directory resides needs to be backed by RAID. Losing this volume means losing NameNode completely. The /dfs/nn path is an example, however you are free to choose your own. You can actually specify several paths with a dfs.name.dir value, separating them by commas. NameNode will mirror the metadata files in each directory specified. If you have a shared network storage available, you can use it as one of the destinations for HDFS metadata. This will provide additional offsite backups. <name>dfs.nameservices</name> <value>sample-cluster</value> The dfs.nameservices variable specifies the logical name of the NameNode cluster and should be replaced by something that makes sense to you, such as prod-cluster or stage-cluster. The value of dfs.nameservices must match the value of fs.default.name from the core-site.xml file. <name>dfs.ha.namenodes.sample-cluster</name> <value>nn1,nn2</value> Here, we specify the NameNodes that make up our HA cluster setup. These are logical names, not real server hostnames or IPs. These logical names will be referenced in other configuration variables. <name>dfs.namenode.rpc-address.sample-cluster.nn1</name> <value>nn1.hadoop.test.com:8020</value> <name>dfs.namenode.rpc-address.sample-cluster.nn2</name> <value>nn2.hadoop.test.com:8020</value> This pair of variables provide mapping from logical names like nn1 and nn2 to the real host and port value. By default, NameNode daemons use port 8020 for communication with clients and each other. Make sure this port is open for the cluster nodes. <name>dfs.namenode.http-address.sample-cluster.nn1</name> <value>nn1.hadoop.test.com:50070</value> <name>dfs.namenode.http-address.sample-cluster.nn2</name> <value>nn2.hadoop.test.com:50070</value> Each NameNode daemon runs a built-in HTTP server, which will be used by the NameNode web interface to expose various metrics and status information about HDFS operations. Additionally, standby NameNode uses HTTP calls to periodically copy the fsimage file from the primary server, perform the checkpoint operation, and ship it back. <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://nn1.hadoop.test.com:8485;nn2.hadoop.test.com:8485; jt1.hadoop.test.com:8485/sample-cluster</value> The dfs.namenode.shared.edits.dir variable specifies the setup of the JournalNode cluster. In our configuration, there are three JournalNodes running on nn1, nn2, and nn3. Both primary and standby nodes will use this variable to identify which hosts they should contact to send or receive new changes from editlog. <name>dfs.journalnode.edits.dir</name> <value>/dfs/journal</value> JournalNodes need to persist editlog changes that are being submitted to them by the active NameNode. The dfs.journalnode.edits.dir variable specifies the location on the local filesystem where editlog changes will be stored. Keep in mind that this path must exist on all JournalNodes and the ownership of all directories must be set to hdfs:hdfs (user and group). <name>dfs.client.failover.proxy.provider.sample-cluster</name> <value>org.apache.hadoop.hdfs.server.namenode.ha. ConfiguredFailoverProxyProvider</value> In an HA setup, clients that access HDFS need to know which NameNode to contact for their requests. The dfs.client.failover.proxy.provider.sample-cluster variable specifies the Java class name, which will be used by clients for determining the active NameNode. At the moment, there is only ConfiguredFailoverProxyProvider available. <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> The dfs.ha.automatic-failover.enabled variable indicates if the NameNode cluster will use a manual or automatic failover. <name>dfs.ha.fencing.methods</name> <value>sshfence shell(/bin/true) </value> Orchestrating failover in a cluster setup is a complicated task involving multiple steps. One of the common problems that is not unique to the Hadoop cluster, but affects any distributed systems, is a "split-brain" scenario. Split-brain is a case where two NameNodes decide they both play an active role and start writing changes to the editlog. To prevent such an issue from occurring, the HA configuration maintains a marker in ZooKeeper, clearly stating which NameNode is active, and JournalNodes accepts writes only from that node. To be absolutely sure that the two NameNodes don't become active at the same time, a technique called fencing is used during failover. The idea is to force the shutdown of the active NameNode before transferring the active state to a standby. There are two fencing methods currently available: sshfence and shell. sshfence. These require a passwordless ssh access as a user that starts the NameNode daemon, from the active NameNode to the standby and vice versa. By default, this is the hdfs user. The fencing process checks if there is anyone listening on a NameNode port using the nc command, and if the port is found busy, it tries to kill the NameNode process. Another option for dfs.ha.fencing.methods is shell. This will execute the specified shell script to perform fencing. It is important to understand that failover will fail if fencing fails. In our case, we specified two options, the second one always returns success. This is done for workaround cases where the primary NameNode machine goes down and the ssh method will fail, and no failover will be performed. We want to avoid this, so the second option would be to failover anyway, even without fencing, which, as already mentioned, is safe with our setup. To achieve this, we specify two fencing methods, which will be tried by ZKFC in the order of: if the first one fails, the second one will be tried. In our case, the second one will always return success and failover will be initiated, even if the server running the primary NameNode is not available via ssh. <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/var/lib/hadoop-hdfs/.ssh/id_rsa</value> The last option we will need to configure for NameNode HA setup is the ssh key, which will be used by sshfence. Make sure you change the ownership for this file to hdfs user. Two keys need to be generated, one for the primary and one for the secondary NameNode. It is a good idea to test ssh access as an hdfs user in both directions to make sure it is working fine. The hdfs-site.xml configuration file is now all set for testing the HA setup. Don't forget to sync these configuration files to all the nodes in the cluster. The next thing that needs to be done is to start JournalNodes. Execute this command on nn1, nn2, and jt1 a root user: # service hadoop-hdfs-journalnode start With CDH, it is recommended to always use the service command instead of calling scripts in /etc/init.d/ directly. This is done to guarantee that all environment variables are set up properly before the daemon is started. Always check the logfiles for daemons. Now, we need to initially format HDFS. For this, run the following command on nn1: # sudo -u hdfs hdfs namenode –format This is the initial setup of the NameNode, so we don't have to worry about affecting any HDFS metadata, but be careful with this command, because it will destroy any previous metadata entries. There is no strict requirement to run format command on nn1, but to make it easier to follow, let's assume we want nn1 to become an active NameNode. Format command will also format the storage for the JournalNodes. The next step is to create an entry for the HA cluster in ZooKeeper, and start NameNode and ZKFC on the first NameNode. In our case, this is nn1: # sudo -u hdfs hdfs zkfc -formatZK # service hadoop-hdfs-namenode start # service hadoop-hdfs-zkfc start Check the ZKFC log file (by default, it is in /var/log/hadoop-hdfs/) to make sure nn1 is now an active NameNode: INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at nn1.hadoop.test.com/192.168.0.100:8020 active... INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at nn1.hadoop.test.com/192.168.0.100:8020 to active state To activate the secondary NameNode, an operation called bootstrapping needs to be performed. To do this, execute the following command on nn2: # sudo -u hdfs hdfs namenode –bootstrapStandby This will pull the current filesystem state from active NameNode and synchronize the secondary NameNode with the JournalNodes Quorum. Now, you are ready to start the NameNode daemon and the ZKFC daemon on nn2. Use the same commands that you used for nn1. Check the ZKFC log file to make sure nn2 successfully acquired the secondary NameNode role. You should see the following messages at the end of the logfile: INFO org.apache.hadoop.ha.ZKFailoverController: ZK Election indicated that NameNode at nn2.hadoop.test.com/192.168.0.101:8020 should become standby INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at nn2.hadoop.test.com/192.168.0.101:8020 to standby state This is the last step in configuring NameNode HA. It is a good idea to verify if automatic failover is configured correctly, and if it will behave as expected in the case of a primary NameNode outage. Testing failover in the cluster setup stage is easier and safer than discovering that failover doesn't work during production stage and causing a cluster outage. You can perform a simple test: kill the primary NameNode daemon and verify if the secondary takes over its role. After that, bring the old primary back online and make sure it takes over the secondary role. You can use execute the following command to get the current status of NameNode nn1: # sudo -u hdfs hdfs haadmin -getServiceState nn1 The hdfs haadmin command can also be used to initiate a failover in manual failover setup. At this point, you have a fully configured and functional NameNode HA setup. Summary We saw in this article how to configure Hadoop's NameNode HA. Resources for Article: Further resources on this subject: Advanced Hadoop MapReduce Administration [Article] Managing a Hadoop Cluster [Article] Making Big Data Work for Hadoop and Solr [Article]

0
0
5573

article-image-automated-testing-using-robotium

Packt

19 Nov 2013

10 min read

Automated testing using Robotium

Packt

19 Nov 2013

10 min read

(For more resources related to this topic, see here.) Robotium framework Robotium is an open source automation testing framework that is used to write a robust and powerful black box for Android applications (the emphasis is mostly on black box test cases). It fully supports testing for native and hybrid applications. Native apps are live on the device, that is, designed for a specific platform and can be installed from the Google Play Store, whereas Hybrid apps are partly native and partly web apps. These can also be installed from the app store, but require the HTML to be rendered in the browser. Robotium is mostly used to automate UI test cases and internally uses run-time binding to Graphical User Interface (GUI) components. Robotium is released under the Apache License 2.0. It is free to download and can be easily used by individuals and enterprises and is built on Java and JUnit 3. It will be more appropriate to call Robotium an extension of the Android Test Unit Framework, available at http://developer.android.com/tools/testing/testing_android.html. Robotium can also work without the application, under the test's source code. The test cases written using Robotium can either be executed on the Android Emulator (Android Virtual Device (AVD))—we will see how to create an AVD during installation in the following section—or on a real Android device. Developers can write function, system, and acceptance test scenarios across multiple activities. It is currently the world's leading Automation Testing Framework, and many open source developers are contributing to introduce more and more exciting features in subsequent releases. The following screenshot is of the git repository website for the Robotium project: As Robotium is an open source project, anyone can contribute for the purpose of development and help in enhancing the framework with many more features. The Robotium source code is maintained at GitHub and can be accessed using the following link: https://github.com/jayway/robotium You just need to fork the project. Make all your changes in a clone project and click on Pull Request on your repository to tell core team members which changes to bring in. If you are new to the git environment, you can refer to the GitHub tutorial at the following link: https://help.github.com/ Robotium is like Selenium but for Android. This project was started in January 2010 by Renas Reda. He is the founder and main developer for Robotium. The project initiated with v1.0 and continues to be followed up with new releases due to new requirements. It has support for Android features such as activities, toasts, menus, context menus, web views, and remote controls. Let's see most of the Robotium features and benefits for Android test case developers. Features and benefits Automated testing using Robotium has many features and benefits. The triangularization workflow diagram between the user, Robotium, and the Android device clearly explains use cases between them: The features and benefits of Robotium are as follows: Robotium helps us to quickly write powerful test cases with minimal knowledge of the application under test. Robotium offers APIs to directly interact with UI controls within the Android application such as EditText, TextView, and Button. Robotium officially supports Android 1.6 and above versions. The Android platform is not modified by Robotium. The Robotium test can also be executed using command prompt. Robotium can be integrated smoothly with Maven or Ant. This helps to add Robotium to your project's build automation process. Screenshots can be captured in Robotium (an example screenshot is shown as follows): The test application project and the application project run on the same JVM, that is, Dalvik Virtual Machine (DVM). It's possible to run Robotium without a source code. Robotium can work with other code coverage measurement tools, such as Cobertura and Emma. Robotium can detect the messages that are shown on the screen (Toasts). Robotium supports Android features such as activities, menu, and context menu. Robotium automated tests can be implemented quickly. Robotium is built on JUnit, because of which it inherits all JUnit's features. The Robotium framework automatically handles multiple activities in an Android application. Robotium test cases are prominently readable, in comparison to standard instrumentation tests. Scrolling activity is automatically handled by the Robotium framework. Recent versions of Robotium support hybrid applications. Hybrid applications use WebViews to present the HTML and JavaScript files in full screen, using the native browser rendering engine. API set Web support has been added to the Robotium framework since Robotium 4.0 released. Robotium has full support for hybrid applications. There are some key differences between native and hybrid applications. Let's go through them one by one, as follows: Native Application Hybrid Application Platform dependent Cross platform Run on the device's internal software and hardware Built using HTML5 and JavaScript and wrapped inside a thin native container that provides access to native platform features Need more developers to build apps on different platforms and learning time is more Save development cost and time Excellent performance Less performance The native and hybrid applications are shown as follows: Let's see some of the existing methods in Robotium that support access to web content. They are as follows: searchText (String text) scrollUp/Down () clickOnText (String text) takeScreenshot () waitForText (String text) In the methods specifically added for web support, the class By is used as a parameter. It is an abstract class used as a conjunction with the web methods. These methods are used to select different WebElements by their properties, such as ID and name. The element used in a web view is referred to as a WebElement. It is similar to the WebDriver implemented in Selenium. The following table lists all the methods inside the class By: Method Description className (String className) Select a WebElement by its class name cssSelector (String selectors) Select a WebElement by its CSS selector getValue () Return the value id (String id) Select a WebElement by its id name (String name) Select a WebElement by its name tagName (String tagName) Select a WebElement by its tag name textContent (String textContent) Select a WebElement by its text content xpath (String xpath) Select a WebElement by its xpath Some of the important methods in the Robotium framework, that aim at direct communication with web content in Android applications, are listed as follows: clickOnWebElement(By by): It clicks on the WebElement matching the specified By class object. waitForWebElement(By by): It waits for the WebElement matching the specified By class object. getWebElement(By by, int index): It returns a WebElement matching the specified By class object and index. enterTextInWebElement(By by, String text): It enters the text in a WebElement matching the specified By class object. typeTextInWebElement(By by): It types the text in a WebElement matching the specified By class object. In this method, the program actually types the text letter by letter using the keyboard, whereas enterTextInWebElement directly enters the text in the particular. clearTextInWebElement(By by): It clears the text in a WebElement matching the specified By class object. getCurrentWebElements(By by): It returns the ArrayList of WebElements displayed in the active web view matching the specified By class object. Before actually looking into the hybrid test example, let's gain more information about WebViews. You can get an instance of WebView using the Solo class as follows: WebView wb = solo.getCurrentViews(WebView.class).get(0); Now that you have control of WebView, you can inject your JavaScript code as follows: Wb.loadUrl("<JavaScript>"); This is very powerful, as we can call every function on the current page; thus, it helps automation. Robotium Remote Control using SAFS SAFS tests are not wrapped up as JUnit tests and the SAFS Remote Control of Robotium uses an implementation that is NOT JUnit based. Also, there is no technical requirement for a JUnit on the Remote-Control side of the test. The test setup and deployment of the automation of the target app can be achieved using the SDK tools. These tools are used as part of the test runtime such as adb and aapt. The existing packaging tools can be used to repackage a compiled Robotium test with an alternate AndroidManifest.xml file, which can change the target application at runtime. SAFS is a general-purpose, data-driven framework. The only thing that should be provided by the user is the target package name or APK path arguments. The test will extract and redeploy the modified packages automatically and then launch the actual test. Traditional JUnit/Robotium users might not have, or see the need for, this general-purpose nature, but that is likely because it was necessary for the previous Android tests to be JUnit tests. It is required for the test to target one specific application. The Remote Control application is application specific. That's why the test app with the Remote Control installed in the device no longer needs to be an application. The Remote Control in Robotium means there are two test applications to build for any given test. They are as follows: Traditional on-device Robotium/JUnit test app Remote Control app These two build projects have entirely different dependencies and build scripts. The on-device test app has the traditional Robotium/Android/JUnit dependencies and build scripts, while the Remote Control app only has dependencies on the TCP sockets for communications and Robotium Remote Control API. The implementation for the remote-controlled Robotium can be done in the following two pieces: On Device: ActivityInstrumentationTestCase2.setup() is initialized when Robotium's Solo class object is to be used for the RobotiumTestRunner (RTR). The RTR has a Remote Control Listener and routes remote control calls and data to the appropriate Solo class methods and returns any results, as needed, to the Remote Controller. The on-device implementation may exploit test-result asserts if that is desirable. Remote Controller: The RemoteSolo API duplicates the traditional Solo API, but its implementation largely pushes the data through the Remote Control to the RTR, and then receives results from the Remote Controller. The Remote Control implementation may exploit any number of options for asserting, handling, or otherwise reporting or tracking the test results for each call. As you can see, the Remote-Control side only requires a RemoteSolo API without any specific JUnit context. It can be wrapped in a JUnit context if the tester desires it, but it is not necessary to be in a JUnit context. The sample code and installation of Robotium Remote Control can be accessed in the following link: http://code.google.com/p/robotium/wiki/RemoteControl Summary Thus this article introduced us to the Robotium framework, its different features, its benefits in the world of automated testing, the API set of the Robotium Framework, and how to implement the Robotium Remote Control using SAFS. Resources for Article: Further resources on this subject: So, what is Spring for Android? [Article] Android Native Application API [Article] Introducing an Android platform [Article]

0
0
6648

Building Responsive Image Sliders

Securing the Hadoop Ecosystem

Understanding outside-in

Understanding Web-based Applications and Other Multimedia Forms

Learning Data Analytics with R and Hadoop

Using memcached with Python

Getting Started with Fortigate: Troubleshooting

Limits of Game Data Analysis

Reducing Data Size

Foundations

Trending Topics

Issues and Wikis in GitLab

Styling the Forms

Overview of Process Management in Microsoft Visio 2013

Setting Up NameNode HA

Automated testing using Robotium

Create a Free Account To Continue Reading

Sign in to activate your 7-day free access