How-To Tutorials

(For more resources related to this topic, see here.) One of the most frequently asked questions on help forums probably is, "How can I learn Google Apps Script?" The answer is almost always the same: learn JavaScript and follow the numerous tutorials available on the Internet. No doubt, it is one of the possible ways to learn but it is also one of the most difficult ways. I shall express my opinion on that subject at the end of this article, but let us first summarize what we really need to be able to use Google Apps Script efficiently. The first and most important thing we must have is a clear idea of what we want to achieve. This seems a bit silly because we think, "Oh well, of course I know what I want; I just don't know how to do it!" As a matter of fact, this is often not the case. Let us have an example: a colleague asked me recently how he could count the time he was spending at school for meetings and other administrative tasks, not taking into account his hours as a teacher. This was supposed to be a simple problem as everyone in our school has a personal calendar in which all the events that we are invited to are recorded. So, he began to search for a way to collect every possible event from his calendar to a spreadsheet and from there—since he can definitely use a spreadsheet—he intended to do some data filtering to get the result he wanted. I told him to have a look at the Google Apps Script documentation and see what tools he had, to pick up data from calendars and import them into a spreadsheet. A few days later, he came back to me complaining that he didn't find any appropriate methods to do what he needs to. And, in a way, he was right; nowhere is such a workflow explained and it is actually not surprising. One can't imagine compiling all the possible workflows into a single help resource; there are definitely too many different use cases, each of them needing a particular approach. We had a discussion where I told him to think about his research as a series of simple and accurate parts and steps before trying to get the whole process in one stroke. The following is what he told me another few days later: "I knew nothing about this macro language, so I discovered that it is based on JavaScript with the addition of Google's own services that use a similar syntax and that the whole thing is composed of functions calling each other and having parameters. Then, I examined the calendar service and saw that it needs so-called date objects to choose a start and end date. Date object methods are pretty well explained on Mozilla's page, so once I got that I had an array of events, I thought what the heck is an array of objects? You gave me the link to this w3schools site, so I took a look at their definition; that was enough for me to go further and discover that I could use a loop to handle each event separately. Google documentation shows all the methods to get events details; that part was easy and now I have all my calendar events with dates, hours, description, title. All of it! I tell you." I'm not going to transcribe all of our conversation—it finally took a couple of hours—but towards the end, he was describing the process so well that the actual writing of his script was almost just a formality. With the help of the Content assist (autocomplete) feature of the script editor and a couple of browser tabs left open on JavaScript and Google documentation, he managed to write his script in one day. Of course, the script was not perfect and by no way optimized his speed or gave  nice-looking results, but it worked and he had the data he was looking for. At that point, he could post his script on a help forum if something went wrong or try to improve another version if he's a perfectionist, but that depends only on his will to go further or not. I would simply say one thing: you will learn what you need. If you don't need it, don't try to learn it as you will forget it faster than you learned it. If you do, then be prepared to need something else right after; it is an endless journey! JavaScript versus Google Apps Script The following is stated on the overview of Google Apps Script documentation page: Google Apps Script is a scripting language based on JavaScript that lets you do new and cool things with Google Apps like Docs, Sheets, and Forms. They should use a bigger typeface to make it more visible! The keyword here is based on JavaScript because it does indeed use most of the JavaScript Version 1.6 (with some portions of Version 1.7 and Version 1.8) and its general syntax. But, it has so many other methods that knowing only JavaScript is clearly not sufficient to use it adequately. I would even say that you can learn it step-by-step when you need it, looking for information on a specific item each time you use a new type of object. The following is the code that was used to get the integer part of the result that uses the getTime method: function myAgeInHours(){  var myBirthDate = new Date('1958/02/19 02:00:00').getTime();  myBirthDate = parseInt(myBirthDate/3600000, 10);  var today = parseInt(new Date().getTime()/3600000, 10);  return today-myBirthDate;} We looked at the documentation about the Date object to find the getTime() method and then found parseInt to get the integer part of the result. Well, I'm convinced that this approach is more efficient than spending hours on a site or in a book that shows all JavaScript information from A to Z. We have the opportunity to have powerful search engines in our browsers, so let's use them; they always find the answer for us in less time than it takes to write the question. Concerning methods specific to Google Apps Script, I think the approach should be different. The Google API documentation is pretty well organized and is full of code examples that clearly show us how to use almost every single method. If we start a project in a spreadsheet, it is a good idea to carefully read the section about spreadsheets (https://developers.google.com/apps-script/reference/spreadsheet) at least once and just check if what it says makes any sense. For example, in the Sheet class, I found this description: Returns the range with the top left cell at the given coordinates, and with the given number of rows. The following screenshot displays the same description: If I understand what range and co-ordinates are, then I probably know enough to be able to use that method (getRange(row, column, numRows) or a similar one. You want me to tell you the truth? I didn't know we could get a range this way by simply defining the top-left cell and just the number of rows (only three parameters). I always use the next one in the list, which is shown as follows: The description says: Returns the range with the top left cell at the given coordinates with the given number of rows and columns. So after all this time I spent on dozens of spreadsheet scripts, there still are methods that I can't even imagine exist! That's actually a nice confirmation of what I was suggesting: one doesn't need to know everything to be able to use it but it's always a good idea to read the docs from time to time. Infinite resources JavaScript is a very popular language; there are thousands of websites that show us examples and explain methods and functions. We must add all the forums and Q&A sites that return many results when we search something on Google to these websites (or any other search engine), and that is actually an unforeseen difficulty. It happens quite often that we find false information or code snippets that simply don't work, either because they have typos in them or they are so badly written that they work only in a very specific and peculiar situation. My personal solution is to use only a couple of websites and perform a search on their search engine, avoiding all the sources I'm not sure of. Maybe I miss something at times, but at least the information I get is trustworthy. Last but certainly not least, the help forum recommended by the Google Apps Script team, http://stackoverflow.com/questions/tagged/google-apps-script (with the google-apps-script tag), is certainly the best resource that is available. With more than 5000 questions (as of January, 2014), the help forum probably has threads about every possible use case and an important part of it has answers as well. There are of course other interesting tags: JavaScript, Google docs, Google spreadsheets, and a few even more specific ones. I have rarely seen really bad answers—although it does happen sometimes—simply because so many people read these posts that they generally flag or comment answers that show wrong information. There are also people from Google that regularly keep an eye on it and clarify any ambiguous response. Being a newbie is, by definition, temporary When I began to use Google spreadsheets and scripts, I found the Google Group Help forum (which does not exist anymore) an invaluable source of information and help, so I asked dozens of questions—some of them very basic and naive—and always got answers. After a while, since I was spending hours on this forum reading about every post I found, I began to answer too. I was so proud of being able to answer a question! It was almost like passing an examination; I knew that one of the experts there was going to read what I wrote and evaluate my knowledge; quite stressful but also satisfying when you don't fail! So after a couple of months I gained my first level point (on the Google Group forum, there are no reputation points but levels, starting from 1 for new arriving members up to TC(Top Contributors), whose level is unknown but is generally more than 15 or 20; anyway, that's not important). That little story is just a way to encourage any beginner to spend some time on this forum and consider every question as a challenge and try to answer it. Of course, there is no need to publish your answer every time as there are chances that you may get it all wrong, but just use this as an exercise that will give you more and more expertise. From time to time, you'll be able to be the first or best answerer and gain a few reputation points; consider it as a game, just a funny game where all you can finally win is knowledge and all you can lose is your newbie status—not a bad deal after all! Try to find your own best learning method I'm certainly not pretending that I know the best learning method for anyone. All the tips I presented in the previous section did work for me—and for a few other people I know—but there is no magic formula that would suit everyone. I know that each of us has a different background and follows a different path, but I wanted to say loud and clear that you don't need to have to be a graduate in IT to begin with Google Apps Script nor do you have to spend hours learning rules and conventions. Practice will make it easier everyday and motivation will give you enough energy to complete your projects, from simple ones to more ambiguous ones. Summary This article has given an overview of the many resources available to improve your learning experience. There are certainly more that I don't know of but as I already mentioned a few times before, we have powerful search engines in our browsers to help us. We also have to keep in mind that Google Apps Script will probably be different as compared to what it will be in a couple of years. Resources for Article: Further resources on this subject: Google Apps: Surfing the Web [Article] Developing apps with the Google Speech APIs [Article] Data Modeling and Scalability in Google App [Article]

(For more resources related to this topic, see here.) Shipping ordered products to customers is one of the key parts of the e-commerce flow. In most cases, a shop owner has a contract with a shipping handler where everyone has their own business rules. In a standard Magento, the following shipping handlers are supported: DHL FedEx UPS USPS If your handler is not on the list, have a look if there is a module available at Magento Connect. If not, you can configure a standard shipping method or you can create your own, which we will do in this article. Initializing module configurations In this recipe, we will create the necessary files for a shipping module, which we will extend with more features using the recipes of this article. Getting ready Open your code editor with the Magento project. Also, get access to the backend where we will check some things. How to do it... The following steps describe how we can create the configuration for a shipping module: Create the following folders: app/code/local/Packt/ app/code/local/Packt/Shipme/ app/code/local/Packt/Shipme/etc/ app/code/local/Packt/Shipme/Model/ app/code/local/Packt/Shipme/Model/Carrier Create the module file named Packt_Shipme.xml in the folder app/etc/modules with the following content: <?xml version="1.0"?> <config> <modules> <Packt_Shipme> <active>true</active> <codePool>local</codePool> <depends> <Mage_Shipping /> </depends> </Packt_Shipme> </modules> </config> Create a config.xml file in the folder app/code/local/Packt/Shipme/etc/ with the following content: <?xml version="1.0" encoding="UTF-8"?> <config> <modules> <Packt_Shipme> <version>0.0.1</version> </Packt_Shipme> </modules> <global> <models> <shipme> <class>Packt_Shipme_Model</class> </shipme> </models> </global> <default> <carriers> <shipme> <active>1</active> <model>shipme/carrier_shipme</model> <title>Shipme shipping</title> <express_enabled>1</express_enabled> <express_title>Express delivery</express_title> <express_price>4</express_price> <business_enabled>1</business_enabled> <business_title>Business delivery</business_title> <business_price>5</business_price> </shipme> </carriers> </default> </config> Clear the cache and navigate in the backend to System | Configuration | Advanced Disable Modules Output. Observe that the Packt_Shipme module is on the list. At this point, the module is initialized and working. Now, we have to create a system.xml file where we will put the configuration parameters for our shipping module. Create the file app/code/local/Packt/Shipme/etc/system.xml. In this file, we will create the configuration parameters for our shipping module. When you paste the following code in the file, you will create an extra group in the shipping method's configuration. In this group, we can set the settings for the new shipping method: <?xml version="1.0" encoding="UTF-8"?> <config> <sections> <carriers> <groups> <shipme translate="label" module="shipping"> <label>Shipme</label> <sort_order>15</sort_order> <show_in_default>1</show_in_default> <show_in_website>1</show_in_website> <show_in_store>1</show_in_store> <fields> <!-- Define configuration fields below --> <active translate="label"> <label>Enabled</label> <frontend_type>select</frontend_type> <source_model>adminhtml/ system_config_source_yesno</source_model> <sort_order>10</sort_order> <show_in_default>1</show_in_default> <show_in_website>1</show_in_website> <show_in_store>1</show_in_store> </active> <title translate="label"> <label>Title</label> <frontend_type>text</frontend_type> <sort_order>20</sort_order> <show_in_default>1</show_in_default> <show_in_website>1</show_in_website> <show_in_store>1</show_in_store> </title> </fields> </shipme> </groups> </carriers> </sections> </config> Clear the cache and navigate in the backend to the shipping method configuration page. To do that, navigate to System | Configuration | Sales | Shipping methods. You will see that an extra group is added as shown in the following screenshot: You will see that there is a new shipping method called Shipme. We will extend this configuration with some values. Add the following code under the <fields> tag of the module: <active translate="label"> <label>Enabled</label> <frontend_type>select</frontend_type> <source_model>adminhtml/system_config_source_yesno</source_ model> <sort_order>10</sort_order> <show_in_default>1</show_in_default> <show_in_website>1</show_in_website> <show_in_store>1</show_in_store> </active> <title translate="label"> <label>Title</label> <frontend_type>text</frontend_type> <sort_order>20</sort_order> <show_in_default>1</show_in_default> <show_in_website>1</show_in_website> <show_in_store>1</show_in_store> </title> <express_enabled translate="label"> <label>Enable express</label> <frontend_type>select</frontend_type> <source_model>adminhtml/system_config_source_yesno</source_ model> <sort_order>30</sort_order> <show_in_default>1</show_in_default> <show_in_website>1</show_in_website> <show_in_store>1</show_in_store> </express_enabled> <express_title translate="label"> <label>Title express</label> <frontend_type>text</frontend_type> <sort_order>40</sort_order> <show_in_default>1</show_in_default> <show_in_website>1</show_in_website> <show_in_store>1</show_in_store> </express_title> <express_price translate="label"> <label>Price express</label> <frontend_type>text</frontend_type> <sort_order>50</sort_order> <show_in_default>1</show_in_default> <show_in_website>1</show_in_website> <show_in_store>1</show_in_store> </express_price> <business_enabled translate="label"> <label>Enable business</label> <frontend_type>select</frontend_type> <source_model>adminhtml/system_config_source_yesno</source_ model> <sort_order>60</sort_order> <show_in_default>1</show_in_default> <show_in_website>1</show_in_website> <show_in_store>1</show_in_store> </business_enabled> <business_title translate="label"> <label>Title business</label> <frontend_type>text</frontend_type> <sort_order>70</sort_order> <show_in_default>1</show_in_default> <show_in_website>1</show_in_website> <show_in_store>1</show_in_store> </business_title> <business_price translate="label"> <label>Price business</label> <frontend_type>text</frontend_type> <sort_order>80</sort_order> <show_in_default>1</show_in_default> <show_in_website>1</show_in_website> <show_in_store>1</show_in_store> </business_price> Clear the cache and reload the backend. You will now see the other configurations under the Shipme – Express shipping method as shown in the following screenshot: How it works... The first thing we have done is to create the necessary files to initialize the module. The following files are required to initialize a module: app/etc/modules/Packt_Shipme.xml app/code/local/Packt/Shipme/etc/config.xml In the first file, we will activate the module with the <active> tag. The <codePool> tag describes that the module is located in the local code pool, which represents the folder app/code/local/. In this file, there is also the <depends> tag. First this will check if the Mage_Shipping module is installed or not. If not, Magento will throw an exception. If the module is available, the dependency will load this module after the Mage_Shipping module. This makes it possible to rewrite some values from the Mage_Shipping module. In the second file, config.xml, we configured all the stuff that we will need in this module. These are the following things: The version number (0.0.1) The models Some default values for the configuration values The last thing we did was create a system.xml file so that we can create a custom configuration for the shipping module. The configuration in the system.xml file adds some extra values to the shipping method configuration, which is available in the backend under the menu System | Configuration | Sales | Shipping methods. In this module, we created a new shipping handler called Shipme. Within this handler, you can configure two shipping options: express and business. In the system.xml file, we created the fields to configure the visibility, name, and price of the options. See also In this recipe, we used the system.xml file of the module to create the configuration values. Writing an adapter model A new shipping module is initialized in the previous recipe. What we did in the previous recipe was a preparation to continue with the business part we will see in this recipe. We will add a model with the business logic for the shipping method. The model is called an adapter class because Magento requires an adapter class for each shipping method. This class will extend the Mage_Shipping_Model_Carrier_Abstract class. This class will be used for the following things: Make the shipping method available Calculate the shipping costs Set the title in the frontend of the shipping methods How to do it... Perform the following steps to create the adapter class for the shipping method: Create the folder app/code/local/Packt/Shipme/Model/Carrier if it doesn't already exist. In this folder, create a file named Shipme.php and add the following content to it: <?php class Packt_Shipme_Model_Carrier_Shipme extends Mage_Shipping_Model_Carrier_Abstract implements Mage_Shipping_Model_Carrier_Interface { protected $_code = 'shipme'; public function collectRates (Mage_Shipping_Model_Rate_Request $request) { $result = Mage::getModel('shipping/rate_result'); //Check if express method is enabled if ($this->getConfigData('express_enabled')) { $method = Mage::getModel ('shipping/rate_result_method'); $method->setCarrier($this->_code); $method->setCarrierTitle ($this->getConfigData('title')); $method->setMethod('express'); $method->setMethodTitle ($this->getConfigData('express_title')); $method->setCost ($this->getConfigData('express_price')); $method->setPrice ($this->getConfigData('express_price')); $result->append($method); } //Check if business method is enabled if ($this->getConfigData('business_enabled')) { $method = Mage::getModel ('shipping/rate_result_method'); $method->setCarrier($this->_code); $method->setCarrierTitle ($this->getConfigData('title')); $method->setMethod('business'); $method->setMethodTitle ($this->getConfigData('business_title')); $method->setCost ($this->getConfigData('business_price')); $method->setPrice ($this->getConfigData('business_price')); $result->append($method); } return $result; } public function isActive() { $active = $this->getConfigData('active'); return $active==1 || $active=='true'; } public function getAllowedMethods() { return array('shipme'=>$this->getConfigData('name')); } } Save the file and clear the cache; your adapter model has now created. How it works... The previously created class handles all the business logic that is needed for the shipping method. Because this adapter class is an extension of the Mage_Shipping_Model_Carrier_Abstract class, we can overwrite some methods to customize the business logic of the standard. The first method we overwrite is the isAvailable() function. In this function, we have to return true or false to say that the module is active. In our code, we will activate the module based on the system configuration field active. The second method is the collectRates() function. This function is used to set the right parameters for every shipping method. For every shipping method, we can set the title and price. The class implements the interface Mage_Shipping_Model_Carrier_Interface. In this interface, two functions are declared: the isTrackingAvailable() and getAllowedMethods() functions. We created the function getAllowedMethods() in the adapter class. The isTrackingAvailable() function is declared in the parent class Mage_Shipping_Model_Carrier_Abstract. We configured two options under the Shipme shipping method. These options are called Express delivery and Business delivery. We will check if they are enabled in the configuration and set the configured title and price for each option. The last thing to do is return the right values. We have to return an instance of the class Mage_Shipping_Model_Rate_Result. We created an empty instance of the class, where we will append the methods to when they are available. To add a method, we have to use the function append($method). This function requires an instance of the class Mage_Shipping_Model_Rate_Result_Method that we created in the two if statements.

(For more resources related to this topic, see here.) A flexible penetration testing distribution BackBox Linux is a very young project designed for penetration testing, vulnerability assessment and management. The key focus in using BackBox is to provide an independent security testing platform that can be easily customized with increased performance and stability. BackBox uses a very light desktop manager called XFCE. It includes the most popular security auditing tools that are essential for penetration testers and security advisers. The suite of tools includes web application analysis, network analysis, stress tests, computer sniffing forensic analysis, exploitation, documentation, and reporting. The BackBox repository is hosted on Launchpad and is constantly updated to the latest stable version of its tools. Adding and developing new tools inside the distribution requires it to be compliant with the open source community and particularly the Debian Free Software Guidelines criteria. IT security and penetration testing are dedicated sectors and quite new in the global market. There are a lot of Linux distributions dedicated to security; but if we do some research, we can see that only a couple of distributions are constantly updated. Many newly born projects stop at the first release without continuity and very few of them are updated. BackBox is one of the new players in this field and even though it is only a few years old, it has acquired an enormous user base and now holds the second place in worldwide rankings. It is a lightweight, community-built penetration testing distribution capable of running live in USB mode or as a permanent installation. BackBox now operates on release 3.09 as of September 2013, with a significant increase in users, thus becoming a stable community. BackBox is also significantly used in the professional world. BackBox is built on top of Ubuntu LTS and the 3.09 release uses 12.04 as its core. The desktop manager environment with XFCE and the ISO images are provided for 32-bit and 64-bit platforms (with the availability on Torrents and HTTP downloads from the project's website). The following screenshot shows the main view of the desktop manager, XFCE: The choice of desktop manager, XFCE, plays a very important role in BackBox. It is not only designed to serve the slender environment with medium and low level of resources, but also designed for very low memory. In case of very low memory and other resources (such as CPU, HD, and video), BackBox has an alternative way of booting the system without graphical user interface (GUI) and using command-line only, which requires really minimal amount of resources. With this aim in mind, BackBox is designed to function with pretty old and obsolete hardware to be used as a normal auditing platform. However, BackBox can be used on more powerful systems to perform actions that require the modern multicore processors to reduce ETA of the task such as brute-force attacks, data/password decryption, and password-cracking. Of course, the BackBox team aims to minimize overhead for the aforementioned cases through continuous research and development. Luckily, the majority of the tools included in BackBox can be performed in a shell/console environment and for the ones which require less resource. However, we always have our XFCE interface where we can access user-friendly GUI tools (in particular network analysis tools), which do not require many resources. Relatively, newcomer into the IT security and penetration testing environment, the first release of BackBox was back in September 09, 2010, as a project of the Italian web community. Now on its third major release and close to the next minor release (BackBox Linux 3.13 is planned for the end of January 2014), BackBox has grown rapidly and offers a wide scope for both amateur and professional use. The minimum requirements for BackBox are as follows: A 32-bit or 64-bit processor 512 MB of system memory RAM (256 MB in case there will be no desktop manager usage and only the console) 4.4 GB of disk space for installation Graphics card capable of 800 × 600 resolution (less resolution in case there will be no desktop manager usage) DVD-ROM drive or USB port The following screenshot shows the main view of BackBox with a toolbar at the bottom: The suite of auditing tools in BackBox makes the system complete and ready to use for security professionals of penetration testing. The organization of tools in BackBox. The entire set of BackBox security tools are populated into a single menu called Audit and structured into different subtasks as follows: Information Gathering Vulnerability Assessment Exploitation Privilege Escalation Maintaining Access Documentation & Reporting Social Engineering Stress Testing Forensic Analysis VoIP Analysis Wireless Analysis Miscellaneous We have to run through all the tools in BackBox by giving a short description of each single tool in the Auditing menu. The following screenshot shows the Auditing menu of BackBox: Information Gathering Information Gathering is the first absolute step of any security engineer and/or penetration tester. It is about collecting information on target systems, which can be very useful to start the assessment. Without this step, it will be quite difficult and hard to assess any system. Vulnerability Assessment After you've gathered information by performing the first step, the next step will be to analyze that information and its evaluation. Vulnerability Assessment is the process of identifying the vulnerabilities present in the system and prioritizing them. Exploitation Exploitation is the process where the weakness or bug in the software is used to penetrate the system. This can be done through the usage of an exploit, which is nothing but an automated script that is designed to perform a malicious attack on target systems. Privilege Escalation Privilege Escalation occurs when we have already gained access to the system but with low privileges. It can also be that we have legitimate access but not enough to make effective changes on the system, so we will need to elevate our privileges or gain access to another account with higher privileges. Maintaining Access Maintaining Access is about setting up an environment that will allow us to access the system again without repeating the tasks that we performed to gain access initially. Documentation & Reporting The Documentation & Reporting menu contains the tools that will allow us to collect the information during our assessment and generate a human readable report from them. Reverse Engineering The Reverse Engineering menu contains the suite of tools aimed to reverse the system by analyzing its structure for both hardware and software. Social Engineering Social Engineering is based on a nontechnical intrusion method, mainly on human interaction. It is the ability to manipulate the person and obtain his/her access credentials or the information that can introduce us to such parameters. Stress Testing The Stress Testing menu contains a group of tools aimed to test the stress level of applications and servers. Stress testing is the action where a massive amount of requests (for example, ICMP request) are performed against the target machine to create heavy traffic to overload the system. In this case, the target server is under severe stress and can be taken advantage of. For instance, the running services such as the web server, database or application server (for example, DDoS attack) can be taken down. Forensic Analysis The Forensic Analysis menu contains a great amount of useful tools to perform a forensic analysis on any system. Forensic analysis is the act of carrying out an investigation to obtain evidence from devices. It is a structured examination that aims to rebuild the user's history in a computer device or a server system. VoIP Analysis The voice over IP (VoIP) is a very commonly used protocol today in every part of the world. VoIP analysis is the act of monitoring and analyzing the network traffic with a specific analysis of VoIP calls. So in this section, we have a single tool dedicated to the analysis of VoIP systems. Wireless Analysis The Wireless Analysis menu contains a suite of tools dedicated to the security analysis of wireless protocols. Wireless analysis is the act of analyzing wireless devices to check their safety level. Miscellaneous The Miscellaneous menu contains tools that have different functionalities and can be placed in any section that we mentioned earlier, or in none of them. Services Apart from the Auditing menu, BackBox also has a Services menu. This menu is designed to populate the daemons of the tools, those which need to be manually initialized as a service. Update We have the Update menu that can be found in the main menu, just next to the Services menu. The Update menu contains the automated scripts to allow the users to update the tools that are out of APT automated system. Anonymous BackBox 3.13 has a new menu voice called Anonymous in the main menu. This menu contains a script that makes the user invisible to the network once started. The script populates a set of tools that anonymize the system while navigating, and connects to the global network, Internet. Extras Apart from the security-auditing tools, BackBox also has several privacy-protection tools. The suite of privacy-protection tools includes Tor, Polipo, and the Firefox safe mode that have been configured with a default profile in the private-browsing mode. There are many other useful tools recommended by the team but they are not included in the default ISO image. Therefore, the recommended tools are available in the BackBox repository and can be easily installed with apt-get (automated package installation tool for Debian-like systems). Completeness, accuracy, and support It is obvious that there are many alternatives when it comes to the choice of penetration testing tools for any particular auditing process. The BackBox team is mainly focused on the size of the tool library, performance, and the inclusion of the tools for security and auditing. The amount of tools included in BackBox is subject to accurate selection and testing by a team. Most of the security and penetration testing tools are implemented to perform identical functions. The BackBox team is very careful in the selection process in order to avoid duplicate applications and redundancies. Besides the wiki-based documentation provided for its set of tools, the repository of BackBox can also be imported into any of existing Ubuntu installation (or any of Debian derivative distro) by simply importing the project's Launchpad repository to the source list. Another point that the BackBox team focus their attention on is the size issue. BackBox may not offer the largest number of tools and utilities, but numbers are not equal to the quality. It has the essential tools installed by default that are sufficient to a penetration tester. However, BackBox is not a perfect penetration testing distribution. It is a very young project and aims to offer the best solution to the global community. Links and contacts BackBox is an open community where everybody's help is greatly welcomed. Here is a list of useful links to BackBox information on the Web: The BackBox main and official web page, where we can find general information about the distribution and the organization of the team, is available at http://www.BackBox.org/ The BackBox official blog, where we can find news about BackBox such as release notes and bug correction notifications, is available at http://www.BackBox.org/blog The BackBox official wikipage, where we can find many tutorials for the tools usage that are included in the distribution, is available at http://wiki.BackBox.org/ The BackBox official forum is the main discussion forum, where users can post their problems and also suggestions, is available at http://forum.BackBox.org/ The BackBox official IRC chat room is available at https://kiwiirc.com/client/irc.autistici.org:6667/?nick=BackBox_?#BackBox The BackBox official repository hosted on Launchpad, where the entire packages are located, is available at https://launchpad.net/~BackBox BackBox has also a Wikipedia page, where we can run through a brief history about how the project began, which is available at http://en.wikipedia.org/wiki/BackBox Summary In this article, we became more familiar with the BackBox environment by analyzing its menu structure and the way its tools are organized. We also provided a quick comment on each tool in BackBox. This is the only theoretical information regarding the introduction of BackBox. Resources for Article: Further resources on this subject: Penetration Testing and Setup [article] BackTrack 4: Security with Penetration Testing Methodology [article] Web app penetration testing in Kali [article]

(For more resources related to this topic, see here.) The following is an image of a finished project: Even though you've made your robot mobile by adding wheels or tracks, this mobile platform will only work well on smooth, flat surfaces. Often, you'll want your robot to work in environments where the path is not smooth or flat; perhaps you'll even want your robot to go up stairs or around curbs. In this article, you'll learn how to attach your board, both mechanically and electrically, to a platform with legs so that your projects can be mobile in many more environments. Robots that can walk! What could be more amazing than that? In this article, we will cover the following topics: Connecting Raspberry Pi to a two-legged mobile platform using a servo motor controller Creating a program in Linux so that you can control the movement of the two-legged mobile platform Making your robot truly mobile by adding voice control Gathering the hardware In this article, you'll need to add a legged platform to make your project mobile. For a legged robot, there are a lot of choices for hardware. Some are completely assembled, others require some assembly, and you may even choose to buy the components and construct your own custom mobile platform. Also I'm going to assume that you don't want to do any soldering or mechanical machining yourself, so let's look at several choices of hardware that are available completely assembled or can be assembled using simple tools (a screwdriver and/or pliers). One of the simplest legged mobile platforms is one that has two legs and four servo motors. The following is an image of this type of platform: We'll use this legged mobile platform in this article because it is the simplest to program and the least expensive, requiring only four servos. To construct this platform, you must purchase the parts and then assemble them yourself. Find the instructions and parts list at http://www.lynxmotion.com/images/html/build112.htm. Another easy way to get all the mechanical parts (except servos) is by purchasing a biped robot kit with six DOF (degrees of freedom). This will contain the parts needed to construct your four-servo biped. These six DOF bipeds can be purchased on eBay or at http://www.robotshop.com/2-wheeled-development-platforms-1.html. You'll also need to purchase the servo motors. Servo motors are designed to move at specific angles based on the control signals that you send. For this type of robot, you can use standard-sized servos. I like the Hitec HS-311 or HS-322 for this robot. They are inexpensive but powerful enough in operations. You can get them on Amazon or eBay. The following is an image of an HS-311 servo: You'll need a mobile power supply for Raspberry Pi. I personally like the 5V cell phone rechargeable batteries that are available at almost any place that supplies cell phones. Choose one that comes with two USB connectors; you can use the second port to power your servo controller. The mobile power supply shown in the following image mounts well on the biped hardware platform: You'll also need a USB cable to connect your battery to Raspberry Pi. You should already have one of those. Now that you have the mechanical parts for your legged mobile platform, you'll need some hardware that will turn the control signals from your Raspberry Pi into voltage levels that can control the servo motors. Servo motors are controlled using a signal called PWM. For a good overview of this type of control, see http://pcbheaven.com/wikipages/How_RC_Servos_Works/ or https://www.ghielectronics.com/docs/18/pwm. You can find tutorials that show you how to control servos directly using Raspberry Pi's GPIO (General Purpose Input/Output) pins, for example, those at http://learn.adafruit.com/adafruit-16-channel-servo-driver-with-raspberry-pi/ and http://www.youtube.com/watch?v=ddlDgUymbxc. For ease of use, I've chosen to purchase a servo controller that can talk over a USB and control the servo motor. These controllers protect my board and make controlling many servos easy. My personal favorite for this application is a simple servo motor controller utilizing a USB from Pololu that can control six servo motors—the Micro Maestro 6-Channel USB Servo Controller (Assembled). The following is an image of the unit: Make sure you order the assembled version. This piece of hardware will turn USB commands into voltage levels that control your servo motors. Pololu makes a number of different versions of this controller, each able to control a certain number of servos. Once you've chosen your legged platform, simply count the number of servos you need to control and choose a controller that can control that many servos. In this article, we will use a two-legged, four-servo robot, so I will illustrate the robot using the six-servo version. Since you are going to connect this controller to Raspberry Pi via USB, you'll also need a USB A to mini-B cable. You'll also need a power cable running from the battery to your servo controller. You'll want to purchase a USB to FTDI cable adapter that has female connectors, for example, the PL2303HX USB to TTL to UART RS232 COM cable available on amazon.com. The TTL to UART RS232 cable isn't particularly important, other than that the cable itself provides individual connectors to each of the four wires in a  USB cable. The following is an image of the cable: Now that you have all the hardware, let's walk through a quick tutorial of how a two-legged system with servos works and then some step-by-step instructions to make your project walk. Connecting Raspberry Pi to the mobile platform using a servo controller Now that you have a legged platform and a servo motor controller, you are ready to make your project walk! Before you begin, you'll need some background on servo motors. Servo motors are somewhat similar to DC motors. However, there is an important difference: while DC motors are generally designed to move in a continuous way, rotating 360 degrees at a given speed, servo motors are generally designed to move at angles within a limited set. In other words, in the DC motor world, you generally want your motors to spin at a continuous rotation speed that you control. In the servo world, you want to control the movement of your motor to a specific position. For more information on how servos work, visit http://www.seattlerobotics.org/guide/servos.html or http://www.societyofrobots.com/actuators_servos.shtml. Connecting the hardware To make your project walk, you first need to connect the servo motor controller to the servos. There are two connections you need to make: the first is to the servo motors and the second is to the battery holder. In this section, you'll connect your servo controller to your PC or Linux machine to check to see whether or not everything is working. The steps for that are as follows: Connect the servos to the controller. The following is an image of your two-legged robot and the four different servo connections: In order to be consistent, let's connect your four servos to the connections marked 0 through 3 on the controller using the following configurations: 0: Left foot 1: Left hip 2: Right foot 3: Right hip The following is an image of the back of the controller; it will show you where to connect your servos: Connect these servos to the servo motor controller as follows: The left foot to the 0 to the top connector, the black cable to the outside (-) The left hip to the 1 connector, the black cable out The right foot to the 2 connector, the black cable out The right hip to the 3 connector, the black cable out See the following image indicating how to connect servos to the controller: Now you need to connect the servo motor controller to your battery. You'll use the USB to FTDI UART cable; plug the red and black cables into the power connector on the servo controller, as shown in the following image: Configuring the software Now you can connect the motor controller to your PC or Linux machine to see whether or not you can talk to it. Once the hardware is connected, you can use some of the software provided by Polulu to control the servos. It is easiest to do this using your personal computer or Linux machine. The steps to do so are as follows: Download the Polulu software from http://www.pololu.com/docs/0J40/3.a and install it based on the instructions on the website. Once it is installed, run the software; you should see the window shown in the following screenshot: You will first need to change the Serial mode configuration in Serial Settings, so select the Serial Settings tab; you should see the window shown in the following screenshot: Make sure that USB Chained is selected; this will allow you to connect to and control the motor controller over the USB. Now go back to the main screen by selecting the Status tab; now you can turn on the four servos. The screen should look as shown in the following screenshot: Now you can use the sliders to control the servos. Enable the four servos and make sure that the servo 0 moves the left foot, 1 the left hip, 2 the right foot, and 3 the right hip. You've checked the motor controllers and the servos and you'll now connect the motor controller to Raspberry Pi to control the servos from there. Remove the USB cable from the PC and connect it to Raspberry Pi. The entire system will look as shown in the following image: Let's now talk to the motor controller by downloading the Linux code from Pololu at http://www.pololu.com/docs/0J40/3.b. Perhaps the best way to do this is by logging on to Raspberry Pi using vncserver and opening a VNC Viewer window on your PC. To do this, log in to your Raspberry Pi using PuTTY and then type vncserver at the prompt to make sure vncserver is running. Then, perform the following steps: On your PC, open the VNC Viewer application, enter your IP address, and then click on Connect. Then, enter the password that you created for the vncserver; you should see the Raspberry Pi viewer screen, which should look as shown in the following screenshot: Open a Firefox browser window and go to http://www.pololu.com/docs/0J40/3.b. Click on the Maestro Servo Controller Linux Software link. You will need to download the file maestro_linux_100507.tar.gz to the Download directory. You can also use wget to get this software by typing wget http://www.pololu.com/file/download/maestro-linux-100507.tar.gz?file_id=0J315 in a terminal window. Go to your Download directory, move it to your home directory by typing mv maestro_linux_100507.tar.gz .. and then you can go back to your home directory. Unpack the file by typing tar –xzfv maestro_linux_011507.tar.gz. This will create a directory called maestro_linux. Go to that directory by typing cd maestro_linux and then type ls. You should see the output as shown in the following screenshot: The document README.txt will give you explicit instructions on how to install the software. Unfortunately, you can't run MaestroControlCenter on your Raspberry Pi. Our version of windowing doesn't support the graphics, but you can control your servos using the UscCmd command-line application. First, type ./UscCmd --list and you should see the following screenshot: The unit sees your servo controller. If you just type ./UscCmd, you can see all the commands you could send to your controller. When you run this command, you can see the result as shown in the following screenshot: Notice that you can send a servo a specific target angle, although if the target angle is not within range, it makes it a bit difficult to know where you are sending your servo. Try typing ./UscCmd --servo 0, 10. The servo will most likely move to its full angle position. Type ./UscCmd – servo 0, 0 and it will stop the servo from trying to move. If you haven't run the Maestro Controller tool and set the Serial Settings setting to USB Chained, your motor controller may not respond.

(For more resources related to this topic, see here.) Opening an individual document for translation To open a document for translation in SDL Trados Studio, perform the following steps: In any view, choose File | Open | Translate Single Document or press Ctrl + Shift + O. Browse to the file that you want to open for translation, select it, and click Open. Alternatively, you can open a file using drag and drop. You must be in the Editor view to do this. Drag the file from Windows Explorer into the Navigation pane, shown in the following screenshot: In the Open Document window, shown in the following screenshot, select the desired Source Language and Target Language. If you are using the sample file, please choose English (US) as your source language and a language of your choice as your target language. Select one or more TMs by clicking Add and browsing to select an existing TM. You can also choose to create a new TM at this point by clicking Create (choose New File-based Translation Memory, specify a Name and Location for the file, and click Finish). If you are working with our sample file, please create or select a TM of your own at this point. The following screenshot shows the Open Document window after we add the TM: For any other settings, click the Advanced button at the bottom-left corner of the window. Click OK to open the document for translation in the side-by-side editor. Translating in the side-by-side editor The side-by-side editor is made up of five columns, numbered in the following screenshot: At the top left is a tab showing the name of the active document. The numbers in circles represent the following: Column 1: The segment number. Column 2: The source text, divided into segments when you open the file for translation. Column 3: The segment status and translation origin, indicating what work you have done on each segment at any given point in time, and where the match came from. The icons have the following meanings: Column 4: Where you type the translation. Column 5: Information to indicate the context of each segment within the structure of the original document. For example, in the sample file, the H in Segment 1 shows that the text is formatted as a heading in the original MS Word document. To find out what the icons in the segment status column and the information in the document structure column mean, move your mouse pointer over that part of the segment to display a tool tip or click on it for more detailed information. Translating the text To begin translating, click in the first target segment and type the translation. As soon as you start typing, the status symbol changes from (Not Translated) to (Draft), showing that you have edited the segment but not stored it in the TM yet, as shown in the following screenshot. Segment 1 of the sample file is a heading, as indicated by the letter H on the right. Notice that the visual formatting of the text as displayed by SDL Trados Studio is replicated when you type the translation. When you are happy with your translation, press Ctrl + Enter to store the translated segment in the TM and move to the next segment that needs translating (pressing Enter alone has no effect). Alternatively, in the Home tab, click the Confirm button, shown on the left of the following screenshot: This action is generally described as confirming the segment. The status symbol changes from (Draft), to (Translated) to indicate that the segment has been confirmed. Segments that you translate or edit must be confirmed in this way, or they will not be stored in the TM. The default confirm action (Ctrl + Enter) actually moves you to the next unconfirmed segment, skipping any confirmed segments in between. To show more options for confirming segments, as shown in the preceding screenshot, click the drop-down arrow under the Confirm button. To go to the next segment down, whether confirmed or not, choose Confirm and Move to Next Segment (Ctrl + Alt + Enter). If you are translating a file that produces lots of 100% matches that you do not wish to check immediately, choose Confirm and Translate until Next Fuzzy Match (Ctrl + Alt + F). You will then move down the bilingual file, automatically confirming any 100% matches, and only stopping at the next match that is less than 100%. Now translate and confirm Segment 2. This moves you into Segment 3, which is a fuzzy or partial match as indicated by the figure 82% in the following screenshot: Typing accented characters The ability to type accented characters in SDL Trados Studio is dependent on the keyboard settings in MS Windows, as with any other application that you might run on MS Windows. If you are using an English language keyboard and want to type accented characters in the target segment, you can use the Alt codes (such as Alt + 0233 for é). It is also possible to change the keyboard to follow the target language layout, via the Control Panel in MS Windows. The Translation Results window Whenever you move into a new segment (as from Segment 2 to Segment 3), the TM (or TMs if more than one is active) are searched for matches, and the highest match appears in the target segment (this action is called Lookup). If a match is found, the results are displayed in the Translation Results window, and an icon appears in the segment status column in the side-by-side editor to show the match level. By default, if no matches are found, the target segment remains empty, and the Translation Results window displays the text No matches found. The Translation Results window displays the text in the current segment in the white area at the top, and any match from the TM underneath it, as shown in the following screenshot. The blue and red text in the source segment indicates the words that need to be added to and deleted from the new segment compared to the match from the TM (in a similar form to that used in Track Changes in MS Word). In this case, for example, we need to add quite long and delete short in the translation. Edit the target segment to make the translation correct, and then confirm. When you edit and confirm the segment, the fuzzy match icon changes to a transparent background, as shown in the following screenshot. Notice that the fuzzy match value remains even after you confirm the segment. Thus, the percentage values displayed always indicate the value of the match as originally offered by the TM (the translation origin). Inserting matches from the TM The following screenshot shows the sample file before we edit and confirm Segment 4: Each match in the Translation Results window has a number, as shown on the left of the following screenshot. As you will see when you get to Segment 4 (which we will now edit and confirm) the highest match (with the number 1 in the column on the left) is automatically inserted whenever you move into an empty target segment. To insert a different match instead, press Ctrl and the numbers on the main keyboard. For example, to insert match number 2, press Ctrl + 2. To insert the match currently highlighted in blue in the Translation Results window, click the Apply Translation button or choose Home | Apply Translation (Ctrl + T). You can also scroll the list of matches to insert other matches via the Select Previous Match (Alt + Pg Up) and Select Next Match (Alt + Pg Dn) buttons. Notice that the yellow bar at the bottom of the Translation Results window shows the name of the TM providing the match in the segment that is highlighted in blue, as in the preceding screenshot. Summary In this article we learned the basic process of opening a document in SDL Trados Studio and translating it. We opened an individual document for translation and translated it in side-b-side editor. Resources for Article: Using Sprites for Animation [Article] About Test Studio [Article] Connecting to Microsoft SQL Server Compact 3.5 with Visual Studio [Article]

(For more resources related to this topic, see here.) Let's revisit the mockup of how our client would like the pricing tables to look on desktop-sized screens: Let's see how close we can get to the desired result, and what we can work out for other viewport sizes. Setting up the variables, files, and markup As shown in the preceding screenshot, there are a few tables in this design. We can begin by adjusting a few fundamental variables for all tables. These are found in _variables.less. Search for the tables section and adjust the variables for background, accented rows, and borders as desired. I've made these adjustments as shown in the following lines of code: // Tables // ------------------------- ... @table-bg: transparent; // overall background-color @table-bg-accent: hsla(0,0,1%,.1); // for striping @table-bg-hover: hsla(0,0,1%,.2); @table-bg-active: @table-bg-hover; @table-border-color: #ccc; // table and cell border Save the file, compile it to CSS, and refresh to see the result as shown in the following screenshot: That's a start. Now we need to write the more specific styles. The _page-contents.less file is now growing long, and the task before us is extensive and highly focused on table styles. To carry the custom styles, let's create a new LESS file for these pricing tables: Create _pricing-tables.less in the main less folder. Import it into __main.less just after _page-contents.less as shown in the following line: @import "_pricing-tables.less"; Open _pricing-tables.less in your editor and begin writing your new styles. But before we begin writing styles, let's review the markup that we'll be working with. We have the following special classes already provided in the markup on the parent element of each respective table:   package package-basic package package-premium package package-pro   Thus, for the first table, you'll see the following markup on its parent div: <div class="package package-basic col-lg-4"> <table class="table table-striped"> ... Similarly, we'll use package package-premium and package package-pro for the second and third table, respectively. These parent containers obviously also provide basic layout instructions using the col-md-4 class to set up a three-column layout in medium viewports. Next, we will observe the markup for each table. We see that the basic table and table-striped classes have been applied: <table class="table table-striped"> The table uses the <thead> element for its top-most block. Within this, there is <th> spanning two columns, with an <h2> heading for the package name and <div class="price"> to markup the dollar amount: <thead><tr><th colspan="2"><h2>Basic Plan</h2><div class="price">$19</div></th></tr></thead> Next is the tfoot tag with the Sign up Now! button: <tfoot><tr><td colspan="2"><a href="#" class="btn">Sign upnow!</a></td></tr></tfoot> Then is the tbody tag with the list of features laid out in a straightforward manner in rows with two columns: <tbody><tr><td>Feature</td><td>Name</td></tr><tr><td>Feature</td><td>Name</td></tr><tr><td>Feature</td><td>Name</td></tr><tr><td>Feature</td><td>Name</td></tr><tr><td>Feature</td><td>Name</td></tr></tbody> And finally, of course, the closing tags for the table and parent div tags: </table></div><!-- /.package .package-basic --> Each table repeats this basic structure. This gives us what we need to start work! Beautifying the table head To beautify the thead element of all of our tables, we'll do the following: Align the text at the center Add a background color—for now, add a gray color that is approximately a midtone similar to the colors we'll apply to the final version Turn the font color white Convert the h2 heading to uppercase Increase the size of the price table Add the necessary padding all around the tables We can apply many of these touches with the following lines of code. We'll specify the #signup section as the context for these special table styles: #signup {table {border: 1px solid @table-border-color;thead th {text-align: center;background-color: @gray-light;color: #fff;padding-top: 12px;padding-bottom: 32px;h2 {text-transform: uppercase;}}}} In short, we've accomplished everything except increasing the size of the price tables. We can get started on this by adding the following lines of code, which are still nested within our #signup table selector: .price { font-size: 7em; line-height: 1;} This yields the following result: This is close to our desired result, but we need to decrease the size of the dollar sign. To give ourselves control over that character, let's go to the markup and wrap a span tag around it: <em class="price"><span>$</span>19</em> Remember to do the same for the other two tables. With this new bit of markup in place, we can nest this within our styles for .price: .price {...span {font-size: .5em;vertical-align: super;} These lines reduce the dollar sign to half its size and align it at the top. Now to recenter the result, we need to add a bit of negative margin to the parent .price selector: .price {margin-left: -0.25em;... The following screenshot shows the result: Styling the table body and foot By continuing to focus on the styles that apply to all three pricing tables, let's make the following adjustments: Add left and right padding to the list of features Stretch the button to full width Increase the button size We can accomplish this by adding the following rules: #signup {table {...tbody {td {padding-left: 16px;padding-right: 16px;}}a.btn {.btn-lg;display: block;width: 100%;background-color: @gray-light;color: #fff;}}} Save the file, compile it to CSS, and refresh the browser. You should see the following result: We're now ready to add styles to differentiate our three packages.

(For more resources related to this topic, see here.) For those hoping to find a magical catch-all formula that will work in every scenario, you'll have to keep looking. Remember every environment is unique, and even where similarities may arise, the use case your organization has, will most likely be different from another organization. Beyond your specific VM resource requirements, the hosts you are installing ESXi on will also vary; the hardware available to you will affect your consolidation ratio (the number of virtual machines you can fit on a single host). For example, if you have 10 servers that you want to virtualize, and you have determined each requires 4 GB of RAM, you might easily virtualize those 10 servers on a host with 48 GB of memory. However, if your host only has 16 GB of memory, you may need two or three hosts in order to achieve the required performance. Another important aspect to consider is when to collect resource utilization statistics about your servers. Think about the requirements you have for a specific server; let's use your finance department as an example. You can certainly collect resource statistics over a period of time in the middle of the month, and that might work just fine; however, the people in your finance department are more likely to utilize the system heavily during the first few days of the month as they are working on their month end processes. If you collect resource statistics on the 15th, you might miss a huge increase in resource utilization requirements, which could lead to the system not working as expected, making unhappy users. One last thing before we jump into some example statistics; you should consider collecting these statistics over at least two periods for each server: First, during the normal business hours of your organization or the specific department, during a time when systems are likely to be heavily utilized The second round should include an entire day or week so you are aware of the impact of after hours tasks such as backups and anti-virus scans on your environment It's important to have a strong understanding of the use cases for all the systems you will be virtualizing. If you are running your test during the middle of the month, you might miss the increase of traffic for systems utilized heavily only at the end of the month, for example, accounting systems. The more information you collect, the better prepared you will be to determine your resource utilization requirements. There are quite a few commercial tools available to help determine the specific resource requirements for your environment. In fact, if you have an active project and/or budget, check with your server and storage vendor as they can most likely provide tools to assess your environment over a period of time to help you collect this information. If you work with a VMware Partner or the VMware Professional Services Organization (PSO), you could also work with them to run a tool called VMware Capacity Planner. This tool is only available to partners who have passed the corresponding partner exams. For purposes of this article, however, we will look at the statistics we can capture natively within an operating system, for example, using Performance Monitor on Windows and the sar command in Linux. If you are an OS X user, you might be wondering why we are not touching OS X. This is because while Apple allows virtualizing OS X 10.5 and later, it is only supported on the Apple hardware and is not likely an everyday use case. If your organization requires virtualizing OSX, ESXi 5.1 is supported on specific Mac Pro desktops with Intel Xeon 5600 series processors and 5.0 is supported on Xserve using Xeon 5500 series processors. The current Apple license agreement allows virtualizing OSX 10.5 and up; of course, you should check for the latest agreement to ensure you are adhering to the license agreement. Monitoring common resource statistics From a statistics perspective, there are four main types of resources you generally monitor: CPU, memory, disk, and network. Unless you have a very chatty application, network utilization is generally very low, but this doesn't mean we won't check on it; however, we probably won't dedicate as much time to it as we do for CPU, memory, and disk. As we think about the CPU and memory, we generally look at utilization in terms of percentages. When we look at example servers, you will see that having an accurate inventory of the physical server is important so we can properly gauge the virtual CPU and memory requirements when we virtualize. If a physical server has dual quad core CPUs and 16 GB of memory, it does not necessarily mean we want to provide the same amount of virtual resources. Disk performance is where many people spend the least amount of time, and those people generally have the most headaches after they have virtualized. Disk performance is probably the most critical aspect to think about when you are planning your virtualization project. Most people only think of storage in terms of storage capacity, generally gigabytes (GB) or terabytes (TB). However, from a server perspective, we are mostly concerned with the amount of input and output per second, otherwise known as IOPS and throughput. We break down IOPS in into reads and writes per second and then their ratio by comparing one with the other. Understanding your I/O patterns will help you design your storage architecture to properly support all your applications. Storage design and understanding is an art and science by itself. Sample workload Let's break this down into a practical example so we can see how we are applying these concepts. In this example, we will look at two different types of servers that are likely to have various resource requirements: Windows Active Directory Domain Controller and a CentOS Apache web server. In this scenario, let's assume that each of these server operating systems and applications are running on dedicated hardware, that is, they are not yet virtual machines. The first step you should take, if you do not have this already, is to document the physical systems, their components, and other relevant information such as computer or DNS name, IP address (es), location, and so on. For larger environments, you may also want to document installed software, user groups or departments, and so on. Collecting statistics on Windows On Windows servers, your first step would be to start performance monitoring. Perform the following steps to do so: Navigate to Start | Run and enter perfmon. Once the Performance Monitor window opens, expand Monitoring Tools and click on Performance Monitor. Here, you could start adding various counters; however, as of Windows 2008/Windows 7, Performance Monitor includes Data Collector Sets. Expand the Data Collector Sets folder and then the System folder; right-click on System Performance and select Start. Performance Monitor will start to collect key statistics about your system and its resource utilization. When you are satisfied that you have collected an appropriate amount of data, click on System Performance and select Stop. Your reports will be saved into the Reports folder; navigate to Reports| System, click on the System Performance folder, and finally double-click on the report to see the report. In the following screenshot for our domain controller, you can see we were using 10 percent of the total CPU resources available, 54 percent of the memory, a low 18 IOPS, and 0 percent of the available network resources (this is not really uncommon; I have had busy application servers that barely break 2 percent). Now let's compare what we are utilizing with the actual physical resources available to the server. This server has two dual core processors (four total cores) running at 2 GHz per core (8 GHz total available), 4 GB of memory, two 200 GB SAS drives configured in a RAID 1, and a 1 Gbps network card. Here, performance monitor shows averages, but you should also investigate peak usage. If you scroll down in the report, you will find a menu labeled CPU. Navigate to CPU | Process. Here you will see quite a bit of data, more than the space we have to review in this book; however, if you scroll down, you will see a section called Processor User Time by CPU. Here, your mean (that is, average) column should match fairly closely to the report overview provided for the total, but we also want to look at any spikes we may have encountered. As you can see, this CPU had one core that received a maximum of 35 percent utilization, slightly more than the average suggested. If we take the average CPU utilization at 10 percent of the total CPU, it means we will theoretically require only 800 MHz of CPU power, something a single physical core could easily support. The, memory is also using only half of what is available, so we can most likely reduce the amount of memory to 3 GB and still have room for various changes in operating conditions we might have not encountered during our collection window. Finally, having only 18 IOPS used means that we have plenty of performance left in the drives; even a SATA 7200 RPM drive can provide around 80 IOPS. Collecting statistics on Linux Now let's look at the Linux web server to see how we can collect this same set of information using sar in an additional package with sysstat that can monitor resource utilization over time. This is similar to what you might get from top or iotop. The sysstat package can easily be added to your system by running yum install sysstat, as it is a part of the base repository (yum install sysstat is command format). Once the sysstat package is installed, it will start collecting information about resource utilization every 10 minutes and keep this information for a period of seven days. To see the information, you just need to run the sar command; there are different options to display different sets of information , which we will look at next. Here, we can see that our system is idle right now by viewing the %idle column. A simple way to generate some load on your system is to run dd if=/dev/zero of=/dev/null, which will spike your CPU load to 100 percent, so, don't do this on production systems! Let's look at the output with some CPU load. In this example, you can see that the CPU was under load for about half of the 10-minute collection window. One problem here is that unless the CPU spike, in this case to 100 percent, was not consistent for at least 10 minutes, we would potentially miss these spikes using sar with a 10-minute window. This is easily changed by editing /etc/cron.d/sysstat, which tells the system to run this every 10 minutes. During a collection window, one or two minutes may provide more valuable detail. In this example, you can see I am now logging at a five-minute interval instead of 10, so I will have a better chance to find maximum CPU usage during my monitoring period. Now, we are not only concerned with the CPU, but we also want to see memory and disk utilization. To access those statistics, run sar with the following options: The sar –r command will show RAM (memory) statistics. At a basic level, the items we are concerned with here would be the percentage of memory used, which we could use to determine how much memory is actually being utilized. The sar –b command will show disk I/O. From a disk perspective, sar –b will tell us the total number of transactions per second (tps), read transactions per second (rtps), and write transactions per second (wtps). As you can see, you are able to natively collect quite a bit of relevant data about resource utilization on our systems. However, without the help of a vendor or VMware PSO who has access to VMware Capacity Planner, another commercial tool, or a good automation system, this can become difficult to do on a large scale (hundreds or thousands of servers), but certainly not impossible. Resources for Article: Further resources on this subject: Windows 8 with VMware View [Article] Troubleshooting Storage Contention [Article] Networking Performance Design [Article]

(For more resources related to this topic, see here.) Terminology In this article, I will be guiding you through many examples using Hammer. There are a handful of terms that will recur many times that you will need to know. It might be a good idea to bookmark this page, so you can flip back and refresh your memory. Brush A brush is a piece of world geometry created with block tool. Brushes make up the basic building blocks of a map and must be convex. A convex object's faces cannot see each other, while a concave object's faces can. Imagine if you're lying on the pitched roof of a generic house. You wouldn't be able to see the other side of the roof because the profile of the house is a convex pentagon. If you moved the point of the roof down inside the house, you would be able to see the other side of the roof because the profile would then be concave. This can be seen in the following screenshot: Got it? Good. Don't think you're limited by this; you can always create convex shapes out of more than one brush. Since we're talking about houses, brushes are used to create the walls, floor, ceiling, and roof. Brushes are usually geometrically very simple; upon creation, common brushes have six faces and eight vertices like a cube. The brushes are outlined in the following screenshot: Not all brushes have six sides; however, the more advanced brushwork techniques can create brushes that have (almost) as many sides as you want. You need to be careful while making complex brushes with the more advanced tools though. This is because it can be quite easy to make an invalid solid if you're not careful. An invalid solid will cause errors during compilation and will make your map unplayable. A concave brush is an example of an invalid solid. World brushes are completely static or unmoving. If you want your brushes to have a special function, they need to be turned into entities. Entity An entity is anything in the game that has a special function. Entities come in two flavors: brush-based and point-based. A brush-based entity, as you can probably guess, is created from a brush. A sliding door or a platform lift are examples of brush-based entities. Point-based entities, on the other hand, are created in a point in space with the entity tool. Lights, models, sounds, and script events are all point-based entities. In the following figure, the models or props are highlighted: World The world is everything inside of a map that you create. Brushes, lights, triggers, sounds, models, and so on are all part of the world. They must all be contained within a sealed map made out of world brushes. Void The void is nothing or everything that isn't the world. The world must be sealed off from the void in order to function correctly when compiled and played in game. Imagine the void as outer space or a vacuum. World brushes seal off the world from the void. If there are any gaps in world brushes (or if there are any entities floating in the void), this will create a leak, and the engine will not be able to discern what the world is and what the void is. If a leak exists in your map, the engine will not know what is supposed to be seen during the compile! The map will compile, but performance-reducing side effects such as bland lighting and excess rendered polygons will plague your map. Settings If at any point in your mapping experience, Hammer doesn't seem to be operating the way you want it to be, go to Tools | Options, and see if there's any preferences you would like to change. You can customize general settings or options related to the 2D and 3D views. If you're coming from another editor, perhaps there's a setting that will make your Hammer experience similar to what you're used to. Loading Hammer for the first time You'll be opening SteamsteamappscommonHalf-Life 2bin often, so you may want to create a desktop shortcut for easier access. Run Hammer.bat from the bin folder to launch Valve Hammer Editor, Valve's map (level) creator, so you can start creating a map. Hammer will prompt you to choose a game configuration, so choose which game you want to map for. I will be using Half-Life 2: Episode Two in the following examples: When you first open up Hammer, you will see a blank gray screen surrounded by a handful of menus, as shown in the following screenshot: Like every other Microsoft Windows application, there is a main menu in the top-left corner of the screen that lets you open, save, or create a document. As far as Hammer is concerned, our documents are maps, and they are saved with the .vmf file extension. So let's open the File menu and load an example map so we can poke around a bit. Load the map titled Chapter2_Example.vmf. The Hammer overview In this section, we will be learning to recognize the different areas of Hammer and what they do. Being familiar with your environment is important! Viewports There are four main windows or viewports in Hammer, as shown in the following screenshot: By default, the top-left window is the 3D view or camera view. The top-right window is the top (x/y) view, the bottom-right window is the side (x/z) view, and the bottom-left window is the front (y/z) view. If you would like to change the layout of the windows, simply click on the top-left corner of any window to change what is displayed. In this article, I will be keeping the default layout but that does not mean you have to! Set up Hammer any way you'd like. For instance, if you would prefer your 3D view to be larger, grab the cross at the middle of the four screens and drag to extend the areas. The 3D window has a few special 3D display types such as Ray-Traced Preview and Lightmap Grid. We will be learning more about these later, but for now, just know that 3D Ray-traced Preview simulates the way light is cast. It does not mimic what you would actually see in-game, but it can be a good first step before compile to see what your lighting may look like. The 3D Lighting Preview will open in a new window and will update every time a camera is moved or a light entity is changed. You cannot navigate directly in the lighting preview window, so you will need to use the 2D cameras to change the viewing perspective. The Map toolbar Located to the left of the screen, the Map toolbar holds all the mapping tools. You will probably use this toolbar the most. The tools will each be covered in depth later on, but here's a basic overview, as shown in the following screenshot, starting from the first tool: The Selection Tool The Selection Tool is pretty self-explanatory; use this tool to select objects. The hot key for this is Shift + S. This is the tool that you will probably use the most. This tool selects objects in the 3D and 2D views and also lets you drag selection boxes in the 2D views. The Magnify Tool The Magnify Tool will zoom in and out in any view. You could also just use the mouse wheel if you have one or the + and – keys on the numerical keypad for the 2D views. The hot key for the magnify tool is Shift + G. The Camera Tool The Camera Tool enables 3D view navigation and lets you place multiple different cameras into the 2D views. Its hot key is Shift + C. The Entity Tool The Entity Tool places entities into the map. If clicked on the 3D view, an entity is placed on the closest brush to the mouse cursor. If used in the 2D view, a crosshair will appear noting the origin of the entity, and the Enter key will add it to the map at the origin. The entity placed in the map is specified by the object bar. The hot key is Shift + E. The Block Tool The Block Tool creates brushes. Drag a box in any 2D view and hit the Enter key to create a brush within the bounds of the box. The object bar specifies which type of brush will be created. The default is box and the hot key for this is Shift + B. The Texture Tool The Texture Tool allows complete control over how you paint your brushes. For now, just know where it is and what it does; the hot key is Shift + A. The Apply Current Texture Tool Clicking on the Apply Current Texture icon will apply the selected texture to the selected brush or brushes. The Decal Tool The Decal Tool applies decals and little detail textures to brushes and the hot key is Shift + D. The Overlay Tool The Overlay Tool is similar to the decal tool. However, overlays are a bit more powerful than decals. Shift + O will be the hot key. The Clipping Tool The Clipping Tool lets you slice brushes into two or more pieces, and the hot key is Shift + X. The Vertex manipulation Tool The Vertex manipulation Tool, or VM tool, allows you to move the individual vertices and edges of brushes any way you like. This is one of the most powerful tools you have in your toolkit! Using this tool improperly, however, is the easiest way to corrupt your map and ruin your day. Not to worry though, we'll learn about this in great detail later on. The hot key is Shift + V. The selection mode bar The selection mode bar (located at the top-right corner by default) lets you choose what you want to select. If Groups is selected, you will select an entire group of objects (if they were previously grouped) when you click on something. The Objects selection mode will only select individual objects within groups. Solids will only select solid objects. The texture bar Located just beneath the selection mode toolbar, the texture bar, as shown, in the following screenshot, shows a thumbnail preview of your currently selected (active) texture and has two buttons that let you select or replace a texture. What a nifty tool, eh? The filter control bar The filter control bar controls your VisGroups (short for visual groups). VisGroups separate your map objects into different categories, similar to layers in the image editing software. To make your mapping life a bit easier, you can toggle visibility of object groups. If, for example, you're trying to sculpt some terrain but keep getting your view blocked by tree models, you can just uncheck the Props box, as shown in the following screenshot, to hide all the trees! There are multiple automatically generated VisGroups such as entities, displacements, and nodraws that you can easily filter through. Don't think you're limited to this though; you can create your own VisGroup with any selection at any time. The object bar The object bar lets you control what type of brush you are creating with the brush tool. This is also where you turn brushes into brush-based entities and create and place prefabs, as shown in the following screenshot: Navigating in 3D You will be spending most of your time in the main four windows, so now let's get comfortable navigating in them, starting with the 3D viewport. Looking around Select the camera tool on the map tools bar. It's the third one down on the list and looks like a red 35 mm camera. Holding the left mouse button while in the 3D view will allow you to look around from a stationary point. Holding the right mouse button will allow you to pan left, right, up, and down in the 3D space. Holding both left and right mouse buttons down together will allow you to move forward and backwards as well as pan left and right. Scrolling the mouse wheel will move the camera forward and backwards. Practice flying down the example map hallway. While looking down the hallway, hold the right mouse button to rise up through the grate and see the top of the map. If you would prefer another method of navigating in 3D, you can use the W, S, A, and D keys to move around while the left mouse button is pressed. Just like your normal FPS game, W moves forward in the direction of the camera, S moves backwards, and A and D move left and right, respectively. You can also move the mouse to look around while moving. Being comfortable with the 3D view is necessary in order to become proficient in creating and scripting 3D environments. As with everything, practice makes perfect, so don't be discouraged if you find yourself hitting the wrong buttons. Having the camera tool selected is not necessary to navigate in 3D. With any tool selected, hold Space bar while the cursor is in the 3D window to activate the 3D navigation mode. While holding Space bar, the navigation functions exactly as it does as if the camera tool was selected. Releasing Space bar will restore normal functionality to the currently selected tool. This can be a huge time saver down the line when you're working on very technical object placements. Multiple cameras If you find yourself bouncing around between different areas of the map, or even changing angles near the same object, you can create multiple cameras and juggle between them. With the camera tool selected, hold Shift and drag a line with the left mouse button in any 2D viewport. The start of the line will be the camera origin, and the end of the line will be the camera's target. Whenever you create a new camera, the newly created camera becomes active and displays its view in the 3D viewport. To cycle between cameras, press the Page Up and Page Down buttons, or click on a camera in any 2D view. Camera locations are stored in the map file when you save, so you don't have to worry about losing them when you exit. Pressing the Delete key with a camera selected will remove the active camera. If you delete the only camera, your view will snap to the origin (0, 0, 0) but you will still be able to look around and create other cameras. In essence, you will always have at least one camera in your map. Selecting objects in the 3D viewport To select an object in the 3D viewport, you must have the selection tool active, as shown in the following screenshot. A quick way to activate the selection tool is to hit the Esc key while any tool is selected, or use the Shift + S hot key. A selected brush or a group of brushes will be highlighted in red with yellow edges as shown in the following screenshot: To deselect anything within the 3D window, click on any other brush, or on the background (void), or simply hit the Esc key. If you want to select an object behind another object, press and hold the left mouse button on the front object. This will cycle through all the objects that are located behind the cursor. You will be able to see the selected objects changing in the 2D and 3D windows about once per second. Simply release the mouse button to complete your selection. To select multiple brushes or objects in the 3D window, hold Ctrl and left-click on multiple brushes. Clicking on a selected object while Ctrl is held will deselect the object. If you've made a mistake choosing objects, you can undo your selections with Ctrl + Z or navigate to Edit | Undo.

(For more resources related to this topic, see here.) Creating a Bridge In WABS, a bridge is used for transporting a message from one place to another. Unlike BizTalk Server, a message entering a bridge will always be routed to one destination only. All bridges have an HTTPS entry point. Additional entry points, such as FTP and SFTP, can be added. The following is a list of what can happen in a bridge: Receiving a message (message format could be XML, Flat File, or any custom format such as JSON, since custom Message Inspectors can be created). Message Inspectors are not covered in this article, but several examples can be found on the Internet. Refer to the following link on how to include custom code in bridges: http://msdn.microsoft.com/en-us/library/windowsazure/dn232389.aspx Decode the message to XML (from Flat File or custom formats). Validate the message with the Schemas specified in the bridge. Enrich the message by writing metadata to the message (similar to Property Promotion in BizTalk Server). Transform the message using Maps. Encode the message from XML to the target format (Flat File or custom format). A bridge can also send messages to another bridge for further processing. A bridge will always act as one whole step, meaning that if a bridge cannot submit to its destination, the message will not be removed from the source. Let us create a very simple bridge that picks up a text message from an FTP folder and drops the file in another FTP location. For this example you will need an FTP site and three folders: In: This is used for submitting files to the bridge Out: This is used for dropping files from the bridge AltOut: This is used as an alternative file drop from the bridge Also, a username that has full access to these folders and a corresponding password will be required. Open Visual Studio, create a new project, and choose Visual C# | BizTalk Services | BizTalk Service. Choose the Location C:BTS2013CertGuideChapter09Projects, name the Solution Chapter09.Example01, and name the Project Chapter09. Example01.SimpleBridge. Click on OK. You should now have a project with a blank canvas where a text reads Drag a bridge to this diagram…, and a toolbox that resembles the following screenshot: Drag a Pass-Through Bridge from the toolbox to the empty canvas. Check the Properties for the created bridge and notice that the default Entity Name and Relative Address have been given the name PassThroughBridge1. Change both names to MySimpleBridge. We now need to create an FTP source so that the bridge will automatically pick up messages, whenever they are submitted from this source. In the toolbox, choose Sources | FTP Source and drag it onto the canvas to the left of your bridge. Once dragged onto the canvas, notice that the source has a yellow exclamation mark, indicating that the source needs to be connected to a bridge. To do this, choose Bridges | Connector in the toolbox. The connector is not used by dragging it to the canvas, but rather just selecting it and then dragging and connecting from the FTP Source to the bridge. You need to be precise when doing this, connecting from the red dot to the other red dot, as shown in the following screenshot: Rename the FTPSource1 by selecting it, and change FTPSource1 to MySimpleFTPPickup under Properties | Entity Name. We now need to configure the FTP Source; select the source and configure the following properties. Parameter Value File Mask *.* Folder Path In Initial Status Stop Password [The FTP User's password] Server Address [The FTP Server address] Use SSL False Username [The FTP Username] Your FTP Source properties should now look somewhat similar to this: Note that the Initial Status is by default Start, which will have the source polling from the FTP folder as soon as the project is deployed. To gain better control of when the polling starts, we will set it to Stop instead and then manually start it using a PowerShell command. In the toolbox, choose Destinations | FTP Destination and drag it onto the canvas on the right side of the bridge. Connect the bridge to the destination. Change the name of the FTP Destination from FTPDestination1 to MySimpleMainFTPDest. Set the appropriate properties, making it point to the Out folder; with a few variations your properties should resemble the following: As of now, the only place we can configure the FTP properties are in Visual Studio. In the future, other configuration options in the Azure Portal might be possible. This also means that, for now, if a password changes, you will need to change this in the Visual Studio project, and then redeploy the project. Try building your project. You should receive a warning about the FTP destination name not being set and two errors stating something about filter conditions. Let's fix these issues. Filter Condition and Route Ordering A bridge can have from one to many destinations. Unlike a publish-subscribe engine, the message will only be routed to one of these destinations. This is done by setting a filter condition on each connection to the destinations and then prioritizing the destinations in the bridge's Route Ordering Table property. All destination connections need a filter condition (which is also the reason we received the two errors before), and a bridge will always have each of its destinations in a prioritized Route Ordering Table. When a message leaves the bridge, the filter condition for the destination connections will be evaluated in the order they appear in the Route Ordering Table. The first positive match will get the message, and the algorithm will stop, so that no more than one destination will get the message. In our rather simple example from before, all we have is one destination, and we naturally want all messages to evaluate to that destination's Filter Condition. Select the connection from the bridge to the FTP destination and locate the Filter Condition property. Click on the ellipsis of the Filter Condition property, and click on the Match All radio button. Click on OK. Notice that the Filter Condition now reads 1 = 1, which will naturally always evaluate to true!   Setting the FTP filename We are now left with a single warning message when building the project: FTP Filename property needs to be specified at the Route Action stage Although this is only a warning and the project will build and deploy, the solution would not work since the FTP destination will not have any name to assign to the file it is writing to the Out folder. For now, we will just hardcode the output.txt filename. Later, we will change the solution so that the original filename is used, by using the Enrich feature in the bridge. To set the FTP filename used by the FTP destination, we need to create a Route Action on the connection to the destination, by using the following steps: Select the connection from the bridge to the FTP destination and locate the Route Action property. Click on the ellipsis, and then click on Add. In Property (Read From), select Expression and type 'output.txt'. Notice that single quotes are needed around the name. In Property (Write To), choose Ftp and FileName. Click on OK twice. Build the project again, and verify that we are left with no warnings. Deploying a Bridge To deploy a project to the BizTalk Service in Azure, we need the following: The name of the BizTalk Service The name of the Access Control Namespace gathered earlier The Default Issuer The Default Key To deploy, perform the following steps: Click somewhere on an empty space on the canvas where the bridge, FTP source, and destination reside. Under Properties, locate the BizTalk Service URL. Replace servicename with the name of your BizTalk Service. Right-click on the project and click on Deploy. In Acs Namespace, type the namespace of the Acs. Set Shared Secret to the token fetched previously from the Azure Portal. Leave the Refresh server after…checkbox unchecked, as we will refresh the service manually from PowerShell if needed. Click on Deploy. Confirm that two items were successfully deployed (the bridge and the source). Now we need to verify that the bridge and the source have been deployed in the WABS, and then start the source so that it will commence picking up files from the In FTP folder. Summary This article has dealt with some of the basics of BizTalk Server and BizTalk Services in Azure. We have seen how to create accounts and Servers in Azure, how to use them, how to set up an EDI agreement for exchanging messages with partners in the cloud, and even how to route messages from Azure to your local on-premise applications. Resources for Article: Microsoft Biztalk server 2010 patterns: Operating Biztalk [Article] Microsoft DAC 2012 [Article] Setting up a BizTalk Server Environment [Article]

This article, written by Khaled Tannir, the author of Optimizing Hadoop for MapReduce, discusses two of the most important aspects to consider while optimizing Hadoop for MapReduce: sizing and configuring the Hadoop cluster correctly. Sizing your Hadoop cluster Hadoop's performance depends on multiple factors based on well-configured software layers and well-dimensioned hardware resources that utilize its CPU, Memory, hard drive (storage I/O) and network bandwidth efficiently. Planning the Hadoop cluster remains a complex task that requires a minimum knowledge of the Hadoop architecture and may be out the scope of this book. This is what we are trying to make clearer in this section by providing explanations and formulas in order to help you to best estimate your needs. We will introduce a basic guideline that will help you to make your decision while sizing your cluster and answer some How to plan questions about cluster's needs such as the following: How to plan my storage? How to plan my CPU? How to plan my memory? How to plan the network bandwidth? While sizing your Hadoop cluster, you should also consider the data volume that the final users will process on the cluster. The answer to this question will lead you to determine how many machines (nodes) you need in your cluster to process the input data efficiently and determine the disk/memory capacity of each one. Hadoop is a Master/Slave architecture and needs a lot of memory and CPU bound. It has two main components: JobTracker: This is the critical component in this architecture and monitors jobs that are running on the cluster TaskTracker: This runs tasks on each node of the cluster To work efficiently, HDFS must have high throughput hard drives with an underlying filesystem that supports the HDFS read and write pattern (large block). This pattern defines one big read (or write) at a time with a block size of 64 MB, 128 MB, up to 256 MB. Also, the network layer should be fast enough to cope with intermediate data transfer and block. HDFS is itself based on a Master/Slave architecture with two main components: the NameNode / Secondary NameNode and DataNode components. These are critical components and need a lot of memory to store the file's meta information such as attributes and file localization, directory structure, names, and to process data. The NameNode component ensures that data blocks are properly replicated in the cluster. The second component, the DataNode component, manages the state of an HDFS node and interacts with its data blocks. It requires a lot of I/O for processing and data transfer. Typically, the MapReduce layer has two main prerequisites: input datasets must be large enough to fill a data block and split in smaller and independent data chunks (for example, a 10 GB text file can be split into 40,960 blocks of 256 MB each, and each line of text in any data block can be processed independently). The second prerequisite is that it should consider the data locality, which means that the MapReduce code is moved where the data lies, not the opposite (it is more efficient to move a few megabytes of code to be close to the data to be processed, than moving many data blocks over the network or the disk). This involves having a distributed storage system that exposes data locality and allows the execution of code on any storage node. Concerning the network bandwidth, it is used at two instances: during the replication process and following a file write, and during the balancing of the replication factor when a node fails. The most common practice to size a Hadoop cluster is sizing the cluster based on the amount of storage required. The more data into the system, the more will be the machines required. Each time you add a new node to the cluster, you get more computing resources in addition to the new storage capacity. Let's consider an example cluster growth plan based on storage and learn how to determine the storage needed, the amount of memory, and the number of DataNodes in the cluster. Daily data input 100 GB Storage space used by daily data input = daily data input * replication factor = 300 GB HDFS replication factor 3 Monthly growth 5% Monthly volume = (300 * 30) + 5% =  9450 GB After one year = 9450 * (1 + 0.05)^12 = 16971 GB Intermediate MapReduce data 25% Dedicated space = HDD size * (1 - Non HDFS reserved space per disk / 100 + Intermediate MapReduce data / 100) = 4 * (1 - (0.25 + 0.30)) = 1.8 TB (which is the node capacity) Non HDFS reserved space per disk 30% Size of a hard drive disk 4 TB Number of DataNodes needed to process: Whole first month data = 9.450 / 1800 ~= 6 nodes The 12th month data = 16.971/ 1800 ~= 10 nodes Whole year data = 157.938 / 1800 ~= 88 nodes Do not use RAID array disks on a DataNode. HDFS provides its own replication mechanism. It is also important to note that for every disk, 30 percent of its capacity should be reserved to non-HDFS use. It is easy to determine the memory needed for both NameNode and Secondary NameNode. The memory needed by NameNode to manage the HDFS cluster metadata in memory and the memory needed for the OS must be added together. Typically, the memory needed by Secondary NameNode should be identical to NameNode. Then you can apply the following formulas to determine the memory amount: NameNode memory 2 GB - 4 GB Memory amount = HDFS cluster management memory + NameNode memory + OS memory Secondary NameNode memory 2 GB - 4 GB OS memory 4 GB - 8 GB HDFS memory 2 GB - 8 GB At least NameNode (Secondary NameNode) memory = 2 + 2 + 4 = 8 GB It is also easy to determine the DataNode memory amount. But this time, the memory amount depends on the physical CPU's core number installed on each DataNode. DataNode process memory 4 GB - 8 GB Memory amount = Memory per CPU core * number of CPU's core + DataNode process memory + DataNode TaskTracker memory + OS memory DataNode TaskTracker memory 4 GB - 8 GB OS memory 4 GB - 8 GB CPU's core number 4+ Memory per CPU core 4 GB - 8 GB At least DataNode memory = 4*4 + 4 + 4 + 4 = 28 GB Regarding how to determine the CPU and the network bandwidth, we suggest using the now-a-days multicore CPUs with at least four physical cores per CPU. The more physical CPU's cores you have, the more you will be able to enhance your job's performance (according to all rules discussed to avoid underutilization or overutilization). For the network switches, we recommend to use equipment having a high throughput (such as 10 GB) Ethernet intra rack with N x 10 GB Ethernet inter rack. Configuring your cluster correctly To run Hadoop and get a maximum performance, it needs to be configured correctly. But the question is how to do that. Well, based on our experiences, we can say that there is not one single answer to this question. The experiences gave us a clear indication that the Hadoop framework should be adapted for the cluster it is running on and sometimes also to the job. In order to configure your cluster correctly, we recommend running a Hadoop job(s) the first time with its default configuration to get a baseline. Then, you will check the resource's weakness (if it exists) by analyzing the job history logfiles and report the results (measured time it took to run the jobs). After that, iteratively, you will tune your Hadoop configuration and re-run the job until you get the configuration that fits your business needs. The number of mappers and reducer tasks that a job should use is important. Picking the right amount of tasks for a job can have a huge impact on Hadoop's performance. The number of reducer tasks should be less than the number of mapper tasks. Google reports one reducer for 20 mappers; the others give different guidelines. This is because mapper tasks often process a lot of data, and the result of those tasks are passed to the reducer tasks. Often, a reducer task is just an aggregate function that processes a minor portion of the data compared to the mapper tasks. Also, the correct number of reducers must also be considered. The number of mappers and reducers is related to the number of physical cores on the DataNode, which determines the maximum number of jobs that can run in parallel on DataNode. In a Hadoop cluster, master nodes typically consist of machines where one machine is designed as a NameNode, and another as a JobTracker, while all other machines in the cluster are slave nodes that act as DataNodes and TaskTrackers. When starting the cluster, you begin starting the HDFS daemons on the master node and DataNode daemons on all data nodes machines. Then, you start the MapReduce daemons: JobTracker on the master node and the TaskTracker daemons on all slave nodes. The following diagram shows the Hadoop daemon's pseudo formula: When configuring your cluster, you need to consider the CPU cores and memory resources that need to be allocated to these daemons. In a huge data context, it is recommended to reserve 2 CPU cores on each DataNode for the HDFS and MapReduce daemons. While in a small and medium data context, you can reserve only one CPU core on each DataNode. Once you have determined the maximum mapper's slot numbers, you need to determine the reducer's maximum slot numbers. Based on our experience, there is a distribution between the Map and Reduce tasks on DataNodes that give good performance result to define the reducer's slot numbers the same as the mapper's slot numbers or at least equal to two-third mapper slots. Let's learn to correctly configure the number of mappers and reducers and assume the following cluster examples: Cluster machine Nb Medium data size Large data size DataNode CPU cores 8 Reserve 1 CPU core Reserve 2 CPU cores DataNode TaskTracker daemon 1 1 1 DataNode HDFS daemon 1 1 1 Data block size 128 MB 256 MB DataNode CPU % utilization 95% to 120% 95% to 150% Cluster nodes 20 40 Replication factor 2 3 We want to use the CPU resources at least 95 percent, and due to Hyper-Threading, one CPU core might process more than one job at a time, so we can set the Hyper-Threading factor range between 120 percent and 170 percent. Maximum mapper's slot numbers on one node in a large data context = number of physical cores - reserved core * (0.95 -> 1.5) Reserved core = 1 for TaskTracker + 1 for HDFS Let's say the CPU on the node will use up to 120% (with Hyper-Threading) Maximum number of mapper slots = (8 - 2) * 1.2 = 7.2 rounded down to 7 Let's apply the 2/3 mappers/reducers technique: Maximum number of reducers slots = 7 * 2/3 = 5 Let's define the number of slots for the cluster: Mapper's slots: = 7 * 40 = 280 Reducer's slots: = 5 * 40 = 200 The block size is also used to enhance performance. The default Hadoop configuration uses 64 MB blocks, while we suggest using 128 MB in your configuration for a medium data context as well and 256 MB for a very large data context. This means that a mapper task can process one data block (for example, 128 MB) by only opening one block. In the default Hadoop configuration (set to 2 by default), two mapper tasks are needed to process the same amount of data. This may be considered as a drawback because initializing one more mapper task and opening one more file takes more time. Summary In this article, we learned about sizing and configuring the Hadoop cluster for optimizing it for MapReduce. Resources for Article: Further resources on this subject: Hadoop Tech Page Hadoop and HDInsight in a Heartbeat Securing the Hadoop Ecosystem Advanced Hadoop MapReduce Administration

(For more resources related to this topic, see here.) Code quality analysis The fact that you can compile your code does not mean your code is good. It does not even mean it will work. There are many things that can easily break your code. A good example is an unhandled NullReferenceException. You will be able to compile your code, you will be able to run your application, but there will be a problem. ReSharper v8 comes with more than 1400 code analysis rules and more than 700 quick fixes, which allow you to fix detected problems. What is really cool is that ReSharper provides you with code inspection rules for all supported languages. This means that ReSharper not only improves your C# or VB.NET code, but also HTML, JavaScript, CSS, XAML, XML, ASP.NET, ASP.NET MVC, and TypeScript. Apart from finding possible errors, code quality analysis rules can also improve the readability of your code. ReSharper can detect code, which is unused and mark it as grayed, prompts you that maybe you should use auto properties or objects and collection initializers, or use the var keyword instead of an explicit type name. ReSharper provides you with five severity levels for rules and allows you to configure them according to your preference. Code inspection rules can be configured in the ReSharper's Options window. A sample view of code inspection rules with the list of available severity levels is shown in the following screenshot: Background analysis One of the best features in terms of code quality in ReSharper is Background analysis. This means that all the rules are checked as you are writing your code. You do not need to compile your project to see the results of the analysis. ReSharper will display appropriate messages in real time. Solution-wide inspections By default, the described rules are checked locally, which means that they should be checked in the current class. Because of this, ReSharper can mark some code as unused if it is used only locally; for example, there can be any unused private method or some part of code inside your method. These two cases are shown in the following screenshot: Additionally, for local analysis, ReSharper can check some rules in your entire project. To do this, you need to enable Solution-wide inspections. The easiest way to enable Solution-wide inspections is to double-click the circle icon in the bottom-right corner of Visual Studio, as seen in the following screenshot: With enabled Solution-wide inspections, ReSharper can mark the public methods or returned values that are unused. Please note that running Solution-wide inspections can hit Visual Studio’s performance in big projects. In such cases, it is better to disable this feature. Disabling code inspections With ReSharper v8, you can easily mark some part of your code as code that should not be checked by ReSharper. You can do this by adding the following comments: // ReSharper disable all // [your code] // ReSharper restore all All code between these two comments will be skipped by ReSharper in code inspections. Of course, instead of the all word, you can use the name of any ReSharper rule such as UseObjectOrCollectionInitializer. You can also disable ReSharper analysis for a single line with the following comment: // ReSharper disable once UseObjectOrCollectionInitializer ReSharper can generate these comments for you. If ReSharper highlights some issue, then just press Alt + Enter and select Options for “YOUR_RULE“ inspection, as shown in the following screenshot: Code Issues You can also an ad-hoc run code analysis. An ad-hoc analysis can be run on the solution or project level. To run ad-hoc analysis, just navigate to RESHARPER | Inspect | Code Issues in Solution or RESHARPER | Inspect | Code Issues in Current Project from the Visual Studio toolbar. This will display a dialog box that shows us the progress of analysis and will finally display the results in the Inspection Results window. You can filter and group the displayed issues as and when you need to. You can also quickly go to a place where the issue occurs just by double-clicking on it. A sample report is shown in the following screenshot: Eliminating errors and code smells We think you will agree that the code analysis provided by ReSharper is really cool and helps create better code. What is even cooler is that ReSharper provides you with features that can fix some issues automatically. Quick fixes Most errors and issues found by ReSharper can be fixed just by pressing Alt + Enter. This will display a list of the available solutions and let you select the best one for you. Fix in scope Quick fixes described above allow you to fix the issues in one particular place. However, sometimes there are issues that you would like to fix in every file in your project or solution. A great example is removing unused using statements or the this keyword. With ReSharper v8, you do not need to fix such issues manually. Instead, you can use a new feature called Fix in scope. You start as usual by pressing Alt + Enter but instead of just selecting some solution, you can select more options by clicking the small arrow on the right from the available options. A sample usage of the Fix in scope feature is shown in the following screenshot: This will allow you to fix the selected issue with just one click! Structural Search and Replace Even though ReSharper contains a lot of built-in analysis, it also allows you to create your own analyses. You can create your own patterns that will be used to search some structures in your code. This feature is called Structural Search and Replace (SSR). To open the Search with Pattern window, navigate to RESHARPER | Find | Search with Pattern…. A sample window is shown in the following screenshot: You can see two things here: On the left, there is place to write you pattern On the right, there is place to define placeholders In the preceding example, we were looking for if statements to compare them with some false expression. You can now simply click on the Find button and ReSharper will display every code that matches this pattern. Of course, you can also save your patterns. You can create new search patterns from the code editor. Just select some code, click on the right mouse button and select Find Similar Code….This will automatically generate the pattern for this code, which you can easily adjust to your needs. SSR allows you not only to find code based on defined patterns, but also replace it with different code. Click on the Replace button available on the top in the preceding screenshot. This will display a new section on the left called Replace pattern. There, you can write code that will be placed instead of code that matches the defined pattern. For the pattern shown, you can write the following code: if (false = $value$) { $statement$ } This will simply change the order of expressions inside the if statement. The saved patterns can also be presented as Quick fixes. Simply navigate to RESHARPER | Options | Code Inspection | Custom Patterns and set proper severity for your pattern, as shown in the following screenshot: This will allow you to define patterns in the code editor, which is shown in the following screenshot: Code Cleanup ReSharper also allows you to fix more than one issue in one run. Navigate to RESHARPER | Tools | Cleanup Code… from the Visual Studio toolbar or just press Ctrl + E, Ctrl + C. This will display the Code Cleanup window, which is shown in the following screenshot: By clicking on the Run button, ReSharper will fix all issues configured in the selected profile. By default, there are two patterns: Full Cleanup Reformat Code You can add your own pattern by clicking on the Edit Profiles button. Summary Code quality analysis is a very powerful feature in ReSharper. As we have described in this article, ReSharper not only prompts you when something is wrong or can be written better, but also allows you to quickly fix these issues. If you do not agree with all rules provided by ReSharper, you can easily configure them to meet your needs. There are many rules that will open your eyes and show you that you can write better code. With ReSharper, writing better, cleaner code is as easy as just pressing Alt + Enter. Resources for Article: Further resources on this subject: Ensuring Quality for Unit Testing with Microsoft Visual Studio 2010 [Article] Getting Started with Code::Blocks [Article] Core .NET Recipes [Article]

(For more resources related to this topic, see here.) Now that we have high availability and failover at the namespace level between datacenters, we need to achieve the same for the Mailbox server role. This is accomplished in a similar way to Exchange 2010, by extending a DAG across two or more datacenters. An organization's SLA covering failure and disaster recovery scenarios is what mostly influences a DAG's design. Every aspect needs to be considered: the number of DAGs to be deployed, the number of members in the DAG(s), the number of database copies, if site resilience is to be used and whether it has to be used for all users or just a subset, if the multiple site solution will be active/active or active/passive, and so on. As to the latter, there are generally three main scenarios when considering a two-datacenter model. Scenario 1 – active/passive In an active/passive configuration, all users' databases are mounted in an active (primary) datacenter, with a passive (standby) datacenter used only in the event of a disaster affecting the active datacenter; this is shown in the following diagram: There are several reasons why organizations might choose this model. Usually, it is because the passive datacenter is not as well equipped as the active one and, as such, is not capable of efficiently hosting all the services provided in the active datacenter. Sometimes, it is simply due to the fact that most or all users are closer to the active datacenter. In this example, a copy of each database in the New York datacenter is replicated to the MBX4 server in the New Jersey datacenter so they can be used in a disaster recovery scenario to provide messaging services to users. By also having database replicas in New York, we provide intrasite resilience. For example, if server MBX1 goes down, database DB1, which is currently mounted on server MBX1, will automatically failover to servers MBX2 or MBX3 without users even realizing or being affected. In some failure scenarios where a server shutdown is initiated (for example, when an Uninterruptible Power Supply (UPS) issues a shutdown command to the server), Exchange tries to activate another copy of the database(s) that the server is hosting, before the shutdown is complete. In case of a hard failure (for example, hardware failure), it will be the other servers detecting the problem and automatically mounting the affected database(s) on another server. In this scenario, we could lose up to two Mailbox servers in New York before having to perform a datacenter switchover. As New York is considered the primary datacenter, the witness server is placed in New York. If, for some reason, the primary site is lost, the majority of the quorum voters is lost, so the entire DAG goes offline. At this stage, administrators have to perform a datacenter switchover, just like in Exchange 2010. However, because the recovery of a DAG is decoupled from the recovery of the namespace, it becomes much easier to perform the switchover, assuming a global namespace is being used. All that the administrators need to do is run the following three cmdlets to get the DAG up and running again in New Jersey: Set the failed servers in the New York site as shown: Stop-DatabaseAvailabilityGroup <DAG_Name> -ActiveDirectorySite NewYork On the remaining DAG members, stop the Cluster service by running the following code line: Stop-Clussvc Activate the DAG members in New Jersey using the following code line: Restore-DatabaseAvailabilityGroup <DAG_Name> -ActiveDirectorySite NewJersey It is true that placing the witness server in the passive datacenter (when a DAG has the same number of nodes in both datacenters) would allow Exchange to automatically failover the DAG to the passive datacenter if the active site went down. However, there is a major disadvantage of doing this: if the passive site were to go down, even though it does not host any active databases, the entire DAG would go offline as the members in the active datacenter would not have quorum. This is why, in this scenario, it is recommended to always place the witness server in the active site. In order to prevent databases in New Jersey from being automatically mounted by Exchange, the Set-MailboxServer cmdlet can be used together with the DatabaseCopyAutoActivationPolicy parameter to specify the type of automatic activation on selected Mailbox servers. This parameter can be configured to any of the following values: Blocked: Prevents databases from being automatically activated on selected server(s). IntrasiteOnly: Only allows incoming mailbox database copies to be activated if the source server is on the same AD site, thus preventing cross-site activation or failover. Unrestricted: Allows mailbox database copies on selected server(s) from being activated independent of the location of the source database. This is the default value. For the preceding example, we would run the following cmdlet in order to prevent database copies from being automatically activated on MBX4: Set-MailboxServer MBX4 -DatabaseCopyAutoActivationPolicy Blocked As New York and New Jersey are on different AD sites, setting the DatabaseCopyAutoActivationPolicy parameter to IntrasiteOnly would achieve the same result. In either case, when performing a database switchover, administrators need to first remove the restriction on the target server, as shown in the following code, otherwise they will not be able to mount any databases. Set-MailboxServer MBX4 -DatabaseCopyAutoActivationPolicy Unrestricted Scenario 2 – active/active In this configuration, users' mailboxes are hosted across both datacenters. This is a very common scenario for deployments with a user population close to both locations. If Exchange fails for users in either of the datacenters, its services are activated on the other datacenter. Instead of simply having some active databases on the MBX4 server (refer to the preceding diagram), multiple DAG members are deployed in the New Jersey datacenter in order to provide protection against additional failures and additional capacity so it can support the entire user population in case the New York datacenter fails. By having more than one member in each datacenter, we are able to provide both intrasite and intersite resilience. Proper planning is crucial, especially capacity planning, so that each server is capable of hosting all workloads, including protocol request handling, processing, and data rendering from other servers without impacting the performance. In this example, the DAG is extended across both datacenters to provide site resilience for users on both sites. However, this particular scenario has a single point of failure: the network connection (most likely a WAN) between the datacenters. Remember that the majority of the voters must be active and able to talk to each other in order to maintain quorum. In the preceding diagram, the majority of voters are located in the New York datacenter, meaning that a WAN outage would cause a service failure for users whose mailboxes are mounted in New Jersey. This happens because when the WAN connection fails, only the DAG members in the New York datacenter are able to maintain the quorum. As such, servers in New Jersey will automatically bring their active database copies offline. In order to overcome this single point of failure, multiple DAGs should be implemented, with each DAG having a majority of voters in different datacenters, as shown in the following diagram: In this example, we would configure DAG1 with its witness server in New York and DAG2 with its witness server in New Jersey. By doing so, if a WAN outage happens, replication will fail, but all users will still have messaging services as both DAGs continue to retain quorum and are able to provide services to their local user population. Scenario 3 – third datacenter The third scenario is the only one to provide automatic DAG failover between datacenters. It involves splitting a DAG across two datacenters, as in the previous scenarios, with the difference that the witness server is placed in a third location. This allows it to be arbitrated by members of the DAG in both datacenters independent of the network state between the two sites. As such, it is a key to place the witness server in a location that is isolated from possible network failures that might affect either location containing the DAG members. This was fully supported in Exchange 2010, but the downside was that the solution itself would not get automatically failed over as the namespace would still need to be manually switched over. For this reason, it was not recommended. Going back to the advantage of the namespace in Exchange 2013 not needing to move with the DAG, this entire process now becomes automatic as shown in the following diagram: However, even though this scenario is now recommended, special consideration needs to be taken into account for when the network link between the two datacenters hosting Exchange mailboxes fail. For example, even though the DAG will continue to be fully operational on both datacenters, CASs in New York will not be able to proxy requests to servers in New Jersey. As such, proper planning is necessary in order to minimize user impact in such an event. One way of doing this is to ensure that DNS servers that are local to users only resolve the namespace to the VIP on the same site. This would cancel the advantages of single and global namespace, but as a workaround during an outage, it would reduce cross-site connections, ensuring users are not affected. Windows Azure Microsoft has been testing the possibility of placing a DAG's witness server in a Windows Azure IaaS (Infrastructure as a Service) environment. However, this infrastructure does not yet support the necessary network components to cater to this scenario. At the time of writing this book, Azure supports two types of networks: single site-to-site VPN (a network connecting two locations) and one or more point-to-site VPNs (a network connecting a single VPN client to a location). The issue is that in order for a server to be placed in Azure and configured as a witness server, two site-to-site VPNs would be required, connecting each datacenter hosting Exchange servers to Azure, which is not possible today. As such, the use of Azure to place a DAG's witness server is not supported at this stage. Using Datacenter Activation Coordination (DAC) DAC is a DAG setting that is disabled by default. It is used to control the startup behavior of databases. Let us suppose the following scenario: a DAG split across two datacenters, like the one shown in the first diagram of the section Scenario 2 – active/active, and the primary datacenter suffers a complete power outage. All servers and the WAN link are down, so a decision is made to activate the standby datacenter. Usually in such scenarios, WAN connectivity is not instantly restored when the primary datacenter gets its power back. When this happens, members of the DAG in the primary datacenter are powered up but are not able to communicate with other members in the standby datacenter that is currently active. As the primary datacenter contains most of the DAG quorum voters (or so it should), when the power is restored, the DAG members located in the primary datacenter have the majority; so, they have quorum. The issue with this is that with quorum, they can mount their databases (assuming everything required to do so is operational, such as storage), which causes discrepancy with the actual active databases mounted in the standby datacenter. So now we have the exact same databases mounted simultaneously in separate servers. This is commonly known as split brain syndrome. DAC was specifically created to prevent a split brain scenario. It does so through the Datacenter Activation Coordination Protocol (DACP) protocol. Once such a failure occurs, when the DAG is recovered, it will not automatically mount databases even if it has quorum. DACP is instead used to evaluate the DAG's current state and if databases should be mounted or not in each server by the Active Manager. Active Manager uses memory to store a bit (a 0 or a 1); so, the DAG knows if it can mount databases that are assigned as active on the local server. When DAC is enabled, every time the Active Manager is started, it sets the bit to 0, meaning it is not allowed to mount any databases. The server is then forced to establish communication with the other DAG members in order to get another server to tell it if it is allowed to mount its local databases or not. The answer from the other members is simply their bit setting in the DAG. If a server replies that it has its bit set to 1, the server is permitted to mount databases and also set its own bit to 1. On the other hand, when restoring the DAG in the preceding scenario, all the members of the DAG in the primary datacenter will have their DACP bit set to 0. As such, none of the servers powering up in the recovered primary datacenter are allowed to mount any databases because none of them are able to communicate with a server that has a DACP bit set to 1. Besides dealing for split brain scenarios, enabling DAC mode allows administrators to use the site resilience built-in cmdlets to carry out datacenter switchovers: Stop-DatabaseAvailabilityGroup Restore-DatabaseAvailabilityGroup Start-DatabaseAvailabilityGroup When DAC mode is disabled, both Exchange and cluster management tools need to be used when performing datacenter switchovers. Enabling the DAC mode DAC can only be enabled or disabled using the Set-DatabaseAvailabilityGroup cmdlet together with the DatacenterActivationMode parameter. To enable DAC mode, this parameter is set to DagOnly and to disable it, it is set to Off: Set-DatabaseAvailabilityGroup <DAG_Name> -DatacenterActivationMode DagOnly Deciding where to place witness servers When designing and configuring a DAG, it is important to consider the location of the witness server. As we have seen, this is very much dependent on the business requirements and what is available to the organization. As already discussed, Exchange 2013 allows scenarios that were not previously recommended, such as placing a witness server on a third location. The following table summarizes the general recommendations around the placement of the witness servers according to different deployment scenarios: Scenario Place Witness Server In... # DAGs # Datacenters 1 1 The datacenter where the DAG members are located. 1 2 The primary datacenter (refer to diagrams of section Scenario 1 –active/passive and Scenario 2 – active/active). A third location that is isolated from possible network failures that might affect either datacenter containing DAG members. 2+ 1 The datacenter where the DAG members are located. The same witness server can be used for multiple DAGs. A DAG member can be used as a witness server for another DAG. 2+ 2 The datacenter where the DAG members are located. The same witness server can be used for multiple DAGs. A DAG member can be used as a witness server for another DAG. A third location that is isolated from possible network failures that might affect either datacenter containing DAG members. 1 or 2+ 3+ The datacenter where administrators want the majority of voters to be. A third location that is isolated from possible network failures that might affect either datacenter containing DAG members. Summary Throughout this article, we explored all the great enhancements made to Exchange 2013 in regards to site resilience for the Mailbox server. As the recovery of a DAG is no longer tied together with the recovery of the client access namespace, each one of these components can be easily switched over between different datacenters without affecting the other component. This also allows administrators to place a witness server in a third datacenter in order to provide automatic failover for a DAG on either of the datacenters it is split across, something not recommended in Exchange 2010. Resources for Article: Further resources on this subject: Microsoft DAC 2012 [article] Choosing Lync 2013 Clients [article] Installing Microsoft Dynamics NAV [article]

(For more resources related to this topic, see here.) Creating animations with CreateJS As you may already know, creating animations in web browsers during web development is a difficult job because you have to write code that has to work in all browsers; this is called browser compatibility. The good news is that CreateJS provides modules to write and develop animations in web browsers without thinking about browser compatibility. CreateJS modules can do this job very well and all you need to do is work with CreateJS API. Understanding TweenJS TweenJS is one of the modules of CreateJS that helps you develop animations in web browsers. We will now introduce TweenJS. The TweenJS JavaScript library provides a simple but powerful tweening interface. It supports tweening of both numeric object properties and CSS style properties, and allows you to chain tweens and actions together to create complex sequences.—TweenJS API Documentation What is tweening? Let us understand precisely what tweening means: Inbetweening or tweening is the process of generating intermediate frames between two images to give the appearance that the first image evolves smoothly into the second image.—Wikipedia The same as other CreateJS subsets, TweenJS contains many functions and methods; however, we are going to work with and create examples for specific basic methods, based on which you can read the rest of the documentation of TweenJS to create more complex animations. Understanding API and methods of TweenJS In order to create animations in TweenJS, you don't have to work with a lot of methods. There are a few functions that help you to create animations. Following are all the methods with a brief description: get: It returns a new tween instance. to: It queues a tween from the current values to the target properties. set: It queues an action to set the specified properties on the specified target. wait: It queues a wait (essentially an empty tween). call: It queues an action to call the specified function. play: It queues an action to play (un-pause) the specified tween. pause: It queues an action to pause the specified tween. The following is an example of using the Tweening API: var tween = createjs.Tween.get(myTarget).to({x:300},400). set({label:"hello!"}).wait(500).to({alpha:0,visible:false},1000). call(onComplete); The previous example will create a tween, which: Tweens the target to an x value of 300 with duration 400ms and sets its label to hello!. Waits 500ms. Tweens the target's alpha property to 0with duration 1s and sets the visible property to false. Finally, calls the onComplete function. Creating a simple animation Now, it's time to create our simplest animation with TweenJS. It is a simple but powerful API, which gives you the ability to develop animations with method chaining. Scenario The animation has a red ball that comes from the top of the Canvas element and then drops down. In the preceding screenshot, you can see all the steps of our simple animation; consequently, you can predict what we need to do to prepare this animation. In our animation,we are going to use two methods: get and to. The following is the complete source code for our animation: var canvas = document.getElementById("canvas"); var stage = new createjs.Stage(canvas); var ball = new createjs.Shape(); ball.graphics.beginFill("#FF0000").drawCircle(0, 0, 50); ball.x = 200; ball.y = -50; var tween = createjs.Tween.get(ball) to({ y: 300 }, 1500, createjs.Ease.bounceOut); stage.addChild(ball); createjs.Ticker.addEventListener("tick", stage); In the second and third line of the JavaScript code snippet, two variables are declared, namely; the canvas and stage objects. In the next line, the ball variable is declared, which contains our shape object. In the following line, we drew a red circle with the drawCircle method. Then, in order to set the coordinates of our shape object outside the viewport, we set the x axis to -50 px. After this, we created a tween variable, which holds the Tween object; then, using the TweenJS method chaining, the to method is called with duration of 1500 ms and the y property set to 300 px. The third parameter of the to method represents the ease function of tween, which we set to bounceOut in this example. In the following lines, the ball variable is added to Stage and the tick event is added to the Ticker class to keep Stage updated while the animation is playing. You can also find the Canvas element in line 30, using which all animations and shapes are rendered in this element. Transforming shapes CreateJS provides some functions to transform shapes easily on Stage. Each DisplayObject has a setTransform method that allows the transforming of a DisplayObject (like a circle). The following shortcut method is used to quickly set the transform properties on the display object. All its parameters are optional. Omitted parameters will have the default value set. setTransform([x=0] [y=0] [scaleX=1] [scaleY=1] [rotation=0] [skewX=0] [skewY=0] [regX=0] [regY=0]) Furthermore, you can change all the properties via DisplayObject directly (like scaleY and scaleX) as shown in the following example: displayObject.setTransform(100, 100, 2, 2); An example of Transforming function As an instance of using the shape transforming feature with CreateJS, we are going to extend our previous example: var angle = 0; window.ball; var canvas = document.getElementById("canvas"); var stage = new createjs.Stage(canvas); ball = new createjs.Shape(); ball.graphics.beginFill("#FF0000").drawCircle(0, 0, 50); ball.x = 200; ball.y = 300; stage.addChild(ball); function tick(event) { angle += 0.025; var scale = Math.cos(angle); ball.setTransform(ball.x, ball.y, scale, scale); stage.update(event); } createjs.Ticker.addEventListener("tick", tick); In this example, we have a red circle, similar to the previous example of tweening. We set the coordinates of the circle to 200 and 300 and added the circle to the stage object. In the next line, we have a tick function that transforms the shape of the circle. Inside this function, we have an angle variable that increases with each call. We then set the ballX and ballY coordinate to the cosine of the angle variable. The transforming done is similar to the following screenshot: This is a basic example of transforming shapes in CreateJS, but obviously, you can develop and create better transforming by playing with a shape's properties and values. Summary In this article, we covered how to use animation and transform objects on the page using CreateJS. Resources for Article: Further resources on this subject: Introducing a feature of IntroJs [Article] So, what is Node.js? [Article] So, what is Ext JS? [Article]

(For more resources related to this topic, see here.) A health check is a runtime test for our application. We are going to create a health check that tests the creation of new contacts using the Jersey client. The health check results are accessible through the admin port of our application, which by default is 8081. How to do it… To add a health check perform the following steps: Create a new package called com.dwbook.phonebook.health and a class named NewContactHealthCheck in it: import javax.ws.rs.core.MediaType; import com.codahale.metrics.health.HealthCheck; import com.dwbook.phonebook.representations.Contact; import com.sun.jersey.api.client.*; public class NewContactHealthCheck extends HealthCheck { private final Client client; public NewContactHealthCheck(Client client) { super(); this.client = client; } @Override protected Result check() throws Exception { WebResource contactResource = client .resource("http://localhost:8080/contact"); ClientResponse response = contactResource.type( MediaType.APPLICATION_JSON).post( ClientResponse.class, new Contact(0, "Health Check First Name", "Health Check Last Name", "00000000")); if (response.getStatus() == 201) { return Result.healthy(); } else { return Result.unhealthy("New Contact cannot be created!"); } } } Register the health check with the Dropwizard environment by using the HealthCheckRegistry#register() method within the #run() method of the App class. You will first need to import com.dwbook.phonebook.health.NewContactHealthCheck. The HealthCheckRegistry can be accessed using the Environment#healthChecks() method: // Add health checks e.healthChecks().register ("New Contact health check", new NewContactHealthCheck(client)); After building and starting your application, navigate with your browser to http://localhost:8081/healthcheck: The results of the defined health checks are presented in the JSON format. In case the custom health check we just created or any other health check fails, it will be flagged as "healthy": false, letting you know that your application faces runtime problems. How it works… We used exactly the same code used by our client class in order to create a health check; that is, a runtime test that confirms that the new contacts can be created by performing HTTP POST requests to the appropriate endpoint of the ContactResource class. This health check gives us the required confidence that our web service is functional. All we need for the creation of a health check is a class that extends HealthCheck and implements the #check() method. In the class's constructor, we call the parent class's constructor specifying the name of our check—the one that will be used to identify our health check. In the #check() method, we literally implement a check. We check that everything is as it should be. If so, we return Result.healthy(), else we return Result.unhealthy(), indicating that something is going wrong. Summary This article showed what a health check is and demonstrated how to add a health check. The health check we created tested the creation of new contacts using the Jersey client. Resources for Article: Further resources on this subject: RESTful Web Services – Server-Sent Events (SSE) [Article] Connecting to a web service (Should know) [Article] Web Services and Forms [Article]

(For more resources related to this topic, see here.) Graphics are points, lines, or polygons that are drawn on top of your map in a layer that is independent of any other data layer associated with a map service. Most people associate a graphic object with the symbol that is displayed on a map to represent the graphic. However, each graphic in ArcGIS Server can be composed of up to four objects, including the geometry of the graphic, the symbology associated with the graphic, attributes that describe the graphic, and an info template that defines the format of the info window that appears when a graphic is clicked on. Although a graphic can be composed of up to four objects, it is not always necessary for this to happen. The objects you choose to associate with your graphic will be dependent on the needs of the application that you are building. For example, in an application that displays GPS coordinates on a map, you may not need to associate attributes or display info window for the graphic. However, in most cases, you will be defining the geometry and symbology for a graphic. Graphics are temporary objects stored in a separate layer on the map. They are displayed while an application is in use and are removed when the session is complete. The separate layer, called the graphics layer, stores all the graphics associated with your map. Just as with the other types of layers, GraphicsLayer also inherits from the Layer class. Therefore, all the properties, methods, and events found in the Layer class will also be present in GraphicsLayer. Graphics are displayed on top of any other layers that are present in your application. An example of point and polygon graphics is provided in the following screenshot. These graphics can be created by users or drawn by the application in response to the tasks that have been submitted. For example, a business analysis application might provide a tool that allows the user to draw a freehand polygon to represent a potential trade area. The polygon graphic would be displayed on top of the map, and could then be used as an input to a geoprocessing task that pulls demographic information pertaining to the potential trade area. Many ArcGIS Server tasks return their results as graphics. The QueryTask object can perform both attribute and spatial queries. The results of a query are then returned to the application in the form of a FeatureSet object, which is simply an array of features. You can then access each of these features as graphics and plot them on the map using a looping structure. Perhaps you'd like to find and display all land parcels that intersect the 100 year flood plain. A QueryTask object could perform the spatial query and then return the results to your application, where they would then be displayed as polygon graphics on the map. In this article, we will cover the following topics: The four parts of a graphic Creating geometry for graphics Symbolizing graphics Assigning attributes to graphics Displaying graphic attributes in an info window Creating graphics Adding graphics to the graphics layer The four parts of a graphic A graphic is composed of four items: Geometry, Symbol, Attributes, and InfoTemplate, as shown in the following diagram: A graphic has a geometric representation that describes where it is located. The geometry, along with a symbol, defines how the graphic is displayed. A graphic can also have attributes that provide descriptive information about the graphic. Attributes are defined as a set of name-value pairs. For example, a graphic depicting a wildfire location could have attributes that describe the name of the fire along with the number of acres burned. The info template defines what attributes should be displayed in the info window that appears when the graphic appears, along with how they should be displayed. After their creation, the graphic objects must be stored inside a GraphicsLayer object, before they can be displayed on the map. This GraphicsLayer object functions as a container for all the graphics that will be displayed. All the elements of a graphic are optional. However, the geometry and symbology of a graphic are almost always assigned. Without these two items, there would be nothing to display on the map, and there isn't much point in having a graphic unless you're going to display it. The following figure shows the typical process of creating a graphic and adding it to the graphics layer. In this case, we are applying the geometry of the graphic as well as a symbol to depict the graphic. However, we haven't specifically assigned attributes or an info template to this graphic.