How-To Tutorials

(For more resources related to this topic, see here.) Introduction to the YouTube API YouTube provides three different APIs for a client application to access. The following figure shows the three different APIs provided by YouTube: Configuring a YouTube API In the Google Developers Console, we need to create a client project. We will be creating a new project, called PacktYoutubeapi. The URL for the Google Developers Console is https://console.developers.google.com. The following screenshot shows the pop-up window that appears when you want to create a new client project in the Developers Console: After the successful creation of the new client project, it will be available in the Console's project list. The following screenshot shows our new client project listed in the Developers Console: There is an option available to enable access to the YouTube API for our application. The following screenshot shows the YouTube API listed in the Developers Console. By default, the status of this API is OFF for the application. To enable this API for our application, we need to toggle the STATUS button to ON. The following screenshot shows the status of the YouTube API, which is ON for our application: To access YouTube API methods, we need to create an API key for our client application. You can find the option to create a public API key in the APIs & auth section. The following screenshot shows the Credentials subsection where you can create an API key: In the preceding screenshot, you can see a button to create a new API key. After clicking on this button, it provides some choices to create an API key, and after the successful creation of an API key, the key will be listed in the Credentials section. The following screenshot shows the API key generated for our application: Searching for a YouTube video In this section, we will learn about integrating a YouTube-related search video. YouTube Data API Version 3.0 is the new API to access YouTube data. It requires the API key that has been created in the previous section. The main steps that we have to follow to do a YouTube search are: After adding the YouTube Search button, click on it to trigger the search process. The script reads the data-booktitle attribute to get the title. This will serve as a keyword for the search. Check the following screenshot for the HTML markup showing the data-booktitle attribute: Then, it creates an AJAX request to make an asynchronous call to the YouTube API, and returns a promise object. After the successful completion of the AJAX call, the promise object is resolved successfully. Once the data is available, we fetch the jQuery template for the search results and compile it with a script function. We then link it to the search data returned by the AJAX call and generate the HTML markup for rendering. The base URL for the YouTube search is through a secure HTTP protocol, https://www.googleapis.com/youtube/v3/search. It takes different parameters as input for the search and filter criteria. Some of the important parameters are field and part. The part parameter The part parameter is about accessing a resource from a YouTube API. It really helps the application to choose resource components that your application actually uses. The following figure shows some of the resource components: The fields parameter The fields parameter is used to filter out the exact fields that are needed by the client application. This is really helpful to reduce the size of the response. For example, fields = items(id, snippet(title)) will result in a small footprint of a response containing an ID and a title. The YouTube button markup We have added a button in our jQuery product template to display the search option in the product. The following code shows the updated template: <script id="aProductTemplate" type="text/x-jquery-tmpl"> <div class="ts-product panel panel-default"> <div class="panel-head"> <div class="fb-like" data-href="${url}" datalayout=" button_count" data-action="like" data-show-faces="true" datashare=" true"> </div> </div> <div class="panel-body"> <span class="glyphicon glyphicon-certificate ts-costicon"> <label>${cost}$</label> </span> <img class="img-responsive" src ="${url}"> <h5>${title}</h5> </div> <div class="panel-footer"> <button type="button" class="btn btn-danger btn-block packt-youtube-button" data-bookTitle="${title}">YouTube Search</ button> <button type="button" class="btn btn-info btn-block">Buy</ button> <button type="button" class="btn btn-info btn-block twitme" data-bookTitle="${title}" data-imgURI="${url}">Tweet</button> <div class="g-plus-button"> <div class="g-plusone" data-width="180" datahref="${ url}"></div> </div> </div> </div> </script> The following screenshot shows the updated product markup with a YouTube button added to the product template:

(For more resources related to this topic, see here.) An evaluation framework for .NET Framework APIs Understanding the .NET Framework in its entirety, including keeping track of the APIs that are available in various versions (for example, 3.5, 4, 4.5, 4.5.1, and so on, and platforms such as Windows 8, Windows Phone 8, and Silverlight 5) is a near impossible undertaking. What software developers and architects need is a high-level framework to logically partition the .NET Framework and identify the APIs that should be used to address a given requirement or category of requirements. API boundaries in the .NET Framework can be a little fuzzy. Some logical APIs span multiple assemblies and namespaces. Some are nicely contained within a neat hierarchy within a single root namespace. To confuse matters even further, single assemblies might contain portions of multiple APIs. The most practical way to distinguish an API is to use the API's root namespace or the namespace that contains the majority of the API's implementation. We will point out the cases where an API spans multiple namespaces or there are peculiarities in the namespaces of an API. Evaluation framework dimensions The dimensions for .NET Framework API evaluation framework are as follows: Maturity: This dimension indicates how long the API has been available, how long it has been part of the .NET Framework, and what the API's expected longevity is. It is also a measure of how relevant the API is or an indication that an API has been subsumed by newer and better APIs. Productivity: This dimension is an indication of how the use of the API will impact developer productivity. This dimension is measured by how easy the API is to learn and use, how well known the API is within the developer community, how simple or complex it is to use, the richness of the tool support (primarily in Visual Studio), and how abstract the API is, that is, whether it is declarative or imperative. Performance: This dimension indicates whether the API was designed with performance, resource utilization, user interface responsiveness, or scalability in mind; alternatively, it indicates whether convenience, ease of use, or code pithiness were the primary design criteria, which often comes at the expense of the former. Availability: This dimension indicates whether the API is available only on limited versions of the .NET Framework and Microsoft operating systems, or whether it is available everywhere that managed code is executed, including third-party implementations on non-Microsoft operating systems, for example, Mono on Linux. Evaluation framework ratings Each dimension of the API evaluation framework is given a four-level rating. Let's take a look at the ratings for each of the dimensions. The following table describes the ratings for Maturity: Rating Glyph Description Emerging This refers to a new API that was either added to the .NET Framework in the last release or is a candidate for addition in an upcoming release that has not gained widespread adoption yet. This also includes APIs that are not officially part of the .NET Framework. New and promising This is an API that has been in the .NET Framework for a couple of releases; it is already being used by the community in production systems, but it has yet to hit the mainstream. This rating may also include Microsoft APIs that are not officially part of .NET, but show a great deal of promise, or are being used extensively in production. Tried and tested This is an API that has been in the .NET Framework for multiple releases, has attained very broad adoption, has been refined and improved with each release, and is probably not going to be subsumed by a new API or deprecated in a later version of the Framework. Showing its age The API is no longer relevant or has been subsumed by a superior API, entirely deprecated in recent versions of .NET, or metamorphosedmerged into a new API. The following table describes the ratings for Productivity: Rating Glyph Description Decrease This is a complex API that is difficult to learn and use and not widely understood within the .NET developer community. Typically, these APIs are imperative, that is, they expose the underlying plumbing that needs to be understood to correctly use the API, and there is little or no tooling provided in Visual Studio. Using this API results in lowered developer productivity. No or little impact This API is fairly well known and used by the .NET developer community, but its use will have little effect on productivity, either because of its complexity, steep learning curve, and lack of tool support, or because there is simply no alternative API Increase This API is well known and used by the .NET developer community, is easy to learn and use, has good tool support, and is typically declarative; that is, the API allows the developer to express the behavior they want without requiring an understanding of the underlying plumbing, and in minimal lines of code too. Significant increase This API is very well known and used in the .NET developer community, is very easy to learn, has excellent tool support, and is declarative and pithy. Its use will significantly improve developer productivity. Performance and Scalability Rating Glyph Description Decrease The API was designed for developer productivity or convenience and will more than likely result in the slower execution of code and the increased usage of system resources (when compared to the use of other .NET APIs that provide the same or similar capabilities). Do not use this API if performance is a concern. No or little impact The API strikes a good balance between performance and developer productivity. Using it should not significantly impact the performance or scalability of your application. If performance is a concern, you can use the API, but do so with caution and make sure you measure its impact. Increase The API has been optimized for performance or scalability, and it generally results in faster, more scalable code that uses fewer system resources. It is safe to use in performance-sensitive code paths if best practices are followed. Significant increase The API was designed and written from the ground up with performance and scalability in mind. The use of this API will result in a significant improvement of performance and scalability over other APIs. The following table describes the ratings for Availability: Rating Glyph Description Rare The API is available in limited versions of the .NET Framework and on limited operating systems. Avoid this API if you are writing code that needs to be portable across all platforms. Limited This API is available on most versions of the .NET Framework and Microsoft operating systems. It is generally safe to use, unless you are targeting very old versions of .NET or Microsoft operating systems. Microsoft Only This API is available on all versions of the .NET Framework and all Microsoft operating systems. It is safe to use if you are on the Microsoft platform and are not targeting third-party CLI implementations, such as Mono.   Universal The API is available on all versions of .NET, including those from third parties, and it is available on all operating systems, including non-Microsoft operating systems. It is always safe to use this API. The .NET Framework The rest of this article will highlight some of the more commonly used APIs within the .NET Framework and rate each of these APIs using the Evaluation framework described previously. The Base Class Library The Base Class Library (BCL) is the heart of the .NET Framework. It contains base types, collections, and APIs to work with events and attributes; console, file, and network I/O; and text, XML, threads, application domains, security, debugging, tracing, serialization, interoperation with native COM and Win32 APIs, and the other core capabilities that most .NET applications need. The BCL is contained within the mscorlib.dll, System.dll, and System.Core.dll assemblies The mscorlib.dll assembly is loaded during the CLR Bootstrap(not by the CLR Loader), contains all non optional APIs and types, and is universally available in every .NET process, such as Silverlight, Windows Phone, and ASP.NET. Optional BCL APIs and types are available in System.dll and System.Core.dll, which are loaded on demand by the CLR Loader, as with all other managed assemblies. It would be a rare exception, however, when a .NET application does not use either of these two aforementioned assemblies since they contain some very useful APIs. When creating any project type in Visual Studio, these assemblies will be referenced by default. For the purpose of this framework, we will treat all of the BCL as a logical unit and not differentiate the nonoptional APIs (that is, the ones contained within mscorlib.dll), from the optional ones. Despite being a significant subset of the .NET Framework, the BCL contains a significant number of namespaces and APIs. The next sections describe a partial list of some of the more notable namespaces/APIs within the BCL, with an evaluation for each: System namespace System.Text namespace System.IO namespace System.Net namespace System.Collections namespace System.Collections.Generic namespace System.Collections.Concurrent namespace System.Linq namespace System.Xml namespace System.Xml.Linq namespace System.Security.Cryptography namespace System.Threading namespace System.Threading.Tasks namespace System.ServiceProcess namespace System.ComponentModel.Composition namespace System.ComponentModel.DataAnnotations namespace ADO.NET Most computer programs are meaningless without appropriate data to operate over. Accessing this data in an efficient way has become one of the greatest challenges modern developers face as the datasets have grown in size, from megabytes, to gigabytes, to terabytes, and now petabytes, in the most extreme cases, for example, Google's search database is around a petabyte. Though relational databases no longer hold the scalability high ground, a significant percentage of the world's data still resides in them and will probably continue to do so for the foreseeable future. ADO.NET contains a number of APIs to work with relational data and data provider APIs to access Microsoft SQL Server, Oracle Database, OLEDB, ODBC, and SQL Server Compact Edition. System.Data namespace System.Data.Entity namespace System.Data.Linq namespace System.Data.Services namespace Windows Forms Windows Forms (WinForms) was the original API for developing the user interface (UI) of Windows desktop applications with the .NET Framework. It was released in the first version of .NET and every version since then. System.Windows.Forms namespace The WinForms API is contained within the System.Windows.Forms namespace. Though WinForms is a managed API, it is actually a fairly thin façade over earlier, unmanaged APIs, primarily Win32 and User32, and any advanced use of WinForms requires a good understanding of these underlying APIs. The advanced customizations of WinForms controls often require the use of the System.Drawing API, which is also just a managed shim over the unmanaged GDI+ API. Many new applications are still developed using WinForms, despite its age and the alternative .NET user interface APIs that are available. It is a very well understood API, is very stable, and has been optimized for performance (though it is not GPU-accelerated like WPF or WinRT). There are a significant number of vendors who produce feature-rich, high-quality, third-party WinForms controls, and WinForms is available in every version of .NET and on most platforms, including Mono. WinForms is clearly showing its age, particularly when its capabilities are compared to those of WPF and WinRT, but it is still a viable API for applications that exclusively target the desktop and where a sophisticated modern UI is not necessary. The following table shows the evaluation of the System.Windows.Forms namespace: Maturity Productivity Performance Availability Windows Presentation Foundation Windows Presentation Foundation (WPF) is an API, introduced in .NET 3.0, for developing rich user interfaces for .NET applications, with no dependencies on legacy Windows APIs and with support for GPU-accelerated 3D rendering, animation, and media playback. If you want to play a video on a clickable button control on the surface of an animated, 3D rotating cube and the only C# code you want to write is the button click event handler, then WPF is the API for the job. See the WPFSample code for a demonstration. System.Windows namespace The System.Windows namespace contains the Windows Presentation Foundation API. WPF includes many of the "standard" controls that are in WinForms, for example, Button, Label, CheckBox, ComboBox, and so on. However, it also includes APIs to create, animate, and render 3D graphics; render multimedia; draw bitmap and vector graphics; and perform animation. WPF addresses many of the limitations of Windows Forms, but this power comes at a price. WPF introduces a number of novel concepts that developers will need to master, including a new, declarative UI markup called Extensible Application Markup Language (XAML), new event handling, data binding and control theming mechanisms, and a variant of the Model-view-controller (MVC) pattern called Model View ViewModel (MVVM); that said, the use of this pattern is optional but highly recommended. WPF has significantly more moving parts than WinForms, if you ignore the underlying native Windows APIs that WinForm abstracts. Microsoft, though, has gone to some lengths to make the WPF development experience easier for both UI designers and developers. Developers using WPF can choose to design and develop user interfaces using XAML, any of the .NET languages, or most often a combination of the two. Visual Studio and Expression Blend provide rich WYSIWYG designers to create WPF controls and interfaces and hide the complexities of the underlying XAML. Direct tweaking of the XAML is sometimes required for precise adjustments. WPF is now a mature, stable API that has been highly optimized for performance—all of its APIs are GPU accelerated. Though it is probably not as well known as WinForms, it has become relatively well known within the developer community, particularly because Silverlight, which is Microsoft's platform for developing rich web and mobile applications, uses a subset of WPF. Many of the third-party control vendors who produce WinForm controls now also produce equivalent WPF controls. The tools for creating WPF applications, predominantly Visual Studio and Expression Blend, are particularly good, and there are also a number of good third-party and open source tools to work with XAML. The introduction of WinRT and the increasingly powerful capabilities of web browser technologies, including HTML5, CSS3, JavaScript, WebGL, and GPU-acceleration, raise valid questions about the long-term future of WPF and Silverlight. Microsoft seems to be continuing to promote the use of WPF, and even WinRT supports a variant of the XAML markup language, so it should remain a viable API for a while. The following table shows the evaluation of the System.Windows namespace: Maturity Productivity Performance Availability ASP.NET The .NET Framework was originally designed to be Microsoft's first web development platform, and it included APIs to build both web applications and web services. These APIs were, and still are, part of the ASP.NET web development framework that lives in the System.Web namespace. ASP.NET has come a very long way since the first release of .NET, and it is the second most widely used and popular web framework in the world today (see http://trends.builtwith.com/framework). The ASP.NET platform provides a number of complimentary APIs that can be used to develop web applications, including Web Forms, web services, MVC, web pages, Web API, and SignalR. System.Web.Forms namespace System.Web.Mvc namespace System.Web.WebPages namespace System.Web.Services namespace Microsoft.AspNet.SignalR namespace Windows Communication Foundation One of the major selling points of the first release of .NET was that the platform had support for web services baked in, in the form of ASP.NET Web Services. Web Services have come a very long way since SOAP was invented in 1998 and the first release of .NET, and WCF has subsumed the limited capabilities of ASP.NET Web Services with a far richer platform. WCF has also subsumed the original .NET Remoting (System.Runtime.Remoting), MSMQ (System.Messaging), and Enterprise Services (System.EnterpriseServices) APIs. System.ServiceModel namespace The root namespace for WCF is System.ServiceModel. This API includes support for most of the WS-* web services standards and non-HTTP or XML-based services, including MSMQ and TCP services that use binary or Message Transmission Optimization Mechanism (MTOM) message encoding. Address, Binding, and Contract (ABC) of WCF are very well understood by the majority of the developer community, though deep technical knowledge of WCF's inner workings is rarer. The use of attributes to declare service and data contracts and a configuration-over-code approach makes the WCF API highly declarative, and creating sophisticated services that use advanced WS-* capabilities is relatively easy. WCF is very stable and can be used to create high-performance distributed applications. WCF is available on all recent versions of .NET, though not all platforms include the server components of WCF. Partial support for WCF is also available on third-party CLI implementations, such as Mono. REST-based web services, that serve relatively simple XML or JSON, have become very popular, and though WCF fairly recently added support for REST, these capabilities have now evolved into the ASP.NET Web API. The following table shows the evaluation of the System.ServiceModel namespace: Maturity Productivity Performance Availability Windows Workflow Foundation Windows Workflow Foundation (WF) is a workflow framework that was introduced in .NET 3.0, and that brings the power and flexibility of declarative workflow or business process design and execution to .NET applications. System.Activities namespace The System.Activities namespace contains the Windows Workflow Foundation API. WF includes a workflow runtime, a hosting API, a number of basic workflow activities, APIs to create custom activities, and a workflow designer control, which was originally a WinForms control but is now a WPF control as of .NET 4.0. WF also uses a variant of the same XAML markup, which WPF and WinRT use, to represent workflows; that said, an excellent designer, hosted by default in Visual Studio, should mean that you never have to directly modify the XAML. The adoption of the first few versions of the WF API was limited, but WF was completely rewritten for .NET 4.0, and many of the shortcomings of the original version were entirely addressed. WF is now a mature, stable, best-of-breed workflow API, with a proven track record. The previous implementation of WF is still available in current versions of the .NET Framework, for migration and interoperation purposes, and is in the System.Workflow namespace. WF is used by SharePoint Server, Windows Server AppFabric, Windows Azure AppFabric, Office 365, Visual Studio Team Foundation Server (MSBuild), and a number of other Microsoft products and services. Windows Server AppFabric and Windows Azure AppFabric enable a new class of scalable SOA server and cloud application called a Workflow Service, which is a combination of the capabilities of WCF and WF. WF has a relatively small but strong following within the .NET developer community. There are also a number of third-party and open source WF activity libraries and tools available. Though applications composed using workflows typically have poorer performance than those that are implemented entirely in code, the flexibility and significantly increased developer productivity (particularly when it comes to modifying existing processes) that workflows give you are often worth the performance price. That said, Microsoft has made significant investments in optimizing the performance of WF, and it should be more than adequate for most enterprise application scenarios. Though versions of WF are available on other CLI platforms, the availability of WF 4.x is limited to Microsoft platforms and .NET 4.0 and higher. The evaluation of the System.Workflow namespace shown in the following table is for the most recent version of WF (the use of versions of WF prior to 4.0 is not recommended for new applications): Maturity Productivity Performance Availability Summary There is more to the .NET Framework than has been articulated in this primer; it includes many useful APIs that have not even been mentioned here, for example, System.Media, System.Speech, and the Windows Identity Framework. There are also a number of very powerful APIs developed by Microsoft (and Microsoft Research) that are not (yet) officially part of the .NET Framework; for example, Reactive Extensions, Microsoft Solver Foundation, Windows Azure APIs, and the new .NET for Windows Store Apps APIs are worth looking into. Resources for Article:   Further resources on this subject: Content Based Routing on Microsoft Platform [article] Building the Content Based Routing Solution on Microsoft Platform [article] Debatching Bulk Data on Microsoft Platform [article]

(For more resources related to this topic, see here.) Prepare for lift off We will begin by creating a new project in Unity. Let's start our project by performing the following steps: First, create a new project and name it MenuInRPG. Click on the Create new Project button, as shown in the following screenshot: Next, import the assets package by going to Assets | Import Package | Custom Package...; choose Chapter2Package.unityPackage, which we just downloaded, and then click on the Import button in the pop-up window link, as shown in the following screenshot: Wait until it's done, and you will see the MenuInRPGGame and SimplePlatform folders in the Window view. Next, click on the arrow in front of the SimplePlatform folder to bring up the drop-down options and you will see the Scenes folder and the SimplePlatform_C# and SimplePlatform_JS scenes, as shown in the following screenshot: Next, double-click on the SimplePlatform_C# (for a C# user) and SimplePlatform_JS (for a Unity JavaScript user) scenes, as shown in the preceding screenshot, to open the scene that we will work on in this project. When you double-click on either of the SimplePlatform scenes, Unity will display a pop-up window asking whether you want to save the current scene or not. As we want to use the SimplePlatform scene, just click on the Don't Save button to open up the SimplePlatform scene, as shown in the following screenshot: Then, go to the MenuInRPGGame/Resources/UI folder and click on the first file to make sure that the Texture Type and Format fields are selected correctly, as shown in the following screenshot: Why do we set it up in this way? This is because we want to have a UI graphic to look as close to the source image as possible. However, we set the Format field to Truecolor, which will make the size of the image larger than Compress, but will show the right color of the UI graphics. Next, we need to set up the Layers and Tags configurations; for this, go to Edit | Project Settings | Tags and set them as follows: Tags Element 0 UI Element 1 Key Element 2 RestartButton Element 3 Floor Element 4 Wall Element 5 Background Element 6 Door Layers User Layer Background User Layer Level Use Layer UI At last, we will save this scene in the MenuInRPGGame/Scenes folder, and name it MenuInRPG by going to File | Save Scene as... and then save it. Engage thrusters Now we are ready to create a GUI skin; for this, perform the following steps: Let's create a new GUISkin object by going to Assets | Create | GUISkin, and we will see New GUISkin in our Project window. Name the GUISkin object as MenuSkin. Then, click on MenuSkin and go to its Inspector window. We will see something similar to the following screenshot: You will see many properties here, but don't be afraid, because this is the main key to creating custom graphics for our UI. Font is the base font for the GUI skin. From Box to Scroll View, each property is GUIStyle, which is used for creating our custom UI. The Custom Styles property is the array of GUIStyle that we can set up to apply extra styles. Settings are the setups for the entire GUI. Next, we will set up the new font style for our menu UI; go to the Font line in the Inspector view, click the circle icon, and select the Federation Kalin font. Now, you have set up the base font for GUISkin. Next, click on the arrow in front of the Box line to bring up a drop-down list. We will see all the properties, as shown in the following screenshot: For more information and to learn more about these properties, visit http://unity3d.com/support/documentation/Components/class-GUISkin.html. Name is basically the name of this style, which by default is box (the default style of GUI.Box). Next, we will be seeing our custom UI to this GUISkin; click on the arrow in front of Normal to bring up the drop-down list, and you will see two parameters—Background and Text Color. Click on the circle icon on the right-hand side of the Background line to bring up the Select Texture2D window and choose the boxNormal texture, or you can drag the boxNormal texture from the MenuInRPG/Resources/UI folder and drop it to the Background space. We can also use the search bar in the Select Texture2D window or the Project view to find our texture by typing boxNormal in the search bar, as shown in the following screenshot: Then, under the Text Color line, we leave the color as the default color—because we don't need any text to be shown in this style—and repeat the previous step with On Normal by using the boxNormal texture. Next, click on the arrow in front of Active under Background. Choose the boxActive texture, and repeat this step for On Active. Then, go to each property in the Box style and set the following parameters: Border: Left: 14, Right: 14, Top: 14, Bottom: 14 Padding: Left: 6, Right: 6, Top: 6, Bottom: 6 For other properties of this style, we will leave them as default. Next, we go to the following properties in the MenuSkin inspector and set them as follows: Label Normal | Text Color R 27, G: 95, B: 104, A: 255 Window Normal | Background myWindow On Normal | Background myWindow Border Left: 27, Right: 27, Top: 55, Bottom: 96 Padding Left: 30, Right: 30, Top: 60, Bottom: 30 Horizontal Scrollbar Normal | Background horScrollBar Border Left: 4, Right: 4, Top: 4, Bottom: 4 Horizontal Scrollbar Thumb Normal | Background horScrollBarThumbNormal Hover | Background horScrollBarThumbHover Border Left: 4, Right: 4, Top: 4, Bottom: 4 Horizontal Scrollbar Left Button Normal | Background arrowLNormal Hover | Background arrowLHover Fixed Width 14 Fixed Height 15 Horizontal Scrollbar Right Button Normal | Background arrowRNormal Hover | Background arrowRHover Fixed Width 14 Fixed Height 15 Vertical Scrollbar Normal | Background verScrollBar Border Left: 4, Right: 4, Top: 4, Bottom: 4 Padding Left: 0, Right: 0, Top: 0, Bottom: 0 Vertical Scrollbar Thumb Normal | Background verScrollBarThumbNormal Hover | Background verScrollBarThumbHover Border Left: 4, Right: 4, Top: 4, Bottom: 4 Vertical Scrollbar Up Button Normal | Background arrowUNormal Hover | Background arrowUHover Fixed Width 16 Fixed Height 14 Vertical Scrollbar Down Button Normal | Background arrowDNormal Hover | Background arrowDHover Fixed Width 16 Fixed Height 14 We have finished setting up of the default styles. Now we will go to the Custom Styles property and create our custom GUIStyle to use for this menu; go to Custom Styles and under Size, change the value to 6. Then, we will see Element 0 to Element 5. Next, we go to the first element or Element 0; under Name, type Tab Button, and we will see Element 0 change to Tab Button. Set it as follows: Tab Button (or Element 0) Name Tab Button Normal Background tabButtonNormal Text Color R: 27, G: 62, B: 67, A: 255 Hover Background tabButtonHover Text Color R: 211, G: 166, B: 9, A: 255 Active Background tabButtonActive Text Color R: 27, G: 62, B: 67, A: 255 On Normal Background tabButtonActive Text Color R: 27, G: 62, B: 67, A: 255 Border Left: 12, Right: 12, Top: 12, Bottom: 4 Padding Left: 6, Right: 6, Top: 6, Bottom: 4 Font Size 14 Alignment Middle Center Fixed Height 31 The settings are shown in the following screenshot: For the Text Color value, we can also use the eyedropper tool next to the color box to copy the same color, as we can see in the following screenshot: We have finished our first style, but we still have five styles left, so let's carry on with Element 1 with the following settings: Exit Button (or Element 1) Name Exit Button Normal | Background buttonCloseNormal Hover | Background buttonCloseHover Fixed Width 26 Fixed Height 22 The settings for Exit Button are showed in the following screenshot: The following styles will create a style for Element 2: Text Item (or Element 2) Name Text Item Normal | Text Color R: 27, G: 95, B: 104, A: 255 Alignment Middle Left Word Wrap Check The settings for Text Item are shown in the following screenshot: To set up the style for Element 3, the following settings should be done: Text Amount (or Element 3) Name Text Amount Normal | Text Color R: 27, G: 95, B: 104, A: 255 Alignment Middle Right Word Wrap Check The settings for Text Amount are shown in the following screenshot: The following settings should be done to create Selected Item: Selected Item (or Element 4) Name Selected Item Normal | Text Color R: 27, G: 95, B: 104, A: 255 Hover Background itemSelectHover Text Color R: 27, G: 95, B: 104, A: 255 Active Background itemSelectHover Text Color R: 27, G: 95, B: 104, A: 255 On Normal Background itemSelectActive Text Color R: 27, G: 95, B: 104, A: 255 Border Left: 6, Right: 6, Top: 6, Bottom: 6 Margin Left: 2, Right: 2, Top: 2, Bottom: 2 Padding Left: 4, Right: 4, Top: 4, Bottom: 4 Alignment Middle Center Word Wrap Check The settings are shown in the following screenshot: To create the Disabled Click style, the following settings should be done: Disabled Click (or Element 5) Name Disabled Click Normal Background itemSelectNormal Text Color R: 27, G: 95, B: 104, A: 255 Border Left: 6, Right: 6, Top: 6, Bottom: 6 Margin Left: 2, Right: 2, Top: 2, Bottom: 2 Padding Left: 4, Right: 4, Top: 4, Bottom: 4 Alignment Middle Center Word Wrap Check The settings for Disabled Click are shown in the following screenshot: And now, we have finished this step.

(For more resources related to this topic, see here.) So I don't think it's possible to go to a conference these days and not see a talk on mobile or wireless. (They tend to schedule the streams to have both mobile and wireless talks at the same time—the sneaky devils. There is no escaping the wireless knowledge!) So, it makes sense that we work out some ways of training people how to skill up on these technologies. We're going to touch on some older vulnerabilities that you don't see very often, but as always, when you do, it's good to know how to insta-win. Wireless environment setup This article is a bit of an odd one, because with Wi-Fi and mobile, it's much harder to create a safe environment for your testers to work in. For infrastructure and web app tests, you can simply say, "it's on the network, yo" and they'll get the picture. However, Wi-Fi and mobile devices are almost everywhere in places that require pen testing. It's far too easy for someone to get confused and attempt to pwn a random bystander. While this sounds hilarious, it is a serious issue if that occurs. So, adhere to the following guidelines for safer testing: Where possible, try and test away from other people and networks. If there is an underground location nearby, testing becomes simpler as floors are more effective than walls for blocking Wi-Fi signals (contrary to the firmly held beliefs of anyone who's tried to improve their home network signal). If you're an individual who works for a company, or you know, has the money to make a Faraday cage, then by all means do the setup in there. I'll just sit here and be jealous. Unless it's pertinent to the test scenario, provide testers with enough knowledge to identify the devices and networks they should be attacking. A good way to go is to provide the Mac address as they very rarely collide. (Mac randomizing tools be damned.) If an evil network has to be created, name it something obvious and reduce the access to ensure that it is visible to as few people as possible. The naming convention we use is Connectingtomewillresultin followed by pain, death, and suffering. While this steers away the majority of people, it does appear to attract the occasional fool, but that's natural selection for you. Once again, but it is worth repeating, don't use your home network. Especially in this case, using your home equipment could expose you to random passersby or evil neighbors. I'm pretty sure my neighbor doesn't know how to hack, but if he does, I'm in for a world of hurt. Software We'll be using Kali Linux as the base for this article as we'll be using the tools provided by Kali to set up our networks for attack. Everything you need is built into Kali, but if you happen to be using another build such as Ubuntu or Debian, you will need the following tools: Iwtools (apt-get install iw): This is the wireless equivalent of ifconfig that allows the alteration of wireless adapters, and provides a handy method to monitor them. Aircrack suite (apt-get install aircrack-ng): The basic tools of wireless attacking are available in the Aircrack suite. This selection of tools provides a wide range of services, including cracking encryption keys, monitoring probe requests, and hosting rogue networks. Hostapd (apt-get install hostapd): Airbase-ng doesn't support WPA-2 networks, so we need to bring in the serious programs for serious people. This can also be used to host WEP networks, but getting Aircrack suite practice is not to be sniffed at. Wireshark (apt-get install wireshark): Wireshark is one of the most widely used network analytics tools. It's not only used by pen testers, but also by people who have CISSP and other important letters after their names. This means that it's a tool that you should know about. dnschef (https://thesprawl.org/projects/dnschef/): Thanks to Duncan Winfrey, who pointed me in this direction. DNSchef is a fantastic resource for doing DNS spoofing. Other alternatives include DNS spoof and Metasploit's Fake DNS. Crunch (apt-get install crunch): Crunch generates strings in a specified order. While it seems very simple, it's incredibly useful. Use with care though; it has filled more than one unwitting user's hard drive. Hardware You want to host a dodgy network. The first question to ask yourself, after the question you already asked yourself about software, is: is your laptop/PC capable of hosting a network? If your adapter is compatible with injection drivers, you should be fine. A quick check is to boot up Kali Linux and run sudo airmon-ng start <interface>. This will put your adapter in promiscuous mode. If you don't have the correct drivers, it'll throw an error. Refer to a potted list of compatible adapters at http://www.aircrack-ng.org/doku.php?id=compatibility_drivers. However, if you don't have access to an adapter with the required drivers, fear not. It is still possible to set up some of the scenarios. There are two options. The first and most obvious is "buy an adapter." I can understand that you might not have a lot of cash kicking around, so my advice is to pick up an Edimax ew-7711-UAN—it's really cheap and pretty compact. It has a short range and is fairly low powered. It is also compatible with Raspberry Pi and BeagleBone, which is awesome but irrelevant. The second option is a limited solution. Most phones on the market can be used as wireless hotspots and so can be used to set up profiles for other devices for the phone-related scenarios in this article. Unfortunately, unless you have a rare and epic phone, it's unlikely to support WEP, so that's out of the question. There are solutions for rooted phones, but I wouldn't instruct you to root your phone, and I'm most certainly not providing a guide to do so. Realistically, in order to create spoofed networks effectively and set up these scenarios, a computer is required. Maybe I'm just not being imaginative enough.

(For more resources related to this topic, see here.) Configuration targets In this section, we look at the different layers that can be configured. The layers are: SYSTEM: This layer is system-wide and found in /etc/gitconfig GLOBAL: This layer is global for the user and found in ~/.gitconfig LOCAL: This layer is local to the current repository and found in .git/config Getting ready We will use the jgit repository for this example, as shown in the following command: $ git clone https://git.eclipse.org/r/jgit/jgit $ cd jgit How to do it... In the previous example, we saw how we could use the command git config --list to list configuration entries. This list is actually made from three different levels of configuration that Git offers: system-wide configuration, SYSTEM; global configuration for the user, GLOBAL; and local repository configuration, LOCAL. For each of these configuration layers, we can query the existing configuration. On a Windows box with a default installation of the Git extensions, the different configuration layers will look approximately like the following: $ git config --list --system core.symlinks=false core.autocrlf=true color.diff=auto color.status=auto color.branch=auto color.interactive=true pack.packsizelimit=2g help.format=html http.sslcainfo=/bin/curl-ca-bundle.crt sendemail.smtpserver=/bin/msmtp.exe diff.astextplain.textconv=astextplain rebase.autosquash=true $ git config --list --global merge.tool=kdiff3 mergetool.kdiff3.path=C:/Program Files (x86)/KDiff3/kdiff3.exe diff.guitool=kdiff3 difftool.kdiff3.path=C:/Program Files (x86)/KDiff3/kdiff3.exe core.editor="C:/Program Files (x86)/GitExtensions/GitExtensions.exe" fileeditor core.autocrlf=true credential.helper=!"C:/Program Files (x86)/GitExtensions/GitCredentialWinStore/git-credential-winst ore.exe" user.name=Aske Olsson user.email=aske.olsson@switch-gears.dk $ git config --list --local core.repositoryformatversion=0 core.filemode=false core.bare=false core.logallrefupdates=true core.symlinks=false core.ignorecase=true core.hidedotfiles=dotGitOnly remote.origin.url=https://git.eclipse.org/r/jgit/jgit remote.origin.fetch=+refs/heads/*:refs/remotes/origin/* branch.master.remote=origin branch.master.merge=refs/heads/master We can also query a single key and limit the scope to one of the three layers, by using the following command: $ git config --global user.email aske.olsson@switch-gears.dk We can set the e-mail address of the user to a different one for the current repository: $ git config --local user.email aske@switch-gears.dk Now, listing the GLOBAL layer user.email will return aske.olsson@switch-gears.dk, listing LOCAL gives aske@switch-gears.dk, and listing user.email without specifying the layer gives the effective value that is used in the operations on this repository, in this case, the LOCAL value aske@switch-gears.dk. The effective value is the value, which takes precedence when needed. When two or more values are specified for the same key, but on different layers, the lowest layer takes precedence. When a configuration value is needed, Git will first look in the LOCAL configuration. If not found here, the GLOBAL configuration is queried. If it is not found in the GLOBAL configuration, the SYSTEM configuration is used. If none of this works, the default value in Git is used. In the previous example, user.email is specified in both the GLOBAL and LOCAL layers. Hence, the LOCAL layer will be used. How it works... Querying the three layers of configuration simply returns the content of the configuration files: /etc/gitconfig for system-wide configuration, ~/.gitconfig for user-specific configuration, and .git/config for repository-specific configuration. When not specifying the configuration layer, the returned value will be the effective value. There's more... Instead of setting all the configuration values on the command line by the key value, it is possible to set them by just editing the configuration file directly. Open the configuration file in your favorite editor and set the configuration you need, or use the built-in git config -e repository to edit the configuration directly in the Git-configured editor. You can set the editor to the editor of your choice either by changing the $EDITOR environment variable or with the core.editor configuration target, for example: $ git config --global core.editor vim Querying the existing configuration In this example, we will look at how we can query the existing configuration and set the configuration values. Getting ready We'll use jgit again by using the following command: $ cd jgit How to do it... To view all the effective configurations for the current Git repository, run the following command: $ git config --list user.name=Aske Olsson user.email=askeolsson@switch-gears.dk core.repositoryformatversion=0 core.filemode=false core.bare=false core.logallrefupdates=true remote.origin.url=https://git.eclipse.org/r/jgit/jgit remote.origin.fetch=+refs/heads/*:refs/remotes/origin/* branch.master.remote=origin branch.master.merge=refs/heads/master The previous output will of course reflect the user running the command. Instead of Aske Olsson as the name and the e-mail, the output should reflect your settings. If we are just interested in a single configuration item, we can just query it by its section.key or section.subsection.key: $ git config user.name Aske Olsson $ git config remote.origin.url https://git.eclipse.org/r/jgit/jgit How it works... Git's configuration is stored in plaintext files, and works like a key-value storage. You can set/query by key and get the value back. An example of the text-based configuration file is shown as follows (from the jgit repository): $ cat .git/config [core] repositoryformatversion = 0 filemode = false bare = false logallrefupdates = true [remote "origin"] url = https://git.eclipse.org/r/jgit/jgit fetch = +refs/heads/*:refs/remotes/origin/* [branch "master"] remote = origin merge = refs/heads/master There's more... It is also easy to set configuration values. Just use the same syntax as when querying the configuration except add an argument to the value. To set a new e-mail address on the LOCAL layer, we can execute the following command line: git config user.email askeolsson@example.com The LOCAL layer is the default if nothing else is specified. If you require whitespaces in the value, you can enclose the string in quotation marks, as you would do when configuring your name: git config user.name "Aske Olsson" You can even set your own configuration, which does not have any effect on the core Git, but can be useful for scripting/builds and so on: $ git config my.own.config "Whatever I need" List the value $ git config my.own.config Whatever I need It is also very easy to delete/unset configuration entries: $ git config --unset my.own.config List the value $ git config my.own.config Templates In this example, we will see how to create a template commit message that will be displayed in the editor when creating a commit. The template is only for the local user and not distributed with the repository in general. Getting ready In this example, we will use the example repository: $ git clone https://github.com/dvaske/data-model.git $ cd data-model We'll use the following code as a commit message template for commit messages: Short description of commit Longer explanation of the motivation for the change Fixes-Bug: Enter bug-id or delete line Implements-Requirement: Enter requirement-id or delete line Save the commit message template in $HOME/.gitcommitmsg.txt. The filename isn't fixed and you can choose a filename of your liking. How to do it... To let Git know about our new commit message template, we can set the configuration variable commit.template to point at the file we just created with that template; we'll do it globally so it is applicable to all our repositories: $ git config --global commit.template $HOME/.gitcommitmsg.txt Now, we can try to change a file, add it, and create a commit. This will bring up our preferred editor with the commit message template preloaded: $ git commit Short description of commit Longer explanation of the motivation for the change Fixes-Bug: Enter bug-id or delete line Implements-Requirement: Enter requirement-id or delete line # Please enter the commit message for your changes. Lines starting # with '#' will be ignored, and an empty message aborts the commit. # On branch master # Changes to be committed: # (use "git reset HEAD <file>..." to unstage) # # modified: another-file.txt # ~ ~ ".git/COMMIT_EDITMSG" 13 lines, 396 characters We can now edit the message according to our commit and save to complete the commit. How it works... When commit.template is set, Git simply uses the content of the template file as a starting point for all commit messages. This is quite convenient if you have a commit-message policy as it greatly increases the chances of the policy being followed. You can even have different templates tied to different repositories, since you can just set the configuration at the local level. A .git directory template Sometimes, having a global configuration isn't enough. You will also need to trigger the execution of scripts (aka Git hooks), exclude files, and so on. It is possible to achieve this with the template option set to git init. It can be given as a command-line option to git clone and git init, or as the $GIT_TEMPLATE_DIR environment variable, or as the configuration option init.templatedir. It defaults to /usr/share/git-core/templates. The template option works by copying files in the template directory to the .git ($GIT_DIR) folder after it has been created. The default directory contains sample hooks and some suggested exclude patterns. In the following example, we'll see how we can set up a new template directory, and add a commit message hook and exclude file. Getting ready First, we will create the template directory. We can use any name we want, and we'll use ~/.git_template, as shown in the following command: $ mkdir ~/.git_template Now, we need to populate the directory with some template files. This could be a hook or an exclude file. We will create one hook file and an exclude file. The hook file is located in .git/hooks/name-of-hook and the exclude file in .git/info/exclude. Create the two directories needed hooks and info, as shown in the following command: $ mkdir ~/.git_template/{hooks,info} To keep the sample hooks provided by the default template directory (the Git installation), we copy the files in the default template directory to the new one. When we use our newly created template directory, we'll override the default one. So, copying the default files to our template directory will make sure that except for our specific changes the template directory is similar to the default one, as shown in the following command: $ cd ~/.git_template/hooks $ cp /usr/share/git-core/templates/hooks/* . We'll use the commit-msg hook as the example hook: #!/bin/sh MSG_FILE="$1" echo "nHi from the template commit-msg hook" >> $MSG_FILE The hook is very simple and will just add Hi from the template commit-msg hook to the end of the commit message. Save it as commit-msg in the ~/.git_template/hooks directory and make it executable by using the following command: chmod +x ~/.git_template/hooks/commit-msg Now that the commit message hook is done, let's also add an exclude file to the example. The exclude file works like the .gitignore file, but is not tracked in the repository. We'll create an exclude file that excludes all the *.txt files, as follows: $ echo *.txt > ~/.git_template/info/exclude Now, our template directory is ready for use. How to do it... Our template directory is ready and we can use it, as described earlier, as a command-line option, an environment variable or, as in this example, to be set as a configuration: $ git config --global init.templatedir ~/.git_template Now, all Git repositories we create using init or clone will have the default files of the template directory. We can test if it works by creating a new repository as follows: $ git init template-example $ cd template-example Let's try to create a .txt file and see what git status tells us. It should be ignored by the exclude file from the template directory: $ echo "this is the readme file" > README.txt $ git status The exclude file worked! You can put in the file endings yourself or just leave it blank and keep to the .gitignore files. To test if the commit-msg hook also works, let us try to create a commit. First, we need a file to commit. So, let's create that and commit it as follows: $ echo "something to commit" > somefile $ git add somefile $ git commit –m "Committed something" We can now check the history with git log: $ git log -1 commit 1f7d63d7e08e96dda3da63eadc17f35132d24064 Author: Aske Olsson <aske.olsson@switch-gears.dk> Date: Mon Jan 6 20:14:21 2014 +0100 Committed something Hi from the template commit-msg hook How it works... When Git creates a new repository, either via init or clone, it will copy the files from the template directory to the new repository when creating the directory structure. The template directory can be defined either by a command-line argument, environment variable, or configuration option. If nothing is specified, the default template directory will be used (distributed with the Git installation). By setting the configuration as a --global option, the template directory defined will apply to all of the user's (new) repositories. This is a very nice way to distribute the same hooks across repositories, but it also has some drawbacks. As the files in the template directory are only copied to the Git repositories, updates to the template directory do not affect the existing repositories. This can be solved by running git init in each existing repository to reinitialize the repository, but this can be quite cumbersome. Also, the template directory can enforce hooks on some repositories where you don't want them. This is quite easily solved by simply deleting the hook files in .git/hooks of that repository.

(For more resources related to this topic, see here.) Design patterns are ways to solve a problem and the way to get your intended result in the best possible manner. So, design patterns are not only ways to create a large and robust system, but they also provide great architectures in a friendly manner. In software engineering, a design pattern is a general repeatable and optimized solution to a commonly occurring problem within a given context in software design. It is a description or template for how to solve a problem, and the solution can be used in different instances. The following are some of the benefits of using design patterns: Maintenance Documentation Readability Ease in finding appropriate objects Ease in determining object granularity Ease in specifying object interfaces Ease in implementing even for large software projects Implements the code reusability concept If you are not familiar with design patterns, the best way to begin understanding is observing the solutions we use for commonly occurring, everyday life problems. Let's take a look at the following image: Many different types of power plugs exist in the world. So, we need a solution that is reusable, optimized, and cheaper than buying a new device for different power plug types. In simple words, we need an adapter. Have a look at the following image of an adapter: In this case, an adapter is the best solution that's reusable, optimized, and cheap. But an adapter does not provide us with a solution when our car's wheel blows out. In object-oriented languages, we the programmers use the objects to do whatever we want to have the outcome we desire. Hence, we have many types of objects, situations, and problems. That means we need more than just one approach to solving different kinds of problems. Elements of design patterns The following are the elements of design patterns: Name: This is a handle we can use to describe the problem Problem: This describes when to apply the pattern Solution: This describes the elements, relationships, responsibilities, and collaborations, in a way that we follow to solve a problem Consequences: This details the results and trade-offs of applying the pattern Classification of design patterns Design patterns are generally divided into three fundamental groups: Creational patterns Structural patterns Behavioral patterns Let's examine these in the following subsections. Creational patterns Creational patterns are a subset of design patterns in the field of software development; they serve to create objects. They decouple the design of an object from its representation. Object creation is encapsulated and outsourced (for example, in a factory) to keep the context of object creation independent from concrete implementation. This is in accordance with the rule: "Program on the interface, not the implementation." Some of the features of creational patterns are as follows: Generic instantiation: This allows objects to be created in a system without having to identify a specific class type in code (Abstract Factory and Factory pattern) Simplicity: Some of the patterns make object creation easier, so callers will not have to write large, complex code to instantiate an object (Builder (Manager) and Prototype pattern) Creation constraints: Creational patterns can put bounds on who can create objects, how they are created, and when they are created The following patterns are called creational patterns: The Abstract Factory pattern The Factory pattern The Builder (Manager) pattern The Prototype pattern The Singleton pattern Structural patterns In software engineering, design patterns structure patterns facilitate easy ways for communications between various entities. Some of the examples of structures of the samples are as follows: Composition: This composes objects into a tree structure (whole hierarchies). Composition allows customers to be uniformly treated as individual objects according to their composition. Decorator: This dynamically adds options to an object. A Decorator is a flexible alternative embodiment to extend functionality. Flies: This is a share of small objects (objects without conditions) that prevent overproduction. Adapter: This converts the interface of a class into another interface that the clients expect. Adapter lets those classes work together that would normally not be able to because of the different interfaces. Facade: This provides a unified interface meeting the various interfaces of a subsystem. Facade defines a higher-level interface to the subsystem, which is easier to use. Proxy: This implements the replacement (surrogate) of another object that controls access to the original object. Bridge: This separates an abstraction from its implementation, which can then be independently altered. Behavioral patterns Behavioral patterns are all about a class' objects' communication. Behavioral patterns are those patterns that are most specifically concerned with communication between objects. The following is a list of the behavioral patterns: Chain of Responsibility pattern Command pattern Interpreter pattern Iterator pattern Mediator pattern Memento pattern Observer pattern State pattern Strategy pattern Template pattern Visitor pattern If you want to check out the usage of some patterns in the Laravel core, have a look at the following list: The Builder (Manager) pattern: IlluminateAuthAuthManager and IlluminateSessionSessionManager The Factory pattern: IlluminateDatabaseDatabaseManager and IlluminateValidationFactory The Repository pattern: IlluminateConfigRepository and IlluminateCacheRepository The Strategy pattern: IIlluminateCacheStoreInterface and IlluminateConfigLoaderInterface The Provider pattern: IIlluminateAuthAuthServiceProvider and IlluminateHashHashServiceProvider Summary In this article, we have explained the fundamentals of design patterns. We've also introduced some design patterns that are used in the Laravel Framework. Resources for Article: Further resources on this subject: Laravel 4 - Creating a Simple CRUD Application in Hours [article] Your First Application [article] Creating and Using Composer Packages [article]

The hardware configuration of this project is not really complex. For each motion sensor module you want to build, you'll need to do the following steps. The first one is to plug an XBee module on the XBee shield. Then, you need to plug the shield into your Arduino board, as shown in the following image: Now, you can connect the motion sensor. It has three pins: VCC (for the positive power supply), GND (which corresponds to the reference voltage level), and SIG (which will turn to a digital HIGH state in case any motion is detected). Connect VCC to the Arduino 5V pin, GND to Arduino GND, and SIG to Arduino pin number 8 (the example code uses pin 8, but you could also use any digital pin). You should end up with something similar to this image: You will also need to set a jumper correctly on the board so we can upload a sketch. On the XBee shield, you have a little switch close to the XBee module to choose between the XBee module being connected directly to the Arduino board serial interface (which means you can't upload any sketches anymore) or leaving it disconnected. As we need to upload the Arduino sketch first, you need to put this switch to DLINE, as shown in this image: You will also need to connect the XBee explorer board to your computer at this point. Simply insert one XBee module to the board as shown in the following image: Now that this is done, you can power up everything by connecting the Arduino board and explorer module to your computer via USB cables. If you want to use several XBee motion sensors, you will need to repeat the beginning of the procedure for each of them: assemble one Arduino board with an XBee shield, one XBee module, and one motion sensor. However, you only need one USB XBee module connected to your computer if you have many sensors. Summary In this article, we learned about the hardware configuration required to build wireless XBee motion detectors. We looked at Arduino R3 board, XBee module, and XBee shield and the other important hardware configuration. Resources for Article: Further resources on this subject: Playing with Max 6 Framework [Article] Our First Project – A Basic Thermometer [Article] Sending Data to Google Docs [Article]

(For more resources related to this topic, see here.) Preparing the environment Before jumping into configuring and setting up the cluster network, we have to check some parameters and prepare the environment. To enable sharding for a database or collection, we have to configure some configuration servers that hold the cluster network metadata and shards information. Other parts of the cluster network use these configuration servers to get information about other shards. In production, it's recommended to have exactly three configuration servers in different machines. The reason for establishing each shard on a different server is to improve the safety of data and nodes. If one of the machines crashes, the whole cluster won't be unavailable. For the testing and developing environment, you can host all the configuration servers on a single server.Besides, we have two more parts for our cluster network, shards and mongos, or query routers. Query routers are the interface for all clients. All read/write requests are routed to this module, and the query router or mongos instance, using configuration servers, route the request to the corresponding shard. The following diagram shows the cluster network, modules, and the relation between them: It's important that all modules and parts have network access and are able to connect to each other. If you have any firewall, you should configure it correctly and give proper access to all cluster modules. Each configuration server has an address that routes to the target machine. We have exactly three configuration servers in our example, and the following list shows the hostnames: cfg1.sharding.com cfg2.sharding.com cfg3.sharding.com In our example, because we are going to set up a demo of sharding feature, we deploy all configuration servers on a single machine with different ports. This means all configuration servers addresses point to the same server, but we use different ports to establish the configuration server. For production use, all things will be the same, except you need to host the configuration servers on separate machines. In the next section, we will implement all parts and finally connect all of them together to start the sharding server and run the cluster network. Implementing configuration servers Now it's time to start the first part of our sharding. Establishing a configuration server is as easy as running a mongod instance using the --configsvr parameter. The following scheme shows the structure of the command: mongod --configsvr --dbpath <path> --port <port> If you don't pass the dbpath or port parameters, the configuration server uses /data/configdb as the path to store data and port 27019 to execute the instance. However, you can override the default values using the preceding command. If this is the first time that you have run the configuration server, you might be faced with some issues due to the existence of dbpath. Before running the configuration server, make sure that you have created the path; otherwise, you will see an error as shown in the following screenshot: You can simply create the directory using the mkdir command as shown in the following line of command: mkdir /data/configdb Also, make sure that you are executing the instance with sufficient permission level; otherwise, you will get an error as shown in the following screenshot: The problem is that the mongod instance can't create the lock file because of the lack of permission. To address this issue, you should simply execute the command using a root or administrator permission level. After executing the command using the proper permission level, you should see a result like the following screenshot: As you can see now, we have a configuration server for the hostname cfg1.sharding.com with port 27019 and with dbpath as /data/configdb. Also, there is a web console to watch and control the configuration server running on port 28019. By pointing the web browser to the address http://localhost:28019/, you can see the console. The following screenshot shows a part of this web console: Now, we have the first configuration server up and running. With the same method, you can launch other instances, that is, using /data/configdb2 with port 27020 for the second configuration server, and /data/configdb3 with port 27021 for the third configuration server. Configuring mongos instance After configuring the configuration servers, we should bind them to the core module of clustering. The mongos instance is responsible to bind all modules and parts together to make a complete sharding core. This module is simple and lightweight, and we can host it on the same machine that hosts other modules, such as configuration servers. It doesn't need a separate directory to store data. The mongos process uses port 27017 by default, but you can change the port using the configuration parameters. To define the configuration servers, you can use the configuration file or command-line parameters. Create a new file using your text editor in the /etc/ directory and add the following configuring settings: configdb = cfg1.sharding.com:27019, cfg2.sharding.com:27020 cfg3.sharding.com:27021 To execute and run the mongos instance, you can simply use the following command: mongos -f /etc/mongos.conf After executing the command, you should see an output like the following screenshot: Please note that if you have a configuration server that has been already used in a different sharding network, you can't use the existing data directory. You should create a new and empty data directory for the configuration server. Currently, we have mongos and all configuration servers that work together pretty well. In the next part, we will add shards to the mongos instance to complete the whole network. Managing mongos instance Now it's time to add shards and split whole dataset into smaller pieces. For production use, each shard should be a replica set network, but for the development and testing environment, you can simply add a single mongod instances to the cluster. To control and manage the mongos instance, we can simply use the mongo shell to connect to the mongos and execute commands. To connect to the mongos, you use the following command: mongo --host <mongos hostname> --port <mongos port> For instance, our mongos address is mongos1.sharding.com and the port is 27017. This is depicted in the following screenshot: After connecting to the mongos instance, we have a command environment, and we can use it to add, remove, or modify shards, or even get the status of the entire sharding network. Using the following command, you can get the status of the sharding network: sh.status() The following screenshot illustrates the output of this command: Because we haven't added any shards to sharding, you see an error that says there are no shards in the sharding network. Using the sh.help() command, you can see all commands as shown in the following screenshot: Using the sh.addShard() function, you can add shards to the network. Adding shards to mongos After connecting to the mongos, you can add shards to sharding. Basically, you can add two types of endpoints to the mongos as a shard; replica set or a standalone mongod instance. MongoDB has a sh namespace and a function called addShard(), which is used to add a new shard to an existing sharding network. Here is the example of a command to add a new shard. This is shown in the following screenshot: To add a replica set to mongos you should follow this scheme: setname/server:port For instance, if you have a replica set with the name of rs1, hostname mongod1.replicaset.com, and port number 27017, the command will be as follows: sh.addShard("rs1/mongod1.replicaset.com:27017") Using the same function, we can add standalone mongod instances. So, if we have a mongod instance with the hostname mongod1.sharding.com listening on port 27017, the command will be as follows: sh.addShard("mongod1.sharding.com:27017") You can use a secondary or primary hostname to add the replica set as a shard to the sharding network. MongoDB will detect the primary and use the primary node to interact with sharding. Now, we add the replica set network using the following command: sh.addShard("rs1/mongod1.replicaset.com:27017") If everything goes well, you won't see any output from the console, which means the adding process was successful. This is shown in the following screenshot: To see the status of sharding, you can use the sh.status() command. This is demonstrated in the following screenshot: Next, we will establish another standalone mongod instance and add it to sharding. The port number of mongod is 27016 and the hostname is mongod1.sharding.com. The following screenshot shows the output after starting the new mongod instance: Using the same approach, we will add the preceding node to sharding. This is shown in the following screenshot: It's time to see the sharding status using the sh.status() command: As you can see in the preceding screenshot, now we have two shards. The first one is a replica set with the name rs1, and the second shard is a standalone mongod instance on port 27016. If you create a new database on each shard, MongoDB syncs this new database with the mongos instance. Using the show dbs command, you can see all databases from all shards as shown in the following screenshot: The configuration database is an internal database that MongoDB uses to configure and manage the sharding network. Currently, we have all sharding modules working together. The last and final step is to enable sharding for a database and collection. Summary In this article, we prepared the environment for sharding of a database. We also learned about the implementation of a configuration server. Next, after configuring the configuration servers, we saw how to bind them to the core module of clustering. Resources for Article: Further resources on this subject: MongoDB data modeling [article] Ruby with MongoDB for Web Development [article] Dart Server with Dartling and MongoDB [article]

This article by Nathan Kozyra, author of Mastering Concurrency in Go, tackles one of the Internet's most famous and esteemed challenges and attempt to solve it with core Go packages. (For more resources related to this topic, see here.) We've built a few usable applications; things we can start with and leapfrog into real systems for everyday use. By doing so, we've been able to demonstrate the basic and intermediate-level patterns involved in Go's concurrent syntax and methodology. However, it's about time we take on a real-world problem—one that has vexed developers (and their managers and VPs) for a great deal of the early history of the Web. In addressing and, hopefully, solving this problem, we'll be able to develop a high-performance web server that can handle a very large volume of live, active traffic. For many years, the solution to this problem was solely to throw hardware or intrusive caching systems at the problem; so, alternately, solving it with programming methodology should excite any programmer. We'll be using every technique and language construct we've learned so far, but we'll do so in a more structured and deliberate way than we have up to now. Everything we've explored so far will come into play, including the following points: Creating a visual representation of our concurrent application Utilizing goroutines to handle requests in a way that will scale Building robust channels to manage communication between goroutines and the loop that will manage them Profiling and benchmarking tools (JMeter, ab) to examine the way our event loop actually works Timeouts and concurrency controls—when necessary—to ensure data and request consistency Attacking the C10K problem The genesis of the C10K problem is rooted in serial, blocking programming, which makes it ideal to demonstrate the strength of concurrent programming, especially in Go. When he asked this in 1999, for many server admins and engineers, serving 10,000 concurrent visitors was something that would be solved with hardware. The notion that a single server on common hardware could handle this type of CPU and network bandwidth without falling over seemed foreign to most. The crux of his proposed solutions relied on producing non-blocking code. Of course, in 1999, concurrency patterns and libraries were not widespread. C++ had some polling and queuing options available via some third-party libraries and the earliest predecessor to multithreaded syntaxes, later available through Boost and then C++11. Over the coming years, solutions to the problem began pouring in across various flavors of languages, programming design, and general approaches. Any performance and scalability problem will ultimately be bound to the underlying hardware, so as always, your mileage may vary. Squeezing 10,000 concurrent connections on a 486 processor with 500 MB of RAM will certainly be more challenging than doing so on a barebones Linux server stacked with memory and multiple cores. It's also worth noting that a simple echo server would obviously be able to assume more cores than a functional web server that returns larger amounts of data and accepts greater complexity in requests, sessions, and so on, as we'll be dealing with here. Failing of servers at 10,000 concurrent connections When the Web was born and the Internet commercialized, the level of interactivity was pretty minimal. If you're a graybeard, you may recall the transition from NNTP/IRC and the like and how extraordinarily rudimentary the Web was. To address the basic proposition of [page request] → [HTTP response], the requirements on a web server in the early 1990s were pretty lenient. Ignoring all of the error responses, header reading, and settings, and other essential (but unrelated to the in → out mechanism) functions, the essence of the early servers was shockingly simple, at least compared to the modern web servers. The first web server was developed by the father of the Web, Tim Berners-Lee. Developed at CERN (such as WWW/HTTP itself), CERN httpd handled many of the things you would expect in a web server today—hunting through the code, you'll find a lot of notation that will remind you that the very core of the HTTP protocol is largely unchanged. Unlike most technologies, HTTP has had an extraordinarily long shelf life. Written in C in 1990, it was unable to utilize a lot of concurrency strategies available in languages such as Erlang. Frankly, doing so was probably unnecessary—the majority of web traffic was a matter of basic file retrieval and protocol. The meat and potatoes of a web server were not dealing with traffic, but rather dealing with the rules surrounding the protocol itself. You can still access the original CERN httpd site and download the source code for yourself from http://www.w3.org/Daemon/. I highly recommend that you do so as both a history lesson and a way to look at the way the earliest web server addressed some of the earliest problems. However, the Web in 1990 and the Web when the C10K question was first posed were two very different environments. By 1999, most sites had some level of secondary or tertiary latency provided by third-party software, CGI, databases, and so on, all of which further complicated the matter. The notion of serving 10,000 flat files concurrently is a challenge in itself, but try doing so by running them on top of a Perl script that accesses a MySQL database without any caching layer; this challenge is immediately exacerbated. By the mid 1990s, the Apache web server had taken hold and largely controlled the market (by 2009, it had become the first server software to serve more than 100 million websites). Apache's approach was rooted heavily in the earliest days of the Internet. At its launch, connections were initially handled first in, first out. Soon, each connection was assigned a thread from the thread pool. There are two problems with the Apache server. They are as follows: Blocking connections can lead to a domino effect, wherein one or more slowly resolved connections could avalanche into inaccessibility Apache had hard limits on the number of threads/workers you could utilize, irrespective of hardware constraints It's easy to see the opportunity here, at least in retrospect. A concurrent server that utilizes actors (Erlang), agents (Clojure), or goroutines (Go) seems to fit the bill perfectly. Concurrency does not solve the C10k problem in itself, but it absolutely provides a methodology to facilitate it. The most notable and visible example of an approach to the C10K problem today is Nginx, which was developed using concurrency patterns, widely available in C by 2002 to address—and ultimately solve—the C10k problem. Nginx, today, represents either the #2 or #3 web server in the world, depending on the source. Using concurrency to attack C10K There are two primary approaches to handle a large volume of concurrent requests. The first involves allocating threads per connection. This is what Apache (and a few others) do. On the one hand, allocating a thread to a connection makes a lot of sense—it's isolated, controllable via the application's and kernel's context switching, and can scale with increased hardware. One problem for Linux servers—on which the majority of the Web lives—is that each allocated thread reserves 8 MB of memory for its stack by default. This can (and should) be redefined, but this imposes a largely unattainable amount of memory required for a single server. Even if you set the default stack size to 1 MB, we're dealing with a minimum of 10 GB of memory just to handle the overhead. This is an extreme example that's unlikely to be a real issue for a couple of reasons: first, because you can dictate the maximum amount of resources available to each thread, and second, because you can just as easily load balance across a few servers and instances rather than add 10 GB to 80 GB of RAM. Even in a threaded server environment, we're fundamentally bound to the issue that can lead to performance decreases (to the point of a crash). First, let's look at a server with connections bound to threads (as shown in the following diagram), and visualize how this can lead to logjams and, eventually, crashes: This is obviously what we want to avoid. Any I/O, network, or external process that can impose some slowdown can bring about that avalanche effect we talked about, such that our available threads are taken (or backlogged) and incoming requests begin to stack up. We can spawn more threads in this model, but as mentioned earlier, there are potential risks there too, and even this will fail to mitigate the underlying problem. Taking another approach In an attempt to create our web server that can handle 10,000 concurrent connections, we'll obviously leverage our goroutine/channel mechanism to put an event loop in front of our content delivery to keep new channels recycled or created constantly. For this example, we'll assume we're building a corporate website and infrastructure for a rapidly expanding company. To do this, we'll need to be able to serve both static and dynamic content. The reason we want to introduce dynamic content is not just for the purposes of demonstration—we want to challenge ourselves to show 10,000 true concurrent connections even when a secondary process gets in the way. As always, we'll attempt to map our concurrency strategy directly to goroutines and channels. In a lot of other languages and applications, this is directly analogous to an event loop, and we'll approach it as such. Within our loop, we'll manage the available goroutines, expire or reuse completed ones, and spawn new ones where necessary. In this example visualization, we show how an event loop (and corresponding goroutines) can allow us to scale our connections without employing too many hard resources such as CPU threads or RAM: The most important step for us here is to manage that event loop. We'll want to create an open, infinite loop to manage the creation and expiration of our goroutines and respective channels. As part of this, we will also want to do some internal logging of what's happening, both for benchmarking and debugging our application. Building our C10K web server Our web server will be responsible for taking requests, routing them, and serving either flat files or dynamic files with templates parsed against a few different data sources. As mentioned earlier, if we exclusively serve flat files and remove much of the processing and network latency, we'd have a much easier time with handling 10,000 concurrent connections. Our goal is to approach as much of a real-world scenario as we can—very little of the Web operates on a single server in a static fashion. Most websites and applications utilize databases, CDNs (Content Delivery Networks), dynamic and uncached template parsing, and so on. We need to replicate them whenever possible. For the sake of simplicity, we'll separate our content by type and filter them through URL routing, as follows: /static/[request]: This will serve request.html directly /template/[request]: This will serve request.tpl after its been parsed through Go /dynamic/[request][number]: This will also serve request.tpl and parse it against a database source's record By doing this, we should get a better mixture of possible HTTP request types that could impede the ability to serve large numbers of users simultaneously, especially in a blocking web server environment. You can find Go's exceptional library to generate safe data-driven templating at http://golang.org/pkg/html/template/. By safe, we're largely referring to the ability to accept data and move it directly into templates without worrying about the sort of injection issues that are behind a large amount of malware and cross-site scripting. For the database source, we'll use MySQL here, but feel free to experiment with other databases if you're more comfortable with them. Like the html/template package, we're not going to put a lot of time into outlining MySQL and/or its variants. Benchmarking against a blocking web server It's only fair to add some starting benchmarks against a blocking web server first so that we can measure the effect of concurrent versus nonconcurrent architecture. For our starting benchmarks, we'll eschew any framework, and we'll go with our old stalwart, Apache. For the sake of completeness here, we'll be using an Intel i5 3GHz machine with 8 GB of RAM. While we'll benchmark our final product on Ubuntu, Windows, and OS X here, we'll focus on Ubuntu for our example. Our localhost domain will have three plain HTML files in /static, each trimmed to 80 KB. As we're not using a framework, we don't need to worry about raw dynamic requests, but only about static and dynamic requests in addition to data source requests. For all examples, we'll use a MySQL database (named master) with a table called articles that will contain 10,000 duplicate entries. Our structure is as follows: CREATE TABLE articles ( article_id INT NOT NULL AUTO_INCREMENT, article_title VARCHAR(128) NOT NULL, article_text VARCHAR(128) NOT NULL, PRIMARY KEY (article_id) ) With ID indexes ranging sequentially from 0-10,000, we'll be able to generate random number requests, but for now, we just want to see what kind of basic response we can get out of Apache serving static pages with this machine. For this test, we'll use Apache's ab tool and then gnuplot to sequentially map the request time as the number of concurrent requests and pages; we'll do this for our final product as well, but we'll also go through a few other benchmarking tools for it to get some better details.   Apache's AB comes with the Apache web server itself. You can read more about it at http://httpd.apache.org/docs/2.2/programs/ab.html. You can download it for Linux, Windows, OS X, and more from http://httpd.apache.org/download.cgi. The gnuplot utility is available for the same operating systems at http://www.gnuplot.info/. So, let's see how we did it. Have a look at the following graph: Ouch! Not even close. There are things we can do to tune the connections available (and respective threads/workers) within Apache, but this is not really our goal. Mostly, we want to know what happens with an out-of-the-box Apache server. In these benchmarks, we start to drop or refuse connections at around 800 concurrent connections. More troubling is that as these requests start stacking up, we see some that exceed 20 seconds or more. When this happens in a blocking server, each request behind it is queued; requests behind that are similarly queued and the entire thing starts to fall apart. Even if we cannot hit 10,000 concurrent connections, there's a lot of room for improvement. While a single server of any capacity is no longer the way we expect a web server environment to be designed, being able to squeeze as much performance as possible out of that server, ostensibly with our concurrent, event-driven approach, should be our goal. Handling requests The Gorilla toolkit certainly makes this easier, but we should also know how to intercept the functionality to impose our own custom handler. Here is a simple web router wherein we handle and direct requests using a custom http.Server struct, as shown in the following code: var routes []string type customRouter struct { } func (customRouter) ServeHTTP(rw http.ResponseWriter, r *http.Request) { fmt.Println(r.URL.Path); } func main() { var cr customRouter; server := &http.Server { Addr: ":9000", Handler:cr, ReadTimeout: 10 * time.Second, WriteTimeout: 10 * time.Second, MaxHeaderBytes: 1 << 20, } server.ListenAndServe() } Here, instead of using a built-in URL routing muxer and dispatcher, we're creating a custom server and custom handler type to accept URLs and route requests. This allows us to be a little more robust with our URL handling. In this case, we created a basic, empty struct called customRouter and passed it to our custom server creation call. We can add more elements to our customRouter type, but we really don't need to for this simple example. All we need to do is to be able to access the URLs and pass them along to a handler function. We'll have three: one for static content, one for dynamic content, and one for dynamic content from a database. Before we go so far though, we should probably see what our absolute barebones, HTTP server written in Go, does when presented with the same traffic that we sent Apache's way. By old school, we mean that the server will simply accept a request and pass along a static, flat file. You could do this using a custom router as we did earlier, taking requests, opening files, and then serving them, but Go provides a much simpler mode to handle this basic task in the http.FileServer method. So, to get some benchmarks for the most basic of Go servers against Apache, we'll utilize a simple FileServer and test it against a test.html page (which contains the same 80 KB file that we had with Apache). As our goal with this test is to improve our performance in serving flat and dynamic pages, the actual specs for the test suite are somewhat immaterial. We'd expect that while the metrics will not match from environment to environment, we should see a similar trajectory. That said, it's only fair we supply the environment used for these tests; in this case, we used a MacBook Air with a 1.4 GHz i5 processor and 4 GB of memory. First, we'll do this with our absolute best performance out of the box with Apache, which had 850 concurrent connections and 900 total requests. The results are certainly encouraging as compared to Apache. Neither of our test systems were tweaked much (Apache as installed and basic FileServer in Go), but Go's FileServer handles 1,000 concurrent connections without so much as a blip, with the slowest clocking in at 411 ms. Apache has made a great number of strides pertaining to concurrency and performance options in the last five years, but to get there does require a bit of tuning and testing. The intent of this experiment is not intended to denigrate Apache, which is well tested and established. Instead, it's to compare the out-of-the-box performance of the world's number 1 web server against what we can do with Go. To really get a baseline of what we can achieve in Go, let's see if Go's FileServer can hit 10,000 connections on a single, modest machine out of the box: ab -n 10500 -c 10000 -g test.csv http://localhost:8080/a.html We will get the following output: Success! Go's FileServer by itself will easily handle 10,000 concurrent connections, serving flat, static content. Of course, this is not the goal of this particular project—we'll be implementing real-world obstacles such as template parsing and database access, but this alone should show you the kind of starting point that Go provides for anyone who needs a responsive server that can handle a large quantity of basic web traffic. Routing requests So, let's take a step back and look again at routing our traffic through a traditional web server to include not only our static content, but also the dynamic content. We'll want to create three functions that will route traffic from our customRouter:serveStatic():: read function and serve a flat file serveRendered():, parse a template to display serveDynamic():, connect to MySQL, apply data to a struct, and parse a template. To take our requests and reroute, we'll change the ServeHTTP method for our customRouter struct to handle three regular expressions. For the sake of brevity and clarity, we'll only be returning data on our three possible requests. Anything else will be ignored. In a real-world scenario, we can take this approach to aggressively and proactively reject connections for requests we think are invalid. This would include spiders and nefarious bots and processes, which offer no real value as nonusers.

(For more resources related to this topic, see here.) In this article, we will cover the following topics: What is OSGi and what are its key features? The role of the OSGi framework The OSGi base artifact—the OSGi bundle and the concept of dependencies between bundles The Apache Karaf OSGi container and the provisioning of applications in the container How to manage the provisioning on multiple Karaf instances? What is OSGi? Developers are always looking for very dynamic, flexible, and agile software components. The purposes to do so are as follows: Reuse: This feature states that instead of duplicating the code, a component should be shared by other components, and multiple versions of the same component should be able to cohabit. Visibility: This feature specifies that a component should not use the implementation from another component directly. The implementation should be hidden, and the client module should use the interface provided by another component. Agility: This feature specifies that the deployment of a new version of a component should not require you to restart the platform. Moreover, a configuration change should not require a restart. For instance, it's not acceptable to restart a production platform just to change a log level. A minor change such as a log level should be dynamic, and the platform should be agile enough to reload the components that should be reloaded. Discovery: This feature states that a component should be able to discover other components. It's a kind of Plug and Play system: as soon as a component needs another component, it just looks for it and uses it. OSGi has been created to address the preceding points. The core concept is to force developers to use a very modular architecture in order to reduce complexity. As this paradigm is applicable for most modern systems, OSGi is now used for small embedded devices as well as for very large systems. Different applications and systems use OSGi, for example, desktop applications, application servers, frameworks, embedded devices, and so on. The OSGi framework OSGi is designed to run in Java. In order to provide these features and deploy OSGi applications, a core layer has to be deployed in the Java Virtual Machine (JVM): the OSGi framework. This framework manages the life cycle and the relationship between the different OSGi components and artifacts. The OSGi bundle In OSGi, the components are packaged as OSGi bundles. An OSGi bundle is a simple Java JAR (Java ARchive) file that contains additional metadata used by the OSGi framework. These metadata are stored in the manifest file of the JAR file. The following is the metadata: Manifest-Version: 1.0 Bundle-ManifestVersion: 2 Bundle-Version: 2.1.6 Bundle-Name: My Logger Bundle-SymbolicName: my_logger Export-Package: org.my.osgi.logger;version=2.1 Import-Package: org.apache.log4j;version="[1.2,2)" Private-Package: org.my.osgi.logger.internal We can see that OSGi is very descriptive and verbose. We explicitly describe all the OSGi metadata (headers), including the package that we export or import with a specified version or version range. As the OSGi headers are defined in the META-INF/MANIFEST file contained in the JAR file, it means that an OSGi bundle is a regular JAR file that you can use outside of OSGi. The life cycle layer of the OSGi framework is an API to install, start, stop, update, and uninstall OSGi bundles. Dependency between bundles An OSGi bundle can use other bundles from the OSGi framework in two ways. The first way is static code sharing. When we say that this bundle exports packages, it means a bundle can expose some code for other bundles. On the other hand, when we say that this bundle imports packages, it means a bundle can use code from other bundles. For instance, we have the bundle A (packaged as the bundleA.jar file) with the following META-INF/MANIFEST file: Manifest-Version: 1.0 Bundle-ManifestVersion: 2 Bundle-Version: 1.0.0 Bundle-Name: Bundle A Bundle-SymbolicName: bundle_a Export-Package: com.bundle.a;version=1.0 We can see that the bundle A exposes (exports) the com.bundle.a package with Version 1.0. On the other hand, we have the bundle B (packaged as the bundleB.jar file) with the following META-INF/MANIIFEST file: Manifest-Version: 1.0 Bundle-ManifestVersion: 2 Bundle-Version: 2.0.0 Bundle-Name: Bundle B Bundle-SymbolicName: bundle_b Import-Package: com.bundle.a;version="[1.0,2)" We can see that the bundle B imports (so, it will use) the com.bundle.a package in any version between 1.0 and 2 (excluded). So, this means that the OSGi framework will wire the bundles, as the bundle A provides the package used by the bundle B (so, the constraint is resolved). This mechanism is similar to regular Java applications, but instead of embedding the required JAR files in your application, you can just declare the expected code. The OSGi framework is responsible for the link between the different bundles; it's done by the modules layer of the OSGi framework. This approach is interesting when you want to use code which is not natively designed for OSGi. It's a step forward for the reuse of components. However, it provides a limited answer to the purposes seen earlier in the article, especially visibility and discovery. The second way in which an OSGi bundle can use other bundles from the OSGi framework is more interesting. It uses Service-Oriented Architecture (SOA) for low-level components. Here, more than exposing the code, an OSGi bundle exposes a OSGi service. On the other hand, another bundle can use an OSGi service. The services layer of the OSGi framework provides a service registry and all the plumbing mechanisms to wire the services. The OSGi services provide a very dynamic system, offering a Publish-Find-Bind model for the bundles. The OSGi container The OSGi container provides a set of additional features on top of the OSGi framework, as shown in the following diagram: Apache Karaf provides the following features: It provides the abstraction of the OSGi framework. If you write an OSGi application, you have to package your application tightly coupled with the OSGi framework (such as the Apache Felix framework or Eclipse Equinox). Most of the time, you have to prepare the scripts, configuration files, and so on in order to provide a complete, ready-to-use application. Apache Karaf allows you to focus only on your application. Karaf, by default, provides the packaging (including scripts and so on), and it also abstracts the OSGi framework. Thanks to Karaf, it's very easy to switch from Apache Felix (the default framework in Karaf) to Eclipse Equinox. Provides support for the OSGi Blueprint and Spring frameworks. Apache Karaf allows you to directly use Blueprint or Spring as the dependency framework in your bundles. In the new version of Karaf (starting from Karaf 3.0.1), it also supports new dependency frameworks (such as DS, CDI, and so on). Apache Karaf provides a complete, Unix-like shell console where you have a lot of commands available to manage and monitor your running container. This shell console works on any system supporting Java and provides a complete Unix-like environment, including completion, contextual help, key bindings, and more. You can access the shell console using SSH. Apache Karaf also provides a complete management layer (using JMX) that is remotely accessible, which means you can perform the same actions as you do using the shell commands with several MBeans. In addition to the default root Apache Karaf container, for convenience, Apache Karaf allows you to manage multiple container instances. Apache Karaf provides dedicated commands and MBeans to create the instances, control the instances, and so on. Logging is a key layer for any kind of software container. Apache Karaf provides a powerful and very dynamic logging system powered by Pax Logging. In your OSGi application, you are not coupled to a specific logging framework; you can use the framework of your choice (slf4j, log4j, logback, commons-logging, and so on). Apache Karaf uses a central configuration file irrespective of the logging frameworks in use. All changes in this configuration file are made on the fly; no need to restart anything. Again, Apache Karaf provides commands and MBeans dedicated to log management (changing the log level, direct display of the log in the shell console, and so on). Hot deployment is also an interesting feature provided by Apache Karaf. By default, the container monitors a deploy folder periodically. When a new file is dropped in the deploy folder, Apache Karaf checks the file type and delegates the deployment logic for this file to a deployer. Apache Karaf provides different deployers by default (spring, blueprint, features, war, and so on). If Java Authentication and Authorization Service (JAAS) is the Java implementation of Pluggable Authentication Modules (PAM), it's not very OSGi compliant by default. Apache Karaf leverages JAAS, exposing realm and login modules as OSGi services. Again, Apache Karaf provides dedicated JAAS shell commands and MBeans. The security framework is very flexible, allowing you to define the chain of login modules that you want for authentication. By default, Apache Karaf uses a PropertiesLoginModule using the etc/users.properties file for storage. The security framework also provides support for password encryption (you just have to enable encryption in the etc/org.apache.karaf.jaas.cfg configuration file). The new Apache Karaf version (3.0.0) also provides a complete Role Based Access Control (RBAC) system, allowing you to configure the users who can run commands, call MBeans, and so on. Apache Karaf is an enterprise-ready container and provides features dedicated to enterprise. The following enterprise features are not installed by default (to minimize the size and footprint of the container by default), but a simple command allows you to extend the container with enterprise functionalities: WebContainer allows you to deploy a Web Application Bundle (WAB) or WAR file. Apache Karaf is a complete HTTP server with JSP/servlet support, thanks to Pax Web. Java Naming and Directory Interface (JNDI) adds naming context support in Apache Karaf. You can bind an OSGi service to a JNDI name and look up these services using the name, thanks to Aries and Xbean naming. Java Transaction API (JTA) allows you to add a transaction engine (exposed as an OSGi service) in Apache Karaf, thanks to Aries JTA. Java Persistence API (JPA) allows you to add a persistence adapter (exposed as an OSGi service) in Apache Karaf, thanks to Aries JPA. Ready-to-use persistence engines can also be installed very easily (especially Apache OpenJPA and Hibernate). Java Database Connectivity (JDBC) or Java Message Service (JMS) are convenient features, allowing you to easily create JDBC DataSources or JMS ConnectionFactories and use them directly in the shell console. If you can completely administrate Apache Karaf using the shell commands and the JMX MBeans, you can also install Web Console. This Web Console uses the Felix Web Console and allows you to manage Karaf with a simple browser. Thanks to these features, Apache Karaf is a complete, rich, and enterprise-ready container. We can consider Apache Karaf as an OSGi application server. Provisioning in Apache Karaf In addition, Apache Karaf provides three core functionalities that can be used both internally in Apache Karaf or can be used by external applications deployed in the container: OSGi bundle management Configuration management Provisioning using Karaf Features As we learned earlier, the default artifact in OSGi is the bundle. Again, it's a regular JAR file with additional OSGi metadata in the MANIFEST file. The bundles are directly managed by the OSGi framework, but for convenience, Apache Karaf wraps the usage of bundles in specific commands and MBeans. A bundle has a specific life cycle. Especially when you install a bundle, the OSGi framework tries to resolve all the dependencies required by your bundle to promote it in a resolved state. The following is the life cycle of a bundle: The OSGi framework checks whether other bundles provide the packages imported by your bundle. The equivalent action for the OSGi services is performed when you start your bundle. It means that a bundle may require a lot of other bundles to start and so on for the transitive bundles. Moreover, a bundle may require configuration to work. Apache Karaf proposes a very convenient way to manage the configurations. The etc folder is periodically monitored to discover new configuration files and load the corresponding configurations. On the other hand, you have dedicated shell commands and MBeans to manage configurations (and configuration files). If a bundle requires a configuration to work, you first have to create a configuration file in the etc folder (with the expected filename) or use the config:* shell command or ConfigMBean to create the configuration. Considering that an OSGi application is a set of bundles, the installation of an OSGi application can be long and painful by hand. The deployment of an OSGi application is called provisioning as it gathers the following: The installation of a set of bundles, including transitive bundles The installation of a set of configurations required by these bundles OBR OSGi Bundle Repository (OBR) can be the first option to be considered in order to solve this problem. Apache Karaf can connect to the OBR server. The OBR server stores all the metadata for all the bundles, which includes the capabilities, packages, and services provided by a bundle and the requirements, packages, and services needed by a bundle. When you install a bundle via OBR, the OBR server checks the requirement of the installed bundle and finds the bundles that provide the capabilities matching the requirements. The OBR server can automatically install the bundles required for the first one.

(For more resources related to this topic, see here.) Choosing a web host that meets your security requirements In this article, you'll learn what you, as the site administrator, can do to keep your site safe. However, there are also some basic but critical security measures that your web hosting company should take. You'll probably have a shared hosting account, where multiple sites are hosted on one server computer and each site has access to their part of the available disk space and resources. Although this is much cheaper than hiring a dedicated server to host your site, it does involve some security risks. Good web hosting companies take precautions to minimize these risks. When selecting your web host, it's worth checking if they have a good reputation of keeping their security up to standards. The official Joomla resources site (http://resources.joomla.org/directory/support-services/hosting.html) features hosting companies that fully meet the security requirements of a typical Joomla-powered site. Tip 1 – Download from reliable sources To avoid installing corrupted versions, it's a good idea to download the Joomla software itself only from the official website (www.joomla.org) or from reliable local Joomla community sites. This is also true when downloading third-party extensions. Use only extensions that have a good reputation; you can check the reviews at www.extensions.joomla.org. Preferably, download extensions only from the original developer's website or from Joomla community websites that have a good reputation. Tip 2 – Update regularly The Joomla development team regularly releases updates to fix bugs and security issues. Fortunately, Joomla makes keeping your site up to date effortless. In the backend control panel, you'll find a MAINTENANCE section in the quick icons column displaying the current status of both the Joomla software itself and of installed extensions. This is shown in the following screenshot: If updates are found, the quick icon text will prompt you to update by adding an Update now! link, as shown in the following screenshot: Clicking on this link takes you to the Joomla! Update component (which can also be found by navigating to Components | Joomla! Update). In this window, you'll see the details of the update. Just click on Install the update; the process is fully automated. Before you upgrade Joomla to an updated version, it's a good idea to create a backup of your current site. If anything goes wrong, you can quickly have it up and running again. See the Tip 6 – Protect files and directories section in this article for more information on creating backups. If updates for installed extensions are available, you'll also see a notice stating this in the MAINTENANCE section of the control panel. However, you can also check for updates manually; in the backend, navigate to Extensions | Extension Manager and click on Update in the menu on the left-hand side. Click on Find Updates to search for updates. After you've clicked on the Find Updates button, you'll see a notice informing you whether updates are available or not. Select the update you want to install and click on the Update button. Be patient; you may not see much happening for a while. After completion, a message is displayed that the available updates have been installed successfully. The update functionality only works for extensions that support it. It's to be expected that this feature will be widely supported by extension developers; but for other extensions, you'll still have to manually check for updates by visiting the extension developer's website. The Joomla update packages are stored in your website's tmp directory before the update is installed on your site. After installation, you can remove these files from the tmp directory to avoid running into disc space problems. Tip 3 – Choose a safe administrator username When you install Joomla, you also choose and enter a username for your login account (the critical account of the almighty Super User). Although you can enter any administrator username when installing Joomla, many people enter a generic administrator username, for example, admin. However, this username is far too common, and therefore poses a security risk; hackers only have to guess your password to gain access to your site. If you haven't come up with something different during the installation process, you can change the administrator username later on. You can do this using the following steps: In the backend of the site, navigate to Users | User Manager. Select the Super User user record. In the Edit Profile screen, enter a new Login Name. Be creative! Click on Save & Close to apply changes. Log out and log in to the backend with the new username. Tip 4 – Pick a strong password Pick an administrator password that isn't easy to guess. It's best to have a password that's not too short; 8 or more characters is fine. Use a combination of uppercase letters, lowercase letters, numbers, and special characters. This should guarantee a strong password. Don't use the same username and password you use for other online accounts, and regularly change your password. You can create a new password anytime in the backend User Manager section in the same way as you enter or change a username (see Tip 2 – Update regularly). Tip 5 – Use Two-Factor authentication By default, logging in to the administrative interface of your site just requires the correct combination of username and password. A much more secure way to log in is using the Two-Factor authentication system, a recent addition to Joomla. It requires you to log in with not just your username and password, but also with a six-digit security code. To get this code, you need to use the Google Authenticator app, which is available for most Android, iOS, Blackberry, Windows 8, and Windows Mobile devices. The app doesn't store the six-digit code; it changes the code every 30 seconds. Two-Factor authentication is a great solution, but it does require a little extra effort every time you log in to your site. You need to have the app ready every time you log in to generate a new access code. However, you can selectively choose which users will require this system. You can decide, for example, that only the site administrator has to log in using Two-Factor authentication. Enabling the Two-Factor authentication system of Joomla! To enable Joomla's Two-Factor authentication, where a user has to enter an additional secret key when logging in to the site, follow these steps: To use this system on your site, first enable the Two-Factor authentication plugin that comes with Joomla. In the Joomla backend, navigate to Extensions | Plugin Manager, select Two Factor Authentication – Google Authenticator, and enable it. Next, download and install the Google Authenticator app for your device. Once this is set up, you can set the authentication procedure for any user in the Joomla backend. In the user manager section, click on the account name. Then, click on the Two Factor Authentication tab and select Google Authenticator from the Authentication method dropdown. This is shown in the following screenshot: Joomla will display the account name and the key for this account. Enter these account details in the Google Authenticator app on your device. The app will generate a security code. In the Joomla backend, enter this in the Security Code field, as shown in the following screenshot: Save your changes. From now on, the Joomla login screen will ask you for your username, password, and a secret key. This is shown in the following screenshot: There are other secure login systems available besides the Google Authenticator app. Joomla also supports Yubico's YubiKey, a USB stick that generates an additional password every time it is used. After entering their usual password, the user inserts the YubiKey into the USB port of the computer and presses the YubiKey button. Automatically, the extra one-time password will be entered in Joomla's Secret Key field. For more information on YubiKey, visit http://www.yubico.com. Tip 6 – Protect files and directories Obviously, you don't want everybody to be able to access the Joomla files and folders on the web server. You can protect files and folders by setting access permissions using the Change Mode (CHMOD) commands. Basically, the CHMOD settings tell the web server who has access to a file or folder and who is allowed to read it, write to it, or to execute a file (run it as a program). Once your Joomla site is set up and everything works OK, you can use CHMOD to change permissions. You don't use Joomla to change the CHMOD settings; these are set with FTP software. You can make this work using the following steps: In your FTP program, right-click on the name of the file or directory you want to protect. In this example, we'll use the open source FTP program FileZilla. In the right-click menu, select File Permissions. You'll be presented with a pop-up screen. Here, you can check permissions and change them by selecting the appropriate options, as shown in the following screenshot: As you can see, it's possible to set permissions for the file owner (that's you), for group members (that's likely to be only you too), and for the public (everyone else). The public permissions are the tricky part; you should restrict public permissions as much as possible. When changing the permission settings, the file permissions number (the value in the Numeric value: field in the previous screenshot) will change accordingly. Every combination of settings has its particular number. In the previous example, this number is 644. Click on OK to execute the CHMOD command and set file permissions. Setting file permissions What files should you protect and what CHMOD settings should you choose? Here are a few pointers: By default, permissions for files are set to 644. Don't change this; it's a safe value. For directories, a safe setting is 750 (which doesn't allow any public access). However, some extensions may need access to certain directories. In such cases, the 750 setting may result in error messages. In this case, set permissions to 755. Never leave permissions for a file or directory set to 777. This allows everybody to write data to it. You can also block direct access to critical directories using a .htaccess file. This is a special file containing instructions for the Apache web server. Among other things, it tells the web server who's allowed access to the directory contents. You can add a .htaccess file to any folder on the server by using specific instructions. This is another way to instruct the web server to restrict access. Check the Joomla security documentation at www.joomla.org for instructions.

(For more resources related to this topic, see here.) Essentials of HDFS HDFS is a distributed filesystem that has been designed to run on top of a cluster of industry standard hardware. The architecture of HDFS is such that there is no specific need for high-end hardware. HDFS is a highly fault-tolerant system and can handle failures of nodes in a cluster without loss of data. The primary goal behind the design of HDFS is to serve large data files efficiently. HDFS achieves this efficiency and high throughput in data transfer by enabling streaming access to the data in the filesystem. The following are the important features of HDFS: Fault tolerance: Many computers working together as a cluster are bound to have hardware failures. Hardware failures such as disk failures, network connectivity issues, and RAM failures could disrupt processing and cause major downtime. This could lead to data loss as well slippage of critical SLAs. HDFS is designed to withstand such hardware failures by detecting faults and taking recovery actions as required. The data in HDFS is split across the machines in the cluster as chunks of data called blocks. These blocks are replicated across multiple machines of the cluster for redundancy. So, even if a node/machine becomes completely unusable and shuts down, the processing can go on with the copy of the data present on the nodes where the data was replicated. Streaming data: Streaming access enables data to be transferred in the form of a steady and continuous stream. This means if data from a file in HDFS needs to be processed, HDFS starts sending the data as it reads the file and does not wait for the entire file to be read. The client who is consuming this data starts processing the data immediately, as it receives the stream from HDFS. This makes data processing really fast. Large data store: HDFS is used to store large volumes of data. HDFS functions best when the individual data files stored are large files, rather than having large number of small files. File sizes in most Hadoop clusters range from gigabytes to terabytes. The storage scales linearly as more nodes are added to the cluster. Portable: HDFS is a highly portable system. Since it is built on Java, any machine or operating system that can run Java should be able to run HDFS. Even at the hardware layer, HDFS is flexible and runs on most of the commonly available hardware platforms. Most production level clusters have been set up on commodity hardware. Easy interface: The HDFS command-line interface is very similar to any Linux/Unix system. The commands are similar in most cases. So, if one is comfortable with the performing basic file actions in Linux/Unix, using commands with HDFS should be very easy. The following two daemons are responsible for operations on HDFS: Namenode Datanode The namenode and datanodes daemons talk to each other via TCP/IP. Configuring HDFS All HDFS-related configuration is done by adding/updating the properties in the hdfs-site.xml file that is found in the conf folder under the Hadoop installation folder. The following are the different properties that are part of the hdfs-site.xml file: dfs.namenode.servicerpc-address: This specifies the unique namenode RPC address in the cluster. Services/daemons such as the secondary namenode and datanode daemons use this address to connect to the namenode daemon whenever it needs to communicate. This property is shown in the following code snippet: <property> <name>dfs.namenode.servicerpc-address</name> <value>node1.hcluster:8022</value> </property> dfs.namenode.http-address: This specifies the URL that can be used to monitor the namenode daemon from a browser. This property is shown in the following code snippet: <property> <name>dfs.namenode.http-address</name> <value>node1.hcluster:50070</value> </property> dfs.replication: This specifies the replication factor for data block replication across the datanode daemons. The default is 3 as shown in the following code snippet: <property> <name>dfs.replication</name> <value>3</value> </property> dfs.blocksize: This specifies the block size. In the following example, the size is specified in bytes (134,217,728 bytes is 128 MB): <property> <name>dfs.blocksize</name> <value>134217728</value> </property> fs.permissions.umask-mode: This specifies the umask value that will be used when creating files and directories in HDFS. This property is shown in the following code snippet: <property> <name>fs.permissions.umask-mode</name> <value>022</value> </property> The read/write operational flow in HDFS To get a better understanding of HDFS, we need to understand the flow of operations for the following two scenarios: A file is written to HDFS A file is read from HDFS HDFS uses a single-write, multiple-read model, where the files are written once and read several times. The data cannot be altered once written. However, data can be appended to the file by reopening it. All files in the HDFS are saved as data blocks. Writing files in HDFS The following sequence of steps occur when a client tries to write a file to HDFS: The client informs the namenode daemon that it wants to write a file. The namenode daemon checks to see whether the file already exists. If it exists, an appropriate message is sent back to the client. If it does not exist, the namenode daemon makes a metadata entry for the new file. The file to be written is split into data packets at the client end and a data queue is built. The packets in the queue are then streamed to the datanodes in the cluster. The list of datanodes is given by the namenode daemon, which is prepared based on the data replication factor configured. A pipeline is built to perform the writes to all datanodes provided by the namenode daemon. The first packet from the data queue is then transferred to the first datanode daemon. The block is stored on the first datanode daemon and is then copied to the next datanode daemon in the pipeline. This process goes on till the packet is written to the last datanode daemon in the pipeline. The sequence is repeated for all the packets in the data queue. For every packet written on the datanode daemon, a corresponding acknowledgement is sent back to the client. If a packet fails to write onto one of the datanodes, the datanode daemon is removed from the pipeline and the remainder of the packets is written to the good datanodes. The namenode daemon notices the under-replication of the block and arranges for another datanode daemon where the block could be replicated. After all the packets are written, the client performs a close action, indicating that the packets in the data queue have been completely transferred. The client informs the namenode daemon that the write operation is now complete. The following diagram shows the data block replication process across the datanodes during a write operation in HDFS: Reading files in HDFS The following steps occur when a client tries to read a file in HDFS: The client contacts the namenode daemon to get the location of the data blocks of the file it wants to read. The namenode daemon returns the list of addresses of the datanodes for the data blocks. For any read operation, HDFS tries to return the node with the data block that is closest to the client. Here, closest refers to network proximity between the datanode daemon and the client. Once the client has the list, it connects the closest datanode daemon and starts reading the data block using a stream. After the block is read completely, the connection to datanode is terminated and the datanode daemon that hosts the next block in the sequence is identified and the data block is streamed. This goes on until the last data block for that file is read. The following diagram shows the read operation of a file in HDFS: Understanding the namenode UI Hadoop provides web interfaces for each of its services. The namenode UI or the namenode web interface is used to monitor the status of the namenode and can be accessed using the following URL: http://<namenode-server>:50070/ The namenode UI has the following sections: Overview: The general information section provides basic information of the namenode with options to browse the filesystem and the namenode logs. The following is the screenshot of the Overview section of the namenode UI: The Cluster ID parameter displays the identification number of the cluster. This number is same across all the nodes within the cluster. A block pool is a set of blocks that belong to a single namespace. The Block Pool Id parameter is used to segregate the block pools in case there are multiple namespaces configured when using HDFS federation. In HDFS federation, multiple namenodes are configured to scale the name service horizontally. These namenodes are configured to share datanodes amongst themselves. We will be exploring HDFS federation in detail a bit later. Summary: The following is the screenshot of the cluster's summary section from the namenode web interface: Under the Summary section, the first parameter is related to the security configuration of the cluster. If Kerberos (the authorization and authentication system used in Hadoop) is configured, the parameter will show as Security is on. If Kerberos is not configured, the parameter will show as Security is off. The next parameter displays information related to files and blocks in the cluster. Along with this information, the heap and non-heap memory utilization is also displayed. The other parameters displayed in the Summary section are as follows: Configured Capacity: This displays the total capacity (storage space) of HDFS. DFS Used: This displays the total space used in HDFS. Non DFS Used: This displays the amount of space used by other files that are not part of HDFS. This is the space used by the operating system and other files. DFS Remaining: This displays the total space remaining in HDFS. DFS Used%: This displays the total HDFS space utilization shown as percentage. DFS Remaining%: This displays the total HDFS space remaining shown as percentage. Block Pool Used: This displays the total space utilized by the current namespace. Block Pool Used%: This displays the total space utilized by the current namespace shown as percentage. As you can see in the preceding screenshot, in this case, the value matches that of the DFS Used% parameter. This is because there is only one namespace (one namenode) and HDFS is not federated. DataNodes usages% (Min, Median, Max, stdDev): This displays the usages across all datanodes in the cluster. This helps administrators identify unbalanced nodes, which may occur when data is not uniformly placed across the datanodes. Administrators have the option to rebalance the datanodes using a balancer. Live Nodes: This link displays all the datanodes in the cluster as shown in the following screenshot: Dead Nodes: This link displays all the datanodes that are currently in a dead state in the cluster. A dead state for a datanode daemon is when the datanode daemon is not running or has not sent a heartbeat message to the namenode daemon. Datanodes are unable to send heartbeats if there exists a network connection issue between the machines that host the datanode and namenode daemons. Excessive swapping on the datanode machine causes the machine to become unresponsive, which also prevents the datanode daemon from sending heartbeats. Decommissioning Nodes: This link lists all the datanodes that are being decommissioned. Number of Under-Replicated Blocks: This represents the number of blocks that have not replicated as per the replication factor configured in the hdfs-site.xml file. Namenode Journal Status: The journal status provides location information of the fsimage file and the state of the edits logfile. The following screenshot shows the NameNode Journal Status section: NameNode Storage: The namenode storage table provides the location of the fsimage file along with the type of the location. In this case, it is IMAGE_AND_EDITS, which means the same location is used to store the fsimage file as well as the edits logfile. The other types of locations are IMAGE, which stores only the fsimage file and EDITS, which stores only the edits logfile. The following screenshot shows the NameNode Storage information:

Service Invocation Time for action – creating the book warehousing process Let's create the BookWarehousingBPEL BPEL process: We will open the SOA composite by double-clicking on the Bookstore in the process tree. In the composite, we can see both Bookstore BPEL processes and their WSDL interfaces. We will add a new BPEL process by dragging-and-dropping the BPEL Process service component from the right-hand side toolbar. An already-familiar dialog window will open, where we will specify the BPEL 2.0 version, enter the process name as BookWarehousingBPEL, modify the namespace to http://packtpub.com/Bookstore/BookWarehousingBPEL, and select Synchronous BPEL Process. We will leave all other fields to their default values: Insert image 8963EN_02_03.png Next, we will wire the BookWarehousingBPEL component with the BookstoreABPEL and BookstoreBBPEL components. This way, we will establish a partner link between them. First, we will create the wire to the BookstoreBBPEL component (although the order doesn't matter). To do this, you have to click on the bottom-right side of the component. Once you place the mouse pointer above the component, circles on the edges will appear. You need to start with the circle labelled as Drag to add a new Reference and connect it with the service interface circle, as shown in the following figure: Insert image 8963EN_02_04.png You do the same to wire the BookWarehousingBPEL component with the BookstoreABPEL component: Insert image 8963EN_02_05.png We should see the following screenshot. Please notice that the BookWarehousingBPEL component is wired to the BookstoreABPEL and BookstoreBBPEL components: Insert image 8963EN_02_06.png What just happened? We have added the BookWarehousingBPEL component to our SOA composite and wired it to the BookstoreABPEL and BookstoreBBPEL components. Creating the wires between components means that references have been created and relations between components have been established. In other words, we have expressed that the BookWarehousingBPEL component will be able to invoke the BookstoreABPEL and BookstoreBBPEL components. This is exactly what we want to achieve with our BookWarehousingBPEL process, which will orchestrate both bookstore BPELs. Once we have created the components and wired them accordingly, we are ready to implement the BookWarehousingBPEL process. What just happened? We have implemented the process flow logic for the BookWarehousingBPEL process. It actually orchestrated two other services. It invoked the BookstoreA and BookstoreB BPEL processes that we've developed in the previous chapter. Here, these two BPEL processes acted like services. First, we have created the XML schema elements for the request and response messages of the BPEL process. We have created four variables: BookstoreARequest, BookstoreAResponse, BookstoreBRequest, and BookstoreBResponse. The BPEL code for variable declaration looks like the code shown in the following screenshot: Insert image 8963EN_02_18b.png Then, we have added the <assign> activity to prepare a request for both the bookstore BPEL processes. Then, we have copied the BookData element from inputVariable to the BookstoreARequest and BookstoreBRequest variables. The following BPEL code has been generated: Insert image 8963EN_02_18c.png Next, we have invoked both the bookstore BPEL services using the <invoke> activity. The BPEL source code reads like the following screenshot: Insert image 8963EN_02_18d.png Finally, we have added the <if> activity to select the bookstore with the lowest stock quantity: Insert image 8963EN_02_18e.png With this, we have concluded the development of the book warehousing BPEL process and are ready to deploy and test it. Deploying and testing the book warehousing BPEL We will deploy the project to the SOA Suite process server the same way as we did in the previous chapter, by right-clicking on the project and selecting Deploy. Then, we will navigate through the options. As we redeploy the whole SOA composite, make sure the Overwrite option is selected. You should check if the compilation and the deployment have been successful. Then, we will log in to the Enterprise Manager console, select the project bookstore, and click on the Test button. Be sure to select bookwarehousingbpel_client for the test. After entering the parameters, as shown in the following figure, we can click on the Test Web Service button to initiate the BPEL process, and we should receive an answer (Bookstore A or Bookstore B, depending on the data that we have entered). Remember that our BookWarehousingBPEL process actually orchestrated two other services. It invoked the BookstoreA and BookstoreB BPEL processes. To verify this, we can launch the flow trace (click on the Launch Flow Trace button) in the Enterprise Manager console, and we will see that the two bookstore BPEL processes have been invoked, as shown: Insert image 8963EN_02_19.png An even more interesting view is the flow view, which also shows that both bookstore services have been invoked. Click on BookWarehousingBPEL to open up the audit trail: Insert image 8963EN_02_20.png Understanding partner links When invoking services, we have often mentioned partner links. Partner links denote all the interactions that a BPEL process has with the external services. There are two possibilities: The BPEL process invokes operations on other services. The BPEL process receives invocations from clients. One of the clients is the user of the BPEL process, who makes the initial invocation. Other clients are services, for example, those that have been invoked by the BPEL process, but may return replies. Links to all the parties that BPEL interacts with are called partner links. Partner links can be links to services that are invoked by the BPEL process. These are sometimes called invoked partner links. Partner links can also be links to clients, and can invoke the BPEL process. Such partner links are sometimes called client partner links. Note that each BPEL process has at least one client partner link, because there has to be a client that first invokes the BPEL process. Usually, a BPEL process will also have several invoked partner links, because it will most likely invoke several services. In our case, the BookWarehousingBPEL process has one client partner link and two invoked partner links, BookstoreABPEL and BookstoreBBPEL. You can observe this on the SOA composite design view and on the BPEL process itself, where the client partner links are located on the left-hand side, while the invoked partner links are located on the right-hand side of the design view. Partner link types Describing situations where the service is invoked by the process and vice versa requires selecting a certain perspective. We can select the process perspective and describe the process as requiring portTypeA on the service and providing portTypeB to the service. Alternatively, we select the service perspective and describe the service as offering portTypeA to the BPEL process and requiring portTypeB from the process. Partner link types allow us to model such relationships as a third party. We are not required to take a certain perspective; rather, we just define roles. A partner link type must have at least one role and can have at most two roles. The latter is the usual case. For each role, we must specify a portType that is used for interaction. Partner link types are defined in the WSDLs. If we take a closer look at the BookWarehousingBPEL.wsdl file, we can see the following partner link type definition: Insert image 8963EN_02_22.png If we specify only one role, we express the willingness to interact with the service, but do not place any additional requirements on the service. Sometimes, existing services will not define a partner link type. Then, we can wrap the WSDL of the service and define partner link types ourselves. Now that we have become familiar with the partner link types and know that they belong to WSDL, it is time to go back to the BPEL process definition, more specifically to the partner links. Defining partner links Partner links are concrete references to services that a BPEL business process interacts with. They are specified near the beginning of the BPEL process definition document, just after the <process> tag. Several <partnerLink> definitions are nested within the <partnerLinks> element: <process ...>   <partnerLinks>   <partnerLink ... /> <partnerLink ... />        ...      </partnerLinks>   <sequence>      ... </sequence> </process> For each partner link, we have to specify the following: name: This serves as a reference for interactions via that partner link. partnerLinkType: This defines the type of the partner link. myRole: This indicates the role of the BPEL process itself. partnerRole: This indicates the role of the partner. initializePartnerRole: This indicates whether the BPEL engine should initialize the partner link's partner role value. This is an optional attribute and should only be used with partner links that specify the partner role. We define both roles (myRole and partnerRole) only if the partnerLinkType specifies two roles. If the partnerLinkType specifies only one role, the partnerLink also has to specify only one role—we omit the one that is not needed. Let's go back to our example, where we have defined the BookstoreABPEL partner link type. To define a partner link, we need to specify the partner roles, because it is a synchronous relation. The definition is shown in the following code excerpt: Insert image 8963EN_02_23.png With this, we have concluded the discussion on partner links and partner link types. We will continue with the parallel service invocation. Parallel service invocation BPEL also supports parallel service invocation. In our example, we have invoked Bookstore A and Bookstore B sequentially. This way, we need to wait for the response from the first service and then for the response from the second service. If these services take longer to execute, the response times will be added together. If we invoke both services in parallel, we only need to wait for the duration of the longer-lasting call, as shown in the following screenshot: BPEL has several possibilities for parallel flows, which will be described in detail in Chapter 8, Dynamic Parallel Invocations. Here, we present the basic parallel service invocation using the <flow> activity. Insert image 8963EN_02_24.png To invoke services in parallel, or to perform any other activities in parallel, we can use the <flow> activity. Within the <flow> activity, we can nest an arbitrary number of flows, and all will execute in parallel. Let's try and modify our example so that Bookstore A and B will be invoked in parallel. Time for action – developing parallel flows Let's now modify the BookWarehousingBPEL process so that the BookstoreA and BookstoreB services will be invoked in parallel. We should do the following: To invoke BookstoreA and BookstoreB services in parallel, we need to add the Flow structured activity to the process flow just before the first invocation, as shown in the following screenshot: Insert image 8963EN_02_25.png We see that two parallel branches have been created. We simply drag-and-drop both the invoke activities into the parallel branches: Insert image 8963EN_02_26.png That's all! We can create more parallel branches if we need, by clicking on the Add Sequence icon. What just happened? We have modified the BookWarehousingBPEL process so that the BookstoreA and BookstoreB <invoke> activities are executed in parallel. A corresponding <flow>activity has been created in the BPEL source code. Within the <flow> activity, both <invoke> activities are nested. Please notice that each <invoke> activity is placed within its own <sequence> activity. T his would make sense if we would require more than one activity in each parallel branch. The BPEL source code looks like the one shown in the following screenshot: Insert image 8963EN_02_27.png Deploying and testing the parallel invocation We will deploy the project to the SOA Suite process server the same way we did in the previous sample. Then, we will log in to the Enterprise Manager console, select the project Bookstore, and click on the Test Web Service button. To observe that the services have been invoked in parallel, we can launch the flow trace (click on the Launch Flow Trace button) in the Enterprise Manager console, click on the book warehousing BPEL processes, and activate the flow view, which shows that both bookstore services have been invoked in parallel. Understanding parallel flow To invoke services concurrently, we can use the <flow> construct. In the following example, the three <invoke> operations will perform concurrently: <process ...>    ... <sequence> <!-- Wait for the incoming request to start the process --> <receive ... />   <!-- Invoke a set of related services, concurrently --> <flow> <invoke ... /> <invoke ... /> <invoke ... /> </flow>      ... <!-- Return the response --> <reply ... /> </sequence> </process> We can also combine and nest the <sequence> and <flow> constructs that allow us to define several sequences that execute concurrently. In the following example, we have defined two sequences, one that consists of three invocations, and one with two invocations. Both sequences will execute concurrently: <process ...>    ... <sequence>   <!-- Wait for the incoming request to start the process --> <receive ... />   <!-- Invoke two sequences concurrently --> <flow> <!-- The three invokes below execute sequentially --> <sequence> <invoke ... /> <invoke ... /> <invoke ... /> </sequence> <!-- The two invokes below execute sequentially --> <sequence> <invoke ... /> <invoke ... /> </sequence> </flow>        ... <!-- Return the response --> <reply ... /> </sequence> </process> We can use other activities as well within the <flow> activity to achieve parallel execution. With this, we have concluded our discussion on the parallel invocation. Pop quiz – service invocation Q1. Which activity is used to invoke services from BPEL processes? <receive> <invoke> <sequence> <flow> <process> <reply> Q2. Which parameters do we need to specify for the <invoke> activity? endpointURL partnerLink operationName operation portType portTypeLink Q3. In which file do we declare partner link types? Q4. Which activity is used to execute service invocation and other BPEL activities in parallel? <receive> <invoke> <sequence> <flow> <process> <reply> Summary In this chapter, we learned how to invoke services and orchestrate services. We explained the primary mission of BPEL—service orchestration. It follows the concept of programming-in-the-large. We have developed a BPEL process that has invoked two services and orchestrated them. We have become familiar with the <invoke> activity and understood the service invocation's background, particularly partner links and partner link types. We also learned that from BPEL, it is very easy to invoke services in parallel. To achieve this, we use the <flow> activity. Within <flow>, we can not only nest several <invoke> activities but also other BPEL activities. Now, we are ready to learn about variables and data manipulation, which we will do in the next chapter.  

(For more resources related to this topic, see here.) The primary mechanism that PostgreSQL uses to provide a data durability guarantee is through its Write Ahead Log (WAL). All transactional data is written to this location before ever being committed to database files. Once WAL files are no longer necessary for crash recovery, PostgreSQL will either delete or archive them. For the purposes of a highly available server, we recommend that you keep these important files as long as possible. There are several reasons for this; they are as follows: Archived WAL files can be used for Point In Time Recovery (PITR) If you are using streaming replication, interrupted streams can be re-established by applying WAL files until the replica has caught up WAL files can be reused to service multiple server copies In order to gain these benefits, we need to enable PostgreSQL WAL archiving and save these files until we no longer need them. This section will address our recommendations for long term storage of WAL files. Getting ready In order to properly archive WAL files, we recommend that you provision a server dedicated to backups or file storage. Depending on the transaction volume, an active PostgreSQL database might produce thousands of these on a daily basis. At 16 MB apiece, this is not an idle concern. For instance, for a 1 TB database, we recommend at least 3 TB of storage space. In addition, we will be using rsync as a daemon on this archive server. To install this on a Debian-based server, execute the following command as a root-level user: sudo apt-get install rsync Red-Hat-based systems will need this command instead: sudo yum install rsync xinetd How to do it... Our archive server has a 3 TB mount at the /db directory and is named arc_server on our network. The PostgreSQL source server resides at 192.168.56.10. Follow these steps for long-term storage of important WAL files on an archive server Enable rsync to run as a daemon on the archive server. On Debian based systems, edit the /etc/default/rsync file and change the RSYNC_ENABLE variable to true. On Red-Hat-based systems, edit the /etc/xinet.d/rsync file and change the disable parameter to no. Create a directory to store archived WAL files as the postgres user with these commands: sudo mkdir /db/pg_archived sudo chown postgres:postgres /db/pg_archived Create a file named /etc/rsyncd.conf and fill it with the following contents: [wal_store] path = /db/pg_archived comment = DB WAL Archives uid = postgres gid = postgres read only = false hosts allow = 192.168.56.10 hosts deny = * Start the rsync daemon. Debian-based systems should execute the following command: sudo service rsync start Red-Hat-based systems can start rsync with this command instead: sudo service xinetd start Change the archive_mode and archive_command parameters in postgresql.conf to read the following: archive_mode = on archive_command = 'rsync -aq %p arc_server::wal_store/%f' Restart the PostgreSQL server with a command similar to this: pg_ctl -D $PGDATA restart How it works The rsync utility is normally used to transfer files between two servers. However, we can take advantage of using it as a daemon to avoid connection overhead imposed by using SSH as an rsync protocol. Our first step is to ensure that the service is not disabled in some manner, which would make the rest of this guide moot. Next, we need a place to store archived WAL files on the archive server. Assuming that we have 3 TB of space in the /db directory, we simply claim /db/pg_archived as the desired storage location. There should be enough space to use /db for backups as well, but we won't discuss that here. Next, we create a file named /etc/rsyncd.conf, which will configure how rsync operates as a daemon. Here, we name the /db/pg_archived directory wal_store so that we can address the path by its name when sending files. We give it a human-readable name and ensure that files are owned by the postgres user, as this user also controls most of the PostgreSQL-related services. The next, and possibly the most important step, is to block all hosts but the primary PostgreSQL server from writing to this location. We set hosts deny to *, which blocks every server. Then, we set hosts allow to the primary database server's IP address so that only it has access. If everything goes well, we can start the rsync (or xinetd on Red Hat systems) service and we can see that in the following screenshot: Next, we enable archive_mode by setting it to on. With archive mode enabled, we can specify a command that will execute when PostgreSQL no longer needs a WAL file for crash recovery. In this case, we invoke the rsync command with the -a parameter to preserve elements such as file ownership, timestamps, and so on. In addition, we specify the -q setting to suppress output, as PostgreSQL only checks the command exit status to determine its success. In the archive_command setting, the %p value represents the full path to the WAL file, and %f resolves to the filename. In this context, we're sending the WAL file to the archive server at the wal_store module we defined in rsyncd.conf. Once we restart PostgreSQL, it will start storing all the old WAL files by sending them to the archive server. In case any rsync command fails because the archive server is unreachable, PostgreSQL will keep trying to send it until it is successful. If the archive server is unreachable for too long, we suggest that you change the archive_command setting to store files elsewhere. This prevents accidentally overfilling the PostgreSQL server storage. There's more... As we will likely want to use the WAL files on other servers, we suggest that you make a list of all the servers that could need WAL files. Then, modify the rsyncd.conf file on the archive server and add this section: [wal_fetch] path = /db/pg_archived comment = DB WAL Archive Retrieval uid = postgres gid = postgres read only = true hosts allow = host1, host2, host3 hosts deny = * Now, we can fetch WAL files from any of the hosts in hosts allow. As these are dedicated PostgreSQL replicas, recovery servers, or other defined roles, this makes the archive server a central location for all our WAL needs. See also We suggest that you read more about the archive_mode and archive_command settings on the PostgreSQL site. We've included a link here: http://www.postgresql.org/docs/9.3/static/runtime-config-wal.html The rsyncd.conf file should also have its own manual page. Read it with this command to learn more about the available settings: man rsyncd.conf Summary In this article, we've successfully learned how to secure the WAL stream by following the given steps. Resources for Article: Further resources on this subject: PostgreSQL 9: Reliable Controller and Disk Setup [article] Backup in PostgreSQL 9 [article] Recovery in PostgreSQL 9 [article]

(For more resources related to this topic, see here.) Drinking Mockito Download the latest Mockito binary from the following link and add it to the project dependency: http://code.google.com/p/mockito/downloads/list As of February 2014, the latest Mockito version is 1.9.5. Configuring Mockito To add Mockito JAR files to the project dependency, perform the following steps: Extract the JAR files into a folder. Launch Eclipse. Create an Eclipse project named Chapter04. Go to the Libraries tab in the project build path. Click on the Add External JARs... button and browse to the Mockito JAR folder. Select all JAR files and click on OK. We worked with Gradle and Maven and built a project with the JUnit dependency. In this section, we will add Mockito dependencies to our existing projects. The following code snippet will add a Mockito dependency to a Maven project and download the JAR file from the central Maven repository (http://mvnrepository.com/artifact/org.mockito/mockito-core): <dependency> <groupId>org.mockito</groupId> <artifactId>mockito-core</artifactId> <version>1.9.5</version> <scope>test</scope> </dependency> The following Gradle script snippet will add a Mockito dependency to a Gradle project: testCompile 'org.mockito:mockito-core:1.9.5' Mocking in action This section demonstrates the mock objects with a stock quote example. In the real world, people invest money on the stock market—they buy and sell stocks. A stock symbol is an abbreviation used to uniquely identify shares of a particular stock on a particular market, such as stocks of Facebook are registered on NASDAQ as FB and stocks of Apple as AAPL. We will build a stock broker simulation program. The program will watch the market statistics, and depending on the current market data, you can perform any of the following actions: Buy stocks Sell stocks Hold stocks The domain classes that will be used in the program are Stock, MarketWatcher, Portfolio, and StockBroker. Stock represents a real-world stock. It has a symbol, company name, and price. MarketWatcher looks up the stock market and returns the quote for the stock. A real implementation of a market watcher can be implemented from http://www.wikijava.org/wiki/Downloading_stock_market_quotes_from_Yahoo!_finance. Note that the real implementation will connect to the Internet and download the stock quote from a provider. Portfolio represents a user's stock data such as the number of stocks and price details. Portfolio exposes APIs for getting the average stock price and buying and selling stocks. Suppose on day one someone buys a share at a price of $10.00, and on day two, the customer buys the same share at a price of $8.00. So, on day two the person has two shares and the average price of the share is $9.00. The following screenshot represents the Eclipse project structure. You can download the project from the Packt Publishing website and work with the files: The following code snippet represents the StockBroker class. StockBroker collaborates with the MarketWatcher and Portfolio classes. The perform() method of StockBroker accepts a portfolio and a Stock object: public class StockBroker { private final static BigDecimal LIMIT = new BigDecimal("0.10"); private final MarketWatcher market; public StockBroker(MarketWatcher market) { this.market = market; } public void perform(Portfolio portfolio,Stock stock) { Stock liveStock = market.getQuote(stock.getSymbol()); BigDecimal avgPrice = portfolio.getAvgPrice(stock); BigDecimal priceGained = liveStock.getPrice().subtract(avgPrice); BigDecimal percentGain = priceGained.divide(avgPrice); if(percentGain.compareTo(LIMIT) > 0) { portfolio.sell(stock, 10); }else if(percentGain.compareTo(LIMIT) < 0){ portfolio.buy(stock); } } } Look at the perform method. It takes a portfolio object and a stock object, calls the getQuote method of MarketWatcher, and passes a stock symbol. Then, it gets the average stock price from portfolio and compares the current market price with the average stock price. If the current stock price is 10 percent greater than the average price, then the StockBroker program sells 10 stocks from Portfolio; however, if the current stock price goes down by 10 percent, then the program buys shares from the market to average out the loss. Why do we sell 10 stocks? This is just an example and 10 is just a number; this could be anything you want. StockBroker depends on Portfolio and MarketWatcher; a real implementation of Portfolio should interact with a database, and MarketWatcher needs to connect to the Internet. So, if we write a unit test for the broker, we need to execute the test with a database and an Internet connection. A database connection will take time and Internet connectivity depends on the Internet provider. So, the test execution will depend on external entities and will take a while to finish. This will violate the quick test execution principle. Also, the database state might not be the same across all test runs. This is also applicable for the Internet connection service. Each time the database might return different values, and therefore asserting a specific value in your unit test is very difficult. We'll use Mockito to mock the external dependencies and execute the test in isolation. So, the test will no longer be dependent on real external service, and therefore it will be executed quickly. Mocking objects A mock can be created with the help of a static mock() method as follows: import org.mockito.Mockito; public class StockBrokerTest { MarketWatcher marketWatcher = Mockito.mock(MarketWatcher.class); Portfolio portfolio = Mockito.mock(Portfolio.class); } Otherwise, you can use Java's static import feature and static import the mock method of the org.mockito.Mockito class as follows: import static org.mockito.Mockito.mock; public class StockBrokerTest { MarketWatcher marketWatcher = mock(MarketWatcher.class); Portfolio portfolio = mock(Portfolio.class); } There's another alternative; you can use the @Mock annotation as follows: import org.mockito.Mock; public class StockBrokerTest { @Mock MarketWatcher marketWatcher; @Mock Portfolio portfolio; } However, to work with the @Mock annotation, you are required to call MockitoAnnotations.initMocks( this ) before using the mocks, or use MockitoJUnitRunner as a JUnit runner. The following code snippet uses MockitoAnnotations to create mocks: import static org.junit.Assert.assertEquals; import org.mockito.MockitoAnnotations; public class StockBrokerTest { @Mock MarketWatcher marketWatcher; @Mock Portfolio portfolio; @Before public void setUp() { MockitoAnnotations.initMocks(this); } @Test public void sanity() throws Exception { assertNotNull(marketWatcher); assertNotNull(portfolio); } } The following code snippet uses the MockitoJUnitRunner JUnit runner: import org.mockito.runners.MockitoJUnitRunner; @RunWith(MockitoJUnitRunner.class) public class StockBrokerTest { @Mock MarketWatcher marketWatcher; @Mock Portfolio portfolio; @Test public void sanity() throws Exception { assertNotNull(marketWatcher); assertNotNull(portfolio); } } Before we deep dive into the Mockito world, there are a few things to remember. Mockito cannot mock or spy the following functions: final classes, final methods, enums, static methods, private methods, the hashCode() and equals() methods, anonymous classes, and primitive types. PowerMock (an extension of EasyMock) and PowerMockito (an extension of the Mockito framework) allows you to mock static and private methods; even PowerMockito allows you to set expectations on new invocations for private member classes, inner classes, and local or anonymous classes. However, as per the design, you should not opt for mocking private/static properties—it violates the encapsulation. Instead, you should refactor the offending code to make it testable. Change the Portfolio class, create the final class, and rerun the test; the test will fail as the Portfolio class is final, and Mockito cannot mock a final class. The following screenshot shows the JUnit output: Stubbing methods We read about stubs in ,Test Doubles. The stubbing process defines the behavior of a mock method such as the value to be returned or the exception to be thrown when the method is invoked. The Mockito framework supports stubbing and allows us to return a given value when a specific method is called. This can be done using Mockito.when() along with thenReturn (). The following is the syntax of importing when: import static org.mockito.Mockito.when; The following code snippet stubs the getQuote(String symbol) method of MarcketWatcher and returns a specific Stock object: import static org.mockito.Matchers.anyString; import static org.mockito.Mockito.when; @RunWith(MockitoJUnitRunner.class) public class StockBrokerTest { @Mock MarketWatcher marketWatcher; @Mock Portfolio portfolio; @Test public void marketWatcher_Returns_current_stock_status() { Stock uvsityCorp = new Stock("UV", "Uvsity Corporation", new BigDecimal("100.00")); when (marketWatcher.getQuote( anyString ())). thenReturn (uvsityCorp); assertNotNull(marketWatcher.getQuote("UV")); } } A uvsityCorp stock object is created with a stock price of $100.00 and the getQuote method is stubbed to return uvsityCorp whenever the getQuote method is called. Note that anyString() is passed to the getQuote method, which means whenever the getQuote method will be called with any String value, the uvsityCorp object will be returned. The when() method represents the trigger, that is, when to stub. The following methods are used to represent what to do when the trigger is triggered: thenReturn(x): This returns the x value. thenThrow(x): This throws an x exception. thenAnswer(Answer answer): Unlike returning a hardcoded value, a dynamic user-defined logic is executed. It's more like for fake test doubles, Answer is an interface. thenCallRealMethod(): This method calls the real method on the mock object. The following code snippet stubs the external dependencies and creates a test for the StockBroker class: import com.packt.trading.dto.Stock; import static org.junit.Assert.assertNotNull; import static org.mockito.Matchers.anyString; import static org.mockito.Matchers.isA; import static org.mockito.Mockito.verify; import static org.mockito.Mockito.when; @RunWith(MockitoJUnitRunner.class) public class StockBrokerTest { @Mock MarketWatcher marketWatcher; @Mock Portfolio portfolio; StockBroker broker; @Before public void setUp() { broker = new StockBroker(marketWatcher); } @Test public void when_ten_percent_gain_then_the_stock_is_sold() { //Portfolio's getAvgPrice is stubbed to return $10.00 when(portfolio.getAvgPrice(isA(Stock.class))). thenReturn(new BigDecimal("10.00")); //A stock object is created with current price $11.20 Stock aCorp = new Stock("A", "A Corp", new BigDecimal("11.20")); //getQuote method is stubbed to return the stock when(marketWatcher.getQuote(anyString())).thenReturn(aCorp); //perform method is called, as the stock price increases // by 12% the broker should sell the stocks broker.perform(portfolio, aCorp); //verifying that the broker sold the stocks verify(portfolio).sell(aCorp,10); } } The test method name is when_ten_percent_gain_then_the_stock_is_sold; a test name should explain the intention of the test. We use underscores to make the test name readable. We will use the when_<<something happens>>_then_<<the action is taken>> convention for the tests. In the preceding test example, the getAvgPrice() method of portfolio is stubbed to return $10.00, then the getQuote method is stubbed to return a hardcoded stock object with a current stock price of $11.20. The broker logic should sell the stock as the stock price goes up by 12 percent. The portfolio object is a mock object. So, unless we stub a method, by default, all the methods of portfolio are autostubbed to return a default value, and for the void methods, no action is performed. The sell method is a void method; so, instead of connecting to a database to update the stock count, the autostub will do nothing. However, how will we test whether the sell method was invoked? We use Mockito.verify. The verify() method is a static method, which is used to verify the method invocation. If the method is not invoked, or the argument doesn't match, then the verify method will raise an error to indicate that the code logic has issues. Verifying the method invocation To verify a redundant method invocation, or to verify whether a stubbed method was not called but was important from the test perspective, we should manually verify the invocation; for this, we need to use the static verify method. Why do we use verify? Mock objects are used to stub external dependencies. We set an expectation, and a mock object returns an expected value. In some conditions, a behavior or method of a mock object should not be invoked, or sometimes, we may need to call the method N (a number) times. The verify method verifies the invocation of mock objects. Mockito does not automatically verify all stubbed calls. If a stubbed behavior should not be called but the method is called due to a bug in the code, verify flags the error though we have to verify that manually. The void methods don't return values, so you cannot assert the returned values. Hence, verify is very handy to test the void methods. Verifying in depth The verify() method has an overloaded version that takes Times as an argument. Times is a Mockito framework class of the org.mockito.internal.verification package, and it takes wantedNumberOfInvocations as an integer argument. If 0 is passed to Times, it infers that the method will not be invoked in the testing path. We can pass 0 to Times(0) to make sure that the sell or buy methods are not invoked. If a negative number is passed to the Times constructor, Mockito throws MockitoException - org.mockito.exceptions.base.MockitoException, and this shows the Negative value is not allowed here error. The following methods are used in conjunction with verify: times(int wantedNumberOfInvocations): This method is invoked exactly n times; if the method is not invoked wantedNumberOfInvocations times, then the test fails. never(): This method signifies that the stubbed method is never called or you can use times(0) to represent the same scenario. If the stubbed method is invoked at least once, then the test fails. atLeastOnce(): This method is invoked at least once, and it works fine if it is invoked multiple times. However, the operation fails if the method is not invoked. atLeast(int minNumberOfInvocations): This method is called at least n times, and it works fine if the method is invoked more than the minNumberOfInvocations times. However, the operation fails if the method is not called minNumberOfInvocations times. atMost(int maxNumberOfInvocations): This method is called at the most n times. However, the operation fails if the method is called more than minNumberOfInvocations times. only(): The only method called on a mock fails if any other method is called on the mock object. In our example, if we use verify(portfolio, only()).sell(aCorp,10);, the test will fail with the following output: The test fails in line 15 as portfolio.getAvgPrice(stock) is called. timeout(int millis): This method is interacted in a specified time range. Verifying zero and no more interactions The verifyZeroInteractions(Object... mocks) method verifies whether no interactions happened on the given mocks. The following test code directly calls verifyZeroInteractions and passes the two mock objects. Since no methods are invoked on the mock objects, the test passes: @Test public void verify_zero_interaction() { verifyZeroInteractions(marketWatcher,portfolio); } The verifyNoMoreInteractions(Object... mocks) method checks whether any of the given mocks has any unverified interaction. We can use this method after verifying a mock method to make sure that nothing else was invoked on the mock. The following test code demonstrates verifyNoMoreInteractions: @Test public void verify_no_more_interaction() { Stock noStock = null; portfolio.getAvgPrice(noStock); portfolio.sell(null, 0); verify(portfolio).getAvgPrice(eq(noStock)); //this will fail as the sell method was invoked verifyNoMoreInteractions(portfolio); } The following is the JUnit output: The following are the rationales and examples of argument matchers. Using argument matcher ArgumentMatcher is a Hamcrest matcher with a predefined describeTo() method. ArgumentMatcher extends the org.hamcrest.BaseMatcher package. It verifies the indirect inputs into a mocked dependency. The Matchers.argThat(Matcher) method is used in conjunction with the verify method to verify whether a method is invoked with a specific argument value. ArgumentMatcher plays a key role in mocking. The following section describes the context of ArgumentMatcher. Mock objects return expected values, but when they need to return different values for different arguments, argument matcher comes into play. Suppose we have a method that takes a player name as input and returns the total number of runs (a run is a point scored in a cricket match) scored as output. We want to stub it and return 100 for Sachin and 10 for xyz. We have to use argument matcher to stub this. Mockito returns expected values when a method is stubbed. If the method takes arguments, the argument must match during the execution; for example, the getValue(int someValue) method is stubbed in the following way: when(mockObject.getValue(1)).thenReturn(expected value); Here, the getValue method is called with mockObject.getValue(100). Then, the parameter doesn't match (it is expected that the method will be called with 1, but at runtime, it encounters 100), so the mock object fails to return the expected value. It will return the default value of the return type—if the return type is Boolean, it'll return false; if the return type is object, then null, and so on. Mockito verifies argument values in natural Java style by using an equals() method. Sometimes, we use argument matchers when extra flexibility is required. Mockito provides built-in matchers such as anyInt(), anyDouble(), anyString(), anyList(), and anyCollection(). More built-in matchers and examples of custom argument matchers or Hamcrest matchers can be found at the following link: http://docs.mockito.googlecode.com/hg/latest/org/mockito/Matchers.html Examples of other matchers are isA(java.lang.Class<T> clazz), any(java.lang.Class<T> clazz), and eq(T) or eq(primitive value). The isA argument checks whether the passed object is an instance of the class type passed in the isA argument. The any(T) argument also works in the same way. Why do we need wildcard matchers? Wildcard matchers are used to verify the indirect inputs to the mocked dependencies. The following example describes the context. In the following code snippet, an object is passed to a method and then a request object is created and passed to service. Now, from a test, if we call the someMethod method and service is a mocked object, then from test, we cannot stub callMethod with a specific request as the request object is local to the someMethod: public void someMethod(Object obj){ Request req = new Request(); req.setValue(obj); Response resp = service.callMethod(req); } If we are using argument matchers, all arguments have to be provided by matchers. We're passing three arguments and all of them are passed using matchers: verify(mock).someMethod(anyInt(), anyString(), eq("third argument")); The following example will fail because the first and the third arguments are not passed using matcher: verify(mock).someMethod(1, anyString(), "third argument");