How-To Tutorials

In this article by Massimiliano Pippi, author of the book Python for Google App Engine, in this article, you will learn how to write a web application and seeing the platform in action. Web applications commonly provide a set of features such as user authentication and data storage. App Engine provides the services and tools needed to implement such features. (For more resources related to this topic, see here.) In this article, we will see: Details of the webapp2 framework How to authenticate users Storing data on Google Cloud Datastore Building HTML pages using templates Experimenting on the Notes application To better explore App Engine and Cloud Platform capabilities, we need a real-world application to experiment on; something that's not trivial to write, with a reasonable list of requirements. A good candidate is a note-taking application; we will name it Notes. Notes enable the users to add, remove, and modify a list of notes; a note has a title and a body of text. Users can only see their personal notes, so they must authenticate before using the application. The main page of the application will show the list of notes for logged-in users and a form to add new ones. The code from the helloworld example is a good starting point. We can simply change the name of the root folder and the application field in the app.yaml file to match the new name we chose for the application, or we can start a new project from scratch named notes. Authenticating users The first requirement for our Notes application is showing the home page only to users who are logged in and redirect others to the login form; the users service provided by App Engine is exactly what we need and adding it to our MainHandler class is quite simple: import webapp2 from google.appengine.api import users class MainHandler(webapp2.RequestHandler): def get(self): user = users.get_current_user() if user is not None: self.response.write('Hello Notes!') else: login_url = users.create_login_url(self.request.uri) self.redirect(login_url) app = webapp2.WSGIApplication([ ('/', MainHandler) ], debug=True) The user package we import on the second line of the previous code provides access to users' service functionalities. Inside the get() method of the MainHandler class, we first check whether the user visiting the page has logged in or not. If they have, the get_current_user() method returns an instance of the user class provided by App Engine and representing an authenticated user; otherwise, it returns None as output. If the user is valid, we provide the response as we did before; otherwise, we redirect them to the Google login form. The URL of the login form is returned using the create_login_url() method, and we call it, passing as a parameter the URL we want to redirect users to after a successful authentication. In this case, we want to redirect users to the same URL they are visiting, provided by webapp2 in the self.request.uri property. The webapp2 framework also provides handlers with a redirect() method we can use to conveniently set the right status and location properties of the response object so that the client browsers will be redirected to the login page. HTML templates with Jinja2 Web applications provide rich and complex HTML user interfaces, and Notes is no exception but, so far, response objects in our applications contained just small pieces of text. We could include HTML tags as strings in our Python modules and write them in the response body but we can imagine how easily it could become messy and hard to maintain the code. We need to completely separate the Python code from HTML pages and that's exactly what a template engine does. A template is a piece of HTML code living in its own file and possibly containing additional, special tags; with the help of a template engine, from the Python script, we can load this file, properly parse special tags, if any, and return valid HTML code in the response body. App Engine includes in the Python runtime a well-known template engine: the Jinja2 library. To make the Jinja2 library available to our application, we need to add this code to the app.yaml file under the libraries section: libraries: - name: webapp2 version: "2.5.2" - name: jinja2 version: latest We can put the HTML code for the main page in a file called main.html inside the application root. We start with a very simple page: <!DOCTYPE html> <html> <head lang="en"> <meta charset="UTF-8"> <title>Notes</title> </head> <body> <div class="container"> <h1>Welcome to Notes!</h1> <p> Hello, <b>{{user}}</b> - <a href="{{logout_url}}">Logout</a> </p> </div> </body> </html> Most of the content is static, which means that it will be rendered as standard HTML as we see it but there is a part that is dynamic and whose content depend on which data will be passed at runtime to the rendering process. This data is commonly referred to as template context. What has to be dynamic is the username of the current user and the link used to log out from the application. The HTML code contains two special elements written in the Jinja2 template syntax, {{user}} and {{logout_url}}, that will be substituted before the final output occurs. Back to the Python script; we need to add the code to initialize the template engine before the MainHandler class definition: import os import jinja2 jinja_env = jinja2.Environment( loader=jinja2.FileSystemLoader(os.path.dirname(__file__))) The environment instance stores engine configuration and global objects, and it's used to load templates instances; in our case, instances are loaded from HTML files on the filesystem in the same directory as the Python script. To load and render our template, we add the following code to the MainHandler.get() method: class MainHandler(webapp2.RequestHandler): def get(self): user = users.get_current_user() if user is not None: logout_url = users.create_logout_url(self.request.uri) template_context = { 'user': user.nickname(), 'logout_url': logout_url, } template = jinja_env.get_template('main.html') self.response.out.write( template.render(template_context)) else: login_url = users.create_login_url(self.request.uri) self.redirect(login_url) Similar to how we get the login URL, the create_logout_url() method provided by the user service returns the absolute URI to the logout procedure that we assign to the logout_url variable. We then create the template_context dictionary that contains the context values we want to pass to the template engine for the rendering process. We assign the nickname of the current user to the user key in the dictionary and the logout URL string to the logout_url key. The get_template() method from the jinja_env instance takes the name of the file that contains the HTML code and returns a Jinja2 template object. To obtain the final output, we call the render() method on the template object passing in the template_context dictionary whose values will be accessed, specifying their respective keys in the HTML file with the template syntax elements {{user}} and {{logout_url}}. Handling forms The main page of the application is supposed to list all the notes that belong to the current user but there isn't any way to create such notes at the moment. We need to display a web form on the main page so that users can submit details and create a note. To display a form to collect data and create notes, we put the following HTML code right below the username and the logout link in the main.html template file: {% if note_title %} <p>Title: {{note_title}}</p> <p>Content: {{note_content}}</p> {% endif %} <h4>Add a new note</h4> <form action="" method="post"> <div class="form-group"> <label for="title">Title:</label> <input type="text" id="title" name="title" /> </div> <div class="form-group"> <label for="content">Content:</label> <textarea id="content" name="content"></textarea> </div> <div class="form-group"> <button type="submit">Save note</button> </div> </form> Before showing the form, a message is displayed only when the template context contains a variable named note_title. To do this, we use an if statement, executed between the {% if note_title %} and {% endif %} delimiters; similar delimiters are used to perform for loops or assign values inside a template. The action property of the form tag is empty; this means that upon form submission, the browser will perform a POST request to the same URL, which in this case is the home page URL. As our WSGI application maps the home page to the MainHandler class, we need to add a method to this class so that it can handle POST requests: class MainHandler(webapp2.RequestHandler): def get(self): user = users.get_current_user() if user is not None: logout_url = users.create_logout_url(self.request.uri) template_context = { 'user': user.nickname(), 'logout_url': logout_url, } template = jinja_env.get_template('main.html') self.response.out.write( template.render(template_context)) else: login_url = users.create_login_url(self.request.uri) self.redirect(login_url) def post(self): user = users.get_current_user() if user is None: self.error(401) logout_url = users.create_logout_url(self.request.uri) template_context = { 'user': user.nickname(), 'logout_url': logout_url, 'note_title': self.request.get('title'), 'note_content': self.request.get('content'), } template = jinja_env.get_template('main.html') self.response.out.write( template.render(template_context)) When the form is submitted, the handler is invoked and the post() method is called. We first check whether a valid user is logged in; if not, we raise an HTTP 401: Unauthorized error without serving any content in the response body. Since the HTML template is the same served by the get() method, we still need to add the logout URL and the user name to the context. In this case, we also store the data coming from the HTML form in the context. To access the form data, we call the get() method on the self.request object. The last three lines are boilerplate code to load and render the home page template. We can move this code in a separate method to avoid duplication: def _render_template(self, template_name, context=None): if context is None: context = {} template = jinja_env.get_template(template_name) return template.render(context) In the handler class, we will then use something like this to output the template rendering result: self.response.out.write( self._render_template('main.html', template_context)) We can try to submit the form and check whether the note title and content are actually displayed above the form. Summary Thanks to App Engine, we have already implemented a rich set of features with a relatively small effort so far. We have discovered some more details about the webapp2 framework and its capabilities, implementing a nontrivial request handler. We have learned how to use the App Engine users service to provide users authentication. We have delved into some fundamental details of Datastore and now we know how to structure data in grouped entities and how to effectively retrieve data with ancestor queries. In addition, we have created an HTML user interface with the help of the Jinja2 template library, learning how to serve static content such as CSS files. Resources for Article: Further resources on this subject: Machine Learning in IPython with scikit-learn [Article] Introspecting Maya, Python, and PyMEL [Article] Driving Visual Analyses with Automobile Data (Python) [Article]

In this article by Peter De Tender, the author of Mastering Hyper-V, we will talk about the building blocks for creating your virtualization platform through Hyper-V. We need to clearly define a detailed list of required server hardware, storage hardware, physical and virtual machine operating systems, and anything else we need to be able to build our future virtualization platform. These components are known as the Hyper-V building blocks, and we describe each one of them in the following sections. (For more resources related to this topic, see here.) Physical server hardware One of the first important components when building a virtualization platform is the physical server hardware. One of the key elements to check is the Microsoft certified hardware and software supportability and compatibility list. This list gives a detailed overview of all tested and certified server brands, server types, and their corresponding configuration components. While it is not a requirement to use this kind of machine, we can only recommend it, based on our own experience. Imagine you have a performance issue with one of your applications running inside a VM, being hosted on non-supported hardware, using non-supported physical NICs, and you're not getting decent support from your IT partner or Microsoft on that specific platform, as the hardware is not supported. The landing page for this compatibility list is http://www.windowsservercatalog.com. After checking the compatibility of the server hardware and software, you need to find out which system resources are available for Hyper-V. The following table shows the maximum scaling possibilities for different components of the Hyper-V platform (the original source is Microsoft TechNet Library article at http://technet.microsoft.com/en-us/library/jj680093.aspx.) System Resource Maximum number   Windows 2008 R2 Windows Server 2012 (R2) Host Logical processors on hardware 64 320 Physical memory 1 TB 4 TB Virtual processors per host 512 1,024 Virtual machine Virtual processors per virtual machine 4 64 Memory per virtual machine 64 GB 1 TB Active virtual machines 384 1,024 Virtual disk size 2 TB 64 TB Cluster Nodes 16 64 Virtual machines 1,000 4,000 Physical storage hardware Next to the physical server component, another vital part of the virtualization environment is the storage hardware. In the Hyper-V platform, multiple kinds of storage are supported, that is DAS, NAS, and/or SAN: Direct Attached Storage (DAS): This is directly connected to the server (think of disk which is located inside the server chassis). Network Attached Storage (NAS): This is the storage provided via the network and presented to the Hyper-V server or virtual machines as file shares. This disk type is file-based access. Server 2012 and 2012 R2 make use of SMB 3.0 as file-sharing protocol, which allows us to use plain file shares as virtual machine storage location Storage Area Network (SAN): This is also network-based storage, but relies on block-based access. The volumes are presented as local disks to the host. Popular protocols within SAN environments are iSCSI and Fibre Channel. The key point of consideration when sizing your disk infrastructure is providing enough storage, at the best performance available, and preferably high availability as well. Depending on the virtual machine's required resources, the disk subsystem can be based on high-performant / expensive SSD disks (solid-state drives), performant / medium-priced SAS disks (serial attached SCSI), or slower but cheaper SATA (serial ATA) disks. Or it could even be a combination of all these types. Although a bit outside of Hyper-V as such, one technology that is configured and used a lot in combination with Hyper-V Server 2012 R2, is Storage Spaces. Storage Spaces is new as of Server 2012, and can be considered as a storage virtualization subsystem. Storage Spaces are disk volumes built on top of physical storage pools, which is in fact just a bunch of physical disks (JBOD). A very important point to note is that the aforementioned network-based SAN and NAS storage solutions cannot be a part of Storage Spaces, as it is only configurable for DAS storage. The following schema diagram provides a good overview of the Storage Spaces topology, possibilities, and features: Physical network devices It's easy to understand that your virtual platform is dependent on your physical network devices such as physical (core) switches and physical NICs in the Hyper-V hosts. When configuring Hyper-V, there are a few configurations to keep into consideration. NIC Teaming NIC Teaming is the configuration of multiple physical network interface cards into a single team, mainly used for high availability or higher bandwidth purposes. NIC Teaming as such is no technology of Hyper-V, but Hyper-V can make good use of this operating system feature. When configuring a NIC team, the physical network cards are bundled and presented to the host OS as one or more virtual network adapter(s). Within Hyper-V, two basic sets of algorithms exist where you can choose from during the configuration of Hyper-V networking: Switch-independent mode: In this configuration, the teaming is configured regardless of the switches to which the host is connected. The main advantage in this configuration is the fact the teaming can be configured to use multiple switches (for example, two NICs in the host are connected to switch 1 and 2 NICs are configured to use switch 2). Switch-dependent mode: In this configuration, the underlying switch is part of the teaming configuration; this automatically requires all NICs in the team to be connected to the same switch. NIC Teaming is managed through the Server Manager / NIC Teaming interface or by using PowerShell cmdlets. Depending on your server hardware and brand, the vendor might provide you with specific configuration software to achieve the same. For example, the HP Proliant series of servers allows for HP Team configuration, which is managed by using a specific HP Team tool. Network virtualization Within Hyper-V 2012 R2, network virtualization not only refers to the virtual networking connections that are used by the virtual machines but also refers to the technology that allows for true network isolation to the different networks in which virtual machines operate. This feature set is very important for hosting providers, who run different virtual machines for their customers in an isolated network. You have to make sure that there is no connection possible between the virtual machines from customer A and the virtual machines from customer B. That's exactly the main purpose of network virtualization. Another possible way of configuring network segmentation is by using VLANs. However, this also requires VLAN configuration to be done on the physical switches, where the described network virtualization completely runs inside the virtual network switch of Hyper-V. Server editions and licensing The last component that comprises the Hyper-V building blocks is the server editions and licensing of the physical and virtual machines operating system. Summary In this article, we looked at the various building blocks for building a virtualization platform using Hyper-V.

In this article by Sakis Kasampalis, author of the book Mastering Python Design Patterns, we will see a detailed description of the Chain of Responsibility design pattern with the help of a real-life example as well as a software example. Also, its use cases and implementation are discussed. (For more resources related to this topic, see here.) When developing an application, most of the time we know which method should satisfy a particular request in advance. However, this is not always the case. For example, we can think of any broadcast computer network, such as the original Ethernet implementation [j.mp/wikishared]. In broadcast computer networks, all requests are sent to all nodes (broadcast domains are excluded for simplicity), but only the nodes that are interested in a sent request process it. All computers that participate in a broadcast network are connected to each other using a common medium such as the cable that connects the three nodes in the following figure: If a node is not interested or does not know how to handle a request, it can perform the following actions: Ignore the request and do nothing Forward the request to the next node The way in which the node reacts to a request is an implementation detail. However, we can use the analogy of a broadcast computer network to understand what the chain of responsibility pattern is all about. The Chain of Responsibility pattern is used when we want to give a chance to multiple objects to satisfy a single request, or when we don't know which object (from a chain of objects) should process a specific request in advance. The principle is the same as the following: There is a chain (linked list, tree, or any other convenient data structure) of objects. We start by sending a request to the first object in the chain. The object decides whether it should satisfy the request or not. The object forwards the request to the next object. This procedure is repeated until we reach the end of the chain. At the application level, instead of talking about cables and network nodes, we can focus on objects and the flow of a request. The following figure, courtesy of a title="Scala for Machine Learning" www.sourcemaking.com [j.mp/smchain], shows how the client code sends a request to all processing elements (also known as nodes or handlers) of an application: Note that the client code only knows about the first processing element, instead of having references to all of them, and each processing element only knows about its immediate next neighbor (called the successor), not about every other processing element. This is usually a one-way relationship, which in programming terms means a singly linked list in contrast to a doubly linked list; a singly linked list does not allow navigation in both ways, while a doubly linked list allows that. This chain organization is used for a good reason. It achieves decoupling between the sender (client) and the receivers (processing elements) [GOF95, page 254]. A real-life example ATMs and, in general, any kind of machine that accepts/returns banknotes or coins (for example, a snack vending machine) use the chain of responsibility pattern. There is always a single slot for all banknotes, as shown in the following figure, courtesy of www.sourcemaking.com: When a banknote is dropped, it is routed to the appropriate receptacle. When it is returned, it is taken from the appropriate receptacle [j.mp/smchain], [j.mp/c2chain]. We can think of the single slot as the shared communication medium and the different receptacles as the processing elements. The result contains cash from one or more receptacles. For example, in the preceding figure, we see what happens when we request $175 from the ATM. A software example I tried to find some good examples of Python applications that use the Chain of Responsibility pattern but I couldn't, most likely because Python programmers don't use this name. So, my apologies, but I will use other programming languages as a reference. The servlet filters of Java are pieces of code that are executed before an HTTP request arrives at a target. When using servlet filters, there is a chain of filters. Each filter performs a different action (user authentication, logging, data compression, and so forth), and either forwards the request to the next filter until the chain is exhausted, or it breaks the flow if there is an error (for example, the authentication failed three consecutive times) [j.mp/soservl]. Apple's Cocoa and Cocoa Touch frameworks use Chain of Responsibility to handle events. When a view receives an event that it doesn't know how to handle, it forwards the event to its superview. This goes on until a view is capable of handling the event or the chain of views is exhausted [j.mp/chaincocoa]. Use cases By using the Chain of Responsibility pattern, we give a chance to a number of different objects to satisfy a specific request. This is useful when we don't know which object should satisfy a request in advance. An example is a purchase system. In purchase systems, there are many approval authorities. One approval authority might be able to approve orders up to a certain value, let's say $100. If the order is more than $100, the order is sent to the next approval authority in the chain that can approve orders up to $200, and so forth. Another case where Chain of Responsibility is useful is when we know that more than one object might need to process a single request. This is what happens in an event-based programming. A single event such as a left mouse click can be caught by more than one listener. It is important to note that the Chain of Responsibility pattern is not very useful if all the requests can be taken care of by a single processing element, unless we really don't know which element that is. The value of this pattern is the decoupling that it offers. Instead of having a many-to-many relationship between a client and all processing elements (and the same is true regarding the relationship between a processing element and all other processing elements), a client only needs to know how to communicate with the start (head) of the chain. The following figure demonstrates the difference between tight and loose coupling. The idea behind loosely coupled systems is to simplify maintenance and make it easier for us to understand how they function [j.mp/loosecoup]: Implementation There are many ways to implement Chain of Responsibility in Python, but my favorite implementation is the one by Vespe Savikko [j.mp/savviko]. Vespe's implementation uses dynamic dispatching in a Pythonic style to handle requests [j.mp/ddispatch]. Let's implement a simple event-based system using Vespe's implementation as a guide. The following is the UML class diagram of the system: The Event class describes an event. We'll keep it simple, so in our case an event has only name: class Event: def __init__(self, name): self.name = name def __str__(self): return self.name The Widget class is the core class of the application. The parent aggregation shown in the UML diagram indicates that each widget can have a reference to a parent object, which by convention, we assume is a Widget instance. Note, however, that according to the rules of inheritance, an instance of any of the subclasses of Widget (for example, an instance of MsgText) is also an instance of Widget. The default value of parent is None: class Widget: def __init__(self, parent=None): self.parent = parent The handle() method uses dynamic dispatching through hasattr() and getattr() to decide who is the handler of a specific request (event). If the widget that is asked to handle an event does not support it, there are two fallback mechanisms. If the widget has parent, then the handle() method of parent is executed. If the widget has no parent but a handle_default() method, handle_default() is executed: def handle(self, event): handler = 'handle_{}'.format(event) if hasattr(self, handler): method = getattr(self, handler) method(event) elif self.parent: self.parent.handle(event) elif hasattr(self, 'handle_default'): self.handle_default(event) At this point, you might have realized why the Widget and Event classes are only associated (no aggregation or composition relationships) in the UML class diagram. The association is used to show that the Widget class "knows" about the Event class but does not have any strict references to it, since an event needs to be passed only as a parameter to handle(). MainWIndow, MsgText, and SendDialog are all widgets with different behaviors. Not all these three widgets are expected to be able to handle the same events, and even if they can handle the same event, they might behave differently. MainWIndow can handle only the close and default events: class MainWindow(Widget): def handle_close(self, event): print('MainWindow: {}'.format(event)) def handle_default(self, event): print('MainWindow Default: {}'.format(event)) SendDialog can handle only the paint event: class SendDialog(Widget): def handle_paint(self, event): print('SendDialog: {}'.format(event)) Finally, MsgText can handle only the down event: class MsgText(Widget): def handle_down(self, event): print('MsgText: {}'.format(event)) The main() function shows how we can create a few widgets and events, and how the widgets react to those events. All events are sent to all the widgets. Note the parent relationship of each widget. The sd object (an instance of SendDialog) has as its parent the mw object (an instance of MainWindow). However, not all objects need to have a parent that is an instance of MainWindow. For example, the msg object (an instance of MsgText) has the sd object as a parent: def main(): mw = MainWindow() sd = SendDialog(mw) msg = MsgText(sd) for e in ('down', 'paint', 'unhandled', 'close'): evt = Event(e) print('nSending event -{}- to MainWindow'.format(evt)) mw.handle(evt) print('Sending event -{}- to SendDialog'.format(evt)) sd.handle(evt) print('Sending event -{}- to MsgText'.format(evt)) msg.handle(evt) The following is the full code of the example (chain.py): class Event: def __init__(self, name): self.name = name def __str__(self): return self.name class Widget: def __init__(self, parent=None): self.parent = parent def handle(self, event): handler = 'handle_{}'.format(event) if hasattr(self, handler): method = getattr(self, handler) method(event) elif self.parent: self.parent.handle(event) elif hasattr(self, 'handle_default'): self.handle_default(event) class MainWindow(Widget): def handle_close(self, event): print('MainWindow: {}'.format(event)) def handle_default(self, event): print('MainWindow Default: {}'.format(event)) class SendDialog(Widget): def handle_paint(self, event): print('SendDialog: {}'.format(event)) class MsgText(Widget): def handle_down(self, event): print('MsgText: {}'.format(event)) def main(): mw = MainWindow() sd = SendDialog(mw) msg = MsgText(sd) for e in ('down', 'paint', 'unhandled', 'close'): evt = Event(e) print('nSending event -{}- to MainWindow'.format(evt)) mw.handle(evt) print('Sending event -{}- to SendDialog'.format(evt)) sd.handle(evt) print('Sending event -{}- to MsgText'.format(evt)) msg.handle(evt) if __name__ == '__main__': main() Executing chain.py gives us the following results: >>> python3 chain.py Sending event -down- to MainWindow MainWindow Default: down Sending event -down- to SendDialog MainWindow Default: down Sending event -down- to MsgText MsgText: down Sending event -paint- to MainWindow MainWindow Default: paint Sending event -paint- to SendDialog SendDialog: paint Sending event -paint- to MsgText SendDialog: paint Sending event -unhandled- to MainWindow MainWindow Default: unhandled Sending event -unhandled- to SendDialog MainWindow Default: unhandled Sending event -unhandled- to MsgText MainWindow Default: unhandled Sending event -close- to MainWindow MainWindow: close Sending event -close- to SendDialog MainWindow: close Sending event -close- to MsgText MainWindow: close There are some interesting things that we can see in the output. For instance, sending a down event to MainWindow ends up being handled by the default MainWindow handler. Another nice case is that although a close event cannot be handled directly by SendDialog and MsgText, all the close events end up being handled properly by MainWindow. That's the beauty of using the parent relationship as a fallback mechanism. If you want to spend some more creative time on the event example, you can replace the dumb print statements and add some actual behavior to the listed events. Of course, you are not limited to the listed events. Just add your favorite event and make it do something useful! Another exercise is to add a MsgText instance during runtime that has MainWindow as the parent. Is this hard? Do the same for an event (add a new event to an existing widget). Which is harder? Summary In this article, we covered the Chain of Responsibility design pattern. This pattern is useful to model requests / handle events when the number and type of handlers isn't known in advance. Examples of systems that fit well with Chain of Responsibility are event-based systems, purchase systems, and shipping systems. In the Chain Of Responsibility pattern, the sender has direct access to the first node of a chain. If the request cannot be satisfied by the first node, it forwards to the next node. This continues until either the request is satisfied by a node or the whole chain is traversed. This design is used to achieve loose coupling between the sender and the receiver(s). ATMs are an example of Chain Of Responsibility. The single slot that is used for all banknotes can be considered the head of the chain. From here, depending on the transaction, one or more receptacles is used to process the transaction. The receptacles can be considered the processing elements of the chain. Java's servlet filters use the Chain of Responsibility pattern to perform different actions (for example, compression and authentication) on an HTTP request. Apple's Cocoa frameworks use the same pattern to handle events such as button presses and finger gestures. Resources for Article: Further resources on this subject: Exploring Model View Controller [Article] Analyzing a Complex Dataset [Article] Automating Your System Administration and Deployment Tasks Over SSH [Article]

In this article by Nick Ramirez, author of the book WiX Cookbook, we will see how we tackle the trouble of getting any bit of code from development to production. WiX solves this problem for its own code by allowing it to be built using a variety of workflows. As part of the WiX toolset, we get the compiler and linker needed to create an MSI installer. If we're using Visual Studio then we also get project templates that use these tools on our behalf so that the entire build process is effortless. If we're trying to fit WiX into an automated deployment pipeline, we can either call the compiler and linker from the command line or use ready-made MSBuild tasks. (For more resources related to this topic, see here.) Installing WiX and creating a new project in Visual Studio 2013 It's possible to work with WiX outside of Visual Studio, but within it, you'll benefit from the project templates; IntelliSense and shortcuts to the compiler and linker settings are available on the project's properties. The only downside is that WiX doesn't work with Visual Studio Express. However, its installer will give you the compiler and linker so that you can still get work done even if you're using Notepad to write the markup. SharpDevelop, a free and open source IDE, also supports WiX projects. Getting WiX up and running starts with downloading and running its installer. This is a one-stop shop to update Visual Studio, getting the compiler and linker as well as other utilities to work with MSI packages. WiX supports Visual Studio 2005 and later, including Visual Studio 2013, which we'll cover here. In this recipe, we will download and install WiX and create our first setup project. Getting ready To prepare for this recipe, install Visual Studio 2013 and close it before installing WiX. How to do it… Download and install the WiX toolset to get access to new project templates, IntelliSense, and project properties in Visual Studio. The following steps will guide you: Open a browser, navigate to http://www.wixtoolset.org, and follow the link to the downloads page: Once downloaded, launch the WiX installer and click on Install: After completing the installation, open Visual Studio and go to File | New | Project | Windows Installer XML. Select the Setup Project template from the list of available project types. The version of .NET that's displayed has no bearing on the project since it's comprised of XML mark-up and not .NET code. Give the project a name and click on OK: The project will initially include a file named Product.wxs, which contains the skeleton markup you'll need to create an installer: <?xml version="1.0" encoding="UTF-8"?> <Wix > <Product Id="*" Name="My Software" Language="1033" Version="1.0.0.0" Manufacturer="My Company" UpgradeCode="889e2707-5235-4d97-b178-cf0cb55d8ab8"> <Package InstallerVersion="200" Compressed="yes" InstallScope="perMachine" /> <MajorUpgrade DowngradeErrorMessage="A newer version of [ProductName] is already installed." /> <MediaTemplate /> <Feature Id="ProductFeature" Title="MyFirstWixProject" Level="1"> <ComponentGroupRef Id="ProductComponents" /> </Feature> </Product> <Fragment> <Directory Id="TARGETDIR" Name="SourceDir"> <Directory Id="ProgramFilesFolder"> <Directory Id="INSTALLFOLDER" Name="My Software" /> </Directory> </Directory> </Fragment> <Fragment> <ComponentGroup Id="ProductComponents" Directory="INSTALLFOLDER"> <!-- TODO: Remove the comments around this Component elementand the ComponentRef below in order to add resourcesto this installer. --> <!-- <Component Id="ProductComponent"> --> <!-- TODO: Insert files, registry keys, and other resources here. --> <!-- </Component> --> </ComponentGroup> </Fragment> </Wix> How it works… The WiX team has always worked quickly to keep up with the latest versions of Visual Studio. For example, WiX 3.9 supports Visual Studio 2013. When we launched the installer, it checked which versions of Visual Studio were present and registered its project templates with all that were compatible. Behind the scenes, WiX introduces a new project type that has a .wixproj file extension. This project file contains MSBuild markup, which points to the WiX compiler and linker. Other IDEs, such as SharpDevelop, can take advantage of these project files to build MSI packages too. The Product.wxs file contains everything we need to get started with writing WiX markup. The best coding practices for how to structure a WiX file have been defaulted for you. For example, the Directory elements are separated into a Fragment element so that directories are decoupled from the files that will go into them. A ComponentGroup has been set up with a comment guiding you to add Component elements to it. Each version of WiX brings a better Product.wxs file with it. There's more… If you were curious about what effect changing the version of the .NET framework listed in the drop-down list at the top of the New Project window would have, the answer, at least for setup projects, is nothing at all. A WiX file contains XML and is compiled with a specialized WiX compiler, so the version of .NET that we select will ultimately be ignored. That's not to say that it doesn't make a difference for any of the other project types. For example, C# Custom Action Project will have a dependency on the version of .NET that's selected. Anyone who uses the installer that in turn uses that custom action will need to have that version of .NET installed. Referencing the output of a .NET console application in a WiX project by using a preprocessor variable After setting up our WiX project, the first thing we'll probably want to do is package up the files that we plan to install. Since we're working in Visual Studio, we'll likely want to include the output of other projects such as the .exe file that's created from a console application project. At first, we could try hardcoding the path to the file: <Component Id="cmpMyConsoleAppEXE" Guid="{882DB6AA-1363-4724-8C43-2950E7ABECD4}"> <File Source="..MyConsoleAppbinDebugMyConsoleApp.exe" /> </Component> Although this works, it's a bit brittle and will break if the path to the file changes. Instead, we can use a preprocessor variable to store the path and allow Visual Studio to keep it up-to-date through the power of project references. In this recipe, we'll reference a console application's output and use a preprocessor variable to include that output in our installer. Getting ready To prepare for this recipe, create a new WiX setup project and name it ConsoleAppInstaller. How to do it… Use a preprocessor variable to get the path to a project's output with the following steps: Add a new C# console application to the same solution as the ConsoleAppInstaller setup project by right-clicking on the solution in Solution Explorer, going to Add | New Project… | Visual C# | Console Application and naming it TestApplication. The name matters as we'll be referencing it later: Within the setup project, add a reference to TestApplication by right-clicking on the References node in Solution Explorer, choosing Add Reference..., and finding TestApplication under the Projects tab. Click on Add and then on OK: Within the setup project, open Product.wxs and replace the ComponentGroup markup inside the last fragment with the following code: <ComponentGroup Id="ProductComponents" Directory="INSTALLFOLDER"> <Component Id="cmpTestApplicationEXE" Guid="{6E2A6370-4784-4CF3-B42B-AA2D29EA5B1B}"> <File Source="$(var.TestApplication.TargetDir)TestApplication.exe" /> </Component> </ComponentGroup> Build the project and TestApplication.exe will be included in the MSI file. Note that you must set the EmbedCab attribute on the MediaTemplate element to yes to include the CAB file that WiX creates, which is where our .exe file is stored, inside the MSI. Also, this example assumes that TestApplication.exe is the only file you'd like to include in the installer. Other files, such as DLLs, can be included in the same way though. How it works… When we referenced the C# console application within the WiX setup project, the preprocessor variable $(var.[ProjectName].TargetDir) was made available to us, where ProjectName in this case is TestApplication. TargetDir points to the output directory of the console application project where our compiled TestApplication.exe file can be found. Other preprocessor variables are also made available. For example, $(var.[ProjectName].TargetFileName) gives you the name of the compiled application, which for us would be TestApplication.exe. A full list of these variables can be found at http://wixtoolset.org/documentation/manual/v3/votive/votive_project_references.html. Another benefit of referencing the console application project in this way is that it ensures it is compiled before our setup project is. This way, our installer always includes the most up-to-date version of the application. The GUID used for the Guid attribute on the Component element in this example can be any GUID, not just the one listed. You can generate a new one in Visual Studio by navigating to Tools | Create GUID. Use Registry Format as the GUID's format. More information can be found at http://wixtoolset.org/documentation/manual/v3/howtos/general/generate_guids.html. You can also set the Guid attribute to an asterisk (*) or omit it altogether and WiX will set the GUID for you. You should choose your own if you plan on authoring a patch file for the application in the future or if the contents of Component don't contain an element that can be marked as a KeyPath element. Separating a portion of WIX markup into its own library As a project grows in complexity and size, we may end up with different teams building different parts of the software in relative isolation. Each team may want to control how their module will be installed or, during development, install only the modules that their code depends upon into their dev environment. To handle these scenarios, we can split our installer into chunks of WiX code called setup libraries. A setup library can be compiled independently and plugged into the main, monolithic setup project later. We can also include the library in a team-owned setup project that only contains the modules required by the team. In essence, we can mix and match libraries wherever we need them to create installers for different purposes. You might also want to share some complex installer markup, such as a user interface, with other installers, and a library is the perfect way to do this. Although it's outside the scope of this article, setup libraries are also used when building custom WiX extensions. In this recipe, we'll see how to create a setup library and include it in our setup project. Getting ready To prepare for this recipe, create a setup project and call it SetupLibraryInstaller. How to do it… Add a setup library to the solution and reference it in a setup project. The following steps show how to do this: Add a new setup library to the same solution as the setup project by right-clicking on the solution in Solution Explorer and navigating to Add | New Project... | Windows Installer XML | Setup Library Project. For this example, name the project MySetupLibrary: After it's created, right-click on the MySetupLibrary project in Solution Explorer and go to Add | New Item… | Text File. Name the text file SampleTextFile.txt and click on Add. Our library will install this single text file. Right-click on the MySetupLibrary project in Solution Explorer again and select Properties. Select the Tool Settings tab and add -bf, which stands for bind files, to the librarian textbox, as shown in the following screenshot: Open Library.wxs and replace the existing markup with the following: <?xml version="1.0" encoding="UTF-8"?> <Wix > <Fragment> <DirectoryRef Id="INSTALLFOLDER"> <Directory Id="SampleComponentsDirectory" Name="Sample Components" /> </DirectoryRef> <ComponentGroup Id="SampleComponentGroup" Directory="SampleComponentsDirectory"> <Component Id="cmpSampleTextFileTXT" Guid="{5382BC02-4484-4C9B-8734-A99D20632EA9}"> <File Source="SampleTextFile.txt" /> </Component> </ComponentGroup> <Feature Id="SampleFeature"> <ComponentGroupRef Id="SampleComponentGroup" /> </Feature> </Fragment> </Wix> In the SetupLibraryInstaller project, add a reference to the setup library by right-clicking on the References node in Solution Explorer and selecting Add Reference…. Click on the Projects tab, highlight MySetupLibrary, click on Add, and then on OK. Open Product.wxs and add a FeatureRef element with an ID of SampleFeature. This includes the feature we added to the Library.wxs file of SetupLibrary in our installer. FeatureRef can go after the existing Feature element as follows: <Feature Id="ProductFeature" Title="ConsoleAppInstaller" Level="1"> <ComponentGroupRef Id="ProductComponents" /> </Feature> <FeatureRef Id="SampleFeature"/> How it works… Our setup library contains WiX markup to install a single text file called SampleTextFile.txt. Ordinarily, when you build a library like this, the source files don't get stored within it. Instead, only the WiX markup is compiled without any of the source files it refers to. In that case, we would have had to copy SampleTextFile.txt to the setup project's directory too, so that it can be found at link-time when compiling the installer. However, because we added the -bf flag, which stands for bind files, to the Librarian settings, the text file was serialized and stored within the library. The -bf flag will handle serializing and storing any type of file including executables, images, and other binary data. Setup libraries are compiled into files with a .wixlib extension. The markup we added to the library created a component, directory, and feature for the text file. To integrate the new directory with the existing directory structure as defined by our setup project, we chose to reference INSTALLFOLDER with a DirectoryRef element. Just be sure that there's a corresponding Directory element in your setup project that has this name. At link time, the DirectoryRef element in the library is merged with the Directory element in the setup project by matching their IDs. Once we had this, we were able to add a new subdirectory within the INSTALLFOLDER directory called Sample Components. After installation, we can see that the new directory was created and it contains our text file: To be sure that our library gets compiled before our setup project, we referenced it within the setup project using the References node. Then, to create a link to the library, we included a FeatureRef element in Product.wxs, which had an ID matching the Feature defined in the library. This pulls the Feature with all of its components into the installer. There's more… The setup libraries might contain more than just components, features, and directories. For example, they might define markup for a user interface using a UI element, which could then be linked to our installer with a UIRef element. Basically, if you can find a corresponding *Ref element, such as DirectoryRef, UIRef, ComponentGroupRef, or FeatureRef, then you'll be able to separate that type of element into a library and use its *Ref element to link it to the setup project. Even if you can't find a corresponding *Ref element, as long as you have a reference of some kind, such as Property and PropertyRef, the rest of the elements in the library will be carried along with it into the installer. So, at the very least, you could include a single Property in the library and use that as the link between the library elements and the installer. Compiling a WiX installer on a build machine using MSBuild The WiX Toolset places its compiler and linker in C:Program Files (x86)WiX Toolset v3.9bin. This is fine when compiling on your own machine but becomes a concern when you'd like to share your project with others or have it compile on a build server. WiX will have to be installed on each computer that builds the project. Alternatively, we can store the WiX tools in source control, and then whoever needs to build a setup project can get everything they need by cloning the repository. This will also help us keep a handle on which version of WiX we're compiling against on a project-by-project basis. In this recipe, we'll store the WiX binaries in a fictitious source control directory on the C: drive. We'll then update the .wixproj file of a setup project to use the MSBuild tasks stored there. I will be using a server with the Windows Server 2012 R2 operating system installed on it. You should be able to follow along with other versions of Windows Server. Getting ready To prepare for this recipe, perform the following steps: Install the .NET Framework 3.5. It's needed by the WiX build tasks. In Windows Server 2012 R2, it can be installed as a feature within Server Manager: Next, we'll need the MSBuild engine, which is part of Microsoft Build Tools. It can be downloaded from http://www.microsoft.com/en-us/download/details.aspx?id=40760. After installing MSBuild, add its installation directory to the computer's PATH environment variable. Get there by right-clicking on This PC in file explorer and then going to Properties | Advanced system settings | Environment Variables.... Scroll through the list of system variables until you find the one labeled Path. Highlight it, click on Edit..., and then add the path to the MSBuild directory into the Variable value field, preceded by a semicolon. Then, click on OK: How to do it… Download the WiX binaries and update your setup project to use the included MSBuild tasks: Open a browser, navigate to http://www.wixtoolset.org, and follow the link to the downloads page. Download wix39-binaries.zip: Make sure that the ZIP file is unblocked by right-clicking on it, choosing Properties, clicking on Unblock (if you don't see it, just continue to the next step), and then on OK. Extract the contents of the ZIP file to C:SourceControlWiX39. Perform this step on both the server and on your own development computer so that our WiX projects can be built in both places using the MSBuild tasks from this folder (note that in a real-world scenario, our source control system would be responsible for copying the binaries to each computer): We will build a simple setup project to confirm that we've got everything on the server configured correctly. Create a setup project on your development machine and call it BuildMachineInstaller. Open the BuildMachineInstaller.wixproj file and add the WixToolPath, WixTargetsPath, and WixTasksPath properties as shown, making sure that the value of WixToolPath ends in a backslash: <PropertyGroup> <Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration> <Platform Condition=" '$(Platform)' == '' ">x86</Platform> <ProductVersion>3.9</ProductVersion> <ProjectGuid>f80ca9fc-8e42-406e-92f9-06e484e94d67</ProjectGuid> <SchemaVersion>2.0</SchemaVersion> <OutputName>BuildMachineInstaller</OutputName> <OutputType>Package</OutputType> <WixToolPath>C:SourceControlWiX39</WixToolPath> <WixTargetsPath>$(WixToolPath)wix.targets</WixTargetsPath> <WixTasksPath>$(WixToolPath)WixTasks.dll</WixTasksPath> <WixTargetsPath Condition=" '$(WixTargetsPath)' == '' AND'$(MSBuildExtensionsPath32)' != '' ">$(MSBuildExtensionsPath32)MicrosoftWiXv3.xWix.targets</WixTargetsPath> <WixTargetsPath Condition=" '$(WixTargetsPath)' == ''">$(MSBuildExtensionsPath)MicrosoftWiXv3.xWix.targets</WixTargetsPath> </PropertyGroup> Copy the BuildMachineInstaller solution folder and all of its subfolders to C:SourceControl on the build server. Open a command prompt via Run | cmd, execute the following commands to change the directory to the BuildMachineInstaller folder and compile the solution using MSBuild: cd C:SourceControlBuildMachineInstaller msbuild BuildMachineInstaller.sln How it works… We started with a blank slate of a freshly installed Windows Server 2012 R2 operating system. Therefore, we had to install all the required software including .NET Framework 3.5 and Microsoft Build Tools 2013. The latter gives us the MSBuild engine, whose path we included in the computer's PATH environment variable. Next, we downloaded the WiX binaries and copied them to C:SourceControl. With a source control system, these files could be shared among all computers that need to compile our setup projects. We also had to update our project's .wixproj file so that it knew where to find these WiX binaries. This is accomplished by adding three MSBuild properties: WixToolPath, WixTargetsPath, and WixTasksPath. The first property sets the path to the WiX binaries, the second to the wix.targets file, and the third to WixTasks.dll. With all of this setup out of the way, we opened a command prompt, navigated to the folder where our solution file was on the build server, and compiled it using MSBuild. Building a WiX installer from the command line WiX has excellent integration with Visual Studio, but that shouldn't stop you from using it in other IDEs. We ought to be able to create an installer using only Notepad and the WiX compiler and linker if we wanted to. Luckily, WiX gives us the freedom to do this. In this recipe, we'll write a simple .wxs file and compile it into an MSI package using Candle, which is the WiX compiler, and Light, which is the WiX linker. Getting ready To prepare for this recipe, perform the following steps: Using a text editor such as Notepad, create a file called Product.wxs and add the following markup to it: <?xml version="1.0" encoding="UTF-8"?> <Wix > <Product Id="*" Name="My Software" Language="1033" Manufacturer="My Company" Version="1.0.0.0" UpgradeCode="8c7d85db-b0d1-4a9a-85ea-130836aeef67"> <Package InstallerVersion="200" Compressed="yes" InstallScope="perMachine" /> <MajorUpgrade DowngradeErrorMessage="A newer version of [ProductName] is already installed." /> <MediaTemplate EmbedCab="yes" /> <Feature Id="ProductFeature" Title="The main feature" Level="1"> <ComponentGroupRef Id="ProductComponents" /> </Feature> </Product> <Fragment> <Directory Id="TARGETDIR" Name="SourceDir"> <Directory Id="ProgramFilesFolder"> <Directory Id="INSTALLFOLDER" Name="My Software" /> </Directory> </Directory> </Fragment> <Fragment> <ComponentGroup Id="ProductComponents" Directory="INSTALLFOLDER"> <Component Id="cmpMyTextFileTXT" Guid="{A4540658-09B6-46DA-8880-0B1962E06642}"> <File Source="MyTextFile.txt" /> </Component> </ComponentGroup> </Fragment> </Wix> This installs a text file called MyTextFile.txt. So, add a text file with this name to the same directory as Product.wxs. We will compile the two files from the command line to create an installer. How to do it… Open a command prompt and use candle.exe and light.exe to compile and link our WiX source file: Open a command prompt by navigating to Run | cmd. Change the directory to where the Product.wxs and MyTextFile.txt files are using the following command line: cd C:MyProject Use Candle to compile the .wxs file into a .wixobj file and then place it in an output folder called obj. Be sure to surround the path to Candle, %WIX%bincandle, with quotes since it will contain spaces when it is expanded: "%WIX%bincandle" *.wxs -o obj Use Light to link the text file and the .wixobj file together to form an MSI: "%WIX%binlight" obj*.wixobj -o binCommandLineInstaller.msi How it works… When we installed the WiX toolset, it gave us the WiX compiler, which is candle.exe, and linker, which is light.exe. These are the only tools we need to create an MSI from our WiX source file, Product.wxs. From the command line, we navigated to the directory where our source file was and then used Candle and Light to compile and link the file to create an MSI installer. The first argument we passed to Candle was *.wxs. This selects all the .wxs files in the current directory and includes them in the compilation. Next, the -o argument tells Candle where to send the output of the compilation step. In this case, we sent it to a directory called obj. Note that the directory name ends in a backslash so that Candle knows that it's a directory. If it didn't exist before, it will be created. The output of the Candle command was a file called Product.wixobj. This was an intermediate file that was picked up by light.exe in the next step. The first argument we passed to Light was the location of the .wixobj files: obj*.wixobj. By using an asterisk, we select all the .wixobj files in the obj directory. The -o argument tells Light where to create the MSI file and what to name it. In this case, we create a file called CommandLineInstaller.msi. Another file called CommandLineInstaller.wixpdb was also created. This can be used when building patch files. You can learn more by reading Peter Marcu's blog post WiX: Introducing the WixPdb at http://petermarcu.blogspot.com/2008/02/wix-introducing-wixpdb.html. There are a number of arguments that can be passed to Candle and Light that you might want to get to know. Passing the -? flag to either will give you a list of all the available options: "%WIX%bincandle" -? "%WIX%binlight" -? We used the %WIX% system environment variable to resolve the path to the WiX bin directory, where candle.exe and light.exe are present. This variable is added when you install the WiX toolset and resolves to C:Program Files (x86)WiX Toolset v3.9. It will not be present if you are using the WiX binaries directly without installing the WiX toolset. Summary This article helped you to build your WiX projects right, whether that means from Visual Studio, the command line, or on a build server with automation. We also saw how to reference the output of other projects that were included in the installer and how to separate WiX markup into libraries. Resources for Article: Further resources on this subject: Windows Installer XML (WiX): Adding a User Interface [Article] Getting Started with Windows Installer XML (WiX) [Article] Windows Phone 8 Applications [Article]

In this article by Suryakumar Balakrishnan Nair and Andreas Oehlke, authors of Learning LibGDX Game Development, Second Edition, you will learn how to load a model and create a basic 3D scene. In a game, we need an actual model exported from Blender or any other 3D animation software. (For more resources related to this topic, see here.) Loading a model Copy these three files to the assets folder of the android project: car.g3dj: This is the model file to be used in our example tiretext.jpg and yellowtaxi.jpg: These are the materials for the model Replacing the ModelBuilder class in our ModelTest.java file, we add the following code: assets = new AssetManager(); assets.load("car.g3dj", Model.class); assets.finishLoading(); model = assets.get("car.g3dj", Model.class); instance = new ModelInstance(model); Additionally, a camera input controller is also added to inspect the model from various angles as follows: camController = new CameraInputController(cam); Gdx.input.setInputProcessor(camController); camController.update(); This camera input controller will be updated on each render() by calling camController.update(). The completed MyModelTest.java is as follows: public class MyModelTest extends ApplicationAdapter { public Environment environment; public PerspectiveCamera cam; public CameraInputController camController; public ModelBatch modelBatch; public Model model; public ModelInstance instance; public AssetManager assets ; @Override public void create() { environment = new Environment(); environment.set(new ColorAttribute(ColorAttribute.AmbientLight, 0.4f, 0.4f, 0.4f, 1f)); environment.add(new DirectionalLight().set(0.8f, 0.8f, 0.8f, -1f, -0.8f, -0.2f)); modelBatch = new ModelBatch(); cam = new PerspectiveCamera(67, Gdx.graphics.getWidth(), Gdx.graphics.getHeight()); cam.position.set(1,1,1); cam.lookAt(0, 0, 0); cam.near = 1f; cam.far = 300f; cam.update(); assets = new AssetManager(); assets.load("car.g3dj", Model.class); assets.finishLoading(); model = assets.get("car.g3dj", Model.class); instance = new ModelInstance(model); camController = new CameraInputController(cam); Gdx.input.setInputProcessor(camController); } @Override public void render() { camController.update(); Gdx.gl.glViewport(0, 0, Gdx.graphics.getWidth(), Gdx.graphics.getHeight()); Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT | GL20.GL_DEPTH_BUFFER_BIT); modelBatch.begin(cam); modelBatch.render(instance, environment); modelBatch.end(); } @Override public void dispose() { modelBatch.dispose(); assets.dispose() ; } } The new additions are highlighted. The following is a screenshot of the render scene. Use the W , S , A , D keys and mouse to navigate through the scene. Model formats and the FBX converter LibGDX supports three model formats, namely Wavefront OBJ, G3DJ, and G3DB. Wavefront OBJ models are intended for testing purposes only because this format does not include enough information for complex models. You can export your 3D model as .obj from any 3D animation or modeling software, however LibGDX does not fully support .obj, hence, if you use your own .obj model, then it might not render correctly. The G3DJ is a JSON textual format supported by LibGDX and can be used for debugging, whereas the G3DB is a binary format and is faster to load. One of the most popular model formats supported by any modeling software is FBX. LibGDX provides a tool called FBX converter to convert formats such as .obj and .fbx into the LibGDX supported formats .g3dj and .g3db. To convert car.fbx to a .g3db format, open the command line and call fbx-conv-win32, as shown in the following screenshot: Make sure that the fbx-conv-win32.exe file is in the same folder as car.fbx. Otherwise, you will have to use the full path of the source file to convert. To find out more about FBX converter visit https://github.com/libgdx/fbx-conv and https://github.com/libgdx/libgdx/wiki/3D-animations-and-skinning. Also, you can download FBX converter from http://libgdx.badlogicgames.com/fbx-conv. Creating a basic 3D scene Create a simple scene with a ball and ground, as shown in the following screenshot: Add the following code to MyCollisionTest.java: package com.packtpub.libgdx.collisiontest; import com.badlogic.gdx.ApplicationAdapter; import com.badlogic.gdx.Gdx; ... import com.badlogic.gdx.utils.Array; public class MyCollisionTest extends ApplicationAdapter { PerspectiveCamera cam; ModelBatch modelBatch; Array<Model> models; ModelInstance groundInstance; ModelInstance sphereInstance; Environment environment; ModelBuilder modelbuilder; @Override public void create() { modelBatch = new ModelBatch(); environment = new Environment(); environment.set(new ColorAttribute(ColorAttribute.AmbientLight, 0.4f, 0.4f, 0.4f, 1f)); environment.add(new DirectionalLight().set(0.8f, 0.8f, 0.8f, -1f, -0.8f, -0.2f)); cam = new PerspectiveCamera(67, Gdx.graphics.getWidth(), Gdx.graphics.getHeight()); cam.position.set(0, 10, -20); cam.lookAt(0, 0, 0); cam.update(); models = new Array<Model>(); modelbuilder = new ModelBuilder(); // creating a ground model using box shape float groundWidth = 40; modelbuilder.begin(); MeshPartBuilder mpb = modelbuilder.part("parts", GL20.GL_TRIANGLES, Usage.Position | Usage.Normal | Usage.Color, new Material(ColorAttribute.createDiffuse(Color.WHITE))); mpb.setColor(1f, 1f, 1f, 1f); mpb.box(0, 0, 0, groundWidth, 1, groundWidth); Model model = modelbuilder.end(); models.add(model); groundInstance = new ModelInstance(model); // creating a sphere model float radius = 2f; final Model sphereModel = modelbuilder.createSphere(radius, radius, radius, 20, 20, new Material(ColorAttribute.createDiffuse(Color.RED), ColorAttribute.createSpecular(Color.GRAY), FloatAttribute.createShininess(64f)), Usage.Position | Usage.Normal); models.add(sphereModel); sphereInstance = new ModelInstance(sphereModel); sphereinstance.transform.trn(0, 10, 0); } public void render() { Gdx.gl.glViewport(0, 0, Gdx.graphics.getWidth(), Gdx.graphics.getHeight()); Gdx.gl.glClearColor(0, 0, 0, 1); Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT | GL20.GL_DEPTH_BUFFER_BIT); modelBatch.begin(cam); modelBatch.render(groundInstance, environment); modelBatch.render(sphereInstance, environment); modelBatch.end(); } @Override public void dispose() { modelBatch.dispose(); for (Model model : models) model.dispose(); } } The ground is actually a thin box created using ModelBuilder just like the sphere. Now that we have created a simple 3D scene, let's add some physics using the following code: public class MyCollisionTest extends ApplicationAdapter { ... private btDefaultCollisionConfiguration collisionConfiguration; private btCollisionDispatcher dispatcher; private btDbvtBroadphase broadphase; private btSequentialImpulseConstraintSolver solver; private btDiscreteDynamicsWorld world; private Array<btCollisionShape> shapes = new Array<btCollisionShape>(); private Array<btRigidBodyConstructionInfo> bodyInfos = new Array<btRigidBody.btRigidBodyConstructionInfo>(); private Array<btRigidBody> bodies = new Array<btRigidBody>(); private btDefaultMotionState sphereMotionState; @Override public void create() { ... // Initiating Bullet Physics Bullet.init(); //setting up the world collisionConfiguration = new btDefaultCollisionConfiguration(); dispatcher = new btCollisionDispatcher(collisionConfiguration); broadphase = new btDbvtBroadphase(); solver = new btSequentialImpulseConstraintSolver(); world = new btDiscreteDynamicsWorld(dispatcher, broadphase, solver, collisionConfiguration); world.setGravity(new Vector3(0, -9.81f, 1f)); // creating ground body btCollisionShape groundshape = new btBoxShape(new Vector3(20, 1 / 2f, 20)); shapes.add(groundshape); btRigidBodyConstructionInfo bodyInfo = new btRigidBodyConstructionInfo(0, null, groundshape, Vector3.Zero); this.bodyInfos.add(bodyInfo); btRigidBody body = new btRigidBody(bodyInfo); bodies.add(body); world.addRigidBody(body); // creating sphere body sphereMotionState = new btDefaultMotionState(sphereInstance.transform); sphereMotionState.setWorldTransform(sphereInstance.transform); final btCollisionShape sphereShape = new btSphereShape(1f); shapes.add(sphereShape); bodyInfo = new btRigidBodyConstructionInfo(1, sphereMotionState, sphereShape, new Vector3(1, 1, 1)); this.bodyInfos.add(bodyInfo); body = new btRigidBody(bodyInfo); bodies.add(body); world.addRigidBody(body); } public void render() { Gdx.gl.glViewport(0, 0, Gdx.graphics.getWidth(), Gdx.graphics.getHeight()); Gdx.gl.glClearColor(0, 0, 0, 1); Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT | GL20.GL_DEPTH_BUFFER_BIT); world.stepSimulation(Gdx.graphics.getDeltaTime(), 5); sphereMotionState.getWorldTransform(sphereInstance.transform); modelBatch.begin(cam); modelBatch.render(groundInstance, environment); modelBatch.render(sphereInstance, environment); modelBatch.end(); } @Override public void dispose() { modelBatch.dispose(); for (Model model : models) model.dispose(); for (btRigidBody body : bodies) { body.dispose(); } sphereMotionState.dispose(); for (btCollisionShape shape : shapes) shape.dispose(); for (btRigidBodyConstructionInfo info : bodyInfos) info.dispose(); world.dispose(); collisionConfiguration.dispose(); dispatcher.dispose(); broadphase.dispose(); solver.dispose(); Gdx.app.log(this.getClass().getName(), "Disposed"); } } The highlighted parts are the addition to our previous code. After execution, we see the ball falling and colliding with the ground. Summary In this article, you learned how to load a 3D model of a car and created a basic 3D scene. Resources for Article: Further resources on this subject: Getting Started with GameSalad [article] Sparrow iOS Game Framework - The Basics of Our Game [article] Making Money with Your Game [article]

This article created by Aaron Johns, the author of Mastering Wireless Penetration Testing for Highly Secured Environments introduces Kali Linux and the steps needed to get started. Kali Linux is a security penetration testing distribution built on Debian Linux. It covers many different varieties of security tools, each of which are organized by category. Let's begin by downloading and installing Kali Linux! (For more resources related to this topic, see here.) Downloading Kali Linux Congratulations, you have now started your first hands-on experience in this article! I'm sure you are excited so let's begin! Visit http://www.kali.org/downloads/. Look under the Official Kali Linux Downloads section: In this demonstration, I will be downloading and installing Kali Linux 1.0.6 32 Bit ISO. Click on the Kali Linux 1.0.6 32 Bit ISO hyperlink to download it. Depending on your Internet connection, this may take an hour to download, so please prepare yourself ahead of time so that you do not have to wait on this download. Those who have a slow Internet connection may want to reconsider downloading from a faster source within the local area. Restrictions on downloading may apply in public locations. Please make sure you have permission to download Kali Linux before doing so. Installing Kali Linux in VMware Player Once you have finished downloading Kali Linux, you will want to make sure you have VMware Player installed. VMware Player is where you will be installing Kali Linux. If you are not familiar with VMware Player, it is simply a type of virtualization software that emulates an operating system without requiring another physical system. You can create multiple operating systems and run them simultaneously. Perform the following steps: Let's start off by opening VMware Player from your desktop: VMware Player should open and display a graphical user interface: Click on Create a New Virtual Machine on the right: Select I will install the operating system later and click on Next. Select Linux and then Debian 7 from the drop-down menu: Click on Next to continue. Type Kali Linux for the virtual machine name. Browse for the Kali Linux ISO file that was downloaded earlier then click on Next. Change the disk size from 25 GB to 50 GB and then click on Next: Click on Finish: Kali Linux should now be displaying in your VMware Player library. From here, you can click on Customize Hardware... to increase the RAM or hard disk space, or change the network adapters according to your system's hardware. Click on Play virtual machine: Click on Player at the top-left and then navigate to Removable Devices | CD/DVD IDE | Settings…: Check the box next to Connected, Select Use ISO image file, browse for the Kali Linux ISO, then click on OK. Click on Restart VM at the bottom of the screen or click on Player, then navigate to Power | Restart Guest; the following screen appears: After restarting the virtual machine, you should see the following: Select Live (686-pae) then press Enter It should boot into Kali Linux and take you to the desktop screen: Congratulations! You have successfully installed Kali Linux. Updating Kali Linux Before we can get started with any of the demonstrations in this book, we must update Kali Linux to help keep the software package up to date. Open VMware Player from your desktop. Select Kali Linux and click on the green arrow to boot it. Once Kali Linux has booted up, open a new Terminal window. Type sudo apt-get update and press Enter: Then type sudo apt-get upgrade and press Enter: You will be prompted to specify if you want to continue. Type y and press Enter: Repeat these commands until there are no more updates: sudo apt-get update sudo apt-get upgrade sudo apt-get dist-upgrade Congratulations! You have successfully updated Kali Linux! Summary This was just the introduction to help prepare you before we get deeper into advanced technical demonstrations and hands-on examples. We did our first hands-on work through Kali Linux to install and update it on VMware Player. Resources for Article: Further resources on this subject: Veil-Evasion [article] Penetration Testing and Setup [article] Wireless and Mobile Hacks [article]

In this article by Ken Cherven, author of the book, Mastering Gephi Network Visualization, we will learn how to select the most appropriate types based on the characteristics of your network data. (For more resources related to this topic, see here.) Assessing your graphing needs Now that you have seen the broad array of available layout options and a bit of their respective capabilities, it is time to step back and reconsider what story you want to tell through the data. As you have just seen, there are many directions you can take within Gephi, and there is no absolute standard for right or wrong in your layout selection. However, there are some simple guidelines that can be followed to help narrow the choices. If you are experienced with Gephi or another network analysis tool, you might wish to dive directly into the next section and begin assessing each layout type using your very own dataset; I will not attempt to convince you otherwise. This is a great way to quickly learn the basics of every layout offering and can be a great experience. On the other hand, if you wish to take a more focused approach, I will offer you a brief checklist of considerations that might help to narrow your pool of layout candidates, allowing you to spend more time with those likely to provide the best results. Think of this as akin to shopping for clothes —you could try on every type of clothing on the rack, or you can quickly narrow your choices based on certain criteria—body type, complementary colors, preferred styles, and so on. So let's have a look at some of the basic points to consider while shopping for an appropriate layout: What is the goal of your analysis? Are you attempting to show complementarity within the network, as in the relationships between nodes or sets of nodes, or is the goal to display divisions within the data? Does geography play a critical role in the network? Perhaps you are seeking to sort or rank networks based on some attribute within the data. Each of these factors can play a determining role in which layout algorithm is best for your specific network. Is the dataset small, medium, or large? Admittedly, this is a subjective criteria, but we can put some general bounds around these definitions. In my mind, if the number of nodes is measured in tens or dozens, then this is likely a small dataset that can be easily displayed in a conventional space—the Gephi workspace window or a simple letter-sized paper for a printed version. If, however, the nodes run into the hundreds, we are now moving away from a very simple network and potentially reducing the number of practical layout options. When the number of nodes in a network moves into the thousands and beyond, we have what can practically be considered a large network, at least for display considerations. With datasets of this scope, additional display considerations come into play, such as judicious use of filters, layers, and interactivity. How densely connected is the network? In our previous example using the power grid data, we had a fairly large dataset numbering in the thousands, but one that was not highly connected, at least as compared to social networks. In that case, we might have an easier time selecting and applying an effective layout, while the highly connected nature of social networks presents an additional challenge. Does the network exhibit certain measurable behaviors such as clustering and homophily? In some cases, we might not know this until the network has been visually and programmatically analyzed, but in others we might already know that the data is likely to cluster based on certain attributes that influence the network structure, including geographic proximity, alumni networks, professional associations, and a host of other possibilities. Knowing some of these in advance might help guide us either toward or away from specific layout types. Will the network be displayed on a single level, or will it be bipartite or multipartite? In this case some networks might be hierarchical, with individuals (for example) linking only to an organization, and not to other individuals in the network. There are many instances where we will wish to present hierarchical networks in this fashion. This could be used to display corporate structures, academic hierarchies, player to team relationships, and so on, and requires some different considerations than networks without this structure. Does the data have a temporal element? In simple terms, will the story be told more effectively by viewing network changes over time? This can be very effective in showing diffusion/contagion patterns, random growth, and simple shifts in behavior within a network, for example—were Thomas and James friends at T1, but no longer so at T3 (where T equals time)? If our data has a specific time element, this leads to identify layouts that will best display these changes and tell an effective story. Will the network be interactive on the user end, or will it be static? This can ultimately lead to a different layout selection when users have the ability to navigate a network via the Web. You might have additional considerations, including the speed of the layout algorithm, but the preceding list should help you to narrow the list of practical layouts, allowing you to test the remaining candidates. Actual example – the Miles Davis network Let's walk through a process following the preceding guidelines, and applying them to a project previously created by me. This will help us migrate from the theoretical constructs above to a practical application of many of these principles. The project I'll use as our example traces the studio albums recorded by the legendary jazz trumpeter, Miles Davis—48 in all. Here are the details for this project, following the above progression. Analysis goal The goal of the analysis was to inform viewers, who might or might not be jazz fans, about the remarkable, far reaching recording legacy of Miles Davis. Since the career of Davis moved through many stages, he crossed paths with and employed an incredible number of artists across a diverse range of instruments that ranged far beyond the normal jazz instrumentation. Therefore, part of the goal of the analysis was to expose viewers to this great diversity, and give them the ability to see changes and patterns within the scope of his career. Dataset parameters The dataset in this case is not insignificant—while 48 albums would represent a small network if left on its own, we know from the data that there are typically at least four musicians per recording, and often far more, numbering into the 20s in some cases. Many of the musicians are represented on multiple recordings, but there is still a multiplicative impact on the size of the network, which turns out to have about 350 nodes. While this certainly doesn't rival the enormous datasets often seen in social networks, it is large enough that we need to be thoughtful about the layout and how users will interact with the project. Here is a look at some of the underlying data for the nodes: Miles Davis nodes Notice that the nodes are a combination of an individual musician and a specific instrument, since so many of these musicians play a second (or even third) instrument. The data is then grouped by instrument, which allows you to partition and custom color the data. Now, the following figure illustrates a partial view of the edge's data: Miles Davis data edges In the preceding screenshot, we see only album level connections, with Miles Davis as the source and each album as the target, although the edges are left undirected. If we move further into the edge's data, we can see how the network is structured a bit more clearly: Miles Davis data edge details This data shows some of the musician level connections to specific recordings, as well as the instrument played on that album. This completes the basic structure of the network, as each musician will have an edge connecting them to any and all albums they played on. So this gives us a basic understanding of how the data will be represented in the network—Miles at the core, all albums at a second level, followed by every contributing musician at a tertiary level. Network density We have all seen many highly connected networks with edges crossing between nodes or groups within a graph that become virtually impenetrable for the viewer. Fortunately, this was not a major concern with this network, given its relatively modest size, but it could still play a role in the final layout selection. As always, the goal is to provide clarity and understanding, regardless of the relative size of the network, so minimizing visual clutter is always a priority. Network behaviors Examining the network behaviors can be an interesting exercise, as it often leads us to findings that were not necessarily anticipated. In the case of this project, we know from viewing the data that Miles played with certain musicians on a frequent basis, but would then often play with an entirely new group during his next phase, before switching yet again to a completely unrelated group of musicians. In other words, there were multiple aggregations of musicians who only occasionally intersected with one another. This is very nearly a proxy for homophily, with distinct clusters connected to each other through a single node (Miles Davis in this case) or perhaps a small subset of network members who act as bridges between various clusters. Based on this knowledge, we would anticipate a highly clustered network with a significant level of connectedness within a given cluster, and a limited set of connections between clusters. The next decision to make was how best to display this network. Network display We just saw the underlying data structure, which had a bipartite nature to it, with each musician connecting to one or more albums, rather than to other musicians. Given this type of network, we want to select a layout that eases our ability to see not only the connections between Miles Davis and each recording, but also from each album to all of the participating musicians. This will require a layout that provides enough empty space to make for clear viewing, but also one that manages to combine this with a minimal number of edge crossings. Remember that many of these musicians played on multiple recordings, so they must be positioned in proximity to several albums at the same time, without adding to a cluttered look. After testing several layouts, some of which simply didn't work effectively with the above two needs, I settled on the ARF algorithm for its visual clarity to display this particular network. The ability to see patterns within the network, even prior to adding interactivity, is a plus; if the network passes that test, it should be very effective once users interact with the information. Temporal elements Another interesting aspect of the network that could have been utilized was the timeline for the recordings. With more than four decades of recordings, this could have provided a wealth of information about changes over time in the musicians' network and instrumentation on each album. This element was not highlighted, but it does make its presence felt in the final network, with albums from one period with a consistent cast of musicians occupying one sector of the graph, while other types of albums with many infrequently used musicians land in another area. Interactivity The final decision was whether to make the network interactive, giving users the ability to learn more through self-navigation of the graph. This was considered important from the very start, so that the viewers could see not only the body of work represented by the 48 recordings, but also the evolution of which musicians were involved, as well as shining a light on the wide array of instruments used as Miles' career evolved. After each of these considerations was evaluated, and through a period of testing the network using multiple layouts, I settled on the ARF force-directed layout coupled with the Sigma.js plugin for interactivity. Here's a look at the final output, which includes options using the Sigma.js plugin: The Miles Davis network graph The link to the project can be found at http://visual-baseball.com/gephi/jazz/miles_davis/. I hope this example helps to generate some ideas or at least opens up the possibilities for what Gephi is capable of creating, and that the process illustrated earlier helps to provide at least a foundation for your own work. Summary In this article, you learned how to select the most appropriate types based on the characteristics of your network data. Resources for Article: Further resources on this subject: Data visualization [article] Visualization as a Tool to Understand Data [article] Creating Network Graphs with Gephi [article]

Advanced Programming and Control In this article by Gary Garber, author of the book Learning LEGO MINDSTORMS EV3, we will explore advanced controlling algorithms to use for sensor-based navigation and tracking. We will cover: Proportional distance control with the Ultrasonic Sensor Proportional distance control with the Infrared (IR) Sensor Line following with the Color Sensor Two-level control with the Color Sensor Proportional control with the Color Sensor Proportional integral derivative control Precise turning and course correction with the Gyro Sensor Beacon tracking with the IR sensor Triangulation with two IR beacons (For more resources related to this topic, see here.) Distance controller In this section, we will program the robot to gradually come to a stop using a proportional algorithm. In a proportional algorithm, the robot will gradually slow down as it approaches the desired stopping point. Before we begin, we need to attach a distance sensor to our robot. If you have the Home Edition, you will be using the IR sensor, whereas if you have the Educational Edition, you will use the Ultrasonic Sensor. Because these sensors use reflected beams (infrared light or sound), they need to be placed unobstructed by the other parts of the robot. You could either place the sensor high above the robot or well out in front of many parts of the robot. The design I have shown in the following screenshot allows you to place the sensor in front of the robot. If you are using the Ultrasonic Sensor for FIRST Lego League (a competition that uses a lot of sensor-based navigation) and trying to measure the distance to the border, you will find it is a good idea to place the sensor as low as possible. This is because the perimeter of the playing fields for FIRST LEGO League are made from 3- or 4-inch- high pieces of lumber. Infrared versus Ultrasonic We are going to start out with a simple program and will gradually add complexity to it. If you are using the Ultrasonic Sensor, it should be plugged into port 4, and this program is on the top line. If you are using the IR sensor, it should be plugged into port 1 and this program is at the bottom line. In this program, the robot moves forward until the Wait block tells it to stop 25 units from a wall or other barrier. You will find that the Ultrasonic Sensor can be set to stop in units of inches or centimeters. The Ultrasonic Sensor emits high-frequency sound waves (above the range of human hearing) and measures the time delay between the emission of the sound waves and when the reflection off an object is measured by the sensor. In everyday conditions, we can assume that the speed of sound is constant, and thus the Ultrasonic Sensor can give precise distance measurements to the nearest centimeter. In other programming languages, you could even use the Ultrasonic Sensor to transmit data between two robots. The IR sensor emits infrared light and has an IR-sensitive camera that measures the reflected light. The sensor reading does not give exact distance units because the strength of the signal depends on environmental factors such as the reflectivity of the surface. What the IR sensor loses in precision in proximity measurements, it makes up for in the fact that you can use it to track on the IR beacon, which is a source of infrared light. In other programming languages, you could actually use the IR sensor to track on sources of infrared light other than the beacon (such as humans or animals). In the following screenshot, we have a simple program that will tell the robot to stop a given distance from a barrier using a Wait for the sensor block. The program on the top of the screenshot uses the Ultrasonic Sensor, and the program on the bottom of the screenshot uses the IR sensor. You should only use the program for the sensor you are using. If you are downloading and executing the program from the Packt Publishing website, you should delete the program that you do not need. When you execute the program in the preceding screenshot, you will find that the robot only begins to stop at 25 units from the wall, but cannot stop immediately. To do this, the robot will need to slow down before it gets to the stopping point. Proportional algorithm In the next set of program, we create a loop called Slow Down. Inside this loop, readings from the Ultrasonic or Infrared proximity sensor block are sent to a Math block (to take the negative of the position values so that the robot moves forward) and then sent to the power input of a Move Steering block. We can have the loop end when it reaches our desired stopping distance as shown in the following screenshot: Instead of using the exact values of the output of the sensor block, we can use the difference between the actual position and the desired position to control the Move Steering block, as shown in the following screenshot. This difference is called the error. We call the desired position the setpoint. In the following screenshot, the setpoint is 20. The power is actually proportional to the error or the difference between the positions. When you execute this code, you will also find that if the robot is too close to the wall, it will run in reverse and back up from the wall. We are using an Advanced Math block in the following screenshot. You can see that we are writing a simple equation, -(a-b), into the block text field of the Advanced Math block: You may have also noticed that the robot moves very slowly as it approaches the stopping point. You can change this program by adding gain to the algorithm. If you multiply the difference by a larger factor, it will approach the stopping point quicker. When you execute this program, you will find that if you increase the gain too much, it will overshoot the stopping point and reverse direction. We can adjust these values using the Advanced Math block. We can type in any simple math function we need, as shown in the following screenshot. In this block, the value of a is the measured position, b is the setpoint position, and c is the gain. The equation can be seen in the following screenshot inside the block text field of the Advanced Math block: We can also define the desired gain and setpoint position using variables. We can create two Variable blocks called Gain and Set Point. We can write the value 3 to the Gain variable block and 20 to the Set Point variable block. Inside our loop, we can then read these variables and take the output of the Read Variable block and draw data wires into the Advanced Math block. The basic idea of the proportional algorithm is that the degree of correction needed is proportional to the error. So when our measured value is far from our goal, a large correction is applied. When our measured value is near our goal, only a small correction is applied. The algorithm also allows overcorrections. If the robot moves past the setpoint distance, it will back up. Depending on what you are trying to do, you will need to play around with various values for the gain variable. If the gain is too large, you will overshoot your goal and oscillate around it. If your gain is too small, you will never reach your goal. The response time of the microprocessor also affects the efficiency of the algorithm. You can experiment by inserting a Wait block into the loop and see how this affects the behavior of the robot. If we are merely using the distance sensor to approach a stationary object, then the proportional algorithm will suffice. However, if you were trying to maintain a given distance from a moving object (such as another robot), you might need a more complicated algorithm such as a Proportional Integral Derivative (PID) controller. Next we will build a line follower using the Color Sensor, which will use a PID controller. Line following using the Color Sensor When we are using the Color Sensor in Reflected Light Intensity mode, the sensor emits light and the robot measures the intensity of the reflected light. The brightness of the red LED in the sensor is a constant, but the intensity of the reflection will depend on the reflectivity of the surface, the angle of the sensor relative to the surface, and the distance of the sensor from the surface. If you shine the sensor at a surface, you will notice that a circle of light is generated. As you change the height of the sensor, the diameter of this circle will change because the light emitted from the LED diverges in a cone. As you increase the height, the size of the circle gets larger and the reflected intensity gets smaller. You might think you want the sensor to be as close as possible to the surface. Because there is a finite distance between the LED and the photo diode (which collects the light) of about 5.5 mm, it puts a constraint on the minimum diameter of your circle of light. Ideally, you want the circle of light to have a diameter of about 11 mm, which means placing the sensor about half of a centimeter above the tracking surface. For the caster-bot, you will need the sensor, an axle, two bushings, two long pins, a 5-mod beam, and two axle-pin connectors, as you can see in the following screenshot: You can assemble the sensor attachment in two steps. The sensor attachment settles into the holes in the caster attachment itself as you can see in the following screenshot. This placement is ideal as it allows the caster to do the steering while you do your line tracking. You can build the Color Sensor attachment in four steps. The Color Sensor attachment for the skid-bot will be the most complicated of our designs because we want the sensor to be in front of the robot and the skid is quite long. Again, we will need the pins, axles, bushings, and axle-pin connectors seen in the following screenshot: The Color Sensor attachment will connect directly to the EV3 brick. As you can see in the following screenshot, the attachment will be inserted from below the brick: Next I will describe the attachment for the tread-bot from the Educational kit. Because the tread-bot is slightly higher off the ground, we need to use some pieces such as the thin 1 x 4 mod lift arm that is a half mod in height. This extra millimeter in height can make a huge difference in the signal strength. The pins have trouble gripping the thin lift arm, so I like to use the pins with stop bushings to prevent the lift arm from falling off. The Light Sensor attachment is once again inserted into the underside of the EV3 brick as you can see in the following screenshot: The simplest of our Light Sensor attachments will be the tread-bot for the Home Edition, and you can build this in one step. Similarly, it attaches to the underside of the EV3 brick. Summary In this article, we explored advanced methods of navigations. We used both the Ultrasonic Sensor and the Infrared Sensor to measure distance with a proportional algorithm. Resources for Article: Further resources on this subject: eJOS – Unleashing EV3 [article] Proportional line follower (Advanced) [article] Making the Unit Very Mobile - Controlling Legged Movement [article]

In this article by Saurav Haloi, author of the book a Apache Zookeeper Essentials, we will learn about Apache ZooKeeper is a software project of the Apache Software Foundation; it provides an open source solution to the various coordination problems in large distributed systems. ZooKeeper as a centralized coordination service is distributed and highly reliable, running on a cluster of servers called a ZooKeeper Ensemble. Distributed consensus, group management, presence protocols, and leader election are implemented by the service so that the applications do not need to reinvent the wheel by implementing them on its own. On top of these, the primitives exposed by ZooKeeper can be used by applications to build much more powerful abstractions for solving a wide variety of problems. (For more resources related to this topic, see here.) Apache ZooKeeper is implemented in Java. It ships with C, Java, Perl, and Python client bindings. Community contributed client libraries are available for a plethora of languages like Go, Scala, Erlang, and so on. Apache ZooKeeper is widely used by large number of organizations, such as Yahoo Inc., Twitter, Netflix and Facebook, in their distributed application platforms as a coordination service. In this article we will look into installation and configuration of Apache ZooKeeper, some of the concepts associated with it followed by programming using Python client library of ZooKeeper. We will also read how we can implement some of the important constructs of distributed programming using ZooKeeper. Download and installation ZooKeeper is supported by a wide variety of platforms. GNU/Linux and Oracle Solaris are supported as development and production platforms for both server and client. Windows and Mac OS X are recommended only as development platforms for both server and client. ZooKeeper is implemented in Java and requires Java 6 or later versions to run. Let's download the stable version from one of the mirrors, say Georgia Tech's Apache download mirror (http://b.gatech.edu/1xElxRb) in the following example: $ wgethttp://www.gtlib.gatech.edu/pub/apache/zookeeper/stable/zookeeper-3.4.6.tar.gz$ ls -alh zookeeper-3.4.6.tar.gz-rw-rw-r-- 1 saurav saurav 17M Feb 20 2014 zookeeper-3.4.6.tar.gz Once we have downloaded the ZooKeeper tarball, installing and setting up a standalone ZooKeeper node is pretty simple and straightforward. Let's extract the compressed tar archive into /usr/share: $ tar -C /usr/share -zxf zookeeper-3.4.6.tar.gz$ cd /usr/share/zookeeper-3.4.6/$ lsbin CHANGES.txt contrib docs ivy.xml LICENSE.txtREADME_packaging.txt recipes zookeeper-3.4.6.jar zookeeper-3.4.6.jar.md5build.xml conf dist-maven ivysettings.xml libNOTICE.txt README.txt src zookeeper-3.4.6.jar.asczookeeper-3.4.6.jar.sha1 The location where the ZooKeeper archive is extracted in our case, /usr/share/zookeeper-3.4.6, can be exported as ZK_HOME as follows: $ export ZK_HOME=/usr/share/zookeeper-3.4.6 Configuration Once we have extracted the tarball, the next thing is to configure ZooKeeper. The conf folder holds the configuration files for ZooKeeper. ZooKeeper needs a configuration file called zoo.cfg in the conf folder inside the extracted ZooKeeper folder. There is a sample configuration file that contains some of the configuration parameters for reference. Let's create our configuration file with the following minimal parameters and save it in the conf directory: $ cat conf/zoo.cfgtickTime=2000dataDir=/var/lib/zookeeperclientPort=2181 The configuration parameters' meanings are explained here: tickTime: This is measured in milliseconds; it is used for session registration and to do regular heartbeats by clients with the ZooKeeper service. The minimum session timeout will be twice the tickTime parameter. dataDir: This is the location to store the in-memory state of ZooKeeper; it includes database snapshots and the transaction log of updates to the database. Extracting the ZooKeeper archive won't create this directory, so if this directory doesn't exist in the system, you will need to create it and set writable permission to it. clientPort: This is the port that listens for client connections, so it is where the ZooKeeper clients will initiate a connection. The client port can be set to any number, and different servers can be configured to listen on different ports. The default is 2181. ZooKeeper needs the JAVA_HOME environment variable to be set correctly. To see if this is set in your system, run the following command: $ echo $JAVA_HOME Starting the ZooKeeper server Now, considering that Java is installed and working properly, let's go ahead and start the ZooKeeper server. All ZooKeeper administration scripts to start/stop the server and invoke the ZooKeeper command shell are shipped along with the archive in the bin folder with the following code: $ pwd /usr/share/zookeeper-3.4.6/bin $ ls README.txt zkCleanup.sh zkCli.cmd zkCli.sh zkEnv.cmd zkEnv.sh zkServer.cmd zkServer.sh The scripts with the .sh extension are for Unix platforms (GNU/Linux, Mac OS X, and so on), and the scripts with the .cmd extension are for Microsoft Windows operating systems. To start the ZooKeeper server in a GNU/Linux system, you need to execute the zkServer.sh script as follows. This script gives options to start, stop, restart, and see the status of the ZooKeeper server: $ ./zkServer.sh JMX enabled by default Using config: /usr/share/zookeeper-3.4.6/bin/../conf/zoo.cfg Usage: ./zkServer.sh {start|start-foreground|stop|restart|status|upgrade|print-cmd} To avoid going to the ZooKeeper install directory to run these scripts, you can include it in your PATH variable as follows: export PATH=$PATH:/usr/share/zookeeper-3.4.6/bin Executing zkServer.sh with the start argument will start the ZooKeeper server. A successful start of the server will show the following output: $ zkServer.sh start JMX enabled by default Using config: /usr/share/zookeeper-3.4.6/bin/../conf/zoo.cfg Starting zookeeper ... STARTED To verify that the ZooKeeper server has started, you can use the following ps command: $ ps –ef | grep zookeeper | grep –v grep | awk '{print $2}' 5511 The ZooKeeper server's status can be checked with the zkServer.sh script as follows: $ zkServer.sh status JMX enabled by default Using config: /usr/share/zookeeper-3.4.6/bin/../conf/zoo.cfg Mode: standalone Connecting to ZooKeeper with a Java-based shell To start the Java-based ZooKeeper command-line shell, we simply need to run zkCli.sh of the ZK_HOME/bin folder with the server IP and port as follows: ${ZK_HOME}/bin/zkCli.sh –server zk_server:port In our case, we are running our ZooKeeper server on the same machine, so the ZooKeeper server will be localhost, or the loop-back address will be 127.0.0.1. The default port we configured was 2181: $ zkCli.sh -server localhost:2181 As we connect to the running ZooKeeper instance, we will see the output similar to the following one in the terminal (some output is omitted): Connecting to localhost:2181 ............... ............... Welcome to ZooKeeper! JLine support is enabled ............. WATCHER:: WatchedEvent state:SyncConnected type:None path:null [zk: localhost:2181(CONNECTED) 0] To see a listing of the commands supported by the ZooKeeper Java shell, you can run the help command in the shell prompt: [zk: localhost:2181(CONNECTED) 0] help ZooKeeper -server host:port cmd args connect host:port get path [watch] ls path [watch] set path data [version] rmr path delquota [-n|-b] path quit printwatches on|off create [-s] [-e] path data acl stat path [watch] close ls2 path [watch] history listquota path setAcl path acl getAcl path sync path redo cmdno addauth scheme auth delete path [version] setquota -n|-b val path We can execute a few simple commands to get a feel of the command-line interface. Let's start by running the ls command, which, as in Unix, is used for listing: [zk: localhost:2181(CONNECTED) 1] ls / [zookeeper] Now, the ls command returned a string called zookeeper, which is a znode in the ZooKeeper terminology. We can create a znode through the ZooKeeper shell as follows: To begin with, let's create a HelloWorld znode with empty data. [zk: localhost:2181(CONNECTED) 2] create /HelloWorld "" Created /HelloWorld [zk: localhost:2181(CONNECTED) 3] ls / [zookeeper, HelloWorld] We can delete the znode created by issuing the delete command as follows: [zk: localhost:2181(CONNECTED) 4] delete /HelloWorld [zk: localhost:2181(CONNECTED) 5] ls / [zookeeper] The ZooKeeper data model ZooKeeper allows distributed processes to coordinate with each other through a shared hierarchical namespace of data registers. The namespace looks quite similar to a Unix filesystem. The data registers are known as znodes in the ZooKeeper nomenclature. ZooKeeper has two types of znodes: persistent and ephemeral. There is a third type that you might have heard of, called a sequential znode, which is a kind of a qualifier for the other two types. Both persistent and ephemeral znodes can be sequential znodes as well. The persistent znode As the name suggests, persistent znodes have a lifetime in the ZooKeeper’s namespace until they’re explicitly deleted. A znode can be deleted by calling the delete API call. The ephemeral znode An ephemeral znode is deleted by the ZooKeeper service when the creating client’s session ends. An end to a client’s session can happen because of disconnection due to a client crash or explicit termination of the connection. The sequential znode A sequential znode is assigned a sequence number by ZooKeeper as a part of its name during its creation. The value of a monotonously increasing counter (maintained by the parent znode) is appended to the name of the znode. The ZooKeeper Watches ZooKeeper is designed to be a scalable and robust centralized service for very large distributed applications. A common design anti-pattern associated while accessing such services by clients is through polling or a pull kind of a model. A pull model often suffers from scalability problems when implemented in large and complex distributed systems. To solve this problem, ZooKeeper designers implemented a mechanism where clients can get notifications from the ZooKeeper service instead of polling for events. This resembles a push model, where notifications are pushed to the registered clients of the ZooKeeper service. Clients can register with the ZooKeeper service for any changes associated with a znode. This registration is known as setting a watch on a znode in ZooKeeper terminology. Watches allow clients to get notifications when a znode changes in any way. A watch is a one-time operation, which means that it triggers only one notification. To continue receiving notifications over time, the client must reregister the watch upon receiving each event notification. ZooKeeper watches are a one-time trigger. What this means is that if a client receives a watch event and wants to get notified of future changes, it must set another watch. Whenever a watch is triggered, a notification is dispatched to the client that had set the watch. Watches are maintained in the ZooKeeper server to which a client is connected, and this makes it a fast and lean means of event notification. The watches are triggered for the following three changes to a znode: Any changes to the data of a znode, such as when new data is written to the znode’s data field using the setData operation. Any changes to the children of a znode. For instance, children of a znode are deleted with the delete operation. A znode being created or deleted, which could happen in the event that a new znode is added to a path or an existing one is deleted. Again, ZooKeeper asserts the following guarantees with respect to watches and notifications: ZooKeeper ensures that watches are always ordered in the FIFO manner and that notifications are always dispatched in order Watch notifications are delivered to a client before any other change is made to the same znode The order of the watch events are ordered with respect to the updates seen by the ZooKeeper service ZooKeeper operations ZooKeeper’s data model and its API support the following nine basic operations: Operation Description Operation Event-generating Actions exists A znode is created or deleted, or its data is updated getChildren A child of a znode is created or deleted, or the znode itself is deleted getData A znode is deleted or its data is updated Watches and ZooKeeper operations The read operations in znodes, such as exists, getChildren, and getData, allow watches to be set on them. On the other hand, the watches triggered by znode's write operations, such as create, delete, and setData. ACL operations do not participate in watches. The following are the types of watch events that might occur during a znode state change: NodeChildrenChanged: A znode’s child is created or deleted NodeCreated: A znode is created in a ZooKeeper path NodeDataChanged: The data associated with a znode is updated NodeDeleted: A znode is deleted in a ZooKeeper path Programming with Apache ZooKeeper with Python ZooKeeper is easily programmable and has client binding for a plethora of languages. Its shipped with official Java, C, Perl and Python client libraries. Here we will look at programming ZooKeeper with Python: Apache ZooKeeper is shipped with an official client binding for Python, which is developed on top of the C bindings. It can be found in the contrib/zkpython directory of the ZooKeeper distribution. To build and install the Python binding, refer to the instructions in the README file there. In this section, we will study about another popular Python client library for ZooKeeper, called Kazoo (https://kazoo.readthedocs.org/). Kazoo is a pure Python library for ZooKeeper, which means that unlike the official Python bindings, Kazoo is implemented fully in Python and has no dependency on the C bindings of ZooKeeper. Along with providing both synchronous and asynchronous APIs, the Kazoo library also provides APIs for some distributed data structure primitives such as distributed locks, leader election, distributed queues, and so on. Installation of Kazoo is very simple, which can be done either with pip or easy_install installers: Using pip, Kazoo can be installed with the following command: $ pip install kazoo Using easy_install, Kazoo is installed as follows: $ easy_install kazoo To verify whether Kazoo is installed properly, let's try to connect to the ZooKeeper instance and print the list of znodes in the root path of the tree, as shown in the following screenshot: In the preceding example, we imported the KazooClient, which is the main ZooKeeper client class. Then, we created an object of the class (an instance of KazooClient) by connecting to the ZooKeeper instance that is running on the localhost. Once we called the start() method, it initiates a connection to the ZooKeeper server. Once successfully connected, the instance contains the handle to the ZooKeeper session. Now, when we called the get_children() method on the root path of the ZooKeeper namespace, it returned a list of the children. Finally, we closed the connection by calling the stop() method. A watcher implementation Kazoo provides a higher-level child and data watching API's as a recipe through a module called kazoo.recipe.watchers. This module provides the implementation of DataWatch and ChildrenWatch along with another class called PatientChildrenWatch. The PatientChildrenWatch> class returns values after the children of a node don't change for a period of time, unlike the other two, which return each time an event is generated. Let's look at the implementation of a simple children watcher client, which will generate an event each time a znode is added or deleted from the ZooKeeper path: import signal from kazoo.client import KazooClient from kazoo.recipe.watchers import ChildrenWatch zoo_path = '/MyPath' zk = KazooClient(hosts='localhost:2181') zk.start() zk.ensure_path(zoo_path) @zk.ChildrenWatch(zoo_path) def child_watch_func(children): print "List of Children %s" % children while True: signal.pause() In this simple implementation of a children watcher, we connect to the ZooKeeper server that is running in the localhost, using the following code, and create a path /MyPath: zk.ensure_path(zoo_path) @zk.ChildrenWatch(zoo_path) We then set a children watcher on this path and register a callback method child_watch_func, which prints the current list of children on the event generated in /MyPath. When we run this client watcher in a terminal, it starts listening to events: On another terminal, we will create some znodes in/MyPath with the ZooKeeper shell: We observe that the children watcher client receives these znode creation events, and it prints the list of the current children in the terminal window: Similarly, if we delete the znodes that we just created, the watcher will receive the events and subsequently will print the children listing in the console: The messages shown in the following screenshot are printed in the terminal where the children watcher is running: ZooKeeper recipes In this section, you will learn to develop high-level distributed system constructs and data structures using ZooKeeper. As mentioned earlier, most of these constructs and functions are of utmost importance in building scalable distributed architectures, but they are fairly complicated to implement from scratch. Developers can often get bogged down while implementing these and integrating them with their application logic. In this section, you will learn how to develop algorithms to build some of these high-level functions using ZooKeeper primitives and data model and see how ZooKeeper makes it simple, scalable, and error free, with much lesser code. Barrier Barrier is a type of synchronization method used in distributed systems to block the processing of a set of nodes until a condition is satisfied. It defines a point where all nodes must stop their processing and cannot proceed until all the other nodes reach this barrier. The algorithm to implement a barrier using ZooKeeper is as follows: To start with, a znode is designated to be a barrier znode, say /zk_barrier. The barrier is said to be active in the system if this barrier znode exists . Each client calls the ZooKeeper API's exists() function on /zk_barrier by registering for watch events on the barrier znode (the watch event is set to true). If the exists() method returns false, the barrier no longer exists, and the client proceeds with its computation. Else, if the exists() method returns true, the clients just waits for watch events. Whenever the barrier exit condition is met, the client in charge of the barrier will delete /zk_barrier. The deletion triggers a watch event, and on getting this notification, the client calls the exists() function on /zk_barrier again. Step 7 returns true, and the clients can proceed further. The barrier exists until the barrier znode ceases to exist! In this way, we can implement a barrier using ZooKeeper without much of an effort. The example cited so far is for a simple barrier to stop a group of distributed processes from waiting on some condition and then proceed together when the condition is met. There is another type of barrier that aids in synchronizing the beginning and end of a computation; this is known as double barrier. The logic of a double barrier states that a computation is started when the required number of processes join the barrier. The processes leave after completing the computation, and when the number of processes participating in the barrier become zero, the computation is stated to end. The algorithm for a double barrier is implemented by having a barrier znode that serves the purpose of being a parent for individual process znodes participating in the computation. It's algorithm is outlined as follows: Phase 1: Joining the barrier znode can be done as follows: Suppose the barrier znode is represented by znode/barrier. Every client process registers with the barrier znode by creating an ephemeral znode with /barrier as the parent. In real scenarios, clients might register using their hostnames. The client process sets a watch event for the existence of another znode called ready under the /barrier znode and waits for the node to appear. A number N is predefined in the system; this governs the minimum number of clients to join the barrier before the computation can start. While joining the barrier, each client process finds the number of child znodes of /barrier: M = getChildren(/barrier, watch=false) 5. If M is less than N, the client waits for the watch event registered in step 3. Else, if M is equal to N, then the client process creates the ready znode under /barrier. The creation of the ready znode in step 5 triggers the watch event, and each client starts the computation that they were waiting so far to do. Phase 2: Leaving the barrier can be done as follows: Client processing on finishing the computation deletes the znode it created under /barrier (in step 2 of Phase 1: Joining the barrier). The client process then finds the number of children under /barrier: M = getChildren(/barrier, watch=True) If M is not equal to 0, this client waits for notifications (observe that we have set the watch event to True in the preceding call). If M is equal to 0, then the client exits the barrier znode The preceding procedure suffers from a potential herd effect where all client processes wake up to check the number of children left in the barrier when a notification is triggered. To get away with this, we can use a sequential ephemeral znode to be created in step 2 of Phase 1: Joining the barrier. Every client process watches it's next lowest sequential ephemeral znode to go away as an exit criterion. This way, only a single event is generated for any client completing the computation, and hence, not all clients need to wake up together to check on its exit condition. For a large number of client processes participating in a barrier, the herd effect can negatively impact the scalability of the ZooKeeper service, and developers should be aware of such scenarios. A Java language implementation of a double barrier can be found in the ZooKeeper documentation at http://zookeeper.apache.org/doc/r3.4.6/zookeeperTutorial.html. Queue A distributed queue is a very common data structure used in distributed systems. A special implementation of a queue, called a producer-consumer queue, is where a collection of processes called producers generate or create new items and put them in the queue, while consumer processes remove the items from the queue and process them. The addition and removal of items in the queue follow a strict ordering of first in first out (FIFO). A producer-consumer queue can be implemented using ZooKeeper. A znode will be designated to hold a queue instance, say queue-znode. All queue items are stored as znodes under this znode. Producers add an item to the queue by creating a znode under the queue-znode, and consumers retrieve the items by getting and then deleting a child from the queue-znode. The FIFO order of the items is maintained using sequential property of znode provided by ZooKeeper. When a producer process creates a znode for a queue item, it sets the sequential flag. This lets ZooKeeper append the znode name with a monotonically increasing sequence number as the suffix. ZooKeeper guarantees that the sequence numbers are applied in order and are not reused. The consumer process processes the items in the correct order by looking at the sequence number of the znode. The pseudocode for the algorithm to implement a producer-consumer queue using ZooKeeper is shown here: Let /_QUEUE_ represent the top-level znode for our queue implementation, which is also called the queue-node. Clients acting as producer processes put something into the queue by calling the create() method with the znode name as "queue-" and set the sequence and ephemeral flags if the create() method call is set true: create( “queue-“, SEQUENCE_EPHEMERAL) The sequence flag lets the new znode get a name like queue-N, where N is a monotonically increasing number Clients acting as consumer processes process a getChildren() method call on the queue-node with a watch event set to true: M = getChildren(/_QUEUE_, true) It sorts the children list M, takes out the lowest numbered child znode from the list, starts processing on it by taking out the data from the znode, and then deletes it. The client picks up items from the list and continues processing on them. On reaching the end of the list, the client should check again whether any new items are added to the queue by issuing another get_children() method call. > The algorithm continues when get_children() returns an empty list; this means that no more znodes or items are left under /_QUEUE_. It's quite possible that in step 3, the deletion of a znode by a client will fail because some other client has gained access to the znode while this client was retrieving the item. In such scenarios, the client should retry the delete call. Using this algorithm for implementation of a generic queue, we can also build a priority queue out of it, where each item can have a priority tagged to it. The algorithm and implementation is left as an exercise to the readers. C and Java implementations of the distributed queue recipe are shipped along with the ZooKeeper distribution under the recipes folder. Developers can use this recipe to implement distributed lock in their applications. Kazoo, the Python client library for ZooKeeper, has distributed queue implementations inside the kazoo.recipe.queue module. This queue implementation has priority assignment to the queue items support as well as queue locking support that are built into it. Lock A lock in a distributed system is an important primitive that provides the applications with a means to synchronize their access to shared resources. Distributed locks need to be globally synchronous to ensure that no two clients can hold the same lock at any instance of time. Typical scenarios where locks are inevitable are when the system as a whole needs to ensure that only one node of the cluster is allowed to carry out an operation at a given time, such as: Write to a shared database or file Act as a decision subsystem Process all I/O requests from other nodes   ZooKeeper can be used to implement mutually exclusive locks for processes that run on different servers across different networks and even geographically apart.   To build a distributed lock with ZooKeeper, a persistent znode is designated to be the main lock-znode. Client processes that want to acquire the lock will create an ephemeral znode with a sequential flag set under the lock-znode. The crux of the algorithm is that the lock is owned by the client process whose child znode has the lowest sequence number. ZooKeeper guarantees the order of the sequence number, as sequence znodes are numbered in a monotonically increasing order. Suppose there are three znodes under the lock-znode: l1, l2, and l3. The client process that created l1 will be the owner of the lock. If the client wants to release the lock, it simply deletes l1, and then, the owner of l2 will be the lock owner and so on. The pseudocode for the algorithm to implement a distributed lock service with ZooKeeper is shown here: Let the parent lock node be represented by a persistent znode, /_locknode_, in the Zookeeper tree. Phase 1: Acquire a lock with the following steps: Call the create("/_locknode_/lock-",CreateMode=EPHEMERAL_SEQUENTIAL) method. Call the getChildren("/_locknode_/lock-", false) method on the lock node. Here, the watch flag is set to false, as otherwise, it can lead to a herd effect. If the znode created by the client in step 1 has the lowest sequence number suffix, then the client is owner of the lock, and it exits the algorithm. Call the exists("/_locknode_/, True) method. If the exists() method returns false, go to step 2. If the exists() method returns true, wait for notifications for the watch event set in step 4. Phase 2: Release a lock as follows: The client holding the lock deletes the node, thereby triggering the next client in line to acquire the lock. The client that created the next higher sequence node will be notified and hold the lock. The watch for this event was set in step 4 of Phase 1,:Acquire a lock. While it's not recommended that you use a distributed system with a large number of clients due to the herd effect, if the other clients also need to know about the change of lock ownership, they could set a watch on the /_locknode_ lock node for events of the NodeChildrenChanged type and can determine the current owner. If there was a partial failure in the creation of znode due to connection loss, it's possible that the client won't be able to correctly determine whether it successfully created the child znode. To resolve such a situation, the client can store its session ID in the znode data field or even as a part of the znode name itself. As a client retains the same session ID after a reconnect, it can easily determine whether the child znode was created by it by looking at the session ID. The idea of creating an ephemeral znode prevents a potential dead-lock situation that might arise when a client dies while holding a lock. However, as the property of the ephemeral znode dictates that it gets deleted when the session times out or expires, ZooKeeper will delete the znode created by the dead client, and the algorithm runs as usual. However, if the client hangs for some reason but the ZooKeeper session is still active, then we might get into a deadlock. This can be solved by having a monitor client that triggers an alarm when the lock holding time for a client crosses a predefined time out. The ZooKeeper distribution is shipped with the C and Java language implementation of a distributed lock in the recipes folder. The recipe implements the algorithm you have learned so far and also takes into account the problems associated with partial failure and herd effect. The previous recipe of a mutually exclusive lock can be modified to implement a shared lock as well. Readers can find the algorithm and pseudocode for a shared lock using Zookeeper in the documentation at http://zookeeper.apache.org/doc/r3.4.6/recipes.html#Shared+Locks. More ZooKeeper recipes are available at: http://zookeeper.apache.org/doc/trunk/recipes.html Summary In this article, we read about the fundamentals of Apache ZooKeeper, programming it and how to implement common distributed data structures with ZooKeeper. For more details on Apache ZooKeeper, please visit its project page. Resources for Article: Further resources on this subject: Creating an Apache JMeter™ test workbench [article] Apache Maven and m2eclipse [article] Coverage with Apache Karaf Pax Exam tests [article]

This article by Fabian Vogelsteller, the author of Building Single-page Web Apps with Meteor, explores the full-stack framework of Meteor. Meteor is not just a JavaScript library such as jQuery or AngularJS. It's a full-stack solution that contains frontend libraries, a Node.js-based server, and a command-line tool. All this together lets us write large-scale web applications in JavaScript, on both the server and client, using a consistent API. (For more resources related to this topic, see here.) Even with Meteor being quite young, already a few companies such as https://lookback.io, https://respond.ly and https://madeye.io use Meteor already in their production environment. If you want to see for yourself what's made with Meteor, take a look at http://madewith.meteor.com. Meteor makes it easy for us to build web applications quickly and takes care of the boring processes such as file linking, minifying, and concatenating of files. Here are a few highlights of what is possible with Meteor: We can build complex web applications amazingly fast using templates that automatically update themselves when data changes We can push new code to all clients on the fly while they are using our app Meteor core packages come with a complete account solution, allowing a seamless integration with Facebook, Twitter, and more Data will automatically be synced across clients, keeping every client in the same state in almost real time Latency compensation will make our interface appear super fast while the server response happens in the background With Meteor, we never have to link files with the <script> tags in HTML. Meteor's command-line tool automatically collects JavaScript or CSS files in our application's folder and links them in the index.html file, which is served to clients on initial page load. This makes structuring our code in separate files as easy as creating them. Meteor's command-line tool also watches all files inside our application's folder for changes and rebuilds them on the fly when they change. Additionally, it starts a Meteor server that serves the app's files to the clients. When a file changes, Meteor reloads the site of every client while preserving its state. This is called a hot code reload. In production, the build process also concatenates and minifies our CSS and JavaScript files. By simply adding the less and coffee core packages, we can even write all styles in LESS and code in CoffeeScript with no extra effort. The command-line tool is also the tool for deploying and bundling our app so that we can run it on a remote server. Sounds awesome? Let's take a look at what's needed to use Meteor Adding basic packages Packages in Meteor are libraries that can be added to our projects. The nice thing about Meteor packages is that they are self-contained units, which run out of the box. They mostly add either some templating functionality or provide extra objects in the global namespace of our project. Packages can also add features to Meteor's build process like the stylus package, which lets us write our app's style files with the stylus pre-processor syntax. Writing templates in Meteor Normally when we build websites, we build the complete HTML on the server side. This was quite straightforward; every page is built on the server, then it is sent to the client, and at last JavaScript added some additional animation or dynamic behavior to it. This is not so in single-page apps, where each page needs to be already in the client's browser so that it can be shown at will. Meteor solves that problem by providing templates that exists in JavaScript and can be placed in the DOM at some point. These templates can have nested templates, allowing for and easy way to reuse and structure an app's HTML layout. Since Meteor is so flexible in terms of folder and file structure, any *.html page can contain a template and will be parsed during Meteor's build process. This allows us to put all templates in the my-meteor-blog/client/templates folder. This folder structure is chosen as it helps us organizing templates while our app grows. Meteor template engine is called Spacebars, which is a derivative of the handlebars template engine. Spacebars is built on top of Blaze, which is Meteor's reactive DOM update engine. Meteor and databases Meteor currently uses MongoDB by default to store data on the server, although there are drivers planned for relational databases, too. If you are adventurous, you can try one of the community-built SQL drivers, such as the numtel:mysql package from https://atmospherejs.com/numtel/mysql. MongoDB is a NoSQL database. This means it is based on a flat document structure instead of a relational table structure. Its document approach makes it ideal for JavaScript as documents are written in BJSON, which is very similar to the JSON format. Meteor has a database everywhere approach, which means we have the same API to query the database on the client as well as on the server. Yet, when we query the database on the client, we are only able to access data that we published to a client. MongoDB uses a datastructure called a collection, which is the equivalent of a table in an SQL database. Collections contain documents, where each document has its own unique ID. These documents are JSON-like structures and can contain properties with values, even with multiple dimensions: { "_id": "W7sBzpBbov48rR7jW", "myName": "My Document Name", "someProperty": 123456, "aNestedProperty": { "anotherOne": "With another string" } } These collections are used to store data in the servers MongoDB as well as the client-sides minimongo collections, which is an in-memory database mimicking the behavior of the real MongoDB. The MongoDB API let us use a simple JSON-based query language to get documents from a collection. We can pass additional options to only ask for specific fields or sort the returned documents. These are very powerful features, especially on the client side, to display data in various ways. Data everywhere In Meteor, we can use the browser console to update data, which means we update the database from the client. This works because Meteor automatically syncs these changes to the server and updates the database accordingly. This is happening because we have the autopublish and insecure core packages added to our project by default. The autopublish package publishes automatically all documents to every client, whereas the insecure package allows every client to update database records by its _id field. Obviously, this works well for prototyping but is infeasible for production, as every client could manipulate our database. If we remove the insecure package, we would need to add the "allow and deny" rules to determine what a client is allowed to update and what not; otherwise all updates will get denied. Differences between client and server collections Meteor has a database everywhere approach. This means it provides the same API on the client as on the server. The data flow is controlled using a publication subscription model. On the server sits the real MongoDB database, which stores data persistently. On the client Meteor has a package called minimongo, which is a pure in-memory database mimicking most of MongoDB's query and update functions. Every time a client connects to its Meteor server, Meteor downloads the documents the client subscribed to and stores them in its local minimongo database. From here, they can be displayed in a template or processed by functions. When the client updates a document, Meteor syncs it back to the server, where it is passed through any allow/deny functions before being persistently stored in the database. This works also in the other way, when a document in the server-side database changes, it will get automatically sync to every client that is subscribed to it, keeping every connected client up to date. Syncing data – the current Web versus the new Web In the current Web, most pages are either static files hosted on a server or dynamically generated by a server on a request. This is true for most server-side-rendered websites, for example, those written with PHP, Rails, or Django. Both of these techniques required no effort besides being displayed by the clients; therefore, they are called thin clients. In modern web applications, the idea of the browser has moved from thin clients to fat clients. This means most of the website's logic resides on the client and the client asks for the data it needs. Currently, this is mostly done via calls to an API server. This API server then returns data, commonly in JSON form, giving the client an easy way to handle it and use it appropriately. Most modern websites are a mixture of thin and fat clients. Normal pages are server-side-rendered, where only some functionality, such as a chat box or news feed, is updated using API calls. Meteor, however, is built on the idea that it's better to use the calculation power of all clients instead of one single server. A pure fat client or a single-page app contains the entire logic of a website's frontend, which is send down on the initial page load. The server then merely acts as a data source, sending only the data to the clients. This can happen by connecting to an API and utilizing AJAX calls, or as with Meteor, using a model called publication/subscription. In this model, the server offers a range of publications and each client decides which dataset it wants to subscribe to. Compared with AJAX calls, the developer doesn't have to take care of any downloading or uploading logic. The Meteor client syncs all of the data automatically in the background as soon as it subscribes to a specific dataset. When data on the server changes, the server sends the updated documents to the clients and vice versa, as shown in the following diagram: Summary Meteor comes with more great ways of building pure JavaScript applications such as simple routing and simple ways to make components, which can be packaged for others to use. Meteor's reactivity model, which allows you to rerun any function and template helpers at will, allows for great consistent interfaces and simple dependency tracking, which is a key for large-scale JavaScript applications. If you want to dig deeper, buy the book and read How to build your own blog as single-page web application in a simple step-by-step fashion by using Meteor, the next generation web! Resources for Article: Further resources on this subject: Quick start - creating your first application [article] Meteor.js JavaScript Framework: Why Meteor Rocks! [article] Marionette View Types and Their Use [article]

In this article by Eric Rochester author of the book, Clojure Data Analysis Cookbook, Second Edition, we will cover the following recipes: Loading Incanter's sample datasets Loading Clojure data structures into datasets Viewing datasets interactively with view Converting datasets to matrices Using infix formulas in Incanter Selecting columns with $ Selecting rows with $ Filtering datasets with $where Grouping data with $group-by Saving datasets to CSV and JSON Projecting from multiple datasets with $join (For more resources related to this topic, see here.) Introduction Incanter combines the power to do statistics using a fully-featured statistical language such as R (http://www.r-project.org/) with the ease and joy of Clojure. Incanter's core data structure is the dataset, so we'll spend some time in this article to look at how to use them effectively. While learning basic tools in this manner is often not the most exciting way to spend your time, it can still be incredibly useful. At its most fundamental level, an Incanter dataset is a table of rows. Each row has the same set of columns, much like a spreadsheet. The data in each cell of an Incanter dataset can be a string or a numeric. However, some operations require the data to only be numeric. First you'll learn how to populate and view datasets, then you'll learn different ways to query and project the parts of the dataset that you're interested in onto a new dataset. Finally, we'll take a look at how to save datasets and merge multiple datasets together. Loading Incanter's sample datasets Incanter comes with a set of default datasets that are useful for exploring Incanter's functions. I haven't made use of them in this book, since there is so much data available in other places, but they're a great way to get a feel of what you can do with Incanter. Some of these datasets—for instance, the Iris dataset—are widely used to teach and test statistical algorithms. It contains the species and petal and sepal dimensions for 50 irises. This is the dataset that we'll access today. In this recipe, we'll load a dataset and see what it contains. Getting ready We'll need to include Incanter in our Leiningen project.clj file: (defproject inc-dsets "0.1.0":dependencies [[org.clojure/clojure "1.6.0"]                 [incanter "1.5.5"]]) We'll also need to include the right Incanter namespaces into our script or REPL: (use '(incanter core datasets)) How to do it… Once the namespaces are available, we can access the datasets easily: user=> (def iris (get-dataset :iris))#'user/iris user=> (col-names iris)[:Sepal.Length :Sepal.Width :Petal.Length :Petal.Width :Species]user=> (nrow iris)150 user=> (set ($ :Species iris))#{"versicolor" "virginica" "setosa"} How it works… We use the get-dataset function to access the built-in datasets. In this case, we're loading the Fisher's Iris dataset, sometimes called Anderson's dataset. This is a multivariate dataset for discriminant analysis. It gives petal and sepal measurements for 150 different Irises of three different species. Incanter's sample datasets cover a wide variety of topics—from U.S. arrests to plant growth and ultrasonic calibration. They can be used to test different algorithms and analyses and to work with different types of data. By the way, the names of functions should be familiar to you if you've previously used R. Incanter often uses the names of R's functions instead of using the Clojure names for the same functions. For example, the preceding code sample used nrow instead of count. There's more... Incanter's API documentation for get-dataset (http://liebke.github.com/incanter/datasets-api.html#incanter.datasets/get-dataset) lists more sample datasets, and you can refer to it for the latest information about the data that Incanter bundles. Loading Clojure data structures into datasets While they are good for learning, Incanter's built-in datasets probably won't be that useful for your work (unless you work with irises). Other recipes cover ways to get data from CSV files and other sources into Incanter. Incanter also accepts native Clojure data structures in a number of formats. We'll take look at a couple of these in this recipe. Getting ready We'll just need Incanter listed in our project.clj file: (defproject inc-dsets "0.1.0":dependencies [[org.clojure/clojure "1.6.0"]                 [incanter "1.5.5"]]) We'll also need to include this in our script or REPL: (use 'incanter.core) How to do it… The primary function used to convert data into a dataset is to-dataset. While it can convert single, scalar values into a dataset, we'll start with slightly more complicated inputs. Generally, you'll be working with at least a matrix. If you pass this to to-dataset, what do you get? user=> (def matrix-set (to-dataset [[1 2 3] [4 5 6]]))#'user/matrix-set user=> (nrow matrix-set)2user=> (col-names matrix-set)[:col-0 :col-1 :col-2] All the data's here, but it can be labeled in a better way. Does to-dataset handle maps? user=> (def map-set (to-dataset {:a 1, :b 2, :c 3}))#'user/map-set user=> (nrow map-set)1 user=> (col-names map-set)[:a :c :b] So, map keys become the column labels. That's much more intuitive. Let's throw a sequence of maps at it: user=> (def maps-set (to-dataset [{:a 1, :b 2, :c 3},                                 {:a 4, :b 5, :c 6}]))#'user/maps-setuser=> (nrow maps-set)2user=> (col-names maps-set)[:a :c :b] This is much more useful. We can also create a dataset by passing the column vector and the row matrix separately to dataset: user=> (def matrix-set-2         (dataset [:a :b :c]                         [[1 2 3] [4 5 6]]))#'user/matrix-set-2 user=> (nrow matrix-set-2)2 user=> (col-names matrix-set-2)[:c :b :a] How it works… The to-dataset function looks at the input and tries to process it intelligently. If given a sequence of maps, the column names are taken from the keys of the first map in the sequence. Ultimately, it uses the dataset constructor to create the dataset. When you want the most control, you should also use the dataset. It requires the dataset to be passed in as a column vector and a row matrix. When the data is in this format or when we need the most control—to rename the columns, for instance—we can use dataset. Viewing datasets interactively with view Being able to interact with our data programmatically is important, but sometimes it's also helpful to be able to look at it. This can be especially useful when you do data exploration. Getting ready We'll need to have Incanter in our project.clj file and script or REPL, so we'll use the same setup as we did for the Loading Incanter's sample datasets recipe, as follows. We'll also use the Iris dataset from that recipe. (use '(incanter core datasets)) How to do it… Incanter makes this very easy. Let's take a look at just how simple it is: First, we need to load the dataset, as follows: user=> (def iris (get-dataset :iris)) #'user/iris Then we just call view on the dataset: user=> (view iris) This function returns the Swing window frame, which contains our data, as shown in the following screenshot. This window should also be open on your desktop, although for me, it's usually hiding behind another window: How it works… Incanter's view function takes any object and tries to display it graphically. In this case, it simply displays the raw data as a table. Converting datasets to matrices Although datasets are often convenient, many times we'll want to treat our data as a matrix from linear algebra. In Incanter, matrices store a table of doubles. This provides good performance in a compact data structure. Moreover, we'll need matrices many times because some of Incanter's functions, such as trans, only operate on a matrix. Plus, it implements Clojure's ISeq interface, so interacting with matrices is also convenient. Getting ready For this recipe, we'll need the Incanter libraries, so we'll use this project.clj file: (defproject inc-dsets "0.1.0":dependencies [[org.clojure/clojure "1.6.0"]                 [incanter "1.5.5"]]) We'll use the core and io namespaces, so we'll load these into our script or REPL: (use '(incanter core io)) This line binds the file name to the identifier data-file: (def data-file "data/all_160_in_51.P35.csv") How to do it… For this recipe, we'll create a dataset, convert it to a matrix, and then perform some operations on it: First, we need to read the data into a dataset, as follows: (def va-data (read-dataset data-file :header true)) Then, in order to convert it to a matrix, we just pass it to the to-matrix function. Before we do this, we'll pull out a few of the columns since matrixes can only contain floating-point numbers: (def va-matrix    (to-matrix ($ [:POP100 :HU100 :P035001] va-data))) Now that it's a matrix, we can treat it like a sequence of rows. Here, we pass it to first in order to get the first row, take in order to get a subset of the matrix, and count in order to get the number of rows in the matrix: user=> (first va-matrix) A 1x3 matrix ------------- 8.19e+03 4.27e+03 2.06e+03   user=> (count va-matrix) 591 We can also use Incanter's matrix operators to get the sum of each column, for instance. The plus function takes each row and sums each column separately: user=> (reduce plus va-matrix) A 1x3 matrix ------------- 5.43e+06 2.26e+06 1.33e+06 How it works… The to-matrix function takes a dataset of floating-point values and returns a compact matrix. Matrices are used by many of Incanter's more sophisticated analysis functions, as they're easy to work with. There's more… In this recipe, we saw the plus matrix operator. Incanter defines a full suite of these. You can learn more about matrices and see what operators are available at https://github.com/liebke/incanter/wiki/matrices. Using infix formulas in Incanter There's a lot to like about lisp: macros, the simple syntax, and the rapid development cycle. Most of the time, it is fine if you treat math operators as functions and use prefix notations, which is a consistent, function-first syntax. This allows you to treat math operators in the same way as everything else so that you can pass them to reduce, or anything else you want to do. However, we're not taught to read math expressions using prefix notations (with the operator first). And especially when formulas get even a little complicated, tracing out exactly what's happening can get hairy. Getting ready For this recipe we'll just need Incanter in our project.clj file, so we'll use the dependencies statement—as well as the use statement—from the Loading Clojure data structures into datasets recipe. For data, we'll use the matrix that we created in the Converting datasets to matrices recipe. How to do it… Incanter has a macro that converts a standard math notation to a lisp notation. We'll explore that in this recipe: The $= macro changes its contents to use an infix notation, which is what we're used to from math class: user=> ($= 7 * 4)28user=> ($= 7 * 4 + 3)31 We can also work on whole matrixes or just parts of matrixes. In this example, we perform a scalar multiplication of the matrix: user=> ($= va-matrix * 4)A 591x3 matrix---------------3.28e+04 1.71e+04 8.22e+03 2.08e+03 9.16e+02 4.68e+02 1.19e+03 6.52e+02 3.08e+02...1.41e+03 7.32e+02 3.72e+02 1.31e+04 6.64e+03 3.49e+03 3.02e+04 9.60e+03 6.90e+03 user=> ($= (first va-matrix) * 4)A 1x3 matrix-------------3.28e+04 1.71e+04 8.22e+03 Using this, we can build complex expressions, such as this expression that takes the mean of the values in the first row of the matrix: user=> ($= (sum (first va-matrix)) /           (count (first va-matrix)))4839.333333333333 Or we can build expressions take the mean of each column, as follows: user=> ($= (reduce plus va-matrix) / (count va-matrix))A 1x3 matrix-------------9.19e+03 3.83e+03 2.25e+03 How it works… Any time you're working with macros and you wonder how they work, you can always get at their output expressions easily, so you can see what the computer is actually executing. The tool to do this is macroexpand-1. This expands the macro one step and returns the result. It's sibling function, macroexpand, expands the expression until there is no macro expression left. Usually, this is more than we want, so we just use macroexpand-1. Let's see what these macros expand into: user=> (macroexpand-1 '($= 7 * 4))(incanter.core/mult 7 4)user=> (macroexpand-1 '($= 7 * 4 + 3))(incanter.core/plus (incanter.core/mult 7 4) 3)user=> (macroexpand-1 '($= 3 + 7 * 4))(incanter.core/plus 3 (incanter.core/mult 7 4)) Here, we can see that the expression doesn't expand into Clojure's * or + functions, but it uses Incanter's matrix functions, mult and plus, instead. This allows it to handle a variety of input types, including matrices, intelligently. Otherwise, it switches around the expressions the way we'd expect. Also, we can see by comparing the last two lines of code that it even handles operator precedence correctly. Selecting columns with $ Often, you need to cut the data to make it more useful. One common transformation is to pull out all the values from one or more columns into a new dataset. This can be useful for generating summary statistics or aggregating the values of some columns. The Incanter macro $ slices out parts of a dataset. In this recipe, we'll see this in action. Getting ready For this recipe, we'll need to have Incanter listed in our project.clj file: (defproject inc-dsets "0.1.0":dependencies [[org.clojure/clojure "1.6.0"]                 [incanter "1.5.5"]                [org.clojure/data.csv "0.1.2"]]) We'll also need to include these libraries in our script or REPL: (require '[clojure.java.io :as io]         '[clojure.data.csv :as csv]         '[clojure.string :as str]         '[incanter.core :as i]) Moreover, we'll need some data. This time, we'll use some country data from the World Bank. Point your browser to http://data.worldbank.org/country and select a country. I picked China. Under World Development Indicators, there is a button labeled Download Data. Click on this button and select CSV. This will download a ZIP file. I extracted its contents into the data/chn directory in my project. I bound the filename for the primary data file to the data-file name. How to do it… We'll use the $ macro in several different ways to get different results. First, however, we'll need to load the data into a dataset, which we'll do in steps 1 and 2: Before we start, we'll need a couple of utilities that load the data file into a sequence of maps and makes a dataset out of those: (defn with-header [coll] (let [headers (map #(keyword (str/replace % space -))                      (first coll))]    (map (partial zipmap headers) (next coll))))   (defn read-country-data [filename] (with-open [r (io/reader filename)]    (i/to-dataset      (doall (with-header                (drop 2 (csv/read-csv r))))))) Now, using these functions, we can load the data: user=> (def chn-data (read-country-data data-file)) We can select columns to be pulled out from the dataset by passing the column names or numbers to the $ macro. It returns a sequence of the values in the column: user=> (i/$ :Indicator-Code chn-data) ("AG.AGR.TRAC.NO" "AG.CON.FERT.PT.ZS" "AG.CON.FERT.ZS" … We can select more than one column by listing all of them in a vector. This time, the results are in a dataset: user=> (i/$ [:Indicator-Code :1992] chn-data)   |           :Indicator-Code |               :1992 | |---------------------------+---------------------| |           AG.AGR.TRAC.NO |             770629 | |         AG.CON.FERT.PT.ZS |                     | |           AG.CON.FERT.ZS |                     | |           AG.LND.AGRI.K2 |             5159980 | … We can list as many columns as we want, although the formatting might suffer: user=> (i/$ [:Indicator-Code :1992 :2002] chn-data)   |           :Indicator-Code |               :1992 |               :2002 | |---------------------------+---------------------+---------------------| |           AG.AGR.TRAC.NO |            770629 |                     | |         AG.CON.FERT.PT.ZS |                     |     122.73027213719 | |           AG.CON.FERT.ZS |                     |   373.087159048868 | |           AG.LND.AGRI.K2 |             5159980 |             5231970 | … How it works… The $ function is just a wrapper over Incanter's sel function. It provides a good way to slice columns out of the dataset, so we can focus only on the data that actually pertains to our analysis. There's more… The indicator codes for this dataset are a little cryptic. However, the code descriptions are in the dataset too: user=> (i/$ [0 1 2] [:Indicator-Code :Indicator-Name] chn-data)   |   :Indicator-Code |                                               :Indicator-Name | |-------------------+---------------------------------------------------------------| |   AG.AGR.TRAC.NO |                             Agricultural machinery, tractors | | AG.CON.FERT.PT.ZS |           Fertilizer consumption (% of fertilizer production) | |   AG.CON.FERT.ZS | Fertilizer consumption (kilograms per hectare of arable land) | … See also… For information on how to pull out specific rows, see the next recipe, Selecting rows with $. Selecting rows with $ The Incanter macro $ also pulls rows out of a dataset. In this recipe, we'll see this in action. Getting ready For this recipe, we'll use the same dependencies, imports, and data as we did in the Selecting columns with $ recipe. How to do it… Similar to how we use $ in order to select columns, there are several ways in which we can use it to select rows, shown as follows: We can create a sequence of the values of one row using $, and pass it the index of the row we want as well as passing :all for the columns: user=> (i/$ 0 :all chn-data) ("AG.AGR.TRAC.NO" "684290" "738526" "52661" "" "880859" "" "" "" "59657" "847916" "862078" "891170" "235524" "126440" "469106" "282282" "817857" "125442" "703117" "CHN" "66290" "705723" "824113" "" "151281" "669675" "861364" "559638" "191220" "180772" "73021" "858031" "734325" "Agricultural machinery, tractors" "100432" "" "796867" "" "China" "" "" "155602" "" "" "770629" "747900" "346786" "" "398946" "876470" "" "795713" "" "55360" "685202" "989139" "798506" "") We can also pull out a dataset containing multiple rows by passing more than one index into $ with a vector (There's a lot of data, even for three rows, so I won't show it here): (i/$ (range 3) :all chn-data) We can also combine the two ways to slice data in order to pull specific columns and rows. We can either pull out a single row or multiple rows: user=> (i/$ 0 [:Indicator-Code :1992] chn-data) ("AG.AGR.TRAC.NO" "770629") user=> (i/$ (range 3) [:Indicator-Code :1992] chn-data)   |   :Indicator-Code | :1992 | |-------------------+--------| |   AG.AGR.TRAC.NO | 770629 | | AG.CON.FERT.PT.ZS |       | |   AG.CON.FERT.ZS |       | How it works… The $ macro is the workhorse used to slice rows and project (or select) columns from datasets. When it's called with two indexing parameters, the first is the row or rows and the second is the column or columns. Filtering datasets with $where While we can filter datasets before we import them into Incanter, Incanter makes it easy to filter and create new datasets from the existing ones. We'll take a look at its query language in this recipe. Getting ready We'll use the same dependencies, imports, and data as we did in the Selecting columns with $ recipe. How to do it… Once we have the data, we query it using the $where function: For example, this creates a dataset with a row for the percentage of China's total land area that is used for agriculture: user=> (def land-use          (i/$where {:Indicator-Code "AG.LND.AGRI.ZS"}                    chn-data)) user=> (i/nrow land-use) 1 user=> (i/$ [:Indicator-Code :2000] land-use) ("AG.LND.AGRI.ZS" "56.2891584865366") The queries can be more complicated too. This expression picks out the data that exists for 1962 by filtering any empty strings in that column: user=> (i/$ (range 5) [:Indicator-Code :1962]          (i/$where {:1962 {:ne ""}} chn-data))   |   :Indicator-Code |             :1962 | |-------------------+-------------------| |   AG.AGR.TRAC.NO |             55360 | |   AG.LND.AGRI.K2 |           3460010 | |   AG.LND.AGRI.ZS | 37.0949187612906 | |   AG.LND.ARBL.HA |         103100000 | | AG.LND.ARBL.HA.PC | 0.154858284392508 | Incanter's query language is even more powerful than this, but these examples should show you the basic structure and give you an idea of the possibilities. How it works… To better understand how to use $where, let's break apart the last example: ($i/where {:1962 {:ne ""}} chn-data) The query is expressed as a hashmap from fields to values (highlighted). As we saw in the first example, the value can be a raw value, either a literal or an expression. This tests for inequality. ($i/where {:1962 {:ne ""}} chn-data) Each test pair is associated with a field in another hashmap (highlighted). In this example, both the hashmaps shown only contain one key-value pair. However, they might contain multiple pairs, which will all be ANDed together. Incanter supports a number of test operators. The basic boolean tests are :$gt (greater than), :$lt (less than), :$gte (greater than or equal to), :$lte (less than or equal to), :$eq (equal to), and :$ne (not equal). There are also some operators that take sets as parameters: :$in and :$nin (not in). The last operator—:$fn—is interesting. It allows you to use any predicate function. For example, this will randomly select approximately half of the dataset: (def random-half (i/$where {:Indicator-Code {:$fn (fn [_] (< (rand) 0.5))}}            chnchn-data)) There's more… For full details of the query language, see the documentation for incanter.core/query-dataset (http://liebke.github.com/incanter/core-api.html#incanter.core/query-dataset). Grouping data with $group-by Datasets often come with an inherent structure. Two or more rows might have the same value in one column, and we might want to leverage that by grouping those rows together in our analysis. Getting ready First, we'll need to declare a dependency on Incanter in the project.clj file: (defproject inc-dsets "0.1.0" :dependencies [[org.clojure/clojure "1.6.0"]                  [incanter "1.5.5"]                  [org.clojure/data.csv "0.1.2"]]) Next, we'll include Incanter core and io in our script or REPL: (require '[incanter.core :as i]          '[incanter.io :as i-io]) For data, we'll use the census race data for all the states. You can download it from http://www.ericrochester.com/clj-data-analysis/data/all_160.P3.csv. These lines will load the data into the race-data name: (def data-file "data/all_160.P3.csv") (def race-data (i-io/read-dataset data-file :header true)) How to do it… Incanter lets you group rows for further analysis or to summarize them with the $group-by function. All you need to do is pass the data to $group-by with the column or function to group on: (def by-state (i/$group-by :STATE race-data)) How it works… This function returns a map where each key is a map of the fields and values represented by that grouping. For example, this is how the keys look: user=> (take 5 (keys by-state)) ({:STATE 29} {:STATE 28} {:STATE 31} {:STATE 30} {:STATE 25}) We can get the data for Virginia back out by querying the group map for state 51. user=> (i/$ (range 3) [:GEOID :STATE :NAME :POP100]            (by-state {:STATE 51}))   | :GEOID | :STATE |         :NAME | :POP100 | |---------+--------+---------------+---------| | 5100148 |     51 | Abingdon town |   8191 | | 5100180 |     51 | Accomac town |     519 | | 5100724 |     51 | Alberta town |     298 | Saving datasets to CSV and JSON Once you've done the work of slicing, dicing, cleaning, and aggregating your datasets, you might want to save them. Incanter by itself doesn't have a good way to do this. However, with the help of some Clojure libraries, it's not difficult at all. Getting ready We'll need to include a number of dependencies in our project.clj file: (defproject inc-dsets "0.1.0":dependencies [[org.clojure/clojure "1.6.0"]                 [incanter "1.5.5"]                 [org.clojure/data.csv "0.1.2"]                 [org.clojure/data.json "0.2.5"]]) We'll also need to include these libraries in our script or REPL: (require '[incanter.core :as i]          '[incanter.io :as i-io]          '[clojure.data.csv :as csv]          '[clojure.data.json :as json]          '[clojure.java.io :as io]) Also, we'll use the same data that we introduced in the Selecting columns with $ recipe. How to do it… This process is really as simple as getting the data and saving it. We'll pull out the data for the year 2000 from the larger dataset. We'll use this subset of the data in both the formats here: (def data2000 (i/$ [:Indicator-Code :Indicator-Name :2000] chn-data)) Saving data as CSV To save a dataset as a CSV, all in one statement, open a file and use clojure.data.csv/write-csv to write the column names and data to it: (with-open [f-out (io/writer "data/chn-2000.csv")] (csv/write-csv f-out [(map name (i/col-names data2000))]) (csv/write-csv f-out (i/to-list data2000))) Saving data as JSON To save a dataset as JSON, open a file and use clojure.data.json/write to serialize the file: (with-open [f-out (io/writer "data/chn-2000.json")] (json/write (:rows data2000) f-out)) How it works… For CSV and JSON, as well as many other data formats, the process is very similar. Get the data, open the file, and serialize data into it. There will be differences in how the output function wants the data (to-list or :rows), and there will be differences in how the output function is called (for instance, whether the file handle is the first or second argument). But generally, outputting datasets will be very similar and relatively simple. Projecting from multiple datasets with $join So far, we've been focusing on splitting up datasets, on dividing them into groups of rows or groups of columns with functions and macros such as $ or $where. However, sometimes we'd like to move in the other direction. We might have two related datasets and want to join them together to make a larger one. For example, we might want to join crime data to census data, or take any two related datasets that come from separate sources and analyze them together. Getting ready First, we'll need to include these dependencies in our project.clj file: (defproject inc-dsets "0.1.0" :dependencies [[org.clojure/clojure "1.6.0"]                 [incanter "1.5.5"]                  [org.clojure/data.csv "0.1.2"]]) We'll use these statements for inclusions: (require '[clojure.java.io :as io]          '[clojure.data.csv :as csv]          '[clojure.string :as str]          '[incanter.core :as i]) For our data file, we'll use the same data that we introduced in the Selecting columns with $ recipe: China's development dataset from the World Bank. How to do it… In this recipe, we'll take a look at how to join two datasets using Incanter: To begin with, we'll load the data from the data/chn/chn_Country_en_csv_v2.csv file. We'll use the with-header and read-country-data functions that were defined in the Selecting columns with $ recipe: (def data-file "data/chn/chn_Country_en_csv_v2.csv") (def chn-data (read-country-data data-file)) Currently, the data for each row contains the data for one indicator across many years. However, for some analyses, it will be more helpful to have each row contain the data for one indicator for one year. To do this, let's first pull out the data from 2 years into separate datasets. Note that for the second dataset, we'll only include a column to match the first dataset (:Indicator-Code) and the data column (:2000): (def chn-1990 (i/$ [:Indicator-Code :Indicator-Name :1990]        chn-data)) (def chn-2000 (i/$ [:Indicator-Code :2000] chn-data)) Now, we'll join these datasets back together. This is contrived, but it's easy to see how we will do this in a more meaningful example. For example, we might want to join the datasets from two different countries: (def chn-decade (i/$join [:Indicator-Code :Indicator-Code]            chn-1990 chn-2000)) From this point on, we can use chn-decade just as we use any other Incanter dataset. How it works… Let's take a look at this in more detail: (i/$join [:Indicator-Code :Indicator-Code] chn-1990 chn-2000) The pair of column keywords in a vector ([:Indicator-Code :Indicator-Code]) are the keys that the datasets will be joined on. In this case, the :Indicator-Code column from both the datasets is used, but the keys can be different for the two datasets. The first column that is listed will be from the first dataset (chn-1990), and the second column that is listed will be from the second dataset (chn-2000). This returns a new dataset. Each row of this new dataset is a superset of the corresponding rows from the two input datasets. Summary In this article we have covered covers the basics of working with Incanter datasets. Datasets are the core data structures used by Incanter, and understanding them is necessary in order to use Incanter effectively. Resources for Article: Further resources on this subject: The Hunt for Data [article] Limits of Game Data Analysis [article] Clojure for Domain-specific Languages - Design Concepts with Clojure [article]

The First Step In this article by Tim Chaplin, author of the book AngularJS Test-driven Development, provides an initial introductory walk-through of how to use TDD to build an AngularJS application with a controller, model, and scope. You will be able to begin the TDD journey and see the fundamentals in action. Now, we will switch gears and dive into TDD with AngularJS. This article will be the first step of TDD. This article will focus on the creation of social media comments. It will also focus on the testing associated with controllers and the use of Angular mocks to AngularJS components in a test. (For more resources related to this topic, see here.) Preparing the application's specification Create an application to enter comments. The specification of the application is as follows: Given I am posting a new comment, when I click on the submit button, the comment should be added to the to-do list Given a comment, when I click on the like button, the number of likes for the comment should be increased Now that we have the specification of application, we can create our development to-do list. It won't be easy to create an entire to-do list of the whole application. Based on the user specifications, we have an idea of what needs to be developed. Here is a rough sketch of the UI: Hold yourself back from jumping into the implementation and thinking about how you will use a controller with a service, ng-repeat, and so on. Resist, resist, resist! Although you can think of how this will be developed in the future, it is never clear until you delve into the code, and that is where you start getting into trouble. TDD and its principles are here to help you get your mind and focus in the right place. Setting up the project I will provide a list in the following section for the initial actions to get the project set up. Setting up the directory The following instructions are specific to setting up the project directory: Create a new project directory. Get angular into the project using Bower: bower install angular Get angular-mocks for testing using Bower: bower install angular-mocks Initialize the application's source directory: mkdir app Initialize the test directory: mkdir spec Initialize the unit test directory: mkdir spec/unit Initialize the end-to-end test directory: mkdir spec/e2e Once the initialization is complete, your folder structure should look as follows: Setting up Protractor In this article, we will just discuss the steps at a higher level: Install Protractor in the project: $ npm install protractor Update Selenium WebDriver: $ ./node_modules/protractor/bin/webdriver-manager update Make sure that Selenium has been installed. Copy the example chromeOnly configuration into the root of the project: $ cp ./node_modules/protractor/example/chromeOnlyConf.js . Configure the Protractor configuration using the following steps: Open the Protractor configuration. Edit the Selenium WebDriver location to reflect the relative directory to chromeDriver: chromeDriver: './node_modules/protractor/selenium/chromedriver', Edit the files section to reflect the test directory: specs: ['spec/e2e/**/*.js'], Set the default base URL: baseUrl: 'http://localhost:8080/', Excellent! Protractor should now be installed and set up. Here is the complete configuration: exports.config = { chromeOnly: true, chromeDriver: './node_modules/protractor/selenium/chromedriver', capabilities: { 'browserName': 'chrome' }, baseUrl: 'http://localhost:8080/', specs: ['spec/e2e/**/*.js'], }; Setting up Karma Here is a brief summary of the steps required to install and get your new project set up: Install Karma using the following command: npm install karma -g Initialize the Karma configuration: karma init Update the Karma configuration: files: [ 'bower_components/angular/angular.js', 'bower_components/angular-mocks/angular-mocks.js', 'spec/unit/**/*.js' ], Now that we have set up the project directory and initialized Protractor and Karma, we can dive into the code. Here is the complete karma.conf.js file: module.exports = function(config) { config.set({ basePath: '', frameworks: ['jasmine'], files: [ 'bower_components/angular/angular.js', 'bower_components/angular-mocks/angular-mocks.js', 'spec/unit/**/*.js' ], reporters: ['progress'], port: 9876, autoWatch: true, browsers: ['Chrome'], singleRun: false }); }; Setting up http-server A web server will be used to host the application. As this will just be for local development only, you can use http-server. The http-server module is a simple HTTP server that serves static content. It is available as an npm module. To install http-server in your project, type the following command: $ npm install http-server Once http-server is installed, you can run the server by providing it with the root directory of the web page. Here is an example: $ ./node_modules/http-server/bin/http-server Now that you have http-server installed, you can move on to the next step. Top-down or bottom-up approach From our development perspective, we have to determine where to start. The approaches that we will discuss in this article are as follows: The bottom-up approach: With this approach, we think about the different components we will need (controller, service, module, and so on) and then pick the most logical one and start coding. The top-down approach: With this approach, we work from the user scenario and UI. We then create the application around the components in the application. There are merits to both types of approaches and the choice can be based on your team, existing components, requirements, and so on. In most cases, it is best for you to make the choice based on the least resistance. In this article, the approach of specification is top-down, everything is laid out for us from the user scenario and will allow you to organically build the application around the UI. Testing a controller Before getting into the specification, and the mind-set of the feature being delivered, it is important to see the fundamentals of testing a controller. An AngularJS controller is a key component used in most applications. A simple controller test setup When testing a controller, tests are centered on the controller's scope. The tests confirm either the objects or methods in the scope. Angular mocks provide inject, which finds a particular reference and returns it for you to use. When inject is used for the controller, the controllers scope can be assigned to an outer reference for the entire test to use. Here is an example of what this would look like: describe('',function(){ var scope = {}; beforeEach(function(){ module('anyModule'); inject(function($controller){ $controller('AnyController',{$scope:scope}); }); }); }); In the preceding case, the test's scope object is assigned to the actual scope of the controller within the inject function. The scope object can now be used throughout the test, and is also reinitialized before each test. Initializing the scope In the preceding example, scope is initialized to an object {}. This is not the best approach; just like a page, a controller might be nested within another controller. This will cause inheritance of a parent scope as follows: <body ng-app='anyModule'> <div ng-controller='ParentController'> <div ng-controller='ChildController'> </div> </div> </body> As seen in the preceding code, we have this hierarchy of scopes that the ChildController function has access to. In order to test this, we have to initialize the scope object properly in the inject function. Here is how the preceding scope hierarchy can be recreated: inject(function($controller,$rootScope){ var parentScope = $rootScope.$new(); $controller('ParentController',{$scope:parentScope}); var childScope = parentScope.$new(); $controller('AnyController',{$scope: childScope}); }); There are two main things that the preceding code does: The $rootScope scope is injected into the test. The $rootScope scope is the highest level of scope that exists. Each level of scope is created with the $new() method. This method creates the child scope. In this article, we will use the simplified version and initialize the scope to an empty object; however, it is important to understand how to create the scope when required. Bring on the comments Now that the setup and approach have been decided, we can start our first test. From a testing point of view, as we will be using a top-down approach, we will write our Protractor tests first and then build the application. We will follow the same TDD life cycle we have already reviewed, that is, test first, make it run, and make it better. Test first The scenario given is in a well-specified format already and fits our Protractor testing template: describe('',function(){ beforeEach(function(){ }); it('',function(){ }); }); Placing the scenario in the template, we get the following code: describe('Given I am posting a new comment',function(){ describe('When I push the submit button',function(){ beforeEach(function(){ }); it('Should then add the comment',function(){ }); }); }); Following the 3 A's (Assemble, Act, Assert), we will fit the user scenario in the template. Assemble The browser will need to point to the first page of the application. As the base URL has already been defined, we can add the following to the test: beforeEach(function(){ browser.get('/'); }); Now that the test is prepared, we can move on to the next step, Act. Act The next thing we need to do, based on the user specification, is add an actual comment. The easiest thing is to just put some text into an input box. The test for this, again without knowing what the element will be called or what it will do, is to write it based on what it should be. Here is the code to add the comment section for the application: beforeEach(function(){ ... var commentInput = $('input'); commentInput.sendKeys('a comment'); }); The last assemble component, as part of the test, is to push the Submit button. This can be easily achieved in Protractor using the click function. Even though we don't have a page yet, or any attributes, we can still name the button that will be created: beforeEach(function(){ ... var submitButton = element.all(by.buttonText('Submit')).click(); }); Finally, we will hit the crux of the test and assert the users' expectations. Assert The user expectation is that once the Submit button is clicked, the comment is added. This is a little ambiguous, but we can determine that somehow the user needs to get notified that the comment was added. The simplest approach is to display all comments on the page. In AngularJS, the easiest way to do this is to add an ng-repeat object that displays all comments. To test this, we will add the following: it('Should then add the comment',function(){ var comments = element(by.repeater('comment in comments')).first(); expect(comment.getText()).toBe('a comment'); }); Now, the test has been constructed and meets the user specifications. It is small and concise. Here is the completed test: describe('Given I am posting a new comment',function(){ describe('When I push the submit button',function(){ beforeEach(function(){ //Assemble browser.get('/'); var commentInput = $('input'); commentInput.sendKeys('a comment'); //Act //Act var submitButton = element.all(by.buttonText('Submit')). click(); }); //Assert it('Should then add the comment',function(){ var comments = element(by.repeater('comment in comments')).first(); expect(comment.getText()).toBe('a comment'); }); }); }); Make it run Based on the errors and output of the test, we will build our application as we go. The first step to make the code run is to identify the errors. Before starting off the site, let's create a bare bones index.html page: <!DOCTYPE html> <html> <head> <title></title> </head> <body> </body> </html> Already anticipating the first error, add AngularJS as a dependency in the page: <script type='text/javascript' src='bower_components/angular/angular.js'></script> </body> Now, starting the web server using the following command: $ ./node_modules/http-server/bin/http-server -p 8080 Run Protractor to see the first error: $ ./node_modules/.bin/protractor chromeOnlyConf.js Our first error states that AngularJS could not be found: Error: Angular could not be found on the page http://localhost:8080/ : angular never provided resumeBootstrap This is because we need to add ng-app to the page. Let's create a module and add it to the page. The complete HTML page now looks as follows: <!DOCTYPE html> <html> <head> <title></title> </head> <body> <script src="bower_components/angular/angular.js"></script> </body> </html> Adding the module The first component that you need to define is an ng-app attribute in the index.html page. Use the following steps to add the module: Add ng-app as an attribute to the body tag: <body ng-app='comments'> Now, we can go ahead and create a simple comments module and add it to a file named comments.js: angular.module('comments',[]); Add this new file to index.html: <script src='app/commentController.js'></script> Rerun the Protractor test to get the next error: $ Error: No element found using locator: By.cssSelector('input') The test couldn't find our input locator. You need to add the input to the page. Adding the input Here are the steps you need to follow to add the input to the page: All we have to do is add a simple input tag to the page: <input type='text' /> Run the test and see what the new output is: $ Error: No element found using locator: by.buttonText('Submit') Just like the previous error, we need to add a button with the appropriate text: <button type='button'>Submit</button> Run the test again and the next error is as follows: $ Error: No element found using locator: by.repeater('comment in comments') This appears to be from our expectation that a submitted comment will be available on the page through ng-repeat. To add this to the page, we will use a controller to provide the data for the repeater. Controller As we mentioned in the preceding section, the error is because there is no comments object. In order to add the comments object, we will use a controller that has an array of comments in its scope. Use the following steps to add a comments object in the scope: Create a new file in the app directory named commentController.js: angular.module('comments') .controller('CommentController',['$scope', function($scope){ $scope.comments = []; }]) Add it to the web page after the AngularJS script: <script src='app/commentController.js'></script> Now, we can add commentController to the page: <div ng-controller='CommentController'> Then, add a repeater for the comments as follows: <ul ng-repeat='comment in comments'> <li>{{comment}}</li> </ul> Run the Protractor test and let's see where we are: $ Error: No element found using locator: by.repeater('comment in comments') Hmmm! We get the same error. Let's look at the actual page that gets rendered and see what's going on. In Chrome, go to http://localhost:8080 and open the console to see the page source (Ctrl + Shift + J). You should see something like what's shown in the following screenshot: Notice that the repeater and controller are both there; however, the repeater is commented out. Since Protractor is only looking at visible elements, it won't find the repeater. Great! Now we know why the repeater isn't visible, but we have to fix it. In order for a comment to show up, it has to exist on the controller's comments scope. The smallest change is to add something to the array to initialize it as shown in the following code snippet: .controller('CommentController',['$scope',function($scope){ $scope.comments = ['anything']; }]); Now run the test and we get the following: $ Expected 'anything' to be 'a comment'. Wow! We finally tackled all the errors and reached the expectation. Here is what the HTML code looks like so far: <!DOCTYPE html> <html> <head> <title></title> </head> <body ng-app='comments'> <div ng-controller='CommentController'> <input type='text' /> <ul> <li ng-repeat='comment in comments'> {{comment.value}} </li> </ul> </div> <script src='bower_components/angular/angular.js'></script> <script src='app/comments.js'></script> <script src='app/commentController.js'></script> </body> </html> The comments.js module looks as follows: angular.module('comments',[]); Here is commentController.js: angular.module('comments') .controller('CommentController',['$scope', function($scope){ $scope.comments = []; }]) Make it pass With TDD, you want to add the smallest possible component to make the test pass. Since we have hardcoded, for the moment, the comments to be initialized to anything, change anything to a comment; this should make the test pass. Here is the code to make the test pass: angular.module('comments') .controller('CommentController',['$scope', function($scope){ $scope.comments = ['a comment']; }]); … Run the test, and bam! We get a passing test: $ 1 test, 1 assertion, 0 failures Wait a second! We still have some work to do. Although we got the test to pass, it is not done. We added some hacks just to get the test passing. The two things that stand out are: Clicking on the Submit button, which really doesn't have any functionality Hardcoded initialization of the expected value for a comment The preceding changes are critical steps we need to perform before we move forward. They will be tackled in the next phase of the TDD life cycle, that is, make it better (refactor). Summary In this article, we walked through the TDD techniques of using Protractor and Karma together. As the application was developed, you were able to see where, why, and how to apply the TDD testing tools and techniques. With the bottom-up approach, the specifications are used to build unit tests and then build the UI layer on top of that. In this article, a top-down approach was shown to focus on the user's behavior. The top-down approach tests the UI and then filters the development through the other layers. Resources for Article: Further resources on this subject: AngularJS Project [Article] Role of AngularJS [Article] Creating Our First Animation in AngularJS [Article]

In this article by Aamir Lakhani and Joseph Muniz, authors of the book Penetration Testing with Raspberry Pi, we will see the various LAN- and wireless-based attack scenarios, using tools found in Kali Linux that are optimized for a Raspberry Pi. These scenarios include scanning, analyzing and capturing network traffic. (For more resources related to this topic, see here.) The Raspberry Pi has limited performance capabilities due to its size and processing power. It is highly recommended that you test the following techniques in a lab prior to using a Raspberry Pi for a live penetration test. Network scanning Network reconnaissance is typically time-consuming, yet it is the most important step when performing a penetration test. The more you know about your target, the more likely it is that you will find the fastest and easiest path to success. The best practice is starting with reconnaissance methods that do not require you to interact with your target; however, you will need to make contact eventually. Upon making contact, you will need to identify any open ports on a target system as well as map out the environment to which it's connected. Once you breach a system, typically there are other networks that you can scan to gain deeper access to your target's network. One huge advantage of the Raspberry Pi is its size and mobility. Typically, Kali Linux is used from an attack system outside a target's network; however, tools such as PWNIE Express and small systems that run Kali Linux, such as a Raspberry Pi, can be placed inside a network and be remotely accessed. This gives an attacker a system inside the network, bypassing typical perimeter defenses while performing internal reconnaissance. This approach brings the obvious risks of having to physically place the system on the network as well as create a method to communicate with it remotely without being detected; however, if successful, this can be very effective. Let's look at a few popular methods to scan a target network. We'll continue forward assuming that you have established a foothold on a network and now want to understand the current environment that you have connected to. Nmap The most popular open source tool used to scan hosts and services on a network is Nmap (short for Network Mapper). Nmap's advanced features can detect different applications running on systems as well as offer services such as the OS fingerprinting features. Nmap can be very effective; however, it can also be easily detected unless used properly. We recommend using Nmap in very specific situations to avoid triggering a target's defense systems. For more information on how to use Nmap, visit http://nmap.org/. To use Nmap to scan a local network, open a terminal window and type nmap (target), for example, nmap www.somewebsite.com or nmap 192.168.1.2. There are many other commands that can be used to tune your scan. For example, you can tune how stealthy you want to be or specify to store the results in a particular location. The following screenshot shows the results after running Nmap against www.thesecurityblogger.com. Note that this is an example and is considered a noisy scan. If you simply type in either of the preceding two commands, it is most likely that your target will easily recognize that you are performing an Nmap scan. There are plenty of online resources available to learn how to master the various features for Nmap. Here is a reference list of popular nmap commands: nmap 192.168.1.0/24: This scans the entire class C range nmap -p <port ranges>: This scans specific ports nmap -sP 192.168.1.0/24: This scans the network/find servers and devices that are running nmap –iflist: This shows host interfaces and routes nmap –sV 192.168.1.1: This detects remote services' version numbers nmap –sS 192.168.1.1: This performs a stealthy TCP SYN scan nmap –sO 192.168.1.1: This scans for the IP protocol nmap -192.168.1.1 > output.txt: This saves the output from the scan to the text file nmap –sA 192.168.1.254: This checks whether the host is protected by a firewall nmap –PN 192.168.1.1: This scans the host when it is protected by a firewall nmap --reason 192.168.1.1: This displays the reason a port is in a particular state nmap --open 192.168.1.1: This only shows open or possibly open ports The Nmap GUI software Zenmap is not included in the Kali Linux ARM image. It is also not recommended over using the command line when running Kali Linux on a Raspberry Pi. Wireless security Another attack vector that can be leveraged on a Raspberry Pi with a Wi-Fi adapter is targeting wireless devices such as mobile tablets and laptops. Scanning wireless networks, once they are connected, is similar to how scanning is done on a LAN; however, typically a layer of password decryption is required before you can connect to a wireless network. Also, wireless network identifier known as Service Set Identifier (SSID) might not be broadcasted but will still be visible when you use the right tools. This section will cover how to bypass wireless onboarding defenses so that you can access a target's Wi-Fi network and perform the penetration testing steps. Looking at a Raspberry Pi with Kali Linux, one of the use cases is hiding the system inside or near a target's network and launching wireless attacks remotely. The goal will be to enable the Raspberry Pi to access the network wirelessly and provide a remote connection back to the attacker. The attacker can be nearby using wireless to control the Raspberry Pi until it gains wireless access. Once on the network, a backdoor can be established so that the attacker can communicate with the Raspberry Pi from anywhere in the world and launch attacks. Cracking WPA/WPA2 A commonly found security protocol for protecting wireless networks is Wi-Fi Protected Access (WPA). WPA was later replaced by WPA2 and it will be probably what you will be up against when you perform a wireless penetration test. WPA and WPA2 can be cracked with Aircrack. Kali Linux includes the Aircrack suite, which is one of the most popular applications to break wireless security. Aircrack works by gathering packets seen on a wireless connection to either mathematically analyze the data to crack weaker protocols such as Wired Equivalent Privacy (WEP), or use brute force on the captured data with a wordlist. Cracking WPA/WPA2 can be done due to a weakness in the four-way handshake between the client and the access point. In summary, a client will authenticate to an access point and go through a four-step process. This is the time when the attacker is able to grab the password and use a brute force approach to identify it. The time-consuming part in this is based on how unique the network password is, how extensive your wordlist that will be used to brute force against the password is, and the processing power of the system. Unfortunately, the Raspberry Pi lacks the processing power and the hard drive space to accommodate large wordlist files. So, you might have to crack the password off-box with a tool such as John the Ripper. We recommend this route for most WPA2 hacking attempts. Here is the process to crack a WPA running on a Linksys WRVS4400N wireless router using a Raspberry Pi on-box options. We are using a WPA example so that the time-consuming part can be accomplished quickly with a Raspberry Pi. Most WPA2 cracking examples would take a very long time to run from a Raspberry Pi; however, the steps to be followed are the same to run on a faster off-box system. The steps are as follows: Start Aircrack by opening a terminal and typing airmon-ng; In Aircrack, we need to select the desired interface to use for the attack. In the previous screenshot, wlan0 is my Wi-Fi adapter. This is a USB wireless adapter that has been plugged into my Raspberry Pi. It is recommended that you hide your Mac address while cracking a foreign wireless network. Kali Linux ARM does not come with the program macchanger. So, you should download it by using the sudo apt-get install macchanger command in a terminal window. There are other ways to change your Mac address, but macchanger can provide a spoofed Mac so that your device looks like a common network device such as a printer. This can be an effective way to avoid detection. Next, we need to stop the interface used for the attack so that we can change our Mac address. So, for this example, we will be stopping wlan0 using the following commands: airmon-ng stop wlan0 ifconfig wlan0 down Now, let's change the Mac address of this interface to hide our true identity. Use macchanger to change your Mac to a random value and specify your interface. There are options to switch to another type of device; however, for this example, we will just leave it as a random Mac address using the following command: macchanger -r wlan0 Our random value is b0:43:3a:1f:3a:05 in the following screenshot. Macchanger shows our new Mac as unknown. Now that our Mac is spoofed, let's restart airmon-ng with the following command: airmon-ng start wlan0 We need to locate available wireless networks so that we can pick our target to attack. Use the following command to do this: airodump-ng wlan0 You should now see networks within range of your Raspberry Pi that can be targeted for this attack. To stop the search once you identify a target, press Ctrl + C. You should write down the Mac address, also known as BSSID, and the channel, also known as CH, used by your target network. The following screenshot shows that our target with ESSID HackMePlease is running WPA on CH 6: The next step is running airodump against the Mac address that you just copied. You will need the following things to make this work: The channel being used by the target The Mac address (BSSID) that you copied A name for the file to save your data Let's run the airodump command in the following manner: airodump-ng –c [channel number] –w [name of file] –-bssid [target ssid] wlan0 This will open a new terminal window after you execute it. Keep that window open. Open another terminal window that will be used to connect to the target's wireless network. We will run aireplay using the following command: aireplay-ng-deauth 1 –a [target's BSSID] –c [our BSSID] [interface] For our example, the command will look like the following: aireplay-ng -–deauth 1 –a 00:1C:10:F6:04:C3 –c 00:0f:56:bc:2c:d1 wlan0 The following screenshot shows the launch of the preceding command: You may not get the full handshake when you run this command. If that happens, you will have to wait for a live user to authenticate you to the access point prior to launching the attack. The output on using Aircrack may show you something like Opening [file].cap a few times followed by No valid WPA handshakes found, if you didn't create a full handshake and somebody hasn't authenticated you by that time. Do not proceed to the next step until you capture a full handshake. The last step is to run Aircrack against the captured data to crack the WPA key. Use the –w option to specify the location of a wordlist that will be used to scan against the captured data. You will use the .cap file that was created earlier during step 9, so we will use the name capturefile.cap in our example. We'll do this using the following command: Aircrack-ng –w ./wordlist.lst wirelessattack.cap The Kali Linux ARM image does not include a wordlist.lst file for cracking passwords. Usually, default wordlists are not good anyway. So, it is recommended that you use Google to find an extensive wordlist (see the next section on wordlists for more information). Make sure to be mindful of the hard drive space that you have on the Raspberry Pi, as many wordlists might be too large to be used directly from the Raspberry Pi. The best practice for running process-intensive steps such as brute forcing passwords is to do them off-box on a more powerful system. You will see Aircrack start and begin trying each password in the wordlist file against the captured data. This process could take a while depending on the password you are trying to break, the number of words in your list, and the processing speed of the Raspberry Pi. We found that it ranges from a few hours to days, as it's a very tedious process and is possibly better-suited for an external system with more horsepower than a Raspberry Pi. You may also find that your wordlist doesn't work after waiting a few days to sort through the entire wordlist file. If Aircrack doesn't open and start trying keys against the password, you either didn't specify the location of the .cap file or the location of the wordlist.lst file, or you don't have the captured handshake data. By default, the previous steps store files in the root directory. You can move your wordlist file in the root directory to mimic how we ran the commands in the previous steps since all our files are located in the root directory folder. You can verify this by typing ls to list the current directory files. Make sure that you list the correct directories of each file that are called by each command. If your attack is successful, you should see something like the following screenshot that shows the identified password as sunshine: It is a good idea to perform this last step on a remote machine. You can set up a FTP server and push your .cap files to that FTP server. You can learn more about setting up an FTP server at http://www.raspberrypi.org/forums/viewtopic.php?f=36&t=35661. Creating wordlists There are many sources and tools that can be used to develop a wordlist for your attack. One popular tool called Custom Wordlist Generator (CeWL), allows you to create your own custom dictionary file. This can be extremely useful if you are targeting individuals and want to scrape their blogs, LinkedIn, or other websites for commonly used words. CeWL doesn't come preinstalled on the Kali Linux ARM image, so you will have to download it using apt-get install cewl. To use CeWL, open a terminal window and put in your target website. CeWL will examine the URL and create a wordlist based on all the unique words it finds. In the following example, we are creating a wordlist of commonly used words found on the security blog www.drchaos.com using the following command: cewl www.drchaos.com -w drchaospasswords.txt The following screenshot shows the launch of the preceding command: You can also find many examples of popular wordlists used as dictionary files on the Internet. Here are a few wordlist examples sources that you can use; however, be sure to research Google for other options as well: https://crackstation.net/buy-crackstation-wordlist-password-cracking-dictionary.html https://wiki.skullsecurity.org/Passwords Here is a dictionary that one of the coauthors put together: http://www.drchaos.com/public_files/chaos-dictionary.lst.txt Capturing traffic on the network It is great to get access to a target network. However, typically the next step, once a foothold is established, is to start looking at the data. To do this, you will need a method to capture and view network packets. This means turning your Raspberry Pi into a remotely accessible network tap. Many of these tools could overload and crash your Raspberry Pi. Look out for our recommendations regarding when to use a tuning method to avoid this from happening. Tcpdump Tcpdump is a command line based packet analyzer. You can use tcpdump to intercept and display TCP/IP and other packets that are transmitted and seen attached by the system This means the Raspberry Pi must have access to the network traffic that you intend to view or using tcpdump won't provide you with any useful data. Tcpdump is not installed with the default Kali Linux ARM image, so you will have to install it using the sudo apt-get install tcpdump command. Once installed, you can run tcpdump by simply opening a terminal window and typing sudo tcpdump. The following screenshot shows the traffic flow visible to us after the launch of the preceding command: As the previous screenshot shows, there really isn't much to see if you don't have the proper traffic flowing through the Raspberry Pi. Basically, we're seeing our own traffic while being plugged into an 802.1X-enabled switch, which isn't interesting. Let's look at how to get other system's data through your Raspberry Pi. Running tcpdump consumes a lot of the Raspberry Pi's processing power. We found that this could crash the Raspberry Pi by itself or while using it with other applications. We recommend that you tune your data capture to avoid this from happening. Man-in-the-middle attacks One common method to capture sensitive information is by performing a man-in-the-middle attack. By definition, a man-in-the-middle attack is when an attacker makes independent connections with victims while actively eavesdropping on the communication. This is typically done between a host and the systems. For example, a popular method to capture passwords is to act as a middleman between login credentials passed by a user to a web server. Summary This article introduced us to the various attack scenarios of penetration testing used with the tools available in Kali Linux, over a Raspberry Pi. This article also gave us detailed description of tools like Nmap, CeWL, and tcpdump, which are used for network scanning, creating wordlists, and analyzing network traffic respectively. Resources for Article: Further resources on this subject: Testing Your Speed [Article] Creating a 3D world to roam in [Article] Making the Unit Very Mobile – Controlling the Movement of a Robot with Legs [Article]

In this article by Colton Myers, author of the book Learning SaltStack, we will learn the basic architecture of a Salt deployment. The two main pieces of Salt are the Salt Master and the Salt Minion. The master is the central hub. All minions connect to the master to receive instructions. From the master, you can run commands and apply configuration across hundreds or thousands of minions in seconds. The minion, as mentioned before, connects to the master and treats the master as the source of all truth. Although minions can exist without a master, the full power of Salt is realized when you have minions and the master working together. Salt is built on two major concepts: remote execution and configuration management. In the remote execution system, Salt leverages Python to accomplish complex tasks with single-function calls. The configuration management system in Salt, called States, builds upon the remote execution foundation to create repeatable, enforceable configuration for the minions. With this bird's-eye view in mind, let's get Salt installed so that we can start learning how to use it to make managing our infrastructure easier! (For more resources related to this topic, see here.) Installing Salt The dependencies for running Salt at the time of writing are as follows: Python 2—Version 2.6 or greater (not Python 3-compatible) msgpack-python YAML Jinja2 MarkupSafe Apache Libcloud Requests ZeroMQ—Version 3.2.0 or greater PyZMQ—Version 2.2.0 or greater PyCrypto M2Crypto The easiest way to ensure that the dependencies for Salt are met is to use system-specific package management systems, such as apt on Ubuntu systems, that will handle the dependency-resolution automatically. You can also use a script called Salt-Bootstrap to handle all of the system-specific commands for you. Salt-Bootstrap is an open source project with the goal of creating a Bourne shell-compatible script that will install Salt on any compatible server. The project is managed and hosted by the SaltStack team. You can find more information at https://github.com/saltstack/salt-bootstrap. We will explore each of these methods of installation in turn. Installation with system packages (Ubuntu) The latest release of Salt for Ubuntu is provided in Personal Package Archive (PPA), which is a type of package repository for Ubuntu. The easiest way to access the PPA to install Salt is using the add-apt-repository command, as follows: # sudo add-apt-repository ppa:saltstack/salt If the add-apt-repository command is not found, you can add it by installing the python-software-properties package: sudo apt-get install python-software-properties If you are using Ubuntu Version 12.10 or greater, this step should not be required as the add-apt-repository command should be included in the base system. After you have added the repository, you must update the package management database, as follows: # sudo apt-get update If the system asks whether you should accept a gpg key, press Enter to accept. You should then be able to install the Salt master and the Salt minion with the following command: # sudo apt-get install salt-master salt-minion Assuming there are no errors after running this command, you should be done! Salt is now installed on your machine. Note that we installed both the Salt master and the Salt minion. The term master refers to the central server—the server from which we will be controlling all of our other servers. The term minion refers to the servers connected to and controlled by a master. Installing with Salt-Bootstrap Information about manual installation on other major Linux distributions can be found online, at http://docs.saltstack.com. However, in most cases, it is easier and more straightforward to use a tool called Salt-Bootstrap. In-depth documentation can be found on the project page at https://github.com/saltstack/salt-bootstrap—however, the tool is actually quite easy to use, as follows: # curl -L https://bootstrap.saltstack.com -o install_salt.sh # sudo sh install_salt.sh –h We won't include the help text for Bootstrap here as it would take up too much space. However, it should be noted that, by default, Bootstrap will install only the Salt minion. We want both the Salt minion and the Salt master, which can be accomplished by passing in the -M flag, as follows: # sudo sh install_salt.sh -M The preceding command will result in a fully-functional installation of Salt on your machine! The supported operating system list is extensive, as follows: Amazon Linux AMI 2012.09 Arch Linux CentOS 5/6 Debian 6.x/7.x/8 (git installations only) Fedora 17/18 FreeBSD 9.1/9.2/10 Gentoo Linux Linaro Linux Mint 13/14 OpenSUSE 12.x Oracle Linux 5/6 RHEL 5/6 Scientific Linux 5/6 SmartOS SuSE 11 SP1 and 11 SP2 Ubuntu 10.x/11.x/12.x/13.x/14.x The version of Salt used for the examples in this book is the 2014.7 release. Here is the full version information: # sudo salt --versions-report            Salt: 2014.7.0          Python: 2.7.6          Jinja2: 2.7.2        M2Crypto: 0.21.1 msgpack-python: 0.3.0    msgpack-pure: Not Installed        pycrypto: 2.6.1        libnacl: Not Installed          PyYAML: 3.10          ioflo: Not Installed          PyZMQ: 14.0.1            RAET: Not Installed            ZMQ: 4.0.4            Mako: 0.9.1 It's probable that the version of Salt you installed is a newer release and might have slightly different output. However, the examples should still all work in the latest version of Salt. Configuring Salt Now that we have the master and the minion installed on our machine, we must do a couple of pieces of configuration in order to allow them to talk to each other. Firewall configuration Since Salt minions connect to masters, the only firewall configuration that must be done is on the master. By default, ports 4505 and 4506 must be able to accept incoming connections on the master. The default install of Ubuntu 14.04, used for these examples, actually requires no firewall configuration out-of-the-box to be able to run Salt; the ports required are already open. However, many distributions of Linux come with much more restrictive default firewall settings. The most common firewall software in use by default is iptables. Note that you might also have to change firewall settings on your network hardware if there is network filtering in place outside the software on the machine on which you're working. Firewall configuration is a topic that deserves its own book. However, our needs for the configuration of Salt are fairly simple. First, you must find the set of rules currently in effect for your system. This varies from system to system; for example, the file is located in /etc/sysconfig/iptables on RedHat distributions, while it is located in /etc/iptables/iptables.rules in Arch Linux. Once you find that file, add the following lines to that file, but be sure to do it above the line that says DROP: -A INPUT -m state --state new -m tcp -p tcp --dport 4505 -j ACCEPT -A INPUT -m state --state new -m tcp -p tcp --dport 4506 -j ACCEPT For more information about configuring on your operating system of choice so that your Salt minion can connect successfully to your Salt master, see the Salt documentation at http://docs.saltstack.com/en/latest/topics/tutorials/firewall.html. In version 2014.7.0, a new experimental transport option was introduced in Salt, called RAET. The use of this transport system is beyond the scope of this book. This book will deal exclusively with the default, ZeroMQ-based transport in Salt. Salt minion configuration Out of the box, the Salt minion is configured to connect to a master at the location salt. The reason for this default is that, if DNS is configured correctly such that salt resolves to the master's IP address, no further configuration is needed. The minion will connect successfully to the master. However, in our example, we do not have any DNS configuration in place, so we must configure this ourselves. The minion and master configuration files are located in the /etc/salt/ directory. The /etc/salt/ directory should be created as part of the installation of Salt, assuming you followed the preceding directions. If it does not exist for some reason, please create the directory, and create two files, minion and master, within the directory. Open /etc/salt/minion with your text editor of choice (remember to use sudo!). We will be making a couple of changes to this file. First, find the commented-out line for the configuration option master. It should look like this: #master: salt Uncomment that line and change salt to localhost (as we have this minion connected to the local master). It should look like this: master: localhost If you cannot find the appropriate line in the file, just add the line shown previously to the top of the file. You should also manually configure the minion ID so that you can more easily follow along with the examples in this text. Find the ID line: #id: Uncomment it and set it to myminion: id: myminion Again, if you cannot find the appropriate line in the file, just add the line shown previously to the top of the file. Save and close the file. Without a manually-specified minion ID, the minion will try to intelligently guess what its minion ID should be at startup. For most systems, this will mean the minion ID will be set to the Fully-Qualified Domain Name (FQDN) for the system. Starting the Salt master and Salt minion Now we need to start (or restart) our Salt master and Salt minion. Assuming you're following along on Ubuntu (which I recommend), you can use the following commands: # sudo service salt-minion restart # sudo service salt-master restart Packages in other supported distributions ship with init scripts for Salt. Use whichever service system is available to you to start or restart the Salt minion and Salt master. Accepting the minion key on the master There is one last step remaining before we can run our first Salt commands. We must tell the master that it can trust the minion. To help us with this, Salt comes with the salt-key command to help us manage minion keys: # sudo salt-key Accepted Keys: Unaccepted Keys: myminion Rejected Keys: Notice that our minion, myminion, is listed in the Unaccepted Keys section. This means that the minion has contacted the master and the master has cached that minion's public key, and is waiting for further instructions as to whether to accept the minion or not. If your minion is not showing up in the output of salt-key, it's possible that the minion cannot reach the master on ports 4505 and 4506. Please refer to the Firewall section described previously for more information. Troubleshooting information can also be found in the Salt documentation at http://docs.saltstack.com/en/latest/topics/troubleshooting/. We can inspect the key's fingerprint to ensure that it matches our minion's key, as follows: # sudo salt-key -f myminion Unaccepted Keys: myminion: a8:1f:b0:c2:ab:9d:27:13:60:c9:81:b1:11:a3:68:e1 We can use the salt-call command to run a command on the minion to obtain the minion's key, as follows: # sudo salt-call --local key.finger local:   a8:1f:b0:c2:ab:9d:27:13:60:c9:81:b1:11:a3:68:e1 Since the fingerprints match, we can accept the key on the master, as follows: # sudo salt-key -a myminion The following keys are going to be accepted: Unaccepted Keys: myminion Proceed? [n/Y] Y Key for minion myminion accepted. We can check that the minion key was accepted, as follows: # sudo salt-key Accepted Keys: myminion Unaccepted Keys: Rejected Keys: Success! We are ready to run our first Salt command! Summary We've covered a lot of ground in this article. We've installed the Salt minion and Salt master on our machines and configured them to talk to each other, including accepting the minion's key on the master. Resources for Article: Further resources on this subject: An Introduction to the Terminal [Article] Importing Dynamic Data [Article] Veil-Evasion [Article]

In this article by Stefan Sjogelid, author of book Raspberry Pi for Secret Agents Second Edition. We will be setting up SIP Witch by adding softphones, connect them together, and then we will run the softphone on the Pi. When you're out in the field and need to call in a favor from a fellow agent or report back to HQ, you don't want to depend on the public phone network if you can avoid it. Landlines and cell phones alike can be tapped by all sorts of shady characters and to add insult to injury, you have to pay good money for this service. We can do better. Welcome to the wonderful world of Voice over IP (VoIP). VoIP is a blanket term for any technology capable of delivering speech between two end users over IP networks. There are plenty of services and protocols out there that try to meet this demand, most of which force you to connect through a central server that you don't own or control. We're going to turn the Pi into the central server of our very own phone network. To aid us with this task, we'll deploy GNU SIP Witch—a peer-to-peer VoIP server that uses Session Initiation Protocol (SIP) to route calls between phones. While there are many excellent VoIP servers available (Asterisk, FreeSwitch, and Yate and so on) SIP Witch has the advantage of being very lightweight on the Pi because its only concern is connecting phones and not much else. (For more resources related to this topic, see here.) Setting up SIP Witch Once we have the SIP server up and running we'll be adding one or more software phones or softphones. It's assumed that server and phones will all be on the same network. Let's get started! Install SIP Witch using the following command: pi@raspberrypi ~ $ sudo apt-get install sipwitch Just as the output of the previous command says, we have to define PLUGINS in /etc/default/sipwitch before running SIP Witch. Let's open it up for editing: pi@raspberrypi ~ $ sudo nano /etc/default/sipwitch Find the line that reads #PLUGINS="zeroconf scripting subscriber forward" and remove the # character to uncomment the line. This directive tells SIP Witch that we want the standard plugins to be loaded. Next we'll have a look at the main SIP Witch configuration file: pi@raspberrypi ~ $ sudo nano /etc/sipwitch.conf Note how some blocks of text are between <!-- and --> tags. These are comments in XML documents and are ignored by SIP Witch. Whatever changes you want to make, ensure they go outside of those tags. Now we're going to add a few softphone user accounts. It's up to you how many phones you'd like on your system, but each account needs a username, an extension (short phone number) and a password. Find the <provision> tag, make a new line and add your users: <user id="phone1"> <extension>201</extension> <secret>SecretSauce201</secret> <display>Agent 201</display> </user> <user id="phone2"> <extension>202</extension> <secret>SecretSauce202</secret> <display>Agent 202</display> </user> The user ID will be used as a user/login name later from the softphones. In this default configuration, the extensions can be any number between 201 and 299. The secret is the password that will go together with the username on the softphones. We will look into a better way of storing passwords later in this chapter. Finally, the display string defines an identity to present to other phones when calling. One more thing that we need to configure is how SIP Witch should treat local names. This makes it possible to call a phone by user ID in addition to the extension. Find the <stack> tag, make a new line and add the following directive, but replace [IP address] with the IP address of your Pi: <localnames>[IP address]</localnames> Those are all the changes we need to make to the configuration at the moment. Basic SIP Witch configuration for two phones With our configuration in place, let's start up the SIP Witch service: pi@raspberrypi ~ $ sudo service sipwitch start The SIP Witch server runs in the background and only outputs to a log file viewable with this command: pi@raspberrypi ~ $ sudo cat /var/log/sipwitch.log Now we can use the sipwitch command to interact with the running service. Type sipwitch for a list of all possible commands. Here's a short list of particularly handy ones: Command Description sudo sipwitch dump Shows how the SIP Witch server is currently configured. sudo sipwitch registry Lists all currently registered softphones. sudo sipwitch calls Lists active calls. sudo sipwitch message [extension] "[text]" Sends a text message from the server to an extension. Perfect for sending status updates from the Pi through scripting. Connecting the softphones Running your own telecommunications service is kind of boring without actual phones to make use of it. Fortunately, there are softphone applications available for most common electronic devices out there. The configuration of these phones will be pretty much identical no matter which platform they're running on. This is the basic information that will always need to be specified when configuring your softphone application: User / Login name: phone1 or phone2 in our example configuration Password / Authentication: The user's secret in our configuration Server / Host name / Domain: The IP address of your Pi Once a softphone is successfully registered with the SIP Witch server, you should be able to see that phone listed using the sudo sipwitch registry command. What follows is a list of verified decent softphones that will get the job done. Windows (MicroSIP) MicroSIP is an open source softphone that also supports video calls. Visit http://www.microsip.org/downloads to obtain and install the latest version (MicroSIP-3.8.1.exe at the time of writing).   Configuring the MicroSIP softphone for Windows Right-click on either the status bar in the main application window or the system tray icon to bring up the menu that lets you access the Account settings. Mac OS X (Telephone) Telephone is a basic open source softphone that is easily installed through the Mac App store. Configuring the Telephone softphone for Mac OS X Linux (SFLphone) SFLphone is an open source softphone with packages available for all major distributions and client interfaces for both GNOME and KDE. Use your distribution's package manager to find and install the application. Configuring SFLphone GNOME client in Ubuntu Android (CSipSimple) CSipSimple is an excellent open source softphone available from the Google Play store. When adding your account, use the basic generic wizard. Configuring the CSipSimple softphone on Android iPhone/iPad (Linphone) Linphone is an open source softphone that is easily installed through the iPhone App store. Select I have already a SIP-account to go to the setup assistant. Configuring Linphone on the iPhone Running a softphone on the Pi It's always good to be able to reach your agents directly from HQ, that is, the Pi itself. Proving once again that anything can be done from the command line, we're going to install a softphone called Linphone that will make good use of your USB microphone. This new softphone obviously needs a user ID and password just like the others. We will take this opportunity to look at a better way of storing passwords in SIP Witch. Encrypting SIP Witch passwords Type sudo sipwitch dump to see how SIP Witch is currently configured. Find the accounts: section and note how there's already a user ID named pi with extension 200. This is the result of a SIP Witch feature that automatically assigns an extension number to certain Raspbian user accounts. You may also have noticed that the display string for the pi user looks empty. We can easily fix that by filling in the full name field for the Raspbian pi user account with the following command: pi@raspberrypi ~ $ sudo chfn -f "Agent HQ" pi Now restart the SIP Witch server with sudo service sipwitch restart and verify with sudo sipwitch dump that the display string has changed. So how do we set the password for this automatically added pi user? For the other accounts, we specified the password in clear text inside <secret> tags in /etc/sipwitch.conf. This is not the best solution from a security perspective if your Pi would happen to fall into the wrong hands. Therefore, SIP Witch supports specifying passwords in encrypted digest form. Use the following command to create an encrypted password for the pi user: pi@raspberrypi ~ $ sudo sippasswd pi We can then view the database of SIP passwords that SIP Witch knows about: pi@raspberrypi ~ $ sudo cat /var/lib/sipwitch/digests.db Now you can add digest passwords for your other SIP users as well and then delete all <secret> lines from /etc/sipwitch.conf to be completely free of clear text. Setting up Linphone With our pi user account up and ready to go, let's proceed to set up Linphone: Linphone does actually have a graphical user interface, but we'll specify that we want the command-line only client: pi@raspberrypi ~ $ sudo apt-get install linphone-nogtk Now we fire up the Linphone command-line client: pi@raspberrypi ~ $ linphonec You will immediately receive a warning that reads: Warning: Could not start udp transport on port 5060, maybe this port is already used. That is, in fact, exactly what is happening. The standard communication channel for the SIP protocol is UDP port 5060, and it's already in use by our SIP Witch server. Let's tell Linphone to use port 5062 with this command: linphonec> ports sip 5062 Next we'll want to set up our microphone. Use these three commands to list, show, and select what audio device to use for phone calls: linphonec> soundcard list linphonec> soundcard show linphonec> soundcard use [number] For the softphone to perform reasonably well on the Pi, we'll want to make adjustments to the list of codecs that Linphone will try to use. The job of a codec is to compress audio as much as possible while retaining high quality. This is a very CPU-intensive process, which is why we want to use the codec with the least amount of CPU load on the Pi, namely, PCMU or PCMA. Use the following command to list all currently supported codecs: linphonec> codec list Now use this command to disable all codecs that are not PCMU or PCMA: linphonec> codec disable [number] It's time to register our softphone to the SIP Witch server. Use the following command but replace [IP address] with the IP address of your Pi and [password] with the SIP password you set earlier for the pi user: linphonec> register sip:pi@[IP address] sip:[IP address] [password] That's all you need to start calling your fellow agents from the Pi itself. Type help to get a list of all commands that Linphone accepts. The basic commands are call [user id] to call someone, answer to pick up incoming calls and quit to exit Linphone. All the settings that you've made will be saved to ~/.linphonerc and loaded the next time you start linphonec. Playing files with Linphone Now that you know the Linphone basics, let's explore some interesting features not offered by most other softphones. At any time (except during a call), you can switch Linphone into file mode, which lets us experiment with alternative audio sources. Use this command to enable file mode: linphonec> soundcard use files Do you remember eSpeak from earlier in this chapter? While you rest your throat, eSpeak can provide its soothing voice to carry out entire conversations with your agents. If you haven't already got it, install eSpeak first: pi@raspberrypi ~ $ sudo apt-get install espeak Now we tell Linphone what to say next: linphonec> speak english Greetings! I'm a Linphone, obviously. This sentence will be spoken as soon as there's an established call. So you can either make an outgoing call or answer an incoming call to start the conversation, after which you're free to continue the conversation in Italian: linphonec> speak italian Buongiorno! Mi chiamo Enzo Gorlami. Should you want a message to play automatically when someone calls, just toggle auto answer: linphonec> autoanswer enable How about playing a pre-recorded message or some nice grooves? If you have a WAV or MP3 file that you'd like to play over the phone, it has to be converted to a suitable format first. A simple SoX command will do the trick: pi@raspberrypi ~ $ sox "original file.mp3" -c 1 -r 48000 playme.wav Now we can tell Linphone to play the file: linphonec> play playme.wav Finally, you can also record a call to file. Note that only the remote part of the conversation can be recorded, which makes this feature more suitable for leaving messages and such. Use the following command to record: linphonec> record message.wav Summary In this article, we set up our very own phone network using SIP Witch and connected softphones running on a wide variety of platforms including the Pi itself. Resources for Article: Further resources on this subject: Our First Project – A Basic Thermometer [article] Testing Your Speed [article] Creating a 3D world to roam in [article]