Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7019 Articles
article-image-animation-effects-aspnet-using-jquery
Packt
03 May 2011
9 min read
Save for later

Animation Effects in ASP.NET using jQuery

Packt
03 May 2011
9 min read
  ASP.NET jQuery Cookbook Over 60 practical recipes for integrating jQuery with ASP.NET   Introduction Some useful inbuilt functions in jQuery that we will explore in this article for achieving animation effects are: animate ( properties, [ duration ], [ easing ], [ complete ] ): This method allows us to create custom animation effects on any numeric css property. The parameters supported by this method are: properties: This is the map of css properties to animate, for e.g. width, height, fontSize, borderWidth, opacity, etc. duration: This is the duration of the animation in milliseconds. The constants slow and fast can be used to specify the durations, and they represent 600 ms and 200 ms respectively. easing: This is the easing function to use. Easing indicates the speed of the animation at different points during the animation. jQuery provides inbuilt swing and linear easing functions. Various plugins can be interfaced if other easing functions are required. complete: This indicates the callback function on completion of the animation. fadeIn ( [ duration ], [ callback ] ): This method animates the opacity of the matched elements from 0 to 1 i.e. transparent to opaque. The parameters accepted are: duration: This is the duration of the animation callback: This is the callback function on completion of the animation fadeOut( [ duration ], [ callback ] ): This method animates the opacity of the matched elements from 1 to 0 i.e. opaque to transparent. The parameters accepted are: duration: This is the duration of the animation callback: This is the callback function on completion of the animation slideUp( [ duration ], [ callback ] ): This method animates the height of the matched elements with an upward sliding motion. When the height of the element reaches 0, the css property display of the element is updated to none so that the element is hidden on the page. The parameters accepted are: duration: This is the duration of the animation callback: This is the callback function on completion of the animation slideDown( [ duration ], [ callback ] ): This method animates the height of the matched elements from 0 to the specified maximum height. Thus, the element appears to slide down on the page. The parameters accepted are: duration: This is the duration of the animation callback: This is the callback function on completion of the animation slideToggle( [ duration ], [ callback ] ): This method animates the height of the matched elements. If the element is initially hidden, it will slide down and become completely visible. If the element is initially visible, it will slide up and become hidden on the page. The parameters accepted are: duration: This is the duration of the animation callback: This is the callback function on completion of the animation jQuery.fx.off: If there is a need to disable animations because of a resource constraint or due to difficulties in viewing the animations, then this utility can be used to turn off the animation completely. This is achieved by setting all animated controls to their final state. stop ( [ clearQueue ], [ jumpToEnd ] ): This method stops the currently running animations on the page. The parameters accepted are: clearQueue: This indicates whether any queued up animations are required to be cleared. The default value is false. jumpToEnd: This indicates if the current animation is to be cleared immediately. The default value is false. In this article, we will cover some of the animation effects that can be achieved in ASP.NET using the capabilities of jQuery. Getting started Let's start by creating a new ASP.NET website in Visual Studio and name it Chapter5. Save the jQuery library in a script folder js in the project. To enable jQuery on any web form, drag-and-drop to add the following to the page: <scriptsrc="js/jquery-1.4.1.js" type="text/javascript"></script> Now let's move on to the recipes where we will see different animation techniques using jQuery. Enlarging text on hover In this recipe, we will animate the font size of text content on hover. Getting ready Add a new web form Recipe1.aspx to the current project. Create a Css class for the text content that we want to animate. The font size specified in the css Class is the original font size of the text before animation is applied to it: .enlarge { font-size:12.5px; font-family:Arial,sans-serif; } Add an ASP.NET Label control on the form and set its Css Class to the preceding style: <asp:LabelCssClass="enlarge"runat="server">Lorem ipsum dolor sit ...............</asp:Label> Thus, the ASPX markup of the form is as follows: <form id="form1" runat="server"> <div align="center"> Mouseover to enlarge text:<br /> <fieldset id="content" style="width:500px;height:300px;"> <asp:LabelCssClass="enlarge" runat="server">Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat.</ asp:Label> </fieldset> </div> </form> Thus, initially, the page will display the Label control as follows: We will now animate the font size of the Label on hover on the containing fieldset element. How to do it… In the document.ready() function of the jQuery script block, retrieve the original font size of the Label: var origFontSize = parseFloat($(".enlarge").css('font-size')); The parseFloat() function takes in an input string and returns the first floating point value in the string. It discards any content after the floating point value. For example, if the css property returns 12.5 px, then the function will discard the px. Define the hover event of the containing fieldset element: $("#content").hover( In the mouseenter event handler of the hover method, update the cursor style to pointer: function() { $(".enlarge").css("cursor", "pointer"); Calculate the maximum font size that we want to animate to. In this example, we will set the maximum size to thrice the original: var newFontSize = origFontSize * 3; Animate the fontSize css property of the Label in 300 ms: $(".enlarge").animate({ fontSize: newFontSize }, 300); }, In the mouseleave event handler of the hover method, animate the fontSize to the original value in 300 ms as shown: function() { $(".enlarge").animate({ fontSize: origFontSize }, 300); } ); Thus, the complete jQuery solution is as follows: <script language="javascript" type="text/javascript"> $(document).ready(function() { var origFontSize = parseFloat($(".enlarge").css('fontsize')); $("#content").hover( function() { $(".enlarge").css("cursor", "pointer"); var newFontSize = origFontSize * 3; $(".enlarge").animate({ fontSize: newFontSize }, 300); }, function() { $(".enlarge").animate({ fontSize: origFontSize }, 300); } ); }); </script> How it works… Run the web form. Mouseover on the fieldset area. The text size will animate over the stated duration and change to the maximum specified font size as displayed in the following screenshot: On removing the mouse from the fieldset area, the text size will return back to the original. Creating a fade effect on hover In this recipe, we will create a fade effect on an ASP.NET Image control on hover. We will use the fadeIn and fadeOut methods to achieve the same. Getting ready Add a new web form Recipe2.aspx to the current project. Add an image control to the form: <asp:Image src="images/Image1.jpg" ID="Image1" runat="server" /> Define the properties of the image in the css: #Image1 { width:438px; height:336px; } Thus, the complete ASPX markup of the web form is as follows: <form id="form1" runat="server"> <div align="center"> Mouseover on the image to view fade effect: <fieldset id="content" style="width:480px;height:370px;"> <br /> <asp:Image src="images/Image1.jpg" ID="Image1" runat="server" /> </fieldset> </div> </form> On page load, the image is displayed as follows: We will now create a fade effect on the image on hover on the containing fieldset area. How to do it… In the document.ready() function of the jQuery script block, define the hover event on the containing fieldset area: $("#content").hover( In the mouseenter event handler of the hover method, update the cursor to pointer: function() { $("#Image1").css("cursor", "pointer"); Apply the fadeOut method on the Image control with an animation duration of 1000 ms: $("#Image1").fadeOut(1000); }, In the mouseleave event handler of the hover method, apply the fadeIn method on the Image control with an animation duration of 1000 ms: function() { $("#Image1").fadeIn(1000); } ); Thus, the complete jQuery solution is as follows: <script language="javascript" type="text/javascript"> $(document).ready(function() { $("#content").hover( function() { $("#Image1").css("cursor", "pointer"); $("#Image1").fadeOut(1000); }, function() { $("#Image1").fadeIn(1000); } ); }); </script> How it works... Run the web page. Mouseover on the Image control on the web page. The image will slowly fade away as shown in the following screenshot: On mouseout from the containing fieldset area, the image reappears. Sliding elements on a page In this recipe, we will use the slideUp and slideDown methods for achieving sliding effects on an ASP.NET panel. Getting ready Add a new web form Recipe3.aspx in the current project. Add an ASP.NET panel to the page as follows: <asp:Panel class="slide" runat="server"> Sliding Panel </asp:Panel> The css class for the panel is defined as follows: .slide { font-size:12px; font-family:Arial,sans-serif; display:none; height:100px; background-color:#9999FF; } Add a button control to trigger the sliding effect on the panel: <asp:Button ID="btnSubmit" runat="server" Text="Trigger Slide" /> Thus, the complete ASPX markup of the web form is as follows: <form id="form1" runat="server"> <div align="center"> <fieldset style="width:400px;height:150px;"> <asp:Button ID="btnSubmit" runat="server" Text="Trigger Slide" /> <br /><br/> <asp:Panel class="slide" runat="server"> Sliding Panel </asp:Panel> </fieldset> </div> </form> On page load, the page appears as shown in the following screenshot: We will now use jQuery to slide up and slide down the panel. How to do it… In the document.ready() function of the jQuery script block, define the click event of the button control: $("#btnSubmit").click(function(e) { Prevent default form submission: e.preventDefault(); Check if the ASP.NET panel control is hidden: if ($(".slide").is(":hidden")) The jQuery selector :hidden selects matched elements that are hidden on the page. If yes, then slide down the panel until its height reaches the maximum (100 px) defined in the css property. $(".slide").slideDown("slow"); If the panel is initially visible then slide up so that its height slowly reduces until it becomes 0 and the panel disappears from the page: else $(".slide").slideUp("slow"); }); Thus, the complete jQuery solution is as follows: <script language="javascript" type="text/javascript"> $(document).ready(function() { $("#btnSubmit").click(function(e) { e.preventDefault(); if ($(".slide").is(":hidden")) $(".slide").slideDown("slow"); else $(".slide").slideUp("slow"); }); }); </script>  
Read more
  • 0
  • 0
  • 13230

article-image-installing-openvpn-windows-and-mac
Packt
19 Jan 2010
5 min read
Save for later

Installing OpenVPN on Windows and Mac

Packt
19 Jan 2010
5 min read
Obtaining the software There are only a few prerequisites that have to be met if you want to install OpenVPN on Windows, if you are running versions later than Windows 2000. Mac OS X is required on Apple platforms. Installation of OpenVPN can be done in one of the following ways: For Microsoft Windows operating systems, you have to download the binary .exe file from http://openvpn.net/index.php/open-source/downloads.html or the package containing a graphical user interface from http://openvpn.se/. Those who dare to use the release candidate of version 2.1, or a forthcoming version 2.1 will find that the Windows GUI is already integrated (since OpenVPN 2.1rc13 from October 2008). On Macintosh systems running Mac OS X, there is a graphical installation wizard and management tool called Tunnelblick. Note that OpenVPN versions that are not tagged as stable should never be used in the production environment. There may be security issues and bugs that cause the code to crash or open your complete network to intruders. The stable versions have been tested for stability and security flaws, and will not be published as stable until they meet the developer team's requirements. Installing OpenVPN on Windows If you want to install OpenVPN on Windows, you have to make a choice before downloading. You can install the original OpenVPN software from a link such as http://www.openvpn.net/release/openvpn-2.0.9-install.exe (this is still my preferred suggestion) or install the OpenVPN GUI from http://openvpn.se/files/install_packages/openvpn-2.0.9-gui-1.0.3-install.exe. This package contains the OpenVPN software plus a GUI to bring up or close down tunnels. Especially, if you set up an OpenVPN client—be it a laptop or desktop PC for a home worker, which is only connecting temporarily to your VPN—the Windows user will want to have an easy-to-use, clickable interface. However, if you do not want the users to interact with the VPN tunnels, then the original OpenVPN software will do, and, as mentioned, beginning with release candidate 13 of version 2.1, the GUI is integrated. OpenVPN can be made to run as a service on the Windows PC, which means it is started automatically on startup. It can be configured to enable the tunnel automatically or forced by a click of a mouse. The installation is pretty straightforward and should not pose any problem to the experienced Windows user. The following sections show you a guided installation process. If you are prompted that the driver has not passed Windows Logo testing, click on Install. Downloading and starting installation Download the newest version of the OpenVPN GUI from http://openvpn.se/ to your local drive. Log in as the administrator or a privileged user, and double-click on the downloaded file to start the Setup Wizard. If you are using a desktop firewall, you will be prompted to allow OpenVPN to be installed and connected to the Internet later. The OpenVPN GUI installation wizard, probably the most convenient way to install OpenVPN on Windows, is started. Click on Next to proceed and agree to the terms of the license agreement (I Agree). Even though OpenVPN and the OpenVPN GUI are freely available under the open source General Public License (GPL), you still have to accept a license agreement. You should read the license to make sure that your planned use of OpenVPN conforms to it. Click on I Agree to proceed. Selecting the components and location The next dialog window offers a choice on the top of OpenVPN components that you may want to install. The standard selection of components change makes sense to is suitable for most cases. In this dialog, you have several options to choose from. Even if you normally don't need to make changes here, the following table gives you an overview of the entries and when you should install which feature. The Client Install is a system that only connects to another OpenVPN system, whereas the Server Install is an OpenVPN system that allows incoming connections. As you can see, the only differences are the RSA management and the option to run OpenVPN as a service. Both can be configured using different methods later, such as the configuration file, the Windows system management, or software like xca that we will use to generate and administer certificates. Press Next to continue installation and choose the path that you want to install OpenVPN to. This normally defaults to C:Program FilesOpenVPN, and there are usually very few reasons to change that. Click on Install to confirm. Finishing installation While OpenVPN is installing, you can read its output in the installation window and follow the creation of folders, files, and shortcuts and the installation of drivers (TAP) for networking. Recent Windows systems will warn you about the TUN/TAP driver that is about to be installed. As Microsoft can't validate the origin of the driver, its security subsystem warns you with the following dialog (Windows Server 2008): Click on Install this driver software anyway and see the OpenVPN installer complete the installation. If you've made it so far, you have successfully installed OpenVPN on your Windows system. If you want to read the Readme file, then activate the checkbox Show Readme before you click on Finish.
Read more
  • 0
  • 0
  • 13224

article-image-coding-minecraft
Packt
17 Sep 2013
7 min read
Save for later

Coding with Minecraft

Packt
17 Sep 2013
7 min read
(For more resources related to this topic, see here.) Getting ready Before you begin, you will need a running copy of Minecraft: Pi Edition. Start a game in a new or existing world, and wait for the game world to load. How to do it... Follow these steps to connect to the running Minecraft game: Open a fresh terminal by double-clicking on the LXTerminal icon on the desktop. Type cd ~/mcpi/api/python/mcpi into the terminal. Type python to begin the Python interpreter. Enter the following Python code: import minecraftmc = minecraft.Minecraft.create()mc.postToChat("Hello, world!") In the Minecraft window, you should see a message appear! How it works... First, we used the cd command, which we have seen previously to move to the location where the Python API is located. The application programming interface (API) consists of code provided by the Minecraft developers that handles some of the more basic functionality you might need. We used the ~ character as a shortcut for your home directory (/home/pi). Typing in cd /home/pi/mcpi/api/python/mcpi would have exactly the same effect, but requires more typing. We then start the Python interpreter. An interpreter is a program that executes code line by line as it is being typed. This allows us to get instant feedback on the code we are writing. You may like to explore the IDLE interpreter by typing idle into the terminal instead of python. IDLE is more advanced, and is able to color your code based on its meaning (so you can spot errors more easily), and can graphically suggest functions available for use. Then we started writing real Python code. The first line, import minecraft, gives us access to the necessary parts of the API by loading the minecraft module. There are several Python code files inside the directory we moved to, one of which is called minecraft.py, each containing a different code module. The main module we want access to is called minecraft. We then create a connection to the game using mc = minecraft.Minecraft.create(). mc is the name we have given to the connection, which allows us to use the same connection in any future code. minecraft. tells Python to look in the minecraft module. Minecraft is the name of a class in the minecraft module that groups together related data and functions. create() is a function of the Minecraft class that creates a connection to the game. Finally, we use the connection we have created, and its postToChat method to display a message in Minecraft. The way that our code interacts with the game is completely hidden from us to increase fl exibility: we can use almost exactly the same code to interact with any game of Minecraft: Pi Edition, and it is possible to use many different programming languages. If the developers want to change the way the communication works, they can do so, and it won't affect any of the code we have written. Behind the scenes, some text describing our command is sent across a network connection to the game, where the command is interpreted and performed. By default, the connection is to the very Raspberry Pi that we are running the code on, but it is also possible to send these commands over the Internet from any computer to any network-connected Raspberry Pi running Minecraft. A description of all of these text-based messages can be found in ~/ mcpi/api/spec: the message sent to the game when we wrote mc.postToChat("Hello,world!") was chat.post("Hello, world!"). This way of doing things allows any programming language to communicate with the running Minecraft game. As well as Python, a Java API is included that is capable of all the same tasks, and the community has created versions of the API in several other languages. There's more... There are many more functions provided in the Python API, some of the main ones are described here. You can explore the available commands using Python's help function: after importing the minecraft module, help(minecraft) will list the contents of that module, along with any text descriptions provided by the writer of the module. You can also use help to provide information on classes and functions. It is also possible to create your own API by building on top of the existing functions. For example, if you find yourself wanting to create a lot of spheres, you could write your own function that makes use of those provided, and import your module wherever you need it. The minecraft module The following code assumes that you have imported the minecraft module and created a connection to the running Minecraft game using mc = minecraft.Minecraft.create(). Whenever x, y, and z coordinates are used, x and z are both different directions that follow the ground, and y is the height, with 0 at sea level. Code Description mc.getBlock(x,y,z) This gets the type of a block at a particular location as a number. These numbers are all provided in the block module. mc.setBlock(x,y,z, type) The sets the block at a particular position to a particular type. There is also a setBlocks function that allows a cuboid to be filled - this will be faster than setting blocks individually. mc.getHeight(x,z) This gets the height of the world at the given location. mc.getPlayerEntityIds() This gets a list of IDs of all connected players. mc.saveCheckpoint() This saves the current state of the world. mc.restoreCheckpoint() This restores the state of the world from the saved checkpoint. mc.postToChat(message) This posts a message to the game chat. mc.setting(setting, status) This changes a game setting (such as "world_immutable" or "nametags_visible") to True or False. mc.camera.setPos(x,y,z) This moves the game camera to a particular location. Other options are setNormal(player_id), setFixed(), and setFollow(player_id). mc.player.getPos() This gets the position of the host player. mc.player.setPos(x,y,z) This moves the host player. mc.events.pollBlockHits() This gets a list of all blocks that have been hit since the last time the events were requested. Each event describes the position of the block that was hit. mc.events.clearAll() This clears the list of events. At the time of writing, only block hits are recorded, but more event types may be included in the future. The block module Another useful module is the block module: use import block to gain access to its contents. The block module has a list of all available blocks, and the numbers used to represent them. For example, a block of dirt is represented by the number 3. You can use 3 directly in your code if you like, or you can use the helpful name block.DIRT, which will help to make your code more readable. Some blocks, such as wool, have additional information to describe their color. This data can be provided after the block's ID in all functions. For example, to create a block of red wool, where 14 is the data value representing "red": mc.setBlock(x, y, z, block.WOOL, 14) Full information on the additional data values can be found online at http://www.minecraftwiki.net/wiki/Data values(Pocket_Edition). Summary This article gave us a simple code to interact with the game. It also explained how Python communicates with the game. The other aspects brought forth by the article include overview of other API functions, which can build useful functions on top of existing ones. Resources for Article: Further resources on this subject: Creating a file server (Samba) [Article] Webcam and Video Wizardry [Article] Instant Minecraft Designs – Building a Tudor-style house [Article]
Read more
  • 0
  • 0
  • 13210

article-image-extending-yii
Packt
03 Oct 2016
14 min read
Save for later

Extending Yii

Packt
03 Oct 2016
14 min read
Introduction      In this article by Dmitry Eliseev, the author of the book Yii Application Development Cookbook Third Edition, we will see three Yii extensions—helpers, behaviors, and components. In addition, we will learn how to make your extension reusable and useful for the community and will focus on the many things you should do in order to make your extension as efficient as possible. (For more resources related to this topic, see here.) Helpers There are a lot of built-in framework helpers, like StringHelper in the yiihelpers namespace. It contains sets of helpful static methods for manipulating strings, files, arrays, and other subjects. In many cases, for additional behavior you can create your own helper and put any static functions into one. For example, we will implement a number helper in this recipe. Getting ready Create a new yii2-app-basic application by using composer, as described in the official guide at http://www.yiiframework.com/doc-2.0/guide-start-installation.html. How to do it… Create the helpers directory in your project and write the NumberHelper class: <?php namespace apphelpers; class NumberHelper { public static function format($value, $decimal = 2) { return number_format($value, $decimal, '.', ','); } } Add the actionNumbers method into SiteController: <?php ... class SiteController extends Controller { … public function actionNumbers() { return $this->render('numbers', ['value' => 18878334526.3]); } } Add the views/site/numbers.php view: <?php use apphelpersNumberHelper; use yiihelpersHtml; /* @var $this yiiwebView */ /* @var $value float */ $this->title = 'Numbers'; $this->params['breadcrumbs'][] = $this->title; ?> <div class="site-numbers"> <h1><?= Html::encode($this->title) ?></h1> <p> Raw number:<br /> <b><?= $value ?></b> </p> <p> Formatted number:<br /> <b><?= NumberHelper::format($value) ?></b> </p> </div> Open the action and see this result: In other cases you can specify another count of decimal numbers; for example: NumberHelper::format($value, 3) How it works… Any helper in Yii2 is just a set of functions implemented as static methods in corresponding classes. You can use one to implement any different format of output for manipulations with values of any variable, and for other cases. Note: Usually, static helpers are light-weight clean functions with a small count of arguments. Avoid putting your business logic and other complicated manipulations into helpers . Use widgets or other components instead of helpers in other cases. See also For more information about helpers, refer to http://www.yiiframework.com/doc-2.0/guide-helper-overview.html. And for examples of built-in helpers, see sources in the helpers directory of the framework, refer to https://github.com/yiisoft/yii2/tree/master/framework/helpers. Creating model behaviors There are many similar solutions in today's web applications. Leading products such as Google's Gmail are defining nice UI patterns; one of these is soft delete. Instead of a permanent deletion with multiple confirmations, Gmail allows users to immediately mark messages as deleted and then easily undo it. The same behavior can be applied to any object such as blog posts, comments, and so on. Let's create a behavior that will allow marking models as deleted, restoring models, selecting not yet deleted models, deleted models, and all models. In this recipe we'll follow a test-driven development approach to plan the behavior and test if the implementation is correct. Getting ready Create a new yii2-app-basic application by using composer, as described in the official guide at http://www.yiiframework.com/doc-2.0/guide-start-installation.html. Create two databases for working and for tests. Configure Yii to use the first database in your primary application in config/db.php. Make sure the test application uses a second database in tests/codeception/config/config.php. Create a new migration: <?php use yiidbMigration; class m160427_103115_create_post_table extends Migration { public function up() { $this->createTable('{{%post}}', [ 'id' => $this->primaryKey(), 'title' => $this->string()->notNull(), 'content_markdown' => $this->text(), 'content_html' => $this->text(), ]); } public function down() { $this->dropTable('{{%post}}'); } } Apply the migration to both working and testing databases: ./yii migrate tests/codeception/bin/yii migrate Create a Post model: <?php namespace appmodels; use appbehaviorsMarkdownBehavior; use yiidbActiveRecord; /** * @property integer $id * @property string $title * @property string $content_markdown * @property string $content_html */ class Post extends ActiveRecord { public static function tableName() { return '{{%post}}'; } public function rules() { return [ [['title'], 'required'], [['content_markdown'], 'string'], [['title'], 'string', 'max' => 255], ]; } } How to do it… Let's prepare a test environment, starting with defining the fixtures for the Post model. Create the tests/codeception/unit/fixtures/PostFixture.php file: <?php namespace apptestscodeceptionunitfixtures; use yiitestActiveFixture; class PostFixture extends ActiveFixture { public $modelClass = 'appmodelsPost'; public $dataFile = '@tests/codeception/unit/fixtures/data/post.php'; } Add a fixture data file in tests/codeception/unit/fixtures/data/post.php: <?php return [ [ 'id' => 1, 'title' => 'Post 1', 'content_markdown' => 'Stored *markdown* text 1', 'content_html' => "<p>Stored <em>markdown</em> text 1</p>n", ], ]; Then, we need to create a test case tests/codeception/unit/MarkdownBehaviorTest: . .php: <?php namespace apptestscodeceptionunit; use appmodelsPost; use apptestscodeceptionunitfixturesPostFixture; use yiicodeceptionDbTestCase; class MarkdownBehaviorTest extends DbTestCase { public function testNewModelSave() { $post = new Post(); $post->title = 'Title'; $post->content_markdown = 'New *markdown* text'; $this->assertTrue($post->save()); $this->assertEquals("<p>New <em>markdown</em> text</p>n", $post->content_html); } public function testExistingModelSave() { $post = Post::findOne(1); $post->content_markdown = 'Other *markdown* text'; $this->assertTrue($post->save()); $this->assertEquals("<p>Other <em>markdown</em> text</p>n", $post->content_html); } public function fixtures() { return [ 'posts' => [ 'class' => PostFixture::className(), ] ]; } } Run unit tests: codecept run unit MarkdownBehaviorTest and ensure that tests have not passed Codeception PHP Testing Framework v2.0.9 Powered by PHPUnit 4.8.27 by Sebastian Bergmann and contributors. Unit Tests (2) --------------------------------------------------------------------------- Trying to test ... MarkdownBehaviorTest::testNewModelSave Error Trying to test ... MarkdownBehaviorTest::testExistingModelSave Error --------------------------------------------------------------------------- Time: 289 ms, Memory: 16.75MB Now we need to implement a behavior, attach it to the model, and make sure the test passes. Create a new directory, behaviors. Under this directory, create the MarkdownBehavior class: <?php namespace appbehaviors; use yiibaseBehavior; use yiibaseEvent; use yiibaseInvalidConfigException; use yiidbActiveRecord; use yiihelpersMarkdown; class MarkdownBehavior extends Behavior { public $sourceAttribute; public $targetAttribute; public function init() { if (empty($this->sourceAttribute) || empty($this->targetAttribute)) { throw new InvalidConfigException('Source and target must be set.'); } parent::init(); } public function events() { return [ ActiveRecord::EVENT_BEFORE_INSERT => 'onBeforeSave', ActiveRecord::EVENT_BEFORE_UPDATE => 'onBeforeSave', ]; } public function onBeforeSave(Event $event) { if ($this->owner->isAttributeChanged($this->sourceAttribute)) { $this->processContent(); } } private function processContent() { $model = $this->owner; $source = $model->{$this->sourceAttribute}; $model->{$this->targetAttribute} = Markdown::process($source); } } Let's attach the behavior to the Post model: class Post extends ActiveRecord { ... public function behaviors() { return [ 'markdown' => [ 'class' => MarkdownBehavior::className(), 'sourceAttribute' => 'content_markdown', 'targetAttribute' => 'content_html', ], ]; } } Run the test and make sure it passes: Codeception PHP Testing Framework v2.0.9 Powered by PHPUnit 4.8.27 by Sebastian Bergmann and contributors. Unit Tests (2) --------------------------------------------------------------------------- Trying to test ... MarkdownBehaviorTest::testNewModelSave Ok Trying to test ... MarkdownBehaviorTest::testExistingModelSave Ok --------------------------------------------------------------------------- Time: 329 ms, Memory: 17.00MB That's it. We've created a reusable behavior and can use it for all future projects by just connecting it to a model. How it works… Let's start with the test case. Since we want to use a set of models, we will define fixtures. A fixture set is put into the DB each time the test method is executed. We will prepare unit tests for specifying how the behavior works: First, we test processing new model content. The behavior must convert Markdown text from a source attribute to HTML and store the second one to target attribute. Second, we test updated content of an existing model. After changing Markdown content and saving the model, we must get updated HTML content. Now let's move to the interesting implementation details. In behavior, we can add our own methods that will be mixed into the model that the behavior is attached to. We can also subscribe to our own component events. We are using it to add our own listener: public function events() { return [ ActiveRecord::EVENT_BEFORE_INSERT => 'onBeforeSave', ActiveRecord::EVENT_BEFORE_UPDATE => 'onBeforeSave', ]; } And now we can implement this listener: public function onBeforeSave(Event $event) { if ($this->owner->isAttributeChanged($this->sourceAttribute)) { $this->processContent(); } } In all methods, we can use the owner property to get the object the behavior is attached to. In general we can attach any behavior to your models, controllers, application, and other components that extend the yiibaseComponent class. We can also attach one behavior again and again to model for the processing of different attributes: class Post extends ActiveRecord { ... public function behaviors() { return [ [ 'class' => MarkdownBehavior::className(), 'sourceAttribute' => 'description_markdown', 'targetAttribute' => 'description_html', ], [ 'class' => MarkdownBehavior::className(), 'sourceAttribute' => 'content_markdown', 'targetAttribute' => 'content_html', ], ]; } } Besides, we can also extend the yiibaseAttributeBehavior class, like yiibehaviorsTimestampBehavior, to update specified attributes for any event. See also To learn more about behaviors and events, refer to the following pages: http://www.yiiframework.com/doc-2.0/guide-concept-behaviors.html http://www.yiiframework.com/doc-2.0/guide-concept-events.html For more information about Markdown syntax, refer to http://daringfireball.net/projects/markdown/. Creating components If you have some code that looks like it can be reused but you don't know if it's a behavior, widget, or something else, it's most probably a component. The component should be inherited from the yiibaseComponent class. Later on, the component can be attached to the application and configured using the components section of a configuration file. That's the main benefit compared to using just a plain PHP class. We are also getting behaviors, events, getters, and setters support. For our example, we'll implement a simple Exchange application component that will be able to get currency rates from the http://fixer.io site, attach them to the application, and use them. Getting ready Create a new yii2-app-basic application by using composer, as described in the official guide at http://www.yiiframework.com/doc-2.0/guide-start-installation.html. How to do it… To get a currency rate, our component should send an HTTP GET query to a service URL, like http://api.fixer.io/2016-05-14?base=USD. The service must return all supported rates on the nearest working day: { "base":"USD", "date":"2016-05-13", "rates": { "AUD":1.3728, "BGN":1.7235, ... "ZAR":15.168, "EUR":0.88121 } } The component should extract needle currency from the response in a JSON format and return a target rate. Create a components directory in your application structure. Create the component class example with the following interface: <?php namespace appcomponents; use yiibaseComponent; class Exchange extends Component { public function getRate($source, $destination, $date = null) { } } Implement the component functional: <?php namespace appcomponents; use yiibaseComponent; use yiibaseInvalidConfigException; use yiibaseInvalidParamException; use yiicachingCache; use yiidiInstance; use yiihelpersJson; class Exchange extends Component { /** * @var string remote host */ public $host = 'http://api.fixer.io'; /** * @var bool cache results or not */ public $enableCaching = false; /** * @var string|Cache component ID */ public $cache = 'cache'; public function init() { if (empty($this->host)) { throw new InvalidConfigException('Host must be set.'); } if ($this->enableCaching) { $this->cache = Instance::ensure($this->cache, Cache::className()); } parent::init(); } public function getRate($source, $destination, $date = null) { $this->validateCurrency($source); $this->validateCurrency($destination); $date = $this->validateDate($date); $cacheKey = $this->generateCacheKey($source, $destination, $date); if (!$this->enableCaching || ($result = $this->cache->get($cacheKey)) === false) { $result = $this->getRemoteRate($source, $destination, $date); if ($this->enableCaching) { $this->cache->set($cacheKey, $result); } } return $result; } private function getRemoteRate($source, $destination, $date) { $url = $this->host . '/' . $date . '?base=' . $source; $response = Json::decode(file_get_contents($url)); if (!isset($response['rates'][$destination])) { throw new RuntimeException('Rate not found.'); } return $response['rates'][$destination]; } private function validateCurrency($source) { if (!preg_match('#^[A-Z]{3}$#s', $source)) { throw new InvalidParamException('Invalid currency format.'); } } private function validateDate($date) { if (!empty($date) && !preg_match('#d{4}-d{2}-d{2}#s', $date)) { throw new InvalidParamException('Invalid date format.'); } if (empty($date)) { $date = date('Y-m-d'); } return $date; } private function generateCacheKey($source, $destination, $date) { return [__CLASS__, $source, $destination, $date]; } } Attach our component in the config/console.php or config/web.php configuration files: 'components' => [ 'cache' => [ 'class' => 'yiicachingFileCache', ], 'exchange' => [ 'class' => 'appcomponentsExchange', 'enableCaching' => true, ], // ... db' => $db, ], We can now use a new component directly or via a get method: echo Yii::$app->exchange->getRate('USD', 'EUR'); echo Yii::$app->get('exchange')->getRate('USD', 'EUR', '2014-04-12'); Create a demonstration console controller: <?phpnamespace appcommands;use yiiconsoleController;class ExchangeController extends Controller{ public function actionTest($currency, $date = null) { echo Yii::$app->exchange->getRate('USD', $currency, $date) . PHP_EOL; }} And try to run any commands: $ ./yii exchange/test EUR > 0.90196 $ ./yii exchange/test EUR 2015-11-24 > 0.93888 $ ./yii exchange/test OTHER > Exception 'yiibaseInvalidParamException' with message 'Invalid currency format.' $ ./yii exchange/test EUR 2015/24/11 Exception 'yiibaseInvalidParamException' with message 'Invalid date format.' $ ./yii exchange/test ASD > Exception 'RuntimeException' with message 'Rate not found.' As a result you must see rate values in success cases or specific exceptions in error ones. In addition to creating your own components, you can do more. Overriding existing application components Most of the time there will be no need to create your own application components, since other types of extensions, such as widgets or behaviors, cover almost all types of reusable code. However, overriding core framework components is a common practice and can be used to customize the framework's behavior for your specific needs without hacking into the core. For example, to be able to format numbers using the Yii::app()->formatter->asNumber($value) method instead of the NumberHelper::format method from the Helpers recipe, follow the next steps: Extend the yiii18nFormatter component like the following: <?php namespace appcomponents; class Formatter extends yiii18nFormatter { public function asNumber($value, $decimal = 2) { return number_format($value, $decimal, '.', ','); } } Override the class of the built-in formatter component: 'components' => [ // ... formatter => [ 'class' => 'appcomponentsFormatter, ], // … ], Right now, we can use this method directly: echo Yii::app()->formatter->asNumber(1534635.2, 3); or as a new format for GridView and DetailView widgets: <?= yiigridGridView::widget([ 'dataProvider' => $dataProvider, 'columns' => [ 'id', 'created_at:datetime', 'title', 'value:number', ], ]) ?> You can also extend every existing component without overwriting its source code. How it works… To be able to attach a component to an application it can be extended from the yiibaseComponent class. Attaching is as simple as adding a new array to the components’ section of configuration. There, a class value specifies the component's class and all other values are set to a component through the corresponding component's public properties and setter methods. Implementation itself is very straightforward; We are wrapping http://api.fixer.io calls into a comfortable API with validators and caching. We can access our class by its component name using Yii::$app. In our case, it will be Yii::$app->exchange. See also For official information about components, refer to http://www.yiiframework.com/doc-2.0/guide-concept-components.html. For the NumberHelper class sources, see Helpers recipe. Summary In this article we learnt about the Yii extensions—helpers, behavior, and components. Helpers contains sets of helpful static methods for manipulating strings, files, arrays, and other subjects. Behaviors allow you to enhance the functionality of an existing component class without needing to change the class's inheritance. Components are the main building blocks of Yii applications. A component is an instance of CComponent or its derived class. Using a component mainly involves accessing its properties and raising/handling its events. Resources for Article: Further resources on this subject: Creating an Extension in Yii 2 [article] Atmosfall – Managing Game Progress with Coroutines [article] Optimizing Games for Android [article]
Read more
  • 0
  • 0
  • 13205

article-image-testing-node-and-hapi
Packt
09 Feb 2016
22 min read
Save for later

Testing in Node and Hapi

Packt
09 Feb 2016
22 min read
In this article by John Brett, the author of the book Getting Started with Hapi.js, we are going to explore the topic of testing in node and hapi. We will look at what is involved in writing a simple test using hapi's test runner, lab, how to test hapi applications, techniques to make testing easier, and finally how to achieve the all-important 100% code coverage. (For more resources related to this topic, see here.) The benefits and importance of testing code Technical debt is developmental work that must be done before a particular job is complete, or else it will make future changes much harder to implement later on. A codebase without tests is a clear indication of technical debt. Let's explore this statement in more detail. Even very simple applications will generally comprise: Features, which the end user interacts with Shared services, such as authentication and authorization, that features interact with These will all generally depend on some direct persistent storage or API. Finally, to implement most of these features and services, we will use libraries, frameworks, and modules regardless of language. So, even for simpler applications, we have already arrived at a few dependencies to manage, where a change that causes a break in one place could possibly break everything up the chain. So let's take a common use case, in which a new version of one of your dependencies is released. This could be a new hapi version, a smaller library, your persistent storage engine, MySQL, MongoDB, or even an operating system or language version. SemVer, as mentioned previously, attempts to mitigate this somewhat, but you are taking someone at their word when they say that they have adhered to this correctly, and SemVer is not used everywhere. So, in the case of a break-causing change, will the current application work with this new dependency version? What will fail? What percentage of tests fail? What's the risk if we don't upgrade? Will support eventually be dropped, including security patches? Without a good automated test suite, these have to be answered by manual testing, which is a huge waste of developer time. Development progress stops here every time these tasks have to be done, meaning that these types of tasks are rarely done, building further technical debt. Apart from this, humans are proven to be poor at repetitive tasks, prone to error, and I know I personally don't enjoy testing manually, which makes me poor at it. I view repetitive manual testing like this as time wasted, as these questions could easily be answered by running a test suite against the new dependency so that developer time could be spent on something more productive. Now, let's look at a worse and even more common example: a security exploit has been identified in one of your dependencies. As mentioned previously, if it's not easy to update, you won't do it often, so you could be on an outdated version that won't receive this security update. Now you have to jump multiple versions at once and scramble to test them manually. This usually means many quick fixes, which often just cause more bugs. In my experience, code changes under pressure are what deteriorate the structure and readability in a codebase, lead to a much higher number of bugs, and are a clear sign of poor planning. A good development team will, instead of looking at what is currently available, look ahead to what is in beta and will know ahead of time if they expect to run into issues. The questions asked will be: Will our application break in the next version of Chrome? What about the next version of node? Hapi does this by running the full test suite against future versions of node in order to alert the node community of how planned changes will impact hapi and the node community as a whole. This is what we should all aim to do as developers. A good test suite has even bigger advantages when working in a team or when adding new developers to a team. Most development teams start out small and grow, meaning all the knowledge of the initial development needs to be passed on to new developers joining the team. So, how do tests lead to a benefit here? For one, tests are a great documentation on how parts of the application work for other members of a team. When trying to communicate a problem in an application, a failing test is a perfect illustration of what and where the problem is. When working as a team, for every code change from yourself or another member of the team, you're faced with the preceding problem of changing a dependency. Do we just test the code that was changed? What about the code that depends on the changed code? Is it going to be manual testing again? If this is the case, how much time in a week would be spent on manual testing versus development? Often, with changes, existing functionality can be broken along with new functionality, which is called regression. Having a good test suite highlights this and makes it much easier to prevent. These are the questions and topics that need to be answered when discussing the importance of tests. Writing tests can also improve code quality. For one, identifying dead code is much easier when you have a good testing suite. If you find that you can only get 90% code coverage, what does the extra 10% do? Is it used at all if it's unreachable? Does it break other parts of the application if removed? Writing tests will often improve your skills in writing easily testable code. Software applications usually grow to be complex pretty quickly—it happens, but we always need to be active in dealing with this, or software complexity will win. A good test suite is one of the best tools we have to tackle this. The preceding is not an exhaustive list on the importance or benefits of writing tests for your code, but hopefully it has convinced you of the importance of having a good testing suite. So, now that we know why we need to write good tests, let's look at hapi's test runner lab and assertion library code and how, along with some tools from hapi, they make the process of writing tests much easier and a more enjoyable experience. Introducing hapi's testing utilities The test runner in the hapi ecosystem is called lab. If you're not familiar with test runners, they are command-line interface tools for you to run your testing suite. Lab was inspired by a similar test tool called mocha, if you are familiar with it, and in fact was initially begun as a fork of the mocha codebase. But, as hapi's needs diverged from the original focus of mocha, lab was born. The assertion library commonly used in the hapi ecosystem is code. An assertion library forms the part of a test that performs the actual checks to judge whether a test case has passed or not, for example, checking that the value of a variable is true after an action has been taken. Lets look at our first test script; then, we can take a deeper look at lab and code, how they function under the hood, and some of the differences they have with other commonly used libraries, such as mocha and chai. Installing lab and code You can install lab and code the same as any other module on npm: npm install lab code -–save-dev Note the --save-dev flag added to the install command here. Remember your package.json file, which describes an npm module? This adds the modules to the devDependencies section of your npm module. These are dependencies that are required for the development and testing of a module but are not required for using the module. The reason why these are separated is that when we run npm install in an application codebase, it only installs the dependencies and devDependencies of package.json in that directory. For all the modules installed, only their dependencies are installed, not their development dependencies. This is because we only want to download the dependencies required to run that application; we don't need to download all the development dependencies for every module. The npm install command installs all the dependencies and devDependencies of package.json in the current working directory, and only the dependencies of the other installed module, not devDependencies. To install the development dependencies of a particular module, navigate to the root directory of the module and run npm install. After you have installed lab, you can then run it with the following: ./node_modules/lab/bin/lab test.js This is quite long to type every time, but fortunately due to a handy feature of npm called npm scripts, we can shorten it. If you look at package.json generated by npm init in the first chapter, depending on your version of npm, you may see the following (some code removed for brevity): ... "scripts": { "test": "echo "Error: no test specified" && exit 1" }, ... Scripts are a list of commands related to the project; they can be for testing purposes, as we will see in this example; to start an application; for build steps; and to start extra servers, among many other options. They offer huge flexibility in how these are combined to manage scripts related to a module or application, and I could spend a chapter, or even a book, on just these, but they are outside the scope of this book, so let's just focus on what is important to us here. To get a list of available scripts for a module application, in the module directory, simply run: $ npm run To then run the listed scripts, such as test you can just use: $ npm run test As you can see, this gives a very clean API for scripts and the documentation for each of them in the project's package.json. From this point on in this book, all code snippets will use npm scripts to test or run any examples. We should strive to use these in our projects to simplify and document commands related to applications and modules for ourselves and others. Let's now add the ability to run a test file to our package.json file. This just requires modifying the scripts section to be the following: ... "scripts": { "test": "./node_modules/lab/bin/lab ./test/index.js" }, ... It is common practice in node to place all tests in a project within the test directory. A handy addition to note here is that when calling a command with npm run, the bin directory of every module in your node_modules directory is added to PATH when running these scripts, so we can actually shorten this script to: … "scripts": { "test": "lab ./test/index.js" }, … This type of module install is considered to be local, as the dependency is local to the application directory it is being run in. While I believe this is how we should all install our modules, it is worth pointing it out that it is also possible to install a module globally. This means that when installing something like lab, it is immediately added to PATH and can be run from anywhere. We do this by adding a -g flag to the install, as follows: $ npm install lab code -g This may appear handier than having to add npm scripts or running commands locally outside of an npm script but should be avoided where possible. Often, installing globally requires sudo to run, meaning you are taking a script from the Internet and allowing it to have complete access to your system. Hopefully, the security concerns here are obvious. Other than that, different projects may use different versions of test runners, assertion libraries, or build tools, which can have unknown side effects and cause debugging headaches. The only time I would use globally installed modules are for command-line tools that I may use outside a particular project—for example, a node base terminal IDE such as slap (https://www.npmjs.com/package/slap) or a process manager such as PM2 (https://www.npmjs.com/package/pm2)—but never with sudo! Now that we are familiar with installing lab and code and the different ways or running it inside and outside of npm scripts, let's look at writing our first test script and take a more in-depth look at lab and code. Our First Test Script Let's take a look at what a simple test script in lab looks like using the code assertion library: const Code = require('code'); [1] const Lab = require('lab'); [1] const lab = exports.lab = Lab.script(); [2] lab.experiment('Testing example', () => { [3] lab.test('fails here', (done) => { [4] Code.expect(false).to.be.true(); [4] return done(); [4] }); [4] lab.test('passes here', (done) => { [4] Code.expect(true).to.be.true(); [4] return done(); [4] }); [4] }); This script, even though small, includes a number of new concepts, so let's go through it with reference to the numbers in the preceding code: [1]: Here, we just include the code and lab modules, as we would any other node module. [2]: As mentioned before, it is common convention to place all test files within the test directory of a project. However, there may be JavaScript files in there that aren't tests, and therefore should not be tested. To avoid this, we inform lab of which files are test scripts by calling Lab.script() and assigning the value to lab and exports.lab. [3]: The lab.experiment() function (aliased lab.describe()) is just a way to group tests neatly. In test output, tests will have the experiment string prefixed to the message, for example, "Testing example fails here". This is optional, however. [4]: These are the actual test cases. Here, we define the name of the test and pass a callback function with the parameter function done(). We see code in action here for managing our assertions. And finally, we call the done() function when finished with our test case. Things to note here: lab tests are always asynchronous. In every test, we have to call done() to finish the test; there is no counting of function parameters or checking whether synchronous functions have completed in order to ensure that a test is finished. Although this requires the boilerplate of calling the done() function at the end of every test, it means that all tests, synchronous or asynchronous, have a consistent structure. In Chai, which was originally used for hapi, some of the assertions such as .ok, .true, and .false use properties instead of functions for assertions, while assertions like .equal(), and .above() use functions. This type of inconsistency leads to us easily forgetting that an assertion should be a method call and hence omitting the (). This means that the assertion is never called and the test may pass as a false positive. Code's API is more consistent in that every assertion is a function call. Here is a comparison of the two: Chai: expect('hello').to.equal('hello'); expect(foo).to.exist; Code: expect('hello').to.equal('hello'); expect('foot').to.exist(); Notice the difference in the second exist() assertion. In Chai, you see the property form of the assertion, while in Code, you see the required function call. Through this, lab can make sure all assertions within a test case are called before done is complete, or it will fail the test. So let's try running our first test script. As we already updated our package.json script, we can run our test with the following command: $ npm run test This will generate the following output: There are a couple of things to note from this. Tests run are symbolized with a . or an X, depending on whether they pass or not. You can get a lab list of the full test title by adding the -v or -–verbose flag to our npm test script command. There are lots of flags to customize the running and output of lab, so I recommend using the full labels for each of these, for example, --verbose and --lint instead of -v and -l, in order to save you the time spent referring back to the documentation each time. You may have noticed the No global variable leaks detected message at the bottom. Lab assumes that the global object won't be polluted and checks that no extra properties have been added after running tests. Lab can be configured to not run this check or whitelist certain globals. Details of this are in the lab documentation availbale at https://github.com/hapijs/lab. Testing approaches This is one of the many known approaches to building a test suite, as is BDD (Behavior Driven Development), and like most test runners in node, lab is unopinionated about how you structure your tests. Details of how to structure your tests in a BDD can again be found easily in the lab documentation. Testing with hapi As I mentioned before, testing is considered paramount in the hapi ecosystem, with every module in the ecosystem having to maintain 100% code coverage at all times, as with all module dependencies. Fortunately, hapi provides us with some tools to make the testing of hapi apps much easier through a module called Shot, which simulates network requests to a hapi server. Let's take the example of a Hello World server and write a simple test for it: const Code = require('code'); const Lab = require('lab'); const Hapi = require('hapi'); const lab = exports.lab = Lab.script(); lab.test('It will return Hello World', (done) => { const server = new Hapi.Server(); server.connection(); server.route({ method: 'GET', path: '/', handler: function (request, reply) { return reply('Hello Worldn'); } }); server.inject('/', (res) => { Code.expect(res.statusCode).to.equal(200); Code.expect(res.result).to.equal('Hello Worldn'); done(); }); }); Now that we are more familiar with with what a test script looks like, most of this will look familiar. However, you may have noticed we never started our hapi server. This means the server was never started and no port assigned, but thanks to the shot module (https://github.com/hapijs/shot), we can still make requests against it using the server.inject API. Not having to start a server means less setup and teardown before and after tests and means that a test suite can run quicker as less resources are required. server.inject can still be used if used with the same API whether the server has been started or not. Code coverage As I mentioned earlier in the article, having 100% code coverage is paramount in the hapi ecosystem and, in my opinion, hugely important for any application to have. Without a code coverage target, writing tests can feel like an empty or unrewarding task where we don't know how many tests are enough or how much of our application or module has been covered. With any task, we should know what our goal is; testing is no different, and this is what code coverage gives us. Even with 100% coverage, things can still go wrong, but it means that at the very least, every line of code has been considered and has at least one test covering it. I've found from working on modules for hapi that trying to achieve 100% code coverage actually gamifies the process of writing tests, making it a more enjoyable experience overall. Fortunately, lab has code coverage integrated, so we don't need to rely on an extra module to achieve this. It's as simple as adding the --coverage or -c flag to our test script command. Under the hood, lab will then build an abstract syntax tree so it can evaluate which lines are executed, thus producing our coverage, which will be added to the console output when we run tests. The code coverage tool will also highlight which lines are not covered by tests, so you know where to focus your testing effort, which is extremely useful in identifying where to focus your testing effort. It is also possible to enforce a minimum threshold as to the percentage of code coverage required to pass a suite of tests with lab through the --threshold or -t flag followed by an integer. This is used for all the modules in the hapi ecosystem, and all thresholds are set to 100. Having a threshold of 100% for code coverage makes it much easier to manage changes to a codebase. When any update or pull request is submitted, the test suite is run against the changes, so we can know that all tests have passed and all code covered before we even look at what has been changed in the proposed submission. There are services that even automate this process for us, such as TravisCI (https://travis-ci.org/). It's also worth knowing that the coverage report can be displayed in a number of formats; For a full list of these reporters with explanations, I suggest reading the lab documentation available at https://github.com/hapijs/lab. Let's now look at what's involved in getting 100% coverage for our previous example. First of all, we'll move our server code to a separate file, which we will place in the lib folder and call index.js. It's worth noting here that it's good testing practice and also the typical module structure in the hapi ecosystem to place all module code in a folder called lib and the associated tests for each file within lib in a folder called test, preferably with a one-to-one mapping like we have done here, where all the tests for lib/index.js are in test/index.js. When trying to find out how a feature within a module works, the one-to-one mapping makes it much easier to find the associated tests and see examples of it in use. So, having separated our server from our tests, let's look at what our two files now look like; first, ./lib/index.js: const Hapi = require('hapi'); const server = new Hapi.Server(); server.connection(); server.route({ method: 'GET', path: '/', handler: function (request, reply) { return reply('Hello Worldn'); } }); module.exports = server; The main change here is that we export our server at the end for another file to acquire and start it if necessary. Our test file at ./test/index.js will now look like this: const Code = require('code'); const Lab = require('lab'); const server = require('../lib/index.js'); const lab = exports.lab = Lab.script(); lab.test('It will return Hello World', (done) => { server.inject('/', (res) => { Code.expect(res.statusCode).to.equal(200); Code.expect(res.result).to.equal('Hello Worldn'); done(); }); }); Finally, for us to test our code coverage, we update our npm test script to include the coverage flag --coverage or -c. The final example of this is in the second example of the source code of Chapter 4, Adding Tests and the Importance of 100% Coverage, which is supplied with this book. If you run this, you'll find that we actually already have 100% of the code covered with this one test. An interesting exercise here would be to find out what versions of hapi this code functions correctly with. At the time of writing, this code was written for hapi version 11.x.x on node.js version 4.0.0. Will it work if run with hapi version 9 or 10? You can test this now by installing an older version with the help of the following command: $ npm install hapi@10 This will give you an idea of how easy it can be to check whether your codebase works with different versions of libraries. If you have some time, it would be interesting to see how this example runs on different versions of node (Hint: it breaks on any version earlier than 4.0.0). In this example, we got 100% code coverage with one test. Unfortunately, we are rarely this fortunate when we increase the complexity of our codebase, and so will the complexity of our tests be, which is where knowledge of writing testable code comes in. This is something that comes with practice by writing tests while writing application or module code. Linting Also built into lab is linting support. Linting enforces a code style that is adhered to, which can be specified through an .eslintrc or .jshintrc file. By default, lab will enforce the the hapi style guide rules. The idea of linting is that all code will have the same structure, making it much easier to spot bugs and keep code tidy. As JavaScript is a very flexible language, linters are used regularly to forbid bad practices such as global or unused variables. To enable the lab linter, simply add the linter flag to the test command, which is --lint or -L. I generally stick with the default hapi style guide rules as they are chosen to promote easy-to-read code that is easily testable and forbids many bad practices. However, it's easy to customize the linting rules used; for this, I recommend referring to the lab documentation. Summary In this article, we covered testing in node and hapi and how testing and code coverage are paramount in the hapi ecosystem. We saw justification for their need in application development and where they can make us more productive developers. We also introduced the test runner and code assertion libraries lab and code in the ecosystem. We saw justification for their use and also how to use them to write simple tests and how to use the tools provided in lab and hapi to test hapi applications. We also learned about some of the extra features baked into lab, such as code coverage and linting. We looked at how to test the code coverage of an application and get it to 100% and how the hapi ecosystem applies the hapi styleguide to all modules using lab's linting integration. Resources for Article: Further resources on this subject: Welcome to JavaScript in the full stack[article] A Typical JavaScript Project[article] An Introduction to Mastering JavaScript Promises and Its Implementation in Angular.js[article]
Read more
  • 0
  • 0
  • 13200

article-image-ios-security-overview
Packt
04 Mar 2015
20 min read
Save for later

iOS Security Overview

Packt
04 Mar 2015
20 min read
In this article by Allister Banks and Charles S. Edge, the authors of the book, Learning iOS Security, we will go through an overview of the basic security measures followed in an iOS. Out of the box, iOS is one of the most secure operating systems available. There are a number of factors that contribute to the elevated security level. These include the fact that users cannot access the underlying operating system. Apps also have data in a silo (sandbox), so instead of accessing the system's internals they can access the silo. App developers choose whether to store settings such as passwords in the app or on iCloud Keychain, which is a secure location for such data on a device. Finally, Apple has a number of controls in place on devices to help protect users while providing an elegant user experience. However, devices can be made even more secure than they are now. In this article, we're going to get some basic security tasks under our belt in order to get some basic best practices of security. Where we feel more explanation is needed about what we did on devices, we'll explore a part of the technology itself in this article. This article will cover the following topics: Pairing Backing up your device Initial security checklist Safari and built-in app protection Predictive search and spotlight (For more resources related to this topic, see here.) To kick off the overview of iOS security, we'll quickly secure our systems by initially providing a simple checklist of tasks, where we'll configure a few device protections that we feel everyone should use. Then, we'll look at how to take a backup of our devices and finally, at how to use a built-in web browser and protections around a browser. Pairing When you connect a device to a computer that runs iTunes for the first time, you are prompted to enter a password. Doing so allows you to synchronize the device to a computer. Applications that can communicate over this channel include iTunes, iPhoto, Xcode, and others. To pair a device to a Mac, simply plug the device in (if you have a passcode, you'll need to enter that in order to pair the device.) When the device is plugged in, you'll be prompted on both the device and the computer to establish a trust. Simply tap on Trust on the iOS device, as shown in the following screenshot: Trusting a computer For the computer to communicate with the iOS device, you'll also need to accept the pairing on your computer (although, when you use libimobiledevice, which is the command to pair, does not require doing so, because you use the command line to accept). When prompted, click on Continue to establish the pairing, as seen in the following screenshot (the screenshot is the same in Windows): Trusting a device When a device is paired, a file is created in /var/db/lockdown, which is the UDID of the device with a property list (plist) extension. A property list is an Apple XML file that stores a variety of attributes. In Windows, iOS data is stored in the MobileSync folder, which you can access by navigating to Users(username)AppDataRoamingApple ComputerMobileSync. The information in this file sets up a trust between the computers and includes the following attributes: DeviceCertificate: This certificate is unique to each device. EscrowBag: The key bag of EscrowBag contains class keys used to decrypt the device. HostCertificate: This certificate is for the host who's paired with iOS devices (usually, the same for all files that you've paired devices with, on your computer). HostID: This is a generated ID for the host. HostPrivateKey: This is the private key for your Mac (should be the same in all files on a given computer). RootCertificate: This is the certificate used to generate keys (should be the same in all files on a given computer). RootPrivateKey: This is the private key of the computer that runs iTunes for that device. SystemBUID: This refers to the ID of the computer that runs iTunes. WiFiMACAddress: This is the Mac address of the Wi-Fi interface of the device that is paired to the computer. If you do not have an active Wi-Fi interface, MAC is still used while pairing. Why does this matter? It's important to know how a device interfaces with a computer. These files can be moved between computers and contain a variety of information about a device, including private keys. Having keys isn't all that is required for a computer to communicate with a device. When the devices are interfacing with a computer over USB, if you have a passcode enabled on the device, you will be required to enter that passcode in order to unlock the device. Once a computer is able to communicate with a device, you need to be careful as the backups of a device, apps that get synchronized to a device, and other data that gets exchanged with a device can be exposed while at rest on devices. Backing up your device What do most people do to maximize the security of iOS devices? Before we do anything, we need to take a backup of our devices. This protects the device from us by providing a restore point. This also secures the data from the possibility of losing it through a silly mistake. There are two ways, which are most commonly used to take backups: iCloud and iTunes. As the names imply, the first makes backups for the data on Apple's cloud service and the second on desktop computers. We'll cover how to take a backup on iCloud first. iCloud backups An iCloud account comes with free storage, to back up your Apple devices. An iOS device takes a backup to Apple servers and can be restored when a new device is set up from those same servers (it's a screen that appears during the activation process of a new device. Also, it appears as an option in iTunes if you back up to iTunes over USB—covered later in this article). Setting up and checking the status of iCloud backups is a straightforward process. From the Settings app, tap on iCloud and then Backup. As you can see from the Backup screen, you have two options, iCloud Backup, which enables automatic backups of the device to your iCloud account, and Back Up Now, which runs an immediate backup of the device. iCloud backups Allowing iCloud to take backups on devices is optional. You can disable access to iCloud and iCloud backups. However, doing so is rarely a good idea as you are limiting the functionality of the device and putting the data on your device at risk, if that data isn't backed up another way such as through iTunes. Many people have reservations about storing data on public clouds; especially, data as private as phone data (texts, phone call history, and so on). For more information on Apple's security and privacy around iCloud, refer to http://support.apple.com/en-us/HT202303. If you do not trust Apple or it's cloud, then you can also take a backup of your device using iTunes, described in the next section. Taking backups using iTunes Originally, iTunes was used to take backups for iOS devices. You can still use iTunes and it's likely you will have a second backup even if you are using iCloud, simply for a quick restore if nothing else. Backups are usually pretty small. The reason is that the operating system is not part of backups, since users can't edit any of those files. Therefore, you can use an ipsw file (the operating system) to restore a device. These are accessed through Apple Configurator or through iTunes if you have a restore file waiting to be installed. These can be seen in ~/Library/iTunes, and the name of the device and its software updates, as can be seen in the following screenshot: IPSW files Backups are stored in the ~/Library/Application Support/MobileSync/Backup directory. Here, you'll see a number of directories that are associated with the UDID of the devices, and within those, you'll see a number of files that make up the modular incremental backups beyond the initial backup. It's a pretty smart system and allows you to restore a device at different points in time without taking too long to perform each backup. Backups are stored in the Documents and SettingsUSERNAMEApplication DataApple ComputerMobileSyncBackup directory on Windows XP and in the UsersUSERNAMEAppDataRoamingApple ComputerMobileSyncBackup directory for newer operating systems. To enable an iTunes back up, plug a device into a computer, and then open iTunes. Click on the device for it to show the device details screen. The top section of the screen is for Backups (in the following screenshot, you can set a back up to This computer, which takes a backup on the computer you are on). I would recommend you to always choose the Encrypt iPhone backup option as it forces you to save a password in order to restore the back up. Additionally, you can use the Back Up Now button to kick off the first back up, as shown in the following screenshot: iTunes Viewing iOS data in iTunes To show why it's important to encrypt backups, let's look at what can be pulled out of those backups. There are a few tools that can extract backups, provided you have a password. Here, we'll look at iBackup Extractor to view the backup of your browsing history, calendars, call history, contacts, iMessages, notes, photos, and voicemails. To get started, download iBackup Extractor from http://www.wideanglesoftware.com/ibackupextractor. When you open iBackup Extractor for the first time, simply choose the device backup you wish to extract in iBackup Extractor. As you can see in following screenshot, you will be prompted for a password in order to unlock the Backup key bag. Enter the password to unlock the system. Unlock the backups Note that the file tree in the following screenshot gives away some information on the structure of the iOS filesystem, or at least, the data stored in the backups of the iOS device. For now, simply click on Browser to see a list of files that can be extracted from the backup, as you can see in the next screenshot: View Device Contents Using iBackup Extractor Note the prevalence of SQL databases in the files. Most apps use these types of databases to store data on devices. Also, check out the other options such as extracting notes (many that were possibly deleted), texts (some that have been deleted from devices), and other types of data from devices. Now that we've exhausted backups and proven that you should really put a password in place for your back ups, let's finally get to some basic security tasks to be performed on these devices! Initial security checklist Apple has built iOS to be one of the most secure operating systems in the world. This has been made possible by restricting access to much of the operating system by end users, unless you jailbreak a device. In this article, we won't cover jail-breaking devices much due to the fact that securing the devices then becomes a whole new topic. Instead, we have focused on what you need to do, how you can do those tasks, what the impacts are, and, how to manage security settings based on a policy. The basic steps required to secure an iOS device start with encrypting devices, which is done by assigning a passcode to a device. We will then configure how much inactive time before a device requires a PIN and accordingly manage the privacy settings. These settings allow us to get some very basic security features under our belt, and set the stage to explain what some of the features actually do. Configuring a passcode The first thing most of us need to do on an iOS device is configure a passcode for the device. Several things happen when a passcode is enabled, as shown in the following steps: The device is encrypted. The device then requires a passcode to wake up. An idle timeout is automatically set that puts the device to sleep after a few minutes of inactivity. This means that three of the most important things you can do to secure a device are enabled when you set up a passcode. Best of all, Apple recommends setting up a passcode during the initial set up of new devices. You can manage passcode settings using policies (or profiles as Apple likes to call them in iOS). Best of all—you can set a passcode and then use your fingerprint on the Home button instead of that passcode. We have found that by the time our phone is out of our pocket and if our finger is on the home button, the device is unlocked by the time we check it. With iPhone 6 and higher versions, you can now use that same fingerprint to secure payment information. Check whether a passcode has been configured, and if needed, configure a passcode using the Settings app. The Settings app is by default on the Home screen where many settings on the device, including Wi-Fi networks the device has been joined to, app preferences, mail accounts, and other settings are configured. To set a passcode, open the Settings app and tap on Touch ID & Passcode If a passcode has been set, you will see the Turn Passcode Off (as seen in the following screenshot) option If a passcode has not been set, then you can do so at this screen as well Additionally, you can change a passcode that has been set using the Change Passcode button and define a fingerprint or additional fingerprints that can be used with a touch ID There are two options in the USE TOUCH ID FOR section of the screen. You can choose whether, or not, you need to enter the passcode in order to unlock a phone, which you should use unless the device is also used by small children or as a kiosk. In these cases, you don't need to encrypt or take a backup of the device anyway. The second option is to force the entering of a passcode while using the App Store and iTunes. This can cost you money if someone else is using your device, so let the default value remain, which requires you to enter a passcode to unlock the options. Configure a Passcode The passcode settings are very easy to configure; so, they should be configured when possible. Scroll down on this screen and you'll see several other features, as shown in the next screenshot. The first option on the screen is Simple Passcode. Most users want to use a simple pin with an iOS device. Trying to use alphanumeric and long passcodes simply causes most users to try to circumvent the requirement. To add a fingerprint as a passcode, simply tap on Add a Fingerprint…, which you can see in the preceding screenshot, and follow the onscreen instructions. Additionally, the following can be accessed when the device is locked, and you can choose to turn them off: Today: This shows an overview of upcoming calendar items Notifications View: This shows you the recent push notifications (apps that have updates on the device) Siri: This represents the voice control of the device Passbook: This tool is used to make payments and display tickets for concert venues and meetups Reply with Message: This tool allows you to send a text reply to an incoming call (useful if you're on the treadmill) Each organization can decide whether it considers these options to be a security risk and direct users how to deal with them, or they can implement a policy around these options. Passcode Settings There aren't a lot of security options around passcodes and encryption, because by and large, Apple secures the device by giving you fewer options than you'll actually use. Under the hood, (for example, through Apple Configurator and Mobile Device Management) there are a lot of other options, but these aren't exposed to end users of devices. For the most part, a simple four-character passcode will suffice for most environments. When you complicate passcodes, devices become much more difficult to unlock, and users tend to look for ways around passcode enforcement policies. The passcode is only used on the device, so complicating the passcode will only reduce the likelihood that a passcode would be guessed before swiping open a device, which typically occurs within 10 tries. Finally, to disable a passcode and therefore encryption, simply go to the Touch ID & Passcode option in the Settings app and tap on Turn Passcode Off. Configuring privacy settings Once a passcode is set and the device is encrypted, it's time to configure the privacy settings. Third-party apps cannot communicate with one another by default in iOS. Therefore, you must enable communication between them (also between third-party apps and built-in iOS apps that have APIs). This is a fundamental concept when it comes to securing iOS devices. To configure privacy options, open the Settings app and tap on the entry for Privacy. On the Privacy screen, you'll see a list of each app that can be communicated with by other apps, as shown in the following screenshot: Privacy Options As an example, tap on the Location Services entry, as shown in the next screenshot. Here, you can set which apps can communicate with Location Services and when. If an app is set to While Using, the app can communicate with Location Services when the app is open. If an app is set to Always, then the app can only communicate with Location Services when the app is open and not when it runs in the background. Configure Location Services On the Privacy screen, tap on Photos. Here, you have fewer options because unlike the location of a device, you can't access photos when the app is running in the background. Here, you can enable or disable an app by communicating with the photo library on a device, as seen in the next screenshot: Configure What Apps Can Access Your Camera Roll Each app should be configured in such a way that it can communicate with the features of iOS or other apps that are absolutely necessary. Other privacy options which you can consider disabling include Siri and Handoff. Siri has the voice controls of an iOS. Because Siri can be used even when your phone is locked, consider to disable it by opening the Settings app, tapping on General and then on Siri, and you will be able disable the voice controls. To disable Handoff, you should use the General System Preference pane in any OS X computer paired to an iOS device. There, uncheck the Allow Handoff between this Mac and your iCloud devices option. Safari and built-in App protections Web browsers have access to a lot of data. One of the most popular targets on other platforms has been web browsers. The default browser on an iOS device is Safari. Open the Settings app and then tap on Safari. The Safari preferences to secure iOS devices include the following: Passwords & AutoFill: This is a screen that includes contact information, a list of saved passwords and credit cards used in web browsers. This data is stored in an iCloud Keychain if iCloud Keychain has been enabled in your phone. Favorites: This performs the function of bookmark management. This shows bookmarks in iOS. Open Links: This configures how links are managed. Block Pop-ups: This enables a pop-up blocker. Scroll down and you'll see the Privacy & Security options (as seen in the next screenshot). Here, you can do the following: Do Not Track: By this, you can block the tracking of browsing activity by websites. Block Cookies: A cookie is a small piece of data sent from a website to a visitor's browser. Many sites will send cookies to third-party sites, so the management of cookies becomes an obstacle to the privacy of many. By default, Safari only allows cookies from websites that you visit (Allow from Websites I Visit). Set the Cookies option to Always Block in order to disable its ability to accept any cookies; set the option to Always Allow to accept cookies from any source; and set the option to Allow from Current Website Only to only allow cookies from certain websites. Fraudulent Website Warning: This blocks phishing attacks (sites that only exist to steal personal information). Clear History and Website Data: This clears any cached history, web files, and passwords from the Safari browser. Use Cellular Data: When this option is turned off, it disables web traffic over cellular connections (so web traffic will only work when the phone is connected to a Wi-Fi network). Configure Privacy Settings for Safari There are also a number of advanced options that can be accessed by clicking on the Advanced button, as shown in the following screenshot: Configure the Advanced Safari Options These advanced options include the following: Website Data: This option (as you can see in the next screenshot) shows the amount of data stored from each site that caches files on the device, and allows you to swipe left on these entries to access any files saved for the site. Tap on Remove All Website Data to remove data for all the sites at once. JavaScript: This allows you to disable any JavaScripts from running on sites the device browses. Web Inspector: This shows the device in the Develop menu on a computer connected to the device. If the Web Inspector option has been disabled, use Advanced Preferences in the Safari Preferences option of Safari. View Website Data On Devices Browser security is an important aspect of any operating system. Predictive search and spotlight The final aspect of securing the settings on an iOS device that we'll cover in this article includes predictive search and spotlight. When you use the spotlight feature in iOS, usage data is sent to Apple along with the information from Location Services. Additionally, you can search for anything on a device, including items previously blocked from being accessed. The ability to search for blocked content warrants the inclusion in locking down a device. That data is then used to generate future searches. This feature can be disabled by opening the Settings app, tap on Privacy, then Location Services, and then System Services. Simply slide Spotlight Suggestions to Off to disable the location data from going over that connection. To limit the type of data that spotlight sends, open the Settings app, tap on General, and then on Spotlight Search. Uncheck each item you don't want indexed in the Spotlight database. The following screenshot shows the mentioned options: Configure What Spotlight Indexes These were some of the basic tactical tasks that secure devices. Summary This article was a whirlwind of quick changes that secure a device. Here, we paired devices, took a backup, set a passcode, and secured app data and Safari. We showed how to manually do some tasks that are set via policies. Resources for Article: Further resources on this subject: Creating a Brick Breaking Game [article] New iPad Features in iOS 6 [article] Sparrow iOS Game Framework - The Basics of Our Game [article]
Read more
  • 0
  • 0
  • 13184
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at ₹800/month. Cancel anytime
article-image-storing-and-generating-graphs-stored-data
Packt
01 Oct 2015
20 min read
Save for later

Storing and Generating Graphs of the Stored Data

Packt
01 Oct 2015
20 min read
In this article by Matthijs Kooijman, the author of the book Building Wireless Sensor Networks Using Arduino, we will explore some of the ways to persistently store the collected sensor data and visualize the data using convenient graphs. First, you will see how to connect your coordinator to the Internet and send its data to the Beebotte cloud platform. You will learn how to create a custom dashboard in this platform that can show you the collected data in a convenient graph format. Second, you will see how you can collect and visualize the data on your own computer instead of sending it to the Internet directly. (For more resources related to this topic, see here.) For the first part, you will need some shield to connect your coordinator Arduino to the Internet in addition to the hardware that has been recommended for the coordinator in the previous chapter. This article provides examples for the following two shields: The Arduino Ethernet shield (https://www.arduino.cc/en/Main/ArduinoEthernetShield) The Adafruit CC3000 Wi-Fi shield (https://www.adafruit.com/products/1491) If possible, the Ethernet shield is recommended as its library is small, and it is easier to keep a reliable connection using a wired connection. Additionally, the CC3000 shield conflicts with the SparkFun XBee shield and this would require some modification on the latter's part to make them work together. For the second part, no additional hardware is needed. There are other Ethernet or Wi-Fi shields that you can use. Storing your data in the cloud When it comes to storing your data online somewhere, there are literally dozens of online platforms that offer some kind of data storage service aimed at collecting sensor data. Each of these has different features, complexity, and cost, and you are encouraged to have a look at what is available. Even though a lot of platforms are available, almost none of them are really suited for a hobby sensor network like the one presented in this article. Most platforms support the basic collection of data and offer some web API to access the data but there were two requirements that ruled out most of the platforms: It has to be affordable for a home user with just a bit of data. Ideally, there is a free version to get started. It has to support creating a dashboard that can show data and graphs and also show input elements that can be used to talk back to the network. (This will be used in the next chapter to create an online thermostat). When this article was written, only two platforms seemed completely suitable: Beebotte (https://beebotte.com/) and Adafruit IO (https://io.adafruit.com/). The examples in this article use Beebotte because at the time of writing, Adafruit IO was not publicly available yet and Beebotte currently has some additional features. However, you are encouraged to check out Adafruit IO as an alternative. As both the platforms use the MQTT protocol (explained in the next section), you should be able to reuse the example code with just minimal changes for Adafruit IO. Introducing Beebotte Beebotte, like most of these services, can be seen as a big online database that stores any data you send to it and allows retrieving any data you are interested in. Additionally, you can easily create dashboards that allow you to look at your data and even interact with it through various configurable widgets. By the end of this chapter, you might have a dashboard that looks like this: Before showing you how to talk to Beebotte from your Arduino, some important concepts in the Beebotte system will be introduced: channels, resources, security tokens, and access protocols. The examples in this article serve to get started with Beebotte, but will certainly not cover all features and details. Be sure to check out the extensive documentation on the Beebotte site at https://beebotte.com/overview. Channels and resources All data collected by Beebotte is organized into resources, each representing a single series of data. All data stored in a resource signifies the same thing, such as the temperature in your living room or on/off status of the air conditioner but at different points in time. This kind of data is also often referred to as time-series data. To keep your data organized, Beebotte supports the concept of channels. Essentially, a channel is just a group of resources that somehow belong together. Typically, a channel represents a single device or data source but you are free to group your resources in any way that you see fit. In this example, every sensor module in the network will get its own channel, each containing a resource to store the temperature data and a resource to store the humidity data. Security To be able to access the data stored in your Beebotte account or publish new data, every connection needs to be authenticated. This happens using a secret token or key (similar to a password) that you configure in your Arduino code and proves to the Beebotte server that you are allowed to access the data. There are two kinds of secrets currently supported by Beebotte: Your account secret key: This is a single key for your account, which allows access to all resources and all channels in your account. It additionally allows the creation and deletion of the channels and resources. Channel tokens: Each channel has an associated channel token, which allows reading and writing data from that channel only. Additionally, the channel token can be regenerated if the token is ever compromised. This example uses the account secret key to authenticate the connection. It would be better to use a more limited channel token (to limit the consequences if the token is leaked) but in this example, as the coordinator forwards data for multiple sensor nodes (each of which have their own channel), a channel token does not provide enough access. As an alternative, you could consider using a single channel containing all the resources (named, for example, "Livingroom_Temperature" to still allow grouping) so that you can use the slightly more secure channel tokens. In the future, Beebotte might also support more flexible limited tokens that support writing to more than one channel. The examples in this article use an unencrypted connection, so make sure at least your Wi-Fi connection is encrypted (using WPA or WPA2). If you are working with particularly sensitive information, be sure to consider using SSL/TLS for the connection. Due to limited microcontroller speeds, running SSL/TLS directly on the microcontroller does not seem feasible, so this would need external cryptographic hardware or support on the Wi-Fi / Ethernet shield that is being used. At the time of writing, there does not seem to be any shield that directly supports this but it seems that at least the ESP2866-based shields and the Arduino Yún could be made to support this and the upcoming Arduino Wi-Fi shield 101 might support it as well (but this is out of the scope for this article). Access protocols To store new data and access the existing data over the Internet, a few different access methods are supported by Beebotte: HTTP / REST: The hypertext transfer protocol (HTTP) is the protocol that powers the web. Originally, it was used to let a web browser request a web page from a server, but now HTTP is also commonly used to let all kinds of devices send and request arbitrary data (instead of webpages) to and from servers as well. In this case, the server is commonly said to export an HTTP or REST (Relational state transfer) API.HTTP APIs are convenient, since HTTP is a very widespread protocol and HTTP libraries are available for most programming languages and environments. WebSockets: A downside of HTTP is that it is not very convenient for sending events from the server to the client. A server can only send data to the client after the client sends a request, which means the client must poll for new events continuously.To overcome this, the WebSockets standard was created. WebSockets is a protocol on top of HTTP that keeps a connection (socket) open indefinitely, ready for the server to send new data whenever it wants to, instead of having to wait for the client to request new data. MQTT: The message queuing telemetry transport protocol (MQTT) is a so-called publish/subscribe (pubsub) protocol. The idea is that multiple devices can connect to a central server and each can publish a message to a given topic and also subscribe to any number of topics. Whenever a message is published to a topic, it is automatically forwarded to all the devices that have subscribed to the same topic.MQTT, like WebSockets, keeps a connection open continuously so that both the client and server can send data at any time, making this protocol especially suitable for real-time data and events. MQTT can not be used to access historical data, though. A lot of alternative platforms only support the HTTP access protocol, which works fine to push and access the data and would be suitable for the examples in this chapter but is less suitable to control your network from the Internet, as used in the next chapter. To prepare for this, the examples in this chapter will already use the MQTT protocol, which supports both the use cases efficiently. Sending your data to Beebotte Now that you have learned about some important Beebotte concepts, you are ready to send your collected sensor data to Beebotte. First, you will prepare Beebotte and your Arduino to connect to each other. Then, you will write a sketch for your coordinator to send the data. Finally, you will see how to access and visualize the stored data. Preparing Beebotte Before you can start sending data to Beebotte, you will have to prepare the proper channels and resources to store the data. This example uses two channels: "Livingroom" and "Study", referring to the rooms where the sensors have been placed. You should, of course, use names that reflect your setup and adapt things if you have more or fewer sensors. The first step is to register an account on beebotte.com. Once you have done this, you can access your Control Panel, which will initially show you an empty list of channels: You can create a new channel by clicking the Create New channel button. In the resulting window, you can fill in a name and description for the channel and define the resources. This should look something like this: After creating a channel for every sensor node that you have built, you have prepared Beebotte to receive all the sensor data. The next step is to modify the coordinator sketch to actually send the data. Connecting your Arduino to the Internet In order to let your coordinator send its data to Beebotte, it must be connected to the Internet somehow. There are plenty of shields out there that add wired Ethernet or wireless Wi-Fi connectivity to your Arduino. Wi-Fi shields are typically a lot more expensive than the Ethernet ones but the recently introduced ESP2866 cheap Wi-Fi chipset will likely change that. (However, at the time of writing, no ready-to-use Arduino shield was available yet.) As the code to connect your Arduino to the Internet will be significantly different for each shield, every part of the code will not be discussed in this article. Instead, we will focus on the code that connects to the MQTT server and publishes the data, assuming that the Internet connection is already set up. In the code bundle for this article, two complete examples are available for use with the Arduino Ethernet shield and Adafruit CC3000 shield. These examples should be usable as a starting point to use the other hardware as well. Some things to keep in mind when selecting your hardware have been listed, as follows: Check carefully for conflicting pins. For example, the Adafruit CC3000 Wi-Fi shield uses pins 2 and 3[t1]  to communicate with the shield but you might be using these pins to talk to the XBee module as well (particularly, when using the SparkFun XBee shield). The libraries for the various Wi-Fi shields take up a lot of code space on the Arduino. For example, using the SparkFun or Adafruit CC3000 library along with the Adafruit MQTT library fills up most of the available code space on an Arduino Uno. The Ethernet library is a bit (but not much) smaller. It is not always easy to keep a reliable Wi-Fi connection. In theory, this is a matter of simply reconnecting when the Wi-Fi connection is failing but in practice it can be tricky to implement this completely reliably. Again, this is easier with wired Ethernet as it does not have the same disconnection issues as Wi-Fi. If you use a different hardware than recommended (including the recently announced Arduino Ethernet shield 2), you will likely need a different Arduino library and need to make changes to the example code provided. Writing the sketch To send the data to Beebotte, the Coordinator.ino sketch from the previous chapter needs to be modified. As noted before, only the MQTT code will be shown here. However, the code to establish the Internet connection is included in the full examples in the code bundle (in the Coordinator_Ethernet.ino and Coordinator_CC3000.ino sketches). This example uses the "Adafruit MQTT library" for the MQTT protocol, which can be installed through the Arduino library manager or can be found here: https://github.com/adafruit/Adafruit_MQTT_Library. Depending on the Internet shield, you might need more libraries as well (see the comments in the example sketches). Do not forget to add the appropriate includes for the libraries that you are using. For the MQTT library, this is: #include <Adafruit_MQTT.h> #include <Adafruit_MQTT_Client.h> To set up the MQTT library, you first need to define some settings: const char MQTT_SERVER[] PROGMEM = "mqtt.beebotte.com"; const char MQTT_CLIENTID[] PROGMEM = ""; const char MQTT_USERNAME[] PROGMEM = "your_key_here"; const char MQTT_PASSWORD[] PROGMEM = ""; const int MQTT_PORT = 1883; This defines the connection settings to use for the MQTT connection. These are appropriate for Beebotte; if you are using some other platform, check its documentation for the appropriate settings. Note that here the username must be set to your account's secret key, for example: const char MQTT_USERNAME[] PROGMEM = "840f626930a07c87aa315e27b22448468844edcad03fe34f551ac747533f544f"; If you are using a channel token, it must be set to the token prefixed with "token:", such as: const char MQTT_USERNAME[] PROGMEM = "token:1438006626319_UNJoxdmKBoMFIPt7"; The password is unused and can be empty, just like the client identifier. The port is just the default MQTT port number, 1883. Note that all the string variables are marked with the PROGMEM keyword. This tells the compiler to store the string variables in program memory, just like the F() macro you have seen before does (which also uses the PROGMEM keyword underwater). However, the F() macro can only be used in the functions, which is why these variables use the PROGMEM keyword directly. This also means that the extra checking offered by the F() macro is not available, so be careful not to mix up the normal and PROGMEM strings here as this will not result in any compiler error and instead things will be broken when you run the code. Using the configuration constants defined earlier, you can now define the main mqtt object: Adafruit_MQTT_Client mqtt(&client, MQTT_SERVER, MQTT_PORT, MQTT_CLIENTID, MQTT_USERNAME, MQTT_PASSWORD); There are a few different flavors of this object. For example, there are flavors optimized for the specific hardware and corresponding libraries, and there is also a generic flavor that works with any hardware whose library exposes a generic Client object (like most libraries currently do). The latter flavor, Adafruit_MQTT_Client, is used in this example and should be fine. The &client [t2] part of this line refers to a previously created Client object (not shown here as it depends on the Internet shield used), which is used by the MQTT library to set up the MQTT connection. To actually connect to the MQTT server, a function called connect() will be defined. This function is called to connect once at startup and to reconnect every time when the publishing of some data fails. In the CC3300 version, this function associates to the WiFi access point and then sets up the connection to the MQTT server. On the Ethernet version, where the network is always available after initial initialization, the connect() function only sets up the MQTT connection. The latter version is shown as follows: void connect() { client.stop(); // Ensure any old connection is closed uint8_t ret = mqtt.connect(); if (ret == 0) DebugSerial.println(F("MQTT connected")); else DebugSerial.println(mqtt.connectErrorString(ret)); } This calls mqtt.connect() to connect to the MQTT server and writes a debug message to report the success or failure. Note that mqtt.connect() returns a number as an error code (with 0 meaning OK), which is translated to a human readable error message using the mqtt.connectErrorString() function. Now, to actually publish a single value, there is the publish() function: void publish(const __FlashStringHelper *resource, float value) { // Use JSON to wrap the data, so Beebotte will remember the data // (instead of just publishing it to whoever is currently listening). String data; data += "{"data": "; data += value; data += ", "write": true}"; DebugSerial.print(F("Publishing ")); DebugSerial.print(data); DebugSerial.print(F( " to ")); DebugSerial.println(resource); // Publish data and try to reconnect when publishing data fails if (!mqtt.publish(resource, data.c_str())) { DebugSerial.println(F("Failed to publish, trying reconnect...")); connect(); if (!mqtt.publish(resource, data.c_str())) DebugSerial.println(F("Still failed to publish data")); } } This function takes two parameters: the name of the resource to publish to and the value to publish. Note the type of the resource parameter: __FlashStringHelper* is similar to the more common char* string type but indicates that the string is stored in the program memory instead of RAM. This is also the type returned by the F() macro that you have seen before. Just like the MQTT server configuration values that used the PROGMEM keyword before, the MQTT library also expects the MQTT topic names to be stored in the program memory. The actual value is sent using the JSON (JavaScript object notation) format. For example, for a temperature of 20 degrees, it constructs {"data": 20.00, "write": true}. This format allows, in addition to transmitting the value, indicating that Beebotte should store the value so that it can be retrieved later. If the write value is false, or not present, Beebotte will only forward the value to any other devices currently subscribed to the appropriate topic without saving it for later. This example uses some quick and dirty string concatenation to generate the JSON. If you want something more robust and elegant, have a look at the ArduinoJson library at https://github.com/bblanchon/ArduinoJson. If publishing the data fails, it is likely that the WiFi or MQTT connection has failed and so it attempts to reconnect and publish the data once more. As before, there is a processRxPacket() function that gets called when a radio packet is received through the XBee module: void processRxPacket(ZBRxResponse& rx, uintptr_t) { Buffer b(rx.getData(), rx.getDataLength()); uint8_t type = b.remove<uint8_t>(); XBeeAddress64 addr = rx.getRemoteAddress64(); if (addr == 0x0013A20040DADEE0 && type == 1 && b.len() == 8) { publish(F("Livingroom/Temperature"), b.remove<float>()); publish(F("Livingroom/Humidity"), b.remove<float>()); return; } if (addr == 0x0013A20040E2C832 && type == 1 && b.len() == 8) { publish(F("Study/Temperature"), b.remove<float>()); publish(F("Study/Humidity"), b.remove<float>()); return; } DebugSerial.println(F("Unknown or invalid packet")); printResponse(rx, DebugSerial); } Instead of simply printing the packet contents as before, it figures out who the sender of the packet is and which Beebotte resource corresponds to it and calls the publish() function that was defined earlier. As you can see, the Beebotte resources are identified using "Channel/Resource", resulting in a unique identifier for each resource (which is later used in the MQTT message as the topic identifier). Note that the F() macro is used for the resource names to store them in program memory as this is what publish() and the MQTT library expect. If you run the resulting sketch and everything connects correctly, the coordinator will forward any sensor values that it receives to the Beebotte server. If you wait for (at most) five minutes to pass (or reset the sensor Arduino to have it send a reading right away) and then go to the appropriate channel in your Beebotte control panel, it should look something like this: Visualizing your data To easily allow the visualizing of your data, Beebotte supports dashboards. A dashboard is essentially a web page where you can add graphs, gauges, tables, buttons, and so on (collectively called widgets). These widgets can then display or control the data in one or more previously defined resources. To create such a dashboard, head over to the My Dashboards section of your control panel and click Create Dashboard to start building one. Once you set a name for the dashboard, you can start adding widgets to it. To display the temperature and humidity for all the sensors that you are using, you could use the Multi-line Chart widget. As the temperature and humidity values will be fairly far apart, it makes sense to put them each in a separate chart. Adding the temperature chart could look like this: If you also add a chart for the humidity, it should look like this: Here, only the living room sensor has been powered on, so no data is shown for the study yet. Also, to make the graphs a bit more interesting, some warm and humid air was breathed onto the sensor, causing the big spike in both the charts. There are plenty of other widget types that will prove useful to you. The Beebotte documentation provides information on the supported types, but you are encouraged to just play with the widgets a bit to see what they can add to your project. In the next chapter, you will see how to use the control widgets, which allow sending events and data back to the coordinator to control it. Accessing your data You have been accessing your data through a Beebotte dashboard so far. However, when these dashboards are not powerful enough for your needs or you want to access your data from an existing application, you can also access the recent and historical data through the Beebotte's HTTP or WebSocket API's. This gives you full control over the processing and display of the data without being limited in any way by what Beebotte offers in its dashboards. As creating a custom (web) application is out of the scope of this article, the HTTP and WebSocket API will not be discussed in detail. Instead, you should be able to find extensive documentation on this API on the Beebotte site at https://beebotte.com/overview. Summary In this article we learnt some of the ways to store the collected sensor data and visualize the data using convenient graphs. We also learnt as to how to save our data on the Cloud. Resources for Article: Further resources on this subject: TV Set Constant Volume Controller[article] Internet Connected Smart Water Meter[article] Getting Started with Arduino [article]
Read more
  • 0
  • 0
  • 13175

article-image-penetration-testing
Packt
04 Feb 2015
15 min read
Save for later

Penetration Testing

Packt
04 Feb 2015
15 min read
In this article by Aamir Lakhani and Joseph Muniz, authors of the book Penetration Testing with Raspberry Pi, we will see the various LAN- and wireless-based attack scenarios, using tools found in Kali Linux that are optimized for a Raspberry Pi. These scenarios include scanning, analyzing and capturing network traffic. (For more resources related to this topic, see here.) The Raspberry Pi has limited performance capabilities due to its size and processing power. It is highly recommended that you test the following techniques in a lab prior to using a Raspberry Pi for a live penetration test. Network scanning Network reconnaissance is typically time-consuming, yet it is the most important step when performing a penetration test. The more you know about your target, the more likely it is that you will find the fastest and easiest path to success. The best practice is starting with reconnaissance methods that do not require you to interact with your target; however, you will need to make contact eventually. Upon making contact, you will need to identify any open ports on a target system as well as map out the environment to which it's connected. Once you breach a system, typically there are other networks that you can scan to gain deeper access to your target's network. One huge advantage of the Raspberry Pi is its size and mobility. Typically, Kali Linux is used from an attack system outside a target's network; however, tools such as PWNIE Express and small systems that run Kali Linux, such as a Raspberry Pi, can be placed inside a network and be remotely accessed. This gives an attacker a system inside the network, bypassing typical perimeter defenses while performing internal reconnaissance. This approach brings the obvious risks of having to physically place the system on the network as well as create a method to communicate with it remotely without being detected; however, if successful, this can be very effective. Let's look at a few popular methods to scan a target network. We'll continue forward assuming that you have established a foothold on a network and now want to understand the current environment that you have connected to. Nmap The most popular open source tool used to scan hosts and services on a network is Nmap (short for Network Mapper). Nmap's advanced features can detect different applications running on systems as well as offer services such as the OS fingerprinting features. Nmap can be very effective; however, it can also be easily detected unless used properly. We recommend using Nmap in very specific situations to avoid triggering a target's defense systems. For more information on how to use Nmap, visit http://nmap.org/. To use Nmap to scan a local network, open a terminal window and type nmap (target), for example, nmap www.somewebsite.com or nmap 192.168.1.2. There are many other commands that can be used to tune your scan. For example, you can tune how stealthy you want to be or specify to store the results in a particular location. The following screenshot shows the results after running Nmap against www.thesecurityblogger.com. Note that this is an example and is considered a noisy scan. If you simply type in either of the preceding two commands, it is most likely that your target will easily recognize that you are performing an Nmap scan. There are plenty of online resources available to learn how to master the various features for Nmap. Here is a reference list of popular nmap commands: nmap 192.168.1.0/24: This scans the entire class C range nmap -p <port ranges>: This scans specific ports nmap -sP 192.168.1.0/24: This scans the network/find servers and devices that are running nmap –iflist: This shows host interfaces and routes nmap –sV 192.168.1.1: This detects remote services' version numbers nmap –sS 192.168.1.1: This performs a stealthy TCP SYN scan nmap –sO 192.168.1.1: This scans for the IP protocol nmap -192.168.1.1 > output.txt: This saves the output from the scan to the text file nmap –sA 192.168.1.254: This checks whether the host is protected by a firewall nmap –PN 192.168.1.1: This scans the host when it is protected by a firewall nmap --reason 192.168.1.1: This displays the reason a port is in a particular state nmap --open 192.168.1.1: This only shows open or possibly open ports The Nmap GUI software Zenmap is not included in the Kali Linux ARM image. It is also not recommended over using the command line when running Kali Linux on a Raspberry Pi. Wireless security Another attack vector that can be leveraged on a Raspberry Pi with a Wi-Fi adapter is targeting wireless devices such as mobile tablets and laptops. Scanning wireless networks, once they are connected, is similar to how scanning is done on a LAN; however, typically a layer of password decryption is required before you can connect to a wireless network. Also, wireless network identifier known as Service Set Identifier (SSID) might not be broadcasted but will still be visible when you use the right tools. This section will cover how to bypass wireless onboarding defenses so that you can access a target's Wi-Fi network and perform the penetration testing steps. Looking at a Raspberry Pi with Kali Linux, one of the use cases is hiding the system inside or near a target's network and launching wireless attacks remotely. The goal will be to enable the Raspberry Pi to access the network wirelessly and provide a remote connection back to the attacker. The attacker can be nearby using wireless to control the Raspberry Pi until it gains wireless access. Once on the network, a backdoor can be established so that the attacker can communicate with the Raspberry Pi from anywhere in the world and launch attacks. Cracking WPA/WPA2 A commonly found security protocol for protecting wireless networks is Wi-Fi Protected Access (WPA). WPA was later replaced by WPA2 and it will be probably what you will be up against when you perform a wireless penetration test. WPA and WPA2 can be cracked with Aircrack. Kali Linux includes the Aircrack suite, which is one of the most popular applications to break wireless security. Aircrack works by gathering packets seen on a wireless connection to either mathematically analyze the data to crack weaker protocols such as Wired Equivalent Privacy (WEP), or use brute force on the captured data with a wordlist. Cracking WPA/WPA2 can be done due to a weakness in the four-way handshake between the client and the access point. In summary, a client will authenticate to an access point and go through a four-step process. This is the time when the attacker is able to grab the password and use a brute force approach to identify it. The time-consuming part in this is based on how unique the network password is, how extensive your wordlist that will be used to brute force against the password is, and the processing power of the system. Unfortunately, the Raspberry Pi lacks the processing power and the hard drive space to accommodate large wordlist files. So, you might have to crack the password off-box with a tool such as John the Ripper. We recommend this route for most WPA2 hacking attempts. Here is the process to crack a WPA running on a Linksys WRVS4400N wireless router using a Raspberry Pi on-box options. We are using a WPA example so that the time-consuming part can be accomplished quickly with a Raspberry Pi. Most WPA2 cracking examples would take a very long time to run from a Raspberry Pi; however, the steps to be followed are the same to run on a faster off-box system. The steps are as follows: Start Aircrack by opening a terminal and typing airmon-ng; In Aircrack, we need to select the desired interface to use for the attack. In the previous screenshot, wlan0 is my Wi-Fi adapter. This is a USB wireless adapter that has been plugged into my Raspberry Pi. It is recommended that you hide your Mac address while cracking a foreign wireless network. Kali Linux ARM does not come with the program macchanger. So, you should download it by using the sudo apt-get install macchanger command in a terminal window. There are other ways to change your Mac address, but macchanger can provide a spoofed Mac so that your device looks like a common network device such as a printer. This can be an effective way to avoid detection. Next, we need to stop the interface used for the attack so that we can change our Mac address. So, for this example, we will be stopping wlan0 using the following commands: airmon-ng stop wlan0 ifconfig wlan0 down Now, let's change the Mac address of this interface to hide our true identity. Use macchanger to change your Mac to a random value and specify your interface. There are options to switch to another type of device; however, for this example, we will just leave it as a random Mac address using the following command: macchanger -r wlan0 Our random value is b0:43:3a:1f:3a:05 in the following screenshot. Macchanger shows our new Mac as unknown. Now that our Mac is spoofed, let's restart airmon-ng with the following command: airmon-ng start wlan0 We need to locate available wireless networks so that we can pick our target to attack. Use the following command to do this: airodump-ng wlan0 You should now see networks within range of your Raspberry Pi that can be targeted for this attack. To stop the search once you identify a target, press Ctrl + C. You should write down the Mac address, also known as BSSID, and the channel, also known as CH, used by your target network. The following screenshot shows that our target with ESSID HackMePlease is running WPA on CH 6: The next step is running airodump against the Mac address that you just copied. You will need the following things to make this work: The channel being used by the target The Mac address (BSSID) that you copied A name for the file to save your data Let's run the airodump command in the following manner: airodump-ng –c [channel number] –w [name of file] –-bssid [target ssid] wlan0 This will open a new terminal window after you execute it. Keep that window open. Open another terminal window that will be used to connect to the target's wireless network. We will run aireplay using the following command: aireplay-ng-deauth 1 –a [target's BSSID] –c [our BSSID] [interface] For our example, the command will look like the following: aireplay-ng -–deauth 1 –a 00:1C:10:F6:04:C3 –c 00:0f:56:bc:2c:d1 wlan0 The following screenshot shows the launch of the preceding command: You may not get the full handshake when you run this command. If that happens, you will have to wait for a live user to authenticate you to the access point prior to launching the attack. The output on using Aircrack may show you something like Opening [file].cap a few times followed by No valid WPA handshakes found, if you didn't create a full handshake and somebody hasn't authenticated you by that time. Do not proceed to the next step until you capture a full handshake. The last step is to run Aircrack against the captured data to crack the WPA key. Use the –w option to specify the location of a wordlist that will be used to scan against the captured data. You will use the .cap file that was created earlier during step 9, so we will use the name capturefile.cap in our example. We'll do this using the following command: Aircrack-ng –w ./wordlist.lst wirelessattack.cap The Kali Linux ARM image does not include a wordlist.lst file for cracking passwords. Usually, default wordlists are not good anyway. So, it is recommended that you use Google to find an extensive wordlist (see the next section on wordlists for more information). Make sure to be mindful of the hard drive space that you have on the Raspberry Pi, as many wordlists might be too large to be used directly from the Raspberry Pi. The best practice for running process-intensive steps such as brute forcing passwords is to do them off-box on a more powerful system. You will see Aircrack start and begin trying each password in the wordlist file against the captured data. This process could take a while depending on the password you are trying to break, the number of words in your list, and the processing speed of the Raspberry Pi. We found that it ranges from a few hours to days, as it's a very tedious process and is possibly better-suited for an external system with more horsepower than a Raspberry Pi. You may also find that your wordlist doesn't work after waiting a few days to sort through the entire wordlist file. If Aircrack doesn't open and start trying keys against the password, you either didn't specify the location of the .cap file or the location of the wordlist.lst file, or you don't have the captured handshake data. By default, the previous steps store files in the root directory. You can move your wordlist file in the root directory to mimic how we ran the commands in the previous steps since all our files are located in the root directory folder. You can verify this by typing ls to list the current directory files. Make sure that you list the correct directories of each file that are called by each command. If your attack is successful, you should see something like the following screenshot that shows the identified password as sunshine: It is a good idea to perform this last step on a remote machine. You can set up a FTP server and push your .cap files to that FTP server. You can learn more about setting up an FTP server at http://www.raspberrypi.org/forums/viewtopic.php?f=36&t=35661. Creating wordlists There are many sources and tools that can be used to develop a wordlist for your attack. One popular tool called Custom Wordlist Generator (CeWL), allows you to create your own custom dictionary file. This can be extremely useful if you are targeting individuals and want to scrape their blogs, LinkedIn, or other websites for commonly used words. CeWL doesn't come preinstalled on the Kali Linux ARM image, so you will have to download it using apt-get install cewl. To use CeWL, open a terminal window and put in your target website. CeWL will examine the URL and create a wordlist based on all the unique words it finds. In the following example, we are creating a wordlist of commonly used words found on the security blog www.drchaos.com using the following command: cewl www.drchaos.com -w drchaospasswords.txt The following screenshot shows the launch of the preceding command: You can also find many examples of popular wordlists used as dictionary files on the Internet. Here are a few wordlist examples sources that you can use; however, be sure to research Google for other options as well: https://crackstation.net/buy-crackstation-wordlist-password-cracking-dictionary.html https://wiki.skullsecurity.org/Passwords Here is a dictionary that one of the coauthors put together: http://www.drchaos.com/public_files/chaos-dictionary.lst.txt Capturing traffic on the network It is great to get access to a target network. However, typically the next step, once a foothold is established, is to start looking at the data. To do this, you will need a method to capture and view network packets. This means turning your Raspberry Pi into a remotely accessible network tap. Many of these tools could overload and crash your Raspberry Pi. Look out for our recommendations regarding when to use a tuning method to avoid this from happening. Tcpdump Tcpdump is a command line based packet analyzer. You can use tcpdump to intercept and display TCP/IP and other packets that are transmitted and seen attached by the system This means the Raspberry Pi must have access to the network traffic that you intend to view or using tcpdump won't provide you with any useful data. Tcpdump is not installed with the default Kali Linux ARM image, so you will have to install it using the sudo apt-get install tcpdump command. Once installed, you can run tcpdump by simply opening a terminal window and typing sudo tcpdump. The following screenshot shows the traffic flow visible to us after the launch of the preceding command: As the previous screenshot shows, there really isn't much to see if you don't have the proper traffic flowing through the Raspberry Pi. Basically, we're seeing our own traffic while being plugged into an 802.1X-enabled switch, which isn't interesting. Let's look at how to get other system's data through your Raspberry Pi. Running tcpdump consumes a lot of the Raspberry Pi's processing power. We found that this could crash the Raspberry Pi by itself or while using it with other applications. We recommend that you tune your data capture to avoid this from happening. Man-in-the-middle attacks One common method to capture sensitive information is by performing a man-in-the-middle attack. By definition, a man-in-the-middle attack is when an attacker makes independent connections with victims while actively eavesdropping on the communication. This is typically done between a host and the systems. For example, a popular method to capture passwords is to act as a middleman between login credentials passed by a user to a web server. Summary This article introduced us to the various attack scenarios of penetration testing used with the tools available in Kali Linux, over a Raspberry Pi. This article also gave us detailed description of tools like Nmap, CeWL, and tcpdump, which are used for network scanning, creating wordlists, and analyzing network traffic respectively. Resources for Article: Further resources on this subject: Testing Your Speed [Article] Creating a 3D world to roam in [Article] Making the Unit Very Mobile – Controlling the Movement of a Robot with Legs [Article]
Read more
  • 0
  • 0
  • 13174

article-image-making-game-console-unity-part-2
Eliot Lash
28 Dec 2015
10 min read
Save for later

Making an In-Game Console in Unity Part 2

Eliot Lash
28 Dec 2015
10 min read
In part 1, I started walking you through making a console using uGUI, Unity’s built-in GUI framework. I'm showing you how to implement a simple input parser. Let's continue where we left off here in Part 2. We’re going to program the behavior of the console. I split this into a ConsoleController, which handles parsing and executing commands, and a view component that handles the communication between the ConsoleController and uGUI. This makes the parser code cleaner, and easier to switch to a different GUI system in the future if needed. First, make a new script called ConsoleController. Completely delete its contents and replace them with the following class: /// <summary> /// Handles parsing and execution of console commands, as well as collecting log output. /// Copyright (c) 2014-2015 Eliot Lash /// </summary> using UnityEngine; using System; using System.Collections.Generic; using System.Text; public delegate void CommandHandler(string[] args); public class ConsoleController { #region Event declarations // Used to communicate with ConsoleView public delegate void LogChangedHandler(string[] log); public event LogChangedHandler logChanged; public delegate void VisibilityChangedHandler(bool visible); public event VisibilityChangedHandler visibilityChanged; #endregion /// <summary> /// Object to hold information about each command /// </summary> class CommandRegistration { public string command { get; private set; } public CommandHandler handler { get; private set; } public string help { get; private set; } public CommandRegistration(string command, CommandHandler handler, string help) { this.command = command; this.handler = handler; this.help = help; } } /// <summary> /// How many log lines should be retained? /// Note that strings submitted to appendLogLine with embedded newlines will be counted as a single line. /// </summary> const int scrollbackSize = 20; Queue<string> scrollback = new Queue<string>(scrollbackSize); List<string> commandHistory = new List<string>(); Dictionary<string, CommandRegistration> commands = new Dictionary<string, CommandRegistration>(); public string[] log { get; private set; } //Copy of scrollback as an array for easier use by ConsoleView const string repeatCmdName = "!!"; //Name of the repeat command, constant since it needs to skip these if they are in the command history public ConsoleController() { //When adding commands, you must add a call below to registerCommand() with its name, implementation method, and help text. registerCommand("babble", babble, "Example command that demonstrates how to parse arguments. babble [word] [# of times to repeat]"); registerCommand("echo", echo, "echoes arguments back as array (for testing argument parser)"); registerCommand("help", help, "Print this help."); registerCommand("hide", hide, "Hide the console."); registerCommand(repeatCmdName, repeatCommand, "Repeat last command."); registerCommand("reload", reload, "Reload game."); registerCommand("resetprefs", resetPrefs, "Reset & saves PlayerPrefs."); } void registerCommand(string command, CommandHandler handler, string help) { commands.Add(command, new CommandRegistration(command, handler, help)); } public void appendLogLine(string line) { Debug.Log(line); if (scrollback.Count >= ConsoleController.scrollbackSize) { scrollback.Dequeue(); } scrollback.Enqueue(line); log = scrollback.ToArray(); if (logChanged != null) { logChanged(log); } } public void runCommandString(string commandString) { appendLogLine("$ " + commandString); string[] commandSplit = parseArguments(commandString); string[] args = new string[0]; if (commandSplit.Length < 1) { appendLogLine(string.Format("Unable to process command '{0}'", commandString)); return; } else if (commandSplit.Length >= 2) { int numArgs = commandSplit.Length - 1; args = new string[numArgs]; Array.Copy(commandSplit, 1, args, 0, numArgs); } runCommand(commandSplit[0].ToLower(), args); commandHistory.Add(commandString); } public void runCommand(string command, string[] args) { CommandRegistration reg = null; if (!commands.TryGetValue(command, out reg)) { appendLogLine(string.Format("Unknown command '{0}', type 'help' for list.", command)); } else { if (reg.handler == null) { appendLogLine(string.Format("Unable to process command '{0}', handler was null.", command)); } else { reg.handler(args); } } } static string[] parseArguments(string commandString) { LinkedList<char> parmChars = new LinkedList<char>(commandString.ToCharArray()); bool inQuote = false; var node = parmChars.First; while (node != null) { var next = node.Next; if (node.Value == '"') { inQuote = !inQuote; parmChars.Remove(node); } if (!inQuote && node.Value == ' ') { node.Value = ' n'; } node = next; } char[] parmCharsArr = new char[parmChars.Count]; parmChars.CopyTo(parmCharsArr, 0); return (new string(parmCharsArr)).Split(new char[] {' n'} , StringSplitOptions.RemoveEmptyEntries); } #region Command handlers //Implement new commands in this region of the file. /// <summary> /// A test command to demonstrate argument checking/parsing. /// Will repeat the given word a specified number of times. /// </summary> void babble(string[] args) { if (args.Length < 2) { appendLogLine("Expected 2 arguments."); return; } string text = args[0]; if (string.IsNullOrEmpty(text)) { appendLogLine("Expected arg1 to be text."); } else { int repeat = 0; if (!Int32.TryParse(args[1], out repeat)) { appendLogLine("Expected an integer for arg2."); } else { for(int i = 0; i < repeat; ++i) { appendLogLine(string.Format("{0} {1}", text, i)); } } } } void echo(string[] args) { StringBuilder sb = new StringBuilder(); foreach (string arg in args) { sb.AppendFormat("{0},", arg); } sb.Remove(sb.Length - 1, 1); appendLogLine(sb.ToString()); } void help(string[] args) { foreach(CommandRegistration reg in commands.Values) { appendLogLine(string.Format("{0}: {1}", reg.command, reg.help)); } } void hide(string[] args) { if (visibilityChanged != null) { visibilityChanged(false); } } void repeatCommand(string[] args) { for (int cmdIdx = commandHistory.Count - 1; cmdIdx >= 0; --cmdIdx) { string cmd = commandHistory[cmdIdx]; if (String.Equals(repeatCmdName, cmd)) { continue; } runCommandString(cmd); break; } } void reload(string[] args) { Application.LoadLevel(Application.loadedLevel); } void resetPrefs(string[] args) { PlayerPrefs.DeleteAll(); PlayerPrefs.Save(); } #endregion } I’ve tried to comment where appropriate, but I’ll give you a basic rundown of this class. It maintains a registry of methods that are mapped to string command names, as well as associated help text. This allows the “help” command to print out all the available commands along with extra info on each one. It keeps track of the output scrollback as well as the history of user-entered commands (this is to aid implementation of bash-style command history paging, which is left as an exercise to the reader. Although I have implemented a simple command, ‘!!’ which repeats the most recent command.) When the view receives command input, it passes it to runCommandString() which calls parseArguments() to perform some rudimentary string parsing using a space as a delimiter. It then calls runCommand() which tries to look up the corresponding method in the command registration dictionary, and if it finds it, calling it with the remaining arguments. Commands can call appendLogLine() to write to the in-game console log, and of course execute arbitrary code. Moving on, we will implement the view. Attach a new script to the ConsoleView object (the parent of ConsoleViewContainer) and call it ConsoleView. Replace its contents with the following: /// <summary> /// Marshals events and data between ConsoleController and uGUI. /// Copyright (c) 2014-2015 Eliot Lash /// </summary> using UnityEngine; using UnityEngine.UI; using System.Text; using System.Collections; public class ConsoleView : MonoBehaviour { ConsoleController console = new ConsoleController(); bool didShow = false; public GameObject viewContainer; //Container for console view, should be a child of this GameObject public Text logTextArea; public InputField inputField; void Start() { if (console != null) { console.visibilityChanged += onVisibilityChanged; console.logChanged += onLogChanged; } updateLogStr(console.log); } ~ConsoleView() { console.visibilityChanged -= onVisibilityChanged; console.logChanged -= onLogChanged; } void Update() { //Toggle visibility when tilde key pressed if (Input.GetKeyUp("`")) { toggleVisibility(); } //Toggle visibility when 5 fingers touch. if (Input.touches.Length == 5) { if (!didShow) { toggleVisibility(); didShow = true; } } else { didShow = false; } } void toggleVisibility() { setVisibility(!viewContainer.activeSelf); } void setVisibility(bool visible) { viewContainer.SetActive(visible); } void onVisibilityChanged(bool visible) { setVisibility(visible); } void onLogChanged(string[] newLog) { updateLogStr(newLog); } void updateLogStr(string[] newLog) { if (newLog == null) { logTextArea.text = ""; } else { logTextArea.text = string.Join(" n", newLog); } } /// <summary> /// Event that should be called by anything wanting to submit the current input to the console. /// </summary> public void runCommand() { console.runCommandString(inputField.text); inputField.text = ""; } } The ConsoleView script manages the GUI and forwards events to the ConsoleController. It also watches the ConsoleController and updates the GUI when necessary. Back in the inspector, select ConsoleView. We’re going to hook up the Console View component properties. Drag ConsoleViewContainer into the “View Container” property. Do the same for LogText into “Log Text Area” and InputField into “Input Field.” i Now we’ve just got a bit more hooking up to do. Select InputField, and in the Input Field component, find the “End Edit” event list. Click the plus button and drag ConsoleView in to the new row. In the function list, select ConsoleView > runCommand (). Finally, select EnterBtn and find the “On Click” event list in the Button component. Click the plus button and drag ConsoleView in to the new row. In the function list, select ConsoleView > onCommand (). Now we’re ready to test! Save your scene and run the game. The console should be visible. Type “help” into the input field: Now press the enter/return key. You should see the help text print out like so: Try out another test command, “echo foo bar baz”. It will show you how it splits the command arguments into a string array, printed out as a comma separated list: Also make sure the fallback “Ent” button is working to submit the input. Lastly, check if the console toggle key works: press the backtick/tilde key (located right above the left tab key.) It looks like this: The console should disappear. Press it again and it should reappear. On a mobile device, tapping five fingers at once will toggle the console instead. If you want to use a different means of toggling the console, you can edit this in ConsoleView.Update(). If anything is not working as expected, please go back over the instructions again and try to see if you’ve missed anything. Lastly, we don’t want the console to show when the game first starts. Stop the game and find ConsoleViewContainer in the hierarchy, then disable it by unchecking the box next to its name in the inspector. Now, save and run the game again. The console should be hidden until you press the backtick key. And that’s it! You now have an in-game, interactive console. It’s an extremely versatile debugging tool that’s easy to extend. Use it to implement cheat codes, enable or disable experimental features, obtain diagnostic output, or whatever else you can think of! When you want to create a new console command, just write a new method in ConsoleController “Command handlers” region and add a registerCommand() line for it in the constructor. Use the commands I’ve included as examples. If you want to have other scripts be able to log to the console, you can make the ConsoleController into a service as I described in my article “One-liner Singletons in Unity”. Make the ConsoleController set itself as a service in its constructor, Then have the other script get the ConsoleController instance and call appendLogLine() with its message. I hope having an in-game console will be as useful for you as it has been for me. Lastly, don’t forget to disable or delete the ConsoleView before shipping production builds unless you want your players to have access to all of your debug cheats! About the author Eliot Lash is an independent game developer and consultant with several mobile titles in the works. In the past, he has worked on Tiny Death Star and the Tap Tap Revenge series. You can find him at eliotlash.com.
Read more
  • 0
  • 3
  • 13174

article-image-linear-regression
Packt
12 Jun 2015
19 min read
Save for later

Linear Regression

Packt
12 Jun 2015
19 min read
In this article by Rui Miguel Forte, the author of the book, Mastering Predictive Analytics with R, we'll learn about linear regression. Regression problems involve predicting a numerical output. The simplest but most common type of regression is linear regression. In this article, we'll explore why linear regression is so commonly used, its limitations, and extensions. (For more resources related to this topic, see here.) Introduction to linear regression In linear regression, the output variable is predicted by a linearly weighted combination of input features. Here is an example of a simple linear model:   The preceding model essentially says that we are estimating one output, denoted by ˆy, and this is a linear function of a single predictor variable (that is, a feature) denoted by the letter x. The terms involving the Greek letter β are the parameters of the model and are known as regression coefficients. Once we train the model and settle on values for these parameters, we can make a prediction on the output variable for any value of x by a simple substitution in our equation. Another example of a linear model, this time with three features and with values assigned to the regression coefficients, is given by the following equation:   In this equation, just as with the previous one, we can observe that we have one more coefficient than the number of features. This additional coefficient, β0, is known as the intercept and is the expected value of the model when the value of all input features is zero. The other β coefficients can be interpreted as the expected change in the value of the output per unit increase of a feature. For example, in the preceding equation, if the value of the feature x1 rises by one unit, the expected value of the output will rise by 1.91 units. Similarly, a unit increase in the feature x3 results in a decrease of the output by 7.56 units. In a simple one-dimensional regression problem, we can plot the output on the y axis of a graph and the input feature on the x axis. In this case, the model predicts a straight-line relationship between these two, where β0 represents the point at which the straight line crosses or intercepts the y axis and β1 represents the slope of the line. We often refer to the case of a single feature (hence, two regression coefficients) as simple linear regression and the case of two or more features as multiple linear regression. Assumptions of linear regression Before we delve into the details of how to train a linear regression model and how it performs, we'll look at the model assumptions. The model assumptions essentially describe what the model believes about the output variable y that we are trying to predict. Specifically, linear regression models assume that the output variable is a weighted linear function of a set of feature variables. Additionally, the model assumes that for fixed values of the feature variables, the output is normally distributed with a constant variance. This is the same as saying that the model assumes that the true output variable y can be represented by an equation such as the following one, shown for two input features: Here, ε represents an error term, which is normally distributed with zero mean and constant variance σ2: We might hear the term homoscedasticity as a more formal way of describing the notion of constant variance. By homoscedasticity or constant variance, we are referring to the fact that the variance in the error component does not vary with the values or levels of the input features. In the following plot, we are visualizing a hypothetical example of a linear relationship with heteroskedastic errors, which are errors that do not have a constant variance. The data points lie close to the line at low values of the input feature, because the variance is low in this region of the plot, but lie farther away from the line at higher values of the input feature because of the higher variance. The ε term is an irreducible error component of the true function y and can be used to represent random errors, such as measurement errors in the feature values. When training a linear regression model, we always expect to observe some amount of error in our estimate of the output, even if we have all the right features, enough data, and the system being modeled really is linear. Put differently, even with a true function that is linear, we still expect that once we find a line of best fit through our training examples, our line will not go through all, or even any of our data points because of this inherent variance exhibited by the error component. The critical thing to remember, though, is that in this ideal scenario, because our error component has zero mean and constant variance, our training criterion will allow us to come close to the true values of the regression coefficients given a sufficiently large sample, as the errors will cancel out. Another important assumption relates to the independence of the error terms. This means that we do not expect the residual or error term associated with one particular observation to be somehow correlated with that of another observation. This assumption can be violated if observations are functions of each other, which is typically the result of an error in the measurement. If we were to take a portion of our training data, double all the values of the features and outputs, and add these new data points to our training data, we could create the illusion of having a larger data set; however, there will be pairs of observations whose error terms will depend on each other as a result, and hence our model assumption would be violated. Incidentally, artificially growing our data set in such a manner is never acceptable for any model. Similarly, correlated error terms may occur if observations are related in some way by an unmeasured variable. For example, if we are measuring the malfunction rate of parts from an assembly line, then parts from the same factory might have a correlation in the error, for example, due to different standards and protocols used in the assembly process. Therefore, if we don't use the factory as a feature, we may see correlated errors in our sample among observations that correspond to parts from the same factory. The study of experimental design is concerned with identifying and reducing correlations in error terms, but this is beyond the scope of this book. Finally, another important assumption concerns the notion that the features themselves are statistically independent of each other. It is worth clarifying here that in linear models, although the input features must be linearly weighted, they themselves may be the output of another function. To illustrate this, one may be surprised to see that the following is a linear model of three features, sin(z1), ln(z2), and exp(z3): We can see that this is a linear model by making a few transformations on the input features and then making the replacements in our model: Now, we have an equation that is more recognizable as a linear regression model. If the previous example made us believe that nearly everything could be transformed into a linear model, then the following two examples will emphatically convince us that this is not in fact the case: Both models are not linear models because of the first regression coefficient (β1). The first model is not a linear model because β1 is acting as the exponent of the first input feature. In the second model, β1 is inside a sine function. The important lesson to take away from these examples is that there are cases where we can apply transformations on our input features in order to fit our data to a linear model; however, we need to be careful that our regression coefficients are always the linear weights of the resulting new features. Simple linear regression Before looking at some real-world data sets, it is very helpful to try to train a model on artificially generated data. In an artificial scenario such as this, we know what the true output function is beforehand, something that as a rule is not the case when it comes to real-world data. The advantage of performing this exercise is that it gives us a good idea of how our model works under the ideal scenario when all of our assumptions are fully satisfied, and it helps visualize what happens when we have a good linear fit. We'll begin by simulating a simple linear regression model. The following R snippet is used to create a data frame with 100 simulated observations of the following linear model with a single input feature: Here is the code for the simple linear regression model: > set.seed(5427395)> nObs = 100> x1minrange = 5> x1maxrange = 25> x1 = runif(nObs, x1minrange, x1maxrange)> e = rnorm(nObs, mean = 0, sd = 2.0)> y = 1.67 * x1 - 2.93 + e> df = data.frame(y, x1) For our input feature, we randomly sample points from a uniform distribution. We used a uniform distribution to get a good spread of data points. Note that our final df data frame is meant to simulate a data frame that we would obtain in practice, and as a result, we do not include the error terms, as these would be unavailable to us in a real-world setting. When we train a linear model using some data such as those in our data frame, we are essentially hoping to produce a linear model with the same coefficients as the ones from the underlying model of the data. Put differently, the original coefficients define a population regression line. In this case, the population regression line represents the true underlying model of the data. In general, we will find ourselves attempting to model a function that is not necessarily linear. In this case, we can still define the population regression line as the best possible linear regression line, but a linear regression model will obviously not perform equally well. Estimating the regression coefficients For our simple linear regression model, the process of training the model amounts to an estimation of our two regression coefficients from our data set. As we can see from our previously constructed data frame, our data is effectively a series of observations, each of which is a pair of values (xi, yi) where the first element of the pair is the input feature value and the second element of the pair is its output label. It turns out that for the case of simple linear regression, it is possible to write down two equations that can be used to compute our two regression coefficients. Instead of merely presenting these equations, we'll first take a brief moment to review some very basic statistical quantities that the reader has most likely encountered previously, as they will be featured very shortly. The mean of a set of values is just the average of these values and is often described as a measure of location, giving a sense of where the values are centered on the scale in which they are measured. In statistical literature, the average value of a random variable is often known as the expectation, so we often find that the mean of a random variable X is denoted as E(X). Another notation that is commonly used is bar notation, where we can represent the notion of taking the average of a variable by placing a bar over that variable. To illustrate this, the following two equations show the mean of the output variable y and input feature x: A second very common quantity, which should also be familiar, is the variance of a variable. The variance measures the average square distance that individual values have from the mean. In this way, it is a measure of dispersion, so that a low variance implies that most of the values are bunched up close to the mean, whereas a higher variance results in values that are spread out. Note that the definition of variance involves the definition of the mean, and for this reason, we'll see the use of the x variable with a bar on it in the following equation that shows the variance of our input feature x: Finally, we'll define the covariance between two random variables x and y using the following equation: From the previous equation, it should be clear that the variance, which we just defined previously, is actually a special case of the covariance where the two variables are the same. The covariance measures how strongly two variables are correlated with each other and can be positive or negative. A positive covariance implies a positive correlation; that is, when one variable increases, the other will increase as well. A negative covariance suggests the opposite; when one variable increases, the other will tend to decrease. When two variables are statistically independent of each other and hence uncorrelated, their covariance will be zero (although it should be noted that a zero covariance does not necessarily imply statistical independence). Armed with these basic concepts, we can now present equations for the estimates of the two regression coefficients for the case of simple linear regression: The first regression coefficient can be computed as the ratio of the covariance between the output and the input feature, and the variance of the input feature. Note that if the output feature were to be independent of the input feature, the covariance would be zero and therefore, our linear model would consist of a horizontal line with no slope. In practice, it should be noted that even when two variables are statistically independent, we will still typically see a small degree of covariance due to the random nature of the errors, so if we were to train a linear regression model to describe their relationship, our first regression coefficient would be nonzero in general. Later, we'll see how significance tests can be used to detect features we should not include in our models. To implement linear regression in R, it is not necessary to perform these calculations as R provides us with the lm() function, which builds a linear regression model for us. The following code sample uses the df data frame we created previously and calculates the regression coefficients: > myfit <- lm(y~x1, df)> myfitCall:lm(formula = y ~ x1, data = df) Coefficients:(Intercept)          x1     -2.380       1.641 In the first line, we see that the usage of the lm() function involves first specifying a formula and then following up with the data parameter, which in our case is our data frame. For the case of simple linear regression, the syntax of the formula that we specify for the lm() function is the name of the output variable, followed by a tilde (~) and then by the name of the single input feature. Finally, the output shows us the values for the two regression coefficients. Note that the β0 coefficient is labeled as the intercept, and the β1 coefficient is labeled by the name of the corresponding feature (in this case, x1) in the equation of the linear model. The following graph shows the population line and the estimated line on the same plot: As we can see, the two lines are so close to each other that they are barely distinguishable, showing that the model has estimated the true population line very closely. Multiple linear regression Whenever we have more than one input feature and want to build a linear regression model, we are in the realm of multiple linear regression. The general equation for a multiple linear regression model with k input features is: Our assumptions about the model and about the error component ε remain the same as with simple linear regression, remembering that as we now have more than one input feature, we assume that these are independent of each other. Instead of using simulated data to demonstrate multiple linear regression, we will analyze two real-world data sets. Predicting CPU performance Our first real-world data set was presented by the researchers Dennis F. Kibler, David W. Aha, and Marc K. Albert in a 1989 paper titled Instance-based prediction of real-valued attributes and published in Journal of Computational Intelligence. The data contain the characteristics of different CPU models, such as the cycle time and the amount of cache memory. When deciding between processors, we would like to take all of these things into account, but ideally, we'd like to compare processors on a single numerical scale. For this reason, we often develop programs to benchmark the relative performance of a CPU. Our data set also comes with the published relative performance of our CPUs and our objective will be to use the available CPU characteristics as features to predict this. The data set can be obtained online from the UCI Machine Learning Repository via this link: http://archive.ics.uci.edu/ml/datasets/Computer+Hardware. The UCI Machine Learning Repository is a wonderful online resource that hosts a large number of data sets, many of which are often cited by authors of books and tutorials. It is well worth the effort to familiarize yourself with this website and its data sets. A very good way to learn predictive analytics is to practice using the techniques you learn in this book on different data sets, and the UCI repository provides many of these for exactly this purpose. The machine.data file contains all our data in a comma-separated format, with one line per CPU model. We'll import this in R and label all the columns. Note that there are 10 columns in total, but we don't need the first two for our analysis, as these are just the brand and model name of the CPU. Similarly, the final column is a predicted estimate of the relative performance that was produced by the researchers themselves; our actual output variable, PRP, is in column 9. We'll store the data that we need in a data frame called machine: > machine <- read.csv("machine.data", header = F)> names(machine) <- c("VENDOR", "MODEL", "MYCT", "MMIN", "MMAX", "CACH", "CHMIN", "CHMAX", "PRP", "ERP")> machine <- machine[, 3:9]> head(machine, n = 3)MYCT MMIN MMAX CACH CHMIN CHMAX PRP1 125 256 6000 256   16   128 1982   29 8000 32000   32     8   32 2693   29 8000 32000   32     8   32 220 The data set also comes with the definition of the data columns: Column name Definition MYCT The machine cycle time in nanoseconds MMIN The minimum main memory in kilobytes MMAX The maximum main memory in kilobytes CACH The cache memory in kilobytes CHMIN The minimum channels in units CHMAX The maximum channels in units PRP The published relative performance (our output variable) The data set contains no missing values, so no observations need to be removed or modified. One thing that we'll notice is that we only have roughly 200 data points, which is generally considered a very small sample. Nonetheless, we will proceed with splitting our data into a training set and a test set, with an 85-15 split, as follows: > library(caret)> set.seed(4352345)> machine_sampling_vector <- createDataPartition(machine$PRP,    p = 0.85, list = FALSE)> machine_train <- machine[machine_sampling_vector,]> machine_train_features <- machine[, 1:6]> machine_train_labels <- machine$PRP[machine_sampling_vector]> machine_test <- machine[-machine_sampling_vector,]> machine_test_labels <- machine$PRP[-machine_sampling_vector] Now that we have our data set up and running, we'd usually want to investigate further and check whether some of our assumptions for linear regression are valid. For example, we would like to know whether we have any highly correlated features. To do this, we can construct a correlation matrix with the cor()function and use the findCorrelation() function from the caret package to get suggestions for which features to remove: > machine_correlations <- cor(machine_train_features)> findCorrelation(machine_correlations)integer(0)> findCorrelation(machine_correlations, cutoff = 0.75)[1] 3> cor(machine_train$MMIN, machine_train$MMAX)[1] 0.7679307 Using the default cutoff of 0.9 for a high degree of correlation, we found that none of our features should be removed. When we reduce this cutoff to 0.75, we see that caret recommends that we remove the third feature (MMAX). As the final line of preceding code shows, the degree of correlation between this feature and MMIN is 0.768. While the value is not very high, it is still high enough to cause us a certain degree of concern that this will affect our model. Intuitively, of course, if we look at the definitions of our input features, we will certainly tend to expect that a model with a relatively high value for the minimum main memory will also be likely to have a relatively high value for the maximum main memory. Linear regression can sometimes still give us a good model with correlated variables, but we would expect to get better results if our variables were uncorrelated. For now, we've decided to keep all our features for this data set. Summary In this article, we studied linear regression, a method that allows us to fit a linear model in a supervised learning setting where we have a number of input features and a single numeric output. Simple linear regression is the name given to the scenario where we have only one input feature, and multiple linear regression describes the case where we have multiple input features. Linear regression is very commonly used as a first approach to solving a regression problem. It assumes that the output is a linear weighted combination of the input features in the presence of an irreducible error component that is normally distributed and has zero mean and constant variance. The model also assumes that the features are independent.
Read more
  • 0
  • 0
  • 13160
article-image-debugging-yournet
Packt
21 Feb 2018
11 min read
Save for later

Debugging Your.Net

Packt
21 Feb 2018
11 min read
In this article,Ya nnick Lefebvre, author of Wordpress Plugin Development Cookbook - Second Edition, we will cover the following recipes: Creating a new shortcode with parameters Managing multiple sets of user settings from a single admin page WordPress shortcodes are a simple, yet powerful tool that can be used to automate the insertion of code into web pages. For example, a shortcode could be used to automate the insertion of videos from a third-party platform that is not supported natively by WordPress, or embed content from a popular web site. By following the two code samples found in this article, you will learn how to create a WordPress plugin that defines your own shortcode to be able to quickly embed Twitter feeds on a web site. You will also learn how to create an administration configuration panel to be able to create a set of configurations that can be referenced when using your newly-created shortcode. Creating a new shortcode with parameters While simple shortcodes already provide a lot of potential to output complex content to a page by entering a few characters in the post editor, shortcodes become even more useful when they are coupled with parameters that will be passed to their associated processing function. Using this technique, it becomes very easy to create a shortcode that accelerates the insertion of external content in WordPress posts or pages by only needing to specify the shortcode and the unique identifier of the source element to be displayed. We will illustrate this concept in this recipe by creating a shortcode that will be used to quickly add Twitter feeds to posts or pages. How to do it... Navigate to the WordPress plugin directory of your development installation. Create a new directory called ch2-twitter-embed. Navigate to this directory and create a new text file called ch2-twitter-embed.php. Open the new file in a code editor and add an appropriate header at the top of the plugin file, naming the plugin Chapter 2 - Twitter Embed. Add the following line of code to declare a new shortcode and specify the name of the function that should be called when the shortcode is found in posts or pages: add_shortcode( 'twitterfeed', 'ch2te_twitter_embed_shortcode' ); Add the following code section to provide an implementation for the                     ch2te_twitter_embed_shortcode function:function ch2te_twitter_embed_shortcode( $atts ) { extract( shortcode_atts( array( 'user_name' => 'ylefebvre' ), $atts ) ); if ( !empty( $user_name ) ) { $output = '<a class="twitter-timeline" href="'; $output .= esc_url( 'https://twitter.com/' . $user_name ); $output .= '">Tweets by ' . esc_html( $user_name ); $output .= '</a><script async '; $output .= 'src="//platform.twitter.com/widgets.js"'; $output .= ' charset="utf-8"></script>'; } else { $output = ''; } return $output; } Save and close the plugin file. Log in to the administration page of your development WordPress installation. Click on Plugins in the left-hand navigation menu. Activate your new plugin. Create a new page and use the shortcode [twitterfeed user_name='WordPress'] in the page editor, where WordPress is the Twitter username of the feed to display: Save and view the page to see that the shortcode was replaced by an embedded Twitter feed on your site. Edit the page and remove the user_name parameter and its associated value, only leaving the core [twitterfeed] shortcode in the post and Save. Refresh the page and see that the feed is still being displayed but now shows tweets from another account. How it works... When shortcodes are used with parameters, these extra pieces of data are sent to the associated processing function in the $atts parameter variable. By using a combination of the standard PHP extract and WordPress-specific shortcode_atts functions, our plugin is able to parse the data sent to the shortcode and create an array of identifiers and values that are subsequently transformed into PHP variables that we can use in the rest of our shortcode implementation function. In this specific example, we expect a single variable to be used, called user_name, which will be stored in a PHP variable called $user_name. If the user enters the shortcode without any parameter, a default value of ylefebvre will be assigned to the username variable to ensure that the plugin still works. Since we are going to accept user input in this code, we also verify that the user did not provide an empty string and we use the esc_html and esc_url functions to remove any potentially harmful HTML characters from the input string and make sure that the link destination URL is valid. Once we have access to the twitter username, we can put together the required HTML code that will embed a Twitter feed in our page and display the selected user's tweets. While this example only has one argument, it is possible to define multiple parameters for a shortcode. Managing multiple sets of user settings from a single admin page Throughout this article, you have learned how to create configuration pages to manage single sets of configuration options for our plugins. In some cases, only being able to specify a single set of options will not be enough. For example, looking back at the Twitter embed shortcode plugin that was created, a single configuration panel would only allow users to specify one set of options, such as the desired twitter feed dimensions or the number of tweets to display. A more flexible solution would be to allow users to specify multiple sets of configuration options, which could then be called up by using an extra shortcode parameter (for example, [twitterfeed user_name="WordPress" option_id="2"]). While the first thought that might cross your mind to configure such a plugin is to create a multi-level menu item with submenus to store a number of different settings, this method would produce a very awkward interface for users to navigate. A better way is to use a single panel but give the user a way to select between multiple sets of options to be modified. In this recipe, you will learn how to enhance the previously created Twitter feed shortcode plugin to be able to control the embedded feed size and number of tweets to display from the plugin configuration panel and to give users the ability to specify multiple display sizes. Getting ready You should have already followed the Creating a new shortcode with parameters recipe in the article to have a starting point for this recipe. Alternatively, you can get the resulting code (Chapter 2/ch2-twitter-embed/ch2-twitter-embed.php) from the downloaded code bundle. How to do it... Navigate to the ch2-twitter-embed folder of the WordPress plugin directory of your development installation. Open the ch2-twitter-embed.php file in a text editor. Add the following lines of code to implement an activation callback to initialize plugin options when it is installed or upgraded: ch2te_twitter_embed_shortcode function:function ch2te_twitter_embed_shortcode( $atts ) { extract( shortcode_atts( array( 'user_name' => 'ylefebvre' ), $atts ) ); if ( !empty( $user_name ) ) { $output = '<a class="twitter-timeline" href="'; $output .= esc_url( 'https://twitter.com/' . $user_name ); $output .= '">Tweets by ' . esc_html( $user_name ); $output .= '</a><script async '; $output .= 'src="//platform.twitter.com/widgets.js"'; $output .= ' charset="utf-8"></script>'; } else { $output = ''; } return $output; } Insert the following code segment to register a function to be called when the administration menu is put together. When this happens, the callback function adds an item to the Settings menu and specifies the function to be called to render the configuration page: ch2te_twitter_embed_shortcode function:function ch2te_twitter_embed_shortcode( $atts ) { extract( shortcode_atts( array( 'user_name' => 'ylefebvre' ), $atts ) ); if ( !empty( $user_name ) ) { $output = '<a class="twitter-timeline" href="'; $output .= esc_url( 'https://twitter.com/' . $user_name ); $output .= '">Tweets by ' . esc_html( $user_name ); $output .= '</a><script async '; $output .= 'src="//platform.twitter.com/widgets.js"'; $output .= ' charset="utf-8"></script>'; } else { $output = ''; } return $output; } Add the following code to implement the configuration page rendering function: ch2te_twitter_embed_shortcode function:function ch2te_twitter_embed_shortcode( $atts ) { extract( shortcode_atts( array( 'user_name' => 'ylefebvre' ), $atts ) ); if ( !empty( $user_name ) ) { $output = '<a class="twitter-timeline" href="'; $output .= esc_url( 'https://twitter.com/' . $user_name ); $output .= '">Tweets by ' . esc_html( $user_name ); $output .= '</a><script async '; $output .= 'src="//platform.twitter.com/widgets.js"'; $output .= ' charset="utf-8"></script>'; } else { $output = ''; } return $output; } Add the following block of code to register a function that will process user options when submitted to the site: add_action( 'admin_init', 'ch2te_admin_init' ); function ch2te_admin_init() { add_action( 'admin_post_save_ch2te_options', 'process_ch2te_options' ); } Add the following code to implement the process_ch2te_options function, declared in the previous block of code, and to declare a utility function used to clean the redirection path: function ch2te_twitter_embed_shortcode( $atts ) { extract( shortcode_atts( array( 'user_name' => 'ylefebvre', 'option_id' => '1' ), $atts ) ); if ( intval( $option_id ) < 1 || intval( $option_id ) > 5 ) { $option_id = 1; } $options = ch2te_get_options( $option_id ); if ( !empty( $user_name ) ) { $output = '<a class="twitter-timeline" href="'; $output .= esc_url( 'https://twitter.com/' . $user_name ); $output .= '" data-width="' . $options['width'] . '" '; $output .= 'data-tweet-limit="' . $options['number_of_tweets']; $output .= '">' . 'Tweets by ' . esc_html( $user_name ); $output .= '</a><script async '; $output .= 'src="//platform.twitter.com/widgets.js"'; $output .= ' charset="utf-8"></script>'; } else { $output = ''; } return $output; } Find the ch2te_twitter_embed_shortcode function and modify it as follows to accept the new option_id parameter and load the plugin options to produce the desired output. The changes are identified in bold within the recipe: function ch2te_twitter_embed_shortcode( $atts ) { extract( shortcode_atts( array( 'user_name' => 'ylefebvre', 'option_id' => '1' ), $atts ) ); if ( intval( $option_id ) < 1 || intval( $option_id ) > 5 ) { $option_id = 1; } $options = ch2te_get_options( $option_id ); if ( !empty( $user_name ) ) { $output = '<a class="twitter-timeline" href="'; $output .= esc_url( 'https://twitter.com/' . $user_name ); $output .= '" data-width="' . $options['width'] . '" '; $output .= 'data-tweet-limit="' . $options['number_of_tweets']; $output .= '">' . 'Tweets by ' . esc_html( $user_name ); $output .= '</a><script async '; $output .= 'src="//platform.twitter.com/widgets.js"'; $output .= ' charset="utf-8"></script>'; } else { $output = ''; } return $output; } Save and close the plugin file. Deactivate and then Activate the Chapter 2 - Twitter Embed plugin from the administration interface to execute its activation function and create default settings. Navigate to the Settings menu and select the Twitter Embed submenu item to see the newly created configuration panel with the first set of options being displayed and more sets of options accessible through the drop-down list shown at the top of the page. To select the set of options to be used, add the parameter option_id to the shortcode used to display a Twitter feed, as follows: [twitterfeed user_name="WordPress" option_id="1"] How it works... This recipe shows how we can leverage options arrays to create multiple sets of options simply by creating the name of the options array on the fly. Instead of having a specific option name in the first parameter of the get_option function call, we create a string with an option ID. This ID is sent through as a URL parameter on the configuration page and as a hidden text field when processing the form data. On initialization, the plugin only creates a single set of options, which is probably enough for most casual users of the plugin. Doing so will avoid cluttering the site database with useless options. When the user requests to view one of the empty option sets, the plugin creates a new set of options right before rendering the options page. The rest of the code is very similar to the other examples that we saw in this article, since the way to access the array elements remains the same. Summary In this article, the author has explained about the entire process of how to create a new shortcode with parameters and how to manage multiple sets of user settings from a single admin page.   Resources for Article:   Further resources on this subject:Introduction to WordPress Plugin [article] Wordpress: Buddypress Courseware [article] Anatomy of a WordPress Plugin [article]
Read more
  • 0
  • 0
  • 13152

article-image-android-game-development-unity3d
Packt
23 Nov 2016
8 min read
Save for later

Android Game Development with Unity3D

Packt
23 Nov 2016
8 min read
In this article by Wajahat Karim, author of the book Mastering Android Game Development with Unity, we will be creating addictive fun games by using a very famous game engine called Unity3D. In this article, we will cover the following topics: Game engines and Unity3D Features of Unity3D Basics of Unity game development (For more resources related to this topic, see here.) Game engines and Unity3D A game engine is a software framework designed for the creation and development of video games. Many tools and frameworks are available for game designers and developers to code a game quickly and easily without building from the ground up. As time passed by, game engines became more mature and easy for developers, with feature-rich environments. Starting from native code frameworks for Android such as AndEngine, Cocos2d-x, LibGDX, and so on, game engines started providing clean user interfaces and drag-drop functionalities to make game development easier for developers. These engines include lots of tools which are different in user interface, features, porting, and many more things; but all have one thing in common— they create video games in the end. Unity (http://unity3d.com) is a cross-platform game engine developed by Unity Technologies. It made its first public announcement at Apple Worldwide Developers Conference in 2005, and supported only game development for Mac OS, but since then it has been extended to target more than 15 platforms for desktop, mobile, and consoles. It is notable for its one-click ability to port games on multiple platforms including BlackBerry 10, Windows Phone 8, Windows, OS X, Linux, Android, iOS, Unity Web Player (including Facebook), Adobe Flash, PlayStation 3, PlayStation 4, PlayStation Vita, Xbox 360, Xbox One, Wii U, and Wii. Unity has a fantastic interface, which lets the developers manage the project really efficiently from the word go. It has a nice drag-drop functionality with connecting behavior scripts written in either C#, JavaScript (or UnityScript), or Boo to define the custom logic and functionality with visual objects quite easily. Unity has been proven quite easy to learn for new developers who are just starting out with game development. Now more largely studios have also started using , and that too for good reasons. Unity is one of those engines that provide support for both 2D and 3D games without putting developers in trouble or confusing them. Due to its popularity all over the game development industry, it has a vast collection of online tutorials, great documentation, and a very helping community of developers. Features of Unity3D Unity is a game development ecosystem comprising a powerful rendering engine, intuitive tools, rapid workflows for 2D and 3D games, all-in-one deployment support, and thousands of already created free and paid assets with a helping developer's community. The feature list includes the following: Easy workflow allowing developers to rapidly assemble scenes in an intuitive editor workspace Quality game creation like AAA visuals, high-definition audio, full-throttle action without any glitches on screen Dedicated tools for both 2D and 3D game creation with shared conventions to make it easy for developers A very unique and flexible animation system to create natural animations with very less time-consuming efforts Smooth frame rate with reliable performance on all the platforms where developers publish their games One-click ability to deploy to all platforms from desktops, browsers, and mobiles to consoles within minutes Reduces time of development by using already created reusable assets available at the huge asset store Basics of Unity game development Before delving into details of Unity3D and game development concepts, let's have a look at some of the very basics of Unity 5.0. We will go through the Unity interface, menu items, using assets, creating scenes, and publishing builds. Unity editor interface When you launch Unity 5.0 for the first time, you will be presented with an editor with a few panels on the left, right, and bottom of the screen. The following screenshot shows the editor interface when it's first launched: Fig 1.7 Unity 5 editor interface at first launch First of all, take your time to look over the editor, and become a little familiar with it. The Unity editor is divided into different small panels and views, which can be dragged to customize the workspace according to the developer/designer's needs. Unity 5 comes with some prebuilt workspace layout templates, which can be selected from the Layout drop-down menu at top-right corner of the screen, as shown in the following screenshot: Fig 1.8 Unity 5 editor layouts The layout currently displayed in the editor shown in the preceding screenshot is the Default layout. You can select these layouts, and see how the editor's interface changes, and how the different panels are placed at different positions in each layout. This book uses the 2 by 3 workspace layout for the whole game. The following figure shows the 2 by 3 workspace with the names of the views and panels highlighted: Fig 1.9 Unity 5 2 by 3 Layout with views and panel names As you can see in the preceding figure, the Unity editor contains different views and panels. Every panel and view have a specific purpose, which is described as follows: Scene view The Scene view is the whole stage for the game development, and it contains every asset in the game from a tiny point to any heavy 3D model. The Scene view is used to select and position environments, characters, enemies, the player, camera, and all other objects which can be placed on the stage for the game. All those objects which can be placed and shown in the game are called game objects. The Scene view allows developers to manipulate game objects such as selecting, scaling, rotating, deleting, moving, and so on. It also provides some controls such as navigation and transformation.  In simple words, the Scene view is the interactive sandbox for developers and designers. Game view The Game view is the final representation of how your game will look when published and deployed on the target devices, and it is rendered from the cameras of the scene. This view is connected to the play mode navigation bar in the center at the top of the whole Unity workspace. The play mode navigation bar is shown in the following: figure. Fig 1.14 Play mode bar When the game is played in the editor, this control bar gets changed to blue color. A very interesting feature of Unity is that it allows developers to pause the game and code while running, and the developers can see and change the properties, transforms, and much more at runtime, without recompiling the whole game, for quick workflow. Hierarchy view The Hierarchy view is the first point to select or handle any game object available in the scene. This contains every game object in the current scene. It is tree-type structure, which allows developers to utilize the parent and child concept on the game objects easily. The following figure shows a simple Hierarchy view: Fig 1.16 Hierarchy view Project browser panel This panel looks like a view, but it is called the Project browser panel. It is an embedded files directory in Unity, and contains all the files and folders included in the game project. The following figure shows a simple Project browser panel: Fig 1.17 Project browser panel The left side of the panel shows a hierarchal directory, while the rest of the panel shows the files, or, as they are called, assets in Unity. Unity represents these files with different icons to differentiate these according to their file types. These files can be sprite images, textures, model files, sounds, and so on. You can search any specific file by typing in the search text box. On the right side of search box, there are button controls for further filters such as animation files, audio clip files, and so on. An interesting thing about the Project browser panel is that if any file is not available in the Assets, then Unity starts looking for it on the Unity Asset Store, and presents you with the available free and paid assets. Inspector panel This is the most important panel for development in Unity. Unity structures the whole game in the form of game objects and assets. These game objects further contain components such as transform, colliders, scripts, meshes, and so on. Unity lets developers manage these components of each game object through the Inspector panel. The following figure shows a simple Inspector panel of a game object: Fig 1.18 Inspector panel These components vary in types, for example, Physics, Mesh, Effects, Audio, UI, and so on. These components can be added in any object by selecting it from the Component menu. The following figure shows the Component menu: Fig 1.19 Components menu Summary In this article, you learned about game engines, such as Unity3D, which is used to create games for Android devices. We also discussed the important features of Unity along with the basics of its development environment. Resources for Article: Further resources on this subject: The Game World [article] Customizing the Player Character [article] Animation features in Unity 5 [article]
Read more
  • 0
  • 0
  • 13145

article-image-overview-data-protection-manager-2010
Packt
30 May 2011
9 min read
Save for later

Overview of Data Protection Manager 2010

Packt
30 May 2011
9 min read
  Microsoft Data Protection Manager 2010 A practical step-by-step guide to planning deployment, installation, configuration, and troubleshooting of Data Protection Manager 2010         Read more about this book       (For more resources on Microsoft, see here.) DPM structure In this section we will look at the DPM file structure in order to have a better understanding of where DPM stores its components. We will also look at important processes that DPM runs and what they are used for. There will be some hints and tips that you should know about that will be useful when administering DPM. DPM file locations It is important to know not only how DPM operates, but also to know the structure that is underneath the application. Understanding the structure of where the DPM components are will help you with administering and troubleshooting DPM if the need arises. The following are some important locations: The DPM database backups are stored in the following location. Also when you make backup shadow copies for the replicas these will be stored in this directory. You would make backup show copies of your replicas if you were archiving them using a third-party backup solution: C:Program FilesMicrosoft DPMDPMVolumesShadowCopyDatabase Backups The following directory is where DPM is installed: C:Program FilesMicrosoft DPM The following directory contains PowerShell scripts that come with DPM. There are many scripts that can be used for performing common DPM tasks. C:Program FilesMicrosoft DPMDPMbin The following folder contains the database and files for SQL reporting services: C:Program FilesMicrosoft DPMSQL The following directory contains the SQL DPM database. MDF and LDF files: C:Program FilesMicrosoft DPMDPMDPMDB The following directory stores shadow copy volumes that are recovery points for a data source. These essentially are the changed blocks of VSS (Volume Shadow Copy Service) (Shadow Copy). C:Program FilesMicrosoft DPMDPMVolumesDiffArea The following folder contains mounted replica volumes. Mounted replica volumes are essentially pointers for every protected data object that points to the partition in a DPM storage pool. Think of these mounted replica points as a map from DPM to the protected data on the hard drives where the actual protected data lives. C:Program FilesMicrosoft DPMDPMVolumesReplica DPM processes We are now going to explore DPM processes. The executable files for these are all located in C:Program FilesMicrosoft DPMDPMbin. You can view these processes in Windows Task Manager and they show up in Windows Services as well: The following screenshot shows the DPM services as they appear in Windows Services: We will look at what each of these processes are and what they do. We will also look at the processes that have an impact on the performance of your DPM server. The processes are as follows: DPMAMService.exe: In Windows Services this is listed as the DPM AccessManager Service. This manages access to DPM. DpmWriter.exe: This is a service as well, so you will see it on the services list. This service is used for archiving. It manages the backup shadow copies or replicas, backups of report databases, as well as DPM backups. Msdpm.exe: The DPM service is the core component of DPM. The DPM service manages all core DPM operations, including replica creation, synchronization, and recovery point creation. This service implements and manages synchronization and shadow copy creation for protected file servers. DPMLA.exe: This is the DPM Library Agent Service. DPMRA.exe: This is the DPM Replication Agent. It helps to back up and recover file and application data to DPM. Dpmac.exe: This is known as the DPM Agent Coordinator Service. This manages the installations, uninstallations, and upgrades of DPM protection agents on remote computers that you need to protect. DPM processes that impact DPM performance The Msdpm.exe, MsDpmProtectionAgent.exe, Microsoft$DPM$Acct.exe, and mmc.exe processes take a toll on DPM performance. mmc.exe is a standard Windows service. "MMC" stands for Microsoft Management Console application and is used to display various management plug-ins. Not all but a good amount of Microsoft server applications run in the MMC such as Exchange, ISA, IIS, System Center, and the Microsoft Server Manager. The DPM Administrator Console runs in an MMC as well. mmc.exe can cause high memory usage. The best way to ensure that this process does not overload your memory is to close the DPM Administrator Console when not using it. MsDpmProtectionAgent.exe is the DPM Protection Agent service and affects both CPU and memory usage when DPM jobs and consistency checks are run. There is nothing you can do to get the usage down for this service. You just need to be aware of this and try not to schedule any other resource intensive applications such as antivirus scans at the same time as DPM jobs or consistency checks. Mspdpm.exe is a service that runs synchronization and shadow copy creations as stated previously. Like MsDpmProtectionAgent.exe, Mspdpm.exe also affects CPU and memory usage when running synchronizations and shadow copies. Like MsDpmProtectionAgent.exe there is nothing you can do to the Mspdpm. exe service to reduce memory and CPU usage. Just make sure to keep the system clear of resource intensive applications when the Mspdpm.exe is running jobs. If you are running a local SQL instance for your DPM deployment you will notice a Microsoft$DPM$Acct.exe process. The SQL Server and SQL Agent services use a Microsoft$DPM$Acct account. This normally runs on a high level. This service reserves part of your system's memory for cache. If the system memory goes low, the Microsoft$DPM$Acct.exe process will let go of the memory cache it has reserved. Important DPM terms In this section you will learn some important terms used commonly in DPM. You will need to understand these terms as you begin to administer DPM on a regular basis. You can read the full list of terms at this site: http://technet.microsoft.com/en-us/library/bb795543.aspx We group the terms in a way that each group relates to an area of DPM. The following are some important terms: Bare metal recovery: This is a restore technique that allows one to restore a complete system onto bare metal, without any requirements, to the previous hardware. This allows restoring to dissimilar hardware. Change journal: A feature that tracks changes to NTFS (New Technology File System) volumes, including additions, deletions, and modifications. The change journal exists on the volume as a sparse file. Sparse files are used to make disk space usage more efficient in NTFS. A sparse file allocates disk space only when it is needed. This allows files to be created even when there is insufficient space on a hard drive. These files contain zeroes instead of disk blocks. Consistency check: The process by which DPM checks for and corrects inconsistencies between a protected data source and its replica. A consistency check is only performed when normal mechanisms for recording changes to protected data, and for applying those changes to replicas, have been interrupted. Express full backup: A synchronization operation in which the protection agent transfers a snapshot of all the blocks that have changed since the previous express full backup (or initial replica creation, for the first express full backup). Shadow copy: A point-in-time copy of files and folders that is stored on the DPM server. Shadow copies are sometimes referred to as snapshots. Shadow copy client software: Client software that enables an end-user to independently recover data by retrieving a shadow copy. Replica: A complete copy of the protected data on a single volume, database, or storage group. Each member of a protection group is associated with a replica on the DPM server. Replica creation: The process by which a full copy of data sources, selected for inclusion in a protection group, is transferred to the DPM storage pool. The replica can be created over the network from data on the protected computer or from a tape backup system. Replica creation is an initialization process that is performed for each data source when the data source is added to a protection group. Replica volume: A volume on the DPM server that contains the replica for a protected data source. Custom volume: A volume that is not in the DPM storage pool and is specified to store the replica and recovery points for a protection group member. Dismount: To remove a removable tape or disc from a drive. DPM Alerts log: A log that stores DPM alerts as Windows events so that the alerts can be displayed in Microsoft System Center Operations Manager (SCOM). DPMDB.mdf: The filename of the DPM database, the SQL Server database that stores DPM settings and configuration information. DPMDBReaders group: A group, created during DPM installation, that contains all accounts that have read-only access to the DPM database. The DPMReport account is a member of this group. DPMReport account: The account that the Web and NT services of SQL Server Reporting Services use to access the DPM database. This account is created when an administrator configures DPM reporting. MICROSOFT$DPM$: The name that the DPM setup assigns to the SQL Server instance used by DPM. Microsoft$DPMWriter$ account: The low-privilege account under which DPM runs the DPM Writer service. This account is created during the DPM installation. MSDPMTrustedMachines group: A group that contains the domain accounts for computers that are authorized to communicate with the DPM server. DPM uses this group to ensure that only computers that have the DPM protection agent installed from a specific DPM server can respond to calls from that server. Protection configuration: The collection of settings that is common to a protection group; specifically, the protection group name, disk allocations, replica creation method, and on-the-wire compression. Protection group: A collection of data sources that share the same protection configuration. Protection group member: A data source within a protection group. Protected computer: A computer that contains data sources that are protection group members. Synchronization: The process by which DPM transfers changes from the protected computer to the DPM server, and applies the changes to the replica of the protected volume. Recovery goals: The retention range, data loss tolerance, and frequency of recovery points for protected data. Recovery collection: The aggregate of all recovery jobs associated with a single recovery operation. Recovery point: The date and time of a previous version of a data source that is available for recovery from media that is managed by DPM. Report database: The SQL Server database that stores DPM reporting information (ReportServer.mdf). ReportServer.mdf: In DPM, the filename for the report database—a SQL Server database that stores reporting information. Retention range: Duration of time for which the data should be available for recovery.
Read more
  • 0
  • 0
  • 13140
article-image-working-cloud
Packt
02 May 2016
33 min read
Save for later

Working with the Cloud

Packt
02 May 2016
33 min read
In this article by Gastón C. Hillar , author of book Internet of Things with Python, we will take advantage of many cloud services to publish and visualize data collected for sensors and to establish bi-directional communications between Internet-connected things. (For more resources related to this topic, see here.) Sending and receiving data in real-time through Internet with PubNub We want to send and receive data in real-time through the Internet and a RESTful API is not the most appropriate option to do this. Instead, we will work with a publish/subscribe model-based on a protocol that is lighter than the HTTP protocol. Specifically, we will use a service-based on the MQ Telemetry Transport (MQTT) protocol. The MQTT protocol is a machine-to-machine (M2M) connectivity protocol. MQTT is a lightweight messaging protocol that runs on top of the TCP/IP protocol and works with a publish-subscribe mechanism. It is possible for any device to subscribe to a specific channel (also known as topic) and receive all the messages published to this channel. In addition, the device can publish message to this or other channel. The protocol is becoming very popular in IoT and M2M projects. You can read more about the MQTT protocol in the following Webpage: http://mqtt.org. PubNub provides many cloud-based services and one of them allows us to easily stream data and signal any device in real-time, working with the MQTT protocol under the hoods. We will take advantage of this PubNub service to send and receive data in real-time through Internet and make it easy to control our Intel Galileo Gen 2 board through the Internet. As PubNub provides a Python API with high quality documentation and examples, it is extremely easy to use the service in Python. PubNub defines itself as the global data stream network for IoT, Mobile and Web applications. You can read more about PubNub in its Webpage: http://www.pubnub.com. In our example, we will take advantage of the free services offered by PubNub and we won't use some advanced features and additional services that might empower our IoT project connectivity requirements but also require a paid subscription. PubNub requires us to sign up and create an account with a valid e-mail and a password before we can create an application within PubNub that allows us to start using their free services. We aren't required to enter any credit card or payment information. If you already have an account at PubNub, you can skip the next step. Once you created your account PubNub will redirect you to the admin portal that lists your PubNub applications. It is necessary to generate your PubNub publish and subscribe keys in order to send and receive messages in the network. A new pane will represent the application in the admin portal. The following screenshot shows the Temperature Control application pane in the PubNub admin portal. Click on the Temperature Control pane and PubNub will display the Demo Keyset pane that has been automatically generated for the application. Click on this pane and PubNub will display the publish, subscribe and secret keys. We must copy and paste each of the keys to use them in our code that will publish messages and subscribe to them. The following screenshot shows the prefixes for the keys and the remaining characters have been erased in the image. In order to copy the secret key, you must click on the eye icon at the right-hand side of the key and PubNub will make all the characters visible. Now, we will use pip installer to install PubNub Python SDK 3.7.6. We just need to run the following command in the SSH terminal to install the package. Notice that it can take a few minutes to complete the installation. pip install pubnub The last lines for the output will indicate that the pubnub package has been successfully installed. Don't worry about the error messages related to building wheel and the insecure platform warning. Downloading pubnub-3.7.6.tar.gz Collecting pycrypto>=2.6.1 (from pubnub) Downloading pycrypto-2.6.1.tar.gz (446kB) 100% |################################| 446kB 25kB/s Requirement already satisfied (use --upgrade to upgrade): requests>=2.4.0 in /usr/lib/python2.7/site-packages (from pubnub) Installing collected packages: pycrypto, pubnub Running setup.py install for pycrypto Installing collected packages: pycrypto, pubnub Running setup.py install for pycrypto Running setup.py install for pubnub Successfully installed pubnub-3.7.6 pycrypto-2.6.1 We will use iot_python_chapter_08_03.py (you will find this in the code bundle) code as a baseline to add new features that will allow us to perform the following actions with PubNub messages sent to a specific channel from any device that has a Web browser: Rotate the servo's shaft to display a temperature value in degrees Fahrenheit received as part of the message Display a line of text received as part of the message at the bottom of the OLED matrix We will use the recently installed pubnub module to subscribe to a specific channel and run code when we receive messages in the channel. We will create a MessageChannel class to represent the communications channel, configure the PubNub subscription and declare the code for the callbacks that are going to be executed when certain events are fired. The code file for the sample is iot_python_chapter_09_02.py (you will find this in the code bundle). Remember that we use the code file iot_python_chapter_08_03.py (you will find this in the code bundle)as a baseline, and therefore, we will add the class to the existing code in this file and we will create a new Python file. Don't forget to replace the strings assigned to the publish_key and subscribe_key local variables in the __init__ method with the values you have retrieved from the previously explained PubNub key generation process. import time from pubnub import Pubnub class MessageChannel: command_key = "command" def __init__(self, channel, temperature_servo, oled): self.temperature_servo = temperature_servo self.oled = oled self.channel = channel # Publish key is the one that usually starts with the "pub-c-" prefix # Do not forget to replace the string with your publish key publish_key = "pub-c-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" # Subscribe key is the one that usually starts with the "sub-c" prefix # Do not forget to replace the string with your subscribe key subscribe_key = "sub-c-xxxxxxxx-xxxx-xxxx-xxxx- xxxxxxxxxxxx" self.pubnub = Pubnub(publish_key=publish_key, subscribe_key=subscribe_key) self.pubnub.subscribe(channels=self.channel, callback=self.callback, error=self.callback, connect=self.connect, reconnect=self.reconnect, disconnect=self.disconnect) def callback(self, message, channel): if channel == self.channel: if self.__class__.command_key in message: if message[self.__class__.command_key] == "print_temperature_fahrenheit": self.temperature_servo.print_temperature (message["temperature_fahrenheit"]) elif message[self.__class__.command_key] == "print_information_message": self.oled.print_line(11, message["text"]) print("I've received the following message: {0}".format(message)) def error(self, message): print("Error: " + str(message)) def connect(self, message): print("Connected to the {0} channel". format(self.channel)) print(self.pubnub.publish( channel=self.channel, message="Listening to messages in the Intel Galileo Gen 2 board")) def reconnect(self, message): print("Reconnected to the {0} channel". format(self.channel)) def disconnect(self, message): print("Disconnected from the {0} channel". Format(self.channel)) The MessageChannel class declares the command_key class attribute that defines the key string that defines what the code will understand as the command. Whenever we receive a message that includes the specified key string, we know that the value associated to this key in the dictionary will indicate the command that the message wants the code running in the board to be processed. Each command requires additional key-value pairs that provide the necessary information to execute the command. We have to specify the PubNub channel name, the TemperatureServo instance and the Oled instance in the channel, temperature_servo and oled required arguments. The constructor, that is, the __init__ method, saves the received arguments in three attributes with the same names. The channel argument specifies the PubNub channel to which we are going to subscribe to listen to the messages that other devices send to this channel. We will also publish messages to this channel, and therefore, we will be both a subscriber and a publisher for this channel. In this case, we will only subscribe to one channel. However, it is very important to know that we are not limited to subscribe to a single channel, we might subscribe to many channels. Then, the constructor declares two local variables: publish_key and subscribe_key. These local variables save the publish and subscribe keys we had generated with the PubNub admin portal. Then, the code creates a new Pubnub instance with publish_key and subscribe_key as the arguments, and saves the reference for the new instance in the pubnub attribute. Finally, the code calls the subscribe method for the new instance to subscribe to data on the channel saved in the channel attribute. Under the hoods, the subscribe method makes the client create an open TCP socket to the PubNub network that includes an MQTT broker and starts listening to messages on the specified channel. The call to this method specifies many methods declared in the MessageChannel class for the following named arguments: callback: Specifies the function that will be called when there is a new message received from the channel error: Specifies the function that will be called on an error event connect: Specifies the function that will be called when a successful connection is established with the PubNub cloud reconnect: Specifies the function that will be called when a successful re-connection is completed with the PubNub cloud disconnect: Specifies the function that will be called when the client disconnects from the PubNub cloud This way, whenever one of the previously enumerated events occur, the specified method will be executed. The callback method receives two arguments: message and channel. First, the method checks whether the received channel matches the value in the channel attribute. In this case, whenever the callback method is executed, the value in the channel argument will always match the value in the channel attribute because we just subscribed to one channel. However, in case we subscribe to more than one channel, is is always necessary to check the channel in which the message was sent and in which we are receiving the message. Then, the code checks whether the command_key class attribute is included in the message dictionary. If the expression evaluates to True, it means that the message includes a command that we have to process. However, before we can process the command, we have to check which is the command, and therefore, it is necessary to retrieve the value associated with the key equivalent to the command_key class attribute. The code is capable of running code when the value is any of the following two commands: print_temperature_fahrenheit: The command must specify the temperature value expressed in degrees Fahrenheit in the value of the temperature_fahrenheit key. The code calls the self.temperature_servo.print_temperature method with the temperature value retrieved from the dictionary as an argument. This way, the code moves the servo's shaft-based on the specified temperature value in the message that includes the command. print_information_message: The command must specify the line of text that has to be displayed at the bottom of the OLED matrix in the value of the print_information_message key. The code calls the self.oled.print_line method with 11 and the text value retrieved from the dictionary as arguments. This way, the code displays the text received in the message that includes the command at the bottom of the OLED matrix. No matter whether the message included a valid command or not, the method prints the raw message that it received in the console output. The connect method prints a message indicating that a connection has been established with the channel. Then, the method prints the results of calling the self.pubnub.publish method that publishes a message in the channel name saved in self.channel with the following message: "Listening to messages in the Intel Galileo Gen 2 board". In this case, the call to this method runs with a synchronous execution. We will work with asynchronous execution for this method in our next example. Currently, we are already subscribed to this channel, and therefore, we will receive the previously published message and the callback method will be executed with this message as an argument. However, as the message doesn't include the key that identifies a command, the code in the callback method will just display the received message and it won't process any of the previously analyzed commands. The other methods declared in the MessageChannel class just display information to the console output about the event that has occurred. Now, we will use the previously coded MessageChannel class to create a new version of the __main__ method that uses the PubNub cloud to receive and process commands. The new version doesn't rotate the servo's shaft when the ambient temperature changes, instead, it will do this when it receives the appropriate command from any device connected to PubNub cloud. The following lines show the new version of the __main__ method. The code file for the sample is iot_python_chapter_09_02.py (you will find this in the code bundle). if __name__ == "__main__": temperature_and_humidity_sensor = TemperatureAndHumiditySensor(0) oled = TemperatureAndHumidityOled(0) temperature_servo = TemperatureServo(3) message_channel = MessageChannel("temperature", temperature_servo, oled) while True: temperature_and_humidity_sensor. measure_temperature_and_humidity() oled.print_temperature( temperature_and_humidity_sensor .temperature_fahrenheit, temperature_and_humidity_sensor.temperature_celsius) oled.print_humidity( temperature_and_humidity_sensor.humidity) print("Ambient temperature in degrees Celsius: {0}". format(temperature_and_humidity_sensor .temperature_celsius)) print("Ambient temperature in degrees Fahrenheit: {0}". format(temperature_and_humidity_sensor .temperature_fahrenheit)) print("Ambient humidity: {0}". format(temperature_and_humidity_sensor.humidity)) # Sleep 10 seconds (10000 milliseconds) time.sleep(10) The highlighted line creates an instance of the previously coded MessageChannel class with "temperature", temperature_servo and oled as the arguments. The constructor will subscribe to the temperature channel in the PubNub cloud, and therefore, we must send the messages to this channel in order to send the commands that the code will process with an asynchronous execution. The loop will read the values from the sensor and print the values to the console similar to the previous version of the code, and therefore, we will have code running in the loop and listening to the messages in the temperature channel in the PubNub cloud. We will start the example later because we want to subscribe to the channel in the PubNub debug console before we run the code in the board. Publishing messages with commands through the PubNub cloud Now, we will take advantage of the PubNub console to send messages with commands to the temperature channel and make the Python code running on the board process these commands. In case you have logged out of PubNub, login again and click on the Temperature Control pane in the Admin Portal. PubNub will display the Demo Keyset pane. Click on the Demo Keyset pane and PubNub will display the publish, subscribe and secret keys. This way, we select the keyset that we want to use for our PubNub application. Click on Debug Console on the sidebar located the left-hand side of the screen. PubNub will create a client for a default channel and subscribe to this channel using the secret keys we have selected in the previous step. We want to subscribe to the temperature channel, and therefore, enter temperature in the Default Channel textbox within a pane that includes the Add client button at the bottom. Then, click on Add client and PubNub will add a new pane with a random client name as a title and the channel name, temperature, in the second line. PubNub makes the client subscribe to this channel and we will be able to receive messages published to this channel and send messages to this channel. The following picture shows the pane for the generated client named Client-ot7pi, subscribed to the temperature channel. Notice that the client name will be different when you follow the explained steps. The client pane displays the output generated when PubNub subscribed the client to the channel. PubNub returns a formatted response for each command. In this case, it indicates that the status is equal to Subscribed and the channel name is temperature. [1,"Subscribed","temperature"] Now, it is time to start running the example in the Intel Galileo Gen 2 board. The following line will start the example in the SSH console. python iot_python_chapter_09_02.py After you run the example, go to the Web browser in which you are working with the PubNub debug console. You will see the following message listed in the previously created client: "Listening to messages in the Intel Galileo Gen 2 board" The Python code running in the board published this message, specifically, the connect method in the MessageChannel class sent this message after the application established a connection with the PubNub cloud. The following picture shows the message listed in the previously created client. Notice that the icon at the left-hand side of the text indicates it is a message. The other message was a debug message with the results of subscribing to the channel. At the bottom of the client pane, you will see the following text and the SEND button at the right-hand side: {"text":"Enter Message Here"} Now, we will replace the previously shown text with a message. Enter the following JSON code and click SEND: {"command":"print_temperature_fahrenheit", "temperature_fahrenheit": 50 } The text editor where you enter the message has some issues in certain browsers. Thus, it is convenient to use your favorite text editor to enter the JSON code, copy it and then past it to replace the text that is included by default in the text for the message to be sent. After you click SEND, the following lines will appear in the client log. The first line is a debug message with the results of publishing the message and indicates that the message has been sent. The formatted response includes a number (1 message), the status (Sent) and a time token. The second line is the message that arrives to the channel because we are subscribed to the temperature channel, that is, we also receive the message we sent. [1,"Sent","14594756860875537"] { "command": "print_temperature_fahrenheit", "temperature_fahrenheit": 50 } The following picture shows the messages and debug messages log for the PubNub client after we clicked the SEND button. After you publish the previous message, you will see the following output in the SSH console for the Intel Galileo Gen 2 board. You will notice the servo's shaft rotates to 50 degrees. I've received the following message: {u'command': u'print_temperature_fahrenheit', u'temperature_fahrenheit': 50} Now, enter the following JSON code and click SEND: {"command":"print_information_message", "text": "Client ready"} After you click SEND, the following lines will appear in the client log. The first line is a debug message with the previously explained formatted response with the results of publishing the message and indicates that the message has been sent. The second line is the message that arrives to the channel because we are subscribed to the temperature channel, that is, we also receive the message we sent. [1,"Sent","14594794434885921"] { "command": "print_information_message", "text": "Client ready" } The following picture shows the messages and debug messages log for the PubNub client after we clicked the SEND button. After you publish the previous message, you will see the following output in the SSH console for the Intel Galileo Gen 2 board. You will see the following text displayed at the bottom of the OLED matrix: Client ready. I've received the following message: {u'text': u'Client ready', u'command': u'print_information_message'} When we published the two messages with the commands, we have definitely noticed a problem. We don't know whether the command was processed or not in the code that is running on the IoT device, that is, in the Intel Galileo Gen 2 board. We know that the board started listening messages in the temperature channel, but we don't receive any kind of response from the IoT device after the command has been processed. Working with bi-directional communications We can easily add a few lines of code to publish a message to the same channel in which we are receiving messages to indicate that the command has been successfully processed. We will use our previous example as a baseline and we will create a new version of the MessageChannel class. The code file was iot_python_chapter_09_02.py (you will find this in the code bundle). Don't forget to replace the strings assigned to the publish_key and subscribe_key local variables in the __init__ method with the values you have retrieved from the previously explained PubNub key generation process. The following lines show the new version of the MessageChannel class that publishes a message after a command has been successfully processed. The code file for the sample is iot_python_chapter_09_03.py (you will find this in the code bundle). import time from pubnub import Pubnub class MessageChannel: command_key = "command" successfully_processed_command_key = "successfully_processed_command" def __init__(self, channel, temperature_servo, oled): self.temperature_servo = temperature_servo self.oled = oled self.channel = channel # Do not forget to replace the string with your publish key publish_key = "pub-c-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" # Subscribe key is the one that usually starts with the "sub-c" prefix # Do not forget to replace the string with your subscribe key subscribe_key = "sub-c-xxxxxxxx-xxxx-xxxx-xxxx- xxxxxxxxxxxx" self.pubnub = Pubnub(publish_key=publish_key, subscribe_key=subscribe_key) self.pubnub.subscribe(channels=self.channel, callback=self.callback, error=self.callback, connect=self.connect, reconnect=self.reconnect, disconnect=self.disconnect) def callback_response_message(self, message): print("I've received the following response from PubNub cloud: {0}".format(message)) def error_response_message(self, message): print("There was an error when working with the PubNub cloud: {0}".format(message)) def publish_response_message(self, message): response_message = { self.__class__.successfully_processed_command_key: message[self.__class__.command_key]} self.pubnub.publish( channel=self.channel, message=response_message, callback=self.callback_response_message, error=self.error_response_message) def callback(self, message, channel): if channel == self.channel: print("I've received the following message: {0}".format(message)) if self.__class__.command_key in message: if message[self.__class__.command_key] == "print_temperature_fahrenheit": self.temperature_servo.print_temperature (message["temperature_fahrenheit"]) self.publish_response_message(message) elif message[self.__class__.command_key] == "print_information_message": self.oled.print_line(11, message["text"]) self.publish_response_message(message) def error(self, message): print("Error: " + str(message)) def connect(self, message): print("Connected to the {0} channel". format(self.channel)) print(self.pubnub.publish( channel=self.channel, message="Listening to messages in the Intel Galileo Gen 2 board")) def reconnect(self, message): print("Reconnected to the {0} channel". format(self.channel)) def disconnect(self, message): print("Disconnected from the {0} channel". format(self.channel)) The highlighted lines in the previous code for the new version of the MessageChannel class show the changes we made in the code. First, the code declares the successfully_processed_command_key class attribute that defines the key string that defines what the code will use as a successfully processed command key in a response message published to the channel. Whenever we publish a message that includes the specified key string, we know that the value associated to this key in the dictionary will indicate the command that the board has successfully processed. The code declares the following three new methods: callback_response_message: This method will be used as the callback that will be executed when a successfully processed command response message is published to the channel. The method just prints the formatted response that PubNub returns when a message has been successfully published in the channel. In this case, the message argument doesn't hold the original message that has been published, it holds the formatted response. We use message for the argument name to keep consistency with the PubNub API. error_response_message: This method will be used as the callback that will be executed when an error occurs when trying to publish a successfully processed command response message to the channel. The method just prints the error message that PubNub returns when a message hasn't been successfully published in the channel. publish_response_message: This method receives the message with the command that was successfully processed in the message argument. The code creates a response_message dictionary with the successfully_processed_command_key class attribute as the key and the value of the key specified in the command_key class attribute for the message dictionary as the value. Then, the code calls the self.pubnub.publish method to publish the response_message dictionary to the channel saved in the channel attribute. The call to this method specifies self.callback_response_message as the callback to be executed when the message is successfully published and self.error_response_message as the callback to be executed when an error occurred during the publishing process. When we specify a callback, the publish method works with an asynchronous execution, and therefore, the execution is non-blocking. The publication of the message and the callbacks that are specified will run in a different thread. Now, the callback method defined in the MessageChannel class adds a call to the publish_response_message method with the message that included the command that has been successfully processed (message) as an argument. As previously explained, the publish_response_message method is non-blocking and will return immediately while the successfully processed message is published in another thread. Now, it is time to start running the example in the Intel Galileo Gen 2 board. The following line will start the example in the SSH console. python iot_python_chapter_09_03.py After you run the example, go to the Web browser in which you are working with the PubNub debug console. You will see the following message listed in the previously created client: "Listening to messages in the Intel Galileo Gen 2 board" Enter the following JSON code and click SEND: {"command":"print_temperature_fahrenheit", "temperature_fahrenheit": 90 } After you click SEND, the following lines will appear in the client log. The last message was published by the board to the channel indicates that the print_temperature_fahrenheit command has been successfully processed. [1,"Sent","14595406989121047"] { "command": "print_temperature_fahrenheit", "temperature_fahrenheit": 90 } { "successfully_processed_command": "print_temperature_fahrenheit" } The following picture shows the messages and debug messages log for the PubNub client after we clicked the SEND button: After you publish the previous message, you will see the following output in the SSH console for the Intel Galileo Gen 2 board. You will notice the servo's shaft rotates to 90 degrees. The board also receives the successfully processed command message because it is subscribed to the channel in which the message has been published. I've received the following message: {u'command': u'print_temperature_fahrenheit', u'temperature_fahrenheit': 90} I've received the following response from PubNub cloud: [1, u'Sent', u'14595422426124592'] I've received the following message: {u'successfully_processed_command': u'print_temperature_fahrenheit'} Now, enter the following JSON code and click SEND: {"command":"print_information_message", "text": "2nd message"} After you click Send, the following lines will appear in the client log. The last message published by the board to the channel indicates that the print_information_message command has been successfully processed. [1,"Sent","14595434708640961"] { "command": "print_information_message", "text": "2nd message" } { "successfully_processed_command": "print_information_message" } The following picture shows the messages and debug messages log for the PubNub client after we clicked the SEND button: After you publish the previous message, you will see the following output in the SSH console for the Intel Galileo Gen 2 board. You will see the following text displayed at the bottom of the OLED matrix: 2nd message. The board also receives the successfully processed command message because it is subscribed to the channel in which the message has been published. I've received the following message: {u'text': u'2nd message', u'command': u'print_information_message'} 2nd message I've received the following response from PubNub cloud: [1, u'Sent', u'14595434710438777'] I've received the following message: {u'successfully_processed_command': u'print_information_message'} We can work with the different SDKs provided by PubNub to subscribe and publish to a channel. We can also make different IoT devices talk to themselves by publishing messages to channels and processing them. In this case, we just created a few commands and we didn't add detailed information about the device that has to process the command or the device that has generated a specific message. A more complex API would require commands that include more information and security. Publishing messages to the cloud with a Python PubNub client So far, we have been using the PubNub debug console to publish messages to the temperature channel and make the Python code running in the Intel Galileo Gen 2 board process them. Now, we are going to code a Python client that will publish messages to the temperature channel. This way, we will be able to design applications that can talk to IoT devices with Python code in the publisher and in the subscriber devices. We can run the Python client on another Intel Galileo Gen 2 board or in any device that has Python 2.7.x installed. In addition, the code will run with Python 3.x. For example, we can run the Python client in our computer. We just need to make sure that we install the pubnub module we have previously installed with pip in the Python version that is running in the Yocto Linux for the board. We will create a Client class to represent a PubNub client, configure the PubNub subscription, make it easy to publish a message with a command and the required values for the command and declare the code for the callbacks that are going to be executed when certain events are fired. The code file for the sample is iot_python_chapter_09_04.py (you will find this in the code bundle). Don't forget to replace the strings assigned to the publish_key and subscribe_key local variables in the __init__ method with the values you have retrieved from the previously explained PubNub key generation process. The following lines show the code for the Client class: import time from pubnub import Pubnub class Client: command_key = "command" def __init__(self, channel): self.channel = channel # Publish key is the one that usually starts with the "pub-c-" prefix publish_key = "pub-c-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" # Subscribe key is the one that usually starts with the "sub-c" prefix # Do not forget to replace the string with your subscribe key subscribe_key = "sub-c-xxxxxxxx-xxxx-xxxx-xxxx- xxxxxxxxxxxx" self.pubnub = Pubnub(publish_key=publish_key, subscribe_key=subscribe_key) self.pubnub.subscribe(channels=self.channel, callback=self.callback, error=self.callback, connect=self.connect, reconnect=self.reconnect, disconnect=self.disconnect) def callback_command_message(self, message): print("I've received the following response from PubNub cloud: {0}".format(message)) def error_command_message(self, message): print("There was an error when working with the PubNub cloud: {0}".format(message)) def publish_command(self, command_name, key, value): command_message = { self.__class__.command_key: command_name, key: value} self.pubnub.publish( channel=self.channel, message=command_message, callback=self.callback_command_message, error=self.error_command_message) def callback(self, message, channel): if channel == self.channel: print("I've received the following message: {0}".format(message)) def error(self, message): print("Error: " + str(message)) def connect(self, message): print("Connected to the {0} channel". format(self.channel)) print(self.pubnub.publish( channel=self.channel, message="Listening to messages in the PubNub Python Client")) def reconnect(self, message): print("Reconnected to the {0} channel". format(self.channel)) def disconnect(self, message): print("Disconnected from the {0} channel". format(self.channel)) The Client class declares the command_key class attribute that defines the key string that defines what the code understands as a command in the messages. Our main goal is to build and publish command messages to a specified channel. We have to specify the PubNub channel name in the channel-required argument. The constructor, that is, the __init__ method, saves the received argument in an attribute with the same name. We will be both a subscriber and a publisher for this channel. Then, the constructor declares two local variables: publish_key and subscribe_key. These local variables save the publish and subscribe keys we had generated with the PubNub Admin portal. Then, the code creates a new Pubnub instance with publish_key and subscribe_key as the arguments, and saves the reference for the new instance in the pubnub attribute. Finally, the code calls the subscribe method for the new instance to subscribe to data on the channel saved in the channel attribute. The call to this method specifies many methods declared in the Client class as we did for our previous examples. The publish_command method receives a command name, the key and the value that provide the necessary information to execute the command in the command_name, key and value required arguments. In this case, we don't target the command to a specific IoT device and all the devices that subscribe to the channel and run the code in our previous example will process the commands that we publish. We can use the code as a baseline to work with more complex examples in which generate commands that target specific IoT devices. Obviously, it is also necessary to improve the security. The method creates a dictionary and saves it in the command_message local variable. The command_key class attribute is the first key for the dictionary and the command_name received as an argument, the value that composes the first key-value pair. Then, the code calls the self.pubnub.publish method to publish the command_message dictionary to the channel saved in the channel attribute. The call to this method specifies self.callback_command_message as the callback to be executed when the message is published successfully and self.error_command_message as the callback to be executed when an error occurred during the publishing process. As happened in our previous example, when we specify a callback, the publish method works with an asynchronous execution. Now, we will use the previously coded Client class to write a __main__ method that uses the PubNub cloud to publish two commands that our board will process. The following lines show the code for the __main__ method. The code file for the sample is iot_python_chapter_09_04.py (you will find this in the code bundle). if __name__ == "__main__": client = Client("temperature") client.publish_command( "print_temperature_fahrenheit", "temperature_fahrenheit", 45) client.publish_command( "print_information_message", "text", "Python IoT" ) # Sleep 60 seconds (60000 milliseconds) time.sleep(60) The code in the __main__ method is very easy to understand. The code creates an instance of the Client class with "temperature" as an argument to become both a subscriber and a publisher for this channel in the PubNub cloud. The code saves the new instances in the client local variable. The code calls the publish_command method with the necessary arguments to build and publish the print_temperature_fahrenheit command with a temperature value of 45. The method will publish the command with an asynchronous execution. Then, the code calls the publish_command method again with the necessary arguments to build and publish the print_information_message command with a text value of "Python IoT". The method will publish the second command with an asynchronous execution. Finally, the code sleeps for 1 minute (60 seconds) in order to make it possible for the asynchronous executions to publish the commands successfully. The different callbacks defined in the Client class will be executed as the different events fire. As we are also subscribed to the channel, we will also receive the messages we publish in the temperature channel. Keep the Python code we executed in our previous example running on the board. We want the board to process our commands. In addition, keep the Web browser in which you are working with the PubNub debug console opened because we also want to see all the messages in the log. The following line will start the example for the Python client in any computer or device that you want to use as a client. It is possible to run the code in another SSH terminal in case you want to use the same board as a client. python iot_python_chapter_09_04.py After you run the example, you will see the following output in the Python console that runs the Python client, that is, the iot_python_chapter_09_04.py (you will find this in the code bundle)Python script. Connected to the temperature channel I've received the following response from PubNub cloud: [1, u'Sent', u'14596508980494876'] I've received the following response from PubNub cloud: [1, u'Sent', u'14596508980505581'] [1, u'Sent', u'14596508982165140'] I've received the following message: {u'text': u'Python IoT', u'command': u'print_information_message'} I've received the following message: {u'command': u'print_temperature_fahrenheit', u'temperature_fahrenheit': 45} I've received the following message: Listening to messages in the PubNub Python Client I've received the following message: {u'successfully_processed_command': u'print_information_message'} I've received the following message: {u'successfully_processed_command': u'print_temperature_fahrenheit'} The code used the PubNub Python SDK to build and publish the following two command messages in the temperature channel: {"command":"print_temperature_fahrenheit", "temperature_fahrenheit": "45"} {"command":"print_information_message", "text": "Python IoT"} As we are also subscribed to the temperature channel, we receive the messages we sent with an asynchronous execution. Then, we received the successfully processed command messages for the two command messages. The board has processed the commands and published the messages to the temperature channel. After you run the example, go to the Web browser in which you are working with the PubNub debug console. You will see the following messages listed in the previously created client: [1,"Subscribed","temperature"] "Listening to messages in the Intel Galileo Gen 2 board" { "text": "Python IoT", "command": "print_information_message" } { "command": "print_temperature_fahrenheit", "temperature_fahrenheit": 45 } "Listening to messages in the PubNub Python Client" { "successfully_processed_command": "print_information_message" } { "successfully_processed_command": "print_temperature_fahrenheit" } The following picture shows the last messages displayed in the log for the PubNub client after we run the previous example: You will see the following text displayed at the bottom of the OLED matrix: Python IoT. In addition, the servo's shaft will rotate to 45 degrees. We can use the PubNub SDKs available in different programming languages to create applications and apps that publish and receive messages in the PubNub cloud and interact with IoT devices. In this case, we worked with the Python SDK to create a client that publishes commands. It is possible to create mobile apps that publish commands and easily build an app that can interact with our IoT device. Summary In this article, we combined many cloud-based services that allowed us to easily publish data collected from sensors and visualize it in a Web-based dashboard. We realized that there is always a Python API, and therefore, it is easy to write Python code that interacts with popular cloud-based services. Resources for Article: Further resources on this subject: Internet of Things with BeagleBone[article] The Internet of Things[article] Python Data Structures[article]
Read more
  • 0
  • 0
  • 13133

article-image-working-vmware-infrastructure
Packt
04 Mar 2015
21 min read
Save for later

Working with VMware Infrastructure

Packt
04 Mar 2015
21 min read
In this article by Daniel Langenhan, the author of VMware vRealize Orchestrator Cookbook, we will take a closer look at how Orchestrator interacts with vCenter Server and vRealize Automation (vRA—formerly known as vCloud Automation Center, vCAC). vRA uses Orchestrator to access and automate infrastructure using Orchestrator plugins. We will take a look at how to make Orchestrator workflows available to vRA. We will investigate the following recipes: Unmounting all the CD-ROMs of all VMs in a cluster Provisioning a VM from a template An approval process for VM provisioning (For more resources related to this topic, see here.) There are quite a lot of plugins for Orchestrator to interact with VMware infrastructure and programs: vCenter Server vCloud Director (vCD) vRealize Automation (vRA—formally known as vCloud Automation Center, vCAC) Site Recovery Manager (SRM) VMware Auto Deploy Horizon (View and Virtual Desktops) vRealize Configuration Manager (earlier known as vCenter Configuration Manager) vCenter Update Manager vCenter Operation Manager, vCOPS (only example packages) VMware, as of writing of this article, is still renaming its products. An overview of all plugins and their names and download links can be found at http://www.vcoteam.info/links/plug-ins.html. There are quite a lot of plugins, and we will not be able to cover all of them, so we will focus on the one that is most used, vCenter. Sadly, vCloud Director is earmarked by VMware to disappear for everyone but service providers, so there is no real need to show any workflow for it. We will also work with vRA and see how it interacts with Orchestrator. vSphere automation The interaction between Orchestrator and vCenter is done using the vCenter API. Here is the explanation of the interaction, which you can refer to in the following figure. A user starts an Orchestrator workflow (1) either in an interactive way via the vSphere Web Client, the Orchestrator Web Operator, the Orchestrator Client, or via the API. The workflow in Orchestrator will then send a job (2) to vCenter and receive a task ID back (type VC:Task). vCenter will then start enacting the job (3). Using the vim3WaitTaskEnd action (4), Orchestrator pauses until the task has been completed. If we do not use the wait task, we can't be certain whether the task has ended or failed. It is extremely important to use the vim3WaitTaskEnd action whenever we send a job to vCenter. When the wait task reports that the job has finished, the workflow will be marked as finished. The vCenter MoRef The MoRef (Managed Object Reference) is a unique ID for every object inside vCenter. MoRefs are basically strings; some examples are shown here: VM Network Datastore ESXi host Data center Cluster vm-301 network-312 dvportgroup-242 datastore-101 host-44 data center-21 domain-c41 The MoRefs are typically stored in the attribute .id or .key of the Orchestrator API object. For example, the MoRef of a vSwitch Network is VC:Network.id. To browse for MoRefs, you can use the Managed Object Browser (MOB), documented at https://pubs.vmware.com/vsphere-55/index.jsp#com.vmware.wssdk.pg.doc/PG_Appx_Using_MOB.20.1.html. The vim3WaitTaskEnd action As already said, vim3WaitTaskEnd is one of the most central actions while interacting with vCenter. The action has the following variables: Category Name Type Usage IN vcTask VC:Task Carries the reconfiguration task from the script to the wait task IN progress Boolean Write to the logs the progress of a task in percentage IN pollRate Number How often the action should be checked for task completion in vCenter OUT ActionResult Any Returns the task's result The wait task will check in regular intervals (pollRate) the status of a task that has been submitted to vCenter. The task can have the following states: State Meaning Queued The task is queued and will be executed as soon as possible. Running The task is currently running. If the progress is set to true, the progress in percentage will be displayed in the logs. Success The task is finished successfully. Error The task has failed and an error will be thrown. Other vCenter wait actions There are actually five waiting tasks that come with the vCenter Server plugin. Here's an overview of the other four: Task Description vim3WaitToolsStarted This task waits until the VMware tools are started on a VM or until a timeout is reached. Vim3WaitForPrincipalIP This task waits until the VMware tools report the primary IP of a VM or until a timeout is reached. This typically indicates that the operating system is ready to receive network traffic. The action will return the primary IP. Vim3WaitDnsNameInTools This task waits until the VMware tools report a given DNS name of a VM or until a timeout is reached. The in-parameter addNumberToName is not used and can be set to Null. WaitTaskEndOrVMQuestion This task waits until a task is finished or if a VM develops a question. A vCenter question is related to user interaction. vRealize Automation (vRA) Automation has changed since the beginning of Orchestrator. Before, tools such as vCloud Director or vCloud Automation Center (vCAC)/vRealize Automation (vRA), Orchestrator was the main tool for automating vCenter resources. With version 6.2 of vCloud Automation Center (vCAC), the product has been renamed vRealize Automation. Now vRA is deemed to become the central cornerstone in the VMware automation effort. vRealize Orchestrator (vRO), is used by vRA to interact with and automate VMware and non-VMware products and infrastructure elements. Throughout the various vCAC/vRA interactions, the role of Orchestrator has changed substantially. Orchestrator started off as an extension to vCAC and became a central part of vRA. In vCAC 5.x, Orchestrator was only an extension of the IaaS life cycle. Orchestrator was tied in using the stubs vCAC 6.0 integrated Orchestrator as an XaaS service (Everything as a Service) using the Advanced Service Designer (ASD) In vCAC 6.1, Orchestrator is used to perform all VMware NSX operations (VMware's new network virtualization and automation), meaning that it became even more of a central part of the IaaS services. With vCAC 6.2, the Advance Service Designer (ASD) was enhanced to allow more complex form of designs, allowing better leverage of Orchestrator workflows. As you can see in the following figure, vRA connects to the vCenter Server using an infrastructure endpoint that allows vRA to conduct basic infrastructure actions, such as power operations, cloning, and so on. It doesn't allow any complex interactions with the vSphere infrastructure, such as HA configurations. Using the Advanced Service Endpoints, vRA integrates the Orchestrator (vRO) plugins as additional services. This allows vRA to offer the entire plugin infrastructure as services to vRA. The vCenter Server, AD, and PowerShell plugins are typical integrations that are used with vRA. Using Advance Service Designer (ASD), you can create integrations that use Orchestrator workflows. ASD allows you to offer Orchestrator workflows as vRA catalog items, making it possible for tenants to access any IT service that can be configured with Orchestrator via its plugins. The following diagram shows an example using the Active Directory plugin. The Orchestrator Plugin provides access to the AD services. By creating a custom resource using the exposed AD infrastructure, we can create a service blueprint and resource actions, both of which are based on Orchestrator workflows that use the AD plugin. The other method of integrating Orchestrator into the IaaS life cycle, which was predominately used in vCAC 5.x was to use the stubs. The build process of a VM has several steps; each step can be assigned a customizable workflow (called a stub). You can configure vRA to run an Orchestrator workflow at these stubs in order to facilitate a few customized actions. Such actions could be taken to change the VMs HA or DRS configuration, or to use the guest integration to install or configure a program on a VM. Installation How to install and configure vRA is out of the scope of this article, but take a look at http://www.kendrickcoleman.com/index.php/Tech-Blog/how-to-install-vcloud-automation-center-vcac-60-part-1-identity-appliance.html for more information. If you don't have the hardware or the time to install vRA yourself, you can use the VMware Hands-on Labs, which can be accessed after clicking on Try for Free at http://hol.vmware.com. The vRA Orchestrator plugin Due to the renaming, the vRA plugin is called vRealize Orchestrator vRA Plug-in 6.2.0, however the file you download and use is named o11nplugin-vcac-6.2.0-2287231.vmoapp. The plugin currently creates a workflow folder called vCloud Automation Center. vRA-integrated Orchestrator The vRA appliance comes with an installed and configured vRO instance; however, the best practice for a production environment is to use a dedicated Orchestrator installation, even better would be an Orchestrator cluster. Dynamic Types or XaaS XaaS means Everything (X) as a Service. The introduction of Dynamic Types in Orchestrator Version 5.5.1 does exactly that; it allows you to build your own plugins and interact with infrastructure that has not yet received its own plugin. Take a look at this article by Christophe Decanini; it integrates Twitter with Orchestrator using Dynamic Types at http://www.vcoteam.info/articles/learn-vco/282-dynamic-types-tutorial-implement-your-own-twitter-plug-in-without-any-scripting.html. Read more… To read more about Orchestrator integration with vRA, please take a look at the official VMware documentation. Please note that the official documentation you need to look at is about vRealize Automation, and not about vCloud Automation Center, but, as of writing this article, the documentation can be found at https://www.vmware.com/support/pubs/vrealize-automation-pubs.html. The document called Advanced Service Design deals with vRO and Advanced Service Designer The document called Machine Extensibility discusses customization using subs Unmounting all the CD-ROMs of all VMs in a cluster This is an easy recipe to start with, but one you can really make it work for your existing infrastructure. The workflow will unmount all CD-ROMs from a running VM. A mounted CD-ROM may block a VM from being vMotioned. Getting ready We need a VM that can mount a CD-ROM either as an ISO from a host or from the client. Before you start the workflow, make sure that the VM is powered on and has an ISO connected to it. How to do it... Create a new workflow with the following variables: Name Type Section Use cluster VC:ClusterComputerResource IN Used to input the cluster clusterVMs Array of VC:VirtualMachine Attribute Use to capture all VMs in a cluster Add the getAllVMsOfCluster action to the schema and assign the cluster in-parameter and the clusterVMs attribute to it as actionResult. Now, add a Foreach element to the schema and assign the workflow Disconnect all detachable devices from a running virtual machine. Assign the Foreach element clusterVMs as a parameter. Save and run the workflow. How it works... This recipe shows how fast and easily you can design solutions that help you with everyday vCenter problems. The problem is that VMs that have CD-ROMs or floppies mounted may experience problems using vMotion, making it impossible for them to be used with DRS. The reality is that a lot of admins mount CD-ROMs and then forget to disconnect them. Scheduling this script every evening just before the nighttime backups will make sure that a production cluster is able to make full use of DRS and is therefore better load-balanced. You can improve this workflow by integrating an exclusion list. See also Refer to the example workflow, 7.01 UnMount CD-ROM from Cluster. Provisioning a VM from a template In this recipe, we will build a deployment workflow for Windows and Linux VMs. We will learn how to create workflows and reduce the amount of input variables. Getting ready We need a Linux or Windows template that we can clone and provision. How to do it… We have split this recipe in two sections. In the first section, we will create a configuration element, and in the second, we will create the workflow. Creating a configuration We will use a configuration for all reusable variables. Build a configuration element that contains the following items: Name Type Use productId String This is the Windows product ID—the licensing code joinDomain String This is the Windows domain FQDN to join domainAdmin Credential These are the credentials to join the domain licenseMode VC:CustomizationLicenseDataMode Example, perServer licenseUsers Number This denotes the number of licensed concurrent users inTimezone Enums:MSTimeZone Time zone fullName String Full name of the user orgName String Organization name newAdminPassword String New admin password dnsServerList Array of String List of DNS servers dnsDomain String DNS domain gateway Array of String List of gateways Creating the base workflow Now we will create the base workflow: Create the workflow as shown in the following figure by adding the given elements:      Clone, Windows with single NIC and credential      Clone, Linux with single NIC      Custom decision Use the Clone, Windows… workflow to create all variables. Link up the ones that you have defined in the configuration as attributes. The rest are defined as follows: Name Type Section Use vmName String IN This is the new virtual machine's name vm VC:VirtualMachine IN Virtual machine to clone folder VC:VmFolder IN This is the virtual machine folder datastore VC:Datastore IN This is the datastore in which you store the virtual machine pool VC:ResourcePool IN This is the resource pool in which you create the virtual machine network VC:Network IN This is the network to which you attach the virtual network interface ipAddress String IN This is the fixed valid IP address subnetMask String IN This is the subnet mask template Boolean Attribute For value No, mark new VM as template powerOn Boolean Attribute For value Yes, power on the VM after creation doSysprep Boolean Attribute For value Yes, run Windows Sysprep dhcp Boolean Attribute For value No, use DHCP newVM VC:VirtualMachine OUT This is the newly-created VM The following sub-workflow in-parameters will be set to special values: Workflow In-parameter value Clone, Windows with single NIC and credential host Null joinWorkgroup Null macAddress Null netBIOS Null primaryWINS Null secondaryWINS Null name vmName clientName vmName Clone, Linux with single NIC host Null macAddress Null name vmName clientName vmName Define the in-parameter VM as input for the Custom decision and add the following script. The script will check whether the name of the OS contains the word Microsoft: guestOS=vm.config.guestFullName; System.log(guestOS);if (guestOS.indexOf("Microsoft") >=0){return true;} else {return false} Save and run the workflow. This workflow will now create a new VM from an existing VM and customize it with a fixed IP. How it works… As you can see, creating workflows to automate vCenter deployments is pretty straightforward. Dealing with the various in-parameters of workflows can be quite overwhelming. The best way to deal with this problem is to hide away variables by defining them centrally using a configuration, or define them locally as attributes. Using configurations has the advantage that you can create them once and reuse them as needed. You can even push the concept a bit further by defining multiple configurations for multiple purposes, such as different environments. While creating a new workflow for automation, a typical approach is as follows: Look for a workflow that you need. Run the workflow normally to check out what it actually does. Either create a new workflow that uses the original or duplicate and edit the one you tried, modifying it until it does what you want. A fast way to deal with a lot of variables is to drag every element you need into the schema and then use the binding to create the variables as needed. You may have noticed that this workflow only lets you select vSwitch networks, not distributed vSwitch networks. You can improve this workflow with the following features: Read the existing Sysprep information stored in your vCenter Server Generate different predefined configurations (for example DEV or Prod) There's more... We can improve the workflow by implementing the ability to change the vCPU and the memory of the VM. Follow these steps to implement it: Move the out-parameter newVM to be an attribute. Add the following variables: Name Type Section Use vCPU Number IN This variable denotes the amount of vCPUs Memory Number IN This variable denotes the amount of VM memory vcTask VC:Task Attribute This variable will carry the reconfiguration task from the script to the wait task progress Boolean Attribute Value NO, vim3WaitTaskEnd pollRate Number Attribute Value 5, vim3WaitTaskEnd ActionResult Any Attribute vim3WaitTaskEnd Add the following actions and workflows according to the next figure:      shutdownVMAndForce      changeVMvCPU      vim3WaitTaskEnd      changeVMRAM      Start virtual machine Bind newVM to all the appropriate input parameters of the added actions and workflows. Bind actionResults (VC:tasks) of the change actions to vim3WaitTasks. See also Refer to the example workflows, 7.02.1 Provision VM (Base), 7.02.2 Provision VM (HW custom), as well as the configuration element, 7 VM provisioning. An approval process for VM provisioning In this recipe, we will see how to create a workflow that waits for an approver to approve the VM creation before provisioning it. We will learn how to combine mail and external events in a workflow to make it interact with different users. Getting ready For this recipe, we first need the provisioning workflow that we have created in the Provisioning a VM from a template recipe. You can use the example workflow, 7.02.1 Provision VM (Base). Additionally, we need a functional e-mail system as well as a workflow to send e-mails. You can use the example workflow, 4.02.1 SendMail as well as its configuration item, 4.2.1 Working with e-mail. How to do it… We will split this recipe in three parts. First, we will create a configuration element then, we will create the workflow, and lastly, we will use a presentation to make the workflow usable. Creating a configuration element We will use a configuration for all reusable variables. Build a configuration element that contains the following items: Name Type Use templates Array/VC:VirtualMachine This contains all the VMs that serve as templates folders Array/VC:VmFolder This contains all the VM folders that are targets for VM provisioning networks Array/VC:Network This contains all VM networks that are targets for VM provisioning resourcePools Array/VC:ResourcePool This contains all resource pools that are targets for VM provisioning datastores Array/VC:Datastore This contains all datastores that are targets for VM provisioning daysToApproval Number These are the number of days the approval should be available for approver String This is the e-mail of the approver Please note that you also have to define or use the configuration elements for SendMail, as well as the Provision VM workflows. You can use the examples contained in the example package. Creating a workflow Create a new workflow and add the following variables: Name Type Section Use mailRequester String IN This is the e-mail address of the requester vmName String IN This is the name of the new virtual machine vm VC:VirtualMachine IN This is the virtual machine to be cloned folder VC:VmFolder IN This is the virtual machine folder datastore VC:Datastore IN This is the datastore in which you store the virtual machine pool VC:ResourcePool IN This is the resource pool in which you create the virtual machine network VC:Network IN This is the network to which you attach the virtual network interface ipAddress String IN This is the fixed valid IP address subnetMask String IN This is the subnet mask isExternalEvent Boolean Attribute A value of true defines this event as external mailApproverSubject String Attribute This is the subject line of the mail sent to the approver mailApproverContent String Attribute This is the content of the mail that is sent to the approver mailRequesterSubject String Attribute This is the subject line of the mail sent to the requester when the VM is provisioned mailRequesterContent String Attribute This is the content of the mail that is sent to the requester when the VM is provisioned mailRequesterDeclinedSubject String Attribute This is the subject line of the mail sent to the requester when the VM is declined mailRequesterDeclinedContent String Attribute This is the content of the mail that is sent to the requester when the VM is declined eventName String Attribute This is the name of the external event endDate Date Attribute This is the end date for the wait of external event approvalSuccess Boolean Attribute This checks whether the VM has been approved Now add all the attributes we defined in the configuration element and link them to the configuration. Create the workflow as shown in the following figure by adding the given elements:      Scriptable task      4.02.1 SendMail (example workflow)       Wait for custom event       Decision       Provision VM (example workflow) Edit the scriptable task and bind the following variables to it: In Out vmName ipAddress mailRequester template approver days to approval mailApproverSubject mailApproverContent mailRequesterSubject mailRequesterContent mailRequesterDeclinedSubject mailRequesterDeclinedContent eventName endDate Add the following script to the scriptable task: //construct event name eventName="provision-"+vmName; //add days to today for approval var today = new Date(); var endDate = new Date(today); endDate.setDate(today.getDate()+daysToApproval); //construct external URL for approval var myURL = new URL() ; myURL=System.customEventUrl(eventName, false); externalURL=myURL.url; //mail to approver mailApproverSubject="Approval needed: "+vmName; mailApproverContent="Dear Approver,n the user "+mailRequester+" would like to provision a VM from template "+template.name+".n To approve please click here: "+externalURL; //VM provisioned mailRequesterSubject="VM ready :"+vmName; mailRequesterContent="Dear Requester,n the VM "+vmName+" has been provisioned and is now available under IP :"+ipAddress; //declined mailRequesterDeclinedSubject="Declined :"+vmName; mailRequesterDeclinedContent="Dear Requester,n the VM "+vmName+" has been declined by "+approver; Bind the out-parameter of Wait for customer event to approvalSuccess. Configure the Decision element with approvalSuccess as true. Bind all the other variables to the workflow elements. Improving with the presentation We will now edit the workflow's presentation in order to make it workable for the requester. To do so, follow the given steps: Click on Presentation and follow the steps to alter the presentation, as seen in the following screenshot: Add the following properties to the in-parameters: In-parameter Property Value template Predefined list of elements #templates folder Predefined list of elements #folders datastore Predefined list of elements #datastores pool Predefined list of elements #resourcePools network Predefined list of elements #networks You can now use the General tab of each in-parameter to change the displayed text. Save and close the workflow. How it works… This is a very simplified example of an approval workflow to create VMs. The aim of this recipe is to introduce you to the method and ideas of how to build such a workflow. This workflow will only give a requester the choices that are configured in the configuration element, making the workflow quite safe for users that have only limited knowhow of the IT environment. When the requester submits the workflow, an e-mail is sent to the approver. The e-mail contains a link, which when clicked, triggers the external event and approves the VM. If the VM is approved it will get provisioned, and when the provisioning has finished an e-mail is sent to the requester stating that the VM is now available. If the VM is not approved within a certain timeframe, the requester will receive an e-mail that the VM was not approved. To make this workflow fully functional, you can add permissions for a requester group to the workflow and Orchestrator so that the user can use the vCenter to request a VM. Things you can do to improve the workflow are as follows: Schedule the provisioning to a future date. Use the resources for the e-mail and replace the content. Add an error workflow in case the provisioning fails. Use AD to read out the current user's e-mail and full name to improve the workflow. Create a workflow that lets an approver configure the configuration elements that a requester can chose from. Reduce the selections by creating, for instance, a development and production configuration that contains the correct folders, datastores, networks, and so on. Create a decommissioning workflow that is automatically scheduled so that the VM is destroyed automatically after a given period of time. See also Refer to the example workflow, 7.03 Approval and the configuration element, 7 approval. Summary In this article, we discussed one of the important aspects of the interaction of Orchestrator with vCenter Server and vRealize Automation, that is VM provisioning. Resources for Article: Further resources on this subject: Importance of Windows RDS in Horizon View [article] Metrics in vRealize Operations [article] Designing and Building a Horizon View 6.0 Infrastructure [article]
Read more
  • 0
  • 0
  • 13128
Modal Close icon
Modal Close icon