How-To Tutorials

article-image-building-smart-contract-deployment-platform

15 Jun 2017

21 min read

Building a Smart Contract Deployment Platform

15 Jun 2017

0
0
3861

How-To Tutorials

Packt

15 Jun 2017

11 min read

Your first Unity project

Packt

15 Jun 2017

11 min read

In this article by Tommaso Lintrami, the author of the book Unity 2017 Game Development Essentials - Third Edition, we will see that when starting out in game development, one of the best ways to learn the various parts of the discipline is to prototype your idea. Unity excels in assisting you with this, with its visual scene editor and public member variables that form settings in the Inspector. To get to grips with working in the Unity editor. In this article, you will learn about: Creating a New Project in Unity Working with GameObjects in the SceneView and Hierarchys (For more resources related to this topic, see here.) As Unity comes in two main forms—a standard, free download and a paid Pro developer license.We'll stick to using features that users of the standard free edition will have access to. If you're launching Unity for the very first time, you'll be presented with a Unitydemonstration project. While this is useful to look into the best practices for the development of high-end projects, if you're starting out, looking over some of the assets and scripting may feel daunting, so we'll leave this behind and start from scratch! Take a look at the following steps for setting up your Unity project: In Unity go to File | NewProject and you will be presented with the ProjectWizard.The following screenshot is a Mac version shown: From here select the NEW tab and 3D type of project. Be aware that if at any time you wish to launch Unity and be taken directly to the ProjectWizard, then simply launch the Unity Editor application and immediately hold the Alt key (Mac and Windows). This can be set to the default behavior for launch in the Unity preferences. Click the Set button and choose where you would like to save your new Unity project folder on your hard drive. The new project has been named UGDE after this book, and chosen to store it on my desktop for easy access. The Project Wizard also offers the ability to import many Assetpackages into your new project which are provided free to use in your game development by Unity Technologies. Comprising scripts, ready-made objects, and other artwork, these packages are a useful way to get started in various types of new project. You can also import these packages at any time from the Assets menu within Unity, by selecting ImportPackage, and choosing from the list of available packages. You can also import a package from anywhere on your hard drive by choosing the CustomPackage option here. This import method is also used to share assets with others, and when receiving assets you have downloaded through the AssetStore—see Window | Asset Store to view this part of Unity later. From the list of packages to be imported, select the following (as shown in the previous image): Characters Cameras Effects TerrainAssets Environment When you are happy with your selection, simply choose Create Project at the bottom of this dialog window. Unity will then create your new project and you will see progress bars representing the import of the four packages. A basic prototyping environment To create a simple environment to prototype some game mechanics, we'll begin with a basic series of objects with which we can introduce gameplay that allows the player to aim and shoot at a wall of primitive cubes. When complete, your prototyping environment will feature a floor comprised of a cube primitive, a main camera through which to view the 3D world, and a point light setup to highlight the area where our gameplay will be introduced. It will look like something as shown in the following screenshot: Setting the scene As all new scenes come with a Main Camera object by default, we'll begin by adding a floor for our prototyping environment. On the Hierarchy panel, click the Create button, and from the drop-down menu, choose Cube. The items listed in this drop-down menu can also be found in the GameObject | CreateOther top menu. You will now see an object in the Hierarchy panel called Cube. Select this and press Return (Mac)/F2 (Windows) or double-click the object name slowly (both platforms) to rename this object, type in Floor and press Return (both platforms) to confirm this change. For consistency's sake, we will begin our creation at world zero—the center of the 3D environment we are working in. To ensure that the floor cube you just added is at this position, ensure it is still selected in the Hierarchypanel and then check the Transform component on the Inspector panel, ensuring that the position values for X, Y, and Z are all at 0, if not, change them all to zero either by typing them in or by clicking the cog icon to the right of the component, and selecting ResetPosition from the pop-out menu. Next, we'll turn the cube into a floor, by stretching it out in the X and Z axes. Into the X and Z values under Scale in the Transform component, type a value of 100, leaving Y at a value of 1. Adding simple lighting Now we will highlight part of our prototyping floor by adding a point light. Select the Create button on the Hierarchypanel(or go to Game Object | Create Other) and choose point light. Position the new point light at (0,20,0) using the Position values in the Transform component, so that it is 20 units above the floor. You will notice that this means that the floor is out of range of the light, so expand the range by dragging on the yellow dot handles that intersect the outline of the point light in the SceneView, until the value for range shown in the Light component in the Inspector reaches something around a value of 40, and the light is creating a lit part of the floor object. Bear in mind that most components and visual editing tools in the SceneView are inextricably linked, so altering values such asRangein the Inspector Light component will update the visual display in the SceneView as you type, and stay constant as soon as you pressReturnto confirm the values entered. Another brick in the wall Now let's make a wall of cubes that we can launch a projectile at. We'll do this by creating a single master brick, adding components as necessary, and then duplicating this until our wall is complete. Building the master brick In order to create a template for all of our bricks, we'll start by creating a master object, something to create clones of. This is done as follows: Click the Create button at the top of the Hierarchy, and select Cube. Position this at (0,1,0) using the Position values in the Transform component on the Inspector. Then, focus your view on this object by ensuring it is still selectedin the Hierarchy, by hovering your cursor over the SceneView, and pressing F. Add physics to your Cube object by choosing Component | Physics | Rigidbody from the top menu. This means that your object is now a Rigidbody—it has mass, gravity, and is affected by other objects using the physics engine for realistic reactions in the 3D world. Finally, we'll color this object by creating amaterial. Materials are a way of applying color and imagery to our 3D geometry. To make a new one, go to the Create button on the Project panel and choose Material from the drop-down menu. Press Return (Mac) or F2 (Windows) to rename this asset to Red instead of the default name New Material. You can also right-click in the Materials Project folder and Create| Material or alternatively you can use the editor main menu: Assets | Create | Material With this material selected, the Inspector shows its properties. Click on the color block to the right of MainColor [see image label 1] to open the Color Picker[see image label 2]. This will differ in appearance depending upon whether you are using Mac or Windows. Simply choose a shade of red, and then close the window. The Main Color block should now have been updated. To apply this material, drag it from the Project panel and drop it onto either the cube as seen in the SceneView, or onto the name of the object in the Hierarchy. The material is then applied to the Mesh Renderer component of this object and immediately seen following the other components of the object in the Inspector. Most importantly, your cube should now be red! Adjusting settings using the preview of this material on any object will edit the original asset, as this preview is simply a link to the asset itself, not a newly editable instance. Now that our cube has a color and physics applied through the Rigid body component, it is ready to be duplicated and act as one brick in a wall of many. However, before we do that, let’s have a quick look at the physics in action. With the cube still selected, set the Yposition value to 15 and the Xrotation value to 40 in the Transform component in the Inspector. Press Play at the top of the Unity interface and you should see the cube fall and then settle, having fallen at an angle. The shortcut for Play is Ctrl+Pfor Windows andCommand+Pfor Mac. Press Play again to stop testing. Do not press Pause as this will only temporarily halt the test, and changes made thereafter to the scene will not be saved. Set the Yposition value for the cube back to 1, and set the X Rotation back to 0. Now that we know our brick behaves correctly, let's start creating a row of bricks to form our wall. And snap!—It's a row To help you position objects, Unity allows you to snap to specific increments when dragging—these increments can be redefined by going to Edit | Snap Settings. To use snapping, hold down Command (Mac) or Ctrl (Windows) when using the Translatetool (W) to move objects in theSceneView. So in order to start building thewall, duplicate the cube brick we already have using the shortcut Command+D (Mac) or Ctrl+D (PC), then drag the red axis handle while holding the snapping key. This will snap one unit at a time by default, so snap-move your cube one unit in the X axis so that it sits next to the original cube, shown as follows: Repeat this procedure of duplication and snap-dragging until you have a row of 10 cubes in a line. This is the first row of bricks, and to simplify building the rest of the bricks we will now group this row under an empty object, and then duplicate the parent empty object. Vertex snapping The basic snapping technique used here works well as our cubes are a generic scale of 1, but when scaling more detailed shaped objects, you should use vertex snapping instead. To do this, ensure that the Translate tool is selected and hold down V on the keyboard.Now hover your cursor over a vertex point on your selected object and drag to any other vertex of another object to snap to it. Grouping and duplicating with empty objects Create an empty object by choosing GameObject | Create Empty from the top menu, then position this at (4.5,0.5,-1) using the Transform component in the Inspector. Rename this from the default nameGameObjecttoCubeHolder. Now select all of the cube objects in the Hierarchy by selecting the top one, holding the Shift key, and then selecting the last. Now drag this list of cubes in the Hierarchy onto the empty object named CubeHolder in the Hierarchy in order to make this their parent object.The Hierarchy should now look like this: You'll notice that the parent empty object now has an arrow to the left of its object title, meaning you can expand and collapse it. To save space in the Hierarchy, click the arrow now to hide all of the child objects, and then re-select the CubeHolder. Now that we have a complete row made and parented, we can simply duplicate the parent object, and use snap-dragging to lift a whole new row up in the Y axis. Use the duplicate shortcut (Command/Ctrl + D) as before, then select the Translate tool (W) and use the snap-drag technique (hold command on Mac, Ctrl on PC) outlined earlier to lift by 1 unit in the Y axis by pulling the green axis handle. Repeat this procedure to create eight rows of bricks in all, one on top of the other. It should look something like the following screenshot. Note that in the image all CubeHolderrow objects are selected in the Hierarchy. Summary In this article, you should have become familiar with the basics of using the Unity interface, working with GameObjects. Resources for Article: Further resources on this subject: Components in Unity [article] Component-based approach of Unity [article] Using Specular in Unity [article]

0
0
32538

article-image-wordpress-web-application-framework

Packt

15 Jun 2017

20 min read

WordPress as a Web Application Framework

Packt

15 Jun 2017

20 min read

0
0
42133

Packt

15 Jun 2017

6 min read

Creating Sprites

Packt

15 Jun 2017

6 min read

In this article by Nikhil Malankar, author of Learning Android Game Development, We have almost learnt everything about the basics that we need to create various components in Android and so we can now move on to do some more exciting stuff. Now at this point we will start working on a proper 2D game. It will be a small 2D side scroller game like Mario. But before we do that, lets first talk about games as a development concept. In order to understand more about games, you will need to understand a bit of Game Theory. So before we proceed with creating images and backgrounds on screen lets dive into some game theory. Here are a list of topics we will be covering in this article: Game Theory Working with colors Creating images on screen Making a continuous scrolling background (For more resources related to this topic, see here.) Lets start with the first one. Game Theory If you observe a game carefully in its source code level, you will observe that a game is just a set of illusions to create certain effects and display them on screen. Perhaps, the best example of this can be the game that we are about to develop itself. In order to make your character move ahead you can do either of the two things: Make the character move ahead Make the background move behind Illusions Either of the two things mentioned above will give you an illusion that the character is moving in a certain direction. Also, if you remember Mario properly then you will notice that the clouds and grasses are one and the same. Only their colors were changed. This was because of the memory limitations of the console platform at the time: Image Source: http://hub.packtpub.com/wp-content/uploads/2017/06/mario-clouds-is-grass.jpg Game developers use many such cheats in order to get their game running. Of course, in today's times we don't have to worry much about memory limitations because our mobile device today has the capability of the Apollo 11 rocket which landed on the Moon. Now, keep in mind the above two scenarios, we are going to use one of them in our game to make our character move. We also have to understand that every game is a loop of activities. Unlike an app, you need to draw your game resources every frame per second which will give you the illusion of moving or any effect on screen. This concept is called as Frames Per Second(FPS). Its almost similar to that of the concept of old films where a huge film used to be projected on screen by rolling per frame. Take a look at the following image to understand this concept better: Sprite Sheet of a game character As you can see above, a sprite sheet is simply an image consisting of multiple images within themselves in order to create an animation and thereby a sprite is simply an image. If we want to make our character run we will simply read the file Run_000 and play it all the way sequentially through to Run_009 which will make it appear as though the character is running. Majority of the things that you will be working with when making a game would be based on manipulating your movements. And so, you will need to be clear about your coordinates system because it will come in handy. Be it for firing a bullet out of a gun, character movement or simply turning around to look here and there. All of it is based on the simple component of movement. Game Loop In its core, every game is basically just a loop of events. It is a set up to give calls to various functions and code blocks to execute in order to have draw calls on your screen and thereby making the game playable. Mostly, your game loop comprises of 3 parts: Initialize Update Draw Initializing the game means to set an entry point to your game through which the other two parts can be called. Once your game is initialized you need to start giving calls to your events which can be managed through your update function. The draw function is responsible for drawing all your image data on screen. Everything you see on the screen including your backgrounds, images or even your GUI is the responsibility of the draw method. To say the least, your game loop is the heart of your game. This is just a basic overview of the game loop and there is much more complexity you can add to it. But for now, this much information is sufficient for you to get started. The following image perfectly illustrates what a game loop is: Image Source: https://gamedevelopment.tutsplus.com/articles/gamedev-glossary-what-is-the-game-loop--gamedev-2469 Game Design Document Also, before starting a game it is very essential to create a Game Design Document(GDD). This document serves as a groundwork for the game you will be making. 99% of times when we start making a game we lose track of the features planned for it and deviate from the core game experience. So, it is always recommended to have a GDD in place in order to keep focus. A GDD consists of the following things: Game play mechanics Story (if any) Level design Sound and music UI planning and game controls You can read more about the Game Design Document by going to the following link: https://en.wikipedia.org/wiki/Game_design_document Prototyping When making a game we need to test it simultaneously. A game is one of the most complex pieces of software and if we mess up on one part there is a chance that it might break the entire game as a whole. This process can be called as Prototyping. Making a prototype of your game is one of THE most important aspects of a game because this is where you test out the basic mechanics of your game. A prototype should be a simple working model of your game with basic functionality. It can also be termed as a stripped down version of your game. Summary Congratulations! You have successfully learnt how to create images and work with colors in Android Studio. Resources for Article: Further resources on this subject: Android Game Development with Unity3D [article] Optimizing Games for Android [article] Drawing and Drawables in Android Canvas [article]

0
0
39440

How-To Tutorials

Packt

14 Jun 2017

19 min read

IoT Analytics for the Cloud

Packt

14 Jun 2017

19 min read

In this article by Andrew Minteer, author of the book Analytics for the Internet of Things (IoT), that you understand how your data is transmitted back to the corporate servers, you feel you have more of a handle on it. You also have a reference frame in your head on how it is operating out in the real world. (For more resources related to this topic, see here.) Your boss stops by again. "Is that rolling average job done running yet?", he asks impatiently. It used to run fine and finished in an hour three months ago. It has steadily taken longer and longer and now sometimes does not even finish. Today, it has been going on six hours and you are crossing your fingers. Yesterday it crashed twice with what looked like out of memory errors. You have talked to your IT group and finance group about getting a faster server with more memory. The cost would be significant and likely will take months to complete the process of going through purchasing, putting it on order, and having it installed. Your friend in finance is hesitant to approve it. The money was not budgeted for this fiscal year. You feel bad especially since this is the only analytic job causing you problems. It just runs once a month but produces key data. Not knowing what else to say, you give your boss a hopeful, strained smile and show him your crossed fingers. “It’s still running...that’s good, right?” This article is about the advantages to cloud based infrastructure for handling and analyzing IoT data. We will discuss cloud services including Amazon Web Services (AWS), Microsoft Azure, and Thingworx. You will learn how to implement analytics elastically to enable a wide variety of capabilities. This article will cover: Building elastic analytics Designing for scale Cloud security and analytics Key Cloud Providers Amazon AWS Microsoft Azure PTC ThingWorx Building elastic analytics IoT data volumes increase quickly. Analytics for IoT is particularly compute intensive at times that are difficult to predict. Business value is uncertain and requires a lot of experimentation to find the right implementation. Combine all that together and you need something that scales quickly, is dynamic and responsive to resource needs, and virtually unlimited capacity at just the right time. And all of that needs to be implemented quickly with a low cost and low maintenance needs. Enter the cloud. IoT Analytics and cloud infrastructure fit together like a hand in a glove. What is the cloud infrastructure? The National Institute of Standards and Technology defines five essential characteristics: On-demand self-service: You can provision things like servers and storage as needed and without interacting with someone. Broad network access: Your cloud resources are accessible over the internet (if enabled) by various methods such as web browser or mobile phone. Resource pooling: Cloud providers pool their servers and storage capacity across many customers using a multi-tenant model. Resources, both physical and virtual, are dynamically assigned and reassigned as needed. Specific location of resources is unknown and generally unimportant. Rapid elasticity: Your resources can be elastically created and destroyed. This can happen automatically as needed to meet demand. You can scale outward rapidly. You can also contract rapidly. Supply of resources is effectively unlimited from your viewpoint. Measured service: Resource usage is monitored, controlled, and reported by the cloud provider. You have access to the same information, providing transparency to your utilization. Cloud systems continuously optimize resources automatically. There is a notion of private clouds that exist on premises or custom built by a third party for a specific organization. For our concerns, we will be discussing public clouds only. By and large, most analytics will be done on public clouds so we will concentrate our efforts there. The capacity available at your fingertips on public clouds is staggering. AWS, as of June 2016, has an estimated 1.3 million servers online. These servers are thought to be three times more efficient than enterprise systems. Cloud providers own the hardware and maintain the network and systems required for the available services. You just have to provision what you need to use, typically through a web application. Providers offer different levels of abstractions. They offer lower level servers and storage where you have fine grained control. They also offer managed services that handle the provisioning of servers, networking, and storage for you. These are used in conjunction with each other without much distinction between the two. Hardware failures are handled automatically. Resources are transferred to new hardware and brought back online. The physical components become unimportant when you design for the cloud, it is abstracted away and you can focus on resource needs. The advantages to using the cloud: Speed: You can bring cloud resources online in minutes. Agility: The ability to quickly create and destroy resources leads to ease of experimentation. This increases the agility of analytics organizations. Variety of services: Cloud providers have many services available to support analytics workflows that can be deployed in minutes. These services manage hardware and storage needs for you. Global reach: You can extend the reach of analytics to the other side of the world with a few clicks. Cost control: You only pay for the resources you need at the time you need them. You can do more for less. To get an idea of the power that is at your fingertips, here is an architectural diagram of something NASA built on AWS as part of an outreach program to school children. Source: Amazon Web Services; https://aws.amazon.com/lex/ By speaking voice commands, it will communicate with a Mars Rover replica to retrieve IoT data such as temperature readings. The process includes voice recognition, natural speech generation from text, data storage and processing, interaction with IoT device, networking, security, and ability to send text messages. This was not a years worth of development effort, it was built by tying together cloud based services already in place. And it is not just for big, funded government agencies like NASA. All of these services and many more are available to you today if your analytics runs in the cloud. Elastic analytics concepts What do we mean by Elastic Analytics? Let’s define it as designing your analytics processes so scale is not a concern. You want your focus to be on the analytics and not on the underlying technology. You want to avoid constraining your analytics capability so it will fit within some set hardware limitations. Focus instead on potential value versus costs. Trade hardware constraints for cost constraints. You also want your analytics to be able to scale. It should go from supporting 100 IoT devices to 1 Million IoT devices without requiring any fundamental changes. All that should happen if the costs increase. This reduces complexity and increases maintainability. That translates into lower costs which enables you to do more analytics. More analytics increases the probability of finding value. Finding more value enables even more analytics. Virtuous circle! Some core Elastic Analytics concepts: Separate compute from storage: We are used to thinking about resources like laptop specifications. You buy one device that has 16GB memory and 500GB hard drive because you think that will meet 90% of your needs and it is the top of your budget. Cloud infrastructure abstracts that away. Doing analytics in the cloud is like renting a magic laptop where you can change 4GB memory into 16GB by snapping your fingers. Your rental bill increases for only the time you have it at 16GB. You snap your fingers again and drop it back down to 4GB to save some money. Your hard drive can grow and shrink independently of the memory specification. You are not stuck having to choose a good balance between them. You can match compute needs with requirements. Build for scale from the start: Use software, services, and programming code that can scale from 1 to 1 million without changes. Each analytic process you put in production has continuing maintenance efforts to it that will build up over time as you add more and more. Make it easy on yourself later on. You do not want to have to stop what you are doing to re-architect a process you built a year ago because it hit limits of scale. Make your bottleneck wetware not hardware: By wetware, we mean brain power. “My laptop doesn’t have enough memory to run the job” should never be the problem. It should always be “I haven’t figured it out yet, but I have several possibilities in test as we speak.” Manage to a spend budget not to available hardware: Use as many cloud resources as you need as long as it fits within your spend budget. There is no need to limit analytics to fit within a set number of servers when you run analytics in the cloud. Traditional enterprise architecture purchases hardware ahead of time which incurs a capital expense. Your finance guy does not (usually) like capital expense. You should not like it either, as it means a ceiling has just been set on what you can do (at least in the near term). Managing to spend means keeping an eye on costs, not on resource limitations. Expand when needed and make sure to contract quickly to keep costs down. Experiment, experiment, experiment: Create resources, try things out, kill them off if it does not work. Then try something else. Iterate to the right answer. Scale out resources to run experiments. Stretch when you need to. Bring it back down when you are done. If Elastic Analytics is done correctly, you will find your biggest limitations are Time and Wetware. Not hardware and capital. Design with the endgame in mind Consider how the analytics you develop in the cloud would end up if successful. Would it turn into a regularly updated dashboard? Would it be something deployed to run under certain conditions to predict customer behavior? Would it periodically run against a new set of data and send an alert if an anomaly is detected? When you list out the likely outcomes, think about how easy it would be to transition from the analytics in development to the production version that will be embedded in your standard processes. Choose tools and analytics that make that transition quick and easy. Designing for scale Following some key concepts will help keep changes to your analytics processes to a minimum, as your needs scale. Decouple key components Decoupling means separating functional groups into components so they are not dependent upon each other to operate. This allows functionality to change or new functionality to be added with minimal impact on other components. Encapsulate analytics Encapsulate means grouping together similar functions and activity into distinct units. It is a core principle of object oriented programming and you should employ it in analytics as well. The goal is to reduce complexity and simplify future changes. As your analytics develop, you will have a list of actions that is either transforming the data, running it through a model or algorithm, or reacting to the result. It can get complicated quickly. By encapsulating the analytics, it is easier to know where to make changes when needed down the road. You will also be able reconfigure parts of the process without affecting the other components. Encapsulation process is carried out in the following steps: Make a list of the steps. Organize them into groups. Think which groups are likely to change together. Separate the groups that are independent into their own process It is a good idea to have the data transformation steps separate from the analytical steps if possible. Sometimes the analysis is tightly tied to the data transformation and it does not make sense to separate, but in most cases it can be separated. The action steps based on the analysis results almost always should be separate. Each group of steps will also have its own resource needs. By encapsulating them and separating the processes, you can assign resources independently and scale more efficiently where you need it. You can do more with less. Decouple with message queues Decoupling encapsulated analytics processes with message queues has several advantages. It allows for change in any process without requiring the other ones to adjust. This is because there is no direct link between them. It also builds in some robustness in case one process has a failure. The queue can continue to expand without losing data while the down process restarts and nothing will be lost after things get going again. What is a message queue? Simple diagram of a message queue New data comes into a queue as a message, it goes into line for delivery, and then is delivered to the end server when it gets its turn. The process adding a message is called the publisher and the process receiving the message is called the subscriber. The message queue exists regardless of if the publisher or subscriber is connected and online. This makes it robust against intermittent connections (intentional or unintentional). The subscriber does not have to wait until the publisher is willing to chat and vice versa. The size of the queue can also grow and shrink as needed. If the subscriber gets behind, the queue just grows to compensate until it can catch up. This can be useful if there is a sudden burst in messages by the publisher. The queue will act as a buffer and expand to capture the messages while the subscriber is working through the sudden influx. There is a limit, of course. If the queue reaches some set threshold, it will reject (and you will most likely lose) any incoming messages until the queue gets back under control. A contrived but real world example of how this can happen: Joe Cut-rate (the developer): Hey, when do you want this doo-hickey device to wake up and report? Jim Unawares (the engineer): Every 4 hours Joe Cut-rate: No sweat. I’ll program it to start at 12am UTC, then every 4 hours after. How many of these you gonna sell again? Jim Unawares: About 20 million Joe Cut-rate: Um….friggin awesome! I better hardcode that 12am UTC then, huh? 4 months later Jim Unawares: We’re only getting data from 10% of the devices. And it is never the same 10%. What the heck? Angela the analyst: Every device in the world reports at exactly the same time, first thing I checked. The message queues are filling up since our subscribers can’t process that fast, new messages are dropped. If you hard coded the report time, we’re going to have to get the checkbook out to buy a ton of bandwidth for the queues. And we need to do it NOW since we are losing 90% of the data every 4 hours. You guys didn’t do that, did you? Although queues in practice typically operate with little lag, make sure the origination time of the data is tracked and not just the time the data was pulled off the queue. It can be tempting to just capture the time the message was processed to save space but that can cause problems for your analytics. Why is this important for analytics? If you only have the date and time the message was received by the subscribing server, it may not be as close as you think to the time the message was generated at the originating device. If there are recurring problems with message queues, the spread in time difference would ebb and flow without you being aware of it. You will be using time values extensively in predictive modeling. If the time values are sometimes accurate and sometimes off, the models will have a harder time finding predictive value in your data. Your potential revenue from repurposing the data can also be affected. Customers are unlikely to pay for a service tracking event times for them if it is not always accurate. There is a simple solution. Make sure the time the device sends the data is tracked along with the time the data is received. You can monitor delivery times to diagnose issues and keep a close eye on information lag times. For example, if you notice the delivery time steadily increases just before you get a data loss, it is probably the message queue filling up. If there is no change in delivery time before a loss, it is unlikely to be the queue. Another benefit to using the cloud is (virtually) unlimited queue sizes when use a managed queue service. This makes the situation described much less likely to occur. Distributed computing Also called cluster computing, distributed computing refers to spreading processes across multiple servers using frameworks that abstract the coordination of each individual server. The frameworks make it appear as if you are using one unified system. Under the covers, it could be a few servers (called nodes) to thousands. The framework handles that orchestration for you. Avoid containing analytics to one server The advantage to this for IoT analytics is in scale. You can add resources by adding nodes to the cluster, no change to the analytics code is required. Try and avoid containing analytics to one server (with a few exceptions). This puts a ceiling on scale. When to use distributed and when to use one server There is a complexity cost to distributed computing though. It is not as simple as single server analytics. Even though the frameworks handle a lot of the complexity for you, you still have to think and design your analytics to work across multiple nodes. Some guidelines on when to keep it simple on one server: There is not much need for scale: Your analytics needs little change even if the number of IoT devices and data explodes. For example, the analytics runs a forecast on data already summarized by month. The volume of devices makes little difference in that case. Small data instead of big data: The analytics runs on a small subset of data without much impact from data size. Analytics on random samples is an example. Resource needs are minimal: Even at orders of magnitude more data, you are unlikely to need more than what is available with a standard server. In that case, keep it simple. Assuming change is constant The world of IoT analytics moves quickly. The analytics you create today will change many times over as you get feedback on results and adapt to changing business conditions. Your analytics processes will need to change. Assume this will happen continuously and design for change. That brings us to the concept of continuous delivery. Continuous delivery is a concept from software development. It automates the release of code into production. The idea is to make change a regular process. Bring this concept into your analytics by keeping a set of simultaneous copies that you use to progress through three stages: Development: Keep a copy of your analytics for improving and trying out new things. Test: When ready, merge your improvements into this copy where the functionality stays the same but it is repeatedly tested. The testing ensures it is working as intended. Keeping a separate copy for test allows development to continue on other functionality. Master: This is the copy that goes into production. When you merge things from test to the Master copy, it is the same as putting it into live use. Cloud providers often have a continuous delivery service that can make this process simpler. For any software developer readers out there, this is a simplification of the Git Flow method, which is a little outside the scope of this article. If the author can drop a suggestion, it is worth some additional research to learn Git Flow and apply it to your analytics development in the cloud. Leverage managed services Cloud infrastructure providers, like AWS and Microsoft Azure, offer services for things like message queues, big data storage, and machine learning processing. The services handle the underlying resource needs like server and storage provisioning and also network requirements. You do not have to worry about how this happens under the hood and it scales as big as you need it. They also manage global distribution of services to ensure low latency. The following image shows the AWS regional data center locations combined with the underwater internet cabling. AWS Regional Data Center Locations and Underwater Internet Cables. Source: http://turnkeylinux.github.io/aws-datacenters/ This reduces the amount of things you have to worry about for analytics. It allows you to focus more on the business application and less on the technology. That is a good thing and you should take advantage of it. An example of a managed service is Amazon Simple Queue Service (SQS). SQS is a message queue where the underlying server, storage, and compute needs is managed automatically by AWS systems. You only need to setup and configure it which takes just a few minutes. Summary In this article, we reviewed what is meant by elastic analytics and the advantages to using cloud infrastructure for IoT analytics. Designing for scale was discussed along with distributed computing. The two main cloud providers were introduced, Amazon Web Services and Microsoft Azure. We also reviewed a purpose built software platform, ThingWorx, made for IoT devices, communications, and analysis. Resources for Article: Further resources on this subject: Building Voice Technology on IoT Projects [article] IoT and Decision Science [article] Introducing IoT with Particle's Photon and Electron [article]

0
0
26641

How-To Tutorials

Packt

14 Jun 2017

8 min read

Docker Swarm

Packt

14 Jun 2017

8 min read

In this article by Russ McKendrick, the author of the book Docker Bootcamp, we will cover following topics: Creating a Swarm manually Launching a service (For more resources related to this topic, see here.) Creating a Swarm manually To start off with we need to launch the hosts and to do this, run the following commands, remembering to replace the digital ocean API access token with your own: docker-machine create --driver digitalocean --digitalocean-access-token 57e4aeaff8d7d1a8a8e46132969c2149117081536d50741191c79d8bc083ae73 swarm01 docker-machine create --driver digitalocean --digitalocean-access-token 57e4aeaff8d7d1a8a8e46132969c2149117081536d50741191c79d8bc083ae73 swarm02 docker-machine create --driver digitalocean --digitalocean-access-token 57e4aeaff8d7d1a8a8e46132969c2149117081536d50741191c79d8bc083ae73 swarm03 Once launched, running docker-machine ls should show you a list of your images. Also, this should be reflected in your digital ocean control panel: Now we have our Docker hosts and we need to assign a role to each of the nodes within the cluster. Docker Swarm has two node roles: Manager: A manager is a node which dispatches tasks to the workers, all your interaction with the Swarm cluster will be targeted against a manager node.You can have more than one manger node, however in this example we will be using just one. Worker: Worker nodes accept the tasks dispatched by the manager node, these are where all your services are launched. We will go in to services in more detail once we have our cluster configured. In our cluster, swarm01 will be the manager node with swarm02 and swarm03 being our two worker nodes. We are going to use the docker-machine ssh command to execute commands directly on our three nodes, starting with configuring our manager node. The commands in the walk through will only work with Mac and Linux, commands to run on Windows will be covered at the end of this section. Before we initialize the manager node, we need to capture the IP address of swarm01 as a command-line variable: managerIP=$(docker-machine ip swarm01) Now that we have the IP address, run the following command to check if it is correct: echo $managerIP And then to configure the manager node to run: docker-machine ssh swarm01 docker swarm init --advertise-addr $managerIP You will then receive confirmation that swarm01 is now a manager along with instructions on what to run to add a worker to the cluster: You don’t have to a make a note of the instructions as we will be running the command in a slightly different way. To add our two workers, we need to capture the join token in a similar way we captured the IP address of our manager node using the $managerIP variable, to do this run: joinToken=$(docker-machine ssh swarm01 docker swarm join-token -q worker) Again, you echo the variable out to check that it is valid: echo $joinToken Now it’s time to add our two worker nodes into the cluster by running: docker-machine ssh swarm02 docker swarm join --token $joinToken $managerIP:2377 docker-machine ssh swarm03 docker swarm join --token $joinToken $managerIP:2377 You should see something same as the following terminal output: Connecting your local Docker client to the manager node using: eval $(docker-machine env swarm01) and then running a docker-machine ls again shows. As you can see from the list of hosts,swarm01 is now active but there is nothing in the SWARM column, why is that? Confusingly, there are two different types of Docker Swarm cluster, there is the Legacy Docker Swarm which was managed by Docker machine, and then there is the new Docker Swarm mode which is managed by the Docker engine itself. We have a launched a Docker Swarm mode cluster, this is now the preferred way of launching Swarm, the legacy Docker Swarm is slowly being retired. To get a list of the nodes within our Swarm cluster we need to run the following command: For information on each node you can run the following command (the --pretty flag renders the JSON output from the Docker API): docker node inspect swarm01--pretty You are given a wealth of information about the host, including the fact that it is a manager and it has been launched in digital ocean. Running the same command, but for a worker node shows using similar information: docker node inspect swarm02 --pretty However, as the node is not a manager that section is missing. Before we look at launching services into our cluster we should look at how to launch our cluster using Docker machine on Windows as there are a few differences in the commands used due differences between powershell and bash. First, we need to launch the three hosts: docker-machine.exe create --driver digitalocean --digitalocean-access-token 57e4aeaff8d7d1a8a8e46132969c2149117081536d50741191c79d8bc083ae73 swarm01 docker-machine.exe create --driver digitalocean --digitalocean-access-token 57e4aeaff8d7d1a8a8e46132969c2149117081536d50741191c79d8bc083ae73 swarm02 docker-machine.exe create --driver digitalocean --digitalocean-access-token 57e4aeaff8d7d1a8a8e46132969c2149117081536d50741191c79d8bc083ae73 swarm03 Once the three hosts are up and running: You can create the manager node by running: $managerIP = $(docker-machine.exe ip swarm01) echo $managerIP docker-machine.exe ssh swarm01 docker swarm init --advertise-addr $managerIP Once you have your manager you can add the two worker nodes: $joinIP = “$(docker-machine.exe ip swarm01):2377” echo $joinIP $joinToken = $(docker-machine.exe ssh swarm01 docker swarm join-token -q worker) echo $joinToken docker-machine.exe ssh swarm02 docker swarm join --token $joinToken $joinIP docker-machine.exe ssh swarm03 docker swarm join --token $joinToken $joinIP and then configure your local Docker client to use your manager node and check the cluster status: docker-machine.exe env --shell powershell swarm01 | Invoke-Expression docker-machine.exe ls docker node ls At this stage, no matter which operating system you are using, you should have a three node Docker Swarm cluster in digital ocean, we can now look at a launching service into our cluster. Launching a service Rather than launching containers using the docker container run command you need to create a service A service defines a task which the manager then passes to one of the workers and then a container is launched: docker service create --name cluster -p:80:80/tcp russmckendrick/cluster That’s it, we should now have a single container running on one of our three nodes. To check that the service is running and get a little more information about the service, run the following commands: docker service ls docker service inspect cluster --pretty Now that we have confirmed that our service is running, you will be able to open your browser and enter the IP address of one of your three nodes (which you can get by running docker-machine ls).One of the features of Docker Swarm is it’s routing mesh: A routing mesh? When we exposed the port using the -p:80:80/tcp flag, we did a little more than map port 80 on the host to port 80 on the container, we actually created a Swarm load balancer on port 80 across all of the hosts within the cluster. The Swarm load balancer then directs requests to containers within our cluster. Running the commands as shown following, should show you which tasks are running on which nodes, remember tasks are containers which have been launched by the service: docker node ps swarm01 docker node ps swarm02 docker node ps swarm03 Like me, you probably have your single task running on swarm01: We can make things more interesting by scaling our service to add more tasks, to do this simply run the following commands to scale and check our service: docker service scale cluster=6 docker service ls docker service inspect cluster --pretty As you should see, we now have 6 tasks running within our cluster service. Checking the nodes should show that the tasks are evenly distributed between our three nodes: docker node ps swarm01 docker node ps swarm02 docker node ps swarm03 Hitting refresh in your browser should also update the hostname under the Docker image change, another way of seeing this on Mac and Linux is to run the following command: curl -s http://$(docker-machine ip swarm01)/ | grep class= As you can see from the terminal in following output, our requests are being load balanced between the running tasks: Before we terminate our Docker Swarm cluster let’s look at another way we can launch services, before we do we need to remove the currently running service, to do this simply run: docker service rm cluster Summary In this article we have learned how to create a Swarm manually, and how to launch a service. Resources for Article: Further resources on this subject: Orchestration with Docker Swarm [article] Hands On with Docker Swarm [article] Introduction to Docker [article]

0
0
4741

Packt

14 Jun 2017

10 min read

Configuring the ESP8266

Packt

14 Jun 2017

10 min read

In this article by Marco Schwartz the authors of the book ESP8266 Internet of Things Cookbook, we will learn following recipes: Setting up the Arduino development environment for the ESP8266 Choosing an ESP8266 Required additional components (For more resources related to this topic, see here.) Setting up the Arduino development environment for the ESP8266 To start us off, we will look at how to set up Arduino IDE development environment so that we can use it to program the ESP8266. This will involve installing the Arduino IDE and getting the board definitions for our ESP8266 module. Getting ready The first thing you should do is download the Arduino IDE if you do not already have it installed in your computer. You can do that from this link: https://www.arduino.cc/en/Main/Software. The webpage will appear as shown. It features that latest version of the Arduino IDE. Select your operating system and download the latest version that is available when you access the link (it was 1.6.13 at when this articlewas being written): When the download is complete, install the Arduino IDE and run it on your computer. Now that the installation is complete it is time to get the ESP8266 definitions. Open the preference window in the Arduino IDE from File|Preferences or by pressing CTRL+Comma. Copy this URL: http://arduino.esp8266.com/stable/package_esp8266com_index.json. Paste it in the filed labelled additional board manager URLs as shown in the figure. If you are adding other URLs too, use a comma to separate them: Open the board manager from the Tools|Board menu and install the ESP8266 platform. The board manager will download the board definition files from the link provided in the preferences window and install them. When the installation is complete the ESP8266 board definitions should appear as shown in the screenshot. Now you can select your ESP8266 board from Tools|Board menu: How it works… The Arduino IDE is an open source development environment used for programming Arduino boards and Arduino-based boards. It is also used to upload sketches to other open source boards, such as the ESP8266. This makes it an important accessory when creating Internet of Things projects. Choosing an ESP8266 board The ESP8266 module is a self-contained System On Chip (SOC) that features an integrated TCP/IP protocol stack that allows you to add Wi-Fi capability to your projects. The module is usually mounted on circuit boards that breakout the pins of the ESP8266 chip, making it easy for you program the chip and to interface with input and output devices. ESP8266 boards come in different forms depending on the company that manufactures them. All the boards use Espressif’s ESP8266 chip as the main controller, but have different additional components and different pin configurations, giving each board unique additional features. Therefore, before embarking on your IoT project, take some time to compare and contrast the different types of ESP8266 boards that are available. This way, you will be able to select the board that has features best suited for your project. Available options The simple ESP8266-01 module is the most basic ESP8266 board available in the market. It has 8 pins which include 4 General Purpose Input/Output (GPIO) pins, serial communication TX and RX pins, enable pin and power pins VCC and GND. Since it only has 4 GPIO pins, you can only connect three inputs or outputsto it. The 8-pin header on the ESP8266-01 module has a 2.0mm spacing which is not compatible with breadboards. Therefore, you have to look for another way to connect the ESP8266-01 module to your setup when prototyping. You can use female to male jumper wires to do that: The ESP8266-07 is an improved version of the ESP8266-01 module. It has 16 pins which comprise of 9 GPIO pins, serial communication TX and RX pins, a reset pin, an enable pin and power pins VCC and GND. One of the GPIO pins can be used as an analog input pin.The board also comes with a U.F.L. connector that you can use to plug an external antenna in case you need to boost Wi-Fi signal. Since the ESP8266 has more GPIO pins you can have more inputs and outputs in your project. Moreover, it supports both SPI and I2C interfaces which can come in handy if you want to use sensors or actuators that communicate using any of those protocols. Programming the board requires the use of an external FTDI breakout board based on USB to serial converters such as the FT232RL chip. The pads/pinholes of the ESP8266-07 have a 2.0mm spacing which is not breadboard friendly. To solve this, you have to acquire a plate holder that breaks out the ESP8266-07 pins to a breadboard compatible pin configuration, with 2.54mm spacing between the pins. This will make prototyping easier. This board has to be powered from a 3.3V which is the operating voltage for the ESP8266 chip: The Olimex ESP8266 module is a breadboard compatible board that features the ESP8266 chip. Just like the ESP8266-07 board, it has SPI, I2C, serial UART and GPIO interface pins. In addition to that it also comes with Secure Digital Input/Output (SDIO) interface which is ideal for communication with an SD card. This adds 6 extra pins to the configuration bringing the total to 22 pins. Since the board does not have an on-board USB to serial converter, you have to program it using an FTDI breakout board or a similar USB to serial board/cable. Moreover it has to be powered from a 3.3V source which is the recommended voltage for the ESP8266 chip: The Sparkfun ESP8266 Thing is a development board for the ESP8266 Wi-Fi SOC. It has 20 pins that are breadboard friendly, which makes prototyping easy. It features SPI, I2C, serial UART and GPIO interface pins enabling it to be interfaced with many input and output devices.There are 8 GPIO pins including the I2C interface pins. The board has a 3.3V voltage regulator which allows it to be powered from sources that provide more than 3.3V. It can be powered using a micro USB cable or Li-Po battery. The USB cable also charges the attached Li-Po battery, thanks to the Li-Po battery charging circuit on the board. Programming has to be done via an external FTDI board: The Adafruit feather Huzzah ESP8266 is a fully stand-alone ESP8266 board. It has built in USB to serial interface that eliminates the need for using an external FTDI breakout board to program it. Moreover, it has an integrated battery charging circuit that charges any connected Li-Po battery when the USB cable is connected. There is also a 3.3V voltage regulator on the board that allows the board to be powered with more than 3.3V. Though there are 28 breadboard friendly pins on the board, only 22 are useable. 10 of those pins are GPIO pins and can also be used for SPI as well as I2C interfacing. One of the GPIO pins is an analog pin: What to choose? All the ESP8266 boards will add Wi-Fi connectivity to your project. However, some of them lack important features and are difficult to work with. So, the best option would be to use the module that has the most features and is easy to work with. The Adafruit ESP8266 fits the bill. The Adafruit ESP8266 is completely stand-alone and easy to power, program and configure due to its on-board features. Moreover, it offers many input/output pins that will enable you to add more features to your projects. It is affordable andsmall enough to fit in projects with limited space. There’s more… Wi-Fi isn’t the only technology that we can use to connect out projects to the internet. There are other options such as Ethernet and 3G/LTE. There are shields and breakout boards that can be used to add these features to open source projects. You can explore these other options and see which works for you. Required additional components To demonstrate how the ESP8266 works we will use some addition components. These components will help us learn how to read sensor inputs and control actuators using the GPIO pins. Through this you can post sensor data to the internet and control actuators from the internet resources such as websites. Required components The components we will use include: Sensors DHT11 Photocell Soil humidity Actuators Relay Powerswitch tail kit Water pump Breadboard Jumper wires Micro USB cable Sensors Let us discuss the three sensors we will be using. DHT11 The DHT11 is a digital temperature and humidity sensor. It uses a thermistor and capacitive humidity sensor to monitor the humidity and temperature of the surrounding air and produces a digital signal on the data pin. A digital pin on the ESP8266 can be used to read the data from the sensor data pin: Photocell A photocell is a light sensor that changes its resistance depending on the amount of incident light it is exposed to. They can be used in a voltage divider setup to detect the amount of light in the surrounding. In a setup where the photocell is used in the Vcc side of the voltage divider, the output of the voltage divider goes high when the light is bright and low when the light is dim. The output of the voltage divider is connected to an analog input pin and the voltage readings can be read: Soil humidity sensor The soil humidity sensor is used for measuring the amount of moisture in soil and other similar materials. It has two large exposed pads that act as a variable resistor. If there is more moisture in the soil the resistance between the pads reduces, leading to higher output signal. The output signal is connected to an analog pin from where its value is read: Actuators Let’s discuss about the actuators. Relays A relay is a switch that is operated electrically. It uses electromagnetism to switch large loads using small voltages. It comprises of three parts: a coil, spring and contacts. When the coil is energized by a HIGH signal from a digital pin of the ESP8266 it attracts the contacts forcing them closed. This completes the circuit and turns on the connected load. When the signal on the digital pin goes LOW, the coil is no longer energized and the spring pulls the contacts apart. This opens the circuit and turns of the connected load: Power switch tail kit A power switch tail kit is a device that is used to control standard wall outlet devices with microcontrollers. It is already packaged to prevent you from having to mess around with high voltage wiring. Using it you can control appliances in your home using the ESP8266: Water pump A water pump is used to increase the pressure of fluids in a pipe. It uses a DC motor to rotate a fan and create a vacuum that sucks up the fluid. The sucked fluid is then forced to move by the fan, creating a vacuum again that sucks up the fluid behind it. This in effect moves the fluid from one place to another: Breadboard A breadboard is used to temporarily connect components without soldering. This makes it an ideal prototyping accessory that comes in handy when building circuits: Jumper wires Jumper wires are flexible wires that are used to connect different parts of a circuit on a breadboard: Micro USB cable A micro USB cable will be used to connect the Adafruit ESP8266 board to the compute: Summary In this article we have learned how to setting up the Arduino development environment for the ESP8266,choosing an ESP8266, and required additional components. Resources for Article: Further resources on this subject: Internet of Things with BeagleBone [article] Internet of Things Technologies [article] BLE and the Internet of Things [article]

0
0
44643

How-To Tutorials

Packt

08 Jun 2017

11 min read

Backpropagation Algorithm

Packt

08 Jun 2017

11 min read

In this article by Gianmario Spacagna, Daniel Slater, Phuong Vo.T.H, and Valentino Zocca, the authors of the book Python Deep Learning, we will learnthe Backpropagation algorithmas it is one of the most important topics for multi-layer feed-forward neural networks. (For more resources related to this topic, see here.) Propagating the error back from last to first layer, hence the name Backpropagation. Backpropagation is one of the most difficult algorithms to understand at first, but all is needed is some knowledge of basic differential calculus and the chain rule. For a deep neural network the algorithm to set the weights is called the Backpropagation algorithm. The Backpropagation algorithm We have seen how neural networks can map inputs onto determined outputs, depending on fixed weights. Once the architecture of the neural network has been defined (feed-forward, number of hidden layers, number of neurons per layer), and once the activity function for each neuron has been chosen, we will need to set the weights that in turn will define the internal states for each neuron in the network. We will see how to do that for a 1-layer network and then how to extend it to a deep feed-forward network. For a deep neural network the algorithm to set the weights is called the Backpropagation algorithm, and we will discuss and explain this algorithm for most of this section as it is one of the most important topics for multilayer feed-forward neural networks. First, however, we will quickly discuss this for 1-layer neural networks. The general concept we need to understand is the following: every neural network is an approximation of a function, therefore each neural network will not be equal to the desired function, instead it will differ by some value. This value is called the error and the aim is to minimize this error. Since the error is a function of the weights in the neural network, we want to minimize the error with respect to the weights. The error function is a function of many weights, it is therefore a function of many variables. Mathematically, the set of points where this function is zero represents therefore a hypersurface and to find a minimum on this surface we want to pick a point and then follow a curve in the direction of the minimum. Linear regression To simplify things we are going to introduce matrix notation. Let x be the input, we can think of x as a vector. In the case of linear regression we are going to consider a single output neuron y, the set of weights w is therefore a vector of dimension the same as the dimension of x. The activation value is then defined as the inner product <x, w>. Let's say that for each input value x we want to output a target value t, while for each x the neural network will output a value y defined by the activity function chosen, in this case the absolute value of the difference (y-t) represents the difference between the predicted value and the actual value for the specific input example x. If we have m input values xi, each of them will have a target value ti. In this case we calculate the error using the mean squared error , where each yi is a function of w. The error is therefore a function of w and it is usually denoted with J(w). As mentioned above, this represents a hypersurface of dimension equal to the dimension of w (we are implicitly also considering the bias), and for each wj we need to find a curve that will lead towards the minimum of the surface. The direction in which a curve increases in a certain direction is given by its derivative with respect to that direction, in this case by: And in order to move towards the minimum we need to move in the opposite direction set by for each wj. Let's calculate: If , then and therefore: The notation can sometimes be confusing, especially the first time one sees it. The input is given by vectors xi, where the superscript indicated the ith example. Since x and w are vectors, the subscript indicates the jth coordinate of the vector. yi then represents the output of the neural network given the input xi, while ti represents the target, that is, the desired value corresponding to the input xi. In order to move towards the minimum, we need to move each weight in the direction of its derivative by a small amount l, called the learning rate, typically much smaller than 1, (say 0.1 or smaller). We can therefore drop the 2 in the derivative and incorporate it in the learning rate, to get the update rule therefore given by: or, more in general, we can write the update rule in matrix form as: where ∇ represents the vector of partial derivatives. This process is what is often called gradient descent. One last note, the update can be done after having calculated all the input vectors, however, in some cases, the weights could be updated after each example or after a defined preset number of examples. Logistic regression In logistic regression, the output is not continuous; rather it is defined as a set of classes. In this case, the activation function is not going to be the identity function like before, rather we are going to use the logistic sigmoid function. The logistic sigmoid function, as we have seen before, outputs a real value in (0,1) and therefore it can be interpreted as a probability function, and that is why it can work so well in a 2-class classification problem. In this case, the target can be one of two classes, and the output represents the probability that it be one of those two classes (say t=1).Let’s denote with σ(a), with a the activation value,the logistic sigmoid function, therefore, for each examplex, the probability that the output be the class y, given the weights w, is: We can write that equation more succinctly as: and, since for each sample xi the probabilities are independent, we have that the global probability is: If we take the natural log of the above equation (to turn products into sums), we get: The object is now to maximize this log to obtain the highest probability of predicting the correct results. Usually, this is obtained, as in the previous case, by using gradient descent to minimize the cost function defined by. As before, we calculate the derivative of the cost function with respect to the weights wj to obtain: In general, in case of a multi-class output t, with t a vector (t1, …, tn), we can generalize this equation using J (w) = −log(P( y x,w))= Ei,j ti j, log ( (di)) that brings to the update equation for the weights: This is similar to the update rule we have seen for linear regression. Backpropagation In the case of 1-layer, weight-adjustment was easy, as we could use linear or logistic regression and adjust the weights simultaneously to get a smaller error (minimizing the cost function). For multi-layer neural networks we can use a similar argument for the weights used to connect the last hidden layer to the output layer, as we know what we would like the output layer to be, but we cannot do the same for the hidden layers, as, a priori, we do not know what the values for the neurons in the hidden layers ought to be. What we do, instead, is calculate the error in the last hidden layer and estimate what it would be in the previous layer, propagating the error back from last to first layer, hence the name Backpropagation. Backpropagation is one of the most difficult algorithms to understand at first, but all is needed is some knowledge of basic differential calculus and the chain rule. Let's introduce some notation first. We denote with Jthe cost (error), with y the activity function that is defined on the activation value a (for example y could be the logistic sigmoid), which is a function of the weights w and the input x. Let's also define wi,j the weight between the ith input value and the jth output. Here we define input and output more generically than for 1-layer network, if wi,j connects a pair of successive layers in a feed-forward network, we denote as input the neurons on the first of the two successive layers, and output the neurons on the second of the two successive layers. In order not to make the notation too heavy, and have to denote on which layer each neuron is, we assume that the ith input yi is always in the layer preceding the layer containing the jth output yj The letter y is used to both denote an input and the activity function, and we can easily infer which one we mean by the contest. We also use subscripts i and jwhere we always have ibelonging to the layer preceding the layer containing the element with subscript j. We also use subscripts i and j, where we always have the element with subscript i belonging to the layer preceding the layer containing the element with subscript j. In this example, layer 1 represents the input, and layer 2 the output Using this notation, and the chain-rule for derivatives, for the last layer of our neural network we can write: Since we know that , we have: If y is the logistic sigmoid defined above, we get the same result we have already calculated at the end of the previous section, since we know the cost function and we can calculate all derivatives. For the previous layers the same formula holds: Since we know that and we know that is the derivative of the activity function that we can calculate, all we need to calculate is the derivative . Let's notice that this is the derivative of the error with respect to the activation function in the second layer, and, if we can calculate this derivative for the last layer, and have a formula that allows us to calculate the derivative for one layer assuming we can calculate the derivative for the next, we can calculate all the derivatives starting from the last layer and move backwards. Let us notice that, as we defined the yj, they are the activation values for the neurons in the second layer, but they are also the activity functions, therefore functions of the activation values in the first layer. Therefore, applying the chain rule, we have: and once again we can calculate both and, so , once we knowwe can calculate, and since we can calculate for the last layer, we can move backward and calculate for any layer and therefore for any layer. Summarizing, if we have a sequence of layers where: We then have these two fundamental equations, where the summation in the second equation should read as the sum over all the outgoing connections fromyj to any neuron yk in the successive layer: By using these two equations we can calculate the derivatives for the cost with respect to each layer. If we set , represents the variation of the cost with respect to the activation value, and we can think of as the error at the neuron yj. We can then rewrite as: which implies that . These two equations give an alternate way of seeing Backpropagation, as the variation of the cost with respect to the activation value, and provide a formula to calculate this variation for any layer once we know the variation for the following layer: We can also combine these equations and show that: The Backpropagation algorithm for updating the weights is then given on each layer by: In the last section we will provide a code example that will help understand and apply these concepts and formulas. Summary At the end of this articlewe learnt the post neural networks architecture phaseand the use of the Backpropagation algorithm and we saw see how we can stack many layers to create and use deep feed-forward neural networks, and how a neural network can have many layers, and why inner (hidden) layers are important. Resources for Article: Further resources on this subject: Basics of Jupyter Notebook and Python [article] Jupyter and Python Scripting [article] Getting Started with Python Packages [article]

0
0
48284

Packt

08 Jun 2017

16 min read

Ionic Components

Packt

08 Jun 2017

16 min read

In this article by Gaurav Saini the authors of the book Hybrid Mobile Development with Ionic, we will learn following topics: Building vPlanet Commerce Ionic 2 components (For more resources related to this topic, see here.) Building vPlanet Commerce The vPlanet Commerce app is an e-commerce app which will demonstrate various Ionic components integrated inside the application and also some third party components build by the community. Let’s start by creating the application from scratch using sidemenu template: You now have the basic application ready based on sidemenu template, next immediate step I took if to take reference from ionic-conference-app for building initial components of the application such aswalkthrough. Let’s create a walkthrough component via CLI generate command: $ ionic g page walkthrough As, we get started with the walkthrough component we need to add logic to show walkthrough component only the first time when user installs the application: // src/app/app.component.ts // Check if the user has already seen the walkthrough this.storage.get('hasSeenWalkThrough').then((hasSeenWalkThrough) => { if (hasSeenWalkThrough) { this.rootPage = HomePage; } else { this.rootPage = WalkThroughPage; } this.platformReady(); }) So, we store a boolean value while checking if user has seen walkthrough first time or not. Another important thing we did create Events for login and logout, so that when user logs into the application and we can update Menu items accordingly or any other data manipulation to be done: // src/app/app.component.ts export interface PageInterface { title: string; component: any; icon: string; logsOut?: boolean; index?: number; tabComponent?: any; } export class vPlanetApp { loggedInPages: PageInterface[] = [ { title: 'account', component: AccountPage, icon: 'person' }, { title: 'logout', component: HomePage, icon: 'log-out', logsOut: true } ]; loggedOutPages: PageInterface[] = [ { title: 'login', component: LoginPage, icon: 'log-in' }, { title: 'signup', component: SignupPage, icon: 'person-add' } ]; listenToLoginEvents() { this.events.subscribe('user:login', () => { this.enableMenu(true); }); this.events.subscribe('user:logout', () => { this.enableMenu(false); }); } enableMenu(loggedIn: boolean) { this.menu.enable(loggedIn, 'loggedInMenu'); this.menu.enable(!loggedIn, 'loggedOutMenu'); } // For changing color of Active Menu isActive(page: PageInterface) { if (this.nav.getActive() && this.nav.getActive().component === page.component) { return 'primary'; } return; } } Next we have inside our app.html we have multiple <ion-menu> items depending upon whether user is loggedin or logout: // src/app/app.html <ion-menu id="loggedOutMenu" [content]="content"> <ion-header> <ion-toolbar> <ion-title>{{'menu' | translate}}</ion-title> </ion-toolbar> </ion-header> <ion-content class="outer-content"> <ion-list> <ion-list-header> {{'navigate' | translate}} </ion-list-header> <button ion-item menuClose *ngFor="let p of appPages" (click)="openPage(p)"> <ion-icon item-left [name]="p.icon" [color]="isActive(p)"></ion-icon> {{ p.title | translate }} </button> </ion-list> <ion-list> <ion-list-header> {{'account' | translate}} </ion-list-header> <button ion-item menuClose *ngFor="let p of loggedOutPages" (click)="openPage(p)"> <ion-icon item-left [name]="p.icon" [color]="isActive(p)"></ion-icon> {{ p.title | translate }} </button> <button ion-item menuClose *ngFor="let p of otherPages" (click)="openPage(p)"> <ion-icon item-left [name]="p.icon" [color]="isActive(p)"></ion-icon> {{ p.title | translate }} </button> </ion-list> </ion-content> </ion-menu>  <ion-menu id="loggedInMenu" [content]="content"> <ion-header> <ion-toolbar> <ion-title>Menu</ion-title> </ion-toolbar> </ion-header> <ion-content class="outer-content"> <ion-list> <ion-list-header> {{'navigate' | translate}} </ion-list-header> <button ion-item menuClose *ngFor="let p of appPages" (click)="openPage(p)"> <ion-icon item-left [name]="p.icon" [color]="isActive(p)"></ion-icon> {{ p.title | translate }} </button> </ion-list> <ion-list> <ion-list-header> {{'account' | translate}} </ion-list-header> <button ion-item menuClose *ngFor="let p of loggedInPages" (click)="openPage(p)"> <ion-icon item-left [name]="p.icon" [color]="isActive(p)"></ion-icon> {{ p.title | translate }} </button> <button ion-item menuClose *ngFor="let p of otherPages" (click)="openPage(p)"> <ion-icon item-left [name]="p.icon" [color]="isActive(p)"></ion-icon> {{ p.title | translate }} </button> </ion-list> </ion-content> </ion-menu> As, our app start mainly from app.html so we declare rootPage here:  <ion-nav [root]="rootPage" #content swipeBackEnabled="false"></ion-nav> Let’s now look into what all pages, services, and filter we will be having inside our app. Rather than mentioning it as a bullet list, the best way to know this is going through app.module.ts file which has all the declarations, imports, entryComponents and providers. // src/app/app.modules.ts import { NgModule, ErrorHandler } from '@angular/core'; import { IonicApp, IonicModule, IonicErrorHandler } from 'ionic-angular'; import { TranslateModule, TranslateLoader, TranslateStaticLoader } from 'ng2-translate/ng2-translate'; import { Http } from '@angular/http'; import { CloudSettings, CloudModule } from '@ionic/cloud-angular'; import { Storage } from '@ionic/storage'; import { vPlanetApp } from './app.component'; import { AboutPage } from '../pages/about/about'; import { PopoverPage } from '../pages/popover/popover'; import { AccountPage } from '../pages/account/account'; import { LoginPage } from '../pages/login/login'; import { SignupPage } from '../pages/signup/signup'; import { WalkThroughPage } from '../pages/walkthrough/walkthrough'; import { HomePage } from '../pages/home/home'; import { CategoriesPage } from '../pages/categories/categories'; import { ProductsPage } from '../pages/products/products'; import { ProductDetailPage } from '../pages/product-detail/product-detail'; import { WishlistPage } from '../pages/wishlist/wishlist'; import { ShowcartPage } from '../pages/showcart/showcart'; import { CheckoutPage } from '../pages/checkout/checkout'; import { ProductsFilterPage } from '../pages/products-filter/products-filter'; import { SupportPage } from '../pages/support/support'; import { SettingsPage } from '../pages/settings/settings'; import { SearchPage } from '../pages/search/search'; import { UserService } from '../providers/user-service'; import { DataService } from '../providers/data-service'; import { OrdinalPipe } from '../filters/ordinal'; // 3rd party modules import { Ionic2RatingModule } from 'ionic2-rating'; export function createTranslateLoader(http: Http) { return new TranslateStaticLoader(http, './assets/i18n', '.json'); } // Configure database priority export function provideStorage() { return new Storage(['sqlite', 'indexeddb', 'localstorage'], { name: 'vplanet' }) } const cloudSettings: CloudSettings = { 'core': { 'app_id': 'f8fec798' } }; @NgModule({ declarations: [ vPlanetApp, AboutPage, AccountPage, LoginPage, PopoverPage, SignupPage, WalkThroughPage, HomePage, CategoriesPage, ProductsPage, ProductsFilterPage, ProductDetailPage, SearchPage, WishlistPage, ShowcartPage, CheckoutPage, SettingsPage, SupportPage, OrdinalPipe, ], imports: [ IonicModule.forRoot(vPlanetApp), Ionic2RatingModule, TranslateModule.forRoot({ provide: TranslateLoader, useFactory: createTranslateLoader, deps: [Http] }), CloudModule.forRoot(cloudSettings) ], bootstrap: [IonicApp], entryComponents: [ vPlanetApp, AboutPage, AccountPage, LoginPage, PopoverPage, SignupPage, WalkThroughPage, HomePage, CategoriesPage, ProductsPage, ProductsFilterPage, ProductDetailPage, SearchPage, WishlistPage, ShowcartPage, CheckoutPage, SettingsPage, SupportPage ], providers: [ {provide: ErrorHandler, useClass: IonicErrorHandler}, { provide: Storage, useFactory: provideStorage }, UserService, DataService ] }) export class AppModule {} Ionic components There are many Ionic JavaScript components which we can effectively use while building our application. What's best is to look around for features we will be needing in our application. Let’s get started with Home page of our e-commerce application which will be having a image slider having banners on it. Slides Slides component is multi-section container which can be used in multiple scenarios same astutorial view or banner slider. <ion-slides> component have multiple <ion-slide> elements which can be dragged or swipped left/right. Slides have multiple configuration options available which can be passed in the ion-slides such as autoplay, pager, direction: vertical/horizontal, initialSlide and speed. Using slides is really simple as we just have to include it inside our home.html, no dependency is required for this to be included in the home.ts file: <ion-slides pager #adSlider (ionSlideDidChange)="logLenth()" style="height: 250px"> <ion-slide *ngFor="let banner of banners"> <img [src]="banner"> </ion-slide> </ion-slides> // Defining banners image path export class HomePage { products: any; banners: String[]; constructor() { this.banners = [ 'assets/img/banner-1.webp', 'assets/img/banner-2.webp', 'assets/img/banner-3.webp' ] } } Lists Lists are one of the most used components in many applications. Inside lists we can display rows of information. We will be using lists multiple times inside our application such ason categories page where we are showing multiple sub-categories: // src/pages/categories/categories.html <ion-content class="categories"> <ion-list-header *ngIf="!categoryList">Fetching Categories ....</ion-list-header> <ion-list *ngFor="let cat of categoryList"> <ion-list-header>{{cat.name}}</ion-list-header> <ion-item *ngFor="let subCat of cat.child"> <ion-avatar item-left> <img [src]="subCat.image"> </ion-avatar> <h2>{{subCat.name}}</h2> <p>{{subCat.description}}</p> <button ion-button clear item-right (click)="goToProducts(subCat.id)">View</button> </ion-item> </ion-list> </ion-content> Loading and toast Loading component can be used to indicate some activity while blocking any user interactions. One of the most common cases of using loading component is HTTP/ calls to the server, as we know it takes time to fetch data from server, till then for good user experience we can show some content showing Loading .. or Login wait .. for login pages. Toast is a small pop-up which provides feedback, usually used when some action is performed by the user. Ionic 2 now provides toast component as part of its library, previously we have to use native Cordova plugin for toasts which in either case can now be used also. Loading and toast component both have a method create. We have to provide options while creating these components: // src/pages/login/login.ts import { Component } from '@angular/core'; import { NgForm } from '@angular/forms'; import { NavController, LoadingController, ToastController, Events } from 'ionic-angular'; import { SignupPage } from '../signup/signup'; import { HomePage } from '../home/home'; import { Auth, IDetailedError } from '@ionic/cloud-angular'; import { UserService } from '../../providers/user-service'; @Component({ selector: 'page-user', templateUrl: 'login.html' }) export class LoginPage { login: {email?: string, password?: string} = {}; submitted = false; constructor(public navCtrl: NavController, public loadingCtrl: LoadingController, public auth: Auth, public userService: UserService, public toastCtrl: ToastController, public events: Events) { } onLogin(form: NgForm) { this.submitted = true; if (form.valid) { // start Loader let loading = this.loadingCtrl.create({ content: "Login wait...", duration: 20 }); loading.present(); this.auth.login('basic', this.login).then((result) => { // user is now registered this.navCtrl.setRoot(HomePage); this.events.publish('user:login'); loading.dismiss(); this.showToast(undefined); }, (err: IDetailedError<string[]>) => { console.log(err); loading.dismiss(); this.showToast(err) }); } } showToast(response_message:any) { let toast = this.toastCtrl.create({ message: (response_message ? response_message : "Log In Successfully"), duration: 1500 }); toast.present(); } onSignup() { this.navCtrl.push(SignupPage); } } As, you can see from the previouscode creating a loader and toast is almost similar at code level. The options provided while creating are also similar, we have used loader here while login and toast after that to show the desired message. Setting duration option is good to use, as in case loader is dismissed or not handled properly in code then we will block the user for any further interactions on app. In HTTP calls to server we might get connection issues or failure cases, in that scenario it may end up blocking users. Tabs versussegments Tabs are easiest way to switch between views and organise content at higher level. On the other hand segment is a group of button and can be treated as a local switch tabs inside a particular component mainly used as a filter. With tabs we can build quick access bar in the footer where we can place Menu options such as Home, Favorites, and Cart. This way we can have one click access to these pages or components. On the other hand we can use segments inside the Account component and divide the data displayed in three segments profile, orders and wallet: // src/pages/account/account.html <ion-header> <ion-navbar> <button menuToggle> <ion-icon name="menu"></ion-icon> </button> <ion-title>Account</ion-title> </ion-navbar> <ion-toolbar [color]="isAndroid ? 'primary' : 'light'" no-border-top> <ion-segment [(ngModel)]="account" [color]="isAndroid ? 'light' : 'primary'"> <ion-segment-button value="profile"> Profile </ion-segment-button> <ion-segment-button value="orders"> Orders </ion-segment-button> <ion-segment-button value="wallet"> Wallet </ion-segment-button> </ion-segment> </ion-toolbar> </ion-header> <ion-content class="outer-content"> <div [ngSwitch]="account"> <div padding-top text-center *ngSwitchCase="'profile'" > <img src="http://www.gravatar.com/avatar?d=mm&s=140"> <h2>{{username}}</h2> <ion-list inset> <button ion-item (click)="updatePicture()">Update Picture</button> <button ion-item (click)="changePassword()">Change Password</button> <button ion-item (click)="logout()">Logout</button> </ion-list> </div> <div padding-top text-center *ngSwitchCase="'orders'" > // Order List data to be shown here </div> <div padding-top text-center *ngSwitchCase="'wallet'"> // Wallet statement and transaction here. </div> </div> </ion-content> This is how we define a segment in Ionic, we don’t need to define anything inside the typescript file for this component. On the other hand with tabs we have to assign a component for each tab and also can access its methods via Tab instance. Just to mention, we haven’t used tabs inside our e-commerce application as we are using side menu. One good example will be to look in ionic-conference-app (https://github.com/driftyco/ionic-conference-app) you will find sidemenu and tabs both in single application: / // We currently don’t have Tabs component inside our e-commerce application // Below is sample code about how we can integrate it. <ion-tabs #showTabs tabsPlacement="top" tabsLayout="icon-top" color="primary"> <ion-tab [root]="Home"></ion-tab> <ion-tab [root]="Wishlist"></ion-tab> <ion-tab [root]="Cart"></ion-tab> </ion-tabs> import { HomePage } from '../pages/home/home'; import { WishlistPage } from '../pages/wishlist/wishlist'; import { ShowcartPage } from '../pages/showcart/showcart'; export class TabsPage { @ViewChild('showTabs') tabRef: Tabs; // this tells the tabs component which Pages // should be each tab's root Page Home = HomePage; Wishlist = WishlistPage; Cart = ShowcartPage; constructor() { } // We can access multiple methods via Tabs instance // select(TabOrIndex), previousTab(trimHistory), getByIndex(index) // Here we will console the currently selected Tab. ionViewDidEnter() { console.log(this.tabRef.getSelected()); } } Properties can be checked in the documentation (https://ionicframework.com/docs/v2/api/components/tabs/Tabs/) as, there are many properties available for tabs, like mode, color, tabsPlacement and tabsLayout. Similarly we can configure some tabs properties at Config level also, you will find here what all properties you can configure globally or for specific platform. (https://ionicframework.com/docs/v2/api/config/Config/). Alerts Alerts are the components provided in Ionic for showing trigger alert, confirm, prompts or some specific actions. AlertController can be imported from ionic-angular which allow us to programmatically create and show alerts inside the application. One thing to note here is these are JavaScript pop-up not the native platform pop-up. There is a Cordova plugin cordova-plugin-dialogs (https://ionicframework.com/docs/v2/native/dialogs/) which you can use if native dialog UI elements are required. Currently five types of alerts we can show in Ionic app basic alert, prompt alert, confirmation alert, radio and checkbox alerts: // A radio alert inside src/pages/products/products.html for sorting products <ion-buttons> <button ion-button full clear (click)="sortBy()"> <ion-icon name="menu"></ion-icon>Sort </button> </ion-buttons> // onClick we call sortBy method // src/pages/products/products.ts import { NavController, PopoverController, ModalController, AlertController } from 'ionic-angular'; export class ProductsPage { constructor( public alertCtrl: AlertController ) { sortBy() { let alert = this.alertCtrl.create(); alert.setTitle('Sort Options'); alert.addInput({ type: 'radio', label: 'Relevance', value: 'relevance', checked: true }); alert.addInput({ type: 'radio', label: 'Popularity', value: 'popular' }); alert.addInput({ type: 'radio', label: 'Low to High', value: 'lth' }); alert.addInput({ type: 'radio', label: 'High to Low', value: 'htl' }); alert.addInput({ type: 'radio', label: 'Newest First', value: 'newest' }); alert.addButton('Cancel'); alert.addButton({ text: 'OK', handler: data => { console.log(data); // Here we can call server APIs with sorted data // using the data which user applied. } }); alert.present().then(() => { // Here we place any function that // need to be called as the alert in opened. }); } } Cancel and OK buttons. We have used this here for sorting the products according to relevance price or other sorting values. We can prepare custom alerts also, where we can mention multiple options. Same as in previous example we have five radio options, similarly we can even add a text input box for taking some inputs and submit it. Other than this, while creating alerts remember that there are alert, input and button options properties for all the alerts present in the AlertController component.(https://ionicframework.com/docs/v2/api/components/alert/AlertController/). Some alert options: title:// string: Title of the alert. subTitle:// string(optional): Sub-title of the popup. Message:// string: Message for the alert cssClass:// string: Custom CSS class name inputs:// array: Set of inputs for alert. Buttons:// array(optional): Array of buttons Cards and badges Cards are one of the important component used more often in mobile and web applications. The reason behind cards are so popular because its a great way to organize information and get the users access to quantity of information on smaller screens also. Cards are really flexible and responsive due to all these reasons they are adopted very quickly by developers and companies. We will also be using cards inside our application on home page itself for showing popular products. Let’s see what all different types of cards Ionic provides in its library: Basic cards Cards with header and Footer Cards lists Cards images Background cards Social and map cards Social and map cards are advanced cards, which is build with custom CSS. We can develop similar advance card also. // src/pages/home/home.html <ion-card> <img [src]="prdt.imageUrl"/> <ion-card-content> <ion-card-title no-padding> {{prdt.productName}} </ion-card-title> <ion-row no-padding class="center"> <ion-col> <b>{{prdt.price | currency }}   </b><span class="dis count">{{prdt.listPrice | currency}}</span> </ion-col> </ion-row> </ion-card-content> </ion-card> We have used here image card with a image on top and below we have favorite and view button icons. Similarly, we can use different types of cards where ever its required. Also, at the same time we can customize our cards and mix two types of card using their specific CSS classes or elements. Badges are small component used to show small information, for example showing number of items in cart above the cart icon. We have used it in our e-commerce application for showing the ratings of product. <ion-badge width="25">4.1</ion-badge> Summary In this article we have learned, building vPlanet Commerce and Ionic components. Resources for Article: Further resources on this subject: Lync 2013 Hybrid and Lync Online [article] Optimizing JavaScript for iOS Hybrid Apps [article] Creating Mobile Dashboards [article]

0
0
36198

article-image-erasure-coding-cold-storage

Packt

07 Jun 2017

20 min read

Erasure coding for cold storage

Packt

07 Jun 2017

20 min read

0
0
17450

article-image-microsoft-azure-stack-architecture

Packt

07 Jun 2017

13 min read

The Microsoft Azure Stack Architecture

Packt

07 Jun 2017

13 min read

In this article by Markus Klein and Susan Roesner, authors of the book Azure Stack for Datacenters, we will help you to plan, build, run, and develop your own Azure-based datacenter running Azure Stack technology. The goal is that the technology in your datacenter will be a 100 percent consistent using Azure, which provides flexibility and elasticity to your IT infrastructure. We will learn about: Cloud basics The Microsoft Azure Stack Core management services Using Azure Stack Migrating services to Azure Stack (For more resources related to this topic, see here.) Cloud as the new IT infrastructure Regarding the technical requirements of today's IT, the cloud is always a part of the general IT strategy. It does not depend upon the region in which the company is working in, nor does it depend upon the part of the economy—99.9 percent of all companies have cloud technology already in their environment. The good question for a lot of CIOs is general: "To what extent do we allow cloud services, and what does that mean to our infrastructure?" So it's a matter of compliance, allowance, and willingness. The top 10most important questions for a CIO to prepare for the cloud are as follows: Are we allowed to save our data in the cloud? What classification of data can be saved in the cloud? How flexible are we regarding the cloud? Do we have the knowledge to work with cloud technology? How does our current IT setup and infrastructure fit into the cloud's requirements? Is our current infrastructure already prepared for the cloud? Are we already working with a cloud-ready infrastructure? Is our Internet bandwidth good enough? What does the cloud mean to my employees? Which technology should we choose? Cloud Terminology The definition of the term "cloud" is not simple, but we need to differentiate between the following: Private cloud: This is a highly dynamic IT infrastructure based on virtualization technology that is flexible and scalable. The resources are saved in a privately owned datacenter either in your company or a service provider of your choice. Public cloud: This is a shared offering of IT infrastructure services that are provided via the Internet. Hybrid cloud: This is a mixture of a private and public cloud. Depending on compliance or other security regulations, the services that could be run in a public datacenter are already deployed there, but the services that need to be stored inside the company are running there. The goal is to run these services on the same technology to provide the agility, flexibility, and scalability to move services between public and private datacenters. In general, there are some big players within the cloud market (for example, Amazon Web Services, Google, Azure, and even Alibaba). If a company is quite Microsoft-minded from the infrastructure point of view, they should have a look at Microsoft Azure datacenters. Microsoft started in 2008 with their first datacenter, and today, they invest a billion dollars every month in Azure. As of today, there are about 34 official datacenters around the world that form Microsoft Azure, besides some that Microsoft does not talk about (for example, USGovernment Azure). There are some dedicated datacenters, such as the German Azure cloud, that do not have connectivity to Azure worldwide. Due to compliance requirements, these frontiers need to exist, but the technology of each Azure datacenter is the same although the services offered may vary. The following map gives an overview of the locations (so-called regions) in Azure as of today and provide an idea of which ones will be coming soon: The Microsoft cloud story When Microsoft started their public cloud, they decided that there must be a private cloud stack too, especially, to prepare their infrastructure to run in Azure sometime in the future. The first private cloud solution was the System Center suite, with System Center Orchestrator and Service Provider Foundation (SPF) and Service Manager as the self-service portal solution. Later on, Microsoft launched Windows Azure Pack for Windows Server. Today, Windows Azure Pack is available as a product focused on the private cloud and provides a self-service portal (the well-known old Azure Portal, code name red dog frontend), and it uses the System Center suite as its underlying technology: Microsoft Azure Stack In May2015, Microsoft formally announced a new solution that brings Azure to your datacenter. This solution was named Microsoft Azure Stack. To put it in one sentence: Azure Stack is the same technology with the same APIs and portal as Public Azure, but you could run it in your datacenter or in that of your service provider. With Azure Stack, System Center is completely gone because everything is the way it is in Azure now, and in Azure, there is no System Center at all. This is what the primary focus of this article is. The following diagram gives a current overview of the technical design of Azure Stack compared with Azure: The one and only difference between Azure Stack and Azure is the cloud infrastructure. In Azure, there are thousands of servers that are part of the solution; with Azure Stack, the number is slightly smaller. That's why there is the cloud-inspired infrastructure based on Windows Server, Hyper-V, and Azure technologies as the underlying technology stack. There is no System Center product in this stack anymore. This does not mean that it cannot be there (for example, SCOM for on-premise monitoring), but Azure Stack itself provides all functionality with the solution itself. For stability and functionality, Microsoft decided to provide Azure Stack as a so-called integrated system, so it will come to your door with the hardware stack included. The customer buys Azure Stack as a complete technology stack. At general availability(GA), the hardware OEMs are HPE, DellEMC, and Lenovo. In addition to this, there will be a one-host PoC deployment available for download that could be run as a proof of concept solution on every type of hardware, as soon as it meets the hardware requirements. Technical design Looking at the technical design a bit more in depth, there are some components that we need to dive deeper into: The general basis of Azure Stack is Windows Server 2016 technology, which builds the cloud-inspired infrastructure: Storage Spaces Direct (S2D) VXLAN Nano Server Azure Resource Manager (ARM) Storage Spaces Direct (S2D) Storage Spaces and Scale-Out File Server were technologies that came with Windows Server 2012. The lack of stability in the initial versions and the issues with the underlying hardware was a bad phase. The general concept was a shared storage setup using JBODs controlled from Windows Server 2012 Storage Spaces servers and a magic Scale-Out File Server cluster that acted as the single point of contact for storage: With Windows Server 2016, the design is quite different and the concept relies on a shared-nothing model, even with local attached storage: This is the storage design Azure Stack is coming up with as one of its main pillars. VXLANnetworking technology With Windows Server 2012, Microsoft introduced Software-Defined Networking(SDN)and the NVGRE technology. Hyper-V Network Virtualization supports Network Virtualization using Generic Routing Encapsulation (NVGRE) as the mechanism to virtualize IP addresses. In NVGRE, the virtual machine's packet is encapsulated inside another packet: Hyper-V Network Virtualization supports NVGRE as the mechanism to virtualize IP addresses. In NVGRE, the virtual machine's packet is encapsulated inside another packet. VXLAN comes as the new SDNv2protocol, is RFC compliant, and is supported by most network hardware vendors by default. The Virtual eXtensible Local Area Network (VXLAN) RFC 7348 protocol has been widely adopted in the marketplace, with support from vendors such as Cisco, Brocade, Arista, Dell, and HP. The VXLAN protocol uses UDP as the transport: Nano Server Nano Server offers a minimal-footprint headless version of Windows Server 2016. It completely excludes the graphical user interface, which means that it is quite small, headless, and easy to handle regarding updates and security fixes, but it doesn't provide the GUI expected by customers of Windows Server. Azure Resource Manager (ARM) The “magical” Azure Resource Manager is a 1-1 bit share with ARM from Azure, so it has the same update frequency and features that are available in Azure, too. ARM is a consistent management layer that saves resources, dependencies, inputs, and outputs as an idempotent deployment as a JSON file called an ARM template. This template defines the tastes of a deployment, whether it be VMs, databases, websites, or anything else. The goal is that once a template is designed, it can be run on each Azure-based cloud platform, including Azure Stack. ARM provides cloud consistency with the finest granularity, and the only difference between the clouds is the region the template is being deployed to and the corresponding REST endpoints. ARM not only provides a template for a logical combination of resources within Azure, it manages subscriptions and role-based access control(RBAC) and defines the gallery, metric, and usage data, too. This means quite simply that everything that needs to be done with Azure resources should be done with ARM. Not only does Azure Resource Manager design one virtual machine, it is responsible for setting up one to a bunch of resources that fit together for a specific service. Even ARM templates can be nested; this means they can depend on each other. When working with ARM, you should know the following vocabulary: Resource: Are source is a manageable item available in Azure Resource group: A resource group is the container of resources that fit together within a service Resource provider: A resource provider is a service that can be consumed within Azure Resource manager template: A resource manager template is the definition of a specific service Declarative syntax: Declarative syntax means that the template does not define the way to set up a resource; it just defines how the result and the resource itself has the feature to set up and configure itself to fulfill the syntax to create your own ARM templates, you need to fulfill the following minimum requirements: A test editor of your choice Visual Studio Community Edition Azure SDK Visual Studio Community Edition is available for free from the Internet. After setting these things up, you could start it and define your own templates: Setting up a simple blank template looks like this: There are different ways to get a template so that you can work on and modify it to fit your needs: Visual Studio templates Quick-start templates on GitHub Azure ARM templates You could export the ARM template directly from Azure Portal if the resource has been deployed: After clicking on View template, the following opens up: For further reading on ARM basics, the Getting started with Azure Resource Managerdocument is a good place to begin: http://aka.ms/GettingStartedWithARM. PowerShell Desired State Configuration We talked about ARM and ARM templates that define resources, but they are unable to design the waya VM looks inside, specify which software needs to be installed, and how the deployment should be done. This is why we need to have a look at VMextensions.VMextensions define what should be done after ARM deployment has finished. In general, the extension could be anything that's a script. The best practice is to use PowerShell and its add-on called Desired State Configuration (DSC). DSC defines—quite similarly to ARM—how the software needs to be installed and configured. The great concept is that it also monitors whether the desired state of a virtual machine is changing (for example, because an administrator uninstalls or reconfigures a machine). If it does, it makes sure within minutes whether the original state will be fulfilled again and rolls back the actions to the desired state: Migrating services to Azure Stack If you are running virtual machines today, you're already using a cloud-based technology, although we do not call it cloud today. Basically, this is the idea of a private cloud. If you are running Azure Pack today, you are quite near Azure Stack from the processes point of view but not the technology part. There is a solution called connectors for Azure Pack that lets you have one portal UI for both cloud solutions. This means that the customer can manage everything out of the Azure Stack Portal, although services run in Azure Pack as a legacy solution. Basically, there is no real migration path within Azure Stack. But the way to solve this is quite easy, because you could use every tool that you can use to migrate services to Azure. Azure website migration assistant The Azure website migration assistant will provide a high-level readiness assessment for existing websites. This report outlines sites that are ready to move and elements that may need changes, and it highlights unsupported features. If everything is prepared properly, the tool creates any website and associated database automatically and synchronizes the content. You can learn more about it at https://azure.microsoft.com/en-us/downloads/migration-assistant/: For virtual machines, there are two tools available: Virtual Machines Readiness Assessment Virtual Machines Optimization Assessment Virtual Machines Readiness Assessment The Virtual Machines Readiness Assessment tool will automatically inspect your environment and provide you with a checklist and detailed report on steps for migrating the environment to the cloud. The download location is https://azure.microsoft.com/en-us/downloads/vm-readiness-assessment/. If you run the tool, you will get an output like this: Virtual Machines Optimization Assessment The Virtual Machine Optimization Assessment tool will at first start with a questionnaire and ask several questions about your deployment. Then, it will create an automated data collection and analysis of your Azure VMs. It generates a custom report with tenprioritized recommendations across six focus areas. These areas are security and compliance, performance and scalability, and availability and business continuity. The download location ishttps://azure.microsoft.com/en-us/downloads/vm-optimization-assessment/. Summary Azure Stack provides a real Azure experience in your datacenter. The UI, administrative tools, and even third-party solutions should work properly. The design of Azure Stack is a very small instance of Azure with some technical design modifications, especially regarding the compute, storage, and network resource providers. These modifications give you a means to start small, think big, and deploy large when migrating services directly to public Azure sometime in the future, if needed. The most important tool for planning, describing, defining, and deploying Azure Stack services is Azure Resource Manager, just like in Azure. This provides you a way to create your services just once but deploy them many times. From the business perspective, this means you have better TCO and lower administrative costs. Resources for Article: Further resources on this subject: Deploying and Synchronizing Azure Active Directory [article] What is Azure API Management? [article] Installing and Configuring Windows Azure Pack [article]

0
0
29512

article-image-introduction-sdn-transformation-legacy-sdn

Packt

07 Jun 2017

23 min read

Introduction to SDN - Transformation from legacy to SDN

Packt

07 Jun 2017

23 min read

In this article, by Reza Toghraee, the author of the book, Learning OpenDayLight, we will: What is and what is not SDN Components of a SDN Difference between SDN and overlay The SDN controllers (For more resources related to this topic, see here.) You might have heard about Software-Defined Networking (SDN). If you are in networking industry this is a topic which probably you have studied initially when first time you heard about the SDN.To understand the importance of SDN and SDN controller, let's look at Google. Google silently built its own networking switches and controller called Jupiter. A home grown project which is mostly software driven and supports such massive scale of Google. The SDN base is There is a controller who knows the whole network. OpenDaylight (ODL), is a SDN controller. In other words, it's the central brain of the network. Why we are going towards SDN Everyone who is hearing about SDN, should ask this question that why are we talking about SDN. What problem is it trying to solve? If we look at traditional networking (layer 2, layer 3 with routing protocols such as BGP, OSPF) we are completely dominated by what is so called protocols. These protocols in fact have been very helpful to the industry. They are mostly standard. Different vendor and products can communicate using standard protocols with each other. A Cisco router can establish a BGP session with a Huawei switch or an open source Quagga router can exchange OSPF routes with a Juniper firewall. Routing protocol is a constant standard with solid bases. If you need to override something in your network routing, you have to find a trick to use protocols, even by using a static route. SDN can help us to come out of routing protocol cage, look at different ways to forward traffic. SDN can directly program each switch or even override a route which is installed by routing protocol. There are high-level benefits of using SDN which we explain few of them as follows: An integrated network: We used to have a standalone concept in traditional network. Each switch was managed separately, each switch was running its own routing protocol instance and was processing routing information messages from other neighbors. In SDN, we are migrating to a centralized model, where the SDN controller becomes the single point of configuration of the network, where you will apply the policies and configuration. Scalable layer 2 across layer 3:Having a layer 2 network across multiple layer 3 network is something which all network architects are interested and till date we have been using proprietary methods such as OTV or by using a service provider VPLS service. With SDN, we can create layer 2 networks across multiple switches or layer 3 domains (using VXLAN) and expand the layer 2 networks. In many cloud environment, where the virtual machines are distributed across different hosts in different datacenters, this is a major requirement. Third-party application programmability: This is a very generic term, isn't it? But what I'm referring to is to let other applications communicate with your network. For example,In many new distributed IP storage systems, the IP storage controller has ability to talk to network to provide the best, shortest path to the storage node. With SDN we are letting other applications to control the network. Of course this control has limitation and SDN doesn't allow an application to scrap the whole network. Flexible application based network:In SDN, everything is an application. L2/L3, BGP, VMware Integration, and so on all are applications running in SDN controller. Service chaining:On the fly you add a firewall in the path or a load balancer. This is service insertion. Unified wired and wireless: This is an ideal benefit, to have a controller which supports both wired and wireless network. OpenDaylight is the only controller which supports CAPWAP protocols which allows integration with wireless access points. Components of a SDN A software defined network infrastructure has two main key components: The SDN Controller (only one, could be deployed in a highly available cluster) The SDN enabled switches(multiple switches, mostly in a Clos topology in a datacenter): SDN controller is the single brain of the SDN domain. In fact, an SDN domain is very similar to a chassis based switch. You can imagine supervisor or management module of a chassis based switch as a SDN controller and rest of the line card and I/O cards as SDN switches. The main difference between a SDN network and a chassis based switch is that you can scale out the SDN with multiple switches, where in a chassis based switch you are limited to the number of slots in that chassis: Controlling the fabric It is very important that you understand the main technologies involved in SDN. These methods are used by SDN controllers to manage and control the SDN network. In general, there are two methods available for controlling the fabric: Direct fabric programming: In this method, SDN controller directly communicates with SDN enabled switches via southbound protocols such as OpenFlow, NETCONF and OVSDB. SDN controller programs each switch member with related information about fabric, and how to forward the traffic. Direct fabric programming is the method used by OpenDaylight. Overlay:In overlay method, SDN controller doesn't rely on programing the network switches and routers. Instead it builds an virtual overlay network on top of existing underlay network. The underlay network can be a L2 or L3 network with traditional network switches and router, just providing IP connectivity. SDN controller uses this platform to build the overlay using encapsulation protocols such as VXLAN and NVGRE. VMware NSX uses overlay technology to build and control the virtual fabric. SDN controllers One of the key fundamentals of SDN is disaggregation. Disaggregation of software and hardware in a network and also disaggregation of control and forwarding planes. SDN controller is the main brain and controller of an SDN environment, it's a strategic control point within the network and responsible for communicating information to: Routers and switches and other network devices behind them. SDN controllers uses APIs or protocols (such as OpenFlow or NETCONF) to communicate with these devices. This communication is known as southbound Upstream switches, routers or applications and the aforementioned business logic (via APIs or protocols). This communication is known as northbound. An example for a northbound communication is a BGP session between a legacy router and SDN controller. If you are familiar with chassis based switches like Cisco Catalyst 6500 or Nexus 7k chassis, you can imagine a SDN network as a chassis, with switches and routers as its I/O line cards and SDN controller as its supervisor or management module. Infact SDN is similar to a very scalable chassis where you don't have any limitation on number of physical slots. SDN controller is similar to role of management module of a chassis based switch and it controls all switches via its southbound protocols and APIs. The following table compares the SDN controller and a chassis based switch: SDN Controller Chassis based switch Supports any switch hardware Supports only specific switch line cards Can scale out, unlimited number of switches Limited to number of physical slots in the chassis Supports high redundancy by multiple controllers in a cluster Supports dual management redundancy, active standby Communicates with switches via southbound protocols such as OpenFlow, NETCONF, BGP PCEP Use proprietary protocols between management module and line cards Communicates with routers, switches and applications outside of SDN via northbound protocols such as BGP, OSPF and direct API Communicates with other routers and switches outside of chassis via standard protocols such as BGP, OSPF or APIs. The first protocol that popularized the concept behind SDN was OpenFlow. When conceptualized by networking researchers at Stanford back in 2008, it was meant to manipulate the data plane to optimize traffic flows and make adjustments, so the network could quickly adapt to changing requirements. Version 1.0 of the OpenFlow specification was released in December of 2009; it continues to be enhanced under the management of the Open Networking Foundation, which is a user-led organization focused on advancing the development of open standards and the adoption of SDN technologies. OpenFlow protocol was the first protocol that helped in popularizing SDN. OpenFlow is a protocol designed to update the flow tables in a switch. Allowing the SDN controller to access the forwarding table of each member switch or in other words to connect control plane and data plane in SDN world. Back in 2008, OpenFlow conceptualized by networking researchers at Stanford University, the initial use of OpenFlow was to alter the switch forwarding tables to optimize traffic flows and make adjustments, so the network could quickly adapt to changing requirements. After introduction of OpenFlow, NOX introduced as original OpenFlow controller (still there wasn't concept of SDN controller). NOX was providing a high-level API capable of managing and also developing network control applications. Separate applications were required to run on top of NOX to manage the network.NOX was initially developed by Nicira networks (which acquired by VMware, and finally became part of VMware NSX). NOX introduced along with OpenFlow in 2009. NOX was a closed source product but ultimately it was donated to SDN community which led to multiple forks and sub projects out of original NOX. For example, POX is a sub project of NOX which provides Python support. Both NOX and POX were early controllers. NOX appears an inactive development, however POX is still in use by the research community as it is a Python based project and can be easily deployed. POX is hosted at http://github.com/noxrepo/pox NOX apart from being the first OpenFlow or SDN controller also established a programing model which inherited by other subsequent controllers. The model was based on processing of OpenFlow messages, with each incoming OpenFlow message trigger an event that had to be processed individually. This model was simple to implement but not efficient and robust and couldn't scale. Nicira along with NTT and Google started developing ONIX, which was meant to be a more abstract and scalable for large deployments. ONIX became the base for Nicira (the core of VMware NSX or network virtualization platform) also there are rumors that it is also the base for Google WAN controller. ONIX was planned to become open source and donated to community but for some reasons the main contributors decided to not to do it which forced the SDN community to focus on developing other platforms. Started in 2010, a new controller introduced,the Beacon controller and it became one of the most popular controllers. It born with contribution of developers from Stanford University. Beacon is a Java-based open source OpenFlow controller created in 2010. It has been widely used for teaching, research, and as the basis of Floodlight. Beacon had the first built-in web user-interface which was a huge step forward in the market of SDN controllers. Also it provided a easier method to deploy and run compared to NOX. Beacon was an influence for design of later controllers after it, however it was only supporting star topologies which was one of the limitations on this controller. Floodlight was a successful SDN controller which was built as a fork of Beacon. BigSwitch networks is developing Floodlight along with other developers. In 2013, Beacon popularity started to shrink down and Floodlight started to gain popularity. Floodlight had fixed many issues of Beacon and added lots of additional features which made it one of the most feature rich controllers available. It also had a web interface, a Java-based GUI and also could get integrated with OpenStack using quantum plugin. Integration with OpenStack was a big step forward as it could be used to provide networking to a large pool of virtual machines, compute and storage. Floodlight adoption increased by evolution of OpenStack and OpenStack adopters. This gave Floodlight greater popularity and applicability than other controllers that came before. Most of controllers came after Floodlight also supported OpenStack integration. Floodlight is still supported and developed by community and BigSwitch networks, and is a base for BigCloud Fabric (the BigSwitch's commercial SDN controller). There are other open source SDN controllers which introduced such as Trema (ruby-based from NEC), Ryu (supported by NTT), FlowER, LOOM and the recent OpenMUL. The following table shows the current open source SDN controllers: Active open source SDNcontroller Non-active open source SDN controllers Floodlight Beacon OpenContrail FlowER OpenDaylight NOX LOOM NodeFlow OpenMUL ONOS POX Ryu Trema OpenDaylight OpenDaylight started in early 2013, and was originally led by IBM and Cisco. It was a new collaborative open source project. OpenDaylight hosted under Linux Foundation and draw support and interest from many developers and adopters. OpenDaylight is a platform to provide common foundations and a robust array of services for SDN environments. OpenDaylight uses a controller model which supports OpenFlow as well as other southbound protocols. It is the first open source controller capable of employing non-OpenFlow proprietary control protocols which eventually lets OpenDaylight to integrate with modern and multi-vendor networks. The first release of OpenDaylight in February 2014 with code name of Hydrogen, followed by Helium in September 2014. The Helium release was significant because it marked a change in direction for the platform that has influenced the way subsequent controllers have been architected. The main change was in the service abstraction layer, which is the part of the controller platform that resides just above the southbound protocols, such as OpenFlow, isolating them from the northbound side and where the applications reside. Hydrogen used an API-driven Service Abstraction Layer (AD-SAL), which had limitations specifically, it meant the controller needed to know about every type of device in the network AND have an inventory of drivers to support them. Helium introduced a Model-driven service abstraction layer (MD-SAL), which meant the controller didn't have to account for all the types of equipment installed in the network, allowing it to manage a wide range of hardware and southbound protocols. Helium release made the framework much more agile and adaptable to changes in the applications; an application could now request changes to the model, which would be received by the abstraction layer and forwarded to the network devices. The OpenDaylight platform built on this advancement in its third release, Lithium, which was introduced in June of 2015. This release focused on broadening the programmability of the network, enabling organizations to create their own service architectures to deliver dynamic network services in a cloud environment and craft intent-based policies. Lithium release was worked on by more than 400 individuals, and contributions from Big Switch Networks, Cisco, Ericsson, HP, NEC, and so on, making it one of the fastest growing open source projects ever. The fourth release, Beryllium come out in February of 2016 and the most recent fifth release, Boron released in September 2016. Many vendors have built and developed commercial SDN controller solutions based on OpenDaylight. Each product has enhanced or added features to OpenDaylight to have some differentiating factor. The use of OpenDaylight in different vendor products are: A base, but sell a commercial version with additional proprietary functionality—for example: Brocade, Ericsson, Ciena, and so on. Part of their infrastructure in their Network as a Service (or XaaS) offerings—for example: Telstra, IBM, and so on. Elements for use in their solution—for example: ConteXtream (now part of HP) Open Networking Operating System (ONOS), which was open sourced in December 2014 is focused on serving the needs of service providers. It is not as widely adopted as OpenDaylight, ONOS has been finding success and gaining momentum around WAN use cases. ONOS is backed by numerous organizations including AT&T, Cisco, Fujitsu, Ericsson, Ciena, Huawei, NTT, SK Telecom, NEC, and Intel, many of whom are also participants in and supporters of OpenDaylight. Apart from open source SDN controllers, there are many commercial, proprietary controllers available in the market. Products such as VMware NSX, Cisco APIC, BigSwitch Big Cloud Fabric, HP VAN and NEC ProgrammableFlow are example commercial and proprietary products. The following table lists the commercially available controllers and their relationship to OpenDaylight: ODL-based ODL-friendly Non-ODL based Avaya Cyan (acquired by Ciena) BigSwitch Brocade HP Juniper Ciena NEC Cisco ConteXtream (HP) Nuage Plexxi Coriant PLUMgrid Ericsson Pluribus Extreme Sonus Huawei (also ships non-ODL controller) VMware NSX Core features of SDN Regardless of an open source or a proprietary SDN platform, there are core features and capabilities which requires the SDN platform to support. These capabilities include: Fabric programmability:Providing the ability to redirect traffic, apply filters to packets (dynamically), and leverage templates to streamline the creation of custom applications. Ensuring northbound APIs allow the control information centralized in the controller available to be changed by SDN applications. This will ensure the controller can dynamically adjust the underlying network to optimize traffic flows to use the least expensive path, take into consideration varying bandwidth constraints, meet quality of service (QoS) requirements. Southbound protocol support:Enabling the controller to communicate to switches and routers and manipulate and optimize how they manage the flow of traffic. Currently OpenFlow is the most standard protocol used between different networking vendors, while there are other southbound protocols that can be used. A SDN platform should support different versions of OpenFlow in order to provide compatibility with different switching equipments. External API support:Ensuring the controller can be used within the varied orchestration and cloud environments such as VMware vSphere, OpenStack, and so on. Using APIs the orchestration platform can communicate with SDN platform in order to publish network policies. For example VMware vSphere shall talk to SDN platform to extend the virtual distributed switches(VDS) from virtual environment to the physical underlay network without any requirement form an network engineer to configure the network. Centralized monitoring and visualization:Since SDN controller has a full visibility over the network, it can offer end-to-end visibility of the network and centralized management to improve overall performance, simplify the identification of issues and accelerate troubleshooting. The SDN controller will be able to discover and present a logical abstraction of all the physical links in the network, also it can discover and present a map of connected devices (MAC addresses) which are related to virtual or physical devices connected to the network. The SDN controller support monitoring protocols, such as syslog, snmp and APIs in order to integrate with third-party management and monitoring systems. Performance: Performance in a SDN environment mainly depends on how fast SDN controller fills the flow tables of SDN enabled switches. Most of SDN controllers pre-populate the flow tables on switches to minimize the delay. When a SDN enabled switch receives a packet which doesn't find a matching entry in its flow table, it sends the packet to the SDN controller in order to find where the packet needs to get forwarded to. A robust SDN solution should ensure that the number of requests form switches are minimum and SDN controller doesn't become a bottleneck in the network. High availability and scalability: Controllers must support high availability clusters to ensure reliability and service continuity in case of failure of a controller. Clustering in SDN controller expands to scalability. A modern SDN platform should support scalability in order to add more controller nodes with load balancing in order to increase the performance and availability. Modern SDN controllers support clustering across multiple different geographical locations. Security:Since all switches communicate with SDN controller, the communication channel needs to be secured to ensure unauthorized devices doesn't compromise the network. SDN controller should secure the southbound channels, use encrypted messaging and mutual authentication to provide access control. Apart from that the SDN controller must implement preventive mechanisms to prevent from denial of services attacks. Also deployment of authorization levels and level controls for multi-tenant SDN platforms is a key requirement. Apart from the aforementioned features SDN controllers are likely to expand their function in future. They may become a network operating system and change the way we used to build networks with hardware, switches, SFPs and gigs of bandwidth. The future will look more software defined, as the silicon and hardware industry has already delivered their promises for high performance networking chips of 40G, 100G. Industry needs more time to digest the new hardware and silicons and refresh the equipment with new gears supporting 10 times the current performance. Current SDN controllers In this section, I'm putting the different SDN controllers in a table. This will help you to understand the current market players in SDN and how OpenDaylight relates to them: Vendors/product Based on OpenDaylight? Commercial/open source Description Brocade SDN controller Yes Commercial It's a commercial version of OpenDaylight, fully supported and with extra reliable modules. Cisco APIC No Commercial Cisco Application Policy Infrastructure Controller (APIC) is the unifying automation and management point for the Application Centric Infrastructure (ACI) data center fabric. Cisco uses APIC controller and Nexus 9k switches to build the fabric. Cisco uses OpFlex as main southbound protocol. Erricson SDN controller Yes Commercial Ericsson's SDN controller is a commercial (hardened) version OpenDaylight SDN controller. Domain specific control applications that use the SDN controller as platform form the basis of the three commercial products in our SDN controller portfolio. Juniper OpenContrial /Contrail No Both OpenContrail is opensource, and Contrail itself is a commercial product. Juniper Contrail Networking is an open SDN solution that consists of Contrail controller, Contrail vRouter, an analytics engine, and published northbound APIs for cloud and NFV. OpenContrail is also available for free from Juniper. Contrail promotes and use MPLS in datacenter. NEC Programmable Flow No Commercial NEC provides its own SDN controller and switches. NEC SDN platform is one of choices of enterprises and has lots of traction and some case studies. Avaya SDN Fx controller Yes Commercial Based on OpenDaylight, bundled as a solution package. Big Cloud Fabric No Commercial BigSwitch networks solution is based on Floodlight opensource project. BigCloud Fabric is a robust, clean SDN controller and works with bare metal whitebox switches. BigCloud Fabric includes SwitchLightOS which is a switch operating system can be loaded on whitebox switches with Broadcom Trident 2 or Tomahawk silicons. The benefit of BigCloud Fabric is that you are not bound to any hardware and you can use baremetal switches from any vendor. Ciena's Agility Yes Commercial Ciena's Agility multilayer WAN controller is built atop the open-source baseline of the OpenDaylight Project—an open, modular framework created by a vendor-neutral ecosystem (rather than a vendor-centric ego-system) that will enable network operators to source network services and applications from both Ciena's Agility and others. HP VAN (Virtual Application Network) No Commercial The building block of the HP open SDN ecosystem, the controller allows third-party developers to deliver innovative SDN solutions. Huawei Agile controller Yes and No (based on editions) Commercial Huawei's SDN controller which integrates as a solution with Huawei enterprise switches Nuage No Commercial Nuage Networks VSP provides SDN capabilities for clouds of all sizes. It is implemented as a non-disruptive overlay for all existing virtualized and non-virtualized server and network resources. Pluribus Netvisor No Commercial Netvisor Premium and Open Netvisor Linux are distributed network operating systems. Open Netvisor integrates atraditional, interoperable networking stack (L2/L3/VXLAN) with an SDN distributed controller that runs in everyswitch of the fabric. VMware NSX No Commercial VMware NSX is an Overlay type of SDN, which currently works with VMware vSphere. The plan is to support OpenStack in future. VMware NSX also has built-in firewall, router and L4 load balancers allowing micro segmentation. OpenDaylight as an SDN controller Previously, we went through the role of SDN controller, and a brief history of ODL.ODL is a modular open SDN platform which allows developers to build any network or business application on top of it to drive the network in the way they want. Currently OpenDaylight has reached to its fifth release (Boron, which is the fifth element in periodic table). ODL releases are named based on periodic table elements, started from first release the Hydrogen. ODL has a 6 month release period, with many developers working on expanding the ODL, 2 releases per year is expected from community. For technical readers to understand it more clearly, the following diagram will help: ODL platform has a broad set of use cases for multivendor, brown field, green fields, service providers and enterprises. ODL is a foundation for networks of the future. Service providers are using ODL to migrate their services to a software enabled level with automatic service delivery and coming out of circuit-based mindset of service delivery. Also they work on providing a virtualized CPE with NFV support in order to provide flexible offerings. Enterprises use ODL for many use cases, from datacenter networking, Cloud and NFS, network automation and resource optimization, visibility, control to deploying a fully SDN campus network. ODL uses a MD-SAL which makes it very scalable and lets it incorporate new applications and protocols faster. ODL supports multiple standard and proprietary southbound protocols, for example with full support of OpenFlow and OVSDB, ODL can communicate with any standard hardware (or even the virtual switches such as Open vSwitch(OVS) supporting such protocols). With such support, ODL can be deployed and used in multivendor environments and control hardware from different vendors from a single console no matter what vendor and what device it is, as long as they support standard southbound protocols. ODL uses a micro service architecture model which allows users to control applications, protocols and plugins while deploying SDN applications. Also ODL is able to manage the connection between external consumers and providers. The followingdiagram explains the ODL footprint and different components and projects within the ODL: Micro servicesarchitecture ODL stores its YANG data structure in a common data store and uses messaging infrastructure between different components to enable a model-driven approach to describe the network and functions. In ODL MD-SAL, any SDN application can be integrated as a service and then loaded into the SDN controller. These services (apps) can be chained together in any number and ways to match the application needs. This concept allows users to only install and enable the protocols and services they need which makes the system light and efficient. Also services and applications created by users can be shared among others in the ecosystem since the SDN application deployment for ODL follows a modular design. ODL supports multiple southbound protocols. OpenFlow and OpenFlow extension such as Table Type Patterns (TTP), as well as other protocols including NETCONF, BGP/PCEP, CAPWAP and OVSDB. Also ODL supports Cisco OpFlex protocol: ODL platform provides a framework for authentication, authorization and accounting (AAA), as well as automatic discovery and securing of network devices and controllers. Another key area in security is to use encrypted and authenticated communication trough southbound protocols with switches and routers within the SDN domain. Most of southbound protocols support security encryption mechanisms. Summary In this article we learned about SDN, and why it is important. We reviewed the SDN controller products, the ODL history as well as core features of SDN controllers and market leader controllers. We managed to dive in some details about SDN . Resources for Article: Further resources on this subject: Managing Network Devices [article] Setting Up a Network Backup Server with Bacula [article] Point-to-Point Networks [article]

0
0
28980

article-image-web-application-information-gathering

Packt

05 Jun 2017

4 min read

Web Application Information Gathering

Packt

05 Jun 2017

4 min read

0
0
24882

How-To Tutorials

article-image-booting-android-system-using-pxenfs

Packt

29 May 2017

31 min read

Booting Up Android System Using PXE/NFS

Packt

29 May 2017

31 min read

0
1
40188

How-To Tutorials

article-image-top-10-deep-learning-frameworks

Amey Varangaonkar

25 May 2017

9 min read

Top 10 deep learning frameworks

Amey Varangaonkar

25 May 2017

9 min read

Deep learning frameworks are powering the artificial intelligence revolution. Without them, it would be almost impossible for data scientists to deliver the level of sophistication in their deep learning algorithms that advances in computing and processing power have made possible. Put simply, deep learning frameworks make it easier to build deep learning algorithms of considerable complexity. This follows a wider trend that you can see in other fields of programming and software engineering; open source communities are continually are developing new tools that simplify difficult tasks and minimize arduous ones. The deep learning framework you choose to use is ultimately down to what you're trying to do and how you work already. But to get you started here is a list of 10 of the best and most popular deep learning frameworks being used today. What are the best deep learning frameworks? Tensorflow One of the most popular Deep Learning libraries out there, Tensorflow, was developed by the Google Brain team and open-sourced in 2015. Positioned as a ‘second-generation machine learning system’, Tensorflow is a Python-based library capable of running on multiple CPUs and GPUs. It is available on all platforms, desktop, and mobile. It also has support for other languages such as C++ and R and can be used directly to create deep learning models, or by using wrapper libraries (for e.g. Keras) on top of it. In November 2017, Tensorflow announced a developer preview for Tensorflow Lite, a lightweight machine learning solution for mobile and embedded devices. The machine learning paradigm is continuously evolving - and the focus is now slowly shifting towards developing machine learning models that run on mobile and portable devices in order to make the applications smarter and more intelligent. Learn how to build a neural network with TensorFlow. If you're just starting out with deep learning, TensorFlow is THE go-to framework. It’s Python-based, backed by Google, has a very good documentation, and there are tons of tutorials and videos available on the internet to guide you. You can check out Packt’s TensorFlow catalog here. Keras Although TensorFlow is a very good deep learning library, creating models using only Tensorflow can be a challenge, as it is a pretty low-level library and can be quite complex to use for a beginner. To tackle this challenge, Keras was built as a simplified interface for building efficient neural networks in just a few lines of code and it can be configured to work on top of TensorFlow. Written in Python, Keras is very lightweight, easy to use, and pretty straightforward to learn. Because of these reasons, Tensorflow has incorporated Keras as part of its core API. Despite being a relatively new library, Keras has a very good documentation in place. If you want to know more about how Keras solves your deep learning problems, this interview by our best-selling author Sujit Pal should help you. Read now: Why you should use Keras for deep learning [box type="success" align="" class="" width=""]If you have some knowledge of Python programming and want to get started with deep learning, this is one library you definitely want to check out![/box] Caffe Built with expression, speed, and modularity in mind, Caffe is one of the first deep learning libraries developed mainly by Berkeley Vision and Learning Center (BVLC). It is a C++ library which also has a Python interface and finds its primary application in modeling Convolutional Neural Networks. One of the major benefits of using this library is that you can get a number of pre-trained networks directly from the Caffe Model Zoo, available for immediate use. If you’re interested in modeling CNNs or solve your image processing problems, you might want to consider this library. Following the footsteps of Caffe, Facebook also recently open-sourced Caffe2, a new light-weight, modular deep learning framework which offers greater flexibility for building high-performance deep learning models. Torch Torch is a Lua-based deep learning framework and has been used and developed by big players such as Facebook, Twitter and Google. It makes use of the C/C++ libraries as well as CUDA for GPU processing. Torch was built with an aim to achieve maximum flexibility and make the process of building your models extremely simple. More recently, the Python implementation of Torch, called PyTorch, has found popularity and is gaining rapid adoption. PyTorch PyTorch is a Python package for building deep neural networks and performing complex tensor computations. While Torch uses Lua, PyTorch leverages the rising popularity of Python, to allow anyone with some basic Python programming language to get started with deep learning. PyTorch improves upon Torch’s architectural style and does not have any support for containers - which makes the entire deep modeling process easier and transparent to you. Still wondering how PyTorch and Torch are different from each other? Make sure you check out this interesting post on Quora. Deeplearning4j DeepLearning4j (or DL4J) is a popular deep learning framework developed in Java and supports other JVM languages as well. It is very slick and is very widely used as a commercial, industry-focused distributed deep learning platform. The advantage of using DL4j is that you can bring together the power of the whole Java ecosystem to perform efficient deep learning, as it can be implemented on top of the popular Big Data tools such as Apache Hadoop and Apache Spark. [box type="success" align="" class="" width=""]If Java is your programming language of choice, then you should definitely check out this framework. It is clean, enterprise-ready, and highly effective. If you’re planning to deploy your deep learning models to production, this tool can certainly be of great worth![/box] MXNet MXNet is one of the most languages-supported deep learning frameworks, with support for languages such as R, Python, C++ and Julia. This is helpful because if you know any of these languages, you won’t need to step out of your comfort zone at all, to train your deep learning models. Its backend is written in C++ and cuda, and is able to manage its own memory like Theano. MXNet is also popular because it scales very well and is able to work with multiple GPUs and computers, which makes it very useful for the enterprises. This is also one of the reasons why Amazon made MXNet its reference library for Deep Learning too. In November, AWS announced the availability of ONNX-MXNet, which is an open source Python package to import ONNX (Open Neural Network Exchange) deep learning models into Apache MXNet. Read why MXNet is a versatile deep learning framework here. Microsoft Cognitive Toolkit Microsoft Cognitive Toolkit, previously known by its acronym CNTK, is an open-source deep learning toolkit to train deep learning models. It is highly optimized and has support for languages such as Python and C++. Known for its efficient resource utilization, you can easily implement efficient Reinforcement Learning models or Generative Adversarial Networks (GANs) using the Cognitive Toolkit. It is designed to achieve high scalability and performance and is known to provide high-performance gains when compared to other toolkits like Theano and Tensorflow when running on multiple machines. Here is a fun comparison of TensorFlow versus CNTK, if you would like to know more. deeplearn.js Gone are the days when you required serious hardware to run your complex machine learning models. With deeplearn.js, you can now train neural network models right on your browser! Originally developed by the Google Brain team, deeplearn.js is an open-source, JavaScript-based deep learning library which runs on both WebGL 1.0 and WebGL 2.0. deeplearn.js is being used today for a variety of purposes - from education and research to training high-performance deep learning models. You can also run your pre-trained models on the browser using this library. BigDL BigDL is distributed deep learning library for Apache Spark and is designed to scale very well. With the help of BigDL, you can run your deep learning applications directly on Spark or Hadoop clusters, by writing them as Spark programs. It has a rich deep learning support and uses Intel’s Math Kernel Library (MKL) to ensure high performance. Using BigDL, you can also load your pre-trained Torch or Caffe models into Spark. If you want to add deep learning functionalities to a massive set of data stored on your cluster, this is a very good library to use. [box type="shadow" align="" class="" width=""]Editor's Note: We have removed Theano and Lasagne from the original list due to the Theano retirement announcement. RIP Theano! Before Tensorflow, Caffe or PyTorch came to be, Theano was the most widely used library for deep learning. While it was a low-level library supporting CPU as well as GPU computations, you could wrap it with libraries like Keras to simplify the deep learning process. With the release of version 1.0, it was announced that the future development and support for Theano would be stopped. There would be minimal maintenance to keep it working for the next one year, after which even the support activities on the library would be suspended completely. “Supporting Theano is no longer the best way we can enable the emergence and application of novel research ideas”, said Prof. Yoshua Bengio, one of the main developers of Theano. Thank you Theano, you will be missed! Goodbye Lasagne Lasagne is a high-level deep learning library that runs on top of Theano. It has been around for quite some time now and was developed with the aim of abstracting the complexities of Theano, and provide a more friendly interface to the users to build and train neural networks. It requires Python and finds many similarities to Keras, which we just saw above. However, if we are to find differences between the two, Keras is faster and has a better documentation in place.[/box] There are many other deep learning libraries and frameworks available for use today – DSSTNE, Apache Singa, Veles are just a few worth an honorable mention. Which deep learning frameworks will best suit your needs? Ultimately, it depends on a number of factors. If you want to get started with deep learning, your safest bet would be to use a Python-based framework like Tensorflow, which are quite popular. For seasoned professionals, the efficiency of the trained model, ease of use, speed and resource utilization are all important considerations for choosing the best deep learning framework.

0
0
63122

Building a Smart Contract Deployment Platform

Your first Unity project

WordPress as a Web Application Framework

Creating Sprites

IoT Analytics for the Cloud

Docker Swarm

Configuring the ESP8266

Backpropagation Algorithm

Ionic Components

Erasure coding for cold storage

Trending Topics

The Microsoft Azure Stack Architecture

Introduction to SDN - Transformation from legacy to SDN

Web Application Information Gathering

Booting Up Android System Using PXE/NFS

Top 10 deep learning frameworks

Create a Free Account To Continue Reading

Sign in to activate your 7-day free access