Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7018 Articles
article-image-installing-red-hat-cloudforms-red-hat-openstack
Packt
04 Sep 2015
8 min read
Save for later

Installing Red Hat CloudForms on Red Hat OpenStack

Packt
04 Sep 2015
8 min read
In this article by Sangram Rath, the author of the book Hybrid Cloud Management with Red Hat CloudForms, this article takes you through the steps required to install, configure, and use Red Hat CloudForms on Red Hat Enterprise Linux OpenStack. However, you should be able to install it on OpenStack running on any other Linux distribution. The following topics are covered in this article: System requirements Deploying the Red Hat CloudForms Management Engine appliance Configuring the appliance Accessing and navigating the CloudForms web console (For more resources related to this topic, see here.) System requirements Installing the Red Hat CloudForms Management Engine Appliance requires an existing virtual or cloud infrastructure. The following are the latest supported platforms: OpenStack Red Hat Enterprise Virtualization VMware vSphere The system requirements for installing CloudForms are different for different platforms. Since this book talks about installing it on OpenStack, we will see the system requirements for OpenStack. You need a minimum of: Four VCPUs 6 GB RAM 45 GB disk space The flavor we select to launch the CloudForms instance must meet or exceed the preceding requirements. For a list of system requirements for other platforms, refer to the following links: System requirements for Red Hat Enterprise Virtualization: https://access.redhat.com/documentation/en-US/Red_Hat_CloudForms/3.1/html/Installing_CloudForms_on_Red_Hat_Enterprise_Virtualization/index.html System requirements for installing CloudForms on VMware vSphere: https://access.redhat.com/documentation/en-US/Red_Hat_CloudForms/3.1/html/Installing_CloudForms_on_VMware_vSphere/index.html Additional OpenStack requirements Before we can launch a CloudForms instance, we need to ensure that some additional requirements are met: Security group: Ensure that a rule is created to allow traffic on port 443 in the security group that will be used to launch the appliance. Flavor: Based on the system requirements for running the CloudForms appliance, we can either use an existing flavor, such as m1.large, or create a new flavor for the CloudForms Management Engine Appliance. To create a new flavor, click on the Create Flavor button under the Flavor option in Admin and fill in the required parameters, especially these three: At least four VCPUs At least 6144 MB of RAM At least 45 GB of disk space Key pair: Although, at the VNC console, you can just use the default username and password to log in to the appliance, it is good to have access to a key pair as well, if required, for remote SSH. Deploying the Red Hat CloudForms Management Engine appliance Now that we are aware of the resource and security requirements for Red Hat CloudForms, let's look at how to obtain a copy of the appliance and run it. Obtaining the appliance The CloudForms Management appliance for OpenStack can be downloaded from your Red Hat customer portal under the Red Hat CloudForms product page. You need access to a Red Hat CloudForms subscription to be able to do so. At the time of writing this book, the direct download link for this is https://rhn.redhat.com/rhn/software/channel/downloads/Download.do?cid=20037. For more information on obtaining the subscription and appliance, or to request a trial, visit http://www.redhat.com/en/technologies/cloud-computing/cloudforms. Note If you are unable to get access to Red Hat CloudForms, ManageIQ (the open source version) can also be used for hands-on experience. Creating the appliance image in OpenStack Before launching the appliance, we need to create an image in OpenStack for the appliance, since OpenStack requires instances to be launched from an image. You can create a new Image under Project with the following parameters (see the screenshot given for assistance): Enter a name for the image. Enter the image location in Image Source (HTTP URL). Set the Format as QCOW2. Optionally, set the Minimum Disk size. Optionally, set Minimum Ram. Make it Public if required and Create An Image. Note that if you have a newer release of OpenStack, there may be some additional options, but the preceding are what need to be filled in—most importantly the download URL of the Red Hat CloudForms appliance. Wait for the Status field to reflect as Active before launching the instance, as shown in this screenshot: Launching the appliance instance In OpenStack, under Project, select Instances and then click on Launch Instance. In the Launch Instance wizard enter the following instance information in the Details tab: Select an Availabilty Zone. Enter an Instance Name. Select Flavor. Set Instance Count. Set Instance Boot Source as Boot from image. Select CloudForms Management Engine Appliance under Image Name. The final result should appear similar to the following figure: Under the Access & Security tab, ensure that the correct Key Pair and Security Group tab are selected, like this: For Networking, select the proper networks that will provide the required IP addresses and routing, as shown here: Other options, such as Post-Creation and Advanced Options, are optional and can be left blank. Click on Launch when ready to start creating the instance. Wait for the instance state to change to Running before proceeding to the next step. Note If you are accessing the CloudForms Management Engine from the Internet, a Floating IP address needs to be associated with the instance. This can be done from Project, under Access & Security and then the Floating IPs tab. The Red Hat CloudForms web console The web console provides a graphical user interface for working with the CloudForms Management Engine Appliance. The web console can be accessed from a browser on any machine that has network access to the CloudForms Management Engine server. System requirements The system requirements for accessing the Red Hat CloudForms web console are: A Windows, Linux, or Mac computer A modern browser, such as Mozilla Firefox, Google Chrome, and Internet Explorer 8 or above Adobe Flash Player 9 or above The CloudForms Management Engine Appliance must already be installed and activated in your enterprise environment Accessing the Red Hat CloudForms Management Engine web console Type the hostname or floating IP assigned to the instance prefixed by https in a supported browser to access the appliance. Enter default username as admin and the password as smartvm to log in to the appliance, as shown in this screenshot: You should log in to only one tab in each browser, as the console settings are saved for the active tab only. The CloudForms Management Engine also does not guarantee that the browser's Back button will produce the desired results. Use the breadcrumbs provided in the console. Navigating the web console The web console has a primary top-level menu that provides access to feature sets such as Insight, Control, and Automate, along with menus used to add infrastructure and cloud providers, create service catalogs and view or raise requests. The secondary menu appears below the top primary menu, and its options change based on the primary menu option selected. In certain cases, a third-sublevel menu may also appear for additional options based on the selection in the secondary menu. The feature sets available in Red Hat CloudForms are categorized under eight menu items: Cloud Intelligence: This provides a dashboard view of your hybrid cloud infrastructure for the selected parameters. Whatever is displayed here can be configured as a widget. It also provides additional insights into the hybrid cloud in the form of reports, chargeback configuration and information, timeline views, and an RSS feeds section. Services: This provides options for creating templates and service catalogs that help in provisioning multitier workloads across providers. It also lets you create and approve requests for these service catalogs. Clouds: This option in the top menu lets you add cloud providers; define availability zones; and create tenants, flavors, security groups, and instances. Infrastructure: This option, in a way similar to clouds, lets you add infrastructure providers; define clusters; view, discover, and add hosts; provision VMs; work with data stores and repositories; view requests; and configure the PXE. Control: This section lets you define compliance and control policies for the infrastructure providers using events, conditions, and actions based on the conditions. You can further combine these policies into policy profiles. Another important feature is alerting the administrators, which is configured from here. You can also simulate these policies, import and export them, and view logs. Automate: This menu option lets you manage life cycle tasks such as provisioning and retirement, and automation of resources. You can create provisioning dialogs to provision hosts and virtual machines and service dialogs to provision service catalogs. Dialog import/export, logs, and requests for automation are all managed from this menu option. Optimize: This menu option provides utilization, planning, and bottleneck summaries for the hybrid cloud environment. You can also generate reports for these individual metrics. Configure: Here, you can customize the look of the dashboard; view queued, running tasks and check errors and warnings for VMs and the UI. It let's you configure the CloudForms Management Engine appliance settings such as database, additional worker appliances, SmartProxy, and white labelling. One can also perform tasks maintenance tasks such as updates and manual modification of the CFME server configuration files. Summary In this article, we deployed the Red Hat CloudForms Management Engine Appliance in an OpenStack environment, and you learned where to configure the hostname, network settings, and time zone. We then used the floating IP of the instance to access the appliance from a web browser, and you learned where the different feature sets are and how to navigate around. Resources for Article: Further resources on this subject: Introduction to Microsoft Azure Cloud Services[article] Apache CloudStack Architecture[article] Using OpenStack Swift [article]
Read more
  • 0
  • 0
  • 14012

article-image-working-basic-components-make-threejs-scene
Packt
17 Oct 2013
22 min read
Save for later

Working with the Basic Components That Make Up a Three.js Scene

Packt
17 Oct 2013
22 min read
(For more resources related to this topic, see here.) Creating a scene We know that for a scene to show anything, we need three types of components: Component Description Camera It determines what is rendered on the screen Lights They have an effect on how materials are shown and used when creating shadow effects Objects These are the main objects that are rendered from the perspective of the camera: cubes, spheres, and so on The THREE.Scene() object serves as the container for all these different objects. This object itself doesn't have too many options and functions. Basic functionality of the scene The best way to explore the functionality of the scene is by looking at an example. I'll use this example to explain the various functions and options that a scene has. When we open this example in the browser, the output will look something like the following screenshot: Even though the scene looks somewhat empty, it already contains a couple of objects. By looking at the following source code, we can see that we've used the Scene.add(object) function from the THREE.Scene() object to add a THREE.Mesh (the ground plane that you see), a THREE.SpotLight. and a THREE.AmbientLight object. The THREE.Camera object is added automatically by the Three.js library when you render the scene, but can also be added manually if you prefer. var scene = new THREE.Scene(); var camera = new THREE.PerspectiveCamera(45, window.innerWidth / window.innerHeight, 0.1, 1000); ... var planeGeometry = new THREE.PlaneGeometry(60,40,1,1); var planeMaterial = new THREE.MeshLambertMaterial({color: 0xffffff}); var plane = new THREE.Mesh(planeGeometry,planeMaterial); ... scene.add(plane); var ambientLight = new THREE.AmbientLight(0x0c0c0c); scene.add(ambientLight); ... var spotLight = new THREE.SpotLight( 0xffffff ); ... scene.add( spotLight ); Before we look deeper into the THREE.Scene() object, I'll first explain what you can do in the demonstration, and after that we'll look at some code. Open this example in your browser and look at the controls at the upper-right corner as you can see in the following screenshot: With these controls you can add a cube to the scene, remove the last added cube from the scene, and show all the current objects that the scene contains. The last entry in the control section shows the current number of objects in the scene. What you'll probably notice when you start up the scene is that there are already four objects in the scene. These are the ground plane, the ambient light, the spot light, and the camera that we had mentioned earlier. In the following code fragment, we'll look at each of the functions in the control section and start with the easiest one: the addCube() function: this.addCube = function() { var cubeSize = Math.ceil((Math.random() * 3)); var cubeGeometry = new THREE.CubeGeometry(cubeSize,cubeSize,cubeSize); var cubeMaterial = new THREE.MeshLambertMaterial( {color: Math.random() * 0xffffff }); var cube = new THREE.Mesh(cubeGeometry, cubeMaterial); cube.castShadow = true; cube.name = "cube-" + scene.children.length; cube.position.x=-30 + Math.round( (Math.random() * planeGeometry.width)); cube.position.y= Math.round((Math.random() * 5)); cube.position.z=-20 + Math.round((Math.random() * planeGeometry.height)); scene.add(cube); this.numberOfObjects = scene.children.length; }; This piece of code should be pretty easy to read by now. Not many new concepts are introduced here. When you click on the addCube button, a new THREE.CubeGeometry instance is created with a random size between zero and three. Besides a random size, the cube also gets a random color and position in the scene. A new thing in this piece of code is that we also give the cube a name by using the name attribute. Its name is set to cube-appended with the number of objects currently in the scene (shown by the scene.children.length property). So you'll get names like cube-1, cube-2, cube-3, and so on. A name can be useful for debugging purposes, but can also be used to directly find an object in your scene. If you use the Scene.getChildByName(name) function, you can directly retrieve a specific object and, for instance, change its location. You might wonder what the last line in the previous code snippet does. The numberOfObjects variable is used by our control GUI to list the number of objects in the scene. So whenever we add or remove an object, we set this variable to the updated count. The next function that we can call from the control GUI is removeCube and, as the name implies, clicking on this button removes the last added cube from the scene. The following code snippet shows how this function is defined: this.removeCube = function() { var allChildren = scene.children; var lastObject = allChildren[allChildren.length-1]; if (lastObject instanceof THREE.Mesh) { scene.remove(lastObject); this.numberOfObjects = scene.children.length; } } To add an object to the scene we will use the add() function. To remove an object from the scene we use the not very surprising remove() function. In the given code fragment we have used the children property from the THREE.Scene() object to get the last object that was added. We also need to check whether that object is a Mesh object in order to avoid removing the camera and the lights. After we've removed the object, we will once again update the GUI property that holds the number of objects in the scene. The final button on our GUI is labeled as outputObjects. You've probably already clicked on it and nothing seemed to happen. What this button does is print out all the objects that are currently in our scene and will output them to the web browser Console as shown in the following screenshot: The code to output information to the Console log makes use of the built-in console object as shown: this.outputObjects = function() { console.log(scene.children); } This is great for debugging purposes; especially when you name your objects, it's very useful for finding issues and problems with a specific object in your scene. For instance, the properties of the cube-17 object will look like the following code snippet: __webglActive: true __webglInit: true _modelViewMatrix: THREE.Matrix4 _normalMatrix: THREE.Matrix3 _vector: THREE.Vector3 castShadow: true children: Array[0] eulerOrder: "XYZ" frustumCulled: true geometry: THREE.CubeGeometry id: 20 material: THREE.MeshLambertMaterial matrix: THREE.Matrix4 matrixAutoUpdate: true matrixRotationWorld: THREE.Matrix4 matrixWorld: THREE.Matrix4 matrixWorldNeedsUpdate: false name: "cube-17" parent: THREE.Scene position: THREE.Vector3 properties: Object quaternion: THREE.Quaternion receiveShadow: false renderDepth: null rotation: THREE.Vector3 rotationAutoUpdate: true scale: THREE.Vector3 up: THREE.Vector3 useQuaternion: false visible: true __proto__: Object So far we've seen the following scene-related functionality: Scene.Add(): This method adds an object to the scene Scene.Remove(): This removes an object from the scene Scene.children(): This method gets a list of all the children in the scene Scene.getChildByName(): This gets a specific object from the scene by using the name attribute These are the most important scene-related functions, and most often you won't need any more. There are, however, a couple of helper functions that could come in handy, and I'd like to show them based on the code that handles the cube rotation. We use a render loop to render the scene. Let's look at the same code snippet for this example: function render() { stats.update(); scene.traverse(function(e) { if (e instanceof THREE.Mesh && e != plane ) { e.rotation.x+=controls.rotationSpeed; e.rotation.y+=controls.rotationSpeed; e.rotation.z+=controls.rotationSpeed; } }); requestAnimationFrame(render); renderer.render(scene, camera); } Here we can see that the THREE.Scene.traverse() function is being used. We can pass a function as an argument to the traverse() function. This passed in function will be called for each child of the scene. In the render() function, we will use the traverse() function to update the rotation for each of the cube instances (we will explicitly ignore the ground plane). We could also have done this by iterating over the children property array by using a for loop. Before we dive into the Mesh and Geometry object details, I'd like to show you two interesting properties that you can set on the Scene object: fog and overrideMaterial. Adding the fog effect to the scene The fog property let's you add a fog effect to the complete scene. The farther an object is, the more it will be hidden from sight. The following screenshot shows how the fog property is enabled: Enabling the fog property is really easy to do in the Three.js library. Just add the following line of code after you've defined your scene: scene.fog=new THREE.Fog( 0xffffff, 0.015, 100 ); Here we are defining a white fog (0xffffff). The last two properties can be used to tune how the mist will appear. The 0.015 value sets the near property and the 100 value sets the far property. With these properties you can determine where the mist will start and how fast it will get denser. There is also a different way to set the mist for the scene; for this you will have to use the following definition: scene.fog=new THREE.FogExp2( 0xffffff, 0.015 ); This time we don't specify the near and far properties, but just the color and the mist density. It's best to experiment a bit with these properties in order to get the effect that you want. Using the overrideMaterial property The last property that we will discuss for the scene is the overrideMaterial property, which is used to fix the materials of all the objects. When you use this property as shown in the following code snippet, all the objects that you add to the scene will make use of the same material: scene.overrideMaterial = new THREE.MeshLambertMaterial({color: 0xffffff}); The scene will be rendered as shown in the following screenshot: In the earlier screenshot, you can see that all the cube instances are rendered by using the same material and color. In this example we've used a MeshLambertMaterial object as the material. With this material type, you can create non-shiny looking objects which will respond to the lights that you add to the scene. In this section we've looked at the first of the core concepts of the Three.js library: the scene. The most important thing to remember about the scene is that it is basically a container for all the objects, lights, and cameras that you want to use while rendering. The following table summarizes the most important functions and attributes of the Scene object: Function/Property Description add(object) Adds an object to the scene. You can also use this function, as we'll see later, to create groups of objects. children Returns a list of all the objects that have been added to the scene, including the camera and lights. getChildByName(name) When you create an object, you can give it a distinct name by using the name attribute. The Scene object has a function that you can use to directly return an object with a specific name. remove(object)  If you've got a reference to an object in the scene, you can also remove it from the scene by using this function. traverse(function) The children attribute returns a list of all the children in the scene. With the traverse() function we can also access these children by passing in a callback function. fog This property allows you to set the fog for the scene. It will render a haze that hides the objects that are far away. overrideMaterial With this property you can force all the objects in the scene to use the same material. In the next section we'll look closely at the objects that you can add to the scene. Working with the Geometry and Mesh objects In each of the examples so far you've already seen the geometries and meshes that are being used. For instance, to add a sphere object to the scene we did the following: var sphereGeometry = new THREE.SphereGeometry(4,20,20); var sphereMaterial = new THREE.MeshBasicMaterial({color: 0x7777ff); var sphere = new THREE.Mesh(sphereGeometry,sphereMaterial); We have defined the shape of the object, its geometry, what this object looks like, its material, and combined all of these in a mesh that can be added to a scene. In this section we'll look a bit closely at what the Geometry and Mesh objects are. We'll start with the geometry. The properties and functions of a geometry The Three.js library comes with a large set of out-of-the-box geometries that you can use in your 3D scene. Just add a material, create a mesh variable, and you're pretty much done. The following screenshot, from example 04-geometries.html, shows a couple of the standard geometries available in the Three.js library: In we'll explore all the basic and advanced geometries that the Three.js library has to offer. For now, we'll go into more detail on what the geometry variable actually is. A geometry in Three.js, and in most other 3D libraries, is basically a collection of points in a 3D space and a number of faces connecting all those points together. Take, for example, a cube: A cube has eight corners. Each of these corners can be defined as a combination of x, y, and z coordinates. So, each cube has eight points in a 3D space. In the Three.js library, these points are called vertices. A cube has six sides, with one vertex at each corner. In the Three.js library, each of these sides is called a face. When you use one of the Three.js library-provided geometries, you don't have to define all the vertices and faces yourself. For a cube you only need to define the width, height, and depth. The Three.js library uses that information and creates a geometry with eight vertices at the correct position and the correct face. Even though you'd normally use the Three.js library-provided geometries, or generate them automatically, you can still create geometries completely by hand by defining the vertices and faces. This is shown in the following code snippet: var vertices = [ new THREE.Vector3(1,3,1), new THREE.Vector3(1,3,-1), new THREE.Vector3(1,-1,1), new THREE.Vector3(1,-1,-1), new THREE.Vector3(-1,3,-1), new THREE.Vector3(-1,3,1), new THREE.Vector3(-1,-1,-1), new THREE.Vector3(-1,-1,1) ]; var faces = [ new THREE.Face3(0,2,1), new THREE.Face3(2,3,1), new THREE.Face3(4,6,5), new THREE.Face3(6,7,5), new THREE.Face3(4,5,1), new THREE.Face3(5,0,1), new THREE.Face3(7,6,2), new THREE.Face3(6,3,2), new THREE.Face3(5,7,0), new THREE.Face3(7,2,0), new THREE.Face3(1,3,4), new THREE.Face3(3,6,4), ]; var geom = new THREE.Geometry(); geom.vertices = vertices; geom.faces = faces; geom.computeCentroids(); geom.mergeVertices(); This code shows you how to create a simple cube. We have defined the points that make up this cube in the vertices array. These points are connected to create triangular faces and are stored in the faces array. For instance, the new THREE.Face3(0,2,1) element creates a triangular face by using the points 0, 2, and 1 from the vertices array. In this example we have used a THREE.Face3 element to define the six sides of the cube, that is, two triangles for each face. In the previous versions of the Three.js library, you could also use a quad instead of a triangle. A quad uses four vertices instead of three to define the face. Whether using quads or triangles is better is a much-heated debate in the 3D modeling world. Basically, using quads is often preferred during modeling, since they can be more easily enhanced and smoothed much easier than triangles. For rendering and game engines, though, working with triangles is easier since every shape can be rendered as a triangle. Using these vertices and faces, we can now create our custom geometry, and use it to create a mesh. I've created an example that you can use to play around with the position of the vertices. In example 05-custom-geometry.html, you can change the position of all the vertices of a cube. This is shown in the following screenshot: This example, which uses the same setup as all our other examples, has a render loop. Whenever you change one of the properties in the drop-down control box, the cube is rendered correctly based on the changed position of one of the vertices. This isn't something that works out-of-the-box. For performance reasons, the Three.js library assumes that the geometry of a mesh won't change during its lifetime. To get our example to work we need to make sure that the following is added to the code in the render loop: mesh.geometry.vertices=vertices; mesh.geometry.verticesNeedUpdate=true; mesh.geometry.computeFaceNormals(); In the first line of the given code snippet, we point the vertices of the mesh that you see on the screen to an array of the updated vertices. We don't need to reconfigure the faces, since they are still connected to the same points as they were before. After we've set the updated vertices, we need to tell the geometry that the vertices need to be updated. We can do this by setting the verticesNeedUpdate property of the geometry to true. Finally we will do a recalculation of the faces to update the complete model by using the computeFaceNormals() function. The last geometry functionality that we'll look at is the clone() function. We had mentioned that the geometry defines the form, the shape of an object, and combined with a material we can create an object that can be added to the scene to be rendered by the Three.js library. With the clone() function, as the name implies, we can make a copy of the geometry and, for instance, use it to create a different mesh with a different material. In the same example, that is, 05-custom-geometry.html, you can see a clone button at the top of the control GUI, as seen in the following screenshot: If you click on this button, a clone will be made of the geometry as it currently is, and a new object is created with a different material and is added to the scene. The code for this is rather trivial, but is made a bit more complex because of the materials that I have used. Let's take a step back and first look at the code that was used to create the green material for the cube: var materials = [ new THREE.MeshLambertMaterial( { opacity:0.6, color: 0x44ff44, transparent:true } ), new THREE.MeshBasicMaterial( { color: 0x000000, wireframe: true } ) ]; As you can see, I didn't use a single material, but an array of two materials. The reason is that besides showing a transparent green cube, I also wanted to show you the wireframe, since that shows very clearly where the vertices and faces are located. The Three.js library, of course, supports the use of multiple materials when creating a mesh. You can use the SceneUtils.createMultiMaterialObject() function for this as shown: var mesh = THREE.SceneUtils.createMultiMaterialObject( geom,materials); What the Three.js library does in this function is that it doesn't create one THREE.Mesh instance, but it creates one for each material that you have specified, and puts all of these meshes in a group. This group can be used in the same manner that you've used for the Scene object. You can add meshes, get meshes by name, and so on. For instance, to add shadows to all the children in this group, we will do the following: mesh.children.forEach(function(e) {e.castShadow=true}); Now back to the clone() function that we were discussing earlier: this.clone = function() { var cloned = mesh.children[0].geometry.clone(); var materials = [ new THREE.MeshLambertMaterial( { opacity:0.6, color: 0xff44ff, transparent:true } ), new THREE.MeshBasicMaterial({ color: 0x000000, wireframe: true } ) ]; var mesh2 = THREE.SceneUtils.createMultiMaterialObject(cloned,materials); mesh2.children.forEach(function(e) {e.castShadow=true}); mesh2.translateX(5); mesh2.translateZ(5); mesh2.name="clone"; scene.remove(scene.getChildByName("clone")); scene.add(mesh2); } This piece of JavaScript is called when the clone button is clicked on. Here we clone the geometry of the first child of the cube. Remember, the mesh variable contains two children: a mesh that uses the MeshLambertMaterial and a mesh that uses the MeshBasicMaterial. Based on this cloned geometry, we will create a new mesh, aptly named mesh2. We can move this new mesh by using the translate() function remove the previous clone (if present), and add the clone to the scene. That's enough on geometries for now. The functions and attributes for a mesh We've already learned that, in order to create a mesh, we need a geometry and one or more materials. Once we have a mesh, we can add it to the scene, and it is rendered. There are a couple of properties that you can use to change where and how this mesh appears in the scene. In the first example, we'll look at the following set of properties and functions: Function/Property Description position Determines the position of this object relative to the position of its parent. Most often the parent of an object is a THREE.Scene() object. rotation With this property you can set the rotation of an object around any of its axes. scale This property allows you to scale the object around its x, y, and z axes. translateX(amount)  Moves the object through the specified amount over the x axis. translateY(amount)  Moves the object through the specified amount over the y axis. translateZ(amount)  Moves the object through the specified amount over the z axis. As always, we have an example ready for you that'll allow you to play around with these properties. If you open up the 06-mesh-properties.html example in your browser, you will get a drop-down menu where you can alter all these properties and directly see the result, as shown in the following screenshot: Let me walk you through them; I'll start with the position property. We've already seen this property a couple of times, so let's quickly address it. With this property you can set the x, y, and z coordinates of the object. The position of an object is relative to its parent object, which usually is the scene that you have added the object to. We can set an object's position property in three different ways; each coordinate can be set directly as follows: cube.position.x=10; cube.position.y=3; cube.position.z=1; But we can also set all of them at once: cube.position.set(10,3,1); There is also a third option. The position property is a THREE.Vector3 object. This means that we can also do the following to set this object: cube.postion=new THREE.Vector3(10,3,1) I want to make a quick sidestep before looking at the other properties of this mesh. I had mentioned that this position is set relative to the position of its parent. In the previous section on THREE.Geometry, we made use of the THREE.SceneUtils.createMultiMaterialObject object to create a multimaterial object. I had explained that this doesn't really return a single mesh, but a group that contains a mesh based on the same geometry for each material. In our case, it is a group that contains two meshes. If we change the position of one of the meshes that is created, you can clearly see that there really are two distinct objects. However, if we now move the created group around, the offset will remain the same. These two meshes are shown in the following screenshot: Ok, the next one on the list is the rotation property. You've already seen this property being used a couple of times in this article. With this property, you can set the rotation of the object around one of its axes. You can set this value in the same manner as we did the for the position property. A complete rotation, as you might remember from math class, is two pi. The following code snippet shows how to configure this: cube.rotation.x=0.5*Math.PI; cube.rotation.set(0.5*Math.PI,0,0); cube.rotation = new THREE.Vector3(0.5*Math.PI,0,0); You can play around with this property by using the 06-mesh-properties.html example. The next property on our list is one that we haven't talked about: scale. The name pretty much sums up what you can do with this property. You can scale the object along a specific axis. If you set the scale to values smaller than one, the object will shrink as shown: When you use values larger than one, the object will become larger as shown in the screenshot that follows: The last part of the mesh that we'll look at in this article is the translate functionality. With translate, you can also change the position of an object, but instead of defining the absolute position of where you want the object to be, you will define where the object should move to, relative to its current position. For instance, we've got a sphere object that is added to a scene and its position has been set to (1,2,3). Next, we will translate the object along its x axis by translateX(4). Its position will now be (5,2,3). If we want to restore the object to its original position we will set it to translateX(-4). In the 06-mesh-properties.html example, there is a menu tab called translate. From there you can experiment with this functionality. Just set the translate values for the x, y, and z axes, and click on the translate button. You'll see that the object is being moved to a new position based on these three values.
Read more
  • 0
  • 0
  • 14000

article-image-algorithmic-trading
Packt
29 Apr 2015
16 min read
Save for later

Algorithmic Trading

Packt
29 Apr 2015
16 min read
In this article by James Ma Weiming, author of the book Mastering Python for Finance , we will see how algorithmic trading automates the systematic trading process, where orders are executed at the best price possible based on a variety of factors, such as pricing, timing, and volume. Some brokerage firms may offer an application programming interface (API) as part of their service offering to customers who wish to deploy their own trading algorithms. For developing an algorithmic trading system, it must be highly robust and handle any point of failure during the order execution. Network configuration, hardware, memory management and speed, and user experience are some factors to be considered when designing a system in executing orders. Designing larger systems inevitably add complexity to the framework. As soon as a position in a market is opened, it is subjected to various types of risk, such as market risk. To preserve the trading capital as much as possible, it is important to incorporate risk management measures to the trading system. Perhaps the most common risk measure used in the financial industry is the value-at-risk (VaR) technique. We will discuss the beauty and flaws of VaR, and how it can be incorporated into our trading system that we will develop in this article. In this article, we will cover the following topics: An overview of algorithmic trading List of brokers and system vendors with public API Choosing a programming language for a trading system Setting up API access on Interactive Brokers (IB) trading platform Using the IbPy module to interact with IB Trader WorkStation (TWS) Introduction to algorithmic trading In the 1990s, exchanges had already begun to use electronic trading systems. By 1997, 44 exchanges worldwide used automated systems for trading futures and options with more exchanges in the process of developing automated technology. Exchanges such as the Chicago Board of Trade (CBOT) and the London International Financial Futures and Options Exchange (LIFFE) used their electronic trading systems as an after-hours complement to traditional open outcry trading in pits, giving traders 24-hour access to the exchange's risk management tools. With improvements in technology, technology-based trading became less expensive, fueling the growth of trading platforms that are faster and powerful. Higher reliability of order execution and lower rates of message transmission error deepened the reliance of technology by financial institutions. The majority of asset managers, proprietary traders, and market makers have since moved from the trading pits to electronic trading floors. As systematic or computerized trading became more commonplace, speed became the most important factor in determining the outcome of a trade. Quants utilizing sophisticated fundamental models are able to recompute fair values of trading products on the fly and execute trading decisions, enabling them to reap profits at the expense of fundamental traders using traditional tools. This gave way to the term high-frequency trading (HFT) that relies on fast computers to execute the trading decisions before anyone else can. HFT has evolved into a billion-dollar industry. Algorithmic trading refers to the automation of the systematic trading process, where the order execution is heavily optimized to give the best price possible. It is not part of the portfolio allocation process. Banks, hedge funds, brokerage firms, clearing firms, and trading firms typically have their servers placed right next to the electronic exchange to receive the latest market prices and to perform the fastest order execution where possible. They bring enormous trading volumes to the exchange. Anyone who wishes to participate in low-latency, high-volume trading activities, such as complex event processing or capturing fleeting price discrepancies, by acquiring exchange connectivity may do so in the form of co-location, where his or her server hardware can be placed on a rack right next to the exchange for a fee. The Financial Information Exchange (FIX) protocol is the industry standard for electronic communications with the exchange from the private server for direct market access (DMA) to real-time information. C++ is the common choice of programming language for trading over the FIX protocol, though other languages, such as .NET framework common language and Java can be used. Before creating an algorithmic trading platform, you would need to assess various factors, such as speed and ease of learning before deciding on a specific language for the purpose. Brokerage firms would provide a trading platform of some sort to their customers for them to execute orders on selected exchanges in return for the commission fees. Some brokerage firms may offer an API as part of their service offering to technically inclined customers who wish to run their own trading algorithms. In most circumstances, customers may also choose from a number of commercial trading platforms offered by third-party vendors. Some of these trading platforms may also offer API access to route orders electronically to the exchange. It is important to read the API documentation beforehand to understand the technical capabilities offered by your broker and to formulate an approach in developing an algorithmic trading system. List of trading platforms with public API The following table lists some brokers and trading platform vendors who have their API documentation publicly available: Broker/vendor URL Programming languages supported Interactive Brokers https://www.interactivebrokers.com/en/index.php?f=1325 C++, Posix C++, Java, and Visual Basic for ActiveX E*Trade https://developer.etrade.com Java, PHP, and C++ IG http://labs.ig.com/ REST, Java, FIX, and Microsoft .NET Framework 4.0 Tradier https://developer.tradier.com Java, Perl, Python, and Ruby TradeKing https://developers.tradeking.com Java, Node.js, PHP, R, and Ruby Cunningham trading systems http://www.ctsfutures.com/wiki/T4%20API%2040.MainPage.ashx Microsoft .NET Framework 4.0 CQG http://cqg.com/Products/CQG-API.aspx C#, C++, Excel, MATLAB, and VB.NET Trading technologies https://developer.tradingtechnologies.com Microsoft .NET Framework 4.0 OANDA http://developer.oanda.com REST, Java, FIX, and MT4 Which is the best programming language to use? With many choices of programming languages available to interface with brokers or vendors, the question that comes naturally to anyone starting out in algorithmic trading platform development is: which language should I use? Well, the short answer is that there is really no best programming language. How your product will be developed, the performance metrics to follow, the costs involved, latency threshold, risk measures, and the expected user interface are pieces of the puzzle to be taken into consideration. The risk manager, execution engine, and portfolio optimizer are some major components that will affect the design of your system. Your existing trading infrastructure, choice of operating system, programming language compiler capability, and available software tools poses further constraints on the system design, development, and deployment. System functionalities It is important to define the outcomes of your trading system. An outcome could be a research-based system that might be more concerned with obtaining high-quality data from data vendors, performing computations or running models, and evaluating a strategy through signal generation. Part of the research component might include a data-cleaning module or a backtesting interface to run a strategy with theoretical parameters over historical data. The CPU speed, memory size, and bandwidth are factors to be considered while designing our system. Another outcome could be an execution-based system that is more concerned with risk management and order handling features to ensure timely execution of multiple orders. The system must be highly robust and handle any point of failure during the order execution. As such, network configuration, hardware, memory management and speed, and user experience are some factors to be considered when designing a system in executing orders. A system may contain one or more of these functionalities. Designing larger systems inevitably add complexity to the framework. It is recommended that you choose one or more programming languages that can address and balance the development speed, ease of development, scalability, and reliability of your trading system. Algorithmic trading with Interactive Brokers and IbPy In this section, we will build a working algorithmic trading platform that will authenticate with Interactive Brokers (IB) and log in, retrieve the market data, and send orders. IB is one of the most popular brokers in the trading community and has a long history of API development. There are plenty of articles on the use of the API available on the Web. IB serves clients ranging from hedge funds to retail traders. Although the API does not support Python directly, Python wrappers such as IbPy are available to make the API calls to the IB interface. The IB API is unique to its own implementation, and every broker has its own API handling methods. Nevertheless, the documents and sample applications provided by your broker would demonstrate the core functionality of every API interface that can be easily integrated into an algorithmic trading system if designed properly. Getting Interactive Brokers' Trader WorkStation The official page for IB is https://www.interactivebrokers.com. Here, you can find a wealth of information regarding trading and investing for retail and institutional traders. In this section, we will take a look at how to get the Trader WorkStation X (TWS) installed and running on your local workstation before setting up an algorithmic trading system using Python. Note that we will perform simulated trading on a demonstration account. If your trading strategy turns out to be profitable, head to the OPEN AN ACCOUNT section of the IB website to open a live trading account. Rules, regulations, market data fees, exchange fees, commissions, and other conditions are subjected to the broker of your choice. In addition, market conditions are vastly different from the simulated environment. You are encouraged to perform extensive testing on your algorithmic trading system before running on live markets. The following key steps describe how to install TWS on your local workstation, log in to the demonstration account, and set it up for API use: From IB's official website, navigate to TRADING, and then select Standalone TWS. Choose the installation executable that is suitable for your local workstation. TWS runs on Java; therefore, ensure that Java runtime plugin is already installed on your local workstation. Refer to the following screenshot: When prompted during the installation process, choose Trader_WorkStation_X and IB Gateway options. The Trader WorkStation X (TWS) is the trading platform with full order management functionality. The IB Gateway program accepts and processes the API connections without any order management features of the TWS. We will not cover the use of the IB Gateway, but you may find it useful later. Select the destination directory on your local workstation where TWS will place all the required files, as shown in the following screenshot: When the installation is completed, a TWS shortcut icon will appear together with your list of installed applications. Double-click on the icon to start the TWS program. When TWS starts, you will be prompted to enter your login credentials. To log in to the demonstration account, type edemo in the username field and demouser in the password field, as shown in the following screenshot: Once we have managed to load our demo account on TWS, we can now set up its API functionality. On the toolbar, click on Configure: Under the Configuration tree, open the API node to reveal further options. Select Settings. Note that Socket port is 7496, and we added the IP address of our workstation housing our algorithmic trading system to the list of trusted IP addresses, which in this case is 127.0.0.1. Ensure that the Enable ActiveX and Socket Clients option is selected to allow the socket connections to TWS: Click on OK to save all the changes. TWS is now ready to accept orders and market data requests from our algorithmic trading system. Getting IbPy – the IB API wrapper IbPy is an add-on module for Python that wraps the IB API. It is open source and can be found at https://github.com/blampe/IbPy. Head to this URL and download the source files. Unzip the source folder, and use Terminal to navigate to this directory. Type python setup.py install to install IbPy as part of the Python runtime environment. The use of IbPy is similar to the API calls, as documented on the IB website. The documentation for IbPy is at https://code.google.com/p/ibpy/w/list. A simple order routing mechanism In this section, we will start interacting with TWS using Python by establishing a connection and sending out a market order to the exchange. Once IbPy is installed, import the following necessary modules into our Python script: from ib.ext.Contract import Contractfrom ib.ext.Order import Orderfrom ib.opt import Connection Next, implement the logging functions to handle calls from the server. The error_handler method is invoked whenever the API encounters an error, which is accompanied with a message. The server_handler method is dedicated to handle all the other forms of returned API messages. The msg variable is a type of an ib.opt.message object and references the method calls, as defined by the IB API EWrapper methods. The API documentation can be accessed at https://www.interactivebrokers.com/en/software/api/api.htm. The following is the Python code for the server_handler method: def error_handler(msg):print "Server Error:", msgdef server_handler(msg):print "Server Msg:", msg.typeName, "-", msg We will place a sample order of the stock AAPL. The contract specifications of the order are defined by the Contract class object found in the ib.ext.Contract module. We will create a method called create_contract that returns a new instance of this object: def create_contract(symbol, sec_type, exch, prim_exch, curr):contract = Contract()contract.m_symbol = symbolcontract.m_secType = sec_typecontract.m_exchange = exchcontract.m_primaryExch = prim_exchcontract.m_currency = currreturn contract The Order class object is used to place an order with TWS. Let's define a method called create_order that will return a new instance of the object: def create_order(order_type, quantity, action):order = Order()order.m_orderType = order_typeorder.m_totalQuantity = quantityorder.m_action = actionreturn order After the required methods are created, we can then begin to script the main functionality. Let's initialize the required variables: if __name__ == "__main__":client_id = 100order_id = 1port = 7496tws_conn = None Note that the client_id variable is our assigned integer that identifies the instance of the client communicating with TWS. The order_id variable is our assigned integer that identifies the order queue number sent to TWS. Each new order requires this value to be incremented sequentially. The port number has the same value as defined in our API settings of TWS earlier. The tws_conn variable holds the connection value to TWS. Let's initialize this variable with an empty value for now. Let's use a try block that encapsulates the Connection.create method to handle the socket connections to TWS in a graceful manner: try:# Establish connection to TWS.tws_conn = Connection.create(port=port,clientId=client_id)tws_conn.connect()# Assign error handling function.tws_conn.register(error_handler, 'Error')# Assign server messages handling function.tws_conn.registerAll(server_handler)finally:# Disconnect from TWSif tws_conn is not None:tws_conn.disconnect() The port and clientId parameter fields define this connection. After the connection instance is created, the connect method will try to connect to TWS. When the connection to TWS has successfully opened, it is time to register listeners to receive notifications from the server. The register method associates a function handler to a particular event. The registerAll method associates a handler to all the messages generated. This is where the error_handler and server_handler methods declared earlier will be used for this occasion. Before sending our very first order of 100 shares of AAPL to the exchange, we will call the create_contract method to create a new contract object for AAPL. Then, we will call the create_order method to create a new Order object, to go long 100 shares. Finally, we will call the placeOrder method of the Connection class to send out this order to TWS: # Create a contract for AAPL stock using SMART order routing.aapl_contract = create_contract('AAPL','STK','SMART','SMART','USD')# Go long 100 shares of AAPLaapl_order = create_order('MKT', 100, 'BUY')# Place order on IB TWS.tws_conn.placeOrder(order_id, aapl_contract, aapl_order) That's it! Let's run our Python script. We should get a similar output as follows: Server Error: <error id=-1, errorCode=2104, errorMsg=Market data farmconnection is OK:ibdemo>Server Response: error, <error id=-1, errorCode=2104, errorMsg=Marketdata farm connection is OK:ibdemo>Server Version: 75TWS Time at connection:20141210 23:14:17 CSTServer Msg: managedAccounts - <managedAccounts accountsList=DU15200>Server Msg: nextValidId - <nextValidId orderId=1>Server Error: <error id=-1, errorCode=2104, errorMsg=Market data farmconnection is OK:ibdemo>Server Msg: error - <error id=-1, errorCode=2104, errorMsg=Market datafarm connection is OK:ibdemo>Server Error: <error id=-1, errorCode=2107, errorMsg=HMDS data farmconnection is inactive but should be available upon demand.demohmds>Server Msg: error - <error id=-1, errorCode=2107, errorMsg=HMDS data farmconnection is inactive but should be available upon demand.demohmds> Basically, what the error messages say is that there are no errors and the connections are OK. Should the simulated order be executed successfully during market trading hours, the trade will be reflected in TWS: The full source code of our implementation is given as follows: """ A Simple Order Routing Mechanism """from ib.ext.Contract import Contractfrom ib.ext.Order import Orderfrom ib.opt import Connectiondef error_handler(msg):print "Server Error:", msgdef server_handler(msg):print "Server Msg:", msg.typeName, "-", msgdef create_contract(symbol, sec_type, exch, prim_exch, curr):contract = Contract()contract.m_symbol = symbolcontract.m_secType = sec_typecontract.m_exchange = exchcontract.m_primaryExch = prim_exchcontract.m_currency = currreturn contractdef create_order(order_type, quantity, action):order = Order()order.m_orderType = order_typeorder.m_totalQuantity = quantityorder.m_action = actionreturn orderif __name__ == "__main__":client_id = 1order_id = 119port = 7496tws_conn = Nonetry:# Establish connection to TWS.tws_conn = Connection.create(port=port,clientId=client_id)tws_conn.connect()# Assign error handling function.tws_conn.register(error_handler, 'Error')# Assign server messages handling function.tws_conn.registerAll(server_handler)# Create AAPL contract and send orderaapl_contract = create_contract('AAPL','STK','SMART','SMART','USD')# Go long 100 shares of AAPLaapl_order = create_order('MKT', 100, 'BUY')# Place order on IB TWS.tws_conn.placeOrder(order_id, aapl_contract, aapl_order)finally:# Disconnect from TWSif tws_conn is not None:tws_conn.disconnect() Summary In this article, we were introduced to the evolution of trading from the pits to the electronic trading platform, and learned how algorithmic trading came about. We looked at some brokers offering API access to their trading service offering. To help us get started on our journey in developing an algorithmic trading system, we used the TWS of IB and the IbPy Python module. In our first trading program, we successfully sent an order to our broker through the TWS API using a demonstration account. Resources for Article: Prototyping Arduino Projects using Python Python functions – Avoid repeating code Pentesting Using Python
Read more
  • 0
  • 0
  • 13995

article-image-getting-started-react-and-bootstrap
Packt
13 Dec 2016
18 min read
Save for later

Getting Started with React and Bootstrap

Packt
13 Dec 2016
18 min read
In this article by Mehul Bhat and Harmeet Singh, the authors of the book Learning Web Development with React and Bootstrap, you will learn to build two simple real-time examples: Hello World! with ReactJS A simple static form application with React and Bootstrap There are many different ways to build modern web applications with JavaScript and CSS, including a lot of different tool choices, and there is a lot of new theory to learn. This article will introduce you to ReactJS and Bootstrap, which you will likely come across as you learn about modern web application development. It will then take you through the theory behind the Virtual DOM and managing the state to create interactive, reusable and stateful components ourselves and then push it to a Redux architecture. You will have to learn very carefully about the concept of props and state management and where it belongs. If you want to learn code then you have to write code, in whichever way you feel comfortable. Try to create small components/code samples, which will give you more clarity/understanding of any technology. Now, let's see how this article is going to make your life easier when it comes to Bootstrap and ReactJS. Facebook has really changed the way we think about frontend UI development with the introduction of React. One of the main advantages of this component-based approach is that it is easy to understand, as the view is just a function of the properties and state. We're going to cover the following topics: Setting up the environment ReactJS setup Bootstrap setup Why Bootstrap? Static form example with React and Bootstrap (For more resources related to this topic, see here.) ReactJS React (sometimes called React.js or ReactJS) is an open-source JavaScript library that provides a view for data rendered as HTML. Components have been used typically to render React views that contain additional components specified as custom HTML tags. React gives you a trivial virtual DOM, powerful views without templates, unidirectional data flow, and explicit mutation. It is very methodical in updating the HTML document when the data changes; and provides a clean separation of components on a modern single-page application. From below example, we will have clear idea on normal HTML encapsulation and ReactJS custom HTML tag. JavaScript code: <section> <h2>Add your Ticket</h2> </section> <script> var root = document.querySelector('section').createShadowRoot(); root.innerHTML = '<style>h2{ color: red; }</style>' + '<h2>Hello World!</h2>'; </script> ReactJS code: var sectionStyle = { color: 'red' }; var AddTicket = React.createClass({ render: function() { return (<section><h2 style={sectionStyle}> Hello World!</h2></section>)} }) ReactDOM.render(<AddTicket/>, mountNode); As your app comes into existence and develops, it's advantageous to ensure that your components are used in the right manner. The React app consists of reusable components, which makes code reuse, testing, and separation of concerns easy. React is not only the V in MVC, it has stateful components (stateful components remembers everything within this.state). It handles mapping from input to state changes, and it renders components. In this sense, it does everything that an MVC does. Let's look at React's component life cycle and its different levels. Observe the following screenshot: React isn't an MVC framework; it's a library for building a composable user interface and reusable components. React used at Facebook in production and https://www.instagram.com/ is entirely built on React. Setting up the environment When we start to make an application with ReactJS, we need to do some setup, which just involves an HTML page and includes a few files. First, we create a directory (folder) called Chapter 1. Open it up in any code editor of your choice. Create a new file called index.html directly inside it and add the following HTML5 boilerplate code: <!doctype html> <html class="no-js" lang=""> <head> <meta charset="utf-8"> <title>ReactJS Chapter 1</title> </head> <body> <!--[if lt IE 8]> <p class="browserupgrade">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> to improve your experience.</p> <![endif]--> <!-- Add your site or application content here --> <p>Hello world! This is HTML5 Boilerplate.</p> </body> </html> This is a standard HTML page that we can update once we have included the React and Bootstrap libraries. Now we need to create couple of folders inside the Chapter 1 folder named images, css, and js (JavaScript) to make your application manageable. Once you have completed the folder structure it will look like this: Installing ReactJS and Bootstrap Once we have finished creating the folder structure, we need to install both of our frameworks, ReactJS and Bootstrap. It's as simple as including JavaScript and CSS files in your page. We can do this via a content delivery network (CDN), such as Google or Microsoft, but we are going to fetch the files manually in our application so we don't have to be dependent on the Internet while working offline. Installing React First, we have to go to this URL https://facebook.github.io/react/ and hit the download button. This will give you a ZIP file of the latest version of ReactJS that includes ReactJS library files and some sample code for ReactJS. For now, we will only need two files in our application: react.min.js and react-dom.min.js from the build directory of the extracted folder. Here are a few steps we need to follow: Copy react.min.js and react-dom.min.js to your project directory, the Chapter 1/js folder, and open up your index.html file in your editor. Now you just need to add the following script in your page's head tag section: <script type="text/js" src="js/react.min.js"></script> <script type="text/js" src="js/react-dom.min.js"></script> Now we need to include the compiler in our project to build the code because right now we are using tools such as npm. We will download the file from the following CDN path, https://cdnjs.cloudflare.com/ajax/libs/babel-core/5.8.23/browser.min.js, or you can give the CDN path directly. The Head tag section will look this: <script type="text/js" src="js/react.min.js"></script> <script type="text/js" src="js/react-dom.min.js"></script> <script type="text/js" src="js/browser.min.js"></script> Here is what the final structure of your js folder will look like: Bootstrap Bootstrap is an open source frontend framework maintained by Twitter for developing responsive websites and web applications. It includes HTML, CSS, and JavaScript code to build user interface components. It's a fast and easy way to develop a powerful mobile first user interface. The Bootstrap grid system allows you to create responsive 12-column grids, layouts, and components. It includes predefined classes for easy layout options (fixed-width and full width). Bootstrap has a dozen pre-styled reusable components and custom jQuery plugins, such as button, alerts, dropdown, modal, tooltip tab, pagination, carousal, badges, icons, and much more. Installing Bootstrap Now, we need to install Bootstrap. Visit http://getbootstrap.com/getting-started/#download and hit on the Download Bootstrap button. This includes the compiled and minified version of css and js for our app; we just need the CSS bootstrap.min.css and fonts folder. This style sheet will provide you with the look and feel of all of the components, and is responsive layout structure for our application. Previous versions of Bootstrap included icons as images but, in version 3, icons have been replaced as fonts. We can also customize the Bootstrap CSS style sheet as per the component used in your application: Extract the ZIP folder and copy the Bootstrap CSS from the css folder to your project folder CSS. Now copy the fonts folder of Bootstrap into your project root directory. Open your index.html in your editor and add this link tag in your head section: <link rel="stylesheet" href="css/bootstrap.min.css"> That's it. Now we can open up index.html again, but this time in your browser, to see what we are working with. Here is the code that we have written so far: <!doctype html> <html class="no-js" lang=""> <head> <meta charset="utf-8"> <title>ReactJS Chapter 1</title> <link rel="stylesheet" href="css/bootstrap.min.css"> <script type="text/javascript" src="js/react.min.js"></script> <script type="text/javascript" src="js/react-dom.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/babel-core/5.8.23/browser.min.js"></script> </head> <body> <!--[if lt IE 8]> <p class="browserupgrade">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> to improve your experience.</p> <![endif]--> <!-- Add your site or application content here --> </body> </html> Using React So now we've got the ReactJS and Bootstrap style sheet and in there we've initialized our app. Now let's start to write our first Hello World app using reactDOM.render(). The first argument of the ReactDOM.render method is the component we want to render and the second is the DOM node to which it should mount (append) to: ReactDOM.render( ReactElement element, DOMElement container, [function callback]) In order to translate it to vanilla JavaScript, we use wraps in our React code, <script type"text/babel">, tag that actually performs the transformation in the browser. Let's start out by putting one div tag in our body tag: <div id="hello"></div> Now, add the script tag with the React code: <script type="text/babel"> ReactDOM.render( <h1>Hello, world!</h1>, document.getElementById('hello') ); </script> Let's open the HTML page in your browser. If you see Hello, world! in your browser then we are on good track. In the preceding screenshot, you can see it shows the Hello, world! in your browser. That's great. We have successfully completed our setup and built our first Hello World app. Here is the full code that we have written so far: <!doctype html> <html class="no-js" lang=""> <head> <meta charset="utf-8"> <title>ReactJS Chapter 1</title> <link rel="stylesheet" href="css/bootstrap.min.css"> <script type="text/javascript" src="js/react.min.js"></script> <script type="text/javascript" src="js/react-dom.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/babel-core/5.8.23/browser.min.js"></script> </head> <body> <!--[if lt IE 8]> <p class="browserupgrade">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> to improve your experience.</p> <![endif]--> <!-- Add your site or application content here --> <div id="hello"></div> <script type="text/babel"> ReactDOM.render( <h1>Hello, world!</h1>, document.getElementById('hello') ); </script> </body> </html> Static form with React and Bootstrap We have completed our first Hello World app with React and Bootstrap and everything looks good and as expected. Now it's time do more and create one static login form, applying the Bootstrap look and feel to it. Bootstrap is a great way to make you app responsive grid system for different mobile devices and apply the fundamental styles on HTML elements with the inclusion of a few classes and div's. The responsive grid system is an easy, flexible, and quick way to make your web application responsive and mobile first that appropriately scales up to 12 columns per device and viewport size. First, let's start to make an HTML structure to follow the Bootstrap grid system. Create a div and add a className .container for (fixed width) and .container-fluid for (full width). Use the className attribute instead of using class: <div className="container-fluid"></div> As we know, class and for are discouraged as XML attribute names. Moreover, these are reserved words in many JavaScript libraries so, to have clear difference and identical understanding, instead of using class and for, we can use className and htmlFor create a div and adding the className row. The row must be placed within .container-fluid: <div className="container-fluid"> <div className="row"></div> </div> Now create columns that must be immediate children of a row: <div className="container-fluid"> <div className="row"> <div className="col-lg-6"></div> </div> </div> .row and .col-xs-4 are predefined classes that available for quickly making grid layouts. Add the h1 tag for the title of the page: <div className="container-fluid"> <div className="row"> <div className="col-sm-6"> <h1>Login Form</h1> </div> </div> </div> Grid columns are created by the given specified number of col-sm-* of 12 available columns. For example, if we are using a four column layout, we need to specify to col-sm-3 lead-in equal columns. Col-sm-* Small devices Col-md-* Medium devices Col-lg-* Large devices We are using the col-sm-* prefix to resize our columns for small devices. Inside the columns, we need to wrap our form elements label and input tags into a div tag with the form-group class: <div className="form-group"> <label for="emailInput">Email address</label> <input type="email" className="form-control" id="emailInput" placeholder="Email"/> </div> Forget the style of Bootstrap; we need to add the form-control class in our input elements. If we need extra padding in our label tag then we can add the control-label class on the label. Let's quickly add the rest of the elements. I am going to add a password and submit button. In previous versions of Bootstrap, form elements were usually wrapped in an element with the form-actions class. However, in Bootstrap 3, we just need to use the same form-group instead of form-actions. Here is our complete HTML code: <div className="container-fluid"> <div className="row"> <div className="col-lg-6"> <form> <h1>Login Form</h1> <hr/> <div className="form-group"> <label for="emailInput">Email address</label> <input type="email" className="form-control" id="emailInput" placeholder="Email"/> </div> <div className="form-group"> <label for="passwordInput">Password</label> <input type="password" className="form- control" id="passwordInput" placeholder="Password"/> </div> <button type="submit" className="btn btn-default col-xs-offset-9 col-xs-3">Submit</button> </form> </div> </div> </div> Now create one object inside the var loginFormHTML script tag and assign this HTML them: Var loginFormHTML = <div className="container-fluid"> <div className="row"> <div className="col-lg-6"> <form> <h1>Login Form</h1> <hr/> <div className="form-group"> <label for="emailInput">Email address</label> <input type="email" className="form-control" id="emailInput" placeholder="Email"/> </div> <div className="form-group"> <label for="passwordInput">Password</label> <input type="password" className="form- control" id="passwordInput" placeholder="Password"/> </div> <button type="submit" className="btn btn-default col-xs-offset-9 col-xs-3">Submit</button> </form> </div> </div> We will pass this object in the React.DOM()method instead of directly passing the HTML: ReactDOM.render(LoginformHTML,document.getElementById('hello')); Our form is ready. Now let's see how it looks in the browser: The compiler is unable to parse our HTML because we have not enclosed one of the div tags properly. You can see in our HTML that we have not closed the wrapper container-fluid at the end. Now close the wrapper tag at the end and open the file again in your browser. Note: Whenever you hand-code (write) your HTML code, please double check your "Start Tag" and "End Tag". It should be written/closed properly, otherwise it will break your UI/frontend look and feel. Here is the HTML after closing the div tag: <!doctype html> <html class="no-js" lang=""> <head> <meta charset="utf-8"> <title>ReactJS Chapter 1</title> <link rel="stylesheet" href="css/bootstrap.min.css"> <script type="text/javascript" src="js/react.min.js"></script> <script type="text/javascript" src="js/react-dom.min.js"></script> <script src="js/browser.min.js"></script> </head> <body> <!-- Add your site or application content here --> <div id="loginForm"></div> <script type="text/babel"> var LoginformHTML = <div className="container-fluid"> <div className="row"> <div className="col-lg-6"> <form> <h1>Login Form</h1> <hr/> <div className="form-group"> <label for="emailInput">Email address</label> <input type="email" className="form-control" id="emailInput" placeholder="Email"/> </div> <div className="form-group"> <label for="passwordInput">Password</label> <input type="password" className="form-control" id="passwordInput" placeholder="Password"/> </div> <button type="submit" className="btn btn-default col-xs-offset-9 col-xs-3">Submit</button> </form> </div> </div> </div> ReactDOM.render(LoginformHTML,document.getElementById('loginForm'); </script> </body> </html> Now, you can check your page on browser and you will be able to see the form with below shown look and feel. Now it's working fine and looks good. Bootstrap also provides two additional classes to make your elements smaller and larger: input-lg and input-sm. You can also check the responsive behavior by resizing the browser. That's look great. Our small static login form application is ready with responsive behavior. Some of the benefits are: Rendering your component is very easy Reading component's code would be very easy with help of JSX JSX will also help you to check your layout as well as components plug-in with each other You can test your code easily and it also allows other tools integration for enhancement As we know, React is view layer, you can also use it with other JavaScript frameworks Summary Our simple static login form application and Hello World examples are looking great and working exactly how they should, so let's recap what we've learned in the this article. To begin with, we saw just how easy it is to get ReactJS and Bootstrap installed with the inclusion of JavaScript files and a style sheet. We also looked at how the React application is initialized and started building our first form application. The Hello World app and form application which we have created demonstrates some of React's and Bootstrap's basic features such as the following: ReactDOM Render Browserify Bootstrap With Bootstrap, we worked towards having a responsive grid system for different mobile devices and applied the fundamental styles of HTML elements with the inclusion of a few classes and divs. We also saw the framework's new mobile-first responsive design in action without cluttering up our markup with unnecessary classes or elements. Resources for Article: Further resources on this subject: Getting Started with ASP.NET Core and Bootstrap 4 [article] Frontend development with Bootstrap 4 [article] Gearing Up for Bootstrap 4 [article]
Read more
  • 0
  • 0
  • 13983

article-image-building-our-first-app-7-minute-workout
Packt
14 Nov 2016
27 min read
Save for later

Building Our First App – 7 Minute Workout

Packt
14 Nov 2016
27 min read
In this article by, Chandermani Arora and Kevin Hennessy, the authors of the book Angular 2 By Example, we will build a new app in Angular, and in the process, develop a better understanding of the framework. This app will also help us explore some new capabilities of the framework. (For more resources related to this topic, see here.) The topics that we will cover in this article include the following: 7 Minute Workout problem description: We detail the functionality of the app that we build. Code organization: For our first real app, we will try to explain how to organize code, specifically Angular code. Designing the model: One of the building blocks of our app is its model. We design the app model based on the app's requirements. Understanding the data binding infrastructure: While building the 7 Minute Workout view, we will look at the data binding capabilities of the framework, which include property, attribute, class, style, and event bindings. Let's get started! The first thing we will do is define the scope of our 7 Minute Workout app. What is 7 Minute Workout? We want everyone reading this article to be physically fit. Our purpose is to simulate your grey matter. What better way to do it than to build an app that targets physical fitness! 7 Minute Workout is an exercise/workout plan that requires us to perform a set of twelve exercises in quick succession within the seven minute time span. 7 Minute Workout has become quite popular due to its benefits and the short duration of the workout. We cannot confirm or refute the claims but doing any form of strenuous physical activity is better than doing nothing at all. If you are interested to know more about the workout, then check out http://well.blogs.nytimes.com/2013/05/09/the-scientific-7-minute-workout/. The technicalities of the app include performing a set of 12 exercises, dedicating 30 seconds for each of the exercises. This is followed by a brief rest period before starting the next exercise. For the app that we are building, we will be taking rest periods of 10 seconds each. So, the total duration comes out be a little more than 7 minutes. Once the 7 Minute Workout app ready, it will look something like this: Downloading the code base The code for this app can be downloaded from the GitHub site https://github.com/chandermani/angular2byexample dedicated to this article. Since we are building the app incrementally, we have created multiple checkpoints that map to GitHub branches such as checkpoint2.1, checkpoint2.2, and so on. During the narration, we will highlight the branch for reference. These branches will contain the work done on the app up to that point in time. The 7 Minute Workout code is available inside the repository folder named trainer. So let's get started! Setting up the build Remember that we are building on a modern platform for which browsers still lack support. Therefore, directly referencing script files in HTML is out of question (while common, it's a dated approach that we should avoid anyway). The current browsers do not understand TypeScript; as a matter of fact, even ES 2015 (also known as ES6) is not supported. This implies that there has to be a process that converts code written in TypeScript into standard JavaScript (ES5), which browsers can work with. Hence, having a build setup for almost any Angular 2 app becomes imperative. Having a build process may seem like overkill for a small application, but it has some other advantages as well. If you are a frontend developer working on the web stack, you cannot avoid Node.js. This is the most widely used platform for Web/JavaScript development. So, no prizes for guessing that the Angular 2 build setup too is supported over Node.js with tools such as Grunt, Gulp, JSPM, and webpack. Since we are building on the Node.js platform, install Node.js before starting. While there are quite elaborate build setup options available online, we go for a minimal setup using Gulp. The reason is that there is no one size fits all solution out there. Also, the primary aim here is to learn about Angular 2 and not to worry too much about the intricacies of setting up and running a build. Some of the notable starter sites plus build setups created by the community are as follows: Start site Location angular2-webpack-starter http://bit.ly/ng2webpack angular2-seed http://bit.ly/ng2seed angular-cli— It allows us to generate the initial code setup, including the build configurations, and has good scaffolding capabilities too. http://bit.ly/ng2-cli A natural question arises if you are very new to Node.js or the overall build process: what does a typical Angular build involve? It depends! To get an idea about this process, it would be beneficial if we look at the build setup defined for our app. Let's set up the app's build locally then. Follow these steps to have the boilerplate Angular 2 app up and running: Download the base version of this app from http://bit.ly/ng2be-base and unzip it to a location on your machine. If you are familiar with how Git works, you can just checkout the branch base: git checkout base This code serves as the starting point for our app. Navigate to the trainer folder from command line and execute these commands: npm i -g gulp npm install The first command installs Gulp globally so that you can invoke the Gulp command line tool from anywhere and execute Gulp tasks. A Gulp task is an activity that Gulp performs during the build execution. If we look at the Gulp build script (which we will do shortly), we realize that it is nothing but a sequence of tasks performed whenever a build occurs. The second command installs the app's dependencies (in the form of npm packages). Packages in the Node.js world are third-party libraries that are either used by the app or support the app's building process. For example, Gulp itself is a Node.js package. The npm is a command-line tool for pulling these packages from a central repository. Once Gulp is installed and npm pulls dependencies from the npm store, we are ready to build and run the application. From the command line, enter the following command: gulp play This compiles and runs the app. If the build process goes fine, the default browser window/tab will open with a rudimentary "Hello World" page (http://localhost:9000/index.html). We are all set to begin developing our app in Angular 2! But before we do that, it would be interesting to know what has happened under the hood. The build internals Even if you are new to Gulp, looking at gulpfile.js gives you a fair idea about what the build process is doing. A Gulp build is a set of tasks performed in a predefined order. The end result of such a process is some form of package code that is ready to be run. And if we are building our apps using TypeScript/ES2015 or some other similar language that browsers do not understand natively, then we need an additional build step, called transpilation. Code transpiling As it stands in 2016, browsers still cannot run ES2015 code. While we are quick to embrace languages that hide the not-so-good parts of JavaScript (ES5), we are still limited by the browser's capabilities. When it comes to language features, ES5 is still the safest bet as all browsers support it. Clearly, we need a mechanism to convert our TypeScript code into plain JavaScript (ES5). Microsoft has a TypeScript compiler that does this job. The TypeScript compiler takes the TypeScript code and converts it into ES5-format code that can run in all browsers. This process is commonly referred to as transpiling, and since the TypeScript compiler does it, it's called a transpiler. Interestingly, transpilation can happen at both build/compile time and runtime: Build-time transpilation: Transpilation as part of the build process takes the script files (in our case, TypeScript .ts files) and compiles them into plain JavaScript. Our build setup uses build-time transpilation. Runtime transpilation: This happens in the browser at runtime. We include the raw language-specific script files (.ts in our case), and the TypeScript compiler—which is loaded in the browser beforehand—compiles these script files on the fly.  While runtime transpilation simplifies the build setup process, as a recommendation, it should be limited to development workflows only, considering the additional performance overheard involved in loading the transpiler and transpiling the code on the fly. The process of transpiling is not limited to TypeScript. Every language targeted towards the Web such as CoffeeScript, ES2015, or any other language that is not inherently understood by a browser needs transpilation. There are transpilers for most languages, and the prominent ones (other than TypeScript) are tracuer and babel. To compile TypeScript files, we can install the TypeScript compiler manually from the command line using this: npm install -g typescript Once installed, we can compile any TypeScript file into ES5 format using the compiler (tsc.exe). But for our build setup, this process is automated using the ts2js Gulp task (check out gulpfile.js). And if you are wondering when we installed TypeScript… well, we did it as part of the npm install step, when setting up the code for the first time. The gulp-typescript package downloads the TypeScript compiler as a dependency. With this basic understanding of transpilation, we can summarize what happens with our build setup: The gulp play command kicks off the build process. This command tells Gulp to start the build process by invoking the play task. Since the play task has a dependency on the ts2js task, ts2js is executed first. The ts2js compiles the TypeScript files (.ts) located in src folder and outputs them to the dist folder at the root. Post build, a static file server is started that serves all the app files, including static files (images, videos, and HTML) and script files (check gulp.play task). Thenceforth, the build process keeps a watch on any script file changes (the gulp.watch task) you make and recompiles the code on the fly. livereload also has been set up for the app. Any changes to the code refresh the browser running the app automatically. In case browser refresh fails, we can always do a manual refresh. This is a rudimentary build setup required to run an Angular app. For complex build requirements, we can always look at the starter/seed projects that have a more complete and robust build setup, or build something of our own. Next let's look at the boilerplate app code already there and the overall code organization. Organizing code This is how we are going to organize our code and other assets for the app: The trainer folder is the root folder for the app and it has a folder (static) for the static content (such as images, CSS, audio files, and others) and a folder (src) for the app's source code. The organization of the app's source code is heavily influenced by the design of Angular and the Angular style guide (http://bit.ly/ng2-style-guide) released by the Angular team. The components folder hosts all the components that we create. We will be creating subfolders in this folder for every major component of the application. Each component folder will contain artifacts related to that component, which includes its template, its implementation and other related item. We will also keep adding more top-level folders (inside the src folder) as we build the application. If we look at the code now, the components/app folder has defined a root level component TrainerAppComponent and root level module AppModule. bootstrap.ts contains code to bootstrap/load the application module (AppModule). 7 Minute Workout uses Just In Time (JIT) compilation to compile Angular views. This implies that views are compiled just before they are rendered in the browser. Angular has a compiler running in the browser that compiles these views. Angular also supports the Ahead Of Time (AoT) compilation model. With AoT, the views are compiled on the server side using a server version of the Angular compiler. The views returned to the browser are precompiled and ready to be used. For 7 Minute Workout, we stick to the JIT compilation model just because it is easy to set up as compared to AoT, which requires server-side tweaks and package installation. We highly recommend that you use AoT compilation for production apps due the numerous benefits it offers. AoT can improve the application's initial load time and reduce its size too. Look at the AoT platform documentation (cookbook) at http://bit.ly/ng2-aot to understand how AoT compilation can benefit you. Time to start working on our first focus area, which is the app's model! The 7 Minute Workout model Designing the model for this app requires us to first detail the functional aspects of the 7 Minute Workout app, and then derive a model that satisfies those requirements. Based on the problem statement defined earlier, some of the obvious requirements are as follows: Being able to start the workout. Providing a visual clue about the current exercise and its progress. This includes the following: Providing a visual depiction of the current exercise Providing step-by-step instructions on how to do a specific exercise The time left for the current exercise Notifying the user when the workout ends. Some other valuable features that we will add to this app are as follows: The ability to pause the current workout. Providing information about the next exercise to follow. Providing audio clues so that the user can perform the workout without constantly looking at the screen. This includes: A timer click sound Details about the next exercise Signaling that the exercise is about to start Showing related videos for the exercise in progress and the ability to play them. As we can see, the central theme for this app is workout and exercise. Here, a workout is a set of exercises performed in a specific order for a particular duration. So, let's go ahead and define the model for our workout and exercise. Based on the requirements just mentioned, we will need the following details about an exercise: The name. This should be unique. The title. This is shown to the user. The description of the exercise. Instructions on how to perform the exercise. Images for the exercise. The name of the audio clip for the exercise. Related videos. With TypeScript, we can define the classes for our model. Create a folder called workout-runner inside the src/components folder and copy the model.ts file from the checkpoint2.1 branch folder workout-runner(http://bit.ly/ng2be-2-1-model-ts) to the corresponding local folder. model.ts contains the model definition for our app. The Exercise class looks like this: export class Exercise { constructor( public name: string, public title: string, public description: string, public image: string, public nameSound?: string, public procedure?: string, public videos?: Array<string>) { } } TypeScript Tips Passing constructor parameters with public or private is a shorthand for creating and initializing class members at one go. The ? suffix after nameSound, procedure, and videos implies that these are optional parameters. For the workout, we need to track the following properties: The name. This should be unique. The title. This is shown to the user. The exercises that are part of the workout. The duration for each exercise. The rest duration between two exercises. So, the model class (WorkoutPlan) looks like this: export class WorkoutPlan { constructor( public name: string, public title: string, public restBetweenExercise: number, public exercises: ExercisePlan[], public description?: string) { } totalWorkoutDuration(): number { … } } The totalWorkoutDuration function returns the total duration of the workout in seconds. WorkoutPlan has a reference to another class in the preceding definition—ExercisePlan. It tracks the exercise and the duration of the exercise in a workout, which is quite apparent once we look at the definition of ExercisePlan: export class ExercisePlan { constructor( public exercise: Exercise, public duration: number) { } } These three classes constitute our base model, and we will decide in the future whether or not we need to extend this model as we start implementing the app's functionality. Since we have started with a preconfigured and basic Angular app, you just need to understand how this app bootstrapping is occurring. App bootstrapping Look at the src folder. There is a bootstrap.ts file with only the execution bit (other than imports): platformBrowserDynamic().bootstrapModule(AppModule); The boostrapModule function call actually bootstraps the application by loading the root module, AppModule. The process is triggered by this call in index.html: System.import('app').catch(console.log.bind(console)); The System.import statement sets off the app bootstrapping process by loading the first module from bootstrap.ts. Modules defined in the context of Angular 2, (using @NgModule decorator) are different from modules SystemJS loads. SystemJS modules are JavaScript modules, which can be in different formats adhering to CommonJS, AMD, or ES2015 specifications. Angular modules are constructs used by Angular to segregate and organize its artifacts. Unless the context of discussion is SystemJS, any reference to module implies Angular module. Now we have details on how SystemJS loads our Angular app. App loading with SystemJS SystemJS starts loading the JavaScript module with the call to System.import('app') in index.html. SystemJS starts by loading bootstrap.ts first. The imports defined inside bootstrap.ts cause SystemJS to then load the imported modules. If these module imports have further import statements, SystemJS loads them too, recursively. And finally the platformBrowserDynamic().bootstrapModule(AppModule); function gets executed once all the imported modules are loaded. For the SystemJS import function to work, it needs to know where the module is located. We define this in the file, systemjs.config.js, and reference it in index.html, before the System.import script: <script src="systemjs.config.js"></script> This configuration file contains all of the necessary configuration for SystemJS to work correctly. Open systemjs.config.js, the app parameter to import function points to a folder dist as defined on the map object: var map = { 'app': 'dist', ... } And the next variable, packages, contains settings that hint SystemJS how to load a module from a package when no filename/extension is specified. For app, the default module is bootstrap.js: var packages = { 'app': { main: 'bootstrap.js', defaultExtension: 'js' }, ... }; Are you wondering what the dist folder has to do with our application? Well, this is where our transpiled scripts end up.  As we build our app in TypeScript, the TypeScript compiler converts these .ts script files in the src folder to JavaScript modules and deposits them into the dist folder. SystemJS then loads these compiled JavaScript modules. The transpiled code location has been configured as part of the build definition in gulpfile.js. Look for this excerpt in gulpfile.ts: return tsResult.js .pipe(sourcemaps.write()) .pipe(gulp.dest('dist')) The module specification used by our app can again be verified in gulpfile.js. Take a look at this line: noImplicitAny: true, module: 'system', target: 'ES5', These are TypeScript compiler options, with one being module, that is, the target module definition format. The system module type is a new module format designed to support the exact semantics of ES2015 modules within ES5. Once the scripts are transpiled and the module definitions created (in the target format), SystemJS can load these modules and their dependencies. It's time to get into the thick of action; let's build our first component. Our first component – WorkoutRunnerComponent To implement the WorkoutRunnerComponent, we need to outline the behavior of the application. What we are going to do in the WorkoutRunnerComponent implementation is as follows: Start the workout. Show the workout in progress and show the progress indicator. After the time elapses for an exercise, show the next exercise. Repeat this process until all the exercises are over. Let's start with the implementation. The first thing that we will create is the WorkoutRunnerComponent implementation. Open workout-runner folder in the src/components folder and add a new code file called workout-runner.component.ts to it. Add this chunk of code to the file: import {WorkoutPlan, ExercisePlan, Exercise} from './model' export class WorkoutRunnerComponent { } The import module declaration allows us to reference the classes defined in the model.ts file in WorkoutRunnerComponent. We first need to set up the workout data. Let's do that by adding a constructor and related class properties to the WorkoutRunnerComponent class: workoutPlan: WorkoutPlan; restExercise: ExercisePlan; constructor() { this.workoutPlan = this.buildWorkout(); this.restExercise = new ExercisePlan( new Exercise("rest", "Relax!", "Relax a bit", "rest.png"), this.workoutPlan.restBetweenExercise); } The buildWorkout on WorkoutRunnerComponent sets up the complete workout, as we will see shortly. We also initialize a restExercise variable to track even the rest periods as exercise (note that restExercise is an object of type ExercisePlan). The buildWorkout function is a lengthy function, so it's better if we copy the implementation from the workout runner's implementation available in Git branch checkpoint2.1 (http://bit.ly/ng2be-2-1-workout-runner-component-ts). The buildWorkout code looks like this: buildWorkout(): WorkoutPlan { let workout = new WorkoutPlan("7MinWorkout", "7 Minute Workout", 10, []); workout.exercises.push( new ExercisePlan( new Exercise( "jumpingJacks", "Jumping Jacks", "A jumping jack or star jump, also called side-straddle hop is a physical jumping exercise.", "JumpingJacks.png", "jumpingjacks.wav", `Assume an erect position, with feet together and arms at your side. …`, ["dmYwZH_BNd0", "BABOdJ-2Z6o", "c4DAnQ6DtF8"]), 30)); // (TRUNCATED) Other 11 workout exercise data. return workout; } This code builds the WorkoutPlan object and pushes the exercise data into the exercises array (an array of ExercisePlan objects), returning the newly built workout. The initialization is complete; now, it's time to actually implement the start workout. Add a start function to the WorkoutRunnerComponent implementation, as follows: start() { this.workoutTimeRemaining = this.workoutPlan.totalWorkoutDuration(); this.currentExerciseIndex = 0; this.startExercise(this.workoutPlan.exercises[this.currentExerciseIndex]); } Then declare the new variables used in the function at the top, with other variable declarations: workoutTimeRemaining: number; currentExerciseIndex: number; The workoutTimeRemaining variable tracks the total time remaining for the workout, and currentExerciseIndex tracks the currently executing exercise index. The call to startExercise actually starts an exercise. This how the code for startExercise looks: startExercise(exercisePlan: ExercisePlan) { this.currentExercise = exercisePlan; this.exerciseRunningDuration = 0; let intervalId = setInterval(() => { if (this.exerciseRunningDuration >= this.currentExercise.duration) { clearInterval(intervalId); } else { this.exerciseRunningDuration++; } }, 1000); } We start by initializing currentExercise and exerciseRunningDuration. The currentExercise variable tracks the exercise in progress and exerciseRunningDuration tracks its duration. These two variables also need to be declared at the top: currentExercise: ExercisePlan; exerciseRunningDuration: number; We use the setInterval JavaScript function with a delay of 1 second (1,000 milliseconds) to track the exercise progress by incrementing exerciseRunningDuration. The setInterval invokes the callback every second. The clearInterval call stops the timer once the exercise duration lapses. TypeScript Arrow functions The callback parameter passed to setInterval (()=>{…}) is a lambda function (or an arrow function in ES 2015). Lambda functions are short-form representations of anonymous functions, with added benefits. You can learn more about them at https://basarat.gitbooks.io/typescript/content/docs/arrow-functions.html. As of now, we have a WorkoutRunnerComponent class. We need to convert it into an Angular component and define the component view. Add the import for Component and a component decorator (highlighted code): import {WorkoutPlan, ExercisePlan, Exercise} from './model' import {Component} from '@angular/core'; @Component({ selector: 'workout-runner', template: ` <pre>Current Exercise: {{currentExercise | json}}</pre> <pre>Time Left: {{currentExercise.duration- exerciseRunningDuration}}</pre>` }) export class WorkoutRunnerComponent { As you already know how to create an Angular component. You understand the role of the @Component decorator, what selector does, and how the template is used. The JavaScript generated for the @Component decorator contains enough metadata about the component. This allows Angular framework to instantiate the correct component at runtime. String enclosed in backticks (` `) are a new addition to ES2015. Also called template literals, such string literals can be multi line and allow expressions to be embedded inside (not to be confused with Angular expressions). Look at the MDN article here at http://bit.ly/template-literals for more details. The preceding template HTML will render the raw ExercisePlan object and the exercise time remaining. It has an interesting expression inside the first interpolation: currentExercise | json. The currentExercise property is defined in WorkoutRunnerComponent, but what about the | symbol and what follows it (json)? In the Angular 2 world, it is called a pipe. The sole purpose of a pipe is to transform/format template data. The json pipe here does JSON data formatting. A general sense of what the json pipe does, we can remove the json pipe plus the | symbol and render the template; we are going to do this next. As our app currently has only one module (AppModule), we add the WorkoutRunnerComponent declaration to it. Update app.module.ts by adding the highlighted code: import {WorkoutRunnerComponent} from '../workout-runner/workout-runner.component'; @NgModule({ imports: [BrowserModule], declarations: [TrainerAppComponent, WorkoutRunnerComponent], Now WorkoutRunnerComponent can be referenced in the root component so that it can be rendered. Modify src/components/app/app.component.ts as highlighted in the following code: @Component({ ... template: ` <div class="navbar ...> ... </div> <div class="container ...> <workout-runner></workout-runner> </div>` }) We have changed the root component template and added the workout-runner element to it. This will render the WorkoutRunnerComponent inside our root component. While the implementation may look complete there is a crucial piece missing. Nowhere in the code do we actually start the workout. The workout should start as soon as we load the page. Component life cycle hooks are going to rescue us! Component life cycle hooks Life of an Angular component is eventful. Components get created, change state during their lifetime and finally they are destroyed. Angular provides some life cycle hooks/functions that the framework invokes (on the component) when such event occurs. Consider these examples: When component is initialized, Angular invokes ngOnInit When a component's input properties change, Angular invokes ngOnChanges When a component is destroyed, Angular invokes ngOnDestroy As developers, we can tap into these key moments and perform some custom logic inside the respective component. Angular has TypeScript interfaces for each of these hooks that can be applied to the component class to clearly communicate the intent. For example: class WorkoutRunnerComponent implements OnInit { ngOnInit (){ ... } ... The interface name can be derived by removing the prefix ng from the function names. The hook we are going to utilize here is ngOnInit. The ngOnInit function gets fired when the component's data-bound properties are initialized but before the view initialization starts. Add the ngOnInit function to the WorkoutRunnerComponent class with a call to start the workout: ngOnInit() { this.start(); } And implement the OnInit interface on WorkoutRunnerComponent; it defines the ngOnInit method: import {Component,OnInit} from '@angular/core'; … export class WorkoutRunnerComponent implements OnInit { There are a number of other life cycle hooks, including ngOnDestroy, ngOnChanges, and ngAfterViewInit, that components support; but we are not going to dwell into any of them here. Look at the developer guide (http://bit.ly/ng2-lifecycle) on Life cycle Hooks to learn more about other such hooks. Time to run our app! Open the command line, navigate to the trainer folder, and type this line: gulp play If there are no compilation errors and the browser automatically loads the app (http://localhost:9000/index.html), we should see the following output: The model data updates with every passing second! Now you'll understand why interpolations ({{ }}) are a great debugging tool. This will also be a good time to try rendering currentExercise without the json pipe (use {{currentExercise}}), and see what gets rendered. We are not done yet! Wait long enough on the index.html page and you will realize that the timer stops after 30 seconds. The app does not load the next exercise data. Time to fix it! Update the code inside the setInterval if condition: if (this.exerciseRunningDuration >= this.currentExercise.duration) { clearInterval(intervalId); let next: ExercisePlan = this.getNextExercise(); if (next) { if (next !== this.restExercise) { this.currentExerciseIndex++; } this.startExercise(next); } else { console.log("Workout complete!"); } } The if condition if (this.exerciseRunningDuration >= this.currentExercise.duration) is used to transition to the next exercise once the time duration of the current exercise lapses. We use getNextExercise to get the next exercise and call startExercise again to repeat the process. If no exercise is returned by the getNextExercise call, the workout is considered complete. During exercise transitioning, we increment currentExerciseIndex only if the next exercise is not a rest exercise. Remember that the original workout plan does not have a rest exercise. For the sake of consistency, we have created a rest exercise and are now swapping between rest and the standard exercises that are part of the workout plan. Therefore, currentExerciseIndex does not change when the next exercise is rest. Let's quickly add the getNextExercise function too. Add the function to the WorkoutRunnerComponent class: getNextExercise(): ExercisePlan { let nextExercise: ExercisePlan = null; if (this.currentExercise === this.restExercise) { nextExercise = this.workoutPlan.exercises[this.currentExerciseIndex + 1]; } else if (this.currentExerciseIndex < this.workoutPlan.exercises.length - 1) { nextExercise = this.restExercise; } return nextExercise; } The WorkoutRunnerComponent.getNextExercise returns the next exercise that needs to be performed. Note that the returned object for getNextExercise is an ExercisePlan object that internally contains the exercise details and the duration for which the exercise runs. The implementation is quite self-explanatory. If the current exercise is rest, take the next exercise from the workoutPlan.exercises array (based on currentExerciseIndex); otherwise, the next exercise is rest, given that we are not on the last exercise (the else if condition check). With this, we are ready to test our implementation. So go ahead and refresh index.html. Exercises should flip after every 10 or 30 seconds. Great! The current build setup automatically compiles any changes made to the script files when the files are saved; it also refreshes the browser post these changes. But just in case the UI does not update or things do not work as expected, refresh the browser window. If you are having a problem with running the code, look at the Git branch checkpoint2.1 for a working version of what we have done thus far. Or if you are not using Git, download the snapshot of checkpoint2.1 (a zip file) from http://bit.ly/ng2be-checkpoint2-1. Refer to the README.md file in the trainer folder when setting up the snapshot for the first time. We have are done with controller. Summary We started this article with the aim of creating an Angular app which is  complex. The 7 Minute Workout app fitted the bill, and you learned a lot about the Angular framework while building this app. We started by defining the functional specifications of the 7 Minute Workout app. We then focused our efforts on defining the code structure for the app. To build the app, we started off by defining the model of the app. Once the model was in place, we started the actual implementation, by building an Angular component. Angular components are nothing but classes that are decorated with a framework-specific decorator, @Component. We now have a basic 7 Minute Workout app. For a better user experience, we have added a number of small enhancements to it too, but we are still missing some good-to-have features that would make our app more usable. Resources for Article: Further resources on this subject: Angular.js in a Nutshell [article] Introduction to JavaScript [article] Create Your First React Element [article] <hr noshade="noshade" size="1"
Read more
  • 0
  • 0
  • 13971

article-image-tesla-autonomy-day-takeaways-full-self-driving-computer-robotaxis-launching-next-year-and-more
Bhagyashree R
24 Apr 2019
6 min read
Save for later

Tesla Autonomy Day takeaways: Full Self-Driving computer, Robotaxis launching next year, and more

Bhagyashree R
24 Apr 2019
6 min read
This Monday, Tesla’s “Autonomy Investor Day” kickstarted at its headquarters in Palo Alto. At this invitation-only event, Elon Musk, the CEO of Tesla, with his fellow executives, talked about its new microchip, robotaxis hitting the road by next year, and more. Here are some of the key takeaways from the event: The Full Self-Driving (FSD) computer Tesla shared details of its new custom chip, Full Self-Driving (FSD) computer, previously known as Autopilot Hardware 3.0. Elon Musk, the CEO of Tesla, believes that the FSD computer is “the best chip in the world…objectively.” Tesla replaced Nvidia’s Autopilot 2.5 computer with its own custom chip for Model S and Model X about a month ago. For Model 3 vehicles this change happened about 10 days ago. Musk said, “All cars being produced all have the hardware necessary — computer and otherwise — for full self-driving. All you need to do is improve the software.” FSD is a high-performance, special-purpose chip built by Samsung with main focus on autonomy and safety. It comes with a factor of 21 improvements in frame per second processing as compared to the previous generation Tesla Autopilot hardware, which was powered by Nvidia hardware. The company further shared that retrofits will be offered to current Tesla owners who bought the ‘Full Self-Driving package’ in the next few months. Here’s the new Tesla FSD computer: Credits: Tesla Musk shared that the company has already started working on a next-generation chip. The design of FSD was completed within 2 years of time and Tesla is now about halfway through the design of the next-generation chip. Musk’s claims of building the best chip can be taken with a pinch of salt as it could surely upset some engineers from Nvidia, Mobileye, and other companies who have been in the chip manufacturing market for a long time. Nvidia, in a blog post, along with applauding Tesla for its FSD computer, highlighted “few inaccuracies” in the comparison made by Musk during the event: “It’s not useful to compare the performance of Tesla’s two-chip Full Self Driving computer against NVIDIA’s single-chip driver assistance system. Tesla’s two-chip FSD computer at 144 TOPs would compare against the NVIDIA DRIVE AGX Pegasus computer which runs at 320 TOPS for AI perception, localization and path planning.” While pointing out the “inaccuracies”, Nvidia did miss out the key point here: the power consumption. “Having a system that can do 160 TOPS means little if it uses 500 watts while tesla's 144 TOPS system uses 72 watts,” a Redditor said. Robotaxis will hit the roads in 2020 Musk shared that within the next year or so we will see Tesla’s robotaxis coming into the ride-hailing market giving competition to Uber and Lyft. Musk made a bold claim saying that though, similar to other ride-hailing services, the robotaxis will allow users to hail a Tesla for a ride, they will not have drivers. Musk announced, “I feel very confident predicting that there will be autonomous robotaxis from Tesla next year — not in all jurisdictions because we won’t have regulatory approval everywhere.” He did not share many details on what regulations he was talking about. The service will allow Tesla-owners to add their properly equipped vehicles to Tesla’s own ride-sharing app, following a similar business model as Uber or Airbnb. The company will provide a dedicated number of robotaxis in areas where there are not enough loanable cars. Musk predicted that the average robotaxi will be able to yield $30,000 in gross profit per car, annually. Of this profit, about 25% to 30% will go to Tesla, therefore, an owner will be able to make $21,000 a year. Musk's plans for launching robotaxis next year looks ambitious. Experts and the media are quite skeptical about his plan. The Partners for Automated Vehicle Education (PAVE) industry group tweeted: https://twitter.com/PAVECampaign/status/1120436981220237312 https://twitter.com/RobMcCargow/status/1120961462678245376 Musk says “Anyone relying on lidar is doomed” Musk has been pretty vocal about his dislike towards LIDAR. He calls this technology “a crutch for self-driving cars”. When this topic came up at the event, Musk said: “Lidar is a fool’s errand. Anyone relying on lidar is doomed. Doomed! [They are] expensive sensors that are unnecessary. It’s like having a whole bunch of expensive appendices. Like, one appendix is bad, well now you have a whole bunch of them, it’s ridiculous, you’ll see.” LIDAR, which stands for Light Direction and Ranging, is used by Uber, Waymo, Cruise, and many other self-driving vehicles manufacturing companies. LIDAR projects low-intensity, harmless, and invisible laser beams at a target, or in the case of self-driving cars, all around. The reflected pulses are then measured for return time and wavelength to calculate the distance of an object from the sender. Lidar is capable of producing pretty detailed visualizations of the environment around a self-driving car. However, Tesla believes that this same functionality can be facilitated by cameras. According to Musk, cameras can provide much better resolutions and when combined with the neural net can predict depth very well. Andrej Karpathy, Tesla’s Senior Director of AI, took to the stage to explain the limitations of Lidar. He said, “In that sense, lidar is really a shortcut. It sidesteps the fundamental problems, the important problem of visual recognition, that is necessary for autonomy. It gives a false sense of progress and is ultimately a crutch. It does give, like, really fast demos!”. Karpathy further added, “You were not shooting lasers out of your eyes to get here.” While true, many felt that the reasoning is completely flawed. A Redditor in a discussion thread, said, “Musk's argument that "you drove here using your own two eyes with no lasers coming out of them" is reductive and flawed. It should be obvious to anyone that our eyes are more complex than simple stereo cameras. If the Tesla FSD system can reliably perceive depth at or above the level of the human eye in all conditions, then they have done something truly remarkable. Judging by how Andrej Karpathy deflected the question about how well the system works in snowy conditions, I would assume they have not reached that level.” Check out the live stream of the autonomy day on Tesla’s official website. Tesla v9 to incorporate neural networks for autopilot Tesla is building its own AI hardware for self-driving cars Nvidia Tesla V100 GPUs publicly available in beta on Google Compute Engine and Kubernetes Engine  
Read more
  • 0
  • 0
  • 13954
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at €18.99/month. Cancel anytime
Packt
07 Aug 2013
13 min read
Save for later

.NET 4.5 Parallel Extensions – Async

Packt
07 Aug 2013
13 min read
(For more resources related to this topic, see here.) Creating an async method The TAP is a new pattern for asynchronous programming in .NET Framework 4.5. It is based on a task, but in this case a task doesn't represent work which will be performed on another thread. In this case, a task is used to represent arbitrary asynchronous operations. Let's start learning how async and await work by creating a Windows Presentation Foundation (WPF ) application that accesses the web using HttpClient. This kind of network access is ideal for seeing TAP in action. The application will get the contents of a classic book from the web, and will provide a count of the number of words in the book. How to do it… Let's go to Visual Studio 2012 and see how to use the async and await keywords to maintain a responsive UI by doing the web communications asynchronously. Start a new project using the WPF Application project template and assign WordCountAsync as Solution name . Begin by opening MainWindow.xaml and adding the following XAML to create a simple user interface containing Button and TextBlock: <Window x_Class="WordCountAsync.MainWindow" Title="WordCountAsync" Height="350" Width="525"> <Grid> <Button x_Name="StartButton" Content="Start" HorizontalAlignment="Left" Margin="219,195,0,0" VerticalAlignment="Top" Width="75" RenderTransformOrigin="-0.2,0.45" Click="StartButton_Click"/> <TextBlock x_Name="TextResults" HorizontalAlignment="Left" Margin="60,28,0,0" TextWrapping="Wrap" VerticalAlignment="Top" Height="139" Width="411"/> </Grid> </Window> Next, open up MainWindow.xaml.cs. Go to the Project and add a reference to System.Net.Http. Add the following using directives to the top of your MainWindow class: using System; using System.Linq; using System.Net.Http; using System.Threading.Tasks; using System.Windows; At the top of the MainWindow class, add a character array constant that will be used to split the contents of the book into a word array. char[] delimiters = { ' ', ',', '.', ';', ':', '-', '_', '/', '\u000A' }; Add a button click event for the StartButton and add the async modifier to the method signature to indicate that this will be a async method. Please note that async methods that return void are normally only used for event handlers, and should be avoided. private async void StartButton_Click(object sender, RoutedEventArgs e) { } Next, let's create a async method called GetWordCountAsync that returns Task<int>. This method will create HttpClient and call its GetStringAsync method to download the book contents as a string. It will then use the Split method to split the string into a wordArray. We can return the count of the wordArray as our return value. public async Task<int> GetWordCountAsync() { TextResults.Text += "Getting the word count for Origin of Species...\n"; var client = new HttpClient(); var bookContents = await client.GetStringAsync(@"http://www.gutenberg.org/files/2009/2009.txt"); var wordArray = bookContents.Split(delimiters, StringSplitOptions.RemoveEmptyEntries); return wordArray.Count(); } Finally, let's complete the implementation of our button click event. The Click event handler will just call GetWordCountAsync with the await keyword and display the results to TextBlock. private async void StartButton_Click(object sender, RoutedEventArgs e) { var result = await GetWordCountAsync(); TextResults.Text += String.Format("Origin of Species word count: {0}",result); } In Visual Studio 2012, press F5 to run the project. Click on the Start button, and your application should appear as shown in the following screenshot: How it works… In the TAP, asynchronous methods are marked with an async modifier. The async modifier on a method does not mean that the method will be scheduled to run asynchronously on a worker thread. It means that the method contains control flow that involves waiting for the result of an asynchronous operation, and will be rewritten by the compiler to ensure that the asynchronous operation can resume this method at the right spot. Let me try to put this a little more simply. When you add the async modifier to a method, it indicates that the method will wait on an asynchronous code to complete. This is done with the await keyword. The compiler actually takes the code that follows the await keyword in an async method and turns it into a continuation that will run after the result of the async operation is available. In the meantime, the method is suspended, and control returns to the method's caller. If you add the async modifier to a method, and then don't await anything, it won't cause an error. The method will simply run synchronously. An async method can have one of the three return types: void, Task, or Task<TResult>. As mentioned before, a task in this context doesn't mean that this is something that will execute on a separate thread. In this case, task is just a container for the asynchronous work, and in the case of Task<TResult>, it is a promise that a result value of type TResult will show up after the asynchronous operation completes. In our application, we use the async keyword to mark the button click event handler as asynchronous, and then we wait for the GetWordCountAsync method to complete by using the wait keyword. private async void StartButton_Click(object sender, RoutedEventArgs e) { StartButton.Enabled = false; var result = await GetWordCountAsync(); TextResults.Text += String.Format("Origin of Species word count: {0}", .................. result); StartButton.Enabled = true; } The code that follows the await keyword, in this case, the same line of code that updates TextBlock, is turned by the compiler into a continuation that will run after the integer result is available. If the Click event is fired again while this asynchronous task is in progress, another asynchronous task is created and awaited. To prevent this, it is a common practice to disable the button that is clicked. It is a convention to name an asynchronous method with an Async postfix, as we have done with GetWordCountAsync. Handling Exceptions in asynchronous code So how would you add Exception handling to code that is executed asynchronously? In previous asynchronous patterns, this was very difficult to achieve. In C# 5.0 it is much more straightforward because you just have to wrap the asynchronous function call with a standard try/catch block. On the surface this sounds easy, and it is, but there is more going on behind the scene that will be explained right after we build our next example application. For this recipe, we will return to our classic books word count scenario, and we will be handling an Exception thrown by HttpClient when it tries to get the book contents using an incorrect URL. How to do it… Let's build another WPF application and take a look at how to handle Exceptions when something goes wrong in one of our asynchronous methods. Start a new project using the WPF Application project template and assign AsyncExceptions as Solution name . Begin by opening MainWindow.xaml and adding the following XAML to create a simple user interface containing Button and a TextBlock: <Window x_Class="WordCountAsync.MainWindow" Title="WordCountAsync" Height="350" Width="525"> <Grid> <Button x_Name="StartButton" Content="Start" HorizontalAlignment="Left" Margin="219,195,0,0" VerticalAlignment="Top" Width="75" RenderTransformOrigin="-0.2,0.45" Click="StartButton_Click"/> <TextBlock x_Name="TextResults" HorizontalAlignment="Left" Margin="60,28,0,0" TextWrapping="Wrap" VerticalAlignment="Top" Height="139" Width="411"/> </Grid> </Window> Next, open up MainWindow.xaml.cs. Go to the Project Explorer , right-click on References , click on Framework from the menu on the left side of the Reference Manager , and then add a reference to System.Net.Http. Add the following using directives to the top of your MainWindow class: using System; using System.Linq; using System.Net.Http; using System.Threading.Tasks; using System.Windows; At the top of the MainWindow class, add a character array constant that will be used to split the contents of the book into a word array. char[] delimiters = { ' ', ',', '.', ';', ':', '-', '_', '/', '\u000A' }; Now let's create our GetWordCountAsync method. This method will be very similar to the last recipe, but it will be trying to access the book on an incorrect URL. The asynchronous code will be wrapped in a try/catch block to handle Exception. We will also use a finally block to dispose of HttpClient. public async Task<int> GetWordCountAsync() { ResultsTextBlock.Text += "Getting the word count for Origin of Species...\n"; var client = new HttpClient(); try { var bookContents = await client.GetStringAsync(@"http://www.gutenberg.org/files/2009/No_Book_Here.txt"); var wordArray = bookContents.Split(delimiters, StringSplitOptions.RemoveEmptyEntries); return wordArray.Count(); } catch (Exception ex) { ResultsTextBlock.Text += String.Format("An error has occurred: {0} \n", ex.Message); return 0; } finally { client.Dispose(); } } Finally, let create the Click event handler for our StartButton. This is pretty much the same as the last recipe, just wrapped in a try/catch block. Don't forget to add the async modifier to the method signature. private async void StartButton_Click(object sender, RoutedEventArgs e) { try { var result = await GetWordCountAsync(); ResultsTextBlock.Text += String.Format("Origin of Species word count: {0}", result); } catch(Exception ex) { ResultsTextBlock.Text += String.Format("An error has occurred: {0} \n", ex.Message); } } Now, in Visual Studio 2012, press F5 to run the project. Click on the Start button. Your application should appear as shown in the following screenshot: How it works… Wrapping your asynchronous code in a try/catch block is pretty easy. In fact, it hides some of the complex work Visual Studio 2012 to doing for us. To understand this, you need to think about the context in which your code is running. When the TAP is used in Windows Forms or WPF applications, there's already a context that the code is running in, such as the message loop UI thread. When async calls are made in those applications, the awaited code goes off to do its work asynchronously and the async method exits back to its caller. In other words, the program execution returns to the message loop UI thread. The Console applications don't have the concept of a context. When the code hits an awaited call inside the try block, it will exit back to its caller, which in this case is Main. If there is no more code after the awaited call, the application ends without the async method ever finishing. To alleviate this issue, Microsoft included async compatible context with the TAP that is used for Console apps or unit test apps to prevent this inconsistent behavior. This new context is called GeneralThreadAffineContext. Do you really need to understand these context issues to handle async Exceptions? No, not really. That's part of the beauty of the Task-based Asynchronous Pattern. Cancelling an asynchronous operation In .NET 4.5, asynchronous operations can be cancelled in the same way that parallel tasks can be cancelled, by passing in CancellationToken and calling the Cancel method on CancellationTokenSource. In this recipe, we are going to create a WPF application that gets the contents of a classic book over the web and performs a word count. This time though we are going to set up a Cancel button that we can use to cancel the async operation if we don't want to wait for it to finish. How to do it… Let's create a WPF application to show how we can add cancellation to our asynchronous methods. Start a new project using the WPF Application project template and assign AsyncCancellation as Solution name . Begin by opening MainWindow.xaml and adding the following XAML to create our user interface. In this case, the UI contains TextBlock, StartButton, and CancelButton. <Window x_Class="AsyncCancellation.MainWindow" Title="AsyncCancellation" Height="400" Width="599"> <Grid Width="600" Height="400"> <Button x_Name="StartButton" Content="Start" HorizontalAlignment="Left" Margin="142,183,0,0" VerticalAlignment="Top" Width="75" RenderTransformOrigin="-0.2,0.45" Click="StartButton_Click"/> <Button x_Name="CancelButton" Content="Cancel" HorizontalAlignment="Left" Margin="379,185,0,0" VerticalAlignment="Top" Width="75" Click="CancelButton_Click"/> <TextBlock x_Name="TextResult" HorizontalAlignment="Left" Margin="27,24,0,0" TextWrapping="Wrap" VerticalAlignment="Top" Height="135" Width="540"/> </Grid> </Window> Next, open up MainWindow.xaml.cs, click on the Project Explorer , and add a reference to System.Net.Http. Add the following using directives to the top of your MainWindow class: using System; using System.Linq; using System.Net.Http; using System.Threading.Tasks; using System.Windows; At the top of the MainWindow class, add a character array constant that will be used to split the contents of the book into a word array. char[] delimiters = { ' ', ',', '.', ';', ':', '-', '_', '/', '\u000A' }; Next, let's create the GetWordCountAsync method. This method is very similar to the method explained before. It needs to be marked as asynchronous with the async modifier and it returns Task<int>. This time however, the method takes a CancellationToken parameter. We also need to use the GetAsync method of HttpClient instead of the GetStringAsync method, because the former supports cancellation, whereas the latter does not. We will add a small delay in the method so we have time to cancel the operation before the download completes. public async Task<int> GetWordCountAsync(CancellationToken ct) { TextResult.Text += "Getting the word count for Origin of Species...\n"; var client = new HttpClient(); await Task.Delay(500); try { HttpResponseMessage response = await client.GetAsync(@"http://www.gutenberg.org/files/2009/2009.txt", ct); var words = await response.Content.ReadAsStringAsync(); var wordArray = words.Split(delimiters, StringSplitOptions.RemoveEmptyEntries); return wordArray.Count(); } finally { client.Dispose(); } } Now, let's create the Click event handler for our CancelButton. This method just needs to check if CancellationTokenSource is null, and if not, it calls the Cancel method. private void CancelButton_Click(object sender, RoutedEventArgs e) { if (cts != null) { cts.Cancel(); } } Ok, let's finish up by adding a Click event handler for StartButton. This method is the same as explained before, except we also have a catch block that specifically handles OperationCancelledException. Don't forget to mark the method with the async modifier. public async Task<int> GetWordCountAsync(CancellationToken ct) { TextResult.Text += "Getting the word count for Origin of Species...\n"; var client = new HttpClient(); await Task.Delay(500); try { HttpResponseMessage response = await client.GetAsync(@"http://www.gutenberg.org/files/2009/2009.txt", ct); var words = await response.Content.ReadAsStringAsync(); var wordArray = words.Split(delimiters, StringSplitOptions.RemoveEmptyEntries); return wordArray.Count(); } finally { client.Dispose(); } } In Visual Studio 2012, press F5 to run the project Click on the Start button, then the Cancel button. Your application should appear as shown in the following screenshot: How it works… Cancellation is an aspect of user interaction that you need to consider to build a professional async application. In this example, we implemented cancellation by using a Cancel button, which is one of the most common ways to surface cancellation functionality in a GUI application. In this recipe, cancellation follows a very common flow. The caller (start button click event handler) creates a CancellationTokenSource object. private async void StartButton_Click(object sender, RoutedEventArgs e) { cts = new CancellationTokenSource(); ... } The caller calls a cancelable method, and passes CancellationToken from CancellationTokenSource (CancellationTokenSource.Token). public async Task<int> GetWordCountAsync(CancellationToken ct ) { ... HttpResponseMessage response = await client.GetAsync(@"http://www.gutenberg.org/files/2009/2009.txt", ct ); ... } The cancel button click event handler requests cancellation using the CancellationTokenSource object (CancellationTokenSource.Cancel()). private void CancelButton_Click(object sender, RoutedEventArgs e) { if (cts != null) { cts.Cancel(); } } The task acknowledges the cancellation by throwing OperationCancelledException, which we handle in a catch block in the start button click event handler.
Read more
  • 0
  • 0
  • 13941

article-image-scipy-signal-processing
Packt
03 Mar 2015
14 min read
Save for later

SciPy for Signal Processing

Packt
03 Mar 2015
14 min read
In this article by Sergio J. Rojas G. and Erik A Christensen, authors of the book Learning SciPy for Numerical and Scientific Computing - Second Edition, we will focus on the usage of some most commonly used routines that are included in SciPy modules—scipy.signal, scipy.ndimage, and scipy.fftpack, which are used for signal processing, multidimensional image processing, and computing Fourier transforms, respectively. We define a signal as data that measures either a time-varying or spatially varying phenomena. Sound or electrocardiograms are excellent examples of time-varying quantities, while images embody the quintessential spatially varying cases. Moving images are treated with the techniques of both types of signals, obviously. The field of signal processing treats four aspects of this kind of data: its acquisition, quality improvement, compression, and feature extraction. SciPy has many routines to treat effectively tasks in any of the four fields. All these are included in two low-level modules (scipy.signal being the main module, with an emphasis on time-varying data, and scipy.ndimage, for images). Many of the routines in these two modules are based on Discrete Fourier Transform of the data. SciPy has an extensive package of applications and definitions of these background algorithms, scipy.fftpack, which we will start covering first. (For more resources related to this topic, see here.) Discrete Fourier Transforms The Discrete Fourier Transform (DFT from now on) transforms any signal from its time/space domain into a related signal in the frequency domain. This allows us not only to be able to analyze the different frequencies of the data, but also for faster filtering operations, when used properly. It is possible to turn a signal in the frequency domain back to its time/spatial domain; thanks to the Inverse Fourier Transform. We will not go into detail of the mathematics behind these operators, since we assume familiarity at some level with this theory. We will focus on syntax and applications instead. The basic routines in the scipy.fftpack module compute the DFT and its inverse, for discrete signals in any dimension, which are fft and ifft (one dimension), fft2 and ifft2 (two dimensions), and fftn and ifftn (any number of dimensions). All of these routines assume that the data is complex valued. If we know beforehand that a particular dataset is actually real valued, and should offer real-valued frequencies, we use rfft and irfft instead, for a faster algorithm. All these routines are designed so that composition with their inverses always yields the identity. The syntax is the same in all cases, as follows: fft(x[, n, axis, overwrite_x]) The first parameter, x, is always the signal in any array-like form. Note that fft performs one-dimensional transforms. This means in particular, that if x happens to be two-dimensional, for example, fft will output another two-dimensional array where each row is the transform of each row of the original. We can change it to columns instead, with the optional parameter, axis. The rest of parameters are also optional; n indicates the length of the transform, and overwrite_x gets rid of the original data to save memory and resources. We usually play with the integer n when we need to pad the signal with zeros, or truncate it. For higher dimension, n is substituted by shape (a tuple), and axis by axes (another tuple). To better understand the output, it is often useful to shift the zero frequencies to the center of the output arrays with fftshift. The inverse of this operation, ifftshift, is also included in the module. The following code shows some of these routines in action, when applied to a checkerboard image: >>> import numpy >>> from scipy.fftpack import fft,fft2, fftshift >>> import matplotlib.pyplot as plt >>> B=numpy.ones((4,4)); W=numpy.zeros((4,4)) >>> signal = numpy.bmat("B,W;W,B") >>> onedimfft = fft(signal,n=16) >>> twodimfft = fft2(signal,shape=(16,16)) >>> plt.figure() >>> plt.gray() >>> plt.subplot(121,aspect='equal') >>> plt.pcolormesh(onedimfft.real) >>> plt.colorbar(orientation='horizontal') >>> plt.subplot(122,aspect='equal') >>> plt.pcolormesh(fftshift(twodimfft.real)) >>> plt.colorbar(orientation='horizontal') >>> plt.show() Note how the first four rows of the one-dimensional transform are equal (and so are the last four), while the two-dimensional transform (once shifted) presents a peak at the origin, and nice symmetries in the frequency domain. In the following screenshot (obtained from the preceding code), the left-hand side image is fft and the right-hand side image is fft2 of a 2 x 2 checkerboard signal: The scipy.fftpack module also offers the Discrete Cosine Transform with its inverse (dct, idct) as well as many differential and pseudo-differential operators defined in terms of all these transforms: diff (for derivative/integral), hilbert and ihilbert (for the Hilbert transform), tilbert and itilbert (for the h-Tilbert transform of periodic sequences), and so on. Signal construction To aid in the construction of signals with predetermined properties, the scipy.signal module has a nice collection of the most frequent one-dimensional waveforms in the literature: chirp and sweep_poly (for the frequency-swept cosine generator), gausspulse (a Gaussian modulated sinusoid) and sawtooth and square (for the waveforms with those names). They all take as their main parameter a one-dimensional ndarray representing the times at which the signal is to be evaluated. Other parameters control the design of the signal, according to frequency or time constraints. Let's take a look into the following code snippet, which illustrates the use of these one dimensional waveforms that we just discussed: >>> import numpy >>> from scipy.signal import chirp, sawtooth, square, gausspulse >>> import matplotlib.pyplot as plt >>> t=numpy.linspace(-1,1,1000) >>> plt.subplot(221); plt.ylim([-2,2]) >>> plt.plot(t,chirp(t,f0=100,t1=0.5,f1=200))   # plot a chirp >>> plt.subplot(222); plt.ylim([-2,2]) >>> plt.plot(t,gausspulse(t,fc=10,bw=0.5))     # Gauss pulse >>> plt.subplot(223); plt.ylim([-2,2]) >>> t*=3*numpy.pi >>> plt.plot(t,sawtooth(t))                     # sawtooth >>> plt.subplot(224); plt.ylim([-2,2]) >>> plt.plot(t,square(t))                       # Square wave >>> plt.show() Generated by this code, the following diagram shows waveforms for chirp (upper-left), gausspulse (upper-right), sawtooth (lower-left), and square (lower-right): The usual method of creating signals is to import them from the file. This is possible by using purely NumPy routines, for example fromfile: fromfile(file, dtype=float, count=-1, sep='') The file argument may point to either a file or a string, the count argument is used to determine the number of items to read, and sep indicates what constitutes a separator in the original file/string. For images, we have the versatile routine, imread in either the scipy.ndimage or scipy.misc module: imread(fname, flatten=False) The fname argument is a string containing the location of an image. The routine infers the type of file, and reads the data into an array, accordingly. In case the flatten argument is turned to True, the image is converted to gray scale. Note that, in order to work, the Python Imaging Library (PIL) needs to be installed. It is also possible to load .wav files for analysis, with the read and write routines from the wavfile submodule in the scipy.io module. For instance, given any audio file with this format, say audio.wav, the command, rate,data = scipy.io.wavfile.read("audio.wav"), assigns an integer value to the rate variable, indicating the sample rate of the file (in samples per second), and a NumPy ndarray to the data variable, containing the numerical values assigned to the different notes. If we wish to write some one-dimensional ndarray data into an audio file of this kind, with the sample rate given by the rate variable, we may do so by issuing the following command: >>> scipy.io.wavfile.write("filename.wav",rate,data) Filters A filter is an operation on signals that either removes features or extracts some component. SciPy has a very complete set of known filters, as well as the tools to allow construction of new ones. The complete list of filters in SciPy is long, and we encourage the reader to explore the help documents of the scipy.signal and scipy.ndimage modules for the complete picture. We will introduce in these pages, as an exposition, some of the most used filters in the treatment of audio or image processing. We start by creating a signal worth filtering: >>> from numpy import sin, cos, pi, linspace >>> f=lambda t: cos(pi*t) + 0.2*sin(5*pi*t+0.1) + 0.2*sin(30*pi*t)    + 0.1*sin(32*pi*t+0.1) + 0.1*sin(47* pi*t+0.8) >>> t=linspace(0,4,400); signal=f(t) We first test the classical smoothing filter of Wiener and Kolmogorov, wiener. We present in a plot, the original signal (in black) and the corresponding filtered data, with a choice of a Wiener window of the size 55 samples (in blue). Next, we compare the result of applying the median filter, medfilt, with a kernel of the same size as before (in red): >>> from scipy.signal import wiener, medfilt >>> import matplotlib.pylab as plt >>> plt.plot(t,signal,'k') >>> plt.plot(t,wiener(signal,mysize=55),'r',linewidth=3) >>> plt.plot(t,medfilt(signal,kernel_size=55),'b',linewidth=3) >>> plt.show() This gives us the following graph showing the comparison of smoothing filters (wiener is the one that has its starting point just below 0.5 and medfilt has its starting point just above 0.5): Most of the filters in the scipy.signal module can be adapted to work in arrays of any dimension. But in the particular case of images, we prefer to use the implementations in the scipy.ndimage module, since they are coded with these objects in mind. For instance, to perform a median filter on an image for smoothing, we use scipy.ndimage.median_filter. Let's see an example. We will start by loading Lena to the array and corrupting the image with Gaussian noise (zero mean and standard deviation of 16): >>> from scipy.stats import norm     # Gaussian distribution >>> import matplotlib.pyplot as plt >>> import scipy.misc >>> import scipy.ndimage >>> plt.gray() >>> lena=scipy.misc.lena().astype(float) >>> plt.subplot(221); >>> plt.imshow(lena) >>> lena+=norm(loc=0,scale=16).rvs(lena.shape) >>> plt.subplot(222); >>> plt.imshow(lena) >>> denoised_lena = scipy.ndimage.median_filter(lena,3) >>> plt.subplot(224); >>> plt.imshow(denoised_lena) The set of filters for images come in two flavors—statistical and morphological. For example, among the filters of statistical nature, we have the Sobel algorithm oriented to detection of edges (singularities along curves). Its syntax is as follows: sobel(image, axis=-1, output=None, mode='reflect', cval=0.0) The optional parameter, axis, indicates the dimension in which the computations are performed. By default, this is always the last axis (-1). The mode parameter, which is one of the strings 'reflect', 'constant', 'nearest', 'mirror', or 'wrap', indicates how to handle the border of the image, in case there is insufficient data to perform the computations there. In case the mode is 'constant', we may indicate the value to use in the border, with the cval parameter. Let's look into the following code snippet, which illustrates the use of the sobel filter: >>> from scipy.ndimage.filters import sobel >>> import numpy >>> lena=scipy.misc.lena() >>> sblX=sobel(lena,axis=0); sblY=sobel(lena,axis=1) >>> sbl=numpy.hypot(sblX,sblY) >>> plt.subplot(223); >>> plt.imshow(sbl) >>> plt.show() The following screenshot illustrates Lena (upper-left) and noisy Lena (upper-right) with the preceding two filters in action—edge map with sobel (lower-left) and median filter (lower-right): Morphology We also have the possibility of creating and applying filters to images based on mathematical morphology, both to binary and gray-scale images. The four basic morphological operations are opening (binary_opening), closing (binary_closing), dilation (binary_dilation), and erosion (binary_erosion). Note that the syntax for each of these filters is very simple, since we only need two ingredients—the signal to filter and the structuring element to perform the morphological operation. Let's take a look into the general syntax for these morphological operations: binary_operation(signal, structuring_element) We may use combinations of these four basic morphological operations to create more complex filters for removal of holes, hit-or-miss transforms (to find the location of specific patterns in binary images), denoising, edge detection, and many more. The SciPy module also allows for creating some common filters using the preceding syntax. For instance, for the location of the letter e in a text, we could use the following command instead: >>> binary_hit_or_miss(text, letterE) For comparative purposes, let's use this command in the following code snippet: >>> import numpy >>> import scipy.ndimage >>> import matplotlib.pylab as plt >>> from scipy.ndimage.morphology import binary_hit_or_miss >>> text = scipy.ndimage.imread('CHAP_05_input_textImage.png') >>> letterE = text[37:53,275:291] >>> HitorMiss = binary_hit_or_miss(text, structure1=letterE,    origin1=1) >>> eLocation = numpy.where(HitorMiss==True) >>> x=eLocation[1]; y=eLocation[0] >>> plt.imshow(text, cmap=plt.cm.gray, interpolation='nearest') >>> plt.autoscale(False) >>> plt.plot(x,y,'wo',markersize=10) >>> plt.axis('off') >>> plt.show() The output for the preceding lines of code is generated as follows: For gray-scale images, we may use a structuring element (structuring_element) or a footprint. The syntax is, therefore, a little different: grey_operation(signal, [structuring_element, footprint, size, ...]) If we desire to use a completely flat and rectangular structuring element (all ones), then it is enough to indicate the size as a tuple. For instance, to perform gray-scale dilation of a flat element of size (15,15) on our classical image of Lena, we issue the following command: >>> grey_dilation(lena, size=(15,15)) The last kind of morphological operations coded in the scipy.ndimage module perform distance and feature transforms. Distance transforms create a map that assigns to each pixel, the distance to the nearest object. Feature transforms provide with the index of the closest background element instead. These operations are used to decompose images into different labels. We may even choose different metrics such as Euclidean distance, chessboard distance, and taxicab distance. The syntax for the distance transform (distance_transform) using a brute force algorithm is as follows: distance_transform_bf(signal, metric='euclidean', sampling=None, return_distances=True, return_indices=False,                      distances=None, indices=None) We indicate the metric with the strings such as 'euclidean', 'taxicab', or 'chessboard'. If we desire to provide the feature transform instead, we switch return_distances to False and return_indices to True. Similar routines are available with more sophisticated algorithms—distance_transform_cdt (using chamfering for taxicab and chessboard distances). For Euclidean distance, we also have distance_transform_edt. All these use the same syntax. Summary In this article, we explored signal processing (any dimensional) including the treatment of signals in frequency space, by means of their Discrete Fourier Transforms. These correspond to the fftpack, signal, and ndimage modules. Resources for Article: Further resources on this subject: Signal Processing Techniques [article] SciPy for Computational Geometry [article] Move Further with NumPy Modules [article]
Read more
  • 0
  • 0
  • 13934

article-image-setting-environment
Packt
04 Jan 2017
14 min read
Save for later

Setting Up the Environment

Packt
04 Jan 2017
14 min read
In this article by Sohail Salehi, the author of the book Angular 2 Services, you can ask the two fundamental questions, when a new development tool is announced or launched are: how different is the new tool from other competitor tools and how enhanced it is when compared to its own the previous versions? If we are going to invest our time in learning a new framework, common sense says we need to make sure we get a good return on our investment. There are so many good articles out there about cons and pros of each framework. To me, choosing Angular 2 boils down to three aspects: The foundation: Angular is introduced and supported by Google and targeted at “ever green” modern browsers. This means we, as developers, don't need to lookout for hacky solutions in each browser upgrade, anymore. The browser will always be updated to the latest version available, letting Angular worry about the new changes and leaving us out of it. This way we can focus more on our development tasks. The community: Think about the community as an asset. The bigger the community, the wider range of solutions to a particular problem. Looking at the statistics, Angular community still way ahead of others and the good news is this community is leaning towards being more involved and more contributing on all levels. The solution: If you look at the previous JS frameworks, you will see most of them focusing on solving a problem for a browser first, and then for mobile devices. The argument for that could be simple: JS wasn't meant to be a language for mobile development. But things have changed to a great extent over the recent years and people now use mobile devices more than before. I personally believe a complex native mobile application – which is implemented in Java or C – is more performant, as compared to its equivalent implemented in JS. But the thing here is that not every mobile application needs to be complex. So business owners have started asking questions like: Why do I need a machine-gun to kill a fly? (For more resources related to this topic, see here.) With that question in mind, Angular 2 chose a different approach. It solves the performance challenges faced by mobile devices first. In other words, if your Angular 2 application is fast enough on mobile environments, then it is lightning fast inside the “ever green” browsers. So that is what we are going to do in this article. First we are going to learn about Angular 2 and the main problem it is going to solve. Then we talk a little bit about the JavaScript history and the differences between Angular 2 and AngularJS 1. Introducing “The Sherlock Project” is next and finally we install the tools and libraries we need to implement this project. Introducing Angular 2 The previous JS frameworks we've used already have a fluid and easy workflow towards building web applications rapidly. But as developers what we are struggling with is the technical debt. In simple words, we could quickly build a web application with an impressive UI. But as the product kept growing and the change requests started kicking in, we had to deal with all maintenance nightmares which forces a long list of maintenance tasks and heavy costs to the businesses. Basically the framework that used to be an amazing asset, turned into a hairy liability (or technical debt if you like). One of the major "revamps" in Angular 2 is the removal of a lot of modules resulting in a lighter and faster core. For example, if you are coming from an Angular 1.x background and don't see $scope or $log in the new version, don't panic, they are still available to you via other means, But there is no need to add overhead to the loading time if we are not going to use all modules. So taking the modules out of the core results in a better performance. So to answer the question, one of the main issues Angular 2 addresses is the performance issues. This is done through a lot of structural changes. There is no backward compatibility We don't have backward compatibility. If you have some Angular projects implemented with the previous version (v1.x), depending on the complexity of the project, I wouldn't recommend migrating to the new version. Most likely, you will end up hammering a lot of changes into your migrated Angular 2 project and at the end you will realize it was more cost effective if you would just create a new project based on Angular 2 from scratch. Please keep in mind, the previous versions of AngularJS and Angular 2 share just a name, but they have huge differences in their nature and that is the price we pay for a better performance. Previous knowledge of AngularJS 1.x is not necessary You might wondering if you need to know AngularJS 1.x before diving into Angular 2. Absolutely not. To be more specific, it might be even better if you didn't have any experience using Angular at all. This is because your mind wouldn't be preoccupied with obsolete rules and syntaxes. For example, we will see a lot of annotations in Angular 2 which might make you feel uncomfortable if you come from a Angular 1 background. Also, there are different types of dependency injections and child injectors which are totally new concepts that have been introduced in Angular 2. Moreover there are new features for templating and data-binding which help to improve loading time by asynchronous processing. The relationship between ECMAScript, AtScript and TypeScript The current edition of ECMAScript (ES5) is the one which is widely accepted among all well known browsers. You can think of it as the traditional JavaScript. Whatever code is written in ES5 can be executed directly in the browsers. The problem is most of modern JavaScript frameworks contain features which require more than the traditional JavaScript capabilities. That is why ES6 was introduced. With this edition – and any future ECMAScript editions – we will be able to empower JavaScript with the features we need. Now, the challenge is running the new code in the current browsers. Most browsers, nowadays recognize standard JavaScript codes only. So we need a mediator to transform ES6 to ES5. That mediator is called a transpiler and the technical term for transformations is transpiling. There are many good transpilers out there and you are free to choose whatever you feel comfortable with. Apart from TypeScript, you might want to consider Babel (babeljs.io) as your main transpiler. Google originally planned to use AtScript to implement Angular 2, but later they joined forces with Microsoft and introduced TypeScript as the official transpiler for Angular 2. The following figure summarizes the relationship between various editions of ECMAScript, AtScript and TypeScript. For more details about JavaScript, ECMAScript and how they evolved during the past decade visit the following link: https://en.wikipedia.org/wiki/ECMAScript Setting up tools and getting started! It is important to get the foundation right before installing anything else. Depending on your operating system, install Node.js and its package manager- npm. . You can find a detailed installation manual on Node.js official website. https://nodejs.org/en/ Make sure both Node.js and npm are installed globally (they are accessible system wide) and have the right permissions. At the time of writing npm comes with Node.js out of the box. But in case their policy changes in the future, you can always download the npm and follow the installation process from the following link. https://npmjs.com The next stop would be the IDE. Feel free to choose anything that you are comfortable with. Even a simple text editor will do. I am going to use WebStorm because of its embedded TypeScript syntax support and Angular 2 features which speeds up development process. Moreover it is light weight enough to handle the project we are about to develop. You can download it from here: https://jetbrains.com/webstorm/download We are going to use simple objects and arrays as a place holder for the data. But at some stage we will need to persist the data in our application. That means we need a database. We will use the Google's Firebase Realtime database for our needs. It is fast, it doesn't need to download or setup anything locally and more over it responds instantly to your requests and aligns perfectly with Angular's two-way data-binding. For now just leave the database as it is. You don't need to create any connections or database objects. Setting up the seed project The final requirement to get started would be an Angular 2 seed project to shape the initial structure of our project. If you look at the public source code repositories you can find several versions of these seeds. But I prefer the official one for two reasons: Custom made seeds usually come with a personal twist depending on the developers taste. Although sometimes it might be a great help, but since we are going to build everything from scratch and learn the fundamental concepts, they are not favorable to our project. The official seeds are usually minimal. They are very slim and don't contain overwhelming amount of 3rd party packages and environmental configurations. Speaking about packages, you might be wondering what happened to the other JavaScript packages we needed for this application. We didn't install anything else other than Node and NPM. The next section will answer this question. Setting up an Angular 2 project in WebStorm Assuming you have installed WebStorm, fire the IDE and and checkout a new project from a git repository. Now set the repository URL to: https://github.com/angular/angular2-seed.git and save the project in a folder called “the sherlock project” as shown in the figure below: Hit the Clone button and open the project in WebStorm. Next, click on the package.json file and observe the dependencies. As you see, this is a very lean seed with minimal configurations. It contains the Angular 2 plus required modules to get the project up and running. The first thing we need to do is install all required dependencies defined inside the package.json file. Right click on the file and select the “run npm install” option. Installing the packages for the first time will take a while. In the mean time, explore the devDependencies section of package.json in the editor. As you see, we have all the required bells and whistles to run the project including TypeScript, web server and so on to start the development: "devDependencies": { "@types/core-js": "^0.9.32", "@types/node": "^6.0.38", "angular2-template-loader": "^0.4.0", "awesome-typescript-loader": "^1.1.1", "css-loader": "^0.23.1", "raw-loader": "^0.5.1", "to-string-loader": "^1.1.4", "typescript": "^2.0.2", "webpack": "^1.12.9", "webpack-dev-server": "^1.14.0", "webpack-merge": "^0.8.4" }, We also have some nifty scripts defined inside package.json that automate useful processes. For example, to start a webserver and see the seed in action we can simply execute following command in a terminal: $ npm run server or we can right click on package.json file and select “Show npm Scripts”. This will open another side tab in WebStorm and shows all available scripts inside the current file. Basically all npm related commands (which you can run from command-line) are located inside the package.json file under the scripts key. That means if you have a special need, you can create your own script and add it there. You can also modify the current ones. Double click on start script and it will run the web server and loads the seed application on port 3000. That means if you visit http://localhost:3000 you will see the seed application in your browser: If you are wondering where the port number comes from, look into package.json file and examine the server key under the scripts section:    "server": "webpack-dev-server --inline --colors --progress --display-error-details --display-cached --port 3000 --content-base src", There is one more thing before we move on to the next topic. If you open any .ts file, WebStorm will ask you if you want it to transpile the code to JavaScript. If you say No once and it will never show up again. We don't need WebStorm to transpile for us because the start script is already contains a transpiler which takes care of all transformations for us. Front-end developers versus back-end developers Recently, I had an interesting conversation with a couple of colleagues of mine which worth sharing here in this article. One of them is an avid front-end developer and the other is a seasoned back-end developer. You guessed what I'm going to talk about: The debate between back-end/front-end developers and who is the better half. We have seen these kind of debates between back-end and front-end people in development communities long enough. But the interesting thing which – in my opinion – will show up more often in the next few months (years) is a fading border between the ends (front-end/back-end). It feels like the reason that some traditional front-end developers are holding up their guard against new changes in Angular 2, is not just because the syntax has changed thus causing a steep learning curve, but mostly because they now have to deal with concepts which have existed natively in back-end development for many years. Hence, the reason that back-end developers are becoming more open to the changes introduced in Angular 2 is mostly because these changes seem natural to them. Annotations or child dependency injections for example is not a big deal to back-enders, as much as it bothers the front-enders. I won't be surprised to see a noticeable shift in both camps in the years to come. Probably we will see more back-enders who are willing to use Angular as a good candidate for some – if not all – of their back-end projects and probably we will see more front-enders taking Object-Oriented concepts and best practices more seriously. Given that JavaScript was originally a functional scripting language they probably will try to catch-up with the other camp as fast as they can. There is no comparison here and I am not saying which camp has advantage over the other one. My point is, before modern front-end frameworks, JavaScript was open to be used in a quick and dirty inline scripts to solve problems quickly. While this is a very convenient approach it causes serious problems when you want to scale a web application. Imagine the time and effort you might have to make finding all of those dependent codes and re-factor them to reflect the required changes. When it comes to scalability, we need a full separation between layers and that requires developers to move away from traditional JavaScript and embrace more OOP best practices in their day to day development tasks. That what has been practiced in all modern front-end frameworks and Angular 2 takes it to the next level by completely restructuring the model-view-* concept and opening doors to the future features which eventually will be native part of any web browser. Introducing “The Sherlock Project” During the course of this journey we are going to dive into all new Angular 2 concepts by implementing a semi-AI project called: “The Sherlock Project”. This project will basically be about collecting facts, evaluating and ranking them, and making a decision about how truthful they are. To achieve this goal, we will implement a couple of services and inject them to the project, wherever they are needed. We will discuss one aspect of the project and will focus on one related service. At the end, all services will come together to act as one big project. Summary This article covered a brief introduction to Angular 2. We saw where Angular2 comes from, what problems is it going to solve and how we can benefit from it. We talked about the project that we will be creating with Angular 2. We also saw whic other tools are needed for our project and how to install and configure them. Finally, we cloned an Angular 2 seed project, installed all its dependencies and started a web server to examine the application output in a browser Resources for Article: Further resources on this subject: Introduction to JavaScript [article] Third Party Libraries [article] API with MongoDB and Node.js [article]
Read more
  • 0
  • 0
  • 13931

article-image-introduction-device-management
Packt
20 Feb 2018
10 min read
Save for later

Introduction with Device Management

Packt
20 Feb 2018
10 min read
In this article by Yatish Patil, the author of the book Microsoft Azure IOT Development Cookbook, we will look at device management using different techniques with Azure IoT Hub. We will see the following recipes: Device registry operations Device twins Device direct methods Device jobs (For more resources related to this topic, see here.) Azure IoT Hub has the capabilities that can be used by a developer to build a robust device management. There could be different use cases or scenarios across multiple industries but these device management capabilities, their patterns and the SDK code remains same, saving the significant time in developing and managing as well as maintaining the millions of devices. Device management will be the central part of any IoT solution. The IoT solution is going to help the users to manage the devices remotely, take actions from the cloud based application like disable, update data, run any command, and firmware update. In this article, we are going to perform all these tasks for device management and will start with creating the device. Device registry operations This sample application is focused on device registry operations and how it works, we will create a console application as our first IoT solution and look at the various device management techniques. Getting ready Let’s create a console application to start with IoT: Create a new project in Visual Studio: Create a Console Application Add IoT Hub connectivity extension in Visual Studio: Add the extension for IoT Hub connectivity Now right click on the Solution and go to Add a Connected Services. Select Azure IoT Hub and click Add. Now select Azure subscription and the IoT Hub created: Select IoT Hub for our application Next it will ask you to add device or you can skip this step and click Complete the configuration. How to do it... Create device identity: initialize the Azure IoT Hub registry connection: registryManager = RegistryManager.CreateFromConnectionString(connectionString); Device device = new Device(); try { device = await registryManager.AddDeviceAsync(new Device(deviceId)); success = true; } catch (DeviceAlreadyExistsException) { success = false; } Retrieve device identity by ID: Device device = new Device(); try { device = await registryManager.GetDeviceAsync(deviceId); } catch (DeviceAlreadyExistsException) { return device; } Delete device identity: Device device = new Device(); try { device = GetDevice(deviceId); await registryManager.RemoveDeviceAsync(device); success = true; } catch (Exception ex) { success = false; } List up to 1000 identities: try { var devicelist = registryManager.GetDevicesAsync(1000); return devicelist.Result; } catch (Exception ex) { // Export all identities to Azure blob storage: var blobClient = storageAccount.CreateCloudBlobClient(); string Containername = "iothubdevices"; //Get a reference to a container var container = blobClient.GetContainerReference(Containername); container.CreateIfNotExists(); //Generate a SAS token var storageUri = GetContainerSasUri(container); await registryManager.ExportDevicesAsync(storageUri, "devices1.txt", false); } Import all identities to Azure blob storage: await registryManager.ImportDevicesAsync(storageUri, OutputStorageUri); How it works... Let’s now understand the steps we performed. We initiated by creating a console application and configured it for the Azure IoT Hub solution. The idea behind this is to see the simple operation for device management. In this article, we started with simple operation for provision of the device by adding it to IoT Hub. We need to create connection to the IoT Hub followed by the created object of registry manager which is a part of devices namespace. Once we are connected we can perform operations like, add device, delete device, get device, these methods are asynchronous ones. IoT Hub also provides a way where in it connects with Azure storage blob for bulk operations like export all devices or import all devices, this works on JSON format only, the entire set of IoT devices gets exported in this way. There's more... Device identities are represented as JSON documents. It consists of properties like: deviceId: It represents the unique identification or the IoT device. ETag: A string representing a weak ETag for the device identity. symkey: A composite object containing a primary and a secondary key, stored in base64 format. status: If enabled, the device can connect. If disabled, this device cannot access any device-facing Endpoint. statusReason: A string that can be used to store the reason for the status changes. connectionState: It can be connected or disconnected. Device twins First we need to understand what device twin is and what is the purpose where we can use the device twin in any IoT solution. The device twin is a JSON formatted document that describes the metadata, properties of any device created within IoT Hub. It describes the individual device specific information. The device twin is made up of: tags, desired properties, and the reported properties. The operation that can be done by a IoT solution are basically update this the data, query for any IoT device. Tags hold the device metadata that can be accessed from IoT solution only. Desired properties are set from IoT solution and can be accessed on the device. Whereas the reported properties are set on the device and retrieved at IoT solution end. How to do it... Store device metadata: var patch = new { properties = new { desired = new { deviceConfig = new { configId = Guid.NewGuid().ToString(), DeviceOwner = "yatish", latitude = "17.5122560", longitude = "70.7760470" } }, reported = new { deviceConfig = new { configId = Guid.NewGuid().ToString(), DeviceOwner = "yatish", latitude = "17.5122560", longitude = "70.7760470" } } }, tags = new { location = new { region = "US", plant = "Redmond43" } } }; await registryManager.UpdateTwinAsync(deviceTwin.DeviceId, JsonConvert.SerializeObject(patch), deviceTwin.ETag); Query device metadata: var query = registryManager.CreateQuery("SELECT * FROM devices WHERE deviceId = '" + deviceTwin.DeviceId + "'"); Report current state of device: var results = await query.GetNextAsTwinAsync(); How it works... In this sample, we retrieved the current information of the device twin and updated the desired properties, which will be accessible on the device side. In the code, we will set the co-ordinates of the device with latitude and longitude values, also the device owner name and so on. This same value will be accessible on the device side. In the similar manner, we can set some properties on the device side which will be a part of the reported properties. While using the device twin we must always consider: Tags can be set, read, and accessed only by backend . Reported properties are set by device and can be read by backend. Desired properties are set by backend and can be read by backend. Use version and last updated properties to detect updates when necessary. Each device twin size is limited to 8 KB by default per device by IoT Hub There's more... Device twin metadata always maintains the last updated time stamp for any modifications. This is UTC time stamp maintained in the metadata. Device twin format is JSON format in which the tags, desired, and reported properties are stored, here is sample JSON with different nodes showing how it is stored: "tags": { "$etag": "1234321", "location": { "country": "India" "city": "Mumbai", "zipCode": "400001" } }, "properties": { "desired": { "latitude": 18.75, "longitude": -75.75, "status": 1, "$version": 4 }, "reported": { "latitude": 18.75, "longitude": -75.75, "status": 1, "$version": 4 } } Device direct methods Azure IoT Hub provides a fully managed bi-directional communication between the IoT solution on the backend and the IoT devices in the fields. When there is need for an immediate communication result, a direct method best suites the scenarios. Lets take example in home automation system, one needs to control the AC temperature or on/off the faucet showers. Invoke method from application: public async Task<CloudToDeviceMethodResult> InvokeDirectMethodOnDevice(string deviceId, ServiceClient serviceClient) { var methodInvocation = new CloudToDeviceMethod("WriteToMessage") { ResponseTimeout = TimeSpan.FromSeconds(300) }; methodInvocation.SetPayloadJson("'1234567890'"); var response = await serviceClient.InvokeDeviceMethodAsync(deviceId, methodInvocation); return response; } Method execution on device: deviceClient = DeviceClient.CreateFromConnectionString("", TransportType.Mqtt); deviceClient.SetMethodHandlerAsync("WriteToMessage", new DeviceSimulator().WriteToMessage, null).Wait(); deviceClient.SetMethodHandlerAsync("GetDeviceName", new DeviceSimulator().GetDeviceName, new DeviceData("DeviceClientMethodMqttSample")).Wait(); How it works... Direct method works on request-response interaction with the IoT device and backend solution. It works on timeout basis if no reply within that, it fails. These synchronous requests have by default 30 seconds of timeout, one can modify the timeout and increase up to 3600 depending on the IoT scenarios they have.  The device needs to connect using the MQTT protocol whereas the backend solution can be using HTTP. The JSON data size direct method can work up to 8 KB Device jobs In a typical scenario, device administrator or operators are required to manage the devices in bulk. We look at the device twin which maintains the properties and tags. Conceptually the job is nothing but a wrapper on the possible actions which can be done in bulk. Suppose we have a scenario in which we need to update the properties for multiple devices, in that case one can schedule the job and track the progress of that job. I would like to set the frequency to send the data at every 1 hour instead of every 30 min for 1000 IoT devices. Another example could be to reboot the multiple devices at the same time. Device administrators can perform device registration in bulk using the export and import methods. How to do it... Job to update twin properties. var twin = new Twin(); twin.Properties.Desired["HighTemperature"] = "44"; twin.Properties.Desired["City"] = "Mumbai"; twin.ETag = "*"; return await jobClient.ScheduleTwinUpdateAsync(jobId, "deviceId='"+ deviceId + "'", twin, DateTime.Now, 10); Job status. var twin = new Twin(); twin.Properties.Desired["HighTemperature"] = "44"; twin.Properties.Desired["City"] = "Mumbai"; twin.ETag = "*"; return await jobClient.ScheduleTwinUpdateAsync(jobId, "deviceId='"+ deviceId + "'", twin, DateTime.Now, 10); How it works... In this example, we looked at a job updating the device twin information and we can follow up the job for its status to find out if the job was completed or failed. In this case, instead of having single API calls, a job can be created to execute on multiple IoT devices. The job client object provides the jobs available with the IoT Hub using the connection to it. Once we locate the job using its unique ID we can retrieve the status for it. The code snippet mentioned in the How to do it... preceding recipe, uses the temperature properties and updates the data. The job is scheduled to start execution immediately with 10 seconds of execution timeout set. There's more... For a job, the life cycle begins with initiation from the IoT solution. If any job is in execution, we can query to it and see the status of execution. Another most common scenario where this could be useful is the firmware update, reboot, configuration updates, and so on, apart from the device property read or write. Each device job has properties that helps us working with them. The useful properties are start and end date time, status, and lastly device job statistics which gives the job execution statistics. Summary We have learned the device management using different techniques with Azure IoT Hub in detail. We have explained, how the IoT solution is going to help the users to manage the devices remotely, take actions from the cloud based application like disable, update data, run any command, and firmware update. We also performed different tasks for device management. Resources for Article: Further resources on this subject: Device Management in Zenoss Core Network and System Monitoring: Part 1 [article] Device Management in Zenoss Core Network and System Monitoring: Part 2 [article] Managing Network Devices [article]
Read more
  • 0
  • 0
  • 13929
article-image-article-authorizations-in-sap-hana
Packt
16 Jul 2013
28 min read
Save for later

Authorizations in SAP HANA

Packt
16 Jul 2013
28 min read
(For more resources related to this topic, see here.) Roles In SAP HANA, as in most of SAP's software, authorizations are grouped into roles. A role is a collection of authorization objects, with their associated privileges. It allows us, as developers, to define self-contained units of authorization. In the same way that at the start of this book we created an attribute view allowing us to have a coherent view of our customer data which we could reuse at will in more advanced developments, authorization roles allow us to create coherent developments of authorization data which we can then assign to users at will, making sure that users who are supposed to have the same rights always have the same rights. If we had to assign individual authorization objects to users, we could be fairly sure that sooner or later, we would forget someone in a department, and they would not be able to access the data they needed to do their everyday work. Worse, we might not give quite the same authorizations to one person, and have to spend valuable time correcting our error when they couldn't see the data they needed (or worse, more dangerous and less obvious to us as developers, if the user could see more data than was intended). It is always a much better idea to group authorizations into a role and then assign the role to users, than assign authorizations directly to users. Assigning a role to a user means that when the user changes jobs and needs a new set of privileges; we can just remove the first role, and assign a second one. Since, we're just starting out using authorizations in SAP HANA, let's get into this good habit right from the start. It really will make our lives easier later on. Creating a role Role creation is done, like all other SAP HANA development, in the Studio. If your Studio is currently closed, please open it, and then select the Modeler perspective. In order to create roles, privileges, and users, you will yourself need privileges. Your SAP HANA user will need the ROLE ADMIN, USER ADMIN, and CREATE STRUCTURED PRIVILEGE system privileges in order to do the development work in this article. You will see in the Navigator panel we have a Security folder, as we can see here: Please find the Security folder and then expand this folder. You will see a subfolder called Roles. Right-click on the Roles folder and select New Role to start creating a role. On the screen which will open, you will see a number of tabs representing the different authorization objects we can create, as we can see here: We'll be looking at each of these in turn, in the following sections, so for the moment just give your role Name (BOOKUSER might be appropriate, if not very original). Granted roles Like many other object types in SAP HANA, once you have created a role, you can then use it inside another role. This onion-like arrangement makes authorizations a lot easier to manage. If we had, for example, a company with two teams: Sales   Purchasing   And two countries, say: France   Germany   We could create a role giving access to sales analytic views, one giving purchasing analytic views, one giving access to data for France, and one giving access to data for Germany. We could then create new roles, say Sales-France, which don't actually contain any authorization objects themselves, but contain only the Sales and the France roles. The role definition is much simpler to understand and to maintain than if we had directly created the Sales-France role and a Sales-Germany role with all the underlying objects. Once again, as with other development objects, creating small self-contained roles and reusing them when possible will make your (maintenance) life easier. In the Granted Roles tab we can see the list of subroles this main role contains. Note that this list is only a pointer, you cannot modify the actual authorizations and the other roles given here, you would need to open the individual role and make changes there. Part of roles The Part of Roles tab in the role definition screen is exactly the opposite of the Granted Roles tab. This tab lists all other roles of which this role is a subrole. It is very useful to track authorizations, especially when you find yourself in a situation where a user seems to have too many authorizations and can see data they shouldn't be able to see. You cannot manipulate this list as such, it exists for information only. If you want to make changes, you need to modify the main role of which this role is a subrole. SQL privileges An SQL privilege is the lowest level at which we can define restrictions for using database objects. SQL privileges apply to the simplest objects in the database such as schemas, tables and so on. No attribute, analytical, or calculation view can be seen by SQL privileges. This is not strictly true, though you can consider it so. What we have seen as an analytical view, for example, the graphical definition, the drag and drop, the checkboxes, has been transformed into a real database object in the _SYS_BIC schema upon activation. We could therefore define SQL privileges on this database object if we wanted, but this is not recommended and indeed limits the control we can have over the view. We'll see a little later that SAP HANA has much finer-grained authorizations for views than this. An important thing to note about SQL privileges is that they apply to the object on which they are defined. They restrict access to a given object itself, but do not at any point have any impact on the object's contents. For example, we can decide that one of our users can have access to the CUSTOMER table, but we couldn't restrict their access to only CUSTOMER values from the COUNTRY USA. SQL privileges can control access to any object under the Catalog node in the Navigator panel. Let's add some authorizations to our BOOK schema and its contents. At the top of the SQL Privileges tab is a green plus sign button. Now click on this button to get the Select Catalog Object dialog, shown here: As you can see in the screenshot, we have entered the two letters bo into the filter box at the top of the dialog. As soon as you enter at least two letters into this box, the Studio will attempt to find and then list all database objects whose name contains the two letters you typed. If you continue to type, the search will be refined further. The first item in the list shown is the BOOK schema we created right back at the start of the book in the Chapter 2, SAP HANA Studio - Installation and First Look . Please select the BOOK item, and then click on OK to add it to our new role: The first thing to notice is the warning icon on the SQL Privileges tab itself: This means that your role definition is incomplete, and the role cannot be activated and used as yet. On the right of the screen, a list of checkbox options has appeared. These are the individual authorizations appropriate to the SQL object you have selected. In order to grant rights to a user via a role, you need to decide which of these options to include in the role. The individual authorization names are self-explicit. For example, the CREATE ANY authorization allows creation of new objects inside a schema. The INSERT or SELECT authorization might at first seem unusual for a schema, as it's not an object which can support such instructions. However, the usage is actually quite elegant. If a user has INSERT rights on the schema BOOK, then they have INSERT rights on all objects inside the schema BOOK. Granting rights on the schema itself avoids having to specify the names of all objects inside the schema. It also future-proofs your authorization concept, since new objects created in the schema will automatically inherit from the existing authorizations you have defined. On the far right of the screen, alongside each authorization is a radio button which gives an additional privilege, the possibility for a given user to, in turn, give the rights to a second user. This is an option which should not be given to all users, and so should not be present in all roles you create; the right to attribute privileges to users should be limited to your administrators. If you give just any user the right to pass on their authorizations further, you will soon find that you are no longer able to determine who can do what in your database. For the moment we are creating a simple role to show the working of the authorization concept in SAP HANA, so we will check all the checkboxes, and leave the radio buttons at No : There are some SQL privileges which are necessary for any user to be able to do work in SAP HANA. These are listed below. They give access to the system objects describing the development models we create in SAP HANA, and if a user does not have these privileges, nothing will work at all, the user will not be authorized to do anything. The SQL privileges you will need to add to the role in order to give access to basic SAP HANA system objects are: The SELECT privilege on the _SYS_BI schema   The SELECT privilege on the _SYS_REPO schema   The EXECUTE privilege on the REPOSITORY_REST procedure   Please add these SQL privileges to your role now, in order to obtain the following result: As you can see with the configuration we have just done, SQL privileges allow a user to access a given object and allow specific actions on the object. They do not however allow us to specify particular authorizations to the contents of the object. In order to use such fine-grained rights, we need to create an analytic privilege, and then add it to our role, so let's do that now. Analytic privileges An analytic privilege is an artifact unique to SAP HANA, it is not part of the standard SQL authorization concept. Analytic privileges allow us to restrict access to certain values of a given attribute, analytic, or calculation view. This means that we can create one view, which by default shows all available data, and then restrict what is actually visible to different users. We could restrict visible data by company code, by country, or by region. For example, our users in Europe would be allowed to see and work with data from our customers in Europe, but not those in the USA. An analytic privilege is created through the Quick Launch panel of Modeler , so please open that view now (or switch to the Quick Launch tab if it's already open). You don't need to close the role definition tab that's already open, we can leave it for now, create our analytic privilege, and then come back to the role definition later. From the Quick Launch panel, select Analytic Privilege , and then Create . As usual with SAP HANA, we are asked to give Name , Description , and select a package for our object. We'll call it AP_EU (for analytic privilege, Europe), use the name as the description, and put it into our book package alongside our other developments. As is common in SAP HANA, we have the option of creating an analytic privilege from scratch (Create New ) or copying an existing privilege (Copy From ). We don't currently have any other analytic privileges in our development, so leave Create New selected, then click on Next to go to the second screen of the wizard, shown here: On this page of the dialog, we are prompted to add development models to the analytic privilege. This will then allow us to restrict access to given values of these models. In the previous screenshot, we have added the CUST_REV analytic view to the analytic privilege. This will allow us to restrict access to any value we specify of any of the fields visible in the view. To add a view to the analytic privilege, just find it in the left panel, click on its name and then click on the Add button. Once you have added the views you require for your authorizations, click on the Finish button at the bottom of the window to go to the next step. You will be presented with the analytic privilege development panel, reproduced here: This page allows us to define our analytic privilege completely. On the left we have the list of database views we have included in the analytic privilege. We can add more, or remove one, using the Add and Remove buttons. To the right, we can see the Associated Attributes Restrictions and Assign Restrictions boxes. These are where we define the restrictions to individual values, or sets of values. In the top box, Associated Attributes Restrictions , we define on which attributes we want to restrict access (country code or region, maybe). In the bottom box, Assign Restrictions , we define the individual values on which to restrict (for example, for company code, we could restrict to value 0001, or US22; for region, we could limit access to EU or USA). Let's add a restriction to the REGION field of our CUST_REV view now. Click on the Add button next to the Associated Attributes Restrictions box, to see the Select Object dialog: As can be expected, this dialog lists all the attributes in our analytic view. We just need to select the appropriate attribute and then click on OK to add it to the analytic privilege. Measures in the view are not listed in the dialog. We cannot restrict access to a view according to numeric values. We cannot therefore, make restrictions to customers with a revenue over 1 million Euros, for example. Please add the REGION field to the analytic privilege now. Once the appropriate fields have been added, we can define the restrictions to be applied to them. Click on the REGION field in the Associated Attributes Restrictions box, then on the Add button next to the Assign Restrictions box, to define the restrictions we want to apply. As we can see, restrictions can be defined according to the usual list of comparison operators. These are the same operators we used earlier to define a restricted column in our analytic views. In our example, we'll be restricting access to those lines with a REGION column equal to EU, so we'll select Equal . In the Value column, we can either type the appropriate value directly, or use the value help button, and the familiar Value Help Dialog which will appear, to select the value from those available in the view. Please add the EU value, either by typing it or by having SAP HANA find it for us, now. There is one more field which needs to be added to our analytic privilege, and the reason behind might seem at first a little strange. This point is valid for SAP HANA SP5, up to and including (at least) release 50 of the software. If this point turns out to be a bug, then it might not be necessary in later versions of the software. The field on which we want to restrict user actions (REGION) is not actually part of the analytic view itself. REGION, if you recall, is a field which is present in CUST_REV , thanks to the included attribute view CUST_ATTR . In its current state, the analytic privilege will not work, because no fields from the analytic view are actually present in the analytic privilege. We therefore need to add at least one of the native fields of the analytic view to the analytic privilege. We don't need to do any restriction on the field; however it needs to be in the privilege for everything to work as expected. This is hinted at in SAP Note 1809199, SAP HANA DB: debugging user authorization errors. Only if a view is included in one of the cube restrictions and at least one of its attribute is employed by one of the dimension restrictions, access to the view is granted by this analytical privilege. Not an explicit description of the workings of the authorization concept, but close. Our analytic view CUST_REV contains two native fields, CURRENCY and YEAR. You can add either of these to the analytic privilege. You do not need to assign any restrictions to the field; it just needs to be in the privilege. Here is the state of the analytic privilege when development work on it is finished: The Count column lists the number of restrictions in effect for the associated field. For the CURRENCY field, no restrictions are defined. We just need (as always) to activate our analytic privilege in order to be able to use it. The activation button is the same one as we have used up until now to activate the modeling views, the round green button with the right-facing white arrow at the top-right of the panel, which you can see on the preceding screenshot. Please activate the analytic privilege now. Once that has been done, we can add it to our role. Return to the Role tab (if you left it open) or reopen the role now. If you closed the role definition tab earlier, you can get back to our role by opening the Security node in the Navigator panel, then opening Roles, and double-clicking on the BOOKUSER role. In the Analytic Privileges tab of the role definition screen, click on the green plus sign at the top, to add an analytic privilege to our role. The analytic privilege we have just created is called AP_EU, so type ap_eu into the search box at the top of the dialog window which will open. As soon as you have typed at least two characters, SAP HANA will start searching for matching analytic privileges, and your AP_EU privilege will be listed, as we can see here: Click on OK to add the privilege to the role. We will see in a minute the effect our analytic privilege has on the rights of a particular user, but for the moment we can take a look at the second-to-last tab in the role definition screen, System Privileges . System privileges As its name suggests, system privileges gives to a particular user the right to perform specific actions on the SAP HANA system itself, not just on a given table or view. These are particular rights which should not be given to just any user, but should be reserved to those users who need to perform a particular task. We'll not be adding any of these privileges to our role, however we'll take a look at the available options and what they are used for. Click on the green plus-sign button at the top of the System Privileges tab to see a list of the available privileges. By default the dialog will do a search on all available values; there are only fifteen or so, but you can as usual filter them down if you require using the filter box at the top of the dialog: For a full list of the system privileges available and their uses, please refer to the SAP HANA SQL Reference, available on the help.sap.com website at http://help.sap.com/hana/html/sql_grant.html. Package privileges The last tab in the role definition screen concerns Package Privileges . These allow a given user to access those objects in a package. In our example, the package is called book, so if we add the book package to our role in the Package Privileges tab, we will see the following result: Assigning package privileges is similar to assigning SQL privileges we saw earlier. We first add the required object (here our book package), then we need to indicate exactly which rights we give to the role. As we can see in the preceding screenshot, we have a series of checkboxes on the right-hand side of the window. At least one of these checkboxes must be checked in order to save the role. The individual rights have names which are fairly self-explanatory. REPO.READ gives access to read the package, whereas REPO.EDIT_NATIVE_OBJECTS allows modification of objects, for example. The role we are creating is destined for an end user who will need to see the data in a role, but should not need to modify the data models in any way (and in fact we really don't want them to modify our data models, do we?). We'll just add the REPO.READ privilege, on our book package, to our role. Again we can decide whether the end user can in turn assign this privilege to others. And again, we don't need this feature in our role. At this point, our role is finished. We have given access to the SQL objects in the BOOK schema, created an analytic privilege which limits access to the Europe region in our CUST_REV model, and given read-only access to our book package. After activation (always) we'll be able to assign our role to a test user, and then see the effect our authorizations have on what the user can do and see. Please activate the role now. Users Users are probably the most important part of the authorization concept. They are where all our problems begin, and their attempts to do and see things they shouldn't are the main reason we have to spend valuable time defining authorizations in the first place. In technical terms, a user is just another database object. They are created, modified, and deleted in the same way a modeling view is. They have properties (their name and password, for example), and it is by modifying these properties that we influence the actions that the person who connects using the user can perform. Up until now we have been using the SYSTEM user (or the user that your database administrator assigned to you). This user is defined by SAP, and has basically the authorizations to do anything with the database. Use of this user is discouraged by SAP, and the author really would like to insist that you don't use it for your developments. Accidents happen, and one of the great things about authorizations is that they help to prevent accidents. If you try to delete an important object with the SYSTEM user, you will delete it, and getting it back might involve a database restore. If however you use a development user with less authorization, then you wouldn't have been allowed to do the deletion, saving a lot of tears. Of course, the question then arises, why have you been using the SYSTEM user for the last couple of hundred pages of development. The answer is simple: if the author had started the book with the authorizations article, not many readers would have gotten past page 10. Let's create a new user now, and assign the role we have just created. From the Navigator panel, open the Security node, right-click on User , and select New User from the menu to obtain the user creation screen as shown in the following screenshot: Defining a user requires remarkably little information: User Name : The login that the user will use. Your company might have a naming convention for users. Users might even already have a standard login they use to connect to other systems in your enterprise. In our example, we'll create a user with the (once again rather unimaginative) name of BOOKU.   Authentication : How will SAP HANA know that the user connecting with the name of ANNE really is Anne? There are three (currently) ways of authenticating a user with SAP HANA. Password : This is the most common authentication system, SAP HANA will ask Anne for her password when she connects to the system. Since Anne is the only person who knows her password, we can be sure that Anne really is ANNE, and let her connect and do anything the user ANNE is allowed to do. Passwords in SAP HANA have to respect a certain format. By default this format is one capital, one lowercase, one number, and at least eight characters. You can see and change the password policy in the system configuration. Double-click on the system name in the Navigator panel, click on the Configuration tab, type the word pass into the filter box at the top of the tab, and scroll down to indexserver.ini and then password policy . The password format in force on your system is listed as password_layout . By default this is A1a, meaning capitals, numbers, and lowercase letters are allowed. The value can also contain the # character, meaning that special characters must also be contained in the password. The only special characters allowed by SAP HANA are currently the underscore, dollar sign, and the hash character. Other password policy defaults are also listed on this screen, such as maximum_password_lifetime (the time after which SAP HANA will force you to change your password).   Kerberos and SAML : These authentication systems need to be set up by your network administrator and allow single sign-on in your enterprise. This means that SAP HANA will be able to see the Windows username that is connecting to the system. The database will assume that the authentication part (deciding whether Anne really is ANNE) has already been done by Windows, and let the user connect.     Session Client : As we saw when we created attribute and analytic views back at the start of the book, SAP HANA understands the notion of client, referring to a partition system of the SAP ERP database. In the SAP ERP, different users can work in different Clients. In our development, we filtered on Client 100. A much better way of handling filtering is to define the default client for a user when we define their account. The Session Client field can be filled with the ERP Client in which the user works. In this way we do not need to filter on the analytic models, we can leave their client value at Dynamic in the view, and the actual value to use will be taken from the user record. Once again this means maintenance of our developments is a lot simpler. If you like, you can take a few minutes at the end of this article to create a user with a session client value of 100, then go back and reset our attribute and analytic views' default client value to Dynamic, reactivate everything, and then do a data preview with your test user. The result should be identical to that obtained when the view was filtered on client 100. However, if you then create a second user with a session client of 200, this second user will see different data.   We'll create a user with a password login, so type a password for your user now. Remember to adhere to the password policy in force on your system. Also note that the user will be required to change their password on first login. At the bottom of the user definition screen, as we can see from the preceding screenshot, we have a series of tabs corresponding to the different authorizations we can assign to our user. These are the same tabs we saw earlier when defining a role. As explained at the beginning of this article, it is considered best practice to assign authorizations to a role and then the role to a user, rather than assign authorizations directly to a user; this makes maintenance easier. For this reason we will not be looking at the different tabs for assigning authorizations to our user, other than the first one, Granted Roles . The Granted Roles tab lists, and allows adding and removing roles from the list assigned to the user. By default when we create a user, they have no roles assigned, and hence have no authorizations at all in the system. They will be able to log in to SAP HANA but will be able to do no development work, and will see no data from the system. Please click on the green plus sign button in the Granted Roles tab of the user definition screen, to add a role to the user account. You will be provided with the Select Role dialog, shown in part here: This dialog has the familiar search box at the top, so typing the first few letters of a role name will bring up a list of matching roles. Here our role was called BOOKUSER, so please do a search for it, then select it in the list and click on OK to add it to the user account. Once that is done, we can test our user to verify that we can perform the necessary actions with the role and user we have just created. We just need, as with all objects in SAP HANA, to activate the user object first. As usual, this is done with the round green button with the right-facing white arrow at the top-right of the screen. Please do this now. Testing our user and role The only real way to check if the authorizations we have defined are appropriate to the business requirements is to create a user and then try out the role to see what the user can and cannot see and do in the system. The first thing to do is to add our new user to the Studio so we can connect to SAP HANA using this new user. To do this, in the Navigator panel, right click on the SAP HANA system name, and select Add Additional User from the menu which appears. This will give you the Add additional user dialog, shown in the following screenshot:     Enter the name of the user you just created (BOOKU) and the password you assigned to the user. You will be required to change the password immediately: Click on Finish to add the user to the Studio. You will see immediately in the Navigator panel that we can now work with either our SYSTEM user, or our BOOKU user: We can also see straight away that BOOKU is missing the privileges to perform or manage data backups; the Backup node is missing from the list for the BOOKU user. Let's try to do something with our BOOKU user and see how the system reacts. The way the Studio lets you handle multiple users is very elegant, since the tree structure of database objects is duplicated, one per user, you can see immediately how the different authorization profiles affect the different users. Additionally, if you request a data preview from the CUST_REV analytic view in the book package under the BOOKU user's node in the Navigator panel, you will see the data according to the BOOKU user's authorizations. Requesting the same data preview from the SYSTEM user's node will see the data according to SYSTEM's authorizations. Let's do a data preview on the CUST_REV view with the SYSTEM user, for reference: As we can see, there are 12 rows of data retrieved, and we have data from the EU and NAR regions. If we ask for the same data preview using our BOOKU user, we can see much less data: BOOKU can only see nine of the 12 data rows in our view, as no data from the NAR region is visible to the BOOKU user. This is exactly the result we aimed to achieve using our analytic privilege, in our role, assigned to our user. Summary In this article, we have taken a look at the different aspects of the authorization concept in SAP HANA. We examined the different authorization levels available in the system, from SQL privileges, analytic privileges, system privileges, and package privileges. We saw how to add these different authorization concepts to a role, a reusable group of authorizations. We went on to create a new user in our SAP HANA system, examining the different types of authentications available, and the assignment of roles to users. Finally, we logged into the Studio with our new user account, and found out the first-hand effect our authorizations had on what the user could see and do. In the next article, we will be working with hierarchical data, seeing what hierarchies can bring to our reporting applications, and how to make the best use of them. Resources for Article : Further resources on this subject: SAP Netweaver: Accessing the MDM System [Article] SAP HANA integration with Microsoft Excel [Article] Exporting SAP BusinessObjects Dashboards into Different Environments [Article]
Read more
  • 0
  • 2
  • 13915

article-image-ibm-websphere-mq-commands
Packt
25 Aug 2010
10 min read
Save for later

IBM WebSphere MQ commands

Packt
25 Aug 2010
10 min read
(For more resources on IBM, see here.) After reading this article, you will not be a WebSphere MQ expert, but you will have brought your knowledge of MQ to a level where you can have a sensible conversation with your site's MQ administrator about what the Q replication requirements are. An introduction to MQ In a nutshell, WebSphere MQ is an assured delivery mechanism, which consists of queues managed by Queue Managers. We can put messages onto, and retrieve messages from queues, and the movement of messages between queues is facilitated by components called Channels and Transmission Queues. There are a number of fundamental points that we need to know about WebSphere MQ: All objects in WebSphere MQ are case sensitive We cannot read messages from a Remote Queue (only from a Local Queue) We can only put a message onto a Local Queue (not a Remote Queue) It does not matter at this stage if you do not understand the above points, all will become clear in the following sections. There are some standards regarding WebSphere MQ object names: Queue names, processes and Queue Manager names can have a maximum length of 48 characters Channel names can have a maximum length of 20 characters The following characters are allowed in names: A-Z,a-z,0-9, and . / _ % symbols There is no implied structure in a name — dots are there for readability Now let's move on to look at MQ queues in a little more detail. MQ queues MQ queues can be thought of as conduits to transport messages between Queue Managers. There are four different types of MQ queues and one related object. The four different types of queues are: Local Queue (QL), Remote Queue (QR), Transmission Queue (TQ), and Dead Letter Queue, and the related object is a Channel (CH). One of the fundamental processes of WebSphere MQ is the ability to move messages between Queue Managers. Let's take a high-level look at how messages are moved, as shown in the following diagram: When the application Appl1 wants to send a message to application Appl2, it opens a queue - the local Queue Manager (QM1) determines if it is a Local Queue or a Remote Queue. When Appl1 issues an MQPUT command to put a message onto the queue, then if the queue is local, the Queue Manager puts the message directly onto that queue. If the queue is a Remote Queue, then the Queue Manager puts the message onto a Transmission Queue. The Transmission Queue sends the message using the Sender Channel on QM1 to the Receiver Channel on the remote Queue Manager (QM2). The Receiver Channel puts the message onto a Local Queue on QM2. Appl2 issues a MQGET command to retrieve the message from this queue. Now let's move on to look at the queues used by Q replication and in particular, unidirectional replication, as shown in the following diagram. What we want to show here is the relationship between Remote Queues, Transmission Queues, Channels, and Local Queues. As an example, let's look at the path a message will take from Q Capture on QMA to Q Apply on QMB. (Move the mouse over the image to enlarge.) Note that in this diagram the Listeners are not shown. Q Capture puts the message onto a remotely-defned queue on QMA (the local Queue Manager for Q Capture). This Remote Queue (CAPA.TO.APPB.SENDQ.REMOTE) is effectively a "place holder" and points to a Local Queue (CAPA.TO.APPB.RECVQ) on QMB and specifes the Transmission Queue (QMB.XMITQ) it should use to get there. The Transmission Queue has, as part of its defnition, the Channel (QMA.TO.QMB) to use. The Channel QMA.TO.QMB has, as part of its defnition, the IP address and Listening port number of the remote Queue Manager (note that we do not name the remote Queue Manager in this defnition—it is specifed in the defnition for the Remote Queue). The defnition for unidirectional Replication Queue Map (circled queue names) is: SENDQ: CAPA.TO.APPB.SENDQ.REMOTE on the source RECVQ: CAPA.TO.APPB.RECVQ on the target ADMINQ: CAPA.ADMINQ.REMOTE on the target Let's look at the Remote Queue defnition for CAPA.TO.APPB.SENDQ.REMOTE, shown next. On the left-hand side are the defnitions on QMA, which comprise the Remote Queue, the Transmission Queue, and the Channel defnition. The defnitions on QMB are on the right-hand side and comprise the Local Queue and the Receiver Channel. Let's break down these defnitions to the core values to show the relationship between the different parameters, as shown next: We defne a Remote Queue by matching up the superscript numbers in the defnitions in the two Queue Managers: For defnitions on QMA, QMA is the local system and QMB is the remote system. For defnitions on QMB, QMB is the local system and QMA is the remote system. Remote Queue Manager name Name of the queue on the remote system Transmission Queue name Port number that the remote system is listening on The IP address of the Remote Queue Manager Local Queue Manager name Channel name Queue names: QMB: Decide on the Local Queue name on QMB—CAPA.TO.APPB.RECVQ. QMA: Decide on the Remote Queue name on QMB—CAPA.TO.APPB.SENDQ.REMOTE. Channels: QMB: Defne a Receiver Channel on QMB, QMA.TO.QMB—make sure the channel type (CHLTYPE) is RCVR. The Channel names on QMA and QMB have to match: QMA.TO.QMB. QMA: Defne a Sender Channel, which takes the messages from the Transmission Queue QMB.XMITQ and which points to the IP address and Listening port number of QMB. The Sender Channel name must be QMA.TO.QMB. Let's move on from unidirectional replication to bidirectional replication. The bidirectional queue diagram is shown next, which is a cut-down version of the full diagram of the The WebSphere MQ layer and just shows the queue names and types without the details. The principles in bidirectional replication are the same as for unidirectional replication. There are two Replication Queue Maps—one going from QMA to QMB (as unidirectional replication) and one going from QMB to QMA. MQ queue naming standards The naming of the WebSphere MQ queues is an important part of Q replication setup. It may be that your site already has a naming standard for MQ queues, but if it does not, then here are some thoughts on the subject (WebSphere MQ naming standards were discussed in the Introduction to MQ section): Queues are related to Q Capture and Q Apply programs, so it would be useful to have that fact refected in the name of the queues. A Q Capture needs a local Restart Queue and we use the name CAPA.RESTARTQ. Each Queue Manager can have a Dead Letter Queue. We use the prefx DEAD.LETTER.QUEUE with a suffx of the Queue Manager name giving DEAD.LETTER.QUEUE.QMA. Receive Queues are related to Send Queues. For every Send Queue, we need a Receive Queue. Our Send Queue names are made up of where they are coming from, Q Capture on QMA (CAPA), and where they are going to, Q Apply on QMB (APPB), and we also want to put in that it is a Send Queue and that it is a remote defnition, so we end up with CAPA.TO.APPB.SENDQ.REMOTE. The corresponding Receive Queue will be called CAPA.TO.APPB.RECVQ. Transmission Queues should refect the names of the "to" Queue Manager. Our Transmission Queue on QMA is called QMB.XMITQ, refecting the Queue Manager that it is going to, and that it is a Transmission Queue. Using this naming convention on QMB, the Transmission Queue is called QMA.XMITQ. Channels should refect the names of the "from" and "to" Queue Managers. Our Sender Channel defnition on QMA is QMA.TO.QMB refecting that it is a channel from QMA to QMB and the Receiver Channel on QMB is also called QMA.TO.QMB. The Receiver Queue on QMA is called QMB.TO.QMA for a Sender Channel of the same name on QMB. A Replication Queue Map defnition requires a local Send Queue, and a remote Receive Queue and a remote Administration Queue. The Send Queue is the queue that Q Capture writes to, the Receive Queue is the queue that Q Apply reads from, and the Administration Queue is the queue that Q Apply writes messages back to Q Capture with. MQ queues required for different scenarios This section lists the number of Local and Remote Queues and Channels that are needed for each type of replication scenario. The queues and channels required for unidirectional replication (including replicating to a Stored Procedure) and Event Publishing are shown in the following tables. Note that the queues and channels required for Event Publishing are a subset of those required for unidirectional replication, but creating extra queues and not using them is not a problem. The queues and channels required for unidirectional (including replicating to a Stored Procedure) are shown in the following table: QMA (7) QMB (7) 3 Local Queues: CAPA.ADMINQ CAPA.RESTARTQ DEAD.LETTER.QUEUE.QMA 1 Remote Queue: CAPA.TO.APPB.SENDQ.REMOTE 1 Transmission Queue: QMB.XMITQ 1 Sender Channel: QMA.TO.QMB 1 Receiver Channel: QMB.TO.QMA 2 Local Queues: CAPA.TO.APPB.REVCQ DEAD.LETTER.QUEUE.QMB 1 Remote Queue: CAPA.ADMINQ.REMOTE 1 Transmission Queue: QMA.XMITQ 1 Sender Channel: QMB.TO.QMA 1 Receiver Channel: QMA.TO.QMB 1 Model Queue: IBMQREP.SPILL.MODELQ   The queues required for Event Publishing are shown in the following table: QMA (7) QMB (7) 3 Local Queues: CAPA.ADMINQ CAPA.RESTARTQ DEAD.LETTER.QUEUE.QMA 1 Remote Queue: CAPA.TO.APPB.SENDQ.REMOTE 1 Transmission Queue: QMB.XMITQ 1 Sender Channel: QMA.TO.QMB 1 Receiver Channel: QMB.TO.QMA 2 Local Queues: CAPA.TO.APPB.REVCQ DEAD.LETTER.QUEUE.QMB 1 Receiver Channel: QMA.TO.QMB The queues and channels required for bidirectional/P2P two-way replication are shown in the following table: QMA (10) QMB (10) 4 Local Queues: CAPA.ADMINQ CAPA.RESTARTQ DEAD.LETTER.QUEUE.QMA CAPB.TO.APPA.RECVQ 2 Remote Queues: CAPA.TO.APPB.SENDQ.REMOTE CAPB.ADMINQ.REMOTE 1 Transmission Queue: QMB.XMITQ 1 Sender Channel: QMA.TO.QMB 1 Receiver Channel: QMB.TO.QMA 1 Model Queue: IBMQREP.SPILL.MODELQ 4 Local Queues: CAPB.ADMINQ CAPB.RESTARTQ DEAD.LETTER.QUEUE.QMB CAPA.TO.APPB.RECVQ 2 Remote Queues: CAPB.TO.APPA.SENDQ.REMOTE CAPA.ADMINQ.REMOTE 1 Transmission Queue: QMA.XMITQ 1 Sender Channel: QMB.TO.QMA 1 Receiver Channel: QMA.TO.QMB 1 Model Queue: IBMQREP.SPILL.MODELQ The queues and channels required for P2P three-way replication are shown in the following table: QMA (16) QMB (16) QMC (16) 5 Local Queues: CAPA.ADMINQ CAPA.RESTARTQ DEAD.LETTER.QUEUE. QMA CAPB.TO.APPA.RECVQ CAPC.TO.APPA.RECVQ 4 Remote Queues: CAPA.TO.APPB. SENDQ.REMOTE CAPB.ADMINQ.REMOTE CAPA.TO.APPC. SENDQ.REMOTE CAPC.ADMINQ.REMOTE 2 Transmission Queues: QMB.XMITQ QMC.XMITQ 2 Sender Channels: QMA.TO.QMC QMA.TO.QMB 2 Receiver Channels: QMC.TO.QMA QMB.TO.QMA 1 Model Queue: IBMQREP.SPILL. MODELQ 5 Local Queues: CAPB.ADMINQ CAPB.RESTARTQ DEAD.LETTER.QUEUE. QMB CAPA.TO.APPB.RECVQ CAPC.TO.APPB.RECVQ 4 Remote Queues: CAPB.TO.APPA. SENDQ.REMOTE CAPA.ADMINQ.REMOTE CAPB.TO.APPC. SENDQ.REMOTE CAPC.ADMINQ.REMOTE 2 Transmission Queues: QMA.XMITQ QMC.XMITQ 2 Sender Channels: QMB.TO.QMA QMB.TO.QMC 2 Receiver Channels: QMA.TO.QMB QMC.TO.QMB 1 Model Queue: IBMQREP.SPILL. MODELQ 5 Local Queues: CAPC.ADMINQ CAPC.RESTARTQ DEAD.LETTER. QUEUE.QMC CAPA.TO.APPC.RECVQ CAPB.TO.APPC.RECVQ 4 Remote Queues: CAPC.TO.APPA. SENDQ.REMOTE CAPA.ADMINQ.REMOTE CAPC.TO.APPB. SENDQ.REMOTE CAPB.ADMINQ.REMOTE 2 Transmission Queues: QMA.XMITQ QMB.XMITQ 2 Sender Channels: QMC.TO.QMA QMC.TO.QMB 2 Receiver Channels: QMA.TO.QMC QMB.TO.QMC 1 Model Queue: IBMQREP.SPILL. MODELQ
Read more
  • 0
  • 0
  • 13908

article-image-starting-out-backbox-linux
Packt
17 Feb 2014
11 min read
Save for later

Starting Out with BackBox Linux

Packt
17 Feb 2014
11 min read
(For more resources related to this topic, see here.) A flexible penetration testing distribution BackBox Linux is a very young project designed for penetration testing, vulnerability assessment and management. The key focus in using BackBox is to provide an independent security testing platform that can be easily customized with increased performance and stability. BackBox uses a very light desktop manager called XFCE. It includes the most popular security auditing tools that are essential for penetration testers and security advisers. The suite of tools includes web application analysis, network analysis, stress tests, computer sniffing forensic analysis, exploitation, documentation, and reporting. The BackBox repository is hosted on Launchpad and is constantly updated to the latest stable version of its tools. Adding and developing new tools inside the distribution requires it to be compliant with the open source community and particularly the Debian Free Software Guidelines criteria. IT security and penetration testing are dedicated sectors and quite new in the global market. There are a lot of Linux distributions dedicated to security; but if we do some research, we can see that only a couple of distributions are constantly updated. Many newly born projects stop at the first release without continuity and very few of them are updated. BackBox is one of the new players in this field and even though it is only a few years old, it has acquired an enormous user base and now holds the second place in worldwide rankings. It is a lightweight, community-built penetration testing distribution capable of running live in USB mode or as a permanent installation. BackBox now operates on release 3.09 as of September 2013, with a significant increase in users, thus becoming a stable community. BackBox is also significantly used in the professional world. BackBox is built on top of Ubuntu LTS and the 3.09 release uses 12.04 as its core. The desktop manager environment with XFCE and the ISO images are provided for 32-bit and 64-bit platforms (with the availability on Torrents and HTTP downloads from the project's website). The following screenshot shows the main view of the desktop manager, XFCE: The choice of desktop manager, XFCE, plays a very important role in BackBox. It is not only designed to serve the slender environment with medium and low level of resources, but also designed for very low memory. In case of very low memory and other resources (such as CPU, HD, and video), BackBox has an alternative way of booting the system without graphical user interface (GUI) and using command-line only, which requires really minimal amount of resources. With this aim in mind, BackBox is designed to function with pretty old and obsolete hardware to be used as a normal auditing platform. However, BackBox can be used on more powerful systems to perform actions that require the modern multicore processors to reduce ETA of the task such as brute-force attacks, data/password decryption, and password-cracking. Of course, the BackBox team aims to minimize overhead for the aforementioned cases through continuous research and development. Luckily, the majority of the tools included in BackBox can be performed in a shell/console environment and for the ones which require less resource. However, we always have our XFCE interface where we can access user-friendly GUI tools (in particular network analysis tools), which do not require many resources. Relatively, newcomer into the IT security and penetration testing environment, the first release of BackBox was back in September 09, 2010, as a project of the Italian web community. Now on its third major release and close to the next minor release (BackBox Linux 3.13 is planned for the end of January 2014), BackBox has grown rapidly and offers a wide scope for both amateur and professional use. The minimum requirements for BackBox are as follows: A 32-bit or 64-bit processor 512 MB of system memory RAM (256 MB in case there will be no desktop manager usage and only the console) 4.4 GB of disk space for installation Graphics card capable of 800 × 600 resolution (less resolution in case there will be no desktop manager usage) DVD-ROM drive or USB port The following screenshot shows the main view of BackBox with a toolbar at the bottom: The suite of auditing tools in BackBox makes the system complete and ready to use for security professionals of penetration testing. The organization of tools in BackBox. The entire set of BackBox security tools are populated into a single menu called Audit and structured into different subtasks as follows: Information Gathering Vulnerability Assessment Exploitation Privilege Escalation Maintaining Access Documentation & Reporting Social Engineering Stress Testing Forensic Analysis VoIP Analysis Wireless Analysis Miscellaneous We have to run through all the tools in BackBox by giving a short description of each single tool in the Auditing menu. The following screenshot shows the Auditing menu of BackBox: Information Gathering Information Gathering is the first absolute step of any security engineer and/or penetration tester. It is about collecting information on target systems, which can be very useful to start the assessment. Without this step, it will be quite difficult and hard to assess any system. Vulnerability Assessment After you've gathered information by performing the first step, the next step will be to analyze that information and its evaluation. Vulnerability Assessment is the process of identifying the vulnerabilities present in the system and prioritizing them. Exploitation Exploitation is the process where the weakness or bug in the software is used to penetrate the system. This can be done through the usage of an exploit, which is nothing but an automated script that is designed to perform a malicious attack on target systems. Privilege Escalation Privilege Escalation occurs when we have already gained access to the system but with low privileges. It can also be that we have legitimate access but not enough to make effective changes on the system, so we will need to elevate our privileges or gain access to another account with higher privileges. Maintaining Access Maintaining Access is about setting up an environment that will allow us to access the system again without repeating the tasks that we performed to gain access initially. Documentation & Reporting The Documentation & Reporting menu contains the tools that will allow us to collect the information during our assessment and generate a human readable report from them. Reverse Engineering The Reverse Engineering menu contains the suite of tools aimed to reverse the system by analyzing its structure for both hardware and software. Social Engineering Social Engineering is based on a nontechnical intrusion method, mainly on human interaction. It is the ability to manipulate the person and obtain his/her access credentials or the information that can introduce us to such parameters. Stress Testing The Stress Testing menu contains a group of tools aimed to test the stress level of applications and servers. Stress testing is the action where a massive amount of requests (for example, ICMP request) are performed against the target machine to create heavy traffic to overload the system. In this case, the target server is under severe stress and can be taken advantage of. For instance, the running services such as the web server, database or application server (for example, DDoS attack) can be taken down. Forensic Analysis The Forensic Analysis menu contains a great amount of useful tools to perform a forensic analysis on any system. Forensic analysis is the act of carrying out an investigation to obtain evidence from devices. It is a structured examination that aims to rebuild the user's history in a computer device or a server system. VoIP Analysis The voice over IP (VoIP) is a very commonly used protocol today in every part of the world. VoIP analysis is the act of monitoring and analyzing the network traffic with a specific analysis of VoIP calls. So in this section, we have a single tool dedicated to the analysis of VoIP systems. Wireless Analysis The Wireless Analysis menu contains a suite of tools dedicated to the security analysis of wireless protocols. Wireless analysis is the act of analyzing wireless devices to check their safety level. Miscellaneous The Miscellaneous menu contains tools that have different functionalities and can be placed in any section that we mentioned earlier, or in none of them. Services Apart from the Auditing menu, BackBox also has a Services menu. This menu is designed to populate the daemons of the tools, those which need to be manually initialized as a service. Update We have the Update menu that can be found in the main menu, just next to the Services menu. The Update menu contains the automated scripts to allow the users to update the tools that are out of APT automated system. Anonymous BackBox 3.13 has a new menu voice called Anonymous in the main menu. This menu contains a script that makes the user invisible to the network once started. The script populates a set of tools that anonymize the system while navigating, and connects to the global network, Internet. Extras Apart from the security-auditing tools, BackBox also has several privacy-protection tools. The suite of privacy-protection tools includes Tor, Polipo, and the Firefox safe mode that have been configured with a default profile in the private-browsing mode. There are many other useful tools recommended by the team but they are not included in the default ISO image. Therefore, the recommended tools are available in the BackBox repository and can be easily installed with apt-get (automated package installation tool for Debian-like systems). Completeness, accuracy, and support It is obvious that there are many alternatives when it comes to the choice of penetration testing tools for any particular auditing process. The BackBox team is mainly focused on the size of the tool library, performance, and the inclusion of the tools for security and auditing. The amount of tools included in BackBox is subject to accurate selection and testing by a team. Most of the security and penetration testing tools are implemented to perform identical functions. The BackBox team is very careful in the selection process in order to avoid duplicate applications and redundancies. Besides the wiki-based documentation provided for its set of tools, the repository of BackBox can also be imported into any of existing Ubuntu installation (or any of Debian derivative distro) by simply importing the project's Launchpad repository to the source list. Another point that the BackBox team focus their attention on is the size issue. BackBox may not offer the largest number of tools and utilities, but numbers are not equal to the quality. It has the essential tools installed by default that are sufficient to a penetration tester. However, BackBox is not a perfect penetration testing distribution. It is a very young project and aims to offer the best solution to the global community. Links and contacts BackBox is an open community where everybody's help is greatly welcomed. Here is a list of useful links to BackBox information on the Web: The BackBox main and official web page, where we can find general information about the distribution and the organization of the team, is available at http://www.BackBox.org/ The BackBox official blog, where we can find news about BackBox such as release notes and bug correction notifications, is available at http://www.BackBox.org/blog The BackBox official wikipage, where we can find many tutorials for the tools usage that are included in the distribution, is available at http://wiki.BackBox.org/ The BackBox official forum is the main discussion forum, where users can post their problems and also suggestions, is available at http://forum.BackBox.org/ The BackBox official IRC chat room is available at https://kiwiirc.com/client/irc.autistici.org:6667/?nick=BackBox_?#BackBox The BackBox official repository hosted on Launchpad, where the entire packages are located, is available at https://launchpad.net/~BackBox BackBox has also a Wikipedia page, where we can run through a brief history about how the project began, which is available at http://en.wikipedia.org/wiki/BackBox Summary In this article, we became more familiar with the BackBox environment by analyzing its menu structure and the way its tools are organized. We also provided a quick comment on each tool in BackBox. This is the only theoretical information regarding the introduction of BackBox. Resources for Article: Further resources on this subject: Penetration Testing and Setup [article] BackTrack 4: Security with Penetration Testing Methodology [article] Web app penetration testing in Kali [article]
Read more
  • 0
  • 0
  • 13907
article-image-optimizing-netscaler-traffic
Packt
14 Oct 2015
11 min read
Save for later

Optimizing NetScaler Traffic

Packt
14 Oct 2015
11 min read
In this article by Marius Sandbu, author of the book Implementing NetScaler VPX™ - Second Edition, explains the purpose of NetScaler is to act as a logistics department; it serves content to different endpoints using different protocols across different types of media, and it can either be a physical device or a device on top of a hypervisor within a private cloud infrastructure. Since there are many factors are at play here, there is room for tuning and improvement. Some of the topics we will go through in this article are as follows: Tuning for virtual environments Tuning TCP traffic (For more resources related to this topic, see here.) Tuning for virtual environments When setting up a NetScaler in a virtual environment, we need to keep in mind that there are many factors that influence how it will perform, for instance, the underlying CPU of the virtual host, NIC throughput and capabilities, vCPU over allocation, NIC teaming, MTU size, and so on. So, always important to remember the hardware requirements when setting up a NetScaler VPX on a virtualization host. Another important factor to keep in mind when setting up a NetScaler VPX is the concepts of Package Engines. By default, when we set up or import a NetScaler, it is set up with two vCPU. The first of these two is dedicated for management purpose and the second vCPU is dedicated to doing all the packet processing, such as content switching, SSL offloading, ICA-proxy, and so on. It is important to note that the second vCPU might be seen as 100% utilized in the hypervisor performance monitoring tools, but the correct way to check if it is being utilized is by using the CLI command stat system Now, by default, VPX 10 and VPX 200 only have support for one packet engine. This is because of the fact that due to its bandwidth limitations it does not require more packet engine CPUs to process the packets. On the other hand, VPX 1000 and VPX 3000 have support for up to 3 packet engines. This, in most cases, is needed to be able to process all the packets that are going through the system, if the bandwidth is going to be utilized to its fullest. In order to add a new packet engine, we need to assign more vCPUs to the VPX and more memory. Packet engines also have the benefit of load balancing processing between them, so instead of having a vCPU that is 100% utilized, we can even the load between multiple vCPUs and get an even better performance and bandwidth. The following chart shows the different editions and support for multiple packet engines: License/Memory 2 GB 4 GB 6 GB 8 GB 10 GB 12 GB VPX 10 1 1 1 1 1 1 VPX 200 1 1 1 1 1 1 VPX 1000 1 2 3 3 3 3 VPX 3000 1 2 3 3 3 3 It is important for us to remember that multiple PEs are only available for VMware, XenServer, and Hyper-V, but not on KVM. If we plan on using NIC-teaming on the underlying virtualization host, there are some important aspects to consider. Most of the different vendors have guidelines which describe the kinds of load balancing techniques that are available in the hypervisor. For instance, Microsoft has a guide that describes their features. You can find the guide at http://www.microsoft.com/en-us/download/details.aspx?id=30160. One of the NIC teaming options, called Switch Independent Dynamic Mode, has an interesting side effect; it replaces the source MAC address of the virtual machine with the one of the primary NIC on the host, hence we might experience packet loss on a VPX. Therefore, it is recommended in most cases that we have LACP/LAG, or in case of Hyper-V, use the HyperVPort distribution feature instead. Note that the features such as SRV-IO or PCI pass through are not supported for NetScaler VPX. NetScaler 11 also introduced the support for Jumbo Frames for the VPX, which allows for a much higher payload in an Ethernet frame. Instead of the traditional 1500 bytes, we can scale up to 9000 bytes of payload. This allows for a much lower overhead since the frames contain more data. This requires that the underlying NIC on the hypervisor supports this feature and is enabled as well, and this in most cases just work for communication with backend resources and not with users accessing public resources. This is because of the fact that most routers and ISP block such high MTU. This feature can be configured on at the Interface level in NetScaler, which can be done under System | Network | Interface then choose Select Interface and click on Edit. Here, we have an option called Maximum Transmission Unit which can be adjusted up to 9,216 Bytes. It is important to note that NetScaler communicates with backend resources using Jumbo frames, and then adjusts the MTU when communicating back with clients. It can also communicate with Jumbo frames in both paths, in case the NetScaler is set up as a backend load balancer. It is important to note that NetScaler only supports Jumbo frames load balancing for the following protocols: TCP TCP based protocols, like HTTP SIP RADIUS TCP tuning Much of the traffic which is going through NetScaler is based on the TCP protocol. Either it is ICA-Proxy, HTTP and so on. TCP is a protocol that provides reliable, error-checked delivery of packets back and forth. This ensures that data is successfully transferred before being processed further. TCP has many features to adjust bandwidth during transfer, congestion checking, adjusting segment sizing, and so on. We will delve a little into all these features in this section. We can adjust the way NetScaler uses TCP using TCP profiles, by default all services and vServers that are created on the NetScaler uses the default TCP profile nstcp_default_profile. Note that these profiles can be found by navigating to System | Profiles | TCP Profiles. Make sure not to alter the default TCP profile without properly consulting the network team, as this affects the way in which TCP works for all default services on the NetScaler. This default profile has most of the different TCP features turned off; this is to ensure compatibility with most infrastructures. The profile has not been adjusted much since it was first added in NetScaler. Citrix also has a lot of other different profiles depending on the use cases, so we are going to look a bit closer on the different options we have here. For instance, the profile nstcp_default_XA_XD_profile, which is intended for ICA-proxy traffic has some differences from the default profile, which are as follows: Window Scaling Selective Acknowledgement Forward Acknowledgement Use of Nagles Algorithm Window Scaling is a TCP option that allows the receiving point to accept more data than it is allowed in the TCP RFC for window size before getting an acknowledgement. By default, the window size is set to accept 65.536 bytes. With Window scaling enabled, it basically bitwise shifts the window size. This is an option that needs to be enabled on both endpoints in order to be used, and will only be sent in the initial three-way handshake. Select Acknowledgement (SACK) is a TCP option that allows for better handling of TCP retransmission. In a scenario of two hosts communicating where SACK is not enabled, and suddenly one of the hosts drops out of the network and loses some packets when they come back online it receives more packets from the other host. In this case the first host will ACK from the last packet it got from the other host before it dropped out. With SACK enabled, it will notify the other host of the last packet it got before it dropped out, and the other packets it received when he got back online. This allows for faster recovery of the communication, since the other host does not need to resend all the packets. Forward Acknowledgement (FACK) is a TCP option which works in conjunction with SACK and helps avoid TCP congestion by measuring the total number of data bytes that are outstanding in the network. Using the information from SACK, it can more precisely calculate how much data it can retransmit. Nagles Algoritm is a TCP feature that tries to cope with the small packet problem. Applications like Telnet often sends each keystroke within its own packet, creating multiple small packets containing only 1 byte of data, which results in a 41-byte packet for one keystroke. The algorithm works by combining a number of small outgoing messages into the same message, thus avoiding an overhead. Since ICA is a protocol that operates with many small packets, which might create congestion, Nagle is enabled in the TCP profile. Also, since many will be connecting using 3G or Wi-Fi, which might, in some cases, be unreliable to change channel, we need options that require the clients to be able to re-establish a connection quickly, which allows the use of SACK and FACK. Note that Nagle might have negative performance on applications that have their own buffering mechanism and operate inside LAN. If we take a look at another profile like nstcp_default_lan, we can see that FACK is disabled; this is because resources needed to calculate the amount of outstanding data in a high-speed network might be too much. Another important aspect of these profiles is the TCP congestion algorithms. For instance, nstcp_default_mobile uses the Westwood congestion algorithm; this is because it is much better at handling large bandwidth-delay paths, such as the wireless. The following congestion algorithms are available in NetScaler: Default (Based on TCP Reno) Westwood (Based on TCP Westwood+) BIC CUBIC Nile (Based on TCP Illinois) It is worth noting here that Westwood is aimed for 3G/4G connections or other slow wireless connections. BIC is aimed for high bandwidth connections with high latency, such as WAN connections. CUBIC is almost like BIC, but not as aggressive when it comes to fast-ramp and retransmissions. However, it is important to note that CUBIC is the default TCP algorithm in Linux kernels from 2.6.19 to 3.1 Nile is a newly created algorithm created by Citrix, which is based upon TCP Illinois and is targeted at high-speed, long-distance networks. It achieves higher throughput than standard TCP and is also compatible with standard TCP. So, here we can customize the algorithm that is better suited for a service. For instance, if we have a vServer that serves content to mobile devices, then we could use the nstcp_default_mobile TCP profile. There are also some other parameters that are important to keep in mind while working with the TCP profile. One of these parameters is multipath TCP. This is feature which allows endpoint which has multiple paths to a service. It is typically a mobile device which has WLAN and 3G capabilities, and allows the device to communicate with a service on a NetScaler using both channels at the same time. This requires that the device supports communication on both methods and that the service or application on the device supports Multipath TCP. So let's take an example of how a TCP profile might look like. If we have a vServer on NetScaler that is used to service an application to mobile devices. Meaning that the most common way that users can access this service is using 3G or Wi-Fi. The web service has its own buffering mechanism meaning that it tries not to send small packets over the link. The application is Multipath-TCP aware. In this scenario, we can leverage the nstcp_default_mobile profile, since it has most of the defaults for a mobile scenario, but we can also enable multipath TCP and create a new profile of it and bind it to the vServer. In order to bind a TCP profile to a vServer, we have go to a particular vServer | Edit | Profiles | TCP Profiles, as shown in the following screenshot: Note that AOL did a presentation of their own TCP customization on NetScaler; the presentation can be found at http://www.slideshare.net/masonke/net-scaler-tcpperformancetuningintheaolnetwork. It is also important to note that TCP should always be done in cooperation with the network team. Summary In this article, we have learned about Tuning for virtual environments and TCP tuning. Resources for Article: Further resources on this subject: Designing, Sizing, Building, and Configuring Citrix VDI-in-a-Box[article] XenMobile™ Solutions Bundle[article] Load balancing MSSQL [article]
Read more
  • 0
  • 0
  • 13890

article-image-defining-applications-policy-file
Packt
23 Aug 2013
21 min read
Save for later

Defining the Application's Policy File

Packt
23 Aug 2013
21 min read
(For more resources related to this topic, see here.) The AndroidManifest.xml file All Android applications need to have a manifest file. This file has to be named as AndroidManifest.xml and has to be placed in the application's root directory. This manifest file is the application's policy file. It declares the application components, their visibility, access rules, libraries, features, and the minimum Android version that the application runs against. The Android system uses the manifest file for component resolution. Thus, the AndroidManfiest.xml file is by far the most important file in the entire application, and special care is required when defining it to tighten up the application's security. The manifest file is not extensible, so applications cannot add their own attributes or tags. The complete list of tags with how these tags can be nested is as follows: <uses-sdk><?xml version="1.0" encoding="utf-8"?> <manifest> <uses-permission /> <permission /> <permission-tree /> <permission-group /> <instrumentation /> <uses-sdk /> <uses-configuration /> <uses-feature /> <supports-screens /> <compatible-screens /> <supports-gl-texture /> <application> <activity> <intent-filter> <action /> <category /> <data /> </intent-filter> <meta-data /> </activity> <activity-alias> <intent-filter> </intent-filter> <meta-data /> </activity-alias> <service> <intent-filter> </intent-filter> <meta-data/> </service> <receiver> <intent-filter> </intent-filter> <meta-data /> </receiver> <provider> <grant-uri-permission /> <meta-data /> <path-permission /> </provider> <uses-library /> </application> </manifest> Only two tags, <manifest> and <application>, are the required tags. There is no specific order to declare components. The <manifest> tag declares the application specific attributes. It is declared as follows: <manifest package="string" android_sharedUserId="string" android_sharedUserLabel="string resource" android_versionCode="integer" android_versionName="string" android_installLocation=["auto" | "internalOnly" | "preferExternal"] > </manifest> An example of the <manifest> tag is shown in the following code snippet. In this example, the package is named com.android.example, the internal version is 10, and the user sees this version as 2.7.0. The install location is decided by the Android system based on where it has room to store the application. <manifest package="com.android.example" android_versionCode="10" android_versionName="2.7.0" android_installLocation="auto" > The attributes of the <manifest> tag are as follows: package: This is the name of the package. This is the Java style namespace of your application, for example, com.android.example. This is the unique ID of your application. If you change the name of a published application, it is considered a new application and auto updates will not work. android:sharedUserId: This attribute is used if two or more applications share the same Linux ID. This attribute is discussed in detail in a later section. android:sharedUserLabel: This is the user readable name of the shared user ID and only makes sense if android:sharedUserId is set. It has to be a string resource. android:versionCode: This is the version code used internally by the application to track revisions. This code is referred to when updating an application with the more recent version. android:versionName: This is the version of the application shown to the user. It can be set as a raw string or as a reference, and is only used for display to users. android:installLocation: This attribute defines the location where an APK will be installed. The application tag is defined as follows: <application android_allowTaskReparenting=["true" | "false"] android_backupAgent="string" android_debuggable=["true" | "false"] android_description="string resource" android_enabled=["true" | "false"] android_hasCode=["true" | "false"] android_hardwareAccelerated=["true" | "false"] android_icon="drawable resource" android_killAfterRestore=["true" | "false"] android_largeHeap=["true" | "false"] android_label="string resource" android_logo="drawable resource" android_manageSpaceActivity="string" android_name="string" android_permission="string" android_persistent=["true" | "false"] android_process="string" android_restoreAnyVersion=["true" | "false"] android_supportsRtl=["true" | "false"] android_taskAffinity="string" android_theme="resource or theme" android_uiOptions=["none" | "splitActionBarWhenNarrow"] > </application> An example of the <application> tag is shown in the following code snippet. In this example, the application name, description, icon, and label are set. The application is not debuggable and the Android system can instantiate the components. <application android_label="@string/app_name" android_description="@string/app_desc" android_icon="@drawable/example_icon" android_enabled="true" android_debuggable="false"> </application> Many attributes of the <application> tag serve as the default values for the components declared within the application. These tags include permission, process, icon, and label. Other attributes such as debuggable and enabled are set for the entire application. The attributes of the <application> tag are discussed as follows: android:allowTaskReparenting: This value can be overridden by the <activity> element. It allows an Activity to re-parent with the Activity it has affinity with, when it is brought to the foreground. android:backupAgent: This attribute contains the name of the backup agent for the application. android:debuggable: This attribute when set to true allows an application to be debugged. This value should always be set to false before releasing the app in the market. android:description: This is the user readable description of an application set as a reference to a string resource. android:enabled: This attribute if set to true, the Android system can instantiate application components. This attribute can be overridden by components. android:hasCode: This attribute if set to true, the application will try to load some code when launching the components. android:hardwareAccelerated: This attribute when set to true allows an application to support hardware accelerated rendering. It was introduced in the API level 11. android:icon: This is the application icon as a reference to a drawable resource. android:killAfterRestore: This attribute if set to true, the application will be terminated once its settings are restored during a full-system restore. android:largeHeap: This attribute lets the Android system create a large Dalvik heap for this application and increases the memory footprint of the application, so this should be used sparingly. android:label: This is the user readable label for the application. android:logo: This is the logo for the application. android:manageSpaceActivity: This value is the name of the Activity that manages the memory for the application. android:name: This attribute contains the fully qualified name of the subclass that will be instantiated before any other component is started. android:permission: This attribute can be overridden by a component and sets the permission that a client should have to interact with the application. android:persistent: Usually used by a system application, this attribute allows the application to be running all the time. android:process: This is the name of the process in which a component will run. This can be overridden by any component's android:process attribute. android:restoreAnyVersion: This attribute lets the backup agent attempt a restore even if the backup currently stored is by a newer application than what is attempting to restore now. android:supportsRtl: This attribute when set to true supports right-to-left layouts. It was added in the API level 17. android:taskAffinity: This attribute lets all activities have affinity with the package name, unless it is set by the Activity explicitly. android:theme: This is a reference to the style resource for the application. android:uiOptions: This attribute if set to none, there are no extra UI options; if set to splitActionBarWhenNarrow, a bar is set at the bottom if constrained for the screen. In the following sections we will discuss how to handle specific requirements using the policy file. Application policy use cases This section discusses how to define the application policies using the manifest file. I have used use cases and we will discuss how to implement these use cases in the policy file. Declaring application permissions An application on the Android platform has to declare what resources it intends to use for proper functioning of the application. These are the permissions that are displayed to the user when they download the application. Application permissions should be descriptive so that users can understand them. Also, as is the general rule with security, it is important to request the minimum permissions required. Application permissions are declared in the manifest file by using the tag <uses-permission>. An example of a location-based manifest file that uses the GPS for retrieving location is shown in the following code snippet: <uses-permissionandroid:name="android. permission.ACCESS_COARSE_LOCATION" /> <uses-permissionandroid:name="android. permission.ACCESS_FINE_LOCATION" /> <uses-permissionandroid:name="android. permission.ACCESS_LOCATION_EXTRA_COMMANDS" /> <uses-permissionandroid:name="android. permission.ACCESS_MOCK_LOCATION" /> <uses-permissionandroid:name="android.permission.INTERNET" /> These permissions will be displayed to the users when they install the app, and can always be checked by going to Application under the settings menu. These permissions are seen in the following screenshot: Declaring permissions for external applications The manifest file also declares the permissions an external application (which does not run with the same Linux ID) needs to access the application components. This can be one of two places in the policy file: in the <application> tag or along with the component in the <activity>, <provider>, <receiver>, and <service> tag. If there are permissions that all components of an application require, then it is easy to specify them in the <application> tag. If a component requires some specific permission, then those can be defined in the specific component tag. Remember, only one permission can be declared in any of the tags. If a component is protected by permission then the component permission overrides the permission declared in the <application> tag. The following is an example of an application that requires external applications to have android.permission.ACCESS_COARSE_LOCATION to access its components and resources: <application android_allowBackup="true" android_icon="@drawable/ic_launcher" android_label="@string/app_name" android_permission="android. permission.ACCESS_COARSE_LOCATION"> If a Service requires that any application component that accesses it should have access to the external storage, then it can be defined as follows: <service android_enabled="true" android_name=".MyService" android_permission="android. permission.WRITE_EXTERNAL_STORAGE"> </service> If a policy file has both the preceding tags then when an external component makes a request to this Service, it should have android.permission.WRITE_EXTERNAL_STORAGE, as this permission will override the permission declared by the application tag. Applications running with the same Linux ID Sharing data between applications is always tricky. It is not easy to maintain data confidentiality and integrity. Proper access control mechanisms have to be put in place based on who has access to how much data. In this section, we will discuss how to share application data with the internal applications (signed by the same developer key). Android is a layered architecture with an application isolation enforced by the operating system itself. Whenever an application is installed on the Android device, the Android system gives it a unique user ID defined by the system. Notice that the two applications, example1 and example2, in the following screenshot are the applications run as separate user IDs, app_49 and app_50: However, an application can request the system for a user ID of its choice. The other application can then request the same user ID as well. This creates tight coupling and does not require components to be made visible to the other application or to create shared content providers. This kind of tight coupling is done in the manifest tags of all applications that want to run in the same process. The following is a snippet of manifest files of the two applications com.example.example1 and com.example.example2 that use the same user ID: <manifest package="com.example.example1" android_versionCode="1" android_versionName="1.0" android_sharedUserId="com.sharedID.example"> <manifest package="com.example.example2" android_versionCode="1" android_versionName="1.0" android_sharedUserId="com.sharedID.example"> The following screenshot is displayed when these two applications are running on the device. Notice that the applications, com.example.example1 and com.example.example2, now have the app ID of app_113. You will notice that the shared UID follows a certain format akin to a package name. Any other naming convention will result in an error such as an installation error: INSTALL_PARSE_FAILED_BAD_SHARED_USER_ID. All applications that share the same UID should have the same certificate. External storage Starting with API Level 8, Android provides support to store Android applications (APK files) on external devices, such as an SD card. This helps to free up internal phone memory. Once the APK is moved to external storage, the only memory taken up by the app is the private data of the application stored on internal memory. It is important to note that even for the SD card resident APKs, the DEX (Dalvik Executable) files, private data directories, and native shared libraries remain on the internal storage. Adding an optional attribute in the manifest file enables this feature. The application info screen for such an application either has a move to the SD card or move to a phone button depending on the current storage location of APK. The user then has an option to move the APK file accordingly. If the external device is un-mounted or the USB mode is set to Mass Storage (where the device is used as a disk drive), all the running activities and services hosted on that external device are immediately killed. The feature to enable storing APK on the external devices is enabled by adding the optional attribute android:installLocation in the application's manifest file in the <manifest> element. The attribute android:installLocation can have the following three values: InternalOnly: The Android system will install the application on the internal storage only. In case of insufficient internal memory, storage errors are returned. PreferExternal: The Android system will try to install the application on the external storage. In case there is not enough external storage, the application will be installed on the internal storage. The user will have the ability to move the app from external to internal storage and vice versa as desired. auto: This option lets the Android system decide the best install location for the application. The default system policy is to install the application on internal storage first. If the system is running low on internal memory, the application is then installed on the external storage. The user will have the ability to move the application from external to internal storage and vice versa as desired. For example, if android:installLocation is set to Auto, then on devices running a version of Android less than 2.2, the system will ignore this feature and APK will only be installed on the internal memory. The following is the code snippet from an application's manifest file with this option: <manifest package="com.example.android" android_versionCode="10" android_versionName="2.7.0" android_installLocation="auto" > The following is a screenshot of the application with the manifest file as specified previously. You will notice that Move to SD card is enabled in this case: In another application, where android:installLocation is not set, the Move to SD Card is disabled as shown in the following screenshot: Setting component visibility Any of the application components namely, activities, services, providers, and receivers can be made discoverable to the external applications. This section discusses the nuances of such scenarios. Any Activity or Service can be made private by setting android:exported=false. This is also the default value for an Activity. See the following two examples of a private Activity: <activity android_name=".Activity1" android_exported="false" /> <activity android_name=".Activity2" /> However, if you add an Intent Filter to the Activity, then the Activity becomes discoverable for the Intent in the Intent Filter. Thus, the Intent Filter should never be relied upon as a security boundary. See the following examples for Intent Filter declaration: <activity android_name=".Activity1" android_label="@string/app_name" > <intent-filter> <action android_name="android.intent.action.MAIN" /> <category android_name="android.intent.category.LAUNCHER" /> </intent-filter> </activity> <activity android_name=".Activity2"> <intent-filter> <action android_name="com.example.android. intent.START_ACTIVITY2" /> </intent-filter> </activity> Both activities and services can also be secured by an access permission required by the external component. This is done using the android:permission attribute of the component tag. A Content Provider can be set up for private access by using android:exported=false. This is also the default value for a provider. In this case, only an application with the same ID can access the provider. This access can be limited even further by setting the android:permission attribute of the provider tag. A Broadcast Receiver can be made private by using android:exported=false. This is the default value of the receiver if it does not contain any Intent Filters. In this case, only the components with the same ID can send a broadcast to the receiver. If the receiver contains Intent Filters then it becomes discoverable and the default value of android:exported is false. Debugging During the development of an application, we usually set the application to be in the debug mode. This lets developers see the verbose logs and can get inside the application to check for errors. This is done in the <application> tag by setting android:debuggable to true. To avoid security leaks, it is very important to set this attribute to false before releasing the application. An example of sensitive information that I have seen in my experience includes usernames and passwords, memory dumps, internal server errors, and even some funny personal notes state of a server and a developer's opinion about a piece of code. The default value of android:debuggable is false. Backup Starting with API level 8, an application can choose a backup agent to back up the device to the cloud or server. This can be set up in the manifest file in the <application> tag by setting android:allowBackup to true and then setting android:backupAgent to a class name. The default value of android:allowBackup is set to true and the application can set it to false if it wants to opt out of the backup. There is no default value for android:backupAgent and a class name should be specified. The security implications of such a backup are debatable as services used to back up the data are different and sensitive data, such as usernames and passwords can be compromised. Putting it all together The following example puts all the learning we have done so far to analyze AndroidManifest.xml provided with an Android SDK sample for RandomMusicPlayer. The manifest file specifies that this is version 1 of the application com.example.android.musicplayer. It runs on SDK 14 but supports backwards up to SDK 7. The application uses two permissions namely, android.permission.INTERNET and android.permission.WAKE_LOCK. The application has one Activity that is the entry point for the application called MainActivity, one Service called MusicService, and one receiver called MusicIntentReceiver. MusicService has defined custom actions called PLAY, REWIND, PAUSE, SKIP, STOP, and TOGGLE_PLAYBACK. The receiver uses the action intent android.media.AUDIO_BECOMING_NOISY and android.media.MEDIA_BUTTON defined by the Android system. None of the components are protected with permissions. An example of an AndroidManifst.xml file is shown in the following screenshot: Example checklist In this section, I have tried to put together an example list that I suggest you refer to whenever you are ready to release a version of your application. This is a very general version and you should adapt it according to your own use case and components. When creating a checklist think about issues that relate to the entire application, those that are specific to a component, and issues that might come up by setting the component and application specification together. Application level In this section, I have listed some questions that you should be asking yourself as you define the application specific preferences. They may affect how your application is viewed, stored, and perceived by users. Some application level questions that you may like to ask are as follows: Do you want to share resources with other applications that you have developed? Did you specify the unique user ID? Did you define this unique ID for another application either intentionally or unintentionally? Does your application require some capabilities such as camera, Bluetooth, and SMS? Does your application need all these permissions? Is there another permission that is more restrictive than the one you have defined? Remember the principle of least privilege Do all the components of your application need this permission or only a few? Check the spellings of all the permissions once again. The application may compile and work even if the permission spelling is incorrect. If you have defined this permission, is this the correct one that you need? At what API level does the application work? What is the minimum API level that your application can support? Are there any external libraries that your application needs? Did you remember to turn off the debug attribute before you release? If you are using a backup agent then remember to mention it here Did you remember to set a version number? This will help you during application upgrade Do you want to set an auto upgrade? Did you remember to sign the application with your release key? Sometimes setting a particular screen orientation will not allow your application to be visible on certain devices. For example, if your application only supports portrait mode then it might not appear for devices with landscape mode only. Where do you want to install the APK? Are there any services that might cease to work if the intent is not received in time? Do you want some other application level settings, such as the ability of the system to restore components? If defining a new permission, think twice if you really want them. Chances are there is already an existing permission that will cover your use case. Component level Some component level questions that you will want to think about in the policy are listed here. These are questions that you should be asking yourself for each component: Did you define all components? If using the third party libraries in your application, did you define all the components that you will use? Was there a particular setting that the third party library expects from your application? Do you want this component to be visible to other applications? Do you need to add some Intent Filters? If the component is not supposed to be visible, did you add Intent Filters? Remember as soon as you add Intent Filters, your component becomes visible. Do other components require some special permission to trigger this component? Verify the spelling of the permission name. Does your application require some capabilities such as camera, Bluetooth, and SMS? Summary In this article, we've learned how to define an applications policy file. The manifest file is the most important artifact of an application and should be defined with utmost care. This manifest file declares the permissions requested by an application and permissions that the external applications need to access its components. With the policy file we also define the storage location of the out APK and the minimum SDK against which the out application will run. The policy file exposes components that are not sensitive to the application. At the end of this article we discussed some sample issues that a developer should be aware of when writing a manifest file. In this article, we've learned about an Android application structure. Resources for Article: Further resources on this subject: So, what is Spring for Android? [Article] Animating Properties and Tweening Pages in Android 3-0 [Article] New Connectivity APIs – Android Beam [Article]
Read more
  • 0
  • 0
  • 13886
Modal Close icon
Modal Close icon