Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7019 Articles
article-image-understanding-material-design
Packt
18 Feb 2016
22 min read
Save for later

Understanding Material Design

Packt
18 Feb 2016
22 min read
Material can be thought of as something like smart paper. Like paper, it has surfaces and edges that reflect light and cast shadows, but unlike paper, material has properties that real paper does not, such as its ability to move, change its shape and size, and merge with other material. Despite this seemingly magical behavior, material should be treated like a physical object with a physicality of its own. Material can be seen as existing in a three-dimensional space, and it is this that gives its interfaces a reassuring sense of depth and structure. Hierarchies become obvious when it is instantly clear whether an object is above or below another. Based largely on age-old principles taken from color theory, animation, traditional print design, and physics, material design provides a virtual space where developers can use surface and light to create meaningful interfaces and movement to design intuitive user interactions. (For more resources related to this topic, see here.) Material properties As mentioned in the introduction, material can be thought of as being bound by physical laws. There are things it can do and things it cannot. It can split apart and heal again, and change color and shape, but it cannot occupy the same space as another sheet of material or rotate around two of its axes. We will be dealing with these properties throughout the book, but it is a good idea to begin with a quick look at the things material can and can't do. The third dimension is fundamental when it comes to material. This is what gives the user the illusion that they are interacting with something more tangible than a rectangle of light. The illusion is generated by the widening and softening of shadows beneath material that is closer to the user. Material exists in virtual space, but a space that, nevertheless, represents the real dimensions of a phone or tablet. The x axis can be thought of as existing between the top and bottom of the screen, the y axis between the right and left edges, and the z axis confined to the space between the back of the handset and the glass of the screen. It is for this reason that material should not rotate around the x or y axes, as this would break the illusion of a space inside the phone. The basic laws of the physics of material are outlined, as follows, in the form of a list: All material is 1 dp thick (along the z axis). Material is solid, only one sheet can exist in one place at a time and material cannot pass through other material. For example, if a card needs to move past another, it must move over it. Elevation, or position along the z axis, is portrayed by shadow, with higher objects having wider, softer shadows. The z axis should be used to prompt interaction. For example, an action button rising up toward the user to demonstrate that it can be used to perform some action. Material does not fold or bend. Material cannot appear to rise higher than the screen surface. Material can grow and shrink along both x and y axes. Material can move along any axis. Material can be spontaneously created and destroyed, but this must not be without movement. The arrivals and departures of material components must be animated. For example, a card growing from the point that it was summoned from or sliding off the screen when dismissed. A sheet of material can split apart anywhere along the x or y axes, and join together again with its original partner or with other material. This covers the basic rules of material behavior but we have said nothing of its content. If material can be thought of as smart paper, then its content can only be described as smart ink. The rules governing how ink behaves are a little simpler: Material content can be text, imagery, or any other form of visual digital content Content can be of any shape or color and behaves independently from its container material It cannot be displayed beyond the edges of its material container It adds nothing to the thickness (z axis) of the material it is displayed on Setting up a development environment The Android development environment consists mainly of two distinct components: the SDK, which provides the code libraries behind Android and Android Studio, and a powerful code editor that is used for constructing and testing applications for Android phones and tablets, Wear, TV, Auto, Glass, and Cardboard. Both these components can both be downloaded as a single package from http://developer.android.com/sdk/index.html. Installing Android Studio The installation is very straightforward. Run the Android Studio bundle and follow the on-screen instructions, installing HAXM hardware acceleration if prompted, and selecting all SDK components, as shown here: Android Studio is dependent on the Java JDK. If you have not previously installed it, this will be detected while you are installing Android Studio, and you will be prompted to download and install it. If for some reason it does not, it can be found at http://www.oracle.com/technetwork/java/javase/downloads/index.html, from where you should download the latest version. This is not quite the end of the installation process. There are still some SDK components that we will need to download manually before we can build our first app. As we will see next, this is done using the Android SDK Manager. Configuring the Android SDK People often refer to Android versions by name, such as Lollipop, or an identity number, such as 5.1.1. As developers, it makes more sense to use the API level, which in the case of Android 5.1.1 would be API level 22. The SDK provides a platform for every API level since API level 8 (Android 2.2). In this section, we will use the SDK Manager to take a closer look at Android platforms, along with the other tools included in the SDK. Start a new Android Studio project or open an existing one with the minimum SDK at 21 or higher. You can then open the SDK manager from the menu via Tools | Android | SDK Manager or the matching icon on the main toolbar. The Android SDK Manager can also be started as a stand alone program. It can be found in the /Android/sdk directory, as can the Android Virtual Device (AVD) manager. As can be seen in the preceding screenshot, there are really three main sections in the SDK: A Tools folder A collection of platforms An Extras folder All these require a closer look. The Tools directory contains exactly what it says, that is, tools. There are a handful of these but the ones that will concern us are the SDK manager that we are using now, and the AVD manager that we will be using shortly to create a virtual device. Open the Tools folder. You should find the latest revisions of the SDK tools and the SDK Platform-tools already installed. If not, select these items, along with the latest Build-tools, that is, if they too have not been installed. These tools are often revised, and it is well worth it to regularly check the SDK manager for updates. When it comes to the platforms themselves, it is usually enough to simply install the latest one. This does not mean that these apps will not work on or be available to devices running older versions, as we can set a minimum SDK level when setting up a project, and along with the use of support libraries, we can bring material design to almost any Android device out there. If you open up the folder for the latest platform, you will see that some items have already been installed. Strictly speaking, the only things you need to install are the SDK platform itself and at least one system image. System images are copies of the hard drives of actual Android devices and are used with the AVD to create emulators. Which images you use will depend on your system and the form factors that you are developing for. In this book, we will be building apps for phones and tablets, so make sure you use one of these at least. Although they are not required to develop apps, the documentation and samples packages can be extremely useful. At the bottom of each platform folder are the Google APIs and corresponding system images. Install these if you are going to include Google services, such as Maps and Cloud, in your apps. You will also need to install the Google support libraries from the Extras directory, and this is what we will cover next. The Extras folder contains various miscellaneous packages with a range of functions. The ones you are most likely to want to download are listed as follows: Android support libraries are invaluable extensions to the SDK that provide APIs that not only facilitate backwards compatibility, but also provide a lot of extra components and functions, and most importantly for us, the design library. As we are developing on Android Studio, we need only install the Android Support Repository, as this contains the Android Support Library and is designed for use with Android. The Google Play services and Google Repository packages are required, along with the Google APIs mentioned a moment ago, to incorporate Google Services into an application. You will most likely need the Google USB Driver if you are intending to test your apps on a real device. How to do this will be explained later in this chapter. The HAXM installer is invaluable if you have a recent Intel processor. Android emulators can be notoriously slow, and this hardware acceleration can make a noticeable difference. Once you have downloaded your selected SDK components, depending on your system and/or project plans, you should have a list of installed packages similar to the one shown next: The SDK is finally ready, and we can start developing material interfaces. All that is required now is a device to test it on. This can, of course, be done on an actual device, but generally speaking, we will need to test our apps on as many devices as possible. Being able to emulate Android devices allows us to do this. Emulating Android devices The AVD allows us to test our designs across the entire range of form factors. There are an enormous number of screen sizes, shapes, and densities around. It is vital that we get to test our apps on as many device configurations as possible. This is actually more important for design than it is for functionality. An app might operate perfectly well on an exceptionally small or narrow screen, but not look as good as we had wanted, making the AVD one of the most useful tools available to us. This section covers how to create a virtual device using the AVD Manager. The AVD Manager can be opened from within Android Studio by navigating to Tools | Android | AVD Manager from the menu or the corresponding icon on the toolbar. Here, you should click on the Create Virtual Device... button. The easiest way to create an emulator is to simply pick a device definition from the list of hardware images and keep clicking on Next until you reach Finish. However, it is much more fun and instructive to either clone and edit an existing profile, or create one from scratch. Click on the New Hardware Profile button. This takes you to the Configure Hardware Profile window where you will be able to create a virtual device from scratch, configuring everything from cameras and sensors, to storage and screen resolution. When you are done, click on Finish and you will be returned to the hardware selection screen where your new device will have been added: As you will have seen from the Import Hardware Profiles button, it is possible to download system images for many devices not included with the SDK. Check the developer sections of device vendor's web sites to see which models are available. So far, we have only configured the hardware for our virtual device. We must now select all the software it will use. To do this, select the hardware profile you just created and press Next. In the following window, select one of the system images you installed earlier and press Next again. This takes us to the Verify Configuration screen where the emulator can be fine-tuned. Most of these configurations can be safely left as they are, but you will certainly need to play with the scale when developing for high density devices. It can also be very useful to be able to use a real SD card. Once you click on Finish, the emulator will be ready to run. An emulator can be rotated through 90 degrees with left Ctrl + F12. The menu can be called with F2, and the back button with ESC. Keyboard commands to emulate most physical buttons, such as call, power, and volume, and a complete list can be found at http://developer.android.com/tools/help/emulator.html. Android emulators are notoriously slow, during both loading and operating, even on quite powerful machines. The Intel hardware accelerator we encountered earlier can make a significant difference. Between the two choices offered, the one that you use should depend on how often you need to open and close a particular emulator. More often than not, taking advantage of your GPU is the more helpful of the two. Apart from this built-in assistance, there are a few other things you can do to improve performance, such as setting lower pixel densities, increasing the device's memory, and building the website for lower API levels. If you are comfortable doing so, set up exclusions in your anti-virus software for the Android Studio and SDK directories. There are several third-party emulators, such as Genymotion, that are not only faster, but also behave more like real devices. The slowness of Android emulators is not necessarily a big problem, as most early development needs only one device, and real devices suffer none of the performance issues found on emulators. As we shall see next, real devices can be connected to our development environment with very little effort. Connecting a real device Using an actual physical device to run and test applications does not have the flexibility that emulators provide, but it does have one or two advantages of its own. Real devices are faster than any emulator, and you can test features unavailable to a virtual device, such as accessing sensors, and making and receiving calls. There are two steps involved in setting up a real phone or tablet. We need to set developer options on the handset and configure the USB connection with our development computer: To enable developer options on your handset, navigate to Settings | About phone. Tap on Build number 7 times to enable Developer options, which will now be available from the previous screen. Open this to enable USB debugging and Allow mock locations. Connect the device to your computer and check that it is connected as a Media device (MTP). Your handset can now be used as a test device. Depending on your We need only install the Google USB. Connect the device to your computer with a USB cable, start Android Studio, and open a project. Depending on your setup, it is quite possible that you are already connected. If not, you can install the Google USB driver by following these steps: From the Windows start menu, open the device manager. Your handset can be found under Other Devices or Portable Devices. Open its Properties window and select the Driver tab. Update the driver with the Google version, which can be found in the sdkextrasgoogleusb_driver directory. An application can be compiled and run from Android Studio by selecting Run 'app' from the Run menu, pressing Shift + F10, or clicking on the green play icon on the toolbar. Once the project has finished building, you will be asked to confirm your choice of device before the app loads and then opens on your handset. With a fully set up development environment and devices to test on, we can now start taking a look at material design, beginning with the material theme that is included as the default in all SDKs with APIs higher than 21. The material theme Since API level 21 (Android 5.0), the material theme has been the built-in user interface. It can be utilized and customized, simplifying the building of material interfaces. However, it is more than just a new look; the material theme also provides the automatic touch feedback and transition animations that we associate with material design. To better understand Android themes and how to apply them, we need to understand how Android styles work, and a little about how screen components, such as buttons and text boxes, are defined. Most individual screen components are referred to as widgets or views. Views that contain other views are called view groups, and they generally take the form of a layout, such as the relative layout we will use in a moment. An Android style is a set of graphical properties defining the appearance of a particular screen component. Styles allow us to define everything from font size and background color, to padding elevation, and much more. An Android theme is simply a style applied across a whole screen or application. The best way to understand how this works is to put it into action and apply a style to a working project. This will also provide a great opportunity to become more familiar with Android Studio. Applying styles Styles are defined as XML files and are stored in the resources (res) directory of Android Studio projects. So that we can apply different styles to a variety of platforms and devices, they are kept separate from the layout code. To see how this is done, start a new project, selecting a minimum SDK of 21 or higher, and using the blank activity template. To the left of the editor is the project explorer pane. This is your access point to every branch of your project. Take a look at the activity_main.xml file, which would have been opened in the editor pane when the project was created. At the bottom of the pane, you will see a Text tab and a Design tab. It should be quite clear, from examining these, how the XML code defines a text box (TextView) nested inside a window (RelativeLayout). Layouts can be created in two ways: textually and graphically. Usually, they are built using a combination of both techniques. In the design view, widgets can be dragged and dropped to form layout designs. Any changes made using the graphical interface are immediately reflected in the code, and experimenting with this is a fantastic way to learn how various widgets and layouts are put together. We will return to both these subjects in detail later on in the book, but for now, we will continue with styles and themes by defining a custom style for the text view in our Hello world app. Open the res node in the project explorer; you can then right-click on the values node and select the New | Values resource file from the menu. Call this file my_style and fill it out as follows: <?xml version="1.0" encoding="utf-8"?> <resources>     <style name="myStyle">         <item name="android:layout_width">match_parent</item>         <item name="android:layout_height">wrap_content</item>         <item name="android:elevation">4dp</item>         <item name="android:gravity">center_horizontal</item>         <item name="android:padding">8dp</item>         <item name="android:background">#e6e6e6</item>         <item name="android:textSize">32sp</item>         <item name="android:textColor">#727272</item>     </style> </resources> This style defines several graphical properties, most of which should be self-explanatory with the possible exception of gravity, which here refers to how content is justified within the view. We will cover measurements and units later in the book, but for now, it is useful to understand dp and sp: Density-independent pixel (dp): Android runs on an enormous number of devices, with screen densities ranging from 120 dpi to 480 dpi and more. To simplify the process of developing for such a wide variety, Android uses a virtual pixel unit based on a 160 dpi screen. This allows us to develop for a particular screen size without having to worry about screen density. Scale-independent pixel (sp): This unit is designed to be applied to text. The reason it is scale-independent is because the actual text size on a user's device will depend on their font size settings. To apply the style we just defined, open the activity_main.xml file (from res/layouts, if you have closed it) and edit the TextView node so that it matches this: <TextView     style="@style/myStyle"     android_text="@string/hello_world" /> The effects of applying this style can be seen immediately from the design tab or preview pane, and having seen how styles are applied, we can now go ahead and create a style to customize the material theme palette. Customizing the material theme One of the most useful features of the material theme is the way it can take a small palette made of only a handful of colors and incorporate these colors into every aspect of a UI. Text and cursor colors, the way things are highlighted, and even system features such as the status and navigation bars can be customized to give our apps brand colors and an easily recognizable look. The use of color in material design is a topic in itself, and there are strict guidelines regarding color, shade, and text, and these will be covered in detail later in the book. For now, we will just look at how we can use a style to apply our own colors to a material theme. So as to keep our resources separate, and therefore easier to manage, we will define our palette in its own XML file. As we did earlier with the my_style.xml file, create a new values resource file in the values directory and call it colors. Complete the code as shown next: <?xml version="1.0" encoding="utf-8"?> <resources>     <color name="primary">#FFC107</color>     <color name="primary_dark">#FFA000</color>     <color name="primary_light">#FFECB3</color>     <color name="accent">#03A9F4</color>     <color name="text_primary">#212121</color>     <color name="text_secondary">#727272</color>     <color name="icons">#212121</color>     <color name="divider">#B6B6B6</color> </resources> In the gutter to the left of the code, you will see small, colored squares. Clicking on these will take you to a dialog with a color wheel and other color selection tools for quick color editing. We are going to apply our style to the entire app, so rather than creating a separate file, we will include our style in the theme that was set up by the project template wizard when we started the project. This theme is called AppTheme, as can be seen by opening the res/values/styles/styles.xml (v21) file. Edit the code in this file so that it looks like the following: <?xml version="1.0" encoding="utf-8"?> <resources>     <style name="AppTheme" parent="android:Theme.Material.Light">         <item name="android:colorPrimary">@color/primary</item>         <item name="android:colorPrimaryDark">@color/primary_dark</item>         <item name="android:colorAccent">@color/accent</item>         <item name="android:textColorPrimary">@color/text_primary</item>         <item name="android:textColor">@color/text_secondary</item>     </style> </resources> Being able to set key colors, such as colorPrimary and colorAccent, allows us to incorporate our brand colors throughout the app, although the project template only shows us how we have changed the color of the status bar and app bar. Try adding radio buttons or text edit boxes to see how the accent color is applied. In the following figure, a timepicker replaces the original text view: The XML for this looks like the following lines: <TimePicker     android_layout_width="wrap_content"     android_layout_height="wrap_content"     android_layout_alignParentBottom="true"     android_layout_centerHorizontal="true" /> For now, it is not necessary to know all the color guidelines. Until we get to them, there is an online material color palette generator at http://www.materialpalette.com/ that lets you try out different palette combinations and download color XML files that can just be cut and pasted into the editor. With a complete and up-to-date development environment constructed, and a way to customize and adapt the material theme, we are now ready to look into how material specific widgets, such as card views, are implemented. Summary The Android SDK, Android Studio, and AVD comprise a sophisticated development toolkit, and even setting them up is no simple task. But, with our tools in place, we were able to take a first look at one of material design's major components: the material theme. We have seen how themes and styles relate, and how to create and edit styles in XML. Finally, we have touched on material palettes, and how to customize a theme to utilize our own brand colors across an app. With these basics covered, we can move on to explore material design further, and in the next chapter, we will look at layouts and material components in greater detail. To learn more about material design, the following books published by Packt Publishing (https://www.packtpub.com/) are recommended: Instant Responsive Web Design (https://www.packtpub.com/web-development/instant-responsive-web-design-instant) Mobile Game Design Essentials (https://www.packtpub.com/game-development/mobile-game-design-essentials) Resources for Article: Further resources on this subject: Speaking Java – Your First Game [article] Metal API: Get closer to the bare metal with Metal API [article] Looking Good – The Graphical Interface [article]
Read more
  • 0
  • 0
  • 10159

article-image-learning-create-and-edit-data-arcgis
Packt
18 Feb 2016
19 min read
Save for later

Learning to Create and Edit Data in ArcGIS

Packt
18 Feb 2016
19 min read
In this article by Daniela Cristiana Docan, author of the book Learning ArcGIS For Desktop, we will look at the ArcGIS editing tools to create and edit feature shapes and attributes. By the end of this article, you will learn the following: Spatially adjusting vector data to real-world units Editing feature shapes and attributes using ArcMap editing tools (For more resources related to this topic, see here.) Editing features in a geodatabase file In this article, we will work with three primary features’ geometries: point, line, and polygon. A point feature is defined by a single pair of x, y coordinates. The line and polygon features are defined by vertices and segments. The vertices represent ordered x, y coordinate pairs that form a feature's shape. The coordinates of a vertex also include the M (measure) and Z (elevation) values. A segment connects only two vertices. A line with two or more segments is called polyline. In ArcGIS, the line and polygon feature classes can store single or multipart features. A multipart feature is composed of two or more disconnected shapes that have only one record in the feature class attribute table. A simple point feature class cannot store multipart features, but a multipoint feature class can store multipoint features. In the editing process, the shape of a feature can be modified by moving, adding, or deleting the vertices that form its sketch. A sketch displays the location of vertices and segments that define the feature's shape. There are six main steps in the feature's shape editing process, which are as follows: Adding the data in a map document Starting an edit session Setting a feature template and snapping properties to edit the layer's feature geometry Creating a new feature by adding a new sketch or select an existing feature and display its sketch Editing a feature’s shape—moving, deleting, or inserting one or more vertices Saving the feature edits and stopping the edit session Using spatial adjustment In this section, we will perform a spatial adjustment for two feature classes imported from a CAD file that don't have a real-world coordinate system associated. We will use the Affine Transformation method that requires at least three displacement links. A displacement link defines the source and destination coordinates used by the transformation method. Before starting any spatial adjustment, be sure that the horizontal accuracy of the destination coordinates is at least the same or better as the horizontal accuracy of source coordinates. Follow these steps to perform a spatial adjustment in the ArcMap application: Start the ArcMap application and open the existing map document, SpatialAdjustment.mxd, from <drive>:LearningArcGISChapter05. In Table Of Contents, right-click on the Polyline layer and navigate to Properties | Source. Note that there is no projection information associated with the layer. Close the Layer Properties window. Repeat the step for the Polygon layer.First, we will add a basemap for the Brussels city from ArcGIS Online. Check whether ArcGIS for Desktop is connected to ArcGIS Online. In the Standard toolbar, from the drop-down menu next to Add Data, click on Add Data from ArcGIS Online. Search for the brusselsdata and select Brussels - Orthophoto 2012. Click on Details to read the information from the Description tab. Click on the yellow button called Add Data, as shown in the following screenshot: Drag the Brussels - Orthophoto 2012 basemap layer to the bottom of Table Of Contents. Right-click on the Brussels-Orthophoto 2012 basemap layer and go to Properties | Source. Inspect the resolution and the coordinate system of the orthophoto map. The layer's coordinate system is Belge_Lambert_1972002C, and the map projection is Lambert_Conformal_Conic.Second, we will use Spatial Adjustment tools to move the Polyline and Polygon layers to their correct coordinates using the Brussels-Orthophoto 2012 basemap layer as the destination coordinate space, as shown in the following screenshot: Add the Spatial Adjustment and Editor toolbars from Customize | Toolbars. To apply a spatial adjustment to the Polyline and Polygon layers, you must start the editing session. In the Editor toolbar, navigate to Editor | Start Editing. In the Spatial Adjustment toolbar, go to Spatial Adjustment | Set Adjust Data. Check All features in these layers[SI1]  and make sure both the Polyline and Polygon layers are selected. Click on OK. Then, in the Spatial Adjustment toolbar, go to Spatial Adjustment | Adjustment Methods. We will use the default transformation method called Transformation-Affine. Go to the Bookmarks menu and select Manage Bookmarks. Click on the Load button and add Source_points.dat and Destination_points.dat from <drive>:LearningArcGISChapter05. Keep Bookmarks Manager open and inspect Source and Destination for all the seven points by selecting the bookmark and clicking on Zoom To button. When finished, close the Bookmarks Manager window.Instead of adding the displacement links manually, you can load them through a links file. In the Spatial Adjustment toolbar, click on Link Table. Go to Spatial Adjustment | Links|Open Links File, navigate to <drive>:LearningArcGISChapter05, and select DisplacementLinks.txt. Now, click on Open. Link Table is automatically updated with seven coordinate pairs. You can modify or delete the individual links. In the Spatial Adjustment toolbar, navigate to Spatial Adjustment | Adjust. Note the RMS Error value: 4.9 meters. For a dataset with the reference scale of 1:15,000, an RMS value of 4.9 meters is a reasonable one. The assumed reference scale and the resulting RMS value should be mentioned in metadata file (at Item Description | Resource | Lineage). Next, close the Link Table. If you want add the displacement links manually, you should set up the snapping environment by checking Vertex Snapping in the Editor | Snapping toolbar. As you can note, the Polyline and Polygon layers were both transformed to match the Brussels-Orthophoto 2012 coordinate space. Inspect the results. In the Editor toolbar, navigate to Editor | Save Edits to save the spatial adjustment applied to the Polyline and Polygon layers.In the second part of the exercise, we will define the reference system for the Brussels_CADToGeodatabase feature dataset. Open the Catalog window and connect to the <drive>:LearningArcGISChapter05 folder. Expand World.gdb. Then, right-click on Brussels_CADToGeodatabase and go to Properties | XY Coordinate System. In the Search text box, type belge. Navigate to Projected Coordinate Systems | National Grids | Europe and select Belge Lambert 1972. Click on the Add To Favorites button. Then, click on Apply.You will see a warning message that tells you that it will modify the existing object as it cannot acquire a schema lock. However, you cannot modify the reference system while you are in an open edit session. Click on OK twice to close all dialog windows. In the Editor toolbar, navigate to Editor | Stop Editing. Repeat the last step to define the reference system for the Brussels_CADToGeodatabase feature dataset. Note that the Polyline and Polygon feature classes automatically inherit the feature dataset's reference system. When finished, save your map document as MySpatialAdjustment.mxd to <drive>:LearningArcGISChapter05. You can find the results of this exercise at <drive>:LearningArcGISChapter05 ResultsSpatialAdjustment. Editing features In this section, you will learn how to create a new feature class from an existing one using the Export Data tool in the ArcMap application. Next, you will explore different edit tools to create new features by splitting or combining the existing features. In the end, you will learn how to modify the feature's shape by adding, moving, and deleting vertices on its sketch. Before you start editing a feature sketch, you should always set the snapping properties, such as the snap type or snapping tolerance value. Snapping helps you create coincidence during the editing process. Snapping allows you to move the mouse pointer exactly to a vertex, edge, start point, or endpoint of a feature, when the pointer location is within a predefined distance of them. This distance is called snapping tolerance. ArcMap calculates the value of the snapping tolerance for you, but you can change it and specify your own value. Follow these steps to start editing features using the ArcMap application: Start the ArcMap application and open a new map document. Open the Catalog window, and navigate to..Chapter05EditingFeaturesWorld.gdb. Expand the Europefeature dataset. Select the EuropeCities_10k feature class and drag it on the ArcMap map display. The map shows the Paris and Brussels cities at a scale of 1:10,000. In Table Of Contents, right-click on theEuropeCities_10k layer and select Open Attribute Table. Dock the Table window to the bottom of the map display by dragging and dropping it on the down arrow of the blue centered target. Click on the Table Options button in the upper-left corner and select Select by Attributes. Build the following expression: CITIES = 'Bruxelles/Brussel'. Then, click on Apply and Close. Note that 32945 records are selected in the attribute table and on the map display. We will save these selected features in a new feature class called Brussels. Right-click on the EuropeCities_10k layer and select Data | Export Data. For Export, accept the Selected features option. Click on the browser button, make sure you are in the ..Chapter05EditingFeaturesWorld.gdbEurope folder, and type Brussels for the name of the feature class. Click on the No button to not add the exported data to map display as a layer. In the Catalog window, the Europe feature dataset is updated with your new polygon feature class. Hide the Catalog window by clicking on the Auto hide button. Then, close the Table window.Next, we will change the projection of the Brussels feature class and save it into the Brussels feature dataset. In the Standard toolbar, click on the ArcToolbox tool. Navigate to ArcToolbox | Data Management Tools | Projections and Transformations and select Projection. Set the following parameters:      Input Dataset or Feature Class: ..EditingFeaturesWorld.gdbEuropeBrussels      Output Dataset or Feature Class: ..EditingFeaturesWorld.gdbBrusselsBrusselsCity      Output Coordinate System: Belge_Lambert_1972      Geographic Transformation: Belge _1972_To_ETRS_1989_2 Click on OK. Click on No to not add the exported data to the map display as a layer. Hide the ArcToolbox window by clicking on the Auto hide button. On the Standard toolbar, click on the New tool to open a new map document. In the New Document dialog window, for Default geodatabase for this map, click on the browser button and navigate to <drive>:LearningArcGISChapter05EditingFeaturesWorld.gdb. Click on Add and OK. Finally, click on No to not save the changes from the last map document. Add the BrusselsCity and Buildings feature classes from the Brussels feature dataset in ArcMap by dragging them on the map display. Change the BrusselsCity symbol to Hollow, Outline Width to 1.5 , and Outline Color to Medium Coral Light. Label the features with the OBJECTID field's values. Change the label to Medium Coral Light. Change the Buildings symbol to Hollow, Outline Width to 1.5, and Outline Color to Solar Yellow. Label the Buildings features with the OBJECTID field's values. Navigate to Add Data | Add Data From ArcGIS Online to add two aerial photos of Brussels: Brussels -Othotophoto 2004 and Brussels -Othotophoto 2012. In Table Of Contents, drag Brussels -Othotophoto 2004 to the bottom of list. From the Bookmarks menu, select Manage Bookmarks. Click on the Load button, navigate to <drive>:LearningArcGISChapter05, and select the EditingFeaturesAttributes.dat ArcGIS Place file. Click on Open. These bookmarks will help us identify particular study areas in the next exercises. Click on Close. When finished, save your map document as MyEditingFeatures.mxd at <drive>:LearningArcGISChapter05EditingFeatures.In the next steps, we will start editing the building footprints from the Buildings layer. These buildings were digitized using an aerial photo from 2004 (Brussels -Othotophoto 2004) as layer reference. We will update the features to reflect the changes that were made from 2004 to 2012 using Brussels -Othotophoto 2012 as the reference layer. First, we will set the Buildings layer as the only selectable layer in the map. At the top of Table Of Contents, click on the List By Selection button. Both the BrusselsCity and Buildings layers will be selectable. Click on the Selectable button next to the BrusselsCity layer to move it to the Not Selectable list. Return to the List By Drawing Order layers’ list mode. From Bookmarks, select Edit 1. You will divide a feature into five separate features, as shown in the following screenshot: In Table Of Contents, right-click onthe Buildings layer and select Open Attribute Table. Dock the Table window to the bottom of the map display. First, we will start an edit session and set the snapping properties. If necessary, add the Editor toolbar. In the Editor toolbar, go to Editor | Start Editing. Second, we will create a feature template. If the Create Features window isn't open, you should navigate to Editor | Editing Windows | Create Features. As the BrusselsCity and Buildings layers reference two feature classes that are stored in the same workspace called World.gdb, the Create Features window displays the default feature templates for both of them. In the Create Features window, double-click on the Buildings template to open the Template Properties window. Inspect the information for the template displays. For Description, type Brussels buildings . For Default Tool, accept Polygon. Click on OK to update the Buildings feature template. At the bottom of the Created Features window, in the Construction Tools list, make sure that the Polygon tool is selected. Next, we will use the default snapping environment to enable snapping to feature vertices. From the Editor menu, navigate to Snapping | Snapping Toolbar. In the Snapping toolbar, turn off all snapping types, except the Vertex Snapping button. During the edit process, your mouse pointer will be moved exactly only to the vertices of a feature. ArcMap allows you to activate the classic snapping environment if you need more control to snapping properties, such as the individual type of snapping on each editable layer or snapping tolerance units (pixels or map units). To activate the classic snapping, in the Editor menu, navigate to Editor | Options | General and take a look at Use classic snapping. From the Editor menu, navigate to Snapping | Snapping Window to manage the individual snapping on layers. To change the snapping tolerance units, go to Editor | Snapping | Options. In the Tools toolbar, click on the Select Feature tool and select the building shown in the preceding screenshot. At the bottom of the Table window, click on the Show selected records button to see only the selected building’s record. In the Editor toolbar, select the Cut Polygons tool. Note that another tool called Straight Segment is selected. You will cut the polygon feature using a multipart polyline. Move the pointer close to the start point (1) indicated before to see the SnapTip  called Buildings: Vertex. Click on the 1 point to add the first vertex of the segment. Click on the 2 point and again on the 3 point. Then, right-click on the third vertex and select Finish Part. If you want to delete a vertex, right-click on it and select Delete Vertex.You can use the Zoom in, Zoom out, and Pan tools while you are splitting the polygon, but you have to select the Cut Polygons tool again to continue adding vertices. Click on the 4, 5, and 6 points, as shown in the preceding screenshot. Right-click on the sixth vertex and select Finish Part. Repeat the steps for the rest of points. Right-click on the twelfth vertex (Sketch: Enpoint) and select Finish Sketch. Five new features are created and selected on the map display. In the Table window, note that the Buildings feature class table is updated with five new building records corresponding to the five polygons on the map display. To unselect the new features on the map, use the Clear Selected Features tool on the Tools toolbar. To save your edits to the data, select Save Edits in the Editor menu.In the next steps, we will continue to update the buildings by erasing the two buildings that have been demolished since 2010, and we will merge the two existing features to represent a larger commercial building, as shown in the following screenshot: We will continue to use the Buildings feature template. From Bookmarks, select Edit 2. Add the Effects toolbar on the map display and select Brussels-Orthophoto 2012 from the drop-down list of layers. Select the Swipe button. Then, click on the map display and drag to see the differences between the two ortophoto maps. Use the Select Feature tool and the Shift key to select two buildings that have the following OBJECTID values: 53 and 52. Right-click on the map display and select Delete. You have a second option of removing them from the Buildings layer using the Delete Selected tool in the Table window. During an edit session, you should use the Edit tool to select a feature by clicking on it. As we don't set Sticky move tolerance at Editor | Options | General, we have to use the Select Feature tool to prevent the accidental moving of the selected features. Use the Select Feature tool to select the buildings with the OBJECTID values 51 and 54. In the Table window, inspect the attribute values of the building with the ID 51. In the Editor toolbar, navigate to Editor | Merge. In the Merge dialog window, click on the first feature Industrial (Buildings). The selected feature flashes on the map. As the first feature in the list is the larger one that has a name, we would like to keep its attribute values. Click on OK. In the Table window, note that the OBJECTID, Type, and Name values were inherited from the previous selected feature. Instead, the SHAPE_Length and SHAPE_Area fields are updated by ArcMap.In the next steps, we will reshape the building with OBJECTID 51, as shown in the following screenshot: To edit the building shape, you must display its sketch. In the Editor toolbar, select the Edit Tool button, double-click on the building to select it, and display the feature's sketch, as shown in the preceding screenshot. The Edit Vertices mini toolbar is automatically added on the map. Drag and draw a box over the four vertices as shown before. Move the pointer over one of the selected vertices until it turns in a compass arrow, right-click, and select Delete Vertices. If you make a mistake, use the Undo Delete tool from the Standard toolbar and repeat the step. To erase the vertices of two or more sketches, you can also draw a box over the vertices using the Delete Vertex tool from the Edit Vertices mini toolbar. Click away from the selected feature to unselect it and see the changes. In the Editor toolbar, select the Edit Tool button again and double-click on the building to display the feature's sketch. On the Edit Vertices toolbar, select Add Vertex. Click on the line segment and add two vertices, as shown in the preceding screenshot. Click on the first new vertex (1) to select it. Move the pointer over the selected vertex until it turns in a compass arrow. Drag and drop the vertex to the new location. Now, click on the second new vertex (2) to select it. You will move this feature's vertex to an exact x, y location. Right-click on vertex and select Move To Fist; then, click on the down arrow next to the X box and check whether Meter is selected. In the X box, select and delete the existing value. Type 154266.227 and press the Tab key. For Y, type 172231.383 and press the Enter key. Note that the vertex has moved to the exact location. Press the F2 key to finish the sketch. Even if your feature shape changes in your map document, the edits of the data is not save in your feature class. To store the updated date in your file geodatabase, you must save your edits during an edit session or when you stop the edit session. In the Editor toolbar, navigate to Editor | Save Edits. Saving your map document does not save the data edits to your geodatabase.In the last steps, we will move three adjacent buildings and reshape one of them, as shown in the following screenshot: From Bookmarks, select Edit 4. In the Editor toolbar, click on the Edit tool. Drag and draw a box over the three buildings, as shown in the preceding screenshot. Note that the Start and End points of the box do not touch the polygon’s features. This way, you can select three adjacent buildings with the Edit tool without accidentally producing small position moves. To move all the buildings 5 meters on the Y axis, from the Editor toolbar, go to Editor | Move. In the Delta X, Y window, type 5 in the second box and press the Enter key. All the three buildings will shift 5 meters on the Yaxis. Next, we will adjust the shape of a building. In the Tool toolbar, click on the Clear Selected Features tool to unselect the buildings. With the Edit tool, select the building with ID 45 and then click on the Reshape Feature tool on the Editor toolbar. Click on the 1 point to add the first vertex of the line segment. Click on the 2 point to add the second vertex, and then click on the 3 point to add the third vertex. Press the F2 key to finish the sketch. Inspect the results. Unselect the building and save the edits. In the Editor toolbar, stop the edit session by navigating to Editing | Stop Editing. When finished, save your changes to the map document. Leave the MyEditingFeatures map document open to continue working with map topology in the next recipe. You can find the results at <drive>:LearningArcGISChapter05 ResultsEditingFeaturesEditingFeatures.mxd. Summary In this article, we identified the main steps in the editing process. You learned how to modify the existing features and add new features. Also, we explored different ArcMap instruments to edit the coincident features—that is, using the Auto-Complete Polygons and map topology tools. Resources for Article: Further resources on this subject: Extracting Real-Time Wildfire Data from ArcGIS Server with the ArcGIS REST API[article] ArcGIS – Advanced ArcObjects[article] ArcGIS Spatial Analyst[article]
Read more
  • 0
  • 0
  • 2711

article-image-build-first-person-shooter
Packt
18 Feb 2016
29 min read
Save for later

Build a First Person Shooter

Packt
18 Feb 2016
29 min read
We will be creating a first person shooter; however, instead of shooting a gun to damage our enemies, we will be shooting a picture in a survival horror environment; similar to the Fatal Frame series of games and the recent indie title DreadOut. To get started on our project, we're first going to look at creating our level or, in this case, our environments starting with the exterior. In the game industry, there are two main roles in level creation: the environment artist and level designer. An environment artist is a person who builds the assets that go into the environment. He/she uses tools such as 3ds Max or Maya to create the model, and then uses other tools such as Photoshop to create textures and normal maps. The level designer is responsible for taking the assets that the environment artist has created and assembling them into an environment for players to enjoy. He/she designs the gameplay elements, creates the scripted events, and tests the gameplay. Typically, a level designer will create environments through a combination of scripting and using a tool that may or may not be in development as the game is being made. In our case, that tool is Unity. One important thing to note is that most companies have their own definition for different roles. In some companies, a level designer may need to create assets and an environment artist may need to create a level layout. There are also some places that hire someone to just do lighting, or just to place meshes (called a mesher) because they're so good at it. (For more resources related to this topic, see here.) Project overview In this article, we take on the role of an environment artist whose been tasked with creating an outdoor environment. We will use assets that I've placed in the example code as well as assets already provided to us by Unity for mesh placement. In addition to this, you will also learn some beginner-level design. Your objectives This project will be split into a number of tasks. It will be a simple step-by-step process from the beginning to end. Here is an outline of our tasks: Creating the exterior environment – Terrain Beautifying the environment – adding water, trees, and grass Building the atmosphere Designing the level layout and background The project setup At this point, I assume you have a fresh installation of Unity and have started it. You can perform the following steps: With Unity started, navigate to File | New Project. Select a project location of your choice somewhere on your hard drive and ensure that you have Setup defaults for set to 3D. Once completed, click on Create. Here, if you see the Welcome to Unity pop up, feel free to close it as we won't be using it. Level design 101 – planning Now just because we are going to be diving straight into Unity, I feel it's important to talk a little more about how level design is done in the gaming industry. While you may think a level designer will just jump into the editor and start playing, the truth is you normally would need to do a ton of planning ahead of time before you even open up your tool. Generally, a level design begins with an idea. This can come from anything; maybe you saw a really cool building, or a photo on the Internet gave you a certain feeling; maybe you want to teach the player a new mechanic. Turning this idea into a level is what a level designer does. Taking all of these ideas, the level designer will create a level design document, which will outline exactly what you're trying to achieve with the entire level from start to end. A level design document will describe everything inside the level; listing all of the possible encounters, puzzles, so on and so forth, which the player will need to complete as well as any side quests that the player will be able to achieve. To prepare for this, you should include as many references as you can with maps, images, and movies similar to what you're trying to achieve. If you're working with a team, making this document available on a website or wiki will be a great asset so that you know exactly what is being done in the level, what the team can use in their levels, and how difficult their encounters can be. Generally, you'll also want a top-down layout of your level done either on a computer or with a graph paper, with a line showing a player's general route for the level with encounters and missions planned out. Of course, you don't want to be too tied down to your design document and it will change as you playtest and work on the level, but the documentation process will help solidify your ideas and give you a firm basis to work from. For those of you interested in seeing some level design documents, feel free to check out Adam Reynolds (Level Designer on Homefront and Call of Duty: World at War) at http://wiki.modsrepository.com/index.php?title=Level_Design:_Level_Design_Document_Example. If you want to learn more about level design, I'm a big fan of Beginning Game Level Design, John Feil (previously my teacher) and Marc Scattergood, Cengage Learning PTR. For more of an introduction to all of game design from scratch, check out Level Up!: The Guide to Great Video Game Design, Scott Rogers, Wiley and The Art of Game Design, Jesse Schell, CRC Press. For some online resources, Scott has a neat GDC talk called Everything I Learned About Level Design I Learned from Disneyland, which can be found at http://mrbossdesign.blogspot.com/2009/03/everything-i-learned-about-game-design.html, and World of Level Design (http://worldofleveldesign.com/) is a good source for learning about level design, though it does not talk about Unity specifically. Exterior environment – terrain When creating exterior environments, we cannot use straight floors for the most part, unless you're creating a highly urbanized area. Our game takes place in a haunted house in the middle of nowhere, so we're going to create a natural landscape. In Unity, the best tool to use to create a natural landscape is the Terrain tool. Unity's terrain system lets us add landscapes, complete with bushes, trees, and fading materials to our game. To show how easy it is to use the terrain tool, let's get started. The first thing that we're going to want to do is actually create the terrain we'll be placing for the world. Let's first create a terrain by navigating to GameObject | Create Other | Terrain: If you are using Unity 4.6 or later, navigate to GameObject | Create  General | Terrain to create the Terrain. At this point, you should see the terrain. Right now, it's just a flat plane, but we'll be adding a lot to it to make it shine. If you look to the right with the Terrain object selected, you'll see the Terrain Editing tools, which can do the following (from left to right): Raise/Lower Height: This option will allow us to raise or lower the height of our terrain up to a certain radius to create hills, rivers, and more. Paint Height: If you already know the exact height that a part of your terrain needs to be, this option will allow you to paint a spot on that location. Smooth Height: This option averages out the area that it is in, and then attempts to smooth out areas and reduce the appearance of abrupt changes. Paint Texture: This option allows us to add textures to the surface of our terrain. One of the nice features of this is the ability to lay multiple textures on top of each other. Place Trees: This option allows us to paint objects in our environment, which will appear on the surface. Unity attempts to optimize these objects by billboarding distant trees so that we can have dense forests without a horrible frame rate. Paint Details: In addition to trees, we can also have small things such as rocks or grass covering the surface of our environment. We can use 2D images to represent individual clumps using bits of randomization to make it appear more natural. Terrain Settings: These are settings that will affect the overall properties of a particular terrain; options such as the size of the terrain and wind can be found here. By default, the entire terrain is set to be at the bottom, but we want to have some ground above and below us; so first, with the terrain object selected, click on the second button to the left of the terrain component (the Paint Height mode). From here, set the Height value under Settings to 100 and then click on the Flatten button. At this point, you should notice the plane moving up, so now everything is above by default. Next, we are going to add some interesting shapes to our world with some hills by painting on the surface. With the Terrain object selected, click on the first button to the left of our Terrain component (the Raise/Lower Terrain mode). Once this is completed, you should see a number of different brushes and shapes that you can select from. Our use of terrain is to create hills in the background of our scene so that it does not seem like the world is completely flat. Under the Settings area, change the Brush Size and Opacity values of your brush to 100 and left-click around the edges of the world to create some hills. You can increase the height of the current hills if you click on top of the previous hill. This is shown in the following screenshot: When creating hills, it's a good idea to look at multiple angles while you're building them, so you can make sure that none are too high or too short. Generally, you want to have taller hills as you go further back, otherwise you cannot see the smaller ones since they would be blocked. In the Scene view, to move your camera around, you can use the toolbar in the top right corner or hold down the right mouse button and drag it in the direction you want the camera to move around in, pressing the W, A, S, and D keys to pan. In addition, you can hold down the middle mouse button and drag it to move the camera around. The mouse wheel can be scrolled to zoom in and out from where the camera is. Even though you should plan out the level ahead of time on something like a piece of graph paper to plan out encounters, you will want to avoid making the level entirely from the preceding section, as the player will not actually see the game with a bird's eye view in the game at all (most likely). Referencing the map from the same perspective of your character will help ensure that the map looks great. To see many different angles at one time, you can use a layout with multiple views of the scene, such as the 4 Split. Once we have our land done, we now want to create some holes in the ground, which we will fill in with water later. This will provide a natural barrier to our world that players will know they cannot pass, so we will create a moat by first changing the Brush Size value to 50 and then holding down the Shift key, and left-clicking around the middle of our texture. In this case, it's okay to use the Top view; remember this will eventually be water to fill in lakes, rivers, and so on, as shown in the following screenshot: At this point, we have done what is referred to in the industry as "greyboxing", making the level in the engine in the simplest way possible but without artwork (also known as "whiteboxing" or "orangeboxing" depending on the company you're working for). At this point in a traditional studio, you'd spend time playtesting the level and iterating on it before an artist or you takes the time to make it look great. However, for our purposes, we want to create a finished project as soon as possible. When doing your own games, be sure to play your level and have others play your level before you polish it. For more information on greyboxing, check out http://www.worldofleveldesign.com/categories/level_design_tutorials/art_of_blocking_in_your_map.php. For an example with images of a greybox to the final level, PC Gamer has a nice article available at http://www.pcgamer.com/2014/03/18/building-crown-part-two-layout-design-textures-and-the-hammer-editor/. This is interesting enough, but being in an all-white world would be quite boring. Thankfully, it's very easy to add textures to everything. However, first we need to have some textures to paint onto the world and for this instance, we will make use of some of the free assets that Unity provides us with. So, with that in mind, navigate to Window | Asset Store. The Asset Store option is home to a number of free and commercial assets that can be used with Unity to help you create your own projects created by both Unity and the community. While we will not be using any unofficial assets, the Asset Store option may help you in the future to save your time in programming or art asset creation. An the top right corner, you'll see a search bar; type terrain assets and press Enter. Once there, the first asset you'll see is Terrain Assets, which is released by Unity Technologies for free. Left-click on it and then once at the menu, click on the Download button: Once it finishes downloading, you should see the Importing Package dialog box pop up. If it doesn't pop up, click on the Import button where the Download button used to be: Generally, you'll only want to select the assets that you want to use and uncheck the others. However, since you're exploring the tools, we'll just click on the Import button to place them all. Close the Asset Store screen; if it's still open, go back into our game view. You should notice the new Terrain Assets folder placed in our Assets folder. Double-click on it and then enter the Textures folder: These will be the textures we will be placing in our environment: Select the Terrain object and then click on the fourth button from the left to select the Paint Texture button. Here, you'll notice that it looks quite similar to the previous sections we've seen. However, there is a Textures section as well, but as of now, there is the information No terrain textures defined in this section. So let's fix that. Click on the Edit Textures button and then select Add Texture. You'll see an Add Terrain Texture dialog pop up. Under the Texture variable, place the Grass (Hill) texture and then click on the Add button: At this point, you should see the entire world change to green if you're far away. If you zoom in, you'll see that the entire terrain uses the Grass (Hill) texture now: Now, we don't want the entire world to have grass. Next, we will add cliffs around the edges where the water is. To do this, add an additional texture by navigating to Edit Textures... | Add Texture. Select Cliff (Layered Rock) as the texture and then select Add. Now if you select the terrain, you should see two textures. With the Cliff (Layered Rock) texture selected, paint the edges of the water by clicking and holding the mouse, and modifying the Brush Size value as needed: We now want to create a path for our player to follow, so we're going to create yet another texture this time using the GoodDirt material. Since this is a path the player may take, I'm going to change the Brush Size value to 8 and the Opacity value to 30, and use the second brush from the left, which is slightly less faded. Once finished, I'm going to paint in some trails that the player can follow. One thing that you will want to try to do is make sure that the player shouldn't go too far before having to backtrack and reward the player for exploration. The following screenshot shows the path: However, you'll notice that there are two problems with it currently. Firstly, it is too big to fit in with the world, and you can tell that it repeats. To reduce the appearance of texture duplication, we can introduce new materials with a very soft opacity, which we place in patches in areas where there is just plain ground. For example, let's create a new texture with the Grass (Meadow) texture. Change the Brush Size value to 16 and the Opacity value to something really low, such as 6, and then start painting the areas that look too static. Feel free to select the first brush again to have a smoother touch up. Now, if we zoom into the world as if we were a character there, I can tell that the first grass texture is way too big for the environment but we can actually change that very easily. Double-click on the texture to change the Size value to (8,8). This will make the texture smaller before it duplicates. It's a good idea to have different textures with different sizes so that the seams of each texture aren't visible to others. The following screenshot shows the size options: Do the same changes as in the preceding step for our Dirt texture as well, changing the Size option to (8,8): With this, we already have a level that looks pretty nice! However, that being said, it's just some hills. To really have a quality-looking title, we are going to need to do some additional work to beautify the environment. Beautifying the environment – adding water, trees, and grass We now have a base for our environment with the terrain, but we're still missing a lot of polish that can make the area stand out and look like a quality environment. Let's add some of those details now: First, let's add water. This time we will use another asset from Unity, but we will not have to go to the Asset Store. Navigate to Assets | Import Package | Water (Basic) and import all of the files included in the package. We will be creating a level for the night time, so navigate to Standard Assets | Water Basic and drag-and-drop the Nighttime Simple Water prefab onto the scene. Once there, set the Position values to (1000,50, 1000) and the Scale values to (1000,1,1000): At this point, you want to repaint your cliff materials to reflect being next to the water better. Next, let's add some trees to make this forest level come to life. Navigate to Terrain Assets | Trees Ambient-Occlusion and drag-and-drop a tree into your world (I'm using ScotsPineTree). By default, these trees do not contain collision information, so our player could just walk through it. This is actually great for areas that the player will not reach as we can add more trees without having to do meaningless calculations, but we need to stop the player from walking into these. To do that, we're going to need to add a collider. To do so, navigate to Component | Physics | Capsule Collider and then change the Radius value to 2. You have to use a Capsule Collider in order to have the collision carried over to the terrain. After this, move our newly created tree into the Assets folder under the Project tab and change its name to CollidingTree. Then, delete the object from the Hierarchy view. With this finished, go back to our Terrain object and then click on the Place Trees mode button. Just like working with painting textures, there are no trees here by default, so navigate to Edit Trees… | Add Tree, add our CollidingTree object created earlier in this step, and then select Add. Next, under the Settings section, change the Tree Density value to 15 and then with our new tree selected, paint the areas on the main island that do not have paths on them. Once you've finished with placing those trees, up the Tree Density value to 50 and then paint the areas that are far away from paths to make it less likely that players go that way. You should also enable Create Tree Collider in the terrain's Terrain Collider component: In our last step to create an environment, let's add some details. The mode next to the Plant Trees mode is Paint Details. Next, click on the Edit Details… button and select Add Grass Texture. Select the Grass texture for the Detail Texture option and then click on Add. In the terrain's Settings mode (the one on the far right), change the Detail Distance value to 250, and then paint the grass where there isn't any dirt along the route in the Paint Details mode: You may not see the results unless you zoom your camera in, which you can do by using the mouse wheel. Don't go too far in though, or the results may not show as well. This aspect of level creation isn't very difficult, just time consuming. However, it's taking time to enter these details that really sets a game apart from the other games. Generally, you'll want to playtest and make sure your level is fun before performing these actions; but I feel it's important to have an idea of how to do it for your future projects. Lastly, our current island is very flat, and while that's okay for cities, nature is random. Go back into the Raise/Lower Height tool and gently raise and lower some areas of the level to give the illusion of depth. Do note that your trees and grass will raise and fall with the changes that you make, as shown in the following screenshot: With this done, let's now add some details to the areas that the player will not be visiting, such as the outer hills. Go into the Place Trees mode and create another tree, but this time select the one without collision and then place it around the edges of the mountains, as shown in the following screenshot: At this point, we have a nice exterior shape created with the terrain tools! If you want to add even more detail to your levels, you can add additional trees and/or materials to the level area as long as it makes sense for them to be there. For more information on the terrain engine that Unity has, please visit http://docs.unity3d.com/Manual/script-Terrain.html. Creating our player Now that we have the terrain and its details, it's hard to get a good picture of what the game looks like without being able to see what it looks like down on the surface, so next we will do just that. However, instead of creating our player from scratch as we've done previously, we will make use of the code that Unity has provided us. We will perform the following steps: Start off by navigating to Assets | Import Package | Character Controller. When the Importing Package dialog comes up, we only need to import the files shown in the following screenshot: Now drag-and-drop the First Person Controller prefab under the Prefabs folder in our Project tab into your world, where you want the player to spawn, setting the Y Position value to above 100. If you see yourself fall through the world instead of hitting the ground when you spawn, then increase the Y Position value until you get there. If you open up the First Person Controller object in the Hierarchy tab, you'll notice that it has a Main Camera object already, so delete the Main Camera object that already exists in the world. Right now, if we played the game, you'd see that everything is dark because we don't have any light. For the purposes of demonstration, let's add a directional light by navigating to GameObject | Create Other | Directional Light. If you are using Unity 4.6 or later, navigate to GameObject | Create  General | Terrain to create the Terrain. Save your scene and hit the Play button to drop into your level: At this point, we have a playable level that we can explore and move around in! Building the atmosphere Now, the base of our world has been created; let's add some effects to make the game even more visually appealing and so it will start to fit in with the survival horror feel that we're going to be giving the game. The first part of creating the atmosphere is to add something for the sky aside from the light blue color that we currently use by default. To fix this, we will be using a skybox. A skybox is a method to create backgrounds to make the area seem bigger than it really is, by putting an image in the areas that are currently being filled with the light blue color, not moving in the same way that the sky doesn't move to us because it's so far away. The reason why we call a skybox a skybox is because it is made up of six textures that will be the inside of the box (one for each side of a cube). Game engines such as Unreal have skydomes, which are the same thing; but they are done with a hemisphere instead of a cube. We will perform the following steps to build the atmosphere: To add in our skybox, we are going to navigate to Assets | Import Package | Skyboxes. We want our level to display the night, so we'll be using the Starry Night Skybox. Just select the StarryNight Skybox.mat file and textures inside the Standard Assets/Skyboxes/Textures/StarryNight/ location, and then click on Import: With this file imported, we need to navigate to Edit | Render Settings next. Once there, we need to set the Skybox Material option to the Starry Night skybox: If you go into the game, you'll notice the level starting to look nicer already with the addition of the skybox, except for the fact that the sky says night while the world says it's daytime. Let's fix that now. Switch to the Game tab so that you can see the changes we'll be making next. While still at the RenderSettings menu, let's turn on the Fog property by clicking on the checkbox with its name and changing the Fog Color value to a black color. You should notice that the surroundings are already turning very dark. Play around with the Fog Density value until you're comfortable with how much the player can see ahead of them; I used 0.005. Fog obscures far away objects, which adds to the atmosphere and saves the rendering power. The denser the fog, the more the game will feel like a horror game. The first game of the Silent Hill franchise used fog to make the game run at an acceptable frame rate due to a large 3D environment it had on early PlayStation hardware. Due to how well it spooked players, it continued to be used in later games even though they could render larger areas with the newer technology. Let's add some lighting tweaks to make the environment that the player is walking in seem more like night. Go into the DirectionalLight properties section and change the Intensity value to 0.05. You'll see the value get darker, as shown in the following screenshot: If for some reason, you'd like to make the world pitch black, you'll need to modify the Ambient Light property to black inside the RenderSettings section. By default, it is dark grey, which will show up even if there are no lights placed in the world. In the preceding example, I increased the Intensity value to make it easier to see the world to make it easier for readers to follow, but in your project, you probably don't want the player to see so far out with such clarity. With this, we now have a believable exterior area at night! Now that we have this basic knowledge, let's add a flashlight so the players can see where they are going. Creating a flashlight Now that our level looks like a dark night, we still want to give our players the ability to see what's in front of them with a flashlight. We will customize the First Person Controller object to fit our needs: Create a spotlight by navigating to GameObject | Create Other | Spotlight. Once created, we are going to make the spotlight a child of the First Person Controller object's Main Camera object by dragging-and-dropping it on top of it. Once a child, change the Transform Position value to (0, -.95, 0). Since positions are relative to your parent's position, this places the light slightly lower than the camera's center, just like a hand holding a flashlight. Now change the Rotation value to (0,0,0) or give it a slight diagonal effect across the scene if you don't want it to look like it's coming straight out: Now, we want the flashlight to reach out into the distance. So we will change the Range value to 1000, and to make the light wider, we will change the Spot Angle value to 45. The effects are shown in the following screenshot: If you have Unity Pro, you can also give shadows to the world based on your lights by setting the Shadow Type property. We now have a flashlight, so the player can focus on a particular area and not worry. Walking / flashlight bobbing animation Now the flashlight looks fine in a screenshot, but if you walk throughout the world, it will feel very static and unnatural. If a person is actually walking through the forest, there will be a slight bob as you walk, and if someone is actually holding a flash light, it won't be stable the entire time because your hand would move. We can solve both of these problems by writing yet another script. We perform the following steps: Create a new folder called Scripts. Inside this folder, create a new C# script called BobbingAnimation. Open the newly created script and use the following code inside it: using UnityEngine; using System.Collections;   /// <summary> /// Allows the attached object to bob up and down through /// movement or /// by default. /// </summary> public class BobbingAnimation : MonoBehaviour {   /// <summary>   /// The elapsed time.   /// </summary>   private float elapsedTime;     /// <summary>   /// The starting y offset from the parent.   /// </summary>   private float startingY;     /// <summary>   /// The controller.   /// </summary>   private CharacterController controller;     /// <summary>   /// How far up and down the object will travel   /// </summary>   public float magnitude = .2f;     /// <summary>   /// How often the object will move up and down   /// </summary>   public float frequency = 10;     /// <summary>   /// Do you always want the object to bob up and down or   /// with movement?   /// </summary>   public bool alwaysBob = false;     /// <summary>   /// Start this instance.   /// </summary>   void Start ()   {     startingY = transform.localPosition.y;     controller = GetComponent<CharacterController> ();   }     /// <summary>   /// Update this instance.   /// </summary>   void Update ()   {          // Only increment elapsedTime if you want the player to     // bob, keeping it the same will keep it still     if(alwaysBob)     {       elapsedTime += Time.deltaTime;     }     else     {       if((Input.GetAxis("Horizontal") != 0.0f) || (Input.GetAxis("Vertical")  != 0.0f) )         elapsedTime += Time.deltaTime;     }         float yOffset = Mathf.Sin(elapsedTime * frequency) * magnitude;       //If we can't find the player controller or we're     // jumping, we shouldn't be bobbing     if(controller && !controller.isGrounded)     {       return;     }       //Set our position     Vector3 pos = transform.position;         pos.y = transform.parent.transform.position.y +             startingY + yOffset;         transform.position = pos;       } } The preceding code will tweak the object it's attached to so that it will bob up and down whenever the player is moving along the x or y axis. I've also added a variable called alwaysBob, which, when true, will make the object always bob. Math is a game developer's best friend, and here we are using sin (pronounced sine). Taking the sin of an angle number gives you the ratio of the length of the opposite side of the angle to the length of the hypotenuse of a right-angled triangle. If that didn't make any sense to you, don't worry. The neat feature of sin is that as the number it takes gets larger, it will continuously give us a value between 0 and 1 that will go up and down forever, giving us a smooth repetitive oscillation. For more information on sine waves, visit http://en.wikipedia.org/wiki/Sine_wave. While we're using the sin just for the player's movement and the flashlight, this could be used in a lot of effects, such as having save points/portals bob up and down, or any kind of object you would want to have slight movement or some special FX. Next, attach the BobbingAnimation component to the Main Camera object, leaving all the values with the defaults. After this, attach the BobbingAnimation component to the spotlight as well. With the spotlight selected, turn the Always Bob option on and change the Magnitude value to .05 and the Frequency value to 3. The effects are shown in the following screenshot: Summary To learn more about FPS game, the following books published by Packt Publishing (https://www.packtpub.com/) are recommended: Building an FPS Game with Unity (https://www.packtpub.com/game-development/building-fps-game-unity) Resources for Article:   Further resources on this subject: Mobile Game Design Best Practices [article] Putting the Fun in Functional Python [article] Using a collider-based system [article]
Read more
  • 0
  • 0
  • 17531

Packt
17 Feb 2016
29 min read
Save for later

Spark – Architecture and First Program

Packt
17 Feb 2016
29 min read
In this article by Sumit Gupta and Shilpi Saxena, the authors of Real-Time Big Data Analytics, we will discuss the architecture of Spark and its various components in detail. We will also briefly talk about the various extensions/libraries of Spark, which are developed over the core Spark framework. (For more resources related to this topic, see here.) Spark is a general-purpose computing engine that initially focused to provide solutions to the iterative and interactive computations and workloads. For example, machine learning algorithms, which reuse intermediate or working datasets across multiple parallel operations. The real challenge with iterative computations was the dependency of the intermediate data/steps on the overall job. This intermediate data needs to be cached in the memory itself for faster computations because flushing and reading from a disk was an overhead, which, in turn, makes the overall process unacceptably slow. The creators of Apache Spark not only provided scalability, fault tolerance, performance, distributed data processing but also provided in-memory processing of distributed data over the cluster of nodes. To achieve this, a new layer abstraction of distributed datasets that is partitioned over the set of machines (cluster) was introduced, which can be cached in the memory to reduce the latency. This new layer of abstraction was known as resilient distributed datasets (RDD). RDD, by definition, is an immutable (read-only) collection of objects partitioned across a set of machines that can be rebuilt if a partition is lost. It is important to note that Spark is capable of performing in-memory operations, but at the same time, it can also work on the data stored on the disk. High-level architecture Spark provides a well-defined and layered architecture where all its layers and components are loosely coupled and integration with external components/libraries/extensions is performed using well-defined contracts. Here is the high-level architecture of Spark 1.5.1 and its various components/layers: The preceding diagram shows the high-level architecture of Spark. Let's discuss the roles and usage of each of the architecture components: Physical machines: This layer represents the physical or virtual machines/nodes on which Spark jobs are executed. These nodes collectively represent the total capacity of the cluster with respect to the CPU, memory, and data storage. Data storage layer: This layer provides the APIs to store and retrieve the data from the persistent storage area to Spark jobs/applications. This layer is used by Spark workers to dump data on the persistent storage whenever the cluster memory is not sufficient to hold the data. Spark is extensible and capable of using any kind of filesystem. RDD, which hold the data, are agnostic to the underlying storage layer and can persist the data in various persistent storage areas, such as local filesystems, HDFS, or any other NoSQL database such as HBase, Cassandra, MongoDB, S3, and Elasticsearch. Resource manager: The architecture of Spark abstracts out the deployment of the Spark framework and its associated applications. Spark applications can leverage cluster managers such as YARN (http://tinyurl.com/pcymnnf) and Mesos (http://mesos.apache.org/) for the allocation and deallocation of various physical resources, such as the CPU and memory for the client jobs. The resource manager layer provides the APIs that are used to request for the allocation and deallocation of available resource across the cluster. Spark core libraries: The Spark core library represents the Spark Core engine, which is responsible for the execution of the Spark jobs. It contains APIs for in-memory distributed data processing and a generalized execution model that supports a wide variety of applications and languages. Spark extensions/libraries: This layer represents the additional frameworks/APIs/libraries developed by extending the Spark core APIs to support different use cases. For example, Spark SQL is one such extension, which is developed to perform ad hoc queries and interactive analysis over large datasets. The preceding architecture should be sufficient enough to understand the various layers of abstraction provided by Spark. All the layers are loosely coupled, and if required, can be replaced or extended as per the requirements. Spark extensions is one such layer that is widely used by architects and developers to develop custom libraries. Let's move forward and talk more about Spark extensions, which are available for developing custom applications/jobs. Spark extensions/libraries In this section, we will briefly discuss the usage of various Spark extensions/libraries that are available for different use cases. The following are the extensions/libraries available with Spark 1.5.1: Spark Streaming: Spark Streaming, as an extension, is developed over the core Spark API. It enables scalable, high-throughput, and fault-tolerant stream processing of live data streams. Spark Streaming enables the ingestion of data from various sources, such as Kafka, Flume, Kinesis, or TCP sockets. Once the data is ingested, it can be further processed using complex algorithms that are expressed with high-level functions, such as map, reduce, join, and window. Finally, the processed data can be pushed out to filesystems, databases, and live dashboards. In fact, Spark Streaming also facilitates the usage Spark's machine learning and graph processing algorithms on data streams. For more information, refer to http://spark.apache.org/docs/latest/streaming-programming-guide.html. Spark MLlib: Spark MLlib is another extension that provides the distributed implementation of various machine learning algorithms. Its goal is to make practical machine learning library scalable and easy to use. It provides implementation of various common machine learning algorithms used for classification, regression, clustering, and many more. For more information, refer to http://spark.apache.org/docs/latest/mllib-guide.html. Spark GraphX: GraphX provides the API to create a directed multigraph with properties attached to each vertex and edges. It also provides the various common operators used for the aggregation and distributed implementation of various graph algorithms, such as PageRank and triangle counting. For more information, refer to http://spark.apache.org/docs/latest/graphx-programming-guide.html. Spark SQL: Spark SQL provides the distributed processing of structured data and facilitates the execution of relational queries, which are expressed in a structured query language. (http://en.wikipedia.org/wiki/SQL). It provides the high level of abstraction known as DataFrames, which is a distributed collection of data organized into named columns. For more information, refer to http://spark.apache.org/docs/latest/sql-programming-guide.html. SparkR: R (https://en.wikipedia.org/wiki/R_(programming_language) is a popular programming language used for statistical computing and performing machine learning tasks. However, the execution of the R language is single threaded, which makes it difficult to leverage in order to process large data (TBs or PBs). R can only process the data that fits into the memory of a single machine. In order to overcome the limitations of R, Spark introduced a new extension: SparkR. SparkR provides an interface to invoke and leverage Spark distributed execution engine from R, which allows us to run large-scale data analysis from the R shell. For more information, refer to http://spark.apache.org/docs/latest/sparkr.html. All the previously listed Spark extension/libraries are part of the standard Spark distribution. Once we install and configure Spark, we can start using APIs that are exposed by the extensions. Apart from the earlier extensions, Spark also provides various other external packages that are developed and provided by the open source community. These packages are not distributed with the standard Spark distribution, but they can be searched and downloaded from http://spark-packages.org/. Spark packages provide libraries/packages for integration with various data sources, management tools, higher level domain-specific libraries, machine learning algorithms, code samples, and other Spark content. Let's move on to the next section where we will dive deep into the Spark packaging structure and execution model, and we will also talk about various other Spark components. Spark packaging structure and core APIs In this section, we will briefly talk about the packaging structure of the Spark code base. We will also discuss core packages and APIs, which will be frequently used by the architects and developers to develop custom applications with Spark. Spark is written in Scala (http://www.scala-lang.org/), but for interoperability, it also provides the equivalent APIs in Java and Python as well. For brevity, we will only talk about the Scala and Java APIs, and for Python APIs, users can refer to https://spark.apache.org/docs/1.5.1/api/python/index.html. A high-level Spark code base is divided into the following two packages: Spark extensions: All APIs for a particular extension is packaged in its own package structure. For example, all APIs for Spark Streaming are packaged in the org.apache.spark.streaming.* package, and the same packaging structure goes for other extensions: Spark MLlib—org.apache.spark.mllib.*, Spark SQL—org.apcahe.spark.sql.*, Spark GraphX—org.apache.spark.graphx.*. For more information, refer to http://tinyurl.com/q2wgar8 for Scala APIs and http://tinyurl.com/nc4qu5l for Java APIs. Spark Core: Spark Core is the heart of Spark and provides two basic components: SparkContext and SparkConfig. Both of these components are used by each and every standard or customized Spark job or Spark library and extension. The terms/concepts Context and Config are not new and more or less they have now become a standard architectural pattern. By definition, a Context is an entry point of the application that provides access to various resources/features exposed by the framework, whereas a Config contains the application configurations, which helps define the environment of the application. Let's move on to the nitty-gritty of the Scala APIs exposed by Spark Core: org.apache.spark: This is the base package for all Spark APIs that contains a functionality to create/distribute/submit Spark jobs on the cluster. org.apache.spark.SparkContext: This is the first statement in any Spark job/application. It defines the SparkContext and then further defines the custom business logic that is is provided in the job/application. The entry point for accessing any of the Spark features that we may want to use or leverage is SparkContext, for example, connecting to the Spark cluster, submitting jobs, and so on. Even the references to all Spark extensions are provided by SparkContext. There can be only one SparkContext per JVM, which needs to be stopped if we want to create a new one. The SparkContext is immutable, which means that it cannot be changed or modified once it is started. org.apache.spark.rdd.RDD.scala: This is another important component of Spark that represents the distributed collection of datasets. It exposes various operations that can be executed in parallel over the cluster. The SparkContext exposes various methods to load the data from HDFS or the local filesystem or Scala collections, and finally, create an RDD on which various operations such as map, filter, join, and persist can be invoked. RDD also defines some useful child classes within the org.apache.spark.rdd.* package such as PairRDDFunctions to work with key/value pairs, SequenceFileRDDFunctions to work with Hadoop sequence files, and DoubleRDDFunctions to work with RDDs of doubles. We will read more about RDD in the subsequent sections. org.apache.spark.annotation: This package contains the annotations, which are used within the Spark API. This is the internal Spark package, and it is recommended that you do not to use the annotations defined in this package while developing our custom Spark jobs. The three main annotations defined within this package are as follows: DeveloperAPI: All those APIs/methods, which are marked with DeveloperAPI, are for advance usage where users are free to extend and modify the default functionality. These methods may be changed or removed in the next minor or major releases of Spark. Experimental: All functions/APIs marked as Experimental are officially not adopted by Spark but are introduced temporarily in a specific release. These methods may be changed or removed in the next minor or major releases. AlphaComponent: The functions/APIs, which are still being tested by the Spark community, are marked as AlphaComponent. These are not recommended for production use and may be changed or removed in the next minor or major releases. org.apache.spark.broadcast: This is one of the most important packages, which are frequently used by developers in their custom Spark jobs. It provides the API for sharing the read-only variables across the Spark jobs. Once the variables are defined and broadcast, they cannot be changed. Broadcasting the variables and data across the cluster is a complex task, and we need to ensure that an efficient mechanism is used so that it improves the overall performance of the Spark job and does not become an overhead. Spark provides two different types of implementations of broadcasts—HttpBroadcast and TorrentBroadcast. The HttpBroadcast broadcast leverages the HTTP server to fetch/retrieve the data from Spark driver. In this mechanism, the broadcast data is fetched through an HTTP Server running at the driver itself and further stored in the executor block manager for faster accesses. The TorrentBroadcast broadcast, which is also the default implementation of the broadcast, maintains its own block manager. The first request to access the data makes the call to its own block manager, and if not found, the data is fetched in chunks from the executor or driver. It works on the principle of BitTorrent and ensures that the driver is not the bottleneck in fetching the shared variables and data. Spark also provides accumulators, which work like broadcast, but provide updatable variables shared across the Spark jobs but with some limitations. You can refer to https://spark.apache.org/docs/1.5.1/api/scala/index.html#org.apache.spark.Accumulator. org.apache.spark.io: This provides implementation of various compression libraries, which can be used at block storage level. This whole package is marked as Developer API, so developers can extend and provide their own custom implementations. By default, it provides three implementations: LZ4, LZF, and Snappy. org.apache.spark.scheduler: This provides various scheduler libraries, which help in job scheduling, tracking, and monitoring. It defines the directed acyclic graph (DAG) scheduler (http://en.wikipedia.org/wiki/Directed_acyclic_graph). The Spark DAG scheduler defines the stage-oriented scheduling where it keeps track of the completion of each RDD and the output of each stage and then computes DAG, which is further submitted to the underlying org.apache.spark.scheduler.TaskScheduler API that executes them on the cluster. org.apache.spark.storage: This provides APIs for structuring, managing, and finally, persisting the data stored in RDD within blocks. It also keeps tracks of data and ensures that it is either stored in memory, or if the memory is full, it is flushed to the underlying persistent storage area. org.apache.spark.util: These are the utility classes used to perform common functions across the Spark APIs. For example, it defines MutablePair, which can be used as an alternative to Scala's Tuple2 with the difference that MutablePair is updatable while Scala's Tuple2 is not. It helps in optimizing memory and minimizing object allocations. Spark execution model – master worker view Let's move on to the next section where we will dive deep into the Spark execution model, and we will also talk about various other Spark components. Spark essentially enables the distributed in-memory execution of a given piece of code. We discussed the Spark architecture and its various layers in the previous section. Let's also discuss its major components, which are used to configure the Spark cluster, and at the same time, they will be used to submit and execute our Spark jobs. The following are the high-level components involved in setting up the Spark cluster or submitting a Spark job: Spark driver: This is the client program, which defines SparkContext. The entry point for any job that defines the environment/configuration and the dependencies of the submitted job is SparkContext. It connects to the cluster manager and requests resources for further execution of the jobs. Cluster manager/resource manager/Spark master: The cluster manager manages and allocates the required system resources to the Spark jobs. Furthermore, it coordinates and keeps track of the live/dead nodes in a cluster. It enables the execution of jobs submitted by the driver on the worker nodes (also called Spark workers) and finally tracks and shows the status of various jobs running by the worker nodes. Spark worker/executors: A worker actually executes the business logic submitted by the Spark driver. Spark workers are abstracted and are allocated dynamically by the cluster manager to the Spark driver for the execution of submitted jobs. The following diagram shows the high-level components and the master worker view of Spark: The preceding diagram depicts the various components involved in setting up the Spark cluster, and the same components are also responsible for the execution of the Spark job. Although all the components are important, but let's briefly discuss the cluster/resource manager, as it defines the deployment model and allocation of resources to our submitted jobs. Spark enables and provides flexibility to choose our resource manager. As of Spark 1.5.1, the following are the resource managers or deployment models that are supported by Spark: Apache Mesos: Apache Mesos (http://mesos.apache.org/) is a cluster manager that provides efficient resource isolation and sharing across distributed applications or frameworks. It can run Hadoop, MPI, Hypertable, Spark, and other frameworks on a dynamically shared pool of nodes. Apache Mesos and Spark are closely related to each other (but they are not the same). The story started way back in 2009 when Mesos was ready and there were talks going on about the ideas/frameworks that can be developed on top of Mesos, and that's exactly how Spark was born. Refer to http://spark.apache.org/docs/latest/running-on-mesos.html for more information on running Spark jobs on Apache Mesos. Hadoop YARN: Hadoop 2.0 (http://tinyurl.com/qypb4xm), also known as YARN, was a complete change in the architecture. It was introduced as a generic cluster computing framework that was entrusted with the responsibility of allocating and managing the resources required to execute the varied jobs or applications. It introduced new daemon services, such as the resource manager (RM), node manager (NM), and application master (AM), which are responsible for managing cluster resources, individual nodes, and respective applications. YARN also introduced specific interfaces/guidelines for application developers where they can implement/follow and submit or execute their custom applications on the YARN cluster. The Spark framework implements the interfaces exposed by YARN and provides the flexibility of executing the Spark applications on YARN. Spark applications can be executed in the following two different modes in YARN: YARN client mode: In this mode, the Spark driver executes the client machine (the machine used for submitting the job), and the YARN application master is just used for requesting the resources from YARN. All our logs and sysouts (println) are printed on the same console, which is used to submit the job. YARN cluster mode: In this mode, the Spark driver runs inside the YARN application master process, which is further managed by YARN on the cluster, and the client can go away just after submitting the application. Now as our Spark driver is executed on the YARN cluster, our application logs/sysouts (println) are also written in the log files maintained by YARN and not on the machine that is used to submit our Spark job. For more information on executing Spark applications on YARN, refer to http://spark.apache.org/docs/latest/running-on-yarn.html. Standalone mode: The Core Spark distribution contains the required APIs to create an independent, distributed, and fault tolerant cluster without any external or third-party libraries or dependencies. Local mode: Local mode should not be confused with standalone mode. In local mode, Spark jobs can be executed on a local machine without any special cluster setup by just passing local[N] as the master URL, where N is the number of parallel threads. Writing and executing our first Spark program In this section, we will install/configure and write our first Spark program in Java and Scala. Hardware requirements Spark supports a variety of hardware and software platforms. It can be deployed on commodity hardware and also supports deployments on high-end servers. Spark clusters can be provisioned either on cloud or on-premises. Though there is no single configuration or standards, which can guide us through the requirements of Spark, but to create and execute Spark examples provided in this article, it would be good to have a laptop/desktop/server with the following configuration: RAM: 8 GB. CPU: Dual core or Quad core. DISK: SATA drives with a capacity of 300 GB to 500 GB with 15 k RPM. Operating system: Spark supports a variety of platforms that include various flavors of Linux (Ubuntu, HP-UX, RHEL, and many more) and Windows. For our examples, we will recommend that you use Ubuntu for the deployment and execution of examples. Spark core is coded in Scala, but it offers several development APIs in different languages, such as Scala, Java, and Python, so that developers can choose their preferred weapon for coding. The dependent software may vary based on the programming languages but still there are common sets of software for configuring the Spark cluster and then language-specific software for developing Spark jobs. In the next section, we will discuss the software installation steps required to write/execute Spark jobs in Scala and Java on Ubuntu as the operating system. Installation of the basic softwares In this section, we will discuss the various steps required to install the basic software, which will help us in the development and execution of our Spark jobs. Spark Perform the following steps to install Spark: Download the Spark compressed tarball from http://d3kbcqa49mib13.cloudfront.net/spark-1.5.1-bin-hadoop2.4.tgz. Create a new directory spark-1.5.1 on your local filesystem and extract the Spark tarball into this directory. Execute the following command on your Linux shell in order to set SPARK_HOME as an environment variable: export SPARK_HOME=<Path of Spark install Dir> Now, browse your SPARK_HOME directory and it should look similar to the following screenshot: Java Perform the following steps to install Java: Download and install Oracle Java 7 from http://www.oracle.com/technetwork/java/javase/install-linux-self-extracting-138783.html. Execute the following command on your Linux shell to set JAVA_HOME as an environment variable: export JAVA_HOME=<Path of Java install Dir> Scala Perform the following steps to install Scala: Download the Scala 2.10.5 compressed tarball from http://downloads.typesafe.com/scala/2.10.5/scala-2.10.5.tgz?_ga=1.7758962.1104547853.1428884173. Create a new directory, Scala 2.10.5, on your local filesystem and extract the Scala tarball into this directory. Execute the following commands on your Linux shell to set SCALA_HOME as an environment variable, and add the Scala compiler to the $PATH system: export SCALA_HOME=<Path of Scala install Dir> Next, execute the command in the following screenshot to ensure that the Scala runtime and Scala compiler is available and the version is 2.10.x: Spark 1.5.1 supports the 2.10.5 version of Scala, so it is advisable to use the same version to avoid any runtime exceptions due to mismatch of libraries. Eclipse Perform the following steps to install Eclipse: Based on your hardware configuration, download Eclipse Luna (4.4) from http://www.eclipse.org/downloads/packages/eclipse-ide-java-eedevelopers/lunasr2: Next, install the IDE for Scala in Eclipse itself so that we can write and compile our Scala code inside Eclipse (http://scala-ide.org/download/current.html). We are now done with the installation of all the required software. Let's move on and configure our Spark cluster. Configuring the Spark cluster The first step to configure the Spark cluster is to identify the appropriate resource manager. We discussed the various resource managers in the Spark execution model – master worker view section (Yarn, Mesos, and standalone). Standalone is the most preferred resource manager for development because it is simple/quick and does not require installation of any other component or software. We will also configure the standalone resource manager for all our Spark examples, and for more information on Yarn and Mesos, refer to the Spark execution model – master worker view section. Perform the following steps to bring up an independent cluster using Spark binaries: The first step to set up the Spark cluster is to bring up the master node, which will track and allocate the systems' resource. Open your Linux shell and execute the following command: $SPARK_HOME/sbin/start-master.sh The preceding command will bring up your master node, and it will also enable a UI, the Spark UI to monitor the nodes/jobs in the Spark cluster, http://<host>:8080/. The <host> is the domain name of the machine on which the master is running. Next, let's bring up our worker node, which will execute our Spark jobs. Execute the following command on the same Linux shell: $SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker <Spark-Master> & In the preceding command, replace the <Spark-Master> with the Spark URL, which is shown at the top of the Spark UI, just beside Spark master at. The preceding command will start the Spark worker process in the background and the same will also be reported in the Spark UI. The Spark UI shown in the preceding screenshot shows the three different sections, providing the following information: Workers: This reports the health of a worker node, which is alive or dead and also provides drill-down to query the status and details logs of the various jobs executed by that specific worker node Running applications: This shows the applications that are currently being executed in the cluster and also provides drill-down and enables viewing of application logs Completed application: This is the same functionality as running applications; the only difference being that it shows the jobs, which are finished We are done!!! Our Spark cluster is up and running and ready to execute our Spark jobs with one worker node. Let's move on and write our first Spark application in Scala and Java and further execute it on our newly created cluster. Coding Spark job in Scala In this section, we will code our first Spark job in Scala, and we will also execute the same job on our newly created Spark cluster and will further analyze the results. This is our first Spark job, so we will keep it simple. We will use the Chicago crimes dataset for August 2015and will count the number of crimes reported in August 2015. Perform the following steps to code the Spark job in Scala for aggregating the number of crimes in August 2015: Open Eclipse and create a Scala project called Spark-Examples. Expand your newly created project and modify the version of the Scala library container to 2.10. This is done to ensure that the version of Scala libraries used by Spark and the custom jobs developed/deployed are the same. Next, open the properties of your project Spark-Examples and add the dependencies for the all libraries packaged with the Spark distribution, which can be found at $SPARK_HOME/lib. Next, create a chapter.six Scala package, and in this package, define a new Scala object by the name of ScalaFirstSparkJob. Define a main method in the Scala object and also import SparkConfand SparkContext. Now, add the following code to the main method of ScalaFirstSparkJob: object ScalaFirstSparkJob { def main(args: Array[String]) { println("Creating Spark Configuration") //Create an Object of Spark Configuration val conf = new SparkConf() //Set the logical and user defined Name of this Application conf.setAppName("My First Spark Scala Application") println("Creating Spark Context") //Create a Spark Context and provide previously created //Object of SparkConf as an reference. val ctx = new SparkContext(conf) //Define the location of the file containing the Crime Data val file = "file:///home/ec2-user/softwares/crime-data/ Crimes_-Aug-2015.csv"; println("Loading the Dataset and will further process it") //Loading the Text file from the local file system or HDFS //and converting it into RDD. //SparkContext.textFile(..) - It uses the Hadoop's //TextInputFormat and file is broken by New line Character. //Refer to http://hadoop.apache.org/docs/r2.6.0/api/org/ apache/hadoop/mapred/TextInputFormat.html //The Second Argument is the Partitions which specify the parallelism. //It should be equal or more then number of Cores in the cluster. val logData = ctx.textFile(file, 2) //Invoking Filter operation on the RDD, and counting the number of lines in the Data loaded in RDD. //Simply returning true as "TextInputFormat" have already divided the data by "\n" //So each RDD will have only 1 line. val numLines = logData.filter(line => true).count() //Finally Printing the Number of lines. println("Number of Crimes reported in Aug-2015 = " + numLines) } } We are now done with the coding! Our first Spark job in Scala is ready for execution. Now, from Eclipse itself, export your project as a .jar fie, name it spark-examples.jar, and save this .jar file in the root of $SPARK_HOME. Next, open your Linux console, go to $SPARK_HOME, and execute the following command: $SPARK_HOME/bin/spark-submit --class chapter.six.ScalaFirstSparkJob --master spark://ip-10-166-191-242:7077 spark-examples.jar In the preceding command, ensure that the value given to --masterparameter is the same as it is shown on your Spark UI. The Spark-submit is a utility script, which is used to submit the Spark jobs to the cluster. As soon as you click on Enter and execute the preceding command, you will see lot of activity (log messages) on the console, and finally, you will see the output of your job at the end: Isn't that simple! As we move forward and discuss Spark more, you will appreciate the ease of coding and simplicity provided by Spark for creating, deploying, and running jobs in a distributed framework. Your completed job will also be available for viewing at the Spark UI: The preceding image shows the status of our first Scala job on the UI. Now let's move forward and develop the same Job using Spark Java APIs. Coding Spark job in Java Perform the following steps to code the Spark job in Java for aggregating the number of crimes in August 2015: Open your Spark-Examples Eclipse project (created in the previous section). Add a new chapter.six.JavaFirstSparkJobJava file, and add the following code snippet: import org.apache.spark.SparkConf; import org.apache.spark.api.java.JavaRDD; import org.apache.spark.api.java.JavaSparkContext; import org.apache.spark.api.java.function.Function; public class JavaFirstSparkJob { public static void main(String[] args) { System.out.println("Creating Spark Configuration"); // Create an Object of Spark Configuration SparkConf javaConf = new SparkConf(); // Set the logical and user defined Name of this Application javaConf.setAppName("My First Spark Java Application"); System.out.println("Creating Spark Context"); // Create a Spark Context and provide previously created //Objectx of SparkConf as an reference. JavaSparkContext javaCtx = new JavaSparkContext(javaConf); System.out.println("Loading the Crime Dataset and will further process it"); String file = "file:///home/ec2-user/softwares/crime-data/Crimes_-Aug-2015.csv"; JavaRDD<String> logData = javaCtx.textFile(file); //Invoking Filter operation on the RDD. //And counting the number of lines in the Data loaded //in RDD. //Simply returning true as "TextInputFormat" have already divided the data by "\n" //So each RDD will have only 1 line. long numLines = logData.filter(new Function<String, Boolean>() { public Boolean call(String s) { return true; } }).count(); //Finally Printing the Number of lines System.out.println("Number of Crimes reported in Aug-2015 = "+numLines); javaCtx.close(); } } Next, compile the preceding JavaFirstSparkJob from Eclipse itself and perform steps 7, 8, and 9 of the previous section in which we executed the Spark Scala job. We are done! Analyze the output on the console; it should be the same as the output of the Scala job, which we executed in the previous section. Troubleshooting – tips and tricks In this section, we will talk about troubleshooting tips and tricks, which are helpful in solving the most common errors encountered while working with Spark. Port numbers used by Spark Spark binds various network ports for communication within the cluster/nodes and also exposes the monitoring information of jobs to developers and administrators. There may be instances where the default ports used by Spark may not be available or may be blocked by the network firewall which in turn will result in modifying the default Spark ports for master/worker or driver. Here is a list of all the ports utilized by Spark and their associated parameters, which need to be configured for any changes (http://spark.apache.org/docs/latest/security.html#configuring-ports-for-network-security). Classpath issues – class not found exception Classpath is the most common issue and it occurs frequently in distributed applications. Spark and its associated jobs run in a distributed mode on a cluster. So, if your Spark job is dependent upon external libraries, then we need to ensure that we package them into a single JAR fie and place it in a common location or the default classpath of all worker nodes or define the path of the JAR file within SparkConf itself: val sparkConf = new SparkConf().setAppName("myapp").setJars(<path of Jar file>)) Other common exceptions In this section, we will talk about few of the common errors/issues/exceptions encountered by architects/developers when they set up Spark or execute Spark jobs: Too many open files: This increases the ulimit on your Linux OS by executingsudo ulimit –n 20000. Version of Scala: Spark 1.5.1 supports Scala 2.10, so if you have multiple versions of Scala deployed on your box, then ensure that all versions are the same, that is, Scala 2.10. Out of memory on workers in standalone mode: This configures SPARK_WORKER_MEMORY in $SPARK_HOME/conf/spark-env.sh. By default, it provides a total memory of 1 G to workers, but at the same time, you should analyze and ensure that you are not loading or caching too much data on worker nodes. Out of memory in applications executed on worker nodes: This configures spark.executor.memory in your SparkConf, as follows: val sparkConf = new SparkConf().setAppName("myapp") .set("spark.executor.memory", "1g") The preceding tips will help you solve basic issues in setting up Spark clusters, but as you move ahead, there could be more complex issues, which are beyond the basic setup, and for all those issues, post your queries at http://stackoverflow.com/questions/tagged/apache-spark or mail at user@spark.apache.org. Summary In this article, we discussed the architecture of Spark and its various components. We also configured our Spark cluster and executed our first Spark job in Scala and Java. Resources for Article:   Further resources on this subject: Data mining [article] Python Data Science Up and Running [article] The Design Patterns Out There and Setting Up Your Environment [article]
Read more
  • 0
  • 0
  • 3188

article-image-introducing-object-oriented-programmng-typescript
Packt
17 Feb 2016
13 min read
Save for later

Introducing Object Oriented Programmng with TypeScript

Packt
17 Feb 2016
13 min read
In this article, we will see how to group our functions in reusable components, such as classes or modules. This article will cover the following topics: SOLID principles Classes Association, aggregation, and composition Inheritance (For more resources related to this topic, see here.) SOLID principles In the early days of software development, developers used to write code with procedural programming languages. In procedural programming languages, the programs follow a top-to-bottom approach and the logic is wrapped with functions. New styles of computer programming, such as modular programming or structured programming, emerged when developers realized that procedural computer programs could not provide them with the desired level of abstraction, maintainability, and reusability. The development community created a series of recommended practices and design patterns to improve the level of abstraction and reusability of procedural programming languages, but some of these guidelines required a certain level of expertise. In order to facilitate adherence to these guidelines, a new style of computer programming known as object-oriented programming (OOP) was created. Developers quickly noticed some common OOP mistakes and came up with five rules that every OOP developer should follow to create a system that is easy to maintain and extend over time. These five rules are known as the SOLID principles. SOLID is an acronym introduced by Michael Feathers, which stands for the following principles: Single responsibility principle (SRP): This principle states that a software component (function, class, or module) should focus on one unique task (have only one responsibility). Open/closed principle (OCP): This principle states that software entities should be designed with application growth (new code) in mind (should be open to extension), but the application growth should require the fewer possible number of changes to the existing code (be closed for modification). Liskov substitution principle (LSP): This principle states that we should be able to replace a class in a program with another class as long as both classes implement the same interface. After replacing the class, no other changes should be required, and the program should continue to work as it did originally. Interface segregation principle (ISP): This principle states that we should split interfaces that are very large (general-purpose interfaces) into smaller and more specific ones (many client-specific interfaces) so that clients will only need to know about the methods that are of interest to them. Dependency inversion principle (DIP): This principle states that entities should depend on abstractions (interfaces) as opposed to depending on concretion (classes). In this article, we will see how to write TypeScript code that adheres to these principles so that our applications are easy to maintain and extend over time. Classes In this section, we will look at some details and OOP concepts through examples. Let's start by declaring a simple class: class Person { public name : string; public surname : string; public email : string; constructor(name : string, surname : string, email : string){ this.email = email; this.name = name; this.surname = surname; } greet() { alert("Hi!"); } } var me : Person = new Person("Remo", "Jansen", "remo.jansen@wolksoftware.com"); We use classes to represent the type of an object or entity. A class is composed of a name, attributes, and methods. The class in the preceding example is named Person and contains three attributes or properties (name, surname, and email) and two methods (constructor and greet). Class attributes are used to describe the object's characteristics, while class methods are used to describe its behavior. A constructor is a special method used by the new keyword to create instances (also known as objects) of our class. We have declared a variable named me, which holds an instance of the Person class. The new keyword uses the Person class's constructor to return an object whose type is Person. A class should adhere to the single responsibility principle (SRP). The Person class in the preceding example represents a person, including all their characteristics (attributes) and behaviors (methods). Now let's add some email as validation logic: class Person { public name : string; public surname : string; public email : string; constructor(name : string, surname : string, email : string) { this.surname = surname; this.name = name; if(this.validateEmail(email)) { this.email = email; } else { throw new Error("Invalid email!"); } } validateEmail() { var re = /S+@S+.S+/; return re.test(this.email); } greet() { alert("Hi! I'm " + this.name + ". You can reach me at " + this.email); } } When an object doesn't follow the SRP and it knows too much (has too many properties) or does too much (has too many methods), we say that the object is a God object. The Person class here is a God object because we have added a method named validateEmail that is not really related to the Person class's behavior. Deciding which attributes and methods should or should not be part of a class is a relatively subjective decision. If we spend some time analyzing our options, we should be able to find a way to improve the design of our classes. We can refactor the Person class by declaring an Email class, responsible for e-mail validation, and use it as an attribute in the Person class: class Email { public email : string; constructor(email : string){ if(this.validateEmail(email)) { this.email = email; } else { throw new Error("Invalid email!"); } } validateEmail(email : string) { var re = /S+@S+.S+/; return re.test(email); } } Now that we have an Email class, we can remove the responsibility of validating the emails from the Person class and update its email attribute to use the type Email instead of string: class Person { public name : string; public surname : string; public email : Email; constructor(name : string, surname : string, email : Email){ this.email = email; this.name = name; this.surname = surname; } greet() { alert("Hi!"); } } Making sure that a class has a single responsibility makes it easier to see what it does and how we can extend/improve it. We can further improve our Person and Email classes by increasing the level of abstraction of our classes. For example, when we use the Email class, we don't really need to be aware of the existence of the validateEmail method; so this method could be invisible from outside the Email class. As a result, the Email class would be much simpler to understand. When we increase the level of abstraction of an object, we can say that we are encapsulating the object's data and behavior. Encapsulation is also known as information hiding. For example, the Email class allows us to use emails without having to worry about e-mail validation because the class will deal with it for us. We can make this clearer by using access modifiers (public or private) to flag as private all the class attributes and methods that we want to abstract from the use of the Email class: class Email { private email : string; constructor(email : string){ if(this.validateEmail(email)) { this.email = email; } else { throw new Error("Invalid email!"); } } private validateEmail(email : string) { var re = /S+@S+.S+/; return re.test(email); } get():string { return this.email; } } We can then simply use the Email class without needing to explicitly perform any kind of validation: var email = new Email("remo.jansen@wolksoftware.com"); Interfaces The feature that we will miss the most when developing large-scale web applications with JavaScript is probably interfaces. We have seen that following the SOLID principles can help us to improve the quality of our code, and writing good code is a must when working on a large project. The problem is that if we attempt to follow the SOLID principles with JavaScript, we will soon realize that without interfaces, we will never be able to write SOLID OOP code. Fortunately for us, TypeScript features interfaces. Traditionally, in OOP, we say that a class can extend another class and implement one or more interfaces. An interface can implement one or more interfaces and cannot extend another class or interface. Wikipedia's definition of interfaces in OOP is as follows: In object-oriented languages, the term interface is often used to define an abstract type that contains no data or code, but defines behaviors as method signatures. Implementing an interface can be understood as signing a contract. The interface is a contract, and when we sign it (implement it), we must follow its rules. The interface rules are the signatures of the methods and properties, and we must implement them. We will see many examples of interfaces later in this article. In TypeScript, interfaces don't strictly follow this definition. The two main differences are that in TypeScript: An interface can extend another interface or class An interface can define data and behaviors as opposed to only behaviors Association, aggregation, and composition In OOP, classes can have some kind of relationship with each other. Now, we will take a look at the three different types of relationships between classes. Association We call association those relationships whose objects have an independent lifecycle and where there is no ownership between the objects. Let's take an example of a teacher and student. Multiple students can associate with a single teacher, and a single student can associate with multiple teachers, but both have their own lifecycles (both can be create and delete independently); so when a teacher leaves the school, we don't need to delete any students, and when a student leaves the school, we don't need to delete any teachers. Aggregation We call aggregation those relationships whose objects have an independent lifecycle, but there is ownership, and child objects cannot belong to another parent object. Let's take an example of a cell phone and a cell phone battery. A single battery can belong to a phone, but if the phone stops working, and we delete it from our database, the phone battery will not be deleted because it may still be functional. So in aggregation, while there is ownership, objects have their own lifecycle. Composition We use the term composition to refer to relationships whose objects don't have an independent lifecycle, and if the parent object is deleted, all child objects will also be deleted. Let's take an example of the relationship between questions and answers. Single questions can have multiple answers, and answers cannot belong to multiple questions. If we delete questions, answers will automatically be deleted. Objects with a dependent life cycle (answers, in the example) are known as weak entities. Sometimes, it can be a complicated process to decide if we should use association, aggregation, or composition. This difficulty is caused in part because aggregation and composition are subsets of association, meaning they are specific cases of association. Inheritance One of the most fundamental object-oriented programming features is its capability to extend existing classes. This feature is known as inheritance and allows us to create a new class (child class) that inherits all the properties and methods from an existing class (parent class). Child classes can include additional properties and methods not available in the parent class. Let's return to our previously declared Person class. We will use the Person class as the parent class of a child class named Teacher: class Person { public name : string; public surname : string; public email : Email; constructor(name : string, surname : string, email : Email){ this.name = name; this.surname = surname; this.email = email; } greet() { alert("Hi!"); } } This example is included in the companion source code. Once we have a parent class in place, we can extend it by using the reserved keyword extends. In the following example, we declare a class called Teacher, which extends the previously defined Person class. This means that Teacher will inherit all the attributes and methods from its parent class: class Teacher extends Person { class Teacher extends Person { teach() { alert("Welcome to class!"); } } Note that we have also added a new method named teach to the class Teacher. If we create instances of the Person and Teacher classes, we will be able to see that both instances share the same attributes and methods with the exception of the teach method, which is only available for the instance of the Teacher class: var teacher = new Teacher("remo", "jansen", new Email("remo.jansen@wolksoftware.com")); var me = new Person("remo", "jansen", new Email("remo.jansen@wolksoftware.com")); me.greet(); teacher.greet(); me.teach(); // Error : Property 'teach' does not exist on type 'Person' teacher.teach(); Sometimes, we will need a child class to provide a specific implementation of a method that is already provided by its parent class. We can use the reserved keyword super for this purpose. Imagine that we want to add a new attribute to list the teacher's subjects, and we want to be able to initialize this attribute through the teacher constructor. We will use the super keyword to explicitly reference the parent class constructor inside the child class constructor. We can also use the super keyword when we want to extend an existing method, such as greet. This OOP language feature that allows a subclass or child class to provide a specific implementation of a method that is already provided by its parent classes is known as method overriding. class Teacher extends Person { public subjects : string[]; constructor(name : string, surname : string, email : Email, subjects : string[]){ super(name, surname, email); this.subjects = subjects; } greet() { super.greet(); alert("I teach " + this.subjects); } teach() { alert("Welcome to Maths class!"); } } var teacher = new Teacher("remo", "jansen", new Email("remo.jansen@wolksoftware.com"), ["math", "physics"]); We can declare a new class that inherits from a class that is already inheriting from another. In the following code snippet, we declare a class called SchoolPrincipal that extends the Teacher class, which extends the Person class: class SchoolPrincipal extends Teacher { manageTeachers() { alert("We need to help students to get better results!"); } } If we create an instance of the SchoolPrincipal class, we will be able to access all the properties and methods from its parent classes (SchoolPrincipal, Teacher, and Person): var principal = new SchoolPrincipal("remo", "jansen", new Email("remo.jansen@wolksoftware.com"), ["math", "physics"]); principal.greet(); principal.teach(); principal.manageTeachers(); It is not recommended to have too many levels in the inheritance tree. A class situated too deeply in the inheritance tree will be relatively complex to develop, test, and maintain. Unfortunately, we don't have a specific rule that we can follow when we are unsure whether we should increase the depth of the inheritance tree (DIT). We should use inheritance in such a way that it helps us to reduce the complexity of our application and not the opposite. We should try to keep the DIT between 0 and 4 because a value greater than 4 would compromise encapsulation and increase complexity. Summary In this article, we saw how to work with classes, and interfaces in depth. We were able to reduce the complexity of our application by using techniques such as encapsulation and inheritance. To learn more about TypeScript, the following books published by Packt Publishing (https://www.packtpub.com/) are recommended: TypeScript Essentials (https://www.packtpub.com/web-development/typescript-essentials) Mastering TypeScript(https://www.packtpub.com/web-development/mastering-typescript) Resources for Article: Further resources on this subject: Writing SOLID JavaScript code with TypeScript [article] Introduction to TypeScript [article] An Introduction to Mastering JavaScript Promises and Its Implementation in Angular.js [article]
Read more
  • 0
  • 0
  • 27935

article-image-using-collider-based-system
Packt
17 Feb 2016
10 min read
Save for later

Using a collider-based system

Packt
17 Feb 2016
10 min read
In this article by Jorge Palacios, the author of the book Unity 5.x Game AI Programming Cookbook, you will learn how to implement agent awareness using a mixed approach that considers the previous learnt sensory-level algorithms. (For more resources related to this topic, see here.) Seeing using a collider-based system This is probably the easiest way to simulate vision. We take a collider, be it a mesh or a Unity primitive, and use it as the tool to determine whether an object is inside the agent's vision range or not. Getting ready It's important to have a collider component attached to the same game object using the script on this recipe, as well as the other collider-based algorithms in this chapter. In this case, it's recommended that the collider be a pyramid-based one in order to simulate a vision cone. The lesser the polygons, the faster it will be on the game. How to do it… We will create a component that is able to see enemies nearby by performing the following steps: Create the Visor component, declaring its member variables. It is important to add the corresponding tags into Unity's configuration: using UnityEngine; using System.Collections; public class Visor : MonoBehaviour { public string tagWall = "Wall"; public string tagTarget = "Enemy"; public GameObject agent; } Implement the function for initializing the game object in case the component is already assigned to it: void Start() { if (agent == null) agent = gameObject; } Declare the function for checking collisions for every frame and build it in the following steps: public void OnCollisionStay(Collision coll) { // next steps here } Discard the collision if it is not a target: string tag = coll.gameObject.tag; if (!tag.Equals(tagTarget)) return; Get the game object's position and compute its direction from the Visor: GameObject target = coll.gameObject; Vector3 agentPos = agent.transform.position; Vector3 targetPos = target.transform.position; Vector3 direction = targetPos - agentPos; Compute its length and create a new ray to be shot soon: float length = direction.magnitude; direction.Normalize(); Ray ray = new Ray(agentPos, direction); Cast the created ray and retrieve all the hits: RaycastHit[] hits; hits = Physics.RaycastAll(ray, length); Check for any wall between the visor and target. If none, we can proceed to call our functions or develop our behaviors to be triggered: int i; for (i = 0; i < hits.Length; i++) { GameObject hitObj; hitObj = hits[i].collider.gameObject; tag = hitObj.tag; if (tag.Equals(tagWall)) return; } // TODO // target is visible // code your behaviour below How it works… The collider component checks every frame to know whether it is colliding with any game object in the scene. We leverage the optimizations to Unity's scene graph and engine, and focus only on how to handle valid collisions. After checking whether a target object is inside the vision range represented by the collider, we cast a ray in order to check whether it is really visible or there is a wall in between. Hearing using a collider-based system In this recipe, we will emulate the sense of hearing by developing two entities; a sound emitter and a sound receiver. It is based on the principles proposed by Millington for simulating a hearing system, and uses the power of Unity colliders to detect receivers near an emitter. Getting ready As with the other recipes based on colliders, we will need collider components attached to every object to be checked and rigid body components attached to either emitters or receivers. How to do it… We will create the SoundReceiver class for our agents and SoundEmitter for things such as alarms: Create the class for the SoundReceiver object: using UnityEngine; using System.Collections; public class SoundReceiver : MonoBehaviour { public float soundThreshold; } We define the function for our own behavior to handle the reception of sound: public virtual void Receive(float intensity, Vector3 position) { // TODO // code your own behavior here } Now, let's create the class for the SoundEmitter object: using UnityEngine; using System.Collections; using System.Collections.Generic; public class SoundEmitter : MonoBehaviour { public float soundIntensity; public float soundAttenuation; public GameObject emitterObject; private Dictionary<int, SoundReceiver> receiverDic; } Initialize the list of receivers nearby and emitterObject in case the component is attached directly: void Start() { receiverDic = new Dictionary<int, SoundReceiver>(); if (emitterObject == null) emitterObject = gameObject; } Implement the function for adding new receivers to the list when they enter the emitter bounds: public void OnCollisionEnter(Collision coll) { SoundReceiver receiver; receiver = coll.gameObject.GetComponent<SoundReceiver>(); if (receiver == null) return; int objId = coll.gameObject.GetInstanceID(); receiverDic.Add(objId, receiver); } Also, implement the function for removing receivers from the list when they are out of reach: public void OnCollisionExit(Collision coll) { SoundReceiver receiver; receiver = coll.gameObject.GetComponent<SoundReceiver>(); if (receiver == null) return; int objId = coll.gameObject.GetInstanceID(); receiverDic.Remove(objId); } Define the function for emitting sound waves to nearby agents: public void Emit() { GameObject srObj; Vector3 srPos; float intensity; float distance; Vector3 emitterPos = emitterObject.transform.position; // next step here } Compute sound attenuation for every receiver: foreach (SoundReceiver sr in receiverDic.Values) { srObj = sr.gameObject; srPos = srObj.transform.position; distance = Vector3.Distance(srPos, emitterPos); intensity = soundIntensity; intensity -= soundAttenuation * distance; if (intensity < sr.soundThreshold) continue; sr.Receive(intensity, emitterPos); } How it works… The collider triggers help register agents in the list of agents assigned to an emitter. The sound emission function then takes into account the agent's distance from the emitter in order to decrease its intensity using the concept of sound attenuation. There is more… We can develop a more flexible algorithm by defining different types of walls that affect sound intensity. It works by casting rays and adding up their values to the sound attenuation: Create a dictionary to store wall types as strings (using tags) and their corresponding attenuation: public Dictionary<string, float> wallTypes; Reduce sound intensity this way: intensity -= GetWallAttenuation(emitterPos, srPos); Define the function called in the previous step: public float GetWallAttenuation(Vector3 emitterPos, Vector3 receiverPos) { // next steps here } Compute the necessary values for ray casting: float attenuation = 0f; Vector3 direction = receiverPos - emitterPos; float distance = direction.magnitude; direction.Normalize(); Cast the ray and retrieve the hits: Ray ray = new Ray(emitterPos, direction); RaycastHit[] hits = Physics.RaycastAll(ray, distance); For every wall type found via tags, add up its value (stored in the dictionary): int i; for (i = 0; i < hits.Length; i++) { GameObject obj; string tag; obj = hits[i].collider.gameObject; tag = obj.tag; if (wallTypes.ContainsKey(tag)) attenuation += wallTypes[tag]; } return attenuation; Smelling using a collider-based system Smelling can be simulated by computing collision between an agent and odor particles, scattered throughout the game level. Getting ready In this recipe based on colliders, we will need collider components attached to every object to be checked, which can be simulated by computing a collision between an agent and odor particles. How to do it… We will develop the scripts needed to represent odor particles and agents able to smell: Create the particle's script and define its member variables for computing its lifespan: using UnityEngine; using System.Collections; public class OdorParticle : MonoBehaviour { public float timespan; private float timer; } Implement the Start function for proper validations: void Start() { if (timespan < 0f) timespan = 0f; timer = timespan; } Implement the timer and destroy the object after its life cycle: void Update() { timer -= Time.deltaTime; if (timer < 0f) Destroy(gameObject); } Create the class for representing the sniffer agent: using UnityEngine; using System.Collections; using System.Collections.Generic; public class Smeller : MonoBehaviour { private Vector3 target; private Dictionary<int, GameObject> particles; } Initialize the dictionary for storing odor particles: void Start() { particles = new Dictionary<int, GameObject>(); } Add to the dictionary the colliding objects that have the odor-particle component attached: public void OnCollisionEnter(Collision coll) { GameObject obj = coll.gameObject; OdorParticle op; op = obj.GetComponent<OdorParticle>(); if (op == null) return; int objId = obj.GetInstanceID(); particles.Add(objId, obj); UpdateTarget(); } Release the odor particles from the local dictionary when they are out of the agent's range or are destroyed: public void OnCollisionExit(Collision coll) { GameObject obj = coll.gameObject; int objId = obj.GetInstanceID(); bool isRemoved; isRemoved = particles.Remove(objId); if (!isRemoved) return; UpdateTarget(); } Create the function for computing the odor centroid according to the current elements in the dictionary: private void UpdateTarget() { Vector3 centroid = Vector3.zero; foreach (GameObject p in particles.Values) { Vector3 pos = p.transform.position; centroid += pos; } target = centroid; } Implement the function for retrieving the odor centroid, if any: public Vector3? GetTargetPosition() { if (particles.Keys.Count == 0) return null; return target; } How it works… Just like the hearing recipe based on colliders, we use the trigger colliders to register odor particles to an agent's perception (implemented using a dictionary). When a particle is included or removed, the odor centroid is computed. However, we implement a function to retrieve that centroid because when no odor particle is registered, the internal centroid position is not updated. There is more… The particle emission logic is left behind to be implemented according to our game's needs and it basically instantiates odor-particle prefabs. Also, it is recommended to attach the rigid body components to the agents. Odor particles are prone to be massively instantiated, reducing the game's performance. Seeing using a graph-based system We will start a recipe oriented to use graph-based logic in order to simulate sense. Again, we will start by developing the sense of vision. Getting ready It is important to grasp the chapter regarding path finding in order to understand the inner workings of the graph-based recipes. How to do it… We will just implement a new file: Create the class for handling vision: using UnityEngine; using System.Collections; using System.Collections.Generic; public class VisorGraph : MonoBehaviour { public int visionReach; public GameObject visorObj; public Graph visionGraph; } Validate the visor object: void Start() { if (visorObj == null) visorObj = gameObject; } Define and start building the function needed to detect visibility of a given set of nodes: public bool IsVisible(int[] visibilityNodes) { int vision = visionReach; int src = visionGraph.GetNearestVertex(visorObj); HashSet<int> visibleNodes = new HashSet<int>(); Queue<int> queue = new Queue<int>(); queue.Enqueue(src); } Implement a breath-first search algorithm: while (queue.Count != 0) { if (vision == 0) break; int v = queue.Dequeue(); List<int> neighbours = visionGraph.GetNeighbors(v); foreach (int n in neighbours) { if (visibleNodes.Contains(n)) continue; queue.Enqueue(v); visibleNodes.Add(v); } } Compare the set of visible nodes with the set of nodes reached by the vision system: foreach (int vn in visibleNodes) { if (visibleNodes.Contains(vn)) return true; } Return false if there is no match between the two sets of nodes: return false; How it works… The recipe uses the breath-first search algorithm in order to discover nodes within its vision reach, and then compares this set of nodes with the set of nodes where the agents reside. Summary In this article, we explained some algorithms for simulating senses and agent awareness. Resources for Article: Further resources on this subject: Animation and Unity3D Physics[article] Unity 3-0 Enter the Third Dimension[article] Animation features in Unity 5[article]
Read more
  • 0
  • 0
  • 31157
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-scraping-web-python-quick-start
Packt
17 Feb 2016
9 min read
Save for later

Scraping the Web with Python - Quick Start

Packt
17 Feb 2016
9 min read
In this article we're going to acquire intelligence data from a variety of sources. We might interview people. We might steal files from a secret underground base. We might search the World Wide Web (WWW). (For more resources related to this topic, see here.) Accessing data from the Internet The WWW and Internet are based on a series of agreements called Request for Comments (RFC). The RFCs define the standards and protocols to interconnect different networks, that is, the rules for internetworking. The WWW is defined by a subset of these RFCs that specifies the protocols, behaviors of hosts and agents (servers and clients), and file formats, among other details. In a way, the Internet is a controlled chaos. Most software developers agree to follow the RFCs. Some don't. If their idea is really good, it can catch on, even though it doesn't precisely follow the standards. We often see this in the way some browsers don't work with some websites. This can cause confusion and questions. We'll often have to perform both espionage and plain old debugging to figure out what's available on a given website. Python provides a variety of modules that implement the software defined in the Internet RFCs. We'll look at some of the common protocols to gather data through the Internet and the Python library modules that implement these protocols. Background briefing – the TCP/IP protocols The essential idea behind the WWW is the Internet. The essential idea behind the Internet is the TCP/IP protocol stack. The IP part of this is the internetworking protocol. This defines how messages can be routed between networks. Layered on top of IP is the TCP protocol to connect two applications to each other. TCP connections are often made via a software abstraction called a socket. In addition to TCP, there's also UDP; it's not used as much for the kind of WWW data we're interested in. In Python, we can use the low-level socket library to work with the TCP protocol, but we won't. A socket is a file-like object that supports open, close, input, and output operations. Our software will be much simpler if we work at a higher level of abstraction. The Python libraries that we'll use will leverage the socket concept under the hood. The Internet RFCs defines a number of protocols that build on TCP/IP sockets. These are more useful definitions of interactions between host computers (servers) and user agents (clients). We'll look at two of these: Hypertext Transfer Protocol (HTTP) and File Transfer Protocol (FTP). Using http.client for HTTP GET The essence of web traffic is HTTP. This is built on TCP/IP. HTTP defines two roles: host and user agent, also called server and client, respectively. We'll stick to server and client. HTTP defines a number of kinds of request types, including GET and POST. A web browser is one kind of client software we can use. This software makes GET and POST requests, and displays the results from the web server. We can do this kind of client-side processing in Python using two library modules. The http.client module allows us to make GET and POST requests as well as PUT and DELETE. We can read the response object. Sometimes, the response is an HTML page. Sometimes, it's a graphic image. There are other things too, but we're mostly interested in text and graphics. Here's a picture of a mysterious device we've been trying to find. We need to download this image to our computer so that we can see it and send it to our informant from http://upload.wikimedia.org/wikipedia/commons/7/72/IPhone_Internals.jpg: Here's a picture of the currency we're supposed to track down and pay with: We need to download this image. Here is the link: http://upload.wikimedia.org/wikipedia/en/c/c1/1drachmi_1973.jpg Here's how we can use http.client to get these two image files: import http.client import contextlib path_list = [ "/wikipedia/commons/7/72/IPhone_Internals.jpg", "/wikipedia/en/c/c1/1drachmi_1973.jpg", ] host = "upload.wikimedia.org" with contextlib.closing(http.client.HTTPConnection( host )) as connection: for path in path_list: connection.request( "GET", path ) response= connection.getresponse() print("Status:", response.status) print("Headers:", response.getheaders()) _, _, filename = path.rpartition("/") print("Writing:", filename) with open(filename, "wb") as image: image.write( response.read() ) We're using http.client to handle the client side of the HTTP protocol. We're also using the contextlib module to politely disentangle our application from network resources when we're done using them. We've assigned a list of paths to the path_list variable. This example introduces list objects without providing any background. It's important that lists are surrounded by [] and the items are separated by ,. Yes, there's an extra , at the end. This is legal in Python. We created an http.client.HTTPConnection object using the host computer name. This connection object is a little like a file; it entangles Python with operating system resources on our local computer plus a remote server. Unlike a file, an HTTPConnection object isn't a proper context manager. As we really like context managers to release our resources, we made use of the contextlib.closing() function to handle the context management details. The connection needs to be closed; the closing() function assures that this will happen by calling the connection's close() method. For all of the paths in our path_list, we make an HTTP GET request. This is what browsers do to get the image files mentioned in an HTML page. We print a few things from each response. The status, if everything worked, will be 200. If the status is not 200, then something went wrong and we'll need to read up on the HTTP status code to see what happened. If you use a coffee shop Wi-Fi connection, perhaps you're not logged in. You might need to open a browser to set up a connection. An HTTP response includes headers that provide some additional details about the request and response. We've printed the headers because they can be helpful in debugging any problems we might have. One of the most useful headers is ('Content-Type', 'image/jpeg'). This confirms that we really did get an image. We used _, _, filename = path.rpartition("/") to locate the right-most / character in the path. Recall that the partition() method locates the left-most instance. We're using the right-most one here. We assigned the directory information and separator to the variable _. Yes, _ is a legal variable name. It's easy to ignore, which makes it a handy shorthand for we don't care. We kept the filename in the filename variable. We create a nested context for the resulting image file. We can then read the body of the response—a collection of bytes—and write these bytes to the image file. In one quick motion, the file is ours. The HTTP GET request is what underlies much of the WWW. Programs such as curl and wget are expansions of this example. They execute batches of GET requests to locate one or more pages of content. They can do quite a bit more, but this is the essence of extracting data from the WWW. Changing our client information An HTTP GET request includes several headers in addition to the URL. In the previous example, we simply relied on the Python http.client library to supply a suitable set of default headers. There are several reasons why we might want to supply different or additional headers. First, we might want to tweak the User-Agent header to change the kind of browser that we're claiming to be. We might also need to provide cookies for some kinds of interactions. For information on the user agent string, see http://en.wikipedia.org/wiki/User_agent_string#User_agent_identification. This information may be used by the web server to determine if a mobile device or desktop device is being used. We can use something like this: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/537.75.14 This makes our Python request appear to come from the Safari browser instead of a Python application. We can use something like this to appear to be a different browser on a desktop computer: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 We can use something like this to appear to be an iPhone instead of a Python application: Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_1 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D201 Safari/9537.53 We make this change by adding headers to the request we're making. The change looks like this: connection.request( "GET", path, headers= { 'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_1 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D201 Safari/9537.53', }) This will make the web server treat our Python application like it's on an iPhone. This might lead to a more compact page of data than might be provided to a full desktop computer that makes the same request. The header information is a structure with the { key: value, } syntax. It's important that dictionaries are surrounded by {}, the keys and values are separated by :, and each key-value pair is separated by ,. Yes, there's an extra , at the end. This is legal in Python. There are many more HTTP headers we can provide. The User-Agent header is perhaps most important to gather different kinds of intelligence data from web servers. You can refer more book related to this topic on the following links: Python for Secret Agents - Volume II: (https://www.packtpub.com/application-development/python-secret-agents-volume-ii) Expert Python Programming: (https://www.packtpub.com/application-development/expert-python-programming) Raspberry Pi for Secret Agents: (https://www.packtpub.com/hardware-and-creative/raspberry-pi-secret-agents) Resources for Article: Further resources on this subject: Python Libraries[article] Optimization in Python[article] Introduction to Object-Oriented Programming using Python, JavaScript, and C#[article]
Read more
  • 0
  • 0
  • 15858

article-image-putting-fun-functional-python
Packt
17 Feb 2016
21 min read
Save for later

Putting the Fun in Functional Python

Packt
17 Feb 2016
21 min read
Functional programming defines a computation using expressions and evaluation—often encapsulated in function definitions. It de-emphasizes or avoids the complexity of state change and mutable objects. This tends to create programs that are more succinct and expressive. In this article, we'll introduce some of the techniques that characterize functional programming. We'll identify some of the ways to map these features to Python. Finally, we'll also address some ways in which the benefits of functional programming accrue when we use these design patterns to build Python applications. Python has numerous functional programming features. It is not a purely functional programming language. It offers enough of the right kinds of features that it confers to the benefits of functional programming. It also retains all optimization power available from an imperative programming language. We'll also look at a problem domain that we'll use for many of the examples in this book. We'll try to stick closely to Exploratory Data Analysis (EDA) because its algorithms are often good examples of functional programming. Furthermore, the benefits of functional programming accrue rapidly in this problem domain. Our goal is to establish some essential principles of functional programming. We'll focus on Python 3 features in this book. However, some of the examples might also work in Python 2. (For more resources related to this topic, see here.) Identifying a paradigm It's difficult to be definitive on what fills the universe of programming paradigms. For our purposes, we will distinguish between just two of the many programming paradigms: Functional programming and Imperative programming. One important distinguishing feature between these two is the concept of state. In an imperative language, like Python, the state of the computation is reflected by the values of the variables in the various namespaces. The values of the variables establish the state of a computation; each kind of statement makes a well-defined change to the state by adding or changing (or even removing) a variable. A language is imperative because each statement is a command, which changes the state in some way. Our general focus is on the assignment statement and how it changes state. Python has other statements, such as global or nonlocal, which modify the rules for variables in a particular namespace. Statements like def, class, and import change the processing context. Other statements like try, except, if, elif, and else act as guards to modify how a collection of statements will change the computation's state. Statements like for and while, similarly, wrap a block of statements so that the statements can make repeated changes to the state of the computation. The focus of all these various statement types, however, is on changing the state of the variables. Ideally, each statement advances the state of the computation from an initial condition toward the desired final outcome. This "advances the computation" assertion can be challenging to prove. One approach is to define the final state, identify a statement that will establish this final state, and then deduce the precondition required for this final statement to work. This design process can be iterated until an acceptable initial state is derived. In a functional language, we replace state—the changing values of variables—with a simpler notion of evaluating functions. Each function evaluation creates a new object or objects from existing objects. Since a functional program is a composition of a function, we can design lower-level functions that are easy to understand, and we will design higher-level compositions that can also be easier to visualize than a complex sequence of statements. Function evaluation more closely parallels mathematical formalisms. Because of this, we can often use simple algebra to design an algorithm, which clearly handles the edge cases and boundary conditions. This makes us more confident that the functions work. It also makes it easy to locate test cases for formal unit testing. It's important to note that functional programs tend to be relatively succinct, expressive, and efficient when compared to imperative (object-oriented or procedural) programs. The benefit isn't automatic; it requires a careful design. This design effort is often easier than functionally similar procedural programming. Subdividing the procedural paradigm We can subdivide imperative languages into a number of discrete categories. In this section, we'll glance quickly at the procedural versus object-oriented distinction. What's important here is to see how object-oriented programming is a subset of imperative programming. The distinction between procedural and object-orientation doesn't reflect the kind of fundamental difference that functional programming represents. We'll use code examples to illustrate the concepts. For some, this will feel like reinventing a wheel. For others, it provides a concrete expression of abstract concepts. For some kinds of computations, we can ignore Python's object-oriented features and write simple numeric algorithms. For example, we might write something like the following to get the range of numbers: s = 0 for n in range(1, 10): if n % 3 == 0 or n % 5 == 0: s += n print(s) We've made this program strictly procedural, avoiding any explicit use of Python's object features. The program's state is defined by the values of the variables s and n. The variable, n, takes on values such that 1 ≤ n < 10. As the loop involves an ordered exploration of values of n, we can prove that it will terminate when n == 10. Similar code would work in C or Java using their primitive (non-object) data types. We can exploit Python's Object-Oriented Programming (OOP) features and create a similar program: m = list() for n in range(1, 10): if n % 3 == 0 or n % 5 == 0: m.append(n) print(sum(m)) This program produces the same result but it accumulates a stateful collection object, m, as it proceeds. The state of the computation is defined by the values of the variables m and n. The syntax of m.append(n) and sum(m) can be confusing. It causes some programmers to insist (wrongly) that Python is somehow not purely Object-oriented because it has a mixture of the function()and object.method() syntax. Rest assured, Python is purely Object-oriented. Some languages, like C++, allow the use of primitive data type such as int, float, and long, which are not objects. Python doesn't have these primitive types. The presence of prefix syntax doesn't change the nature of the language. To be pedantic, we could fully embrace the object model, the subclass, the list class, and add a sum method: class SummableList(list): def sum( self ): s= 0 for v in self.__iter__(): s += v return s If we initialize the variable, m, with the SummableList() class instead of the list() method, we can use the m.sum() method instead of the sum(m) method. This kind of change can help to clarify the idea that Python is truly and completely object-oriented. The use of prefix function notation is purely syntactic sugar. All three of these examples rely on variables to explicitly show the state of the program. They rely on the assignment statements to change the values of the variables and advance the computation toward completion. We can insert the assert statements throughout these examples to demonstrate that the expected state changes are implemented properly. The point is not that imperative programming is broken in some way. The point is that functional programming leads to a change in viewpoint, which can, in many cases, be very helpful. We'll show a function view of the same algorithm. Functional programming doesn't make this example dramatically shorter or faster. Using the functional paradigm In a functional sense, the sum of the multiples of 3 and 5 can be defined in two parts: The sum of a sequence of numbers A sequence of values that pass a simple test condition, for example, being multiples of three and five The sum of a sequence has a simple, recursive definition: def sum(seq): if len(seq) == 0: return 0 return seq[0] + sum(seq[1:]) We've defined the sum of a sequence in two cases: the base case states that the sum of a zero length sequence is 0, while the recursive case states that the sum of a sequence is the first value plus the sum of the rest of the sequence. Since the recursive definition depends on a shorter sequence, we can be sure that it will (eventually) devolve to the base case. The + operator on the last line of the preceeding example and the initial value of 0 in the base case characterize the equation as a sum. If we change the operator to * and the initial value to 1, it would just as easily compute a product. Similarly, a sequence of values can have a simple, recursive definition, as follows: def until(n, filter_func, v): if v == n: return [] if filter_func(v): return [v] + until( n, filter_func, v+1 ) else: return until(n, filter_func, v+1) In this function, we've compared a given value, v, against the upper bound, n. If v reaches the upper bound, the resulting list must be empty. This is the base case for the given recursion. There are two more cases defined by the given filter_func() function. If the value of v is passed by the filter_func() function, we'll create a very small list, containing one element, and append the remaining values of the until() function to this list. If the value of v is rejected by the filter_func() function, this value is ignored and the result is simply defined by the remaining values of the until() function. We can see that the value of v will increase from an initial value until it reaches n, assuring us that we'll reach the base case soon. Here's how we can use the until() function to generate the multiples of 3 or 5. First, we'll define a handy lambda object to filter values: mult_3_5= lambda x: x%3==0 or x%5==0 (We will use lambdas to emphasize succinct definitions of simple functions. Anything more complex than a one-line expression requires the def statement.) We can see how this lambda works from the command prompt in the following example: >>> mult_3_5(3) True >>> mult_3_5(4) False >>> mult_3_5(5) True This function can be used with the until() function to generate a sequence of values, which are multiples of 3 or 5. The until() function for generating a sequence of values works as follows: >>> until(10, lambda x: x%3==0 or x%5==0, 0) [0, 3, 5, 6, 9] We can use our recursive sum() function to compute the sum of this sequence of values. The various functions, such as sum(), until(), and mult_3_5() are defined as simple recursive functions. The values are computed without restoring to use intermediate variables to store state. We'll return to the ideas behind this purely functional recursive function definition in several places. It's important to note here that many functional programming language compilers can optimize these kinds of simple recursive functions. Python can't do the same optimizations. Using a functional hybrid We'll continue this example with a mostly functional version of the previous example to compute the sum of the multiples of 3 and 5. Our hybrid functional version might look like the following: print( sum(n for n in range(1, 10) if n%3==0 or n%5==0) ) We've used nested generator expressions to iterate through a collection of values and compute the sum of these values. The range(1, 10) method is an iterable and, consequently, a kind of generator expression; it generates a sequence of values . The more complex expression, n for n in range(1, 10) if n%3==0 or n%5==0, is also an iterable expression. It produces a set of values . A variable, n, is bound to each value, more as a way of expressing the contents of the set than as an indicator of the state of the computation. The sum() function consumes the iterable expression, creating a final object, 23. The bound variable doesn't change once a value is bound to it. The variable, n, in the loop is essentially a shorthand for the values available from the range() function. The if clause of the expression can be extracted into a separate function, allowing us to easily repurpose this with other rules. We could also use a higher-order function named filter() instead of the if clause of the generator expression. As we work with generator expressions, we'll see that the bound variable is at the blurry edge of defining the state of the computation. The variable, n, in this example isn't directly comparable to the variable, n, in the first two imperative examples. The for statement creates a proper variable in the local namespace. The generator expression does not create a variable in the same way as a for statement does: >>> sum( n for n in range(1, 10) if n%3==0 or n%5==0 ) 23 >>> n Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'n' is not defined Because of the way Python uses namespaces, it might be possible to write a function that can observe the n variable in a generator expression. However, we won't. Our objective is to exploit the functional features of Python, not to detect how those features have an object-oriented implementation under the hood. Looking at object creation In some cases, it might help to look at intermediate objects as a history of the computation. What's important is that the history of a computation is not fixed. When functions are commutative or associative, then changes to the order of evaluation might lead to different objects being created. This might have performance improvements with no changes to the correctness of the results. Consider this expression: >>> 1+2+3+4 10 We are looking at a variety of potential computation histories with the same result. Because the + operator is commutative and associative, there are a large number of candidate histories that lead to the same result. Of the candidate sequences, there are two important alternatives, which are as follows: >>> ((1+2)+3)+4 10 >>> 1+(2+(3+4)) 10 In the first case, we fold in values working from left to right. This is the way Python works implicitly. Intermediate objects 3 and 6 are created as part of this evaluation. In the second case, we fold from right-to-left. In this case, intermediate objects 7 and 9 are created. In the case of simple integer arithmetic, the two results have identical performance; there's no optimization benefit. When we work with something like the list append, we might see some optimization improvements when we change the association rules. Here's a simple example: >>> import timeit >>> timeit.timeit("((([]+[1])+[2])+[3])+[4]") 0.8846941249794327 >>> timeit.timeit("[]+([1]+([2]+([3]+[4])))") 1.0207440659869462 In this case, there's some benefit in working from left to right. What's important for functional design is the idea that the + operator (or add() function) can be used in any order to produce the same results. The + operator has no hidden side effects that restrict the way this operator can be used. The stack of turtles When we use Python for functional programming, we embark down a path that will involve a hybrid that's not strictly functional. Python is not Haskell, OCaml, or Erlang. For that matter, our underlying processor hardware is not functional; it's not even strictly object-oriented—CPUs are generally procedural. All programming languages rest on abstractions, libraries, frameworks and virtual machines. These abstractions, in turn, may rely on other abstractions, libraries, frameworks and virtual machines. The most apt metaphor is this: the world is carried on the back of a giant turtle. The turtle stands on the back of another giant turtle. And that turtle, in turn, is standing on the back of yet another turtle. It's turtles all the way down.                                                                                                             – Anonymous
There's no practical end to the layers of abstractions. More importantly, the presence of abstractions and virtual machines doesn't materially change our approach to designing software to exploit the functional programming features of Python. Even within the functional programming community, there are more pure and less pure functional programming languages. Some languages make extensive use of monads to handle stateful things like filesystem input and output. Other languages rely on a hybridized environment that's similar to the way we use Python. We write software that's generally functional with carefully chosen procedural exceptions. Our functional Python programs will rely on the following three stacks of abstractions: Our applications will be functions—all the way down—until we hit the objects The underlying Python runtime environment that supports our functional programming is objects—all the way down—until we hit the turtles The libraries that support Python are a turtle on which Python stands The operating system and hardware form their own stack of turtles. These details aren't relevant to the problems we're going to solve. A classic example of functional programming As part of our introduction, we'll look at a classic example of functional programming. This is based on the classic paper Why Functional Programming Matters by John Hughes. The article appeared in a paper called Research Topics in Functional Programming, edited by D. Turner, published by Addison-Wesley in 1990. Here's a link to the paper Research Topics in Functional Programming: http://www.cs.kent.ac.uk/people/staff/dat/miranda/whyfp90.pdf This discussion of functional programming in general is profound. There are several examples given in the paper. We'll look at just one: the Newton-Raphson algorithm for locating the roots of a function. In this case, the function is the square root. It's important because many versions of this algorithm rely on the explicit state managed via loops. Indeed, the Hughes paper provides a snippet of the Fortran code that emphasizes stateful, imperative processing. The backbone of this approximation is the calculation of the next approximation from the current approximation. The next_() function takes x, an approximation to the sqrt(n) method and calculates a next value that brackets the proper root. Take a look at the following example: def next_(n, x): return (x+n/x)/2 This function computes a series of values . The distance between the values is halved each time, so they'll quickly get to converge on the value such that, which means . We don't want to call the method next() because this name would collide with a built-in function. We call it the next_() method so that we can follow the original presentation as closely as possible. Here's how the function looks when used in the command prompt: >>> n= 2 >>> f= lambda x: next_(n, x) >>> a0= 1.0 >>> [ round(x,4) for x in (a0, f(a0), f(f(a0)), f(f(f(a0))),) ] [1.0, 1.5, 1.4167, 1.4142] We've defined the f() method as a lambda that will converge on . We started with 1.0 as the initial value for . Then we evaluated a sequence of recursive evaluations: , and so on. We evaluated these functions using a generator expression so that we could round off each value. This makes the output easier to read and easier to use with doctest. The sequence appears to converge rapidly on . We can write a function, which will (in principle) generate an infinite sequence of values converging on the proper square root: def repeat(f, a): yield a for v in repeat(f, f(a)): yield v This function will generate approximations using a function, f(), and an initial value, a. If we provide the next_() function defined earlier, we'll get a sequence of approximations to the square root of the n argument. The repeat() function expects the f() function to have a single argument, however, our next_() function has two arguments. We can use a lambda object, lambda x: next_(n, x), to create a partial version of the next_() function with one of two variables bound. The Python generator functions can't be trivially recursive, they must explicitly iterate over the recursive results, yielding them individually. Attempting to use a simple return repeat(f, f(a)) will end the iteration, returning a generator expression instead of yielding the sequence of values. We have two ways to return all the values instead of returning a generator expression, which are as follows: We can write an explicit for loop as follows: for x in some_iter: yield x. We can use the yield from statement as follows: yield from some_iter. Both techniques of yielding the values of a recursive generator function are equivalent. We'll try to emphasize yield from. In some cases, however, the yield with a complex expression will be more clear than the equivalent mapping or generator expression. Of course, we don't want the entire infinite sequence. We will stop generating values when two values are so close to each other that we can call either one the square root we're looking for. The common symbol for the value, which is close enough, is the Greek letter Epsilon, ε, which can be thought of as the largest error we will tolerate. In Python, we'll have to be a little clever about taking items from an infinite sequence one at a time. It works out well to use a simple interface function that wraps a slightly more complex recursion. Take a look at the following code snippet: def within(ε, iterable): def head_tail(ε, a, iterable): b= next(iterable) if abs(a-b) <= ε: return b return head_tail(ε, b, iterable) return head_tail(ε, next(iterable), iterable) We've defined an internal function, head_tail(), which accepts the tolerance, ε, an item from the iterable sequence, a, and the rest of the iterable sequence, iterable. The next item from the iterable bound to a name b. If , then the two values that are close enough together that we've found the square root. Otherwise, we use the b value in a recursive invocation of the head_tail() function to examine the next pair of values. Our within() function merely seeks to properly initialize the internal head_tail() function with the first value from the iterable parameter. Some functional programming languages offer a technique that will put a value back into an iterable sequence. In Python, this might be a kind of unget() or previous() method that pushes a value back into the iterator. Python iterables don't offer this kind of rich functionality. We can use the three functions next_(), repeat(), and within() to create a square root function, as follows: def sqrt(a0, ε, n): return within(ε, repeat(lambda x: next_(n,x), a0)) We've used the repeat() function to generate a (potentially) infinite sequence of values based on the next_(n,x) function. Our within() function will stop generating values in the sequence when it locates two values with a difference less than ε. When we use this version of the sqrt() method, we need to provide an initial seed value, a0, and an ε value. An expression like sqrt(1.0, .0001, 3) will start with an approximation of 1.0 and compute the value of to within 0.0001. For most applications, the initial a0 value can be 1.0. However, the closer it is to the actual square root, the more rapidly this method converges. The original example of this approximation algorithm was shown in the Miranda language. It's easy to see that there are few profound differences between Miranda and Python. The biggest difference is Miranda's ability to construct cons, a value back into an iterable, doing a kind of unget. This parallelism between Miranda and Python gives us confidence that many kinds of functional programming can be easily done in Python. Summary We've looked at programming paradigms with an eye toward distinguishing the functional paradigm from two common imperative paradigms in details. For more information kindly take a look at the following books, also by Packt Publishing: Learning Python (https://www.packtpub.com/application-development/learning-python) Mastering Python (https://www.packtpub.com/application-development/mastering-python) Mastering Object-oriented Python (https://www.packtpub.com/application-development/mastering-object-oriented-python) Resources for Article: Further resources on this subject: Saying Hello to Unity and Android [article] Using Specular in Unity [article] Unity 3.x Scripting-Character Controller versus Rigidbody [article]
Read more
  • 0
  • 1
  • 6081

article-image-asking-permission-getting-your-head-around-marshmallows-runtime-permissions
Packt
17 Feb 2016
8 min read
Save for later

Asking Permission: Getting your head around Marshmallow's Runtime Permissions

Packt
17 Feb 2016
8 min read
In Android, each application runs with distinct system IDs known as Linux user ID and Group ID. The system parts are also separated into distinct IDs, forming isolated zones for applications—from each other and from the system. As part of this isolated life cycle scheme, accessing services or other applications' data requires that you declare this desire in advance by requesting a permission. This is done by adding the uses-permission element to your AndroidManifest.xml file. Your manifest may have zero or more uses-permission elements, and all of them must be the direct children of the root <manifest> element. Trying to access data or features without proper permission should give out a security exception (using a SecurityException class), informing you about the missing permission in most cases. The sendBroadcast(Intent) method is exceptional as it checks permissions after the method call has returned, so you will not receive an exception if there are permission failures. A permission failure should be printed to the system log. Note that in Android versions prior to Marshmallow, missing permissions were due to missing declarations in the manifest. Hence, it is important that you keep permissions in mind when you come up with the feature list for your app. (For more resources related to this topic, see here.) Understanding Android Marshmallow permissions Android Marshmallow introduces a new application permissions model, allowing a simpler process for users when installing and/or upgrading applications. Applications running on Marshmallow should work according to a new permissions model, where the user can grant or revoke permissions after the installation—permissions are not given until there is user acceptance. Supporting the new permissions model is backward-compatible, which means your apps can still An overview With the Android Marshmallow version, a new application permissions model has been introduced. Let's review it a bit more thoroughly: Declaring permissions: All permissions an app needs are declared in the manifest, which is done to preserve backward compatibility in a manner similar to earlier Android platform versions. Permission groups: As discussed previously, permissions are divided into permission groups based on their functionalities: PROTECTION_NORMAL permissions: Some of the permissions are granted when users install the app. Upon the installation, the system checks your app's manifest and automatically grants permissions that match the PROTECTION_NORMAL group. INTERNET permission: One important permission is the INTERNET permission, which will be granted upon the installation, and the user can't revoke it. App signature permissions granted: The user is not prompted to grant any permissions at the time of installation. Permissions granted by users at runtime: You as an app developer need to request a permission in your app; a system dialog is shown to the user, and the user response is passed back to your app, notifying whether the permission is granted. Permissions can be revoked: Users can revoke permissions that were granted previously. We must learn how to handle these cases, as we'll learn later on. If an app targets an Android Marshmallow version, it must use the new permissions model. Permission groups When working with permissions, we divide them into groups. This division is done for fast user interaction when reviewing and approving permissions. Granting is done only once per permission group. If you add a new permission or request a new permission from the same permission group and the user has already approved that group, the system will grant you the added permission without bothering the user about the approval. For more information on this, visit https://developer.android.com/reference/android/content/pm/PermissionInfo.html#constants[GS1] . When the user installs an app, the app is granted only those permissions that are listed in the manifest that belongs to the PROTECTION_NORMAL group. Requesting permissions from the PROTECTION_SIGNATURE group will be granted only if the application is signed with the same certificate as the app with the declared permission. Apps cannot request signature permissions at runtime. System components automatically receive all the permissions listed in their manifests. Runtime permissions Android Marshmallow showcased a new permissions model where users were able to directly manage app permissions at application runtime. Google has altered the old permissions model, mostly to enable easier and frictionless installations and auto-updates for users as well as for app developers. This allows users to install the app without the need to preapprove each permission the application needs. The user can install the app without going through the phase of checking each permission and declining the installation due to a single permission. Users can grant or revoke permissions for installed apps, leaving the tweaking and the freedom of choice in the users' hands. Most of the applications will need to address these issues when updating the target API to 23. Taking coding permissions into account Well, after all the explanations, we've reached the coding part, and this is where we will get our coding hands dirty. The following are the methods used for coding permissions: Context.checkSelfPermission(): This checks whether your app has been granted a permission Activity.requestPermission(): This requests a permission at runtime Even if your app is not yet targeting Android Marshmallow, you should test your app and prepare to support it. Testing permissions In the Android Marshmallow permissions model, your app must ask the user for individual permissions at runtime. There is limited compatibility support for legacy apps, and you should test your app and also test a version to make sure it's supported. You can use the following test guide and conduct app testing with the new behavior: Map your app's permissions Test flows with permissions granted and revoked The adb command shell can be quite helpful to check for permissions: Listing application permissions and status by group can be done using the following adb command: adb shell pm list permissions -g You can grant or revoke permissions using the following adb syntax: adb shell pm [grant|revoke] <permission.name> You can grant permissions and install apk using the following adb command: adb install -g <path_to_apk> Coding for runtime permissions When we want to adjust our application to the new model, we need to make sure that we organize our steps and leave no permission stranded: Check what platform the app is running on: When running a piece of code that is sensitive at the API level, we start by checking the version/API level that we are running on. By now, you should be familiar with Build.VERSION.SDK_INT. Check whether the app has the required permission: Here, we get ourselves a brand new API call: Context.checkSelfPermission(String permission_name) With this, we silently check whether permissions are granted or not. This method returns immediately, so any permission-related controls/flows should be dealt with by checking this first. Prompting for permissions: We have a new API call, Activity.requestPermissions (String[] permissions, int requestCode). This call triggers the system to show the dialog requesting a permission. This method functions asynchronously. You can request more than one permission at once. The second argument is a simple request code returned in the callback so that you can recognize the calls. This is just like how we've been dealing with startActivityForResult() and onActivityResult() for years. Another new API is Activity.shouldShowRequestPermissionRationale(String permission). This method returns true when you have requested a permission and the user denied the request. It's considered a good practice after verifying that you explain to the user why you need that exact permission. The user can decide to turn down the permission request and select the Don't ask again option; then, this method will return false. The following sample code checks whether the app has permission to read the user's contacts. It requests the permission if required, and the result callback returns to onRequestPermissionsResult: if (checkSelfPermission(Manifest.permission.READ_CONTACTS) != PackageManager.PERMISSION_GRANTED) { requestPermissions(new String[]{Manifest.permission.READ_CONTACTS}, SAMPLE_MATRIXY_READ_CONTACTS); } //Now this is our callback @Override public void onRequestPermissionsResult(int requestCode, String permissions[], int[] grantResults) { switch (requestCode) { case SAMPLE_MATRIXY_READ_CONTACTS: if (grantResults[0] == PackageManager.PERMISSION_GRANTED) { // permission granted - we can continue the feature flow. } else { // permission denied! - we should disable the functionality that depends on this permission. } } } Just to make sure we all know the constants used, here's the explanation: public static final int PERMISSION_DENIED=-1: Since it's API level 1, permission has not been granted to the given package public static final int PERMISSION_GRANTED=0: Since it's API level 1. permission has been granted to the given package. If the user denies your permission request, your app should take the appropriate action, such as notifying the user why this permission is required or explaining that the feature can't work without it. Your app cannot assume user interaction has taken place because the user can choose to reject granting a permission along with the do not show again option; your permission request is automatically rejected and onRequestPermissionsResult gets the result back. Summary To learn more about Android 6 Essentials, the following books published by Packt Publishing (https://www.packtpub.com/) are recommended: Android User Interface Development: Beginner's Guide (https://www.packtpub.com/application-development/android-user-interface-development-beginners-guide) Android Native Development Kit Cookbook (https://www.packtpub.com/application-development/android-native-development-kit-cookbook) Resources for Article: Further resources on this subject: Practical How-To Recipes for Android [article] Working with Xamarin.Android [article] Detecting Shapes Employing Hough Transform [article]
Read more
  • 0
  • 0
  • 2227

article-image-chrome-custom-tabs
Packt
17 Feb 2016
16 min read
Save for later

Chrome Custom Tabs

Packt
17 Feb 2016
16 min read
Well, most of us know tabs from every day Internet browsing. It doesn't really matter which browser you use; all browsers support tabs and multiple tabs' browsing. This allows us to have more than one website open at the same time and navigate between the opened instances. In Android, things are much the same, but when using WebView, you don't have tabs. This article will give highlights about WebView and the new feature of Android 6, Chrome custom tabs. (For more resources related to this topic, see here.) What is WebView? WebView is the part in the Android OS that's responsible for rendering web pages in most Android apps. If you see web content in an Android app, chances are you're looking at WebView. The major exceptions to this rule are some of the Android browsers, such as Chrome, Firefox, and so on. In Android 4.3 and lower, WebView uses code based on Apple's Webkit. In Android 4.4 and higher, WebView is based on the Chromium project, which is the open source base of Google Chrome. In Android 5.0, WebView was decoupled into a separate app that allowed timely updates through Google Play without requiring firmware updates to be issued, and the same technique was used with Google Play services. Now, let's talk again about a simple scenario: we want to display web content (URL-related) in our application. We have two options: either launch a browser or build our own in-app browser using WebView. Both options have trade-offs or disadvantages if we write them down. A browser is an external application and you can't really change its UI; while using it, you push the users to other apps and you may lose them in the wild. On the other hand, using WebView will keep the users tightly inside. However, actually dealing with all possible actions in WebView is quite an overhead. Google heard our rant and came to the rescue with Chrome custom tabs. Now we have better control over the web content in our application, and we can stitch web content into our app in a cleaner, prettier manner. Customization options Chrome custom tabs allow several modifications and tweaks: The toolbar color Enter and exit animations Custom actions for the toolbar and overflow menu Prestarted and prefetched content for faster loading When to use Chrome custom tabs Ever since WebView came out, applications have been using it in multiple ways, embedding content—local static content inside the APK and dynamic content as loading web pages that were not designed for mobile devices at the beginning. Later on we saw the rise of the mobile web era complete with hybrid applications). Chrome custom tabs are a bit more than just loading local content or mobile-compatible web content. They should be used when you load web data and want to allow simple implementation and easier code maintenance and, furthermore, make the web content part of your application—as if it's always there within your app. Among the reasons why you should use custom tabs are the following: Easy implementation: you use the support library when required or just add extras to your View intent. It's that simple. In app UI modifications, you can do the following: Set the toolbar color Add/change the action button Add custom menu items to the overflow menu Set and create custom in/out animations when entering the tab or exiting to the previous location Easier navigation and navigation logic: you can get a callback notifying you about an external navigation, if required. You know when the user navigates to web content and where they should return when done. Chrome custom tabs allow added performance optimizations that you can use: You can keep the engine running, so to speak, and actually give the custom tab a head start to start itself and do some warm up prior to using it. This is done without interfering or taking away precious application resources. You can provide a URL to load in advance in the background while waiting for other user interactions. This speeds up the user-visible page loading time and gives the user a sense of blazing fast application where all the content is just a click away. While using the custom tab, the application won't be evicted as the application level will still be in the foreground even though the tab is on top of it. So, we remain at the top level for the entire usage time (unless a phone call or some other user interaction leads to a change). Using the same Chrome container means that users are already signed in to sites they connected to in the past; specific permissions that were granted previously apply here as well; even fill data, autocomplete, and sync work here. Chrome custom tabs allow us give the users the latest browser implementation on pre-Lollipop devices where WebView is not the latest version. The implementation guide As discussed earlier, we have a couple of features integrated into Chrome custom tabs. The first customizes the UI and interaction with the custom tabs. The second allows pages to be loaded faster and keeps the application alive. Can we use Chrome custom tabs? Before we start using custom tabs, we want to make sure they're supported. Chrome custom tabs expose a service, so the best check for support is to try and bind to the service. Success means that custom tabs are supported and can be used. You can check out this gist, which shows a helper how to to check it, or check the project source code later on at https://gist.github.com/MaTriXy/5775cb0ff98216b2a99d. After checking and learning that support exists, we will start with the UI and interaction part. Custom UI and tab interaction Here, we will use the well-known ACTION_VIEW intent action, and by appending extras to the intent sent to Chrome, we will trigger changes in the UI. Remember that the ACTION_VIEW intent is compatible with all browsers, including Chrome. There are some phones without Chrome out there, or there are instances where the device's default browser isn't Chrome. In these cases, the user will navigate to the specific browser application. Intent is a convenient way to pass that extra data we want Chrome to get. Don't use any of these flags when calling to the Chrome custom tabs: FLAG_ACTIVITY_NEW_TASK FLAG_ACTIVITY_NEW_DOCUMENT Before using the API, we need to add it to our gradle file: compile 'com.android.support:customtabs:23.1.0' This will allow us to use the custom tab support library in our application: CustomTabsIntent.EXTRA_SESSION The preceding code is an extra from the custom tabs support library; it's used to match the session. It must be included in the intent when opening a custom tab. It can be null if there is no need to match any service-side sessions with the intent. We have a sample project to show the options for the UI called ChubbyTabby at https://github.com/MaTriXy/ChubbyTabby. We will go over the important parts here as well. Our main interaction comes from a special builder from the support library called CustomTabsIntent.Builder; this class will help us build the intent we need for the custom tab: CustomTabsIntent.Builder intentBuilder = new CustomTabsIntent.Builder(); //init our Builder //Setting Toolbar Color int color = getResources().getColor(R.color.primary); //we use primary color for our toolbar as well - you can define any color you want and use it. intentBuilder.setToolbarColor(color); //Enabling Title showing intentBuilder.setShowTitle(true); //this will show the title in the custom tab along the url showing at the bottom part of the tab toolbar. //This part is adding custom actions to the over flow menu String menuItemTitle = getString(R.string.menu_title_share); PendingIntent menuItemPendingIntent = createPendingShareIntent(); intentBuilder.addMenuItem(menuItemTitle, menuItemPendingIntent); String menuItemEmailTitle = getString(R.string.menu_title_email); PendingIntent menuItemPendingIntentTwo = createPendingEmailIntent(); intentBuilder.addMenuItem(menuItemEmailTitle, menuItemPendingIntentTwo); //Setting custom Close Icon. intentBuilder.setCloseButtonIcon(mCloseButtonBitmap); //Adding custom icon with custom action for the share action. intentBuilder.setActionButton(mActionButtonBitmap, getString(R.string.menu_title_share), createPendingShareIntent()); //Setting start and exit animation for the custom tab. intentBuilder.setStartAnimations(this, R.anim.slide_in_right, R.anim.slide_out_left); intentBuilder.setExitAnimations(this, android.R.anim.slide_in_left, android.R.anim.slide_out_right); CustomTabActivityHelper.openCustomTab(this, intentBuilder.build(), Uri.parse(URL), new WebviewFallback(), useCustom);  A few things to notice here are as follows: Every menu item uses a pending intent; if you don't know what a pending intent is, head to http://developer.android.com/reference/android/app/PendingIntent.html When we set custom icons, such as close buttons or an action button, for that matter, we use bitmaps and we must decode the bitmap prior to passing it to the builder Setting animations is easy and you can use animations' XML files that you created previously; just make sure that you test the result before releasing the app The following screenshot is an example of a Chrome custom UI and tab: The custom action button As developers, we have full control over the action buttons presented in our custom tab. For most use cases, we can think of a share action or maybe a more common option that your users will perform. The action button is basically a bundle with an icon of the action button and a pending intent that will be called by Chrome when your user hits the action button. The icon should be 24 dp in height and 24-48 dp in width according to specifications: //Adding custom icon with custom action for the share action intentBuilder.setActionButton(mActionButtonBitmap, getString(R.string.menu_title_share), createPendingShareIntent()); Configuring a custom menu By default, Chrome custom tabs usually have a three-icon row with Forward, Page Info, and Refresh on top at all times and Find in page and Open in Browser (Open in Chrome can appear as well) at the footer of the menu. We, developers, have the ability to add and customize up to three menu items that will appear between the icon row and foot items as shown in the following screenshot: The menu we see is actually represented by an array of bundles, each with menu text and a pending intent that Chrome will call on your behalf when the user taps the item: //This part is adding custom buttons to the over flow menu String menuItemTitle = getString(R.string.menu_title_share); PendingIntent menuItemPendingIntent = createPendingShareIntent(); intentBuilder.addMenuItem(menuItemTitle, menuItemPendingIntent); String menuItemEmailTitle = getString(R.string.menu_title_email); PendingIntent menuItemPendingIntentTwo = createPendingEmailIntent(); intentBuilder.addMenuItem(menuItemEmailTitle, menuItemPendingIntentTwo); Configuring custom enter and exit animations Nothing is complete without a few animations to tag along. This is no different, as we have two transitions to make: one for the custom tab to enter and another for its exit; we have the option to set a specific animation for each start and exit animation: //Setting start and exit animation for the custom tab. intentBuilder.setStartAnimations(this,R.anim.slide_in_right, R.anim.slide_out_left); intentBuilder.setExitAnimations(this, android.R.anim.slide_in_left, android.R.anim.slide_out_right); Chrome warm-up Normally, after we finish setting up the intent with the intent builder, we should call CustomTabsIntent.launchUrl (Activity context, Uri url), which is a nonstatic method that will trigger a new custom tab activity to load the URL and show it in the custom tab. This can take up quite some time and impact the impression of smoothness the app provides. We all know that users demand a near-instantaneous experience, so Chrome has a service that we can connect to and ask it to warm up the browser and its native components. Calling this will ask Chrome to perform the following: The DNS preresolution of the URL's main domain The DNS preresolution of the most likely subresources Preconnection to the destination, including HTTPS/TLS negotiation The process to warm up Chrome is as follows: Connect to the service. Attach a navigation callback to get notified upon finishing the page load. On the service, call warmup to start Chrome behind the scenes. Create newSession; this session is used for all requests to the API. Tell Chrome which pages the user is likely to load with mayLaunchUrl. Launch the intent with the session ID generated in step 4. Connecting to the Chrome service Connecting to the Chrome service involves dealing with Android Interface Definition Language (AIDL). If you don't know about AIDL, read http://developer.android.com/guide/components/aidl.html. The interface is created with AIDL, and it automatically creates a proxy service class for you: CustomTabsClient.bindCustomTabsService() So, we check for the Chrome package name; in our sample project, we have a special method to check whether Chrome is present in all variations. After we set the package, we bind to the service and get a CustomTabsClient object that we can use until we're disconnected from the service: pkgName - This is one of several options checking to see if we have a version of Chrome installed can be one of the following static final String STABLE_PACKAGE = "com.android.chrome"; static final String BETA_PACKAGE = "com.chrome.beta"; static final String DEV_PACKAGE = "com.chrome.dev"; static final String LOCAL_PACKAGE = "com.google.android.apps.chrome"; private CustomTabsClient mClient; // Binds to the service. CustomTabsClient.bindCustomTabsService(myContext, pkgName, new CustomTabsServiceConnection() { @Override public void onCustomTabsServiceConnected(ComponentName name, CustomTabsClient client) { // CustomTabsClient should now be valid to use mClient = client; } @Override public void onServiceDisconnected(ComponentName name) { // CustomTabsClient is no longer valid which also invalidates sessions. mClient = null; } });  After we bind to the service, we can call the proper methods we need. Warming up the browser process The method for this is as follows: boolean CustomTabsClient.warmup(long flags) //With our valid client earlier we call the warmup method. mClient.warmup(0); Flags are currently not being used, so we pass 0 for now. The warm-up procedure loads native libraries and the browser process required to support custom tab browsing later on. This is asynchronous, and the return value indicates whether the request has been accepted or not. It returns true to indicate success. Creating a new tab session The method for this is as follows: boolean CustomTabsClient.newSession(ICustomTabsCallback callback) The new tab session is used as the grouping object tying the mayLaunchUrl call, the VIEW intent that we build, and the tab generated altogether. We can get a callback associated with the created session that would be passed for any consecutive mayLaunchUrl calls. This method returns CustomTabsSession when a session is created successfully; otherwise, it returns Null. Setting the prefetching URL The method for this is as follows: boolean CustomTabsSession.mayLaunchUrl (Uri url, Bundle extras, List<Bundle> otherLikelyBundles) This method will notify the browser that a navigation to this URL will happen soon. Make sure that you call warmup() prior to calling this method—this is a must. The most likely URL has to be specified first, and you can send an optional list of other likely URLs (otherLikelyBundles). Lists have to be sorted in a descending order and the optional list may be ignored. A new call to this method will lower the priority of previous calls and can result in URLs not being prefetched. Boolean values inform us whether the operation has been completed successfully. Custom tabs connection callback The method for this is as follows: void CustomTabsCallback.onNavigationEvent (int navigationEvent, Bundle extras) We have a callback triggered upon each navigation event in the custom tab. The int navigationEvent element is one of the six that defines the state the page is in. Refer to the following code for more information: //Sent when the tab has started loading a page. public static final int NAVIGATION_STARTED = 1; //Sent when the tab has finished loading a page. public static final int NAVIGATION_FINISHED = 2; //Sent when the tab couldn't finish loading due to a failure. public static final int NAVIGATION_FAILED = 3; //Sent when loading was aborted by a user action. public static final int NAVIGATION_ABORTED = 4; //Sent when the tab becomes visible. public static final int TAB_SHOWN = 5; //Sent when the tab becomes hidden. public static final int TAB_HIDDEN = 6; private static class NavigationCallback extends CustomTabsCallback { @Override public void onNavigationEvent(int navigationEvent, Bundle extras) { Log.i(TAG, "onNavigationEvent: Code = " + navigationEvent); } } Summary In this article, we learned about a newly added feature, Chrome custom tabs, which allows us to embed web content into our application and modify the UI. Chrome custom tabs allow us to provide a fuller, faster in-app web experience for our users. We use the Chrome engine under the hood, which allows faster loading than regular WebViews or loading the entire Chrome (or another browser) application. We saw that we can preload pages in the background, making it appear as if our data is blazing fast. We can customize the look and feel of our Chrome tab so that it matches our app. Among the changes we saw were the toolbar color, transition animations, and even the addition of custom actions to the toolbar. Custom tabs also benefit from Chrome features such as saved passwords, autofill, tap to search, and sync; these are all available within a custom tab. For developers, integration is quite easy and requires only a few extra lines of code in the basic level. The support library helps with more complex integration, if required. This is a Chrome feature, which means you get it on any Android device where the latest versions of Chrome are installed. Remember that the Chrome custom tab support library changes with new features and fixes, which is the same as other support libraries, so please update your version and make sure that you use the latest API to avoid any issues. To learn more about Chrome custom tabs and Android 6, refer to the following books: Android 6 Essentials (https://www.packtpub.com/application-development/android-6-essentials) Augmented Reality for Android Application Development (https://www.packtpub.com/application-development/augmented-reality-android-application-development) Resources for Article: Further resources on this subject: Android and iOS Apps Testing at a Glance [Article] Working with Xamarin.Android [Article] Mobile Phone Forensics – A First Step into Android Forensics [Article]
Read more
  • 0
  • 0
  • 5817
article-image-openstack-performance-availability
Packt
17 Feb 2016
21 min read
Save for later

OpenStack Performance, Availability

Packt
17 Feb 2016
21 min read
In this article by Tony Campbell, author of the book Troubleshooting OpenStack, we will cover some of the chronic issues that might be early signs of trouble. This article is more about prevention and aims to help you avoid emergency troubleshooting as much as possible. (For more resources related to this topic, see here.) Database Many OpenStack services make heavy use of a database. Production deployments will typically use MySQL or Postgres as the backend database server. As you may have learned, a failing or misconfigured database will quickly lead to trouble in your OpenStack cluster. Database problems can also present more subtle concerns that may grow into huge problems if neglected. Availability This database server can become a single point of failure if your database server is not deployed in a highly available configuration. OpenStack does not require a high availability installation of your database, and as a result, many installations may skip this step. However, production deployments of OpenStack should take care to ensure that their database can survive the failure of a single database server. MySQL with Galera Cluster For installations using the MySQL database engine, there are several options for clustering your installation. One popular method is to leverage Galera Cluster (http://http://galeracluster.com/). Galera Cluster for MySQL leverages synchronous replication and provides a multi-master cluster, which offers high availability for your OpenStack databases. Postgres Installations that use the Postgres database engine have several options such as high availability, load balancing, and replication. These options include block device replication with DRBD, log shipping, Master-Standby replication based on triggers, statement-based replication, and asynchronous multi-master replication. For details, refer to the Postgres High Availability Guide (http://www.postgresql.org/docs/current/static/high-availability.html). Performance Database performance is one of those metrics that can degrade over time. For an administrator who does not pay attention to small problems in this area, these can eventually become large problems. A wise administrator will regularly monitor the performance of their database and will constantly be on the lookout for slow queries, high database loads, and other indications of trouble. MySQL There are several options for monitoring your MySQL server, some of which are commercial and many others that are open source. Administrators should evaluate the options available and select a solution that fits their current set of tools and operating environment. There are several performance metrics you will want to monitor. Show Status The MySQL SHOW STATUS statement can be executed from the mysql command prompt. The output of this statement is server status information with over 300 variables reported. To narrow down the information, you can leverage a LIKE clause on the variable_name to display the sections you are interested in. Here is an abbreviated list of the output instances returned by SHOW STATUS: mysql> SHOW STATUS; +------------------------------------------+-------------+ | Variable_name                            | Value       | +------------------------------------------+-------------+ | Aborted_clients                          | 29          | | Aborted_connects                         | 27          | | Binlog_cache_disk_use                    | 0           | | Binlog_cache_use                         | 0           | | Binlog_stmt_cache_disk_use               | 0           | | Binlog_stmt_cache_use                    | 0           | | Bytes_received                           | 614         | | Bytes_sent                               | 33178       | Mytop Mytop is a command-line utility inspired by the Linux top command. Mytop retrieves data from the MySql SHOW PROCESSLIST command and the SHOW STATUS command. Data from these commands is refreshed, processed, and displayed in the output of the Mytop command. The Mytop output includes a header, which contains summary data followed by a thread section. Mytop header section Here is an example of the header output from the Mytop command: MySQL on localhost (5.5.46)                                                                                                                    load 1.01 0.85 0.79 4/538 23573 up 5+02:19:24 [14:35:24]  Queries: 3.9M     qps:    9 Slow:     0.0         Se/In/Up/De(%):    49/00/08/00  Sorts:     0 qps now:   10 Slow qps: 0.0  Threads:   30 (   1/   4) 40/00/12/00  Cache Hits: 822.0 Hits/s:  0.0 Hits now:   0.0  Ratio:  0.0%  Ratio now:  0.0%  Key Efficiency: 97.3%  Bps in/out:  1.7k/ 3.1k   Now in/out:  1.0k/ 3.9k As demonstrated in the preceding output, the header section for the Mytop command includes the following information: The hostname and MySQL version The server load The MySQL server uptime The total number of queries The average number of queries Slow queries The percentage of Select, Insert, Update, and Delete queries Queries per second Threads Cache hits Key efficiency Mytop thread section They Mytop thread section will list as many threads as it can display. The threads are ordered by the Time column, which displays the threads' idle time:        Id      User         Host/IP         DB       Time    Cmd    State Query                                                                                                                                --      ----         -------         --       ----    ---    ----- ----------                                                                                                                         3461   neutron  174.143.201.98    neutron   5680  Sleep                                                                                                                                            3477    glance  174.143.201.98     glance   1480  Sleep                                                                                                                                            3491      nova  174.143.201.98     nova      880  Sleep                                                                                                                                             3512      nova  174.143.201.98    nova      281  Sleep                                                                                                                                             3487  keystone  174.143.201.98   keystone        280  Sleep                                                                                                                                             3489    glance  174.143.201.98     glance        280  Sleep                                                                                                                                            3511  keystone  174.143.201.98   keystone        280  Sleep                                                                                                                                            3513   neutron  174.143.201.98    neutron        280  Sleep                                                                                                                                             3505  keystone  174.143.201.98   keystone        279  Sleep                                                                                                                                             3514  keystone  174.143.201.98   keystone        141  Sleep                                                                                                                                            ... The Mytop thread section displays the ID of each thread followed by the user and host. Finally, this section will display the database, idle time, and state or command query. Mytop will allow you to keep an eye on the performance of your MySql database server. Percona toolkit The Percona Toolkit is a very useful set of command-line tools for performing MySQL operations and system tasks. The toolkit can be downloaded from Percona at https://www.percona.com/downloads/percona-toolkit/. The output from these tools can be fed into your monitoring system allowing you to effectively monitor your MyQL installation. Postgres Like MySQL, the Postgres database also has a series of tools, which can be leveraged to monitor database performance. In addition to standard Linux troubleshooting tools, such as top and ps, Postgres also offers its own collection of statistics. The PostgreSQL statistics collector The statistics collector in Postgres allows you to collect data about server activity. The statistics collected in this tool are varied and may be helpful for troubleshooting or system monitoring. In order to leverage the statistics collector, you must turn on the functionality in the postgresql.conf file. The settings are commented out by default in the RUNTIME STATISTICS section of the configuration file. Uncomment the lines in the Query/Index Statistics Collector subsection. #------------------------------------------------------------------------------ # RUNTIME STATISTICS #------------------------------------------------------------------------------   # - Query/Index Statistics Collector -   track_activities = on track_counts = on track_io_timing = off track_functions = none                 # none, pl, all track_activity_query_size = 1024       # (change requires restart) update_process_title = on stats_temp_directory = 'pg_stat_tmp' Once the statistics collector is configured, restart the database server or execute a pg_ctl reload for the configuration to take effect. Once the collector has been configured, there will be a series of views created and named with the prefix “pg_stat”. These views can be queried for relevant statistics in the Posgres database server. Database bckups A diligent operator will ensure that a backup of the database for each OpenStack project is created. Since most OpenStack services make heavy use of the database for persisting things such as states and metadata, a corruption or loss of data can render your OpenStack cloud unusable. The current database backups can help rescue you from this fate. MySQL users can use the mysqldump utility to back up all of the OpenStack datbases. mysqldump --opt --all-databases > all_openstack_dbs.sql Similarly, Postgres users can back up all OpenStack databases with a command similar to the following: pg_dumpall > all_openstack_dbs.sql Your cadence for backups will depend on your environment and tolerance for data corruption of loss. You should store these backups in a safe place and occasional deploy test restores from the data to ensure that they work as expected. Monitoring Monitoring is often your early warning system that something is going wrong in your cluster. Your monitoring system can also be a rich source of information when the time comes to troubleshoot issues with the cluster. There are multiple options available for monitoring OpenStack. Many of your current application monitoring platforms will handle OpenStack just as well as any other Linux system. Regardless of the tool you select to for monitoring, there are several parts of OpenStack that you should focus on. Resource monitoring OpenStack is typically deployed on a series of Linux servers. Monitoring the resources on those servers is essential. A set it and forget it attitude is a recipe for disaster. The things you may want to monitor on your host servers include the following: CPU Disk Memory Log file size Network I/O Database Message broker OpenStack qotas OpenStack operators have the option to set usage quotas for each tenant/project. As an administrator, it is helpful to monitor a project’s usage as it pertains to these quotas. Once users reach a quota, they may not be able to deploy additional resources. Users may misinterpret this as an error in the system and report it to you . By keeping an eye on the quotas, your can proactively warn users as they reach their thresholds or you can decide to increase the quotas as appropriate. Some of the services have client commands that can be used to retrieve quota statistics. As an example, we demonstrate the nova absolute-limits command here: nova absolute-limits +--------------------+------+-------+ | Name               | Used | Max   | +--------------------+------+-------+ | Cores              | 1    | 20    | | FloatingIps        | 0    | 10    | | ImageMeta          | -    | 128   | | Instances          | 1    | 10    | | Keypairs           | -    | 100   | | Personality        | -    | 5     | | Personality Size   | -    | 10240 | | RAM                | 512  | 51200 | | SecurityGroupRules | -    | 20    | | SecurityGroups     | 1    | 10    | | Server Meta        | -    | 128   | | ServerGroupMembers | -    | 10    | | ServerGroups       | 0    | 10    | +--------------------+------+-------+ The absolute-limits command in Nova is nice because it displays the project’s current usage alongside the quota maximum, making it easy to notice that a project/tenant is coming close to the limit. RabbitMQ RabbitMQ is the default message broker used in OpenStack installations. However, if it is installed as is out the box, it can become a single point of failure. Administrators should consider clustering RabbitMQ and activating mirrored queues. Summary OpenStack is the leading open source software for running private clouds. Its popularity has grown exponentially since it was founded by Rackspace and NASA. The output of this engaged community is staggering, resulting in plenty of new features finding their way into OpenStack with each release. The project is at a size now where no one can truly know the details of each service. When working with such a complex project, it is inevitable that you will run into problems, bugs, errors, issues, and plain old trouble. Resources for Article:   Further resources on this subject: Concepts for OpenStack [article] Implementing OpenStack Networking and Security [article] Using the OpenStack Dashboard [article]
Read more
  • 0
  • 0
  • 12568

article-image-understanding-docker
Packt
17 Feb 2016
26 min read
Save for later

Understanding Docker

Packt
17 Feb 2016
26 min read
This article will cover the Docker basics that you should already have a pretty good handle on. But if you don't already have the required knowledge at this point, this article will help give you the basics. (For more resources related to this topic, see here.) In this article, we're going to review the following higher level topics with subtopics in each section: Understanding Docker Docker versus typical VMs The Dockerfile and its function Docker networking/linking Docker installers/installation Types of installers and how they operate Controlling your Docker daemon The Kitematic GUI Docker commands Useful commands for Docker, Docker images, and Docker containers Understanding Docker In this section, we will be covering the structure of Docker and the flow of what happens behind the scenes in this world. We will also take a look at Dockerfile and all the magic it can do. Lastly, in this section, we will look at the Docker networking/linking. Difference between Docker and typical VMs First, we must know what exactly Docker is and does. Docker is a container management system that helps easily manage Linux Containers (LXC) in an easier and universal fashion. This lets you create images in virtual environments on your laptop and run commands or operations against them. The actions you do to the containers that you run in these environments locally on your own machine will be the same commands or operations you run against them when they are running in your production environment. This helps in not having to do things differently when you go from a development environment like that on your local machine to a production environment on your server. Now, let's take a look at the differences between Docker containers and the typical virtual machine environments. In the following illustration, we can see the typical Docker setup on the right-hand side versus the typical VM setup on the left-hand side: This illustration gives us a lot of insight into the biggest key benefit of Docker; and that is its no need for a full operating system every time we need to bring up a new container, which cuts down on the overall size of containers. Docker relies on using the host OS's Linux kernel (since almost all the versions of Linux use the standard kernel models) for the OS it was built upon, such as Red Hat, CentOS, Ubuntu, and so on. For this reason, you can have almost any Linux OS as your host operating system (Ubuntu in the previous illustration) and be able to layer other OSes on top of the host. For example, in the earlier illustration, we could have Red Hat running for one app (the one on the left) and Debian running for the other app (the one on the right), but there would never be a need to actually install Red Hat or Debian on the host. Thus, another benefit of Docker is the size of images when they are born. They are not built with the largest piece: the kernel or the operating system. This makes them incredibly small, compact, and easy to ship. Dockerfile Next, let's take a look at the most important file pertaining to Docker: Dockerfile. Dockerfile is the core file that contains instructions to be performed when an image is built. For example, in an Ubuntu-based system, if you want to install the Apache package, you would first do an apt-get update followed by an apt-get install -y apache2. These would be the type of instructions you would find inside a typical Dockerfile. Items such as commands, calls to other scripts, setting environmental variables, adding files, and setting permissions can all be done via Dockerfile. Dockerfile is also where you specify what image is to be used as your base image for the build. Let's take a look at a very basic Dockerfile and then go over the individual pieces that make one up and what they all do: FROM ubuntu:latest MAINTAINER Scott P. Gallagher <email@somewhere.com> RUN apt-get update && apt-get install -y apache2 ADD 000-default.conf /etc/apache2/sites-available/ RUN chown root:root /etc/apache2/sites-available/000-default.conf EXPOSE 80 CMD ["/usr/sbin/apache2ctl", "-D", "FOREGROUND"] These are the typical items you would find in a basic Dockerfile. The first line states the image we want to start off with when we build the container. In this example, we will be using Ubuntu; the item after the colon can be called if you want a specific version of it. In this case, I am just going to say use the latest version of Ubuntu; but you will also specify trusty, precise, raring, and so on. The second line is the line that is relevant to the maintainer of Dockerfile. In this case, I just have my information in there; well, at least, my name is there. This is for people to contact you if they have any questions or find any errors in your file. Typically, most people just include their name and e-mail address. The next line is a typical line you will see while pulling updates and packages in a Ubuntu environment. You might think they should be separate and wonder why they should be put on the same line separated by &&. Well, in the Dockerfile, it helps by only having to run one process to encompass the entire line. If you were to split it into separate lines, it would have to run one process, finish the process, then start the next process, and finish it. With this, it helps speed up the process by pairing the processes together. They still run one after another, but with more efficiency. The next two lines complement each other. The first adds your custom configurations to the path you specified and changes the owner and group owner to the root user. The EXPOSE line will expose the ports to anything external to the container and to the host it is running on. (This will, by default, expose the container externally beyond the host, unless the firewall is enabled and protecting it.) The last line is the command that is run when the container is launched. This particular command in a Dockerfile should only be used once. If it is used more than once, the last CMD in the Dockerfile will be launched upon the container that is running. This also helps emphasize the one process per container rule. The idea is to spread out the processes so that each process runs in its own container, thus the value of the containers will become more understandable. Essentially, something that runs in the foreground, such as the earlier command to keep the Apache running in the foreground. If we were to use CMD ["service apache2 start"], the container would start and then immediately stop. There is nothing to keep the container running. You can also have other instructions, such as ENV to specify the environmental variables that users can pass upon runtime. These are typically used and are useful while using shell scripts to perform actions such as specifying a database to be created in MySQL or setting permission databases. Docker networking/linking Another important aspect that needs to be understood is how Docker containers are networked or linked together. The way they are networked or linked together highlights another important and large benefit of Docker. When a container is created, it creates a bridge network adapter for which it is assigns an address; it is through these network adapters that the communication flows when you link containers together. Docker doesn't have the need to expose ports to link containers. Let's take a look at it with the help of the following illustration: In the preceding illustration, we can see that the typical VM has to expose ports for others to be able to communicate with each other. This can be dangerous if you don't set up your firewalls or, in this case with MySQL, your MySQL permissions correctly. This can also cause unwanted traffic to the open ports. In the case of Docker, you can link your containers together, so there is no need to expose the ports. This adds security to your setup, as there is now a secure connection between your containers. We've looked at the differences between Docker and typical VMs, as well as the Dockerfile structure and the components that make up the file. We also looked at how Docker containers are linked together for security purposes as opposed to typical VMs. Now, let's review the installers for Docker and the structure behind the installation once they are installed, manipulating them to ensure they are operating correctly. Docker installers/installation Installers are one of the first pieces you need to get up and running with Docker on both your local machine as well as your server environments. Let's first take a look at what environments you can install Docker in: Apple OS X (Mac) Windows Linux (various Linux flavors) Cloud (AWS, DigitalOcean, Microsoft Azure, and so on) Types of installers With the various types of installers listed earlier, there are different ways Docker actually operates on the operating system. Docker natively runs on Linux; so if you are using Linux, then it's pretty straightforward how Docker runs right on your system. However, if you are using Windows or Mac OS X, then it operates a little differently, since it relies on using Linux. With these operating systems, they need Linux in some sort of way, thus enters the virtual machine needed to run the Linux part that Docker operates on, which is called boot2docker. The installers for both Windows and Mac OS X are bundled with the boot2docker package alongside the virtual machine software that, by default, is the Oracle VirtualBox. Now, it is worthwhile to note that Docker recently moved away from offering boot2docker. But, I feel, it is important to understand the boot2docker terms and commands in case you run across anyone running the previous version of the Docker installer. This will help you understand what is going on and move forward to the new installer(s). Currently, they are offering up Docker Toolbox that, like the name implies, includes a lot of items that the installer will install for you. The installers for each OS contain different applications with regards to Docker such as: Docker Toolbox piece Mac OS X Windows Docker Client X X Docker Machine X X Docker Compose X   Docker Kitematic X X VirtualBox X X First, let's take a look at the older style commands of boot2docker. Then, we will take a look at the new commands or application that you can use to achieve these outcomes. Controlling the Docker VM (boot2docker) Now, there are ways to run boot2docker on different VM software. But to start off, VirtualBox is the best and easiest way to operate boot2docker: $ boot2docker Usage: boot2docker [<options>] {help|init|up|ssh|save|down|poweroff|reset|restart|config|status|info|ip|shellinit|delete|download|upgrade|version} [<args>] Now, after we have installed Docker on Linux, OS X, or Windows, how do we go about controlling this virtual machine in the events when we need to start it up, restart it, or even shut it down? This is where the boot2docker command-line parameters come into play. As you can see in the earlier illustration, there are a lot of options you can use for your boot2docker instance. The options you will use mostly are up, down, poweroff, restart, status, ip, upgrade, and version. Some of these commands you will use mostly to troubleshoot items when you are trying to see why the Docker commands might hang, or when you run into any other issues with your boot2docker virtual machine. You can see what each command does by executing the following command: $ boot2docker help The most useful command that I have found while troubleshooting is the boot2docker status command: $ boot2docker status Another useful boot2docker command is: $ boot2docker version This command will help see what version of boot2docker you are currently running. This is helpful in knowing when to use the boot2docker upgrade command. The last command we will look at with respect to boot2docker is the boot2docker ip command. This command is very useful when you need to know what IP address is to be used to access the machines you have been running on a particular host: $ boot2docker ip 192.168.59.103 As you can see, the earlier command gives us the IP address of the boot2docker client running on my OS X machine inside VirtualBox. By using this IP, I can now access the containers I might have been running using the IP address alongside any of the open ports I have exposed. Docker Machine – the new boot2docker So, with boot2docker on its way out, there needs to be a new way to do what boot2docker does. This being said, enter Docker Machine. With Docker Machine, you can do the same things you did with boot2docker, but now in Machine. The following table shows the commands you used in boot2docker and what they are now in Machine: Command boot2docker Docker Machine command boot2docker docker-machine help boot2docker help docker-machine help status boot2docker status docker-machine status version boot2docker version docker-machine sionus i ip boot2docker ip docker-machine ip Kitematic Now that we have covered all the basics of controlling your boot2docker VM, let's take a look at another way you can run Docker containers on your local machine. Let's take a look at Kitematic. Kitematic is a recent addition to the Docker portfolio. Up until now, everything we have done has been command line-based. With Kitematic, you can manage your Docker containers through a GUI. Kitematic can be used either on Windows or OS X, just not on Linux; besides who needs a GUI on Linux anyways! Kitematic, just like boot2docker, operates on a VM defaulting to VirtualBox. Pictures are worth a thousand words, so let's take a look at some screenshots of Kitematic: The previous screenshot depicts what you will see when you launch Kitematic for the first time. After you start running the containers, they will show up on the left-hand side column. You can manipulate and get information about them through the GUI. You can search for prebuilt images on the Docker Hub and click on the CREATE button once you have found the one you want to use or test. In the preceding screenshot, we have created and are running the hello-world-nginx image inside Kitematic. We can now use the STOP, RESTART, and EXEC commands against the container as well as view the settings of the running container. In the following screenshot, we can go to settings and view what ports are exposed from the container to the outside: In the following screenshot, you can see that you can use your login credentials to log in to the Docker Hub and view the repositories you have created and pushed there: The Docker commands We have covered the types of installers and what they can be run on. We have also seen how to control the Docker VM that gets created for you and how to use Kitematic. Let's look at some Docker commands that you should be familiar with already. We will start with some common commands and then take a peek at the commands that are used for the Docker images. We will then take a dive into the commands that are used for the containers. The first command we will be taking a look at will be one of the most useful commands not only in Docker but in any command-line utility you use—the help command. It is run simply by executing the command as follows: $ docker help The earlier command will give you a full list of all the Docker commands at your disposal and a brief description of what each command does. For further help with a particular command, you can run the following: $ docker COMMAND --help You will then receive additional information on using the command, such as the switches, arguments, and descriptions of the arguments. Similar to the boot2docker version command we ran earlier, there is also a version command for the Docker daemon: $ docker version Now, this command will give us a little bit more information than the boot2docker command output, as follows: Client version: 1.7.0 Client API version: 1.19 Go version (client): go1.4.2 Git commit (client): 0baf609 OS/Arch (client): darwin/amd64 Server version: 1.7.0 Server API version: 1.19 Go version (server): go1.4.2 Git commit (server): 0baf609 OS/Arch (server): linux/amd64 This is helpful when you want to see the version of the Docker daemon you may be running to see if you need/want to upgrade. The Docker images Next, let's take a dive into the Docker images. You will learn how to view the images you currently have that you can run, search for images on the Docker Hub, and pull them down to your environment, so you can run them. Let's first take a look at the docker images command. Upon running the command, we will get an output similar to the following output: REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE ubuntu 14.10 ab57dbafeeea 11 days ago 194.5 MB ubuntu trusty 6d4946999d4f 11 days ago 188.3 MB ubuntu latest 6d4946999d4f 11 days ago 188.3 MB Your output will differ based on whether you have any images at all in your Docker environment or upon what images you do have. There are a few important pieces you need to understand from the output you see. Let's go over the columns and what is contained in each. The first column you see is the REPOSITORY column; this column contains the name of the repository as it exists in the Docker Hub. If you were to have a repository that was from someone's user account, it may show up as follows: REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE scottpgallagher/mysql latest 57df9c7989a1 9 weeks ago 321.7 MB The next column, the TAG column, will show you different versions of a repository. As you can see in the preceding example with the Ubuntu repository, there are tag names for the different versions. So, if you want to specify a particular version of a repository in your Dockerfile (as we saw earlier), you are able to. This is useful, so you're not always reliant on having to use the latest version of an operating system and can use the one your application supports the best. It can also help you do backward compatibility testing for your application. The next column is labeled IMAGE ID and it is based on a unique 64 hexadecimal digit string of characters. The image ID simplifies this down to the first 12 digits for easier viewing. Imagine if you had to view all 64 bits on one line! You will learn when to use this unique image ID for later tasks. The last two columns are pretty straightforward; the first being the creation date for the repository, followed by the virtual size of the image. The size is very important as you want to keep or use images that are very small in size if you plan to be moving them around a lot. The smaller the image, the faster is the load time; and who doesn't like it faster? Searching for the Docker images Okay, so let's look at how we can search for the images that are in the Docker Hub using the Docker commands. The command we will be looking at is docker search. With the docker search command, you can search based on the different criteria you are looking for. For example, we can search for all the images with the term ubuntu in them and see what all is available. Here is what we would get back in our results; it would go as follows: $ docker search ubuntu We would get back our results: NAME DESCRIPTION STARS OFFICIAL AUTOMATED ubuntu Ubuntu is a Debian-based Linux operating s... 1835 [OK] ubuntu-upstart Upstart is an event-based replacement for ... 26 [OK] tutum/ubuntu Ubuntu image with SSH access. For the root... 25 [OK] torusware/speedus-ubuntu Always updated official Ubuntu docker imag... 25 [OK] ubuntu-debootstrap debootstrap --variant=minbase --components... 10 [OK] rastasheep/ubuntu-sshd Dockerized SSH service, built on top of of... 4 [OK] maxexcloo/ubuntu Docker base image built on Ubuntu with Sup... 2 [OK] nuagebec/ubuntu Simple always updated Ubuntu docker images... 2 [OK] nimmis/ubuntu This is a docker images different LTS vers... 1 [OK] alsanium/ubuntu Ubuntu Core image for Docker 1 [OK] Based on these results, we can now decipher some information. We can see the name of the repository, a reduced description, how many people have starred and think it is a good repository, whether it's an official repository; which means it's been approved by the Docker team, as well as if it's an automated build. An automated build is typically a Docker image that is built automatically when a Git repository it is linked to is updated. The code gets updated, the web hook is called, and a new Docker image is built in the Docker Hub. If we find an image we want to use, we can simply pull it using its repository name with the docker pull command, as follows: $ docker pull tutum/ubuntu The image will be downloaded and show up in our list when we perform the docker images command we ran earlier. We now know how to search for Docker images and pull them down to our machine. What if we want to get rid of them? That's where the docker rmi command comes into play. With the docker rmi command, you can remove unwanted images from your machine(s). So, let's take look at the images we currently have on our machine with the docker images command. We will get the following: REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE ubuntu 14.10 ab57dbafeeea 11 days ago 194.5 MB ubuntu trusty 6d4946999d4f 11 days ago 188.3 MB ubuntu latest 6d4946999d4f 11 days ago 188.3 MB We can see that we have duplicate images here taking up space. We can see this by looking at the image ID and seeing the exact image ID for both ubuntu:trusty and ubuntu:latest. We now know that ubuntu:trusty is the latest Ubuntu image, so there is no need to keep them both around. Let's free up some space by removing ubuntu:trusty and just keeping ubuntu:latest. We do this by using the docker rmi command, as follows: $ docker rmi ubuntu:trusty If you issue the docker images command now, you will see that ubuntu:trusty no longer shows up in your images list and has been removed. Now, you can remove machines based on their image ID as well. But be careful while you do so; in this scenario, not only will you remove ubuntu:trusty, but you will also remove ubuntu:latest as they have the same image ID. Manipulating the Docker images We have gone over the images and know how to obtain and manipulate them in some ways. Next, we are going to take a look at what it takes to fire them up and manipulate them. This is the part where the images become containers! Let's first go over the basics of the docker run command and how to run containers. We will cover some basic docker run items in this article. So, let's just look at how to get images up, running, and turned into containers. The most basic way to run a container is as follows: $ docker run -i -t <image_name>:<tag> /bin/bash Upon closer inspection of the earlier command, we start off with the docker run command, followed by two switches: -i and -t. The -i gives us an interactive shell into the running container, the -t will allocate a pseudo-tty that, while using interactive processes, must be used together with the -i switch. You can also use switches together; for example, -it is commonly used for these two switches. This will help you test the container to see how it operates before running it as a daemon. Once you are comfortable with your container, you can test how it operates in the daemon mode: $ docker run -d <image_name>:<tag> If the container is set up correctly and has an entry point setup, you should be able to see the running container by issuing the docker ps command. You will see something similar to the following: $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES cc1fefcfa098 ubuntu:14.10 "/bin/bash" 3 seconds ago Up 3 seconds boring_mccarthy Based on the earlier command, we get a lot of other important information indicating that the container is running. We can see the container ID, the image name that is running, the command that is running to keep the image alive, when the container started, its current status, if any ports were exposed they would be listed here, as well as the name given to the container. Now, these names are random, unless it is specified otherwise by the --name= switch. You can also the expose the ports on your containers by using the -p switch as follows: $ docker run -d -p <host_port>:<container_port> <image>:<tag> $ docker run -d -p 8080:80 ubuntu:14.10 This will run the ubuntu 14.10 container in the demonized mode, exposing port 8080 on the Docker host to port 80 on the running container: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 55cfdcb6beb6 ubuntu:14.10 "/bin/bash" 2 seconds ago Up 2 seconds 0.0.0.0:8080->80/tcp babbage Now, there will come a time when containers don't want to behave. For this, you can see the issues you have by using the docker logs command. The command is very straightforward. You specify the container you want to see the logs off. For this command, you need to use the container ID or the name of the container from the docker ps output: $ docker logs 55cfdcb6beb6 Or: $ docker logs babbage You can also get this ID when you first initiate the docker run command: $ docker run -d ubuntu:14.10 /bin/bash da92261485db98c7463fffadb43e3f684ea9f47949f287f92408fd0f3e4f2bad Stopping containers Now, let's take a look at how we can stop these containers. For various reasons, we would want to do this. There are a few commands we could use; they are docker kill, docker stop, docker pause, and docker unpause. Let's cover them briefly as they are fairly straightforward. First, let's look at the difference between docker kill and docker stop. The docker kill command will do just that—kill the container immediately. For a graceful shutdown of the container, you would want to use the docker stop command. Mostly, when you are testing, you will be using docker kill. When you're in your production environments, you will want to use docker stop to ensure you don't corrupt any data you might have in the Docker volumes. The commands are used exactly like the docker logs command, where you can use the container ID, the random name given to the container, or the one you might specify with the --name= switch. Now, let's take a dive into how we can execute some commands, view information on our running containers, and manipulate them in a small sense. The first thing we want to take a look at, which will make things a little easier with the upcoming commands, is the docker rename command. With the docker rename command, we can change the name that has been randomly generated for the container. When we performed the docker run command, a random name was assigned to our container; most times, these names are fine. But if you are looking for an easy way to manage the containers, a name can be sometimes easier to remember. For this, you can use the docker rename command as follows: $ docker rename <current_container_name> <new_container_name> Now that we have an easily recognizable and rememberable name, let's take a peek inside our containers with the docker stats and docker top commands, taking them in order: $ docker stats <container_name> CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O web1 0.00% 1.016 MB/2.099 GB 0.05% 0 B/0 B The other command docker top provides a list of all running processes inside the container. Again, we can use the name of the container to pull the information: $ docker top <container_name> We will receive an output similar to the following one based on what processes are running inside the container: UID PID PPID C STIME TTY TIME CMD root 8057 1380 0 13:02 pts/0 00:00:00 /bin/bash We can see who is running the process (in this case, the root user), the command being run (in this case, /bin/bash), as well as the other information that might be useful. Lastly, let's cover how we can remove the containers. The same way we looked at removing images earlier with the docker rmi command, we can use the docker rm command to remove unwanted containers. This is useful if you want to reuse a name you provided to a container: $ docker rm <container_name> Summary In this article, we have gone over the basics of what Docker is and how it is compared to typical virtual machines. We looked at the Dockerfile structure and the networking and linking of containers. We went over the installers, how they operate on different operating systems, and how to control them through the command line. We briefly looked at the latest Docker addition Kitematic for those interested in a GUI version for Windows or OS X. Then, we took a small but deep dive into the basic Docker commands to get you started. Resources for Article: Further resources on this subject: Introduction to Docker[article] Docker in Production[article] Speeding Vagrant Development With Docker[article]
Read more
  • 0
  • 0
  • 50741

article-image-developing-basic-site-nodejs-and-express
Packt
17 Feb 2016
21 min read
Save for later

Developing a Basic Site with Node.js and Express

Packt
17 Feb 2016
21 min read
In this article, we will continue with the Express framework. It's one of the most popular frameworks available and is certainly a pioneering one. Express is still widely used and several developers use it as a starting point. (For more resources related to this topic, see here.) Getting acquainted with Express Express (http://expressjs.com/) is a web application framework for Node.js. It is built on top of Connect (http://www.senchalabs.org/connect/), which means that it implements middleware architecture. In the previous chapter, when exploring Node.js, we discovered the benefit of such a design decision: the framework acts as a plugin system. Thus, we can say that Express is suitable for not only simple but also complex applications because of its architecture. We may use only some of the popular types of middleware or add a lot of features and still keep the application modular. In general, most projects in Node.js perform two functions: run a server that listens on a specific port, and process incoming requests. Express is a wrapper for these two functionalities. The following is basic code that runs the server: var http = require('http'); http.createServer(function (req, res) { res.writeHead(200, {'Content-Type': 'text/plain'}); res.end('Hello Worldn'); }).listen(1337, '127.0.0.1'); console.log('Server running at http://127.0.0.1:1337/'); var http = require('http'); http.createServer(function (req, res) { res.writeHead(200, {'Content-Type': 'text/plain'}); res.end('Hello Worldn'); }).listen(1337, '127.0.0.1'); console.log('Server running at http://127.0.0.1:1337/'); This is an example extracted from the official documentation of Node.js. As shown, we use the native module http and run a server on the port 1337. There is also a request handler function, which simply sends the Hello world string to the browser. Now, let's implement the same thing but with the Express framework, using the following code: var express = require('express'); var app = express(); app.get("/", function(req, res, next) { res.send("Hello world"); }).listen(1337); console.log('Server running at http://127.0.0.1:1337/'); It's pretty much the same thing. However, we don't need to specify the response headers or add a new line at the end of the string because the framework does it for us. In addition, we have a bunch of middleware available, which will help us process the requests easily. Express is like a toolbox. We have a lot of tools to do the boring stuff, allowing us to focus on the application's logic and content. That's what Express is built for: saving time for the developer by providing ready-to-use functionalities. Installing Express There are two ways to install Express. We'll will start with the simple one and then proceed to the more advanced technique. The simpler approach generates a template, which we may use to start writing the business logic directly. In some cases, this can save us time. From another viewpoint, if we are developing a custom application, we need to use custom settings. We can also use the boilerplate, which we get with the advanced technique; however, it may not work for us. Using package.json Express is like every other module. It has its own place in the packages register. If we want to use it, we need to add the framework in the package.json file. The ecosystem of Node.js is built on top of the Node Package Manager. It uses the JSON file to find out what we need and installs it in the current directory. So, the content of our package.json file looks like the following code: { "name": "projectname", "description": "description", "version": "0.0.1", "dependencies": { "express": "3.x" } } These are the required fields that we have to add. To be more accurate, we have to say that the mandatory fields are name and version. However, it is always good to add descriptions to our modules, particularly if we want to publish our work in the registry, where such information is extremely important. Otherwise, the other developers will not know what our library is doing. Of course, there are a bunch of other fields, such as contributors, keywords, or development dependencies, but we will stick to limited options so that we can focus on Express. Once we have our package.json file placed in the project's folder, we have to call npm install in the console. By doing so, the package manager will create a node_modules folder and will store Express and its dependencies there. At the end of the command's execution, we will see something like the following screenshot: The first line shows us the installed version, and the proceeding lines are actually modules that Express depends on. Now, we are ready to use Express. If we type require('express'), Node.js will start looking for that library inside the local node_modules directory. Since we are not using absolute paths, this is normal behavior. If we miss running the npm install command, we will be prompted with Error: Cannot find module 'express'. Using a command-line tool There is a command-line instrument called express-generator. Once we run npm install -g express-generator, we will install and use it as every other command in our terminal. If you use the framework inseveral projects, you will notice that some things are repeated. We can even copy and paste them from one application to another, and this is perfectly fine. We may even end up with our own boiler plate and can always start from there. The command-line version of Express does the same thing. It accepts few arguments and based on them, creates a skeleton for use. This can be very handy in some cases and will definitely save some time. Let's have a look at the available arguments: -h, --help: This signifies output usage information. -V, --version: This shows the version of Express. -e, --ejs: This argument adds the EJS template engine support. Normally, we need a library to deal with our templates. Writing pure HTML is not very practical. The default engine is set to JADE. -H, --hogan: This argument is Hogan-enabled (another template engine). -c, --css: If wewant to use the CSS preprocessors, this option lets us use LESS(short forLeaner CSS) or Stylus. The default is plain CSS. -f, --force: This forces Express to operate on a nonempty directory. Let's try to generate an Express application skeleton with LESS as a CSS preprocessor. We use the following line of command: express --css less myapp A new myapp folder is created with the file structure, as seen in the following screenshot: We still need to install the dependencies, so cd myapp && npm install is required. We will skip the explanation of the generated directories for now and will move to the created app.js file. It starts with initializing the module dependencies, as follows: var express = require('express'); var path = require('path'); var favicon = require('static-favicon'); var logger = require('morgan'); var cookieParser = require('cookie-parser'); var bodyParser = require('body-parser'); var routes = require('./routes/index'); var users = require('./routes/users'); var app = express(); Our framework is express, and path is a native Node.js module. The middleware are favicon, logger, cookieParser, and bodyParser. The routes and users are custom-made modules, placed in local for the project folders. Similarly, as in the Model-View-Controller(MVC) pattern, these are the controllers for our application. Immediately after, an app variable is created; this represents the Express library. We use this variable to configure our application. The script continues by setting some key-value pairs. The next code snippet defines the path to our views and the default template engine: app.set('views', path.join(__dirname, 'views')); app.set('view engine', 'jade'); The framework uses the methods set and get to define the internal properties. In fact, we may use these methods to define our own variables. If the value is a Boolean, we can replace set and get with enable and disable. For example, see the following code: app.set('color', 'red'); app.get('color'); // red app.enable('isAvailable'); The next code adds middleware to the framework. Wecan see the code as follows: app.use(favicon()); app.use(logger('dev')); app.use(bodyParser.json()); app.use(bodyParser.urlencoded()); app.use(cookieParser()); app.use(require('less-middleware')({ src: path.join(__dirname, 'public') })); app.use(express.static(path.join(__dirname, 'public'))); The first middleware serves as the favicon of our application. The second is responsible for the output in the console. If we remove it, we will not get information about the incoming requests to our server. The following is a simple output produced by logger: GET / 200 554ms - 170b GET /stylesheets/style.css 200 18ms - 110b The json and urlencoded middleware are related to the data sent along with the request. We need them because they convert the information in an easy-to-use format. There is also a middleware for the cookies. It populates the request object, so we later have access to the required data. The generated app uses LESS as a CSS preprocessor, and we need to configure it by setting the directory containing the .less files. Eventually, we define our static resources, which should be delivered by the server. These are just few lines, but we've configured the whole application. We may remove or replace some of the modules, and the others will continue working. The next code in the file maps two defined routes to two different handlers, as follows: app.use('/', routes); app.use('/users', users); If the user tries to open a missing page, Express still processes the request by forwarding it to the error handler, as follows: app.use(function(req, res, next) { var err = new Error('Not Found'); err.status = 404; next(err); }); The framework suggests two types of error handling:one for the development environment and another for the production server. The difference is that the second one hides the stack trace of the error, which should be visible only for the developers of the application. As we can see in the following code, we are checking the value of the env property and handling the error differently: // development error handler if (app.get('env') === 'development') { app.use(function(err, req, res, next) { res.status(err.status || 500); res.render('error', { message: err.message, error: err }); }); } // production error handler app.use(function(err, req, res, next) { res.status(err.status || 500); res.render('error', { message: err.message, error: {} }); }); At the end, the app.js file exports the created Express instance, as follows: module.exports = app; To run the application, we need to execute node ./bin/www. The code requires app.js and starts the server, which by default listens on port 3000. #!/usr/bin/env node var debug = require('debug')('my-application'); var app = require('../app'); app.set('port', process.env.PORT || 3000); var server = app.listen(app.get('port'), function() { debug('Express server listening on port ' + server.address().port); }); The process.env declaration provides an access to variables defined in the current development environment. If there is no PORT setting, Express uses 3000 as the value. The required debug module uses a similar approach to find out whether it has to show messages to the console. Managing routes The input of our application is the routes. The user visits our page at a specific URL and we have to map this URL to a specific logic. In the context of Express, this can be done easily, as follows: var controller = function(req, res, next) { res.send("response"); } app.get('/example/url', controller); We even have control over the HTTP's method, that is, we are able to catch POST, PUT, or DELETE requests. This is very handy if we want to retain the address path but apply a different logic. For example, see the following code: var getUsers = function(req, res, next) { // ... } var createUser = function(req, res, next) { // ... } app.get('/users', getUsers); app.post('/users', createUser); The path is still the same, /users, but if we make a POST request to that URL, the application will try to create a new user. Otherwise, if the method is GET, it will return a list of all the registered members. There is also a method, app.all, which we can use to handle all the method types at once. We can see this method in the following code snippet: app.all('/', serverHomePage); There is something interesting about the routing in Express. We may pass not just one but many handlers. This means that we can create a chain of functions that correspond to one URL. For example, it we need to know if the user is logged in, there is a module for that. We can add another method that validates the current user and attaches a variable to the request object, as follows: var isUserLogged = function(req, res, next) { req.userLogged = Validator.isCurrentUserLogged(); next(); } var getUser = function(req, res, next) { if(req.userLogged) { res.send("You are logged in. Hello!"); } else { res.send("Please log in first."); } } app.get('/user', isUserLogged, getUser); The Validator class is a class that checks the current user's session. The idea is simple: we add another handler, which acts as an additional middleware. After performing the necessary actions, we call the next function, which passes the flow to the next handler, getUser. Because the request and response objects are the same for all the middlewares, we have access to the userLogged variable. This is what makes Express really flexible. There are a lot of great features available, but they are optional. At the end of this chapter, we will make a simple website that implements the same logic. Handling dynamic URLs and the HTML forms The Express framework also supports dynamic URLs. Let's say we have a separate page for every user in our system. The address to those pages looks like the following code: /user/45/profile Here, 45 is the unique number of the user in our database. It's of course normal to use one route handler for this functionality. We can't really define different functions for every user. The problem can be solved by using the following syntax: var getUser = function(req, res, next) { res.send("Show user with id = " + req.params.id); } app.get('/user/:id/profile', getUser); The route is actually like a regular expression with variables inside. Later, that variable is accessible in the req.params object. We can have more than one variable. Here is a slightly more complex example: var getUser = function(req, res, next) { var userId = req.params.id; var actionToPerform = req.params.action; res.send("User (" + userId + "): " + actionToPerform) } app.get('/user/:id/profile/:action', getUser); If we open http://localhost:3000/user/451/profile/edit, we see User (451): edit as a response. This is how we can get a nice looking, SEO-friendly URL. Of course, sometimes we need to pass data via the GET or POST parameters. We may have a request like http://localhost:3000/user?action=edit. To parse it easily, we need to use the native url module, which has few helper functions to parse URLs: var getUser = function(req, res, next) { var url = require('url'); var url_parts = url.parse(req.url, true); var query = url_parts.query; res.send("User: " + query.action); } app.get('/user', getUser); Once the module parses the given URL, our GET parameters are stored in the .query object. The POST variables are a bit different. We need a new middleware to handle that. Thankfully, Express has one, which is as follows: app.use(express.bodyParser()); var getUser = function(req, res, next) { res.send("User: " + req.body.action); } app.post('/user', getUser); The express.bodyParser() middleware populates the req.body object with the POST data. Of course, we have to change the HTTP method from .get to .post or .all. If we want to read cookies in Express, we may use the cookieParser middleware. Similar to the body parser, it should also be installed and added to the package.json file. The following example sets the middleware and demonstrates its usage: var cookieParser = require('cookie-parser'); app.use(cookieParser('optional secret string')); app.get('/', function(req, res, next){ var prop = req.cookies.propName }); Returning a response Our server accepts requests, does some stuff, and finally, sends the response to the client's browser. This can be HTML, JSON, XML, or binary data, among others. As we know, by default, every middleware in Express accepts two objects, request and response. The response object has methods that we can use to send an answer to the client. Every response should have a proper content type or length. Express simplifies the process by providing functions to set HTTP headers and sending content to the browser. In most cases, we will use the .send method, as follows: res.send("simple text"); When we pass a string, the framework sets the Content-Type header to text/html. It's great to know that if we pass an object or array, the content type is application/json. If we develop an API, the response status code is probably going to be important for us. With Express, we are able to set it like in the following code snippet: res.send(404, 'Sorry, we cannot find that!'); It's even possible to respond with a file from our hard disk. If we don't use the framework, we will need to read the file, set the correct HTTP headers, and send the content. However, Express offers the .sendfile method, which wraps all these operations as follows: res.sendfile(__dirname + "/images/photo.jpg"); Again, the content type is set automatically; this time it is based on the filename's extension. When building websites or applications with a user interface, we normally need to serve an HTML. Sure, we can write it manually in JavaScript, but it's good practice to use a template engine. This means we save everything in external files and the engine reads the markup from there. It populates them with some data and, at the end, provides ready-to-show content. In Express, the whole process is summarized in one method, .render. However, to work properly, we have to instruct the framework regarding which template engine to use. We already talked about this in the beginning of this chapter. The following two lines of code, set the path to our views and the template engine: app.set('views', path.join(__dirname, 'views')); app.set('view engine', 'jade'); Let's say we have the following template ( /views/index.jade ): h1= title p Welcome to #{title} Express provides a method to serve templates. It accepts the path to the template, the data to be applied, and a callback. To render the previous template, we should use the following code: res.render("index", {title: "Page title here"}); The HTML produced looks as follows: <h1>Page title here</h1><p>Welcome to Page title here</p> If we pass a third parameter, function, we will have access to the generated HTML. However, it will not be sent as a response to the browser. The example-logging system We've seen the main features of Express. Now let's build something real. The next few pages present a simple website where users can read only if they are logged in. Let's start and set up the application. We are going to use Express' command-line instrument. It should be installed using npm install -g express-generator. We create a new folder for the example, navigate to it via the terminal, and execute express --css less site. A new directory, site, will be created. If we go there and run npm install, Express will download all the required dependencies. As we saw earlier, by default, we have two routes and two controllers. To simplify the example, we will use only the first one: app.use('/', routes). Let's change the views/index.jade file content to the following HTML code: doctype html html head title= title link(rel='stylesheet', href='/stylesheets/style.css') body h1= title hr p That's a simple application using Express. Now, if we run node ./bin/www and open http://127.0.0.1:3000, we will see the page. Jade uses indentation to parse our template. So, we should not mix tabs and spaces. Otherwise, we will get an error. Next, we need to protect our content. We check whether the current user has a session created; if not, a login form is shown. It's the perfect time to create a new middleware. To use sessions in Express, install an additional module: express-session. We need to open our package.json file and add the following line of code: "express-session": "~1.0.0" Once we do that, a quick run of npm install will bring the module to our application. All we have to do is use it. The following code goes to app.js: var session = require('express-session'); app.use(session({ secret: 'app', cookie: { maxAge: 60000 }})); var verifyUser = function(req, res, next) { if(req.session.loggedIn) { next(); } else { res.send("show login form"); } } app.use('/', verifyUser, routes); Note that we changed the original app.use('/', routes) line. The session middleware is initialized and added to Express. The verifyUser function is called before the page rendering. It uses the req.session object, and checks whether there is a loggedIn variable defined and if its value is true. If we run the script again, we will see that the show login form text is shown for every request. It's like this because no code sets the session exactly the way we want it. We need a form where users can type their username and password. We will process the result of the form and if the credentials are correct, the loggedIn variable will be set to true. Let's create a new Jade template, /views/login.jade: doctype html html head title= title link(rel='stylesheet', href='/stylesheets/style.css') body h1= title hr form(method='post') label Username: br input(type='text', name='username') br label Password: br input(type='password', name='password') br input(type='submit') Instead of sending just a text with res.send("show login form"); we should render the new template, as follows: res.render("login", {title: "Please log in."}); We choose POST as the method for the form. So, we need to add the middleware that populates the req.body object with the user's data, as follows: app.use(bodyParser()); Process the submitted username and password as follows: var verifyUser = function(req, res, next) { if(req.session.loggedIn) { next(); } else { var username = "admin", password = "admin"; if(req.body.username === username && req.body.password === password) { req.session.loggedIn = true; res.redirect('/'); } else { res.render("login", {title: "Please log in."}); } } } The valid credentials are set to admin/admin. In a real application, we may need to access a database or get this information from another place. It's not really a good idea to place the username and password in the code; however, for our little experiment, it is fine. The previous code checks whether the passed data matches our predefined values. If everything is correct, it sets the session, after which the user is forwarded to the home page. Once you log in, you should be able to log out. Let's add a link for that just after the content on the index page (views/index.jade ): a(href='/logout') logout Once users clicks on this link, they will be forward to a new page. We just need to create a handler for the new route, remove the session, and forward them to the index page where the login form is reflected. Here is what our logging out handler looks like: // in app.js var logout = function(req, res, next) { req.session.loggedIn = false; res.redirect('/'); } app.all('/logout', logout); Setting loggedIn to false is enough to make the session invalid. The redirect sends users to the same content page they came from. However, this time, the content is hidden and the login form pops up. Summary In this article, we learned about one of most widely used Node.js frameworks, Express. We discussed its fundamentals, how to set it up, and its main characteristics. The middleware architecture, which we mentioned in the previous chapter, is the base of the library and gives us the power to write complex but, at the same time, flexible applications. The example we used was a simple one. We required a valid session to provide page access. However, it illustrates the usage of the body parser middleware and the process of registering the new routes. We also updated the Jade templates and saw the results in the browser. For more information on Node.js Refer to the following URLs: https://www.packtpub.com/web-development/instant-nodejs-starter-instant https://www.packtpub.com/web-development/learning-nodejs-net-developers https://www.packtpub.com/web-development/nodejs-essentials Resources for Article: Further resources on this subject: Writing a Blog Application with Node.js and AngularJS [article] Testing in Node and Hapi [article] Learning Node.js for Mobile Application Development [article]
Read more
  • 0
  • 0
  • 14618
article-image-create-your-first-react-element
Packt
17 Feb 2016
22 min read
Save for later

Create Your First React Element

Packt
17 Feb 2016
22 min read
From the 7th to the 13th of November 2016, you can save up to 80% on some of our top ReactJS content - so what are you waiting for? Dive in here before the week ends! As many of you know, creating a simple web application today involves writing the HTML, CSS, and JavaScript code. The reason we use three different technologies is because we want to separate three different concerns: Content (HTML) Styling (CSS) Logic (JavaScript) (For more resources related to this topic, see here.) This separation works great for creating a web page because, traditionally, we had different people working on different parts of our web page: one person structured the content using HTML and styled it using CSS, and then another person implemented the dynamic behavior of various elements on that web page using JavaScript. It was a content-centric approach. Today, we mostly don't think of a website as a collection of web pages anymore. Instead, we build web applications that might have only one web page, and that web page does not represent the layout for our content—it represents a container for our web application. Such a web application with a single web page is called (unsurprisingly) a Single Page Application (SPA). You might be wondering, how do we represent the rest of the content in a SPA? Surely, we need to create an additional layout using HTML tags? Otherwise, how does a web browser know what to render? These are all valid questions. Let's take a look at how it works in this article. Once you load your web page in a web browser, it creates a Document Object Model (DOM) of that web page. A DOM represents your web page in a tree structure, and at this point, it reflects the structure of the layout that you created with only HTML tags. This is what happens regardless of whether you're building a traditional web page or a SPA. The difference between the two is what happens next. If you are building a traditional web page, then you would finish creating your web page's layout. On the other hand, if you are building a SPA, then you would need to start creating additional elements by manipulating the DOM with JavaScript. A web browser provides you with the JavaScript DOM API to do this. You can learn more about it at https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model. However, manipulating (or mutating) the DOM with JavaScript has two issues: Your programming style will be imperative if you decide to use the JavaScript DOM API directly. This programming style leads to a code base that is harder to maintain. DOM mutations are slow because they cannot be optimized for speed, unlike other JavaScript code. Luckily, React solves both these problems for us. Understanding virtual DOM Why do we need to manipulate the DOM in the first place? Because our web applications are not static. They have a state represented by the user interface (UI) that a web browser renders, and that state can be changed when an event occurs. What kind of events are we talking about? There are two types of events that we're interested in: User events: When a user types, clicks, scrolls, resizes, and so on Server events: When an application receives data or an error from a server, among others What happens while handling these events? Usually, we update the data that our application depends on, and that data represents a state of our data model. In turn, when a state of our data model changes, we might want to reflect this change by updating a state of our UI. Looks like what we want is a way of syncing two different states: the UI state and the data model state. We want one to react to the changes in  the other and vice versa. How can we achieve this? One of the ways to sync your application's UI state with an underlying data model's state is two-way data binding. There are different types of two-way data binding. One of them is key-value observing (KVO), which is used in Ember.js, Knockout, Backbone, and iOS, among others. Another one is dirty checking, which is used in Angular. Instead of two-way data binding, React offers a different solution called the virtual DOM. The virtual DOM is a fast, in-memory representation of the real DOM, and it's an abstraction that allows us to treat JavaScript and DOM as if they were reactive. Let's take a look at how it works: Whenever the state of your data model changes, the virtual DOM and React will rerender your UI to a virtual DOM representation. React then calculates the difference between the two virtual DOM representations: the previous virtual DOM representation that was computed before the data was changed and the current virtual DOM representation that was computed after the data was changed. This difference between the two virtual DOM representations is what actually needs to be changed in the real DOM. React updates only what needs to be updated in the real DOM. The process of finding a difference between the two representations of the virtual DOM and rerendering only the updated patches in a real DOM is fast. Also, the best part is, as a React developer, that you don't need to worry about what actually needs to be rerendered. React allows you to write your code as if you were rerendering the entire DOM every time your application's state changes. If you would like to learn more about the virtual DOM, the rationale behind it, and how it can be compared to data binding, then I would strongly recommend that you watch this very informative talk by Pete Hunt from Facebook at https://www.youtube.com/watch?v=-DX3vJiqxm4. Now that we've learnt about the virtual DOM, let's mutate a real DOM by installing React and creating our first React element. Installing React To start using the React library, we need to first install it. I am going to show you two ways of doing this: the simplest one and the one using the npm install command. The simplest way is to add the <script> tag to our ~/snapterest/build/index.html file: For the development version of React, add the following command: <script src="https://cdnjs.cloudflare.com/ajax/libs/react/0.14.0-beta3/react.js"></script> For the production version version of React, add the following command: <script src="https://cdnjs.cloudflare.com/ajax/libs/react/0.14.0-beta3/react.min.js"></script> For our project, we'll be using the development version of React. At the time of writing, the latest version of React library is 0.14.0-beta3. Over time, React gets updated, so make sure you use the latest version that is available to you, unless it introduces breaking changes that are incompatible with the code samples provided in this article. Visit https://github.com/fedosejev/react-essentials to learn about any compatibility issues between the code samples and the latest version of React. We all know that Browserify allows us to import all the dependency modules for our application using the require() function. We'll be using require() to import the React library as well, which means that, instead of adding a <script> tag to our index.html, we'll be using the npm install command to install React: Navigate to the ~/snapterest/ directory and run this command:  npm install --save react@0.14.0-beta3 react-dom@0.14.0-beta3 Then, open the ~/snapterest/source/app.js file in your text editor and import the React and ReactDOM libraries to the React and ReactDOM variables, respectively: var React = require('react'); var ReactDOM = require('react-dom'); The react package contains methods that are concerned with the key idea behind React, that is, describing what you want to render in a declarative way. On the other hand, the react-dom package offers methods that are responsible for rendering to the DOM. You can read more about why developers at Facebook think it's a good idea to separate the React library into two packages at https://facebook.github.io/react/blog/2015/07/03/react-v0.14-beta-1.html#two-packages. Now we're ready to start using the React library in our project. Next, let's create our first React Element! Creating React Elements with JavaScript We'll start by familiarizing ourselves with a fundamental React terminology. It will help us build a clear picture of what the React library is made of. This terminology will most likely update over time, so keep an eye on the official documentation at http://facebook.github.io/react/docs/glossary.html. Just like the DOM is a tree of nodes, React's virtual DOM is a tree of React nodes. One of the core types in React is called ReactNode. It's a building block for a virtual DOM, and it can be any one of these core types: ReactElement: This is the primary type in React. It's a light, stateless, immutable, virtual representation of a DOM Element. ReactText: This is a string or a number. It represents textual content and it's a virtual representation of a Text Node in the DOM. ReactElements and ReactTexts are ReactNodes. An array of ReactNodes is called a ReactFragment. You will see examples of all of these in this article. Let's start with an example of a ReactElement: Add the following code to your ~/snapterest/source/app.js file: var reactElement = React.createElement('h1'); ReactDOM.render(reactElement, document.getElementById('react-application')); Now your app.js file should look exactly like this: var React = require('react'); var ReactDOM = require('react-dom'); var reactElement = React.createElement('h1'); ReactDOM.render(reactElement, document.getElementById('react-application')); Navigate to the ~/snapterest/ directory and run Gulp's default task: gulp You will see the following output: Starting 'default'... Finished 'default' after 1.73 s Navigate to the ~/snapterest/build/ directory, and open index.html in a web browser. You will see a blank web page. Open Developer Tools in your web browser and inspect the HTML markup for your blank web page. You should see this line, among others: <h1 data-reactid=".0"></h1> Well done! We've just created your first React element. Let's see exactly how we did it. The entry point to the React library is the React object. This object has a method called createElement() that takes three parameters: type, props, and children: React.createElement(type, props, children); Let's take a look at each parameter in more detail. The type parameter The type parameter can be either a string or a ReactClass: A string could be an HTML tag name such as 'div', 'p', 'h1', and so on. React supports all the common HTML tags and attributes. For a complete list of HTML tags and attributes supported by React, you can refer to http://facebook.github.io/react/docs/tags-and-attributes.html. A ReactClass is created via the React.createClass() method. The type parameter describes how an HTML tag or a ReactClass is going to be rendered. In our example, we're rendering the h1 HTML tag. The props parameter The props parameter is a JavaScript object passed from a parent element to a child element (and not the other way around) with some properties that are considered immutable, that is, those that should not be changed. While creating DOM elements with React, we can pass the props object with properties that represent the HTML attributes such as class, style, and so on. For example, run the following commands: var React = require('react'); var ReactDOM = require('react-dom'); var reactElement = React.createElement('h1', { className: 'header' }); ReactDOM.render(reactElement, document.getElementById('react-application')); The preceding code will create an h1 HTML element with a class attribute set to header: <h1 class="header" data-reactid=".0"></h1> Notice that we name our property className rather than class. The reason is that the class keyword is reserved in JavaScript. If you use class as a property name, it will be ignored by React, and a helpful warning message will be printed on the web browser's console: Warning: Unknown DOM property class. Did you mean className?Use className instead. You might be wondering what this data-reactid=".0" attribute is doing in our h1 tag? We didn't pass it to our props object, so where did it come from? It is added and used by React to track the DOM nodes; it might be removed in a future version of React. The children parameter The children parameter describes what child elements this element should have, if any. A child element can be any type of ReactNode: a virtual DOM element represented by a ReactElement, a string or a number represented by a ReactText, or an array of other ReactNodes, which is also called ReactFragment. Let's take a look at this example: var React = require('react'); var ReactDOM = require('react-dom'); var reactElement = React.createElement('h1', { className: 'header' }, 'This is React'); ReactDOM.render(reactElement, document.getElementById('react-application')); The following code will create an h1 HTML element with a class attribute and a text node, This is React: <h1 class="header" data-reactid=".0">This is React</h1> The h1 tag is represented by a ReactElement, while the This is React string is represented by a ReactText. Next, let's create a React element with a number of other React elements as it's children: var React = require('react'); var ReactDOM = require('react-dom');   var h1 = React.createElement('h1', { className: 'header', key: 'header' }, 'This is React'); var p = React.createElement('p', { className: 'content', key: 'content' }, "And that's how it works."); var reactFragment = [ h1, p ]; var section = React.createElement('section', { className: 'container' }, reactFragment);   ReactDOM.render(section, document.getElementById('react-application')); We've created three React elements: h1, p, and section. h1 and p both have child text nodes, "This is React" and "And that's how it works.", respectively. The section has a child that is an array of two ReactElements, h1 and p, called reactFragment. This is also an array of ReactNodes. Each ReactElement in the reactFragment array must have a key property that helps React to identify that ReactElement. As a result, we get the following HTML markup: <section class="container" data-reactid=".0">   <h1 class="header" data-reactid=".0.$header">This is React</h1>   <p class="content" data-reactid=".0.$content">And that's how it works.</p> </section> Now we understand how to create React elements. What if we want to create a number of React elements of the same type? Does it mean that we need to call React.createElement('type') over and over again for each element of the same type? We can, but we don't need to because React provides us with a factory function called React.createFactory(). A factory function is a function that creates other functions. This is exactly what React.createFactory(type) does: it creates a function that produces a ReactElement of a given type. Consider the following example: var React = require('react'); var ReactDOM = require('react-dom');   var listItemElement1 = React.createElement('li', { className: 'item-1', key: 'item-1' }, 'Item 1'); var listItemElement2 = React.createElement('li', { className: 'item-2', key: 'item-2' }, 'Item 2'); var listItemElement3 = React.createElement('li', { className: 'item-3', key: 'item-3' }, 'Item 3');   var reactFragment = [ listItemElement1, listItemElement2, listItemElement3 ]; var listOfItems = React.createElement('ul', { className: 'list-of-items' }, reactFragment);   ReactDOM.render(listOfItems, document.getElementById('react-application')); The preceding example produces this HTML: <ul class="list-of-items" data-reactid=".0">   <li class="item-1" data-reactid=".0.$item-1">Item 1</li>   <li class="item-2" data-reactid=".0.$item-2">Item 2</li>   <li class="item-3" data-reactid=".0.$item-3">Item 3</li> </ul> We can simplify it by first creating a factory function: var React = require('react'); var ReactDOM = require('react-dom'); var createListItemElement = React.createFactory('li'); var listItemElement1 = createListItemElement({ className: 'item-1', key: 'item-1' }, 'Item 1'); var listItemElement2 = createListItemElement({ className: 'item-2', key: 'item-2' }, 'Item 2'); var listItemElement3 = createListItemElement({ className: 'item-3', key: 'item-3' }, 'Item 3'); var reactFragment = [ listItemElement1, listItemElement2, listItemElement3 ]; var listOfItems = React.createElement('ul', { className: 'list-of-items' }, reactFragment); ReactDOM.render(listOfItems, document.getElementById('react-application')); In the preceding example, we're first calling the React.createFactory() function and passing a li HTML tag name as a type parameter. Then, the React.createFactory() function returns a new function that we can use as a convenient shorthand to create elements of type li. We store a reference to this function in a variable called createListItemElement. Then, we call this function three times, and each time we only pass the props and children parameters, which are unique for each element. Notice that React.createElement() and React.createFactory() both expect the HTML tag name string (such as li) or the ReactClass object as a type parameter. React provides us with a number of built-in factory functions to create the common HTML tags. You can call them from the React.DOM object; for example, React.DOM.ul(), React.DOM.li(), React.DOM.div(), and so on. Using them, we can simplify our previous example even further: var React = require('react'); var ReactDOM = require('react-dom');   var listItemElement1 = React.DOM.li({ className: 'item-1', key: 'item-1' }, 'Item 1'); var listItemElement2 = React.DOM.li({ className: 'item-2', key: 'item-2' }, 'Item 2'); var listItemElement3 = React.DOM.li({ className: 'item-3', key: 'item-3' }, 'Item 3');   var reactFragment = [ listItemElement1, listItemElement2, listItemElement3 ]; var listOfItems = React.DOM.ul({ className: 'list-of-items' }, reactFragment);   ReactDOM.render(listOfItems, document.getElementById('react-application')); Now we know how to create a tree of ReactNodes. However, there is one important line of code that we need to discuss before we can progress further: ReactDOM.render(listOfItems, document.getElementById('react-application')); As you might have already guessed, it renders our ReactNode tree to the DOM. Let's take a closer look at how it works. Rendering React Elements The ReactDOM.render() method takes three parameters: ReactElement, a regular DOMElement, and a callback function: ReactDOM.render(ReactElement, DOMElement, callback); ReactElement is a root element in the tree of ReactNodes that you've created. A regular DOMElement is a container DOM node for that tree. The callback is a function executed after the tree is rendered or updated. It's important to note that if this ReactElement was previously rendered to a parent DOM Element, then ReactDOM.render() will perform an update on the already rendered DOM tree and only mutate the DOM as it is necessary to reflect the latest version of the ReactElement. This is why a virtual DOM requires fewer DOM mutations. So far, we've assumed that we're always creating our virtual DOM in a web browser. This is understandable because, after all, React is a user interface library, and all the user interfaces are rendered in a web browser. Can you think of a case when rendering a user interface on a client would be slow? Some of you might have already guessed that I am talking about the initial page load. The problem with the initial page load is the one I mentioned at the beginning of this article—we're not creating static web pages anymore. Instead, when a web browser loads our web application, it receives only the bare minimum HTML markup that is usually used as a container or a parent element for our web application. Then, our JavaScript code creates the rest of the DOM, but in order for it to do so it often needs to request extra data from the server. However, getting this data takes time. Once this data is received, our JavaScript code starts to mutate the DOM. We know that DOM mutations are slow. How can we solve this problem? The solution is somewhat unexpected. Instead of mutating the DOM in a web browser, we mutate it on a server. Just like we would with our static web pages. A web browser will then receive an HTML that fully represents a user interface of our web application at the time of the initial page load. Sounds simple, but we can't mutate the DOM on a server because it doesn't exist outside a web browser. Or can we? We have a virtual DOM that is just a JavaScript, and as you know using Node.js, we can run JavaScript on a server. So technically, we can use the React library on a server, and we can create our ReactNode tree on a server. The question is how can we render it to a string that we can send to a client? React has a method called ReactDOMServer.renderToString() just to do this: var ReactDOMServer = require('react-dom/server'); ReactDOMServer.renderToString(ReactElement); It takes a ReactElement as a parameter and renders it to its initial HTML. Not only is this faster than mutating a DOM on a client, but it also improves the Search Engine Optimization (SEO) of your web application. Speaking of generating static web pages, we can do this too with React: var ReactDOMServer = require('react-dom/server'); ReactDOM.renderToStaticMarkup(ReactElement); Similar to ReactDOM.renderToString(), this method also takes a ReactElement as a parameter and outputs an HTML string. However, it doesn't create the extra DOM attributes that React uses internally, it produces shorter HTML strings that we can transfer to the wire quickly. Now you know not only how to create a virtual DOM tree using React elements, but you also know how to render it to a client and server. Our next question is whether we can do it quickly and in a more visual manner. Creating React Elements with JSX When we build our virtual DOM by constantly calling the React.createElement() method, it becomes quite hard to visually translate these multiple function calls into a hierarchy of HTML tags. Don't forget that, even though we're working with a virtual DOM, we're still creating a structure layout for our content and user interface. Wouldn't it be great to be able to visualize that layout easily by simply looking at our React code? JSX is an optional HTML-like syntax that allows us to create a virtual DOM tree without using the React.createElement() method. Let's take a look at the previous example that we created without JSX: var React = require('react'); var ReactDOM = require('react-dom');   var listItemElement1 = React.DOM.li({ className: 'item-1', key: 'item-1' }, 'Item 1'); var listItemElement2 = React.DOM.li({ className: 'item-2', key: 'item-2' }, 'Item 2'); var listItemElement3 = React.DOM.li({ className: 'item-3', key: 'item-3' }, 'Item 3');   var reactFragment = [ listItemElement1, listItemElement2, listItemElement3 ]; var listOfItems = React.DOM.ul({ className: 'list-of-items' }, reactFragment);   ReactDOM.render(listOfItems, document.getElementById('react-application')); Translate this to the one with JSX: var React = require('react'); var ReactDOM = require('react-dom');   var listOfItems = <ul className="list-of-items">                     <li className="item-1">Item 1</li>                     <li className="item-2">Item 2</li>                     <li className="item-3">Item 3</li>                   </ul>; ReactDOM.render(listOfItems, document.getElementById('react-application'));   As you can see, JSX allows us to write HTML-like syntax in our JavaScript code. More importantly, we can now clearly see what our HTML layout will look like once it's rendered. JSX is a convenience tool and it comes with a price in the form of an additional transformation step. Transformation of the JSX syntax into valid JavaScript syntax must happen before our "invalid" JavaScript code is interpreted. We know that the babely module transforms our JSX syntax into a JavaScript one. This transformation happens every time we run our default task from gulpfile.js: gulp.task('default', function () {   return browserify('./source/app.js')         .transform(babelify)         .bundle()         .pipe(source('snapterest.js'))         .pipe(gulp.dest('./build/')); }); As you can see, the .transform(babelify) function call transforms JSX into JavaScript before bundling it with the other JavaScript code. To test our transformation, run this command: gulp Then, navigate to the ~/snapterest/build/ directory, and open index.html in a web browser. You will see a list of three items. The React team has built an online JSX Compiler that you can use to test your understanding of how JSX works at http://facebook.github.io/react/jsx-compiler.html. Using JSX, you might feel very unusual in the beginning, but it can become a very intuitive and convenient tool to use. The best part is that you can choose whether to use it or not. I found that JSX saves me development time, so I chose to use it in this project that we're building. If you choose to not use it, then I believe that you have learned enough in this article to be able to translate the JSX syntax into a JavaScript code with the React.createElement() function calls. If you have a question about what we have discussed in this article, then you can refer to https://github.com/fedosejev/react-essentials and create a new issue. Summary We started this article by discussing the issues with single web page applications and how they can be addressed. Then, we learned what a virtual DOM is and how React allows us to build it. We also installed React and created our first React element using only JavaScript. Then, we also learned how to render React elements in a web browser and on a server. Finally, we looked at a simpler way of creating React elements with JSX. Resources for Article: Further resources on this subject: Changing Views [article] Introduction to Akka [article] ECMAScript 6 Standard [article]
Read more
  • 0
  • 0
  • 17426

article-image-python-data-analysis-utilities
Packt
17 Feb 2016
13 min read
Save for later

Python Data Analysis Utilities

Packt
17 Feb 2016
13 min read
After the success of the book Python Data Analysis, Packt's acquisition editor Prachi Bisht gauged the interest of the author, Ivan Idris, in publishing Python Data Analysis Cookbook. According to Ivan, Python Data Analysis is one of his best books. Python Data Analysis Cookbook is meant for a bit more experienced Pythonistas and is written in the cookbook format. In the year after the release of Python Data Analysis, Ivan has received a lot of feedback—mostly positive, as far as he is concerned. Although Python Data Analysis covers a wide range of topics, Ivan still managed to leave out a lot of subjects. He realized that he needed a library as a toolbox. Named dautil for data analysis utilities, the API was distributed by him via PyPi so that it is installable via pip/easy_install. As you know, Python 2 will no longer be supported after 2020, so dautil is based on Python 3. For the sake of reproducibility, Ivan also published a Docker repository named pydacbk (for Python Data Analysis Cookbook). The repository represents a virtual image with preinstalled software. For practical reasons, the image doesn't contain all the software, but it still contains a fair percentage. This article has the following sections: Data analysis, data science, big data – what is the big deal? A brief history of data analysis with Python A high-level overview of dautil IPython notebook utilities Downloading data Plotting utilities Demystifying Docker Future directions (For more resources related to this topic, see here.) Data analysis, data science, big data – what is the big deal? You've probably seen Venn diagrams depicting data science as the intersection of mathematics/statistics, computer science, and domain expertise. Data analysis is timeless and was there before data science and computer science. You could perform data analysis with a pen and paper and, in more modern times, with a pocket calculator. Data analysis has many aspects with goals such as making decisions or coming up with new hypotheses and questions. The hype, status, and financial rewards surrounding data science and big data remind me of the time when data warehousing and business intelligence were the buzzwords. The ultimate goal of business intelligence and data warehousing was to build dashboards for management. This involved a lot of politics and organizational aspects, but on the technical side, it was mostly about databases. Data science, on the other hand, is not database-centric, and leans heavily on machine learning. Machine learning techniques have become necessary because of the bigger volumes of data. Data growth is caused by the growth of the world's population and the rise of new technologies such as social media and mobile devices. Data growth is in fact probably the only trend that we can be sure will continue. The difference between constructing dashboards and applying machine learning is analogous to the way search engines evolved. Search engines (if you can call them that) were initially nothing more than well-organized collections of links created manually. Eventually, the automated approach won. Since more data will be created in time (and not destroyed), we can expect an increase in automated data analysis. A brief history of data analysis with Python The history of the various Python software libraries is quite interesting. I am not a historian, so the following notes are written from my own perspective: 1989: Guido van Rossum implements the very first version of Python at the CWI in the Netherlands as a Christmas hobby project. 1995: Jim Hugunin creates Numeric, the predecessor to NumPy. 1999: Pearu Peterson writes f2py as a bridge between Fortran and Python. 2000: Python 2.0 is released. 2001: The SciPy library is released. Also, Numarray, a competing library of Numeric, is created. Fernando Perez releases IPython, which starts out as an afternoon hack. NLTK is released as a research project. 2002: John Hunter creates the matplotlib library. 2005: NumPy is released by Travis Oliphant. Initially, NumPy is Numeric extended with features inspired by Numarray. 2006: NumPy 1.0 is released. The first version of SQLAlchemy is released. 2007: The scikit-learn project is initiated as a Google Summer of Code project by David Cournapeau. Cython is forked from Pyrex. Cython is later intensively used in pandas and scikit-learn to improve performance. 2008: Wes McKinney starts working on pandas. Python 3.0 is released. 2011: The IPython 0.12 release introduces the IPython notebook. Packt releases NumPy 1.5 Beginner's Guide. 2012: Packt releases NumPy Cookbook. 2013: Packt releases NumPy Beginner's Guide - Second Edition. 2014: Fernando Perez announces Project Jupyter, which aims to make a language-agnostic notebook. Packt releases Learning NumPy Array and Python Data Analysis. 2015: Packt releases NumPy Beginner's Guide - Third Edition and NumPy Cookbook - Second Edition. A high-level overview of dautil The dautil API that Ivan made for this book is a humble toolbox, which he found useful. It is released under the MIT license. This license is very permissive, so you could in theory use the library in a production system. He doesn't recommend doing this currently (as of January, 2016), but he believes that the unit tests and documentation are of acceptable quality. The library has 3000+ lines of code and 180+ unit tests with a reasonable coverage. He has fixed as many issues reported by pep8 and flake8 as possible. Some of the functions in dautil are on the short side and are of very low complexity. This is on purpose. If there is a second edition (knock on wood), dautil will probably be completely transformed. The API evolved as Ivan wrote the book under high time pressure, so some of the decisions he made may not be optimal in retrospect. However, he hopes that people find dautil useful and, ideally, contribute to it. The dautil modules are summarized in the following table: Module Description LOC dautil.collect Contains utilities related to collections 331 dautil.conf Contains configuration utilities 48 dautil.data Contains utilities to download and load data 468 dautil.db Contains database-related utilities 98 dautil.log_api Contains logging utilities 204 dautil.nb Contains IPython/Jupyter notebook widgets and utilities 609 dautil.options Configures dynamic options of several libraries related to data analysis 71 dautil.perf Contains performance-related utilities 162 dautil.plotting Contains plotting utilities 382 dautil.report Contains reporting utilities 232 dautil.stats Contains statistical functions and utilities 366 dautil.ts Contains Utilities for time series and dates 217 dautil.web Contains utilities for web mining and HTML processing 47 IPython notebook utilities The IPython notebook has become a standard tool for data analysis. The dautil.nb has several interactive IPython widgets to help with Latex rendering, the setting of matplotlib properties, and plotting. Ivan has defined a Context class, which represents the configuration settings of the widgets. The settings are stored in a pretty-printed JSON file in the current working directory, which is named dautil.json. This could be extended, maybe even with a database backend. The following is an edited excerpt (so that it doesn't take up a lot of space) of an example dautil.json: { ... "calculating_moments": { "figure.figsize": [ 10.4, 7.7 ], "font.size": 11.2 }, "calculating_moments.latex": [ 1, 2, 3, 4, 5, 6, 7 ], "launching_futures": { "figure.figsize": [ 11.5, 8.5 ] }, "launching_futures.labels": [ [ {}, { "legend": "loc=best", "title": "Distribution of Means" } ], [ { "legend": "loc=best", "title": "Distribution of Standard Deviation" }, { "legend": "loc=best", "title": "Distribution of Skewness" } ] ], ... }  The Context object can be constructed with a string—Ivan recommends using the name of the notebook, but any unique identifier will do. The dautil.nb.LatexRenderer also uses the Context class. It is a utility class, which helps you number and render Latex equations in an IPython/Jupyter notebook, for instance, as follows: import dautil as dl lr = dl.nb.LatexRenderer(chapter=12, context=context) lr.render(r'delta! = x - m') lr.render(r'm' = m + frac{delta}{n}') lr.render(r'M_2' = M_2 + delta^2 frac{ n-1}{n}') lr.render(r'M_3' = M_3 + delta^3 frac{ (n - 1) (n - 2)}{n^2}/ - frac{3delta M_2}{n}') lr.render(r'M_4' = M_4 + frac{delta^4 (n - 1) / (n^2 - 3n + 3)}{n^3} + frac{6delta^2 M_2}/ {n^2} - frac{4delta M_3}{n}') lr.render(r'g_1 = frac{sqrt{n} M_3}{M_2^{3/2}}') lr.render(r'g_2 = frac{n M_4}{M_2^2}-3.') The following is the result:   Another widget you may find useful is RcWidget, which sets matplotlib settings, as shown in the following screenshot: Downloading data Sometimes, we require sample data to test an algorithm or prototype a visualization. In the dautil.data module, you will find many utilities for data retrieval. Throughout this book, Ivan has used weather data from the KNMI for the weather station in De Bilt. A couple of the utilities in the module add a caching layer on top of existing pandas functions, such as the ones that download data from the World Bank and Yahoo! Finance (the caching depends on the joblib library and is currently not very configurable). You can also get audio, demographics, Facebook, and marketing data. The data is stored under a special data directory, which depends on the operating system. On the machine used in the book, it is stored under ~/Library/Application Support/dautil. The following example code loads data from the SPAN Facebook dataset and computes the clique number: import networkx as nx import dautil as dl fb_file = dl.data.SPANFB().load() G = nx.read_edgelist(fb_file, create_using=nx.Graph(), nodetype=int) print('Graph Clique Number', nx.graph_clique_number(G.subgraph(list(range(2048)))))  To understand what is going on in detail, you will need to read the book. In a nutshell, we load the data and use the NetworkX API to calculate a network metric. Plotting utilities Ivan visualizes data very often in the book. Plotting helps us get an idea about how the data is structured and helps you form hypotheses or research questions. Often, we want to chart multiple variables, but we want to easily see what is what. The standard solution in matplotlib is to cycle colors. However, Ivan prefers to cycle line widths and line styles as well. The following unit test demonstrates his solution to this issue: def test_cycle_plotter_plot(self): m_ax = Mock() cp = plotting.CyclePlotter(m_ax) cp.plot([0], [0]) m_ax.plot.assert_called_with([0], [0], '-', lw=1) cp.plot([0], [1]) m_ax.plot.assert_called_with([0], [1], '--', lw=2) cp.plot([1], [0]) m_ax.plot.assert_called_with([1], [0], '-.', lw=1) The dautil.plotting module currently also has a helper tool for subplots, histograms, regression plots, and dealing with color maps. The following example code (the code for the labels has been omitted) demonstrates a bar chart utility function and a utility function from dautil.data, which downloads stock price data: import dautil as dl import numpy as np import matplotlib.pyplot as plt ratios = [] STOCKS = ['AAPL', 'INTC', 'MSFT', 'KO', 'DIS', 'MCD', 'NKE', 'IBM'] for symbol in STOCKS: ohlc = dl.data.OHLC() P = ohlc.get(symbol)['Adj Close'].values N = len(P) mu = (np.log(P[-1]) - np.log(P[0]))/N var_a = 0 var_b = 0 for k in range(1, N): var_a = (np.log(P[k]) - np.log(P[k - 1]) - mu) ** 2 var_a = var_a / N for k in range(1, N//2): var_b = (np.log(P[2 * k]) - np.log(P[2 * k - 2]) - 2 * mu) ** 2 var_b = var_b / N ratios.append(var_b/var_a - 1) _, ax = plt.subplots() dl.plotting.bar(ax, STOCKS, ratios) plt.show() Refer to the following screenshot for the end result: The code performs a random walk test and calculates the corresponding ratio for a list of stock prices. The data is retrieved whenever you run the code, so you may get different results. Some of you have a finance aversion, but rest assured that this book has very little finance-related content. The following script demonstrates a linear regression utility and caching downloader for World Bank data (the code for the watermark and plot labels has been omitted): import dautil as dl import matplotlib.pyplot as plt import numpy as np wb = dl.data.Worldbank() countries = wb.get_countries()[['name', 'iso2c']] inf_mort = wb.get_name('inf_mort') gdp_pcap = wb.get_name('gdp_pcap') df = wb.download(country=countries['iso2c'], indicator=[inf_mort, gdp_pcap], start=2010, end=2010).dropna() loglog = df.applymap(np.log10) x = loglog[gdp_pcap] y = loglog[inf_mort] dl.options.mimic_seaborn() fig, [ax, ax2] = plt.subplots(2, 1) ax.set_ylim([0, 200]) ax.scatter(df[gdp_pcap], df[inf_mort]) ax2.scatter(x, y) dl.plotting.plot_polyfit(ax2, x, y) plt.show()  The following image should be displayed by the code: The program downloads World Bank data for 2010 and plots the infant mortality rate against the GDP per capita. Also shown is a linear fit of the log-transformed data. Demystifying Docker Docker uses Linux kernel features to provide an extra virtualization layer. It was created in 2013 by Solomon Hykes. Boot2Docker allows us to install Docker on Windows and Mac OS X as well. Boot2Docker uses a VirtualBox VM that contains a Linux environment with Docker. Ivan's Docker image, which is mentioned in the introduction, is based on the continuumio/miniconda3 Docker image. The Docker installation docs are at https://docs.docker.com/index.html. Once you install Boot2Docker, you need to initialize it. This is only necessary once, and Linux users don't need this step: $ boot2docker init The next step for Mac OS X and Windows users is to start the VM: $ boot2docker start Check the Docker environment by starting a sample container: $ docker run hello-world Docker images are organized in a repository, which resembles GitHub. A producer pushes images and a consumer pulls images. You can pull Ivan's repository with the following command. The size is currently 387 MB. $ docker pull ivanidris/pydacbk Future directions The dautil API consists of items Ivan thinks will be useful outside of the context of this book. Certain functions and classes that he felt were only suitable for a particular chapter are placed in separate per-chapter modules, such as ch12util.py. In retrospect, parts of those modules may need to be included in dautil as well. In no particular order, Ivan has the following ideas for future dautil development: He is playing with the idea of creating a parallel library with "Cythonized" code, but this depends on how dautil is received Adding more data loaders as required There is a whole range of streaming (or online) algorithms that he thinks should be included in dautil as well The GUI of the notebook widgets should be improved and extended The API should have more configuration options and be easier to configure Summary In this article, Ivan roughly sketched what data analysis, data science, and big data are about. This was followed by a brief of history of data analysis with Python. Then, he started explaining dautil—the API he made to help him with this book. He gave a high-level overview and some examples of the IPython notebook utilities, features to download data, and plotting utilities. He used Docker for testing and giving readers a reproducible data analysis environment, so he spent some time on that topic too. Finally, he mentioned the possible future directions that could be taken for the library in order to guide anyone who wants to contribute. Resources for Article:   Further resources on this subject: Recommending Movies at Scale (Python) [article] Python Data Science Up and Running [article] Making Your Data Everything It Can Be [article]
Read more
  • 0
  • 0
  • 7911
Modal Close icon
Modal Close icon