Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7018 Articles
article-image-data-types-and-fields
Packt
10 Aug 2015
30 min read
Save for later

Data Types and Fields

Packt
10 Aug 2015
30 min read
In this article by David Studebaker and Christopher Studebaker, authors of the book Programming Microsoft Dynamics™ NAV 2015, explain the design of an application should begin at the simplest level, with the design of the data elements. The type of data our development tool supports has a significant effect on our design. Because NAV is designed for financially-oriented business applications, NAV data types are financially and business oriented. In this article, we will cover many of the data types we use within NAV. For each data type, we will cover some of the more frequently modified field properties and how particular properties, such as Field Class, are used to support application functionality. Field Class is a fundamental property which defines whether the contents of the field are data to be processed or control information to be interpreted. (For more resources related to this topic, see here.) Data types We are going to segregate the data types into several groups. We will first look at Fundamental data types and then at Complex data types. Fundamental data types Fundamental data types are the basic components from which the complex data types are formed. They are grouped into Numeric, String, and Date/Time data types. Numeric data Just like other systems, Microsoft Dynamics NAV 2015 supports several numeric data types. The specifications for each NAV data type are defined for NAV, independent of the supporting SQL Server database rules. However, some data types are stored and handled somewhat differently from a SQL Server point of view than the way they appear to us as NAV developers and users. For more details on the SQL Server-specific representations of various data elements, refer to the Developer and IT Pro Help. Our discussion will focus on NAV representation and handling for each data type. The various numeric data types are as follows: Integer: This is an integer number ranging from -2,147,483,646 to +2,147,483,647 Decimal: This is a decimal number in the range of +/- 999,999,999,999,999.99. Although it is possible to construct larger numbers, errors such as overflow, truncation, or loss of precision might occur. In addition, there is no facility to display or edit larger numbers. Option: This is a special instance of an integer, stored as an integer number ranging from 0 to +2,147,483,647. An option is normally represented in the body of our C/AL code as an option string. We can compare an option to an integer in C/AL rather than using the option string. However, this is not a good practice because it eliminates the self-documenting aspect of an option field. An option string is a set of choices listed in a comma-separated string, one of which is chosen and stored as the current option. Since the maximum length of this string is 250 characters, the practical maximum number of choices for a single option is less than 125. The currently selected choice within the set of options is stored in the option field as the ordinal position of that option within the set. For example, selection of an entry from the option string of red, yellow, and blue would result in the storing of 0 (red), 1 (yellow), and 2 (blue). If red were selected, 0 would be stored in the variable and if blue were selected, 2 would be stored. Quite often, an option string starts with a blank to allow an effective choice of "none chosen". An example of this (blank, Hourly, Daily,…) is as follows: Boolean: A Boolean variable is stored as 1 or 0. In a C/AL code, it is programmatically referred to as True or False, but sometimes, it is referred in properties as Yes or No. Boolean variables may be displayed as Yes or No (language dependent), P or blank, or True or False. BigInteger: 8-byte Integer as opposed to the 4 bytes of Integer. BigIntegers are for very big numbers (from -9,223,372,036,854,775,807 to 9,223,372,036,854,775,807). Char: A numeric code between 0 and 65535 (hexadecimal FFFF) representing a single 16-bit Unicode character. Char variables can operate either as text or numbers. Numeric operations can be done on Char variables. Char variables can also be defined with individual text character values. Char variables cannot be defined as permanent variables in a table; they can only be defined as working storage variables within C/AL objects. Byte: This is a single 8-bit ASCII character with a value ranging from 0 to 255. Byte variables can operate either as text or numbers. Numeric operations can be done on Byte variables. Byte variables can also be defined with individual text character values. Byte variables cannot be defined as permanent variables in a table, but only as working storage variables within C/AL objects. Action: This is a variable returned from a PAGE RUNMODAL function or RUNMODAL (Page) function that specifies what action a user performs on a page. The possible values are OK, Cancel, LookupOK, LookupCancel, Yes, No, RunObject, and RunSystem. ExecutionMode: This specifies the mode in which a session runs. The possible values are Debug or Standard. String data The following are the data types included in String data: Text: This contains any string of alphanumeric characters. In a table, a Text field can be from 1 to 250 characters long. In working storage within an object, a Text variable can be any length if no length is defined. If a maximum length is defined, it must not exceed 1024. NAV 2015 does not require a length to be specified, but if we define a maximum length, it will be enforced. When calculating the length of a record for design purposes (relative to the maximum record length of 8,000 bytes), the full defined field length should be counted. Code: Although the Help says that the length constraints for Code variables are the same as those for text variables, the C/AL Editor enforces length limits of 1 to 250 characters. All of the letters are automatically converted to uppercase when data is entered into a Code variable; any leading or trailing spaces are removed. Date/Time data The following are the data types included in Date/Time data: Date: This contains an integer number, which is interpreted as a date ranging from January 1, 1754 to December 31, 9999. A 0D (numeral zero, letter D) represents an undefined date (stored as a SQL Server DateTime field) that is interpreted as January 1, 1753. According to the Developer and IT Pro Help that, NAV 2015 supports a Date of 1/1/0000 (presumably as a special case for backward compatibility, but it is not supported by SQL Server). A date constant can be written as the letter D preceded by either six digits in the format MMDDYY or eight digits as MMDDYYYY (where M = month, D = day, and Y = year). For example, 011915D or 01192015D both represent January 19, 2015. Later, in DateFormula, we will find D interpreted as day, but here the trailing D is interpreted as the date (data type) constant. When the year is expressed as YY rather than YYYY, the century portion (in this case, 20) is 20 if the two digit year is from 00 to 29, or 19 if the year is from 30 through 99. NAV also defines a special date called the Closing date, which represents the point in time between one day and the next. The purpose of a closing date is to provide a point at the end of a day, after all of the real date- and time-sensitive activity is recorded—the point when accounting closing entries can be recorded. Closing entries are recorded, in effect, at the stroke of midnight between two dates—this is the date of closing accounting books, and it is designed so that one can include or exclude, at the user's option, closing entries in various reports. When sorted by date, the closing date entries will get sorted after all normal entries for a day. For example, the normal date entry for December 31, 2015 would display as 12/31/15 (depending on the date format masking), and the closing date entry would display as C12/31/15. All of the C12/31/15 ledger entries would appear after all normal 12/31/15 ledger entries. The following screenshot shows two 2014 closing date entries mixed with normal entries from December 2015 and January through April 2015. (This data is from Cronus demo. The 2014 Closing entries have an "Opening Entry" description, which shows that these were the first entries for the demo data in the respective accounts. This is not a normal set of production data.) Time: This contains an integer number, which is interpreted on a 24-hour clock, in milliseconds plus 1, from 00:00:00 to 23:59:59:999. A 0T (numeral zero, letter T) represents an undefined time and is stored as 1/1/1753 00:00:00.000. DateTime: This represents a combined Date and Time, stored in Coordinated Universal Time (UTC), and it always displays local time (that is, the local time on our system). DateTime fields do not support NAV "Closing" dates. DateTime is helpful for an application that must support multiple time zones simultaneously. DateTime values can range from January 1, 1754 00:00:00.000 to December 31, 9999 23:59:59.999, but dates earlier than January 1, 1754 cannot be entered (don't test with dates late in 9999 as an intended advance to the year 10000 won't work). Assigning a date of 0DT will yield an undefined or blank DateTime. Duration: This represents the positive or negative difference between two DateTime values, in milliseconds, stored as a BigInteger. Durations are automatically output in the text format as DDD days HH hours MM minutes SS seconds. Complex data types Each complex data type consists of multiple data elements. For ease of reference, we will categorize them into several groups of similar types. Data structure The following data types are in the data structure group: File: This refers to any standard Windows file outside the NAV database. There is a reasonably complete set of functions to allow to create, delete, open, close, read, write, and copy (among other things) data files. For example, we could create our own NAV routines in C/AL to import or export data from or to a file that had been created by some other application. With the three-tier architecture of NAV 2015, business logic runs on the server and not the client. We need to keep this in mind any time we refer to local external files because they will be on the server by default. Use of Universal Naming Convention (UNC) paths can make this easier to manage. Record: This refers to a single data row within a NAV table that consists of individual fields. Quite often, multiple variable instances of a Record (table) are defined in working storage to support a validation process, allowing access to different records within the table at one time in the same function. Objects Page, Report, Codeunit, Query, and XMLPort, each represents an object data type. Object data types are used when there is a need to refer to an object or a function in another object. Examples: Invoking a Report or an XMLPort from a Page or a Report Calling a function for data validation or processing is coded as a function in a Table or a Codeunit Automation The following are Automation data types. (these are not supported by the NAV Web client.) OCX and Automation data types are supported in NAV 2015 for backward compatibility only: OCX: This allows the definition of a variable that represents and allows access to an ActiveX or OCX custom control. Such a control is typically an external application object that we can invoke from our NAV object. Automation: This allows us to define a variable that we can access similar to an OCX. The application must act as an Automation Server and be registered with the NAV client or server that calls it. For example, we can interface from NAV into the various Microsoft Office products (Word, Excel, and so on) by defining them in Automation variables. DotNet: This allows us to define a variable for .NET Framework interface types within an assembly. It supports accessing .NET Framework type members, including methods, properties, and constructors from C/AL. These can be members of the global assembly cache or custom assemblies. Input/Output The following are the Input/Output data types: Dialog: This supports the definition of a simple user interface window without the use of a Page object. Typically, Dialog windows are used to communicate processing progress or allow a brief user response to a go/no-go question, though this latter use could result in bad performance due to locking. There are other user communication tools as well, but they do not use a Dialog type data item. InStream and Outstream: These allow us to read from and write to external files, BLOBS, and objects of the Automation and OCX data types. DateFormula DateFormula provides for the definition and storage of a simple, but clever, set of constructs to support the calculation of runtime-sensitive dates. A DateFormula is stored in a nonlanguage dependent format, thus supporting multilanguage functionality. A DateFormula is a combination of: Numeric multipliers (for example, 1, 2, 3, 4, and so on) Alpha time units (all must be in uppercase) D for a day W for a week WD for day of the week, that is, from day 1 to day 7 (either in the future or in the past but not today). Monday is day 1 and Sunday is day 7. M for calendar month Y for year CM for current month, CY for current year, CW for current week Math symbols interpretation + (plus) as in CM + 10D means the Current Month end plus 10 Days (in other words, the tenth of the next month) – (minus) as in (-WD3) means the date of the previous Wednesday (which is the 3rd day of the past week). Positional notation (D15 means the 15th day of the month and 15D means 15 days) Payment Terms for Invoices support full use of DateFormula. All DateFormula results are expressed as a date based on a reference date. The default reference date is the system date and not the Work Date. Here are some sample DateFormulas and their interpretations (displayed dates are based on the US calendar) with a reference date of July 10, 2015, a Friday: CM is the last day of Current Month, 07/31/15 CM + 10D is the tenth of the next month, 08/10/15 WD6 is the next sixth day of the week, 07/11/15 WD5 is the next fifth day of the week, 07/17/15 CM – M + D is the end of the current month minus one month plus one day, 07/01/15 CM – 5M is the end of the current month minus five months, 02/28/15 Let us take the opportunity to use the DateFormula data type to learn a few NAV development basics. We will do so by experimenting with some hands-on evaluations of several DateFormula values. We will create a table to calculate dates using DateFormula and Reference Dates. To do this, navigate to Tools | Object Designer | Tables. Then, click on the New button and define the fields shown in the following screenshot. Save it as Table 50009, named Date Formula Test. After we are done with this test, we will save this table for some later testing. Now, we will add some simple C/AL code to our table so that when we enter or change either the Reference Date or the DateFormula data, we can calculate a new result date. First, access the new table via the Design button. Then, go to the global variables definition form through the View menu option, the C/AL Globals sub-option, and finally, choose the Functions tab. Type in our new function name as CalculateNewDate on the first blank line, as shown in the following screenshot, and then exit (by means of the Esc key) from this form back to the list of data fields: From the Table Designer form that displays the list of data fields, either press F9 or click on the C/AL Code icon: This will take us to the following screen, where we can see all of the field triggers plus the trigger for the new function that we just defined. The table triggers will not be visible, unless we scroll up to show them. Note that our new function was defined as a LOCAL function. This means that it cannot be accessed from another object unless we change it to a GLOBAL function. Since our goal now is to focus on experimenting with the DateFormula, we will not go into detail and explain the logic of what we are creating. The logic that we're going to code is as follows: When an entry is made (new or changed) in either the "Reference Date" field or in the "Date Formula to Test field", invoke the CalculateNewDate function to calculate a new “Result Date” value based on the entered data. First, you need to create the logic within our new function, CalculateNewDate(), to evaluate and store a Date Result based on the DateFormula and Reference Date that you enter into the table. Just copy the C/AL code exactly as shown in the following screenshot, exit, compile, and save the table: If you get an error message of any type when you close and save the table, you probably have not copied the C/AL code exactly as it is shown in the screenshot. (also shown below for ease of copying.) CalculateNewDate;"Date Result" := CALCDATE("Date Formula to Test","Reference Date for Calculation"); This code will cause the CalculateNewDate()function to be called via the OnValidate trigger when an entry is made in either the Reference Date for Calculation or the Date Formula to Test fields. The function will place the result in the Date Result field. The use of an integer value in the redundantly named Primary Key field allows us to enter any number of records into the table (by manually numbering them 1, 2, 3, and so forth). Let's experiment with several different date and date formula combinations. We will access the table via the Run button. This will cause NAV to generate a default format page and run it in the Role Tailored Client. Enter a Primary Key value of 1 (one). In Reference Date for Calculation, enter either an upper or lower case T for Today and the system date. The same date will appear in the Date Result field because at this point, no Date Formula has been entered. Now, enter 1D (number 1 followed by uppercase or lowercase D (C/SIDE will make it uppercase) in the Date Formula to Test field. We will see that the Date Result field contents are changed to be one day beyond the date in the Reference Date for Calculation field. Now, for another test entry, start with a 2 in the Primary Key field. Again, enter the letter T (for Today) in the Reference Date for Calculation field, and enter the letter W (for Week) in the Date Formula to Test field. We will get an error message telling us that our formulas should include a number. Make the system happy and enter 1W. We will now see a date in the Date Result field that is one week beyond our system date. Set the system's Work Date to a date in the middle of a month. Start another line with the number 3 as the Primary Key, followed by a W (for Work Date) in the Reference Date for Calculation field. Enter cm (or CM or cM or Cm, it doesn't matter) in the Date Formula to Test field. Our result date will be the last day of our Work Date month. Now, enter another line using the Work Date, but enter a formula of –cm (the same as before but with a minus sign). This time, our result date will be the first day of our Work Date month. Note that the DateFormula logic handles month end dates correctly, including a leap year. Try starting with a date in the middle of February 2016 to confirm this. The following screen shows the Date Formula Test window: Now, enter another line with a new Primary Key. Skip over the Reference Date for Calculation field and just enter 1D in the Date Formula to Test field. So, what happens when you do this? We get an error message stating that "You cannot base a date calculation on an undefined date." In other words, NAV cannot make the requested calculation without a Reference Date. Before we put this function into production, we want our code to check for a Reference Date before calculating. We could default an empty date to the System Date or the Work Date and avoid this particular error. The preceding and following screenshots show different sample calculations. Build on these and then experiment. We can create a variety of different algebraic date formulae and get some very interesting results. One NAV user has due dates on Invoices for the tenth of the next month. Invoices are dated at various times during the month than they are actually printed. By using the DateFormula of CM + 10D, the due date is always automatically calculated to be the tenth of the next month. Don't forget to test with WD (weekday), Q (quarter), and Y (year) as well as D (day), W (week), and M (month). For our code to be language independent, we should enter the date formulae with < > delimiters around them (for example, <1D+1W>). NAV will translate the formula into the correct language codes using the installed language layer. Although our focus for the work we just completed was the Date Formula data type, we've accomplished a lot more than simply learning about that one data type: We created a new table just for the purpose of experimenting with a C/AL feature that we might use. This is a technique that comes in handy when we are learning a new feature or trying to decide how it works or how we might use it. We put some critical OnValidate logic in the table. When data is entered in one area, the entry is validated and, if valid, the defined processing is done instantly. We created a common routine as a new LOCAL function. This function is then called from all the places to which it applies. We did our entire test with a table object and a default tabular page that is automatically generated when we Run a table. We didn't have to create a supporting structure to do our testing. Of course, when we design a change to a complicated existing structure, we will have a more complicated testing scenario. One of our goals will always be to simplify our testing scenarios, both to minimize the setup effort and to keep our test narrowly focused on the specific issue. Finally, and most specifically, we saw how NAV tools make a variety of relative date calculations easy. These are very useful in business applications, many aspects of which are date centered. References and other data types The following data types are used for advanced functionality in NAV, sometimes supporting an interface with an external object: RecordID: This contains the object number and primary key of a table. RecordRef: This identifies a row in a table, a record. RecordRef can be used to obtain information about the table, the record, the fields in the record, and the currently active filters on the table. FieldRef: This identifies a field in a table; thus, it allows access to the contents of that field. KeyRef: This identifies a key in a table and the fields in that key. Since the specific record, field, and key references are assigned at runtime, RecordRef, FieldRef, and KeyRef are used to support logic which can run on tables that are not specified at design time. This means that one routine built on these data types can be created to perform a common function for a variety of different tables and table formats. Variant: This defines variables that are typically used to interface with Automation and OCX objects. Variant variables can contain data of various C/AL data types to pass them to an Automation or OCX object as well as external Automation data types that cannot be mapped to C/AL data types. TableFilter: For variables which can only be used for setting security filters from the Permissions table. Transaction Type: This has optional values of UpdateNoLocks, Update, Snapshot, Browse, and Report that define SQL Server behavior for a NAV Report or XMLport transaction from the beginning of the transaction. BLOB: This can contain either specially formatted text, a graphic in the form of a bitmap, or other developer-defined binary data up to 2 GB in size. The term Binary Large Object (BLOB). BLOBs can only be included in tables and not used to define working storage Variables. Refer to Developer and IT Pro Help for additional information. BigText: This can contain large chunks of text up to 2 GB in size. BigText variables can only be defined in the working storage within an object, but they cannot be included in tables. BigText variables cannot be directly displayed or seen in the debugger. There is a group of special functions that can be used to handle BigText data. Refer to Developer and IT Pro Help for additional information. To handle text strings in a single data element that are greater than 250 characters in length, use a combination of BLOB and BigText variables. GUID: This is used to assign a unique identifying number to any database object. Globally Unique Identifier (GUID), a 16-byte binary data type that is used for unique global identification of records, objects, and so on. GUID is generated by an algorithm developed by Microsoft. TestPage: This is used to store a test page, which is a logical representation of a page that does not display a user interface. Test pages are used when you do NAV application testing using the automated testing facility that is part of NAV. Data type usage About forty percent of the data types can be used to define the data either stored in tables or in working storage data definitions (that is, in a Global or Local data definition within an object). Two data types, BLOB and TableFilter, can only be used to define table-stored data, but not working storage data. About sixty percent of the data types can only be used for working storage data definitions. The following list shows which data types can be used for table (persisted) data fields and which ones can be used for working storage (variable) data: FieldClass property options Almost all data fields have a FieldClass property. FieldClass has as much effect on the content and usage of a data field as the data type; in some instances, it has more effect. Now we'll discuss the FieldClass property options now. FieldClass – Normal When the FieldClass is Normal, the field will contain the type of application data that's typically stored in a table—the contents we would expect based on the data type and various properties. FieldClass – FlowField FlowFields must be dynamically calculated. FlowFields are virtual fields stored as metadata; they do not contain data in the conventional sense. A FlowField contains the definition of how to calculate (at runtime) the data that the field represents and a place to store the result of that calculation. Generally, the Editable property for a FlowField is set to No.. Depending on the CalcFormula method, this could be a value, a reference lookup, or a Boolean. When the CalcFormula method is Sum, the FieldClass connects a data field to a previously defined SumIndexField in the table defined in the CalcFormula. The FlowField processing speed will be significantly affected by the key configuration of the table being processed. While we must be careful not to define extra keys, having the right keys defined will have a major effect on system performance and thus, on user satisfaction. A FlowField value is always 0, blank, or false, unless it has been calculated. If a FlowField is displayed directly on a page, it is calculated automatically when the page is rendered. FlowFields are also automatically calculated when they are the subject of predefined filters as part of the properties of a data item in an object. In all other cases, a FlowField must be forced to calculate using the C/AL RecordName.CALCFIELDS(FlowField1, [FlowField2],...) function or by the use of the SETAUTOCALCFIELDS function. This is also true if the underlying data is changed after the initial display of a page (that is, the FlowField must be recalculated to take a data change into account). Because a FlowField does not contain actual data, it cannot be used as a field in a key. In other words, we cannot include a FlowField as part of a key. In addition, we cannot define a FlowField that is based on another FlowField, except in special circumstances. When a field has its FieldClass set to FlowField, another directly associated property becomes available—CalcFormula. (Conversely, the AltSearchField, AutoIncrement, and TestTableRelation properties disappear from view when FieldClass is set to FlowField). The CalcFormula method is the place where we can define the formula for calculating the FlowField. On the CalcFormula property line, there is an ellipsis button. Clicking on that button will bring up the following screen: Click on the drop-down button to show the seven FlowField methods: The seven FlowFields are described in the following table: FlowField Method Field data type   Calculated value as it applies to the specified set of data within a specific column (field) in a table   Sum Decimal The sum total Average Decimal The average value (the sum divided by the row count) Exist Boolean Yes or No / True or False - does an entry exist? Count Integer The number of entries that exist Min Any The smallest value of any entry Max Any The largest value of any entry Lookup Any The value of the specified entry The Reverse Sign control allows us to change the displayed sign of the result for FlowField types Sum and Average only; the underlying data is not changed. If a Reverse Sign is used with the FlowField type Exists, it changes the effective function to does not Exist. Table and Field allow us to define the Table and the Field within that table to which our Calculation Formula will apply. When we make the entries in our Calculation Formula screen, no validation checking is done by the compiler to check whether we have chosen an eligible table and field combination. This checking doesn't occur until runtime. Therefore, when we create a new FlowField, we should test it as soon as we have defined it. The last, but by no means the least significant component of the FlowField calculation formula is the Table Filter. When we click on the ellipsis in the table filter field, the window shown in the following screenshot will appear: When we click on the Field column, we will be invited to select a field from the table that was entered into the Table field earlier. The Type field choice will determine the type of filter. The Value field will have the filter rules defined on this line, which must be consistent with the Type choices described in the following table: Filter type Value Filtering action OnlyMax- Limit Values- Filter Const   A constant which will be defined in the Value field This uses the constant to filter for equally valued entries     Filter   A filter that will be spelled out as a literal in the Value field This applies the filter expression from the Value field     Field   A field from the table within which the FlowField exists This uses the contents of the specified field to filter equally valued entries False False     If the specified field is a FlowFilter and the OnlyMaxLimit parameter is True, then the FlowFilter range will be applied on the basis of only having a MaxLimit, that is, having no bottom limit. This is useful for the date filters for the Balance Sheet data. (Refer to Balance at Date field in the G/L Account table for an example) True False     This causes the contents of the specified field to be interpreted as a filter (See Balance at Date field in the G/L Account table for an example) True or False True FieldClass – FlowFilter FlowFilters control the calculation of FlowFields in the table (when the FlowFilters are included in the CalcFormula). FlowFilters do not contain permanent data, but instead, they contain filters on a per-user basis, with the information stored in that user's instance of the code that is being executed. A FlowFilter field allows a filter to be entered at a parent record level by the user (for example, G/L Account) and applied (through the use of FlowField formulas, for example) to constrain what child data (for example, G/L Entry records) is selected. A FlowFilter allows us to provide flexible data selection functions to the users. The user does not need to have a full understanding of the data structure to apply filtering in intuitive ways to both the primary data table and the subordinate data. Based on our C/AL code design, FlowFilters can be used to apply filtering on multiple tables that are subordinate to a parent table. Of course, it is our responsibility as developers to make good use of this tool. As with many C/AL capabilities, a good way to learn more is by studying standard code designed by the Microsoft developers of NAV and then experimenting. A number of good examples on the use of FlowFilters can be found in the Customer (Table 18) and Item (Table 27) tables. In the Customer table, some of the FlowFields using FlowFilters are Balance, Balance (LCY), Net Change, Net Change (LCY), Sales (LCY), and Profit (LCY) where LCY stands for local currency. The Sales (LCY) FlowField FlowFilter usage is shown in the following screenshot: Similarly constructed FlowFields using FlowFilters in the Item table include Inventory, Net Invoiced Qty., Net Change, Purchases (Qty.) as well as other fields. Throughout the standard code, there are FlowFilters in most of the master table definitions; there are the Date Filters and Global Dimension Filters (global dimensions are user-defined codes that facilitate the segregation of accounting data by groupings such as divisions, departments, projects, customer type, and so on). Other FlowFilters that are widely used in the standard code related to Inventory activity such as Location Filter, Lot No. Filter, Serial No. Filter, and Bin Filter. The following pair of images shows two fields from the Customer table, both with a Data Type of Date. On the left side of the screenshot is the Last Date Modified field (FieldClass of Normal) and on the right side of the screenshot is the Date Filter field (FieldClass of FlowFilter). It's easy to see that the properties of the two fields are very similar, except for the properties that differ because one is a Normal field and the other is a FlowFilter field. Summary In this article, we focused on the basic building blocks of the NAV data structure: fields and their attributes. We reviewed the types of data fields, properties, and trigger elements for each type of field. We walked through a number of examples to illustrate most of these elements though we had postponed the exploration of triggers until later, when we had more knowledge of C/AL. We covered Data Type and FieldClass, properties which determine what kind of data can be stored in a field. Resources for Article: Further resources on this subject: Customization in Microsoft Dynamics CRM [article] What is BI and What are BI Tools for Microsoft Dynamics GP? [article] Learning MS Dynamics AX 2012 Programming [article]
Read more
  • 0
  • 0
  • 8059

article-image-using-arcpy-dataaccess-module-withfeature-classesand-tables
Packt
10 Aug 2015
28 min read
Save for later

Using the ArcPy DataAccess Module withFeature Classesand Tables

Packt
10 Aug 2015
28 min read
In this article by Eric Pimpler, author of the book Programming ArcGIS with Python Cookbook - Second Edition, we will cover the following topics: Retrieving features from a feature class with SearchCursor Filtering records with a where clause Improving cursor performance with geometry tokens Inserting rows with InsertCursor Updating rows with UpdateCursor (For more resources related to this topic, see here.) We'll start this article with a basic question. What are cursors? Cursors are in-memory objects containing one or more rows of data from a table or feature class. Each row contains attributes from each field in a data source along with the geometry for each feature. Cursors allow you to search, add, insert, update, and delete data from tables and feature classes. The arcpy data access module or arcpy.da was introduced in ArcGIS 10.1 and contains methods that allow you to iterate through each row in a cursor. Various types of cursors can be created depending on the needs of developers. For example, search cursors can be created to read values from rows. Update cursors can be created to update values in rows or delete rows, and insert cursors can be created to insert new rows. There are a number of cursor improvements that have been introduced with the arcpy data access module. Prior to the development of ArcGIS 10.1, cursor performance was notoriously slow. Now, cursors are significantly faster. Esri has estimated that SearchCursors are up to 30 times faster, while InsertCursors are up to 12 times faster. In addition to these general performance improvements, the data access module also provides a number of new options that allow programmers to speed up processing. Rather than returning all the fields in a cursor, you can now specify that a subset of fields be returned. This increases the performance as less data needs to be returned. The same applies to geometry. Traditionally, when accessing the geometry of a feature, the entire geometric definition would be returned. You can now use geometry tokens to return a portion of the geometry rather than the full geometry of the feature. You can also use lists and tuples rather than using rows. There are also other new features, such as edit sessions and the ability to work with versions, domains, and subtypes. There are three cursor functions in arcpy.da. Each returns a cursor object with the same name as the function. SearchCursor() creates a read-only SearchCursor object containing rows from a table or feature class. InsertCursor() creates an InsertCursor object that can be used to insert new records into a table or feature class. UpdateCursor() returns a cursor object that can be used to edit or delete records from a table or feature class. Each of these cursor objects has methods to access rows in the cursor. You can see the relationship between the cursor functions, the objects they create, and how they are used, as follows: Function Object created Usage SearchCursor() SearchCursor This is a read-only view of data from a table or feature class InsertCursor() InsertCursor This adds rows to a table or feature class UpdateCursor() UpdateCursor This edits or deletes rows in a table or feature class The SearchCursor() function is used to return a SearchCursor object. This object can only be used to iterate through a set of rows returned for read-only purposes. No insertions, deletions, or updates can occur through this object. An optional where clause can be set to limit the rows returned. Once you've obtained a cursor instance, it is common to iterate the records, particularly with SearchCursor or UpdateCursor. There are some peculiarities that you need to understand when navigating the records in a cursor. Cursor navigation is forward-moving only. When a cursor is created, the pointer of the cursor sits just above the first row in the cursor. The first call to next() will move the pointer to the first row. Rather than calling the next() method, you can also use a for loop to process each of the records without the need to call the next() method. After performing whatever processing you need to do with this row, a subsequent call to next() will move the pointer to row 2. This process continues as long as you need to access additional rows. However, after a row has been visited, you can't go back a single record at a time. For instance, if the current row is row 3, you can't programmatically back up to row 2. You can only go forward. To revisit rows 1 and 2, you would need to either call the reset() method or recreate the cursor and move back through the object. As I mentioned earlier, cursors are often navigated through the use of for loops as well. In fact, this is a more common way to iterate a cursor and a more efficient way to code your scripts. Cursor navigation is illustrated in the following diagram: The InsertCursor() function is used to create an InsertCursor object that allows you to programmatically add new records to feature classes and tables. To insert rows, call the insertRow() method on this object. You can also retrieve a read-only tuple containing the field names in use by the cursor through the fields property. A lock is placed on the table or feature class being accessed through the cursor. Therefore, it is important to always design your script in a way that releases the cursor when you are done. The UpdateCursor() function can be used to create an UpdateCursor object that can update and delete rows in a table or feature class. As is the case with InsertCursor, this function places a lock on the data while it's being edited or deleted. If the cursor is used inside a Python's with statement, the lock will automatically be freed after the data has been processed. This hasn't always been the case. Prior to ArcGIS 10.1, cursors were required to be manually released using the Python del statement. Once an instance of UpdateCursor has been obtained, you can then call the updateCursor() method to update records in tables or feature classes and the deleteRow() method to delete a row. The subject of data locks requires a little more explanation. The insert and update cursors must obtain a lock on the data source they reference. This means that no other application can concurrently access this data source. Locks are a way of preventing multiple users from changing data at the same time and thus, corrupting the data. When the InsertCursor() and UpdateCursor() methods are called in your code, Python attempts to acquire a lock on the data. This lock must be specifically released after the cursor has finished processing so that the running applications of other users, such as ArcMap or ArcCatalog, can access the data sources. If this isn't done, no other application will be able to access the data. Prior to ArcGIS 10.1 and the with statement, cursors had to be specifically unlocked through Python's del statement. Similarly, ArcMap and ArcCatalog also acquire data locks when updating or deleting data. If a data source has been locked by either of these applications, your Python code will not be able to access the data. Therefore, the best practice is to close ArcMap and ArcCatalog before running any standalone Python scripts that use insert or update cursors. In this article, we're going to cover the use of cursors to access and edit tables and feature classes. However, many of the cursor concepts that existed before ArcGIS 10.1 still apply. Retrieving features from a feature class with SearchCursor There are many occasions when you need to retrieve rows from a table or feature class for read-only purposes. For example, you might want to generate a list of all land parcels in a city with a value greater than $100,000. In this case, you don't have any need to edit the data. Your needs are met simply by generating a list of rows that meet some sort of criteria. A SearchCursor object contains a read-only copy of rows from a table or feature class. These objects can also be filtered through the use of a where clause so that only a subset of the dataset is returned. Getting ready The SearchCursor() function is used to return a SearchCursor object. This object can only be used to iterate a set of rows returned for read-only purposes. No insertions, deletions, or updates can occur through this object. An optional where clause can be set to limit the rows returned. In this article, you will learn how to create a basic SearchCursor object on a feature class through the use of the SearchCursor() function. The SearchCursor object contains a fields property along with the next() and reset() methods. The fields property is a read-only structure in the form of a Python tuple, containing the fields requested from the feature class or table. You are going to hear the term tuple a lot in conjunction with cursors. If you haven't covered this topic before, tuples are a Python structure to store a sequence of data similar to Python lists. However, there are some important differences between Python tuples and lists. Tuples are defined as a sequence of values inside parentheses, while lists are defined as a sequence of values inside brackets. Unlike lists, tuples can't grow and shrink, which can be a very good thing in some cases when you want data values to occupy a specific position each time. This is the case with cursor objects that use tuples to store data from fields in tables and feature classes. How to do it… Follow these steps to learn how to retrieve rows from a table or feature class inside a SearchCursor object: Open IDLE and create a new script window. Save the script as C:ArcpyBookCh8SearchCursor.py. Import the arcpy.da module: import arcpy.da Set the workspace: arcpy.env.workspace = "c:/ArcpyBook/Ch8" Use a Python with statement to create a cursor: with arcpy.da.SearchCursor("Schools.shp",("Facility","Name")) as cursor: Loop through each row in SearchCursor and print the name of the school. Make sure you indent the for loop inside the with block:for row in sorted(cursor): print("School name: " + row[1]) The entire script should appear as follows: import arcpy.da arcpy.env.workspace = "c:/ArcpyBook/Ch8" with arcpy.da.SearchCursor("Schools.shp",("Facility","Name")) as cursor:    for row in sorted(cursor):        print("School name: " + row[1]) Save the script. You can check your work by examining the C:ArcpyBookcodeCh8SearchCursor_Step1.py solution file. Run the script. You should see the following output: School name: ALLAN School name: ALLISON School name: ANDREWS School name: BARANOFF School name: BARRINGTON School name: BARTON CREEK School name: BARTON HILLS School name: BATY School name: BECKER School name: BEE CAVE How it works… The with statement used with the SearchCursor() function will create, open, and close the cursor. So, you no longer have to be concerned with explicitly releasing the lock on the cursor as you did prior to ArcGIS 10.1. The first parameter passed into the SearchCursor() function is a feature class, represented by the Schools.shp file. The second parameter is a Python tuple containing a list of fields that we want returned in the cursor. For performance reasons, it is a best practice to limit the fields returned in the cursor to only those that you need to complete the task. Here, we've specified that only the Facility and Name fields should be returned. The SearchCursor object is stored in a variable called cursor. Inside the with block, we use a Python for loop to loop through each school returned. We also use the Python sorted() function to sort the contents of the cursor. To access the values from a field on the row, simply use the index number of the field you want to return. In this case, we want to return the contents of the Name column, which will be the 1 index number, since it is the second item in the tuple of field names that are returned. Filtering records with a where clause By default, SearchCursor will contain all rows in a table or feature class. However, in many cases, you will want to restrict the number of rows returned by some sort of criteria. Applying a filter through the use of a where clause limits the records returned. Getting ready By default, all rows from a table or feature class will be returned when you create a SearchCursor object. However, in many cases, you will want to restrict the records returned. You can do this by creating a query and passing it as a where clause parameter when calling the SearchCursor() function. In this article, you'll build on the script you created in the previous article by adding a where clause that restricts the records returned. How to do it… Follow these steps to apply a filter to a SearchCursor object that restricts the rows returned from a table or feature class: Open IDLE and load the SearchCursor.py script that you created in the previous recipe. Update the SearchCursor() function by adding a where clause that queries the facility field for records that have the HIGH SCHOOL text: with arcpy.da.SearchCursor("Schools.shp",("Facility","Name"), '"FACILITY" = 'HIGH SCHOOL'') as cursor: You can check your work by examining the C:ArcpyBookcodeCh8SearchCursor_Step2.py solution file. Save and run the script. The output will now be much smaller and restricted to high schools only: High school name: AKINS High school name: ALTERNATIVE LEARNING CENTER High school name: ANDERSON High school name: AUSTIN High school name: BOWIE High school name: CROCKETT High school name: DEL VALLE High school name: ELGIN High school name: GARZA High school name: HENDRICKSON High school name: JOHN B CONNALLY High school name: JOHNSTON High school name: LAGO VISTA How it works… The where clause parameter accepts any valid SQL query, and is used in this case to restrict the number of records that are returned. Improving cursor performance with geometry tokens Geometry tokens were introduced in ArcGIS 10.1 as a performance improvement for cursors. Rather than returning the entire geometry of a feature inside the cursor, only a portion of the geometry is returned. Returning the entire geometry of a feature can result in decreased cursor performance due to the amount of data that has to be returned. It's significantly faster to return only the specific portion of the geometry that is needed. Getting ready A token is provided as one of the fields in the field list passed into the constructor for a cursor and is in the SHAPE@<Part of Feature to be Returned> format. The only exception to this format is the OID@ token, which returns the object ID of the feature. The following code example retrieves only the X and Y coordinates of a feature: with arcpy.da.SearchCursor(fc, ("SHAPE@XY","Facility","Name")) as cursor: The following table lists the available geometry tokens. Not all cursors support the full list of tokens. Check the ArcGIS help files for information about the tokens supported by each cursor type. The SHAPE@ token returns the entire geometry of the feature. Use this carefully though, because it is an expensive operation to return the entire geometry of a feature and can dramatically affect performance. If you don't need the entire geometry, then do not include this token! In this article, you will use a geometry token to increase the performance of a cursor. You'll retrieve the X and Y coordinates of each land parcel from the parcels feature class along with some attribute information about the parcel. How to do it… Follow these steps to add a geometry token to a cursor, which should improve the performance of this object: Open IDLE and create a new script window. Save the script as C:ArcpyBookCh8GeometryToken.py. Import the arcpy.da module and the time module: import arcpy.da import time Set the workspace: arcpy.env.workspace = "c:/ArcpyBook/Ch8" We're going to measure how long it takes to execute the code using a geometry token. Add the start time for the script: start = time.clock() Use a Python with statement to create a cursor that includes the centroid of each feature as well as the ownership information stored in the PY_FULL_OW field: with arcpy.da.SearchCursor("coa_parcels.shp",("PY_FULL_OW","SHAPE@XY")) as cursor: Loop through each row in SearchCursor and print the name of the parcel and location. Make sure you indent the for loop inside the with block: for row in cursor: print("Parcel owner: {0} has a location of: {1}".format(row[0], row[1])) Measure the elapsed time: elapsed = (time.clock() - start) Print the execution time: print("Execution time: " + str(elapsed)) The entire script should appear as follows: import arcpy.da import time arcpy.env.workspace = "c:/ArcpyBook/Ch9" start = time.clock() with arcpy.da.SearchCursor("coa_parcels.shp",("PY_FULL_OW", "SHAPE@XY")) as cursor:    for row in cursor:        print("Parcel owner: {0} has a location of: {1}".format(row[0], row[1])) elapsed = (time.clock() - start) print("Execution time: " + str(elapsed)) You can check your work by examining the C:ArcpyBookcodeCh8GeometryToken.py solution file. Save the script. Run the script. You should see something similar to the following output. Note the execution time; your time will vary: Parcel owner: CITY OF AUSTIN ATTN REAL ESTATE DIVISION has a location of: (3110480.5197341456, 10070911.174956793) Parcel owner: CITY OF AUSTIN ATTN REAL ESTATE DIVISION has a location of: (3110670.413783513, 10070800.960865) Parcel owner: CITY OF AUSTIN has a location of: (3143925.0013213265, 10029388.97419636) Parcel owner: CITY OF AUSTIN % DOROTHY NELL ANDERSON ATTN BARRY LEE ANDERSON has a location of: (3134432.983822767, 10072192.047894118) Execution time: 9.08046185109 Now, we're going to measure the execution time if the entire geometry is returned instead of just the portion of the geometry that we need: Save a new copy of the script as C:ArcpyBookCh8GeometryTokenEntireGeometry.py. Change the SearchCursor() function to return the entire geometry using SHAPE@ instead of SHAPE@XY: with arcpy.da.SearchCursor("coa_parcels.shp",("PY_FULL_OW", "SHAPE@")) as cursor: You can check your work by examining the C:ArcpyBookcodeCh8GeometryTokenEntireGeometry.py solution file. Save and run the script. You should see the following output. Your time will vary from mine, but notice that the execution time is slower. In this case, it's only a little over a second slower, but we're only returning 2600 features. If the feature class were significantly larger, as many are, this would be amplified: Parcel owner: CITY OF AUSTIN ATTN REAL ESTATE DIVISION has a location of: <geoprocessing describe geometry object object at 0x06B9BE00> Parcel owner: CITY OF AUSTIN ATTN REAL ESTATE DIVISION has a location of: <geoprocessing describe geometry object object at 0x2400A700> Parcel owner: CITY OF AUSTIN has a location of: <geoprocessing describe geometry object object at 0x06B9BE00> Parcel owner: CITY OF AUSTIN % DOROTHY NELL ANDERSON ATTN BARRY LEE ANDERSON has a location of: <geoprocessing describe geometry object object at 0x2400A700> Execution time: 10.1211390896 How it works… A geometry token can be supplied as one of the field names supplied in the constructor for the cursor. These tokens are used to increase the performance of a cursor by returning only a portion of the geometry instead of the entire geometry. This can dramatically increase the performance of a cursor, particularly when you are working with large polyline or polygon datasets. If you only need specific properties of the geometry in your cursor, you should use these tokens. Inserting rows with InsertCursor You can insert a row into a table or feature class using an InsertCursor object. If you want to insert attribute values along with the new row, you'll need to supply the values in the order found in the attribute table. Getting ready The InsertCursor() function is used to create an InsertCursor object that allows you to programmatically add new records to feature classes and tables. The insertRow() method on the InsertCursor object adds the row. A row in the form of a list or tuple is passed into the insertRow() method. The values in the list must correspond to the field values defined when the InsertCursor object was created. Similar to instances that include other types of cursors, you can also limit the field names returned using the second parameter of the method. This function supports geometry tokens as well. The following code example illustrates how you can use InsertCursor to insert new rows into a feature class. Here, we insert two new wildfire points into the California feature class. The row values to be inserted are defined in a list variable. Then, an InsertCursor object is created, passing in the feature class and fields. Finally, the new rows are inserted into the feature class by using the insertRow() method: rowValues = [(Bastrop','N',3000,(-105.345,32.234)), ('Ft Davis','N', 456, (-109.456,33.468))] fc = "c:/data/wildfires.gdb/California" fields = ["FIRE_NAME", "FIRE_CONTAINED", "ACRES", "SHAPE@XY"] with arcpy.da.InsertCursor(fc, fields) as cursor: for row in rowValues:    cursor.insertRow(row) In this article, you will use InsertCursor to add wildfires retrieved from a .txt file into a point feature class. When inserting rows into a feature class, you will need to know how to add the geometric representation of a feature into the feature class. This can be accomplished by using InsertCursor along with two miscellaneous objects: Array and Point. In this exercise, we will add point features in the form of wildfire incidents to an empty point feature class. In addition to this, you will use Python file manipulation techniques to read the coordinate data from a text file. How to do it… We will be importing the North American wildland fire incident data from a single day in October, 2007. This data is contained in a comma-delimited text file containing one line for each fire incident on this particular day. Each fire incident has a latitude, longitude coordinate pair separated by commas along with a confidence value. This data was derived by automated methods that use remote sensing data to derive the presence or absence of a wildfire. Confidence values can range from 0 to 100. Higher numbers represent a greater confidence that this is indeed a wildfire: Open the file at C:ArcpyBookCh8Wildfire DataNorthAmericaWildfire_2007275.txt and examine the contents.You will notice that this is a simple comma-delimited text file containing the longitude and latitude values for each fire along with a confidence value. We will use Python to read the contents of this file line by line and insert new point features into the FireIncidents feature class located in the C:ArcpyBookCh8 WildfireDataWildlandFires.mdb personal geodatabase. Close the file. Open ArcCatalog. Navigate to C:ArcpyBookCh8WildfireData.You should see a personal geodatabase called WildlandFires. Open this geodatabase and you will see a point feature class called FireIncidents. Right now, this is an empty feature class. We will add features by reading the text file you examined earlier and inserting points. Right-click on FireIncidents and select Properties. Click on the Fields tab.The latitude/longitude values found in the file we examined earlier will be imported into the SHAPE field and the confidence values will be written to the CONFIDENCEVALUE field. Open IDLE and create a new script. Save the script to C:ArcpyBookCh8InsertWildfires.py. Import the arcpy modules: import arcpy Set the workspace: arcpy.env.workspace = "C:/ArcpyBook/Ch8/WildfireData/WildlandFires.mdb" Open the text file and read all the lines into a list: f = open("C:/ArcpyBook/Ch8/WildfireData/NorthAmericaWildfires_2007275.txt","r") lstFires = f.readlines() Start a try block: try: Create an InsertCursor object using a with block. Make sure you indent inside the try statement. The cursor will be created in the FireIncidents feature class: with arcpy.da.InsertCursor("FireIncidents",("SHAPE@XY","CONFIDENCEVALUE")) as cur: Create a counter variable that will be used to print the progress of the script: cntr = 1 Loop through the text file line by line using a for loop. Since the text file is comma-delimited, we'll use the Python split() function to separate each value into a list variable called vals. We'll then pull out the individual latitude, longitude, and confidence value items and assign them to variables. Finally, we'll place these values into a list variable called rowValue, which is then passed into the insertRow() function for the InsertCursor object, and we then print a message: for fire in lstFires:      if 'Latitude' in fire:        continue      vals = fire.split(",")      latitude = float(vals[0])      longitude = float(vals[1])      confid = int(vals[2])      rowValue = [(longitude,latitude),confid]      cur.insertRow(rowValue)      print("Record number " + str(cntr) + " written to feature class")      #arcpy.AddMessage("Record number" + str(cntr) + " written to feature class")      cntr = cntr + 1 Add the except block to print any errors that may occur: except Exception as e: print(e.message) Add a finally block to close the text file: finally: f.close() The entire script should appear as follows: import arcpy   arcpy.env.workspace = "C:/ArcpyBook/Ch8/WildfireData/WildlandFires.mdb" f = open("C:/ArcpyBook/Ch8/WildfireData/NorthAmericaWildfires_2007275.txt","r") lstFires = f.readlines() try: with arcpy.da.InsertCursor("FireIncidents", ("SHAPE@XY","CONFIDENCEVALUE")) as cur:    cntr = 1    for fire in lstFires:      if 'Latitude' in fire:        continue      vals = fire.split(",")      latitude = float(vals[0])      longitude = float(vals[1])      confid = int(vals[2])      rowValue = [(longitude,latitude),confid]      cur.insertRow(rowValue)      print("Record number " + str(cntr) + " written to feature class")      #arcpy.AddMessage("Record number" + str(cntr) + "       written to feature class")      cntr = cntr + 1 except Exception as e: print(e.message) finally: f.close() You can check your work by examining the C:ArcpyBookcodeCh8InsertWildfires.py solution file. Save and run the script. You should see messages being written to the output window as the script runs: Record number: 406 written to feature class Record number: 407 written to feature class Record number: 408 written to feature class Record number: 409 written to feature class Record number: 410 written to feature class Record number: 411 written to feature class Open ArcMap and add the FireIncidents feature class to the table of contents. The points should be visible, as shown in the following screenshot: You may want to add a basemap to provide some reference for the data. In ArcMap, click on the Add Basemap button and select a basemap from the gallery. How it works… Some additional explanation may be needed here. The lstFires variable contains a list of all the wildfires that were contained in the comma-delimited text file. The for loop will loop through each of these records one by one, inserting each individual record into the fire variable. We also include an if statement that is used to skip the first record in the file, which serves as the header. As I explained earlier, we then pull out the individual latitude, longitude, and confidence value items from the vals variable, which is just a Python list object and assign them to variables called latitude, longitude, and confid. We then place these values into a new list variable called rowValue in the order that we defined when we created InsertCursor. Thus, the latitude and longitude pair should be placed first followed by the confidence value. Finally, we call the insertRow() function on the InsertCursor object assigned to the cur variable, passing in the new rowValue variable. We close by printing a message that indicates the progress of the script and also create the except and finally blocks to handle errors and close the text file. Placing the file.close() method in the finally block ensures that it will execute and close the file even if there is an error in the previous try statement. Updating rows with UpdateCursor If you need to edit or delete rows from a table or feature class, you can use UpdateCursor. As is the case with InsertCursor, the contents of UpdateCursor can be limited through the use of a where clause. Getting ready The UpdateCursor() function can be used to either update or delete rows in a table or feature class. The returned cursor places a lock on the data, which will automatically be released if used inside a Python with statement. An UpdateCursor object is returned from a call to this method. The UpdateCursor object places a lock on the data while it's being edited or deleted. If the cursor is used inside a Python with statement, the lock will automatically be freed after the data has been processed. This hasn't always been the case. Previous versions of cursors were required to be manually released using the Python del statement. Once an instance of UpdateCursor has been obtained, you can then call the updateCursor() method to update records in tables or feature classes and the deleteRow() method can be used to delete a row. In this article, you're going to write a script that updates each feature in the FireIncidents feature class by assigning a value of poor, fair, good, or excellent to a new field that is more descriptive of the confidence values using an UpdateCursor. Prior to updating the records, your script will add the new field to the FireIncidents feature class. How to do it… Follow these steps to create an UpdateCursor object that will be used to edit rows in a feature class: Open IDLE and create a new script. Save the script to C:ArcpyBookCh8UpdateWildfires.py. Import the arcpy module: import arcpy Set the workspace: arcpy.env.workspace = "C:/ArcpyBook/Ch8/WildfireData/WildlandFires.mdb" Start a try block: try: Add a new field called CONFID_RATING to the FireIncidents feature class. Make sure to indent inside the try statement: arcpy.AddField_management("FireIncidents","CONFID_RATING", "TEXT","10") print("CONFID_RATING field added to FireIncidents") Create a new instance of UpdateCursor inside a with block: with arcpy.da.UpdateCursor("FireIncidents", ("CONFIDENCEVALUE","CONFID_RATING")) as cursor: Create a counter variable that will be used to print the progress of the script. Make sure you indent this line of code and all the lines of code that follow inside the with block: cntr = 1 Loop through each of the rows in the FireIncidents fire class. Update the CONFID_RATING field according to the following guidelines:     Confidence value 0 to 40 = POOR     Confidence value 41 to 60 = FAIR     Confidence value 61 to 85 = GOOD     Confidence value 86 to 100 = EXCELLENT This can be translated in the following block of code:    for row in cursor:      # update the confid_rating field      if row[0] <= 40:        row[1] = 'POOR'      elif row[0] > 40 and row[0] <= 60:        row[1] = 'FAIR'      elif row[0] > 60 and row[0] <= 85:        row[1] = 'GOOD'      else:        row[1] = 'EXCELLENT'      cursor.updateRow(row)                       print("Record number " + str(cntr) + " updated")      cntr = cntr + 1 Add the except block to print any errors that may occur: except Exception as e: print(e.message) The entire script should appear as follows: import arcpy   arcpy.env.workspace = "C:/ArcpyBook/Ch8/WildfireData/WildlandFires.mdb" try: #create a new field to hold the values arcpy.AddField_management("FireIncidents", "CONFID_RATING","TEXT","10") print("CONFID_RATING field added to FireIncidents") with arcpy.da.UpdateCursor("FireIncidents",("CONFIDENCEVALUE", "CONFID_RATING")) as cursor:    cntr = 1    for row in cursor:      # update the confid_rating field      if row[0] <= 40:        row[1] = 'POOR'      elif row[0] > 40 and row[0] <= 60:        row[1] = 'FAIR'      elif row[0] > 60 and row[0] <= 85:        row[1] = 'GOOD'      else:        row[1] = 'EXCELLENT'      cursor.updateRow(row)                       print("Record number " + str(cntr) + " updated")      cntr = cntr + 1 except Exception as e: print(e.message) You can check your work by examining the C:ArcpyBookcodeCh8UpdateWildfires.py solution file. Save and run the script. You should see messages being written to the output window as the script runs: Record number 406 updated Record number 407 updated Record number 408 updated Record number 409 updated Record number 410 updated Open ArcMap and add the FireIncidents feature class. Open the attribute table and you should see that a new CONFID_RATING field has been added and populated by UpdateCursor: When you insert, update, or delete data in cursors, the changes are permanent and can't be undone if you're working outside an edit session. However, with the new edit session functionality provided by ArcGIS 10.1, you can now make these changes inside an edit session to avoid these problems. We'll cover edit sessions soon. How it works… In this case, we've used UpdateCursor to update each of the features in a feature class. We first used the Add Field tool to add a new field called CONFID_RATING, which will hold new values that we assign based on values found in another field. The groups are poor, fair, good, and excellent and are based on numeric values found in the CONFIDENCEVALUE field. We then created a new instance of UpdateCursor based on the FireIncidents feature class, and returned the two fields mentioned previously. The script then loops through each of the features and assigns a value of poor, fair, good, or excellent to the CONFID_RATING field (row[1]), based on the numeric value found in CONFIDENCEVALUE. A Python if/elif/else structure is used to control the flow of the script based on the numeric value. The value for CONFID_RATING is then committed to the feature class by passing the row variable into the updateRow() method. Summary In this article we studied, how to retrieve features from a feature class with SerchCursor, filtering records with a where clause, improving cursr performance with geoetry tokens, inserting rows with InsertCursor and updating rows with UpdateCursor. Resources for Article: Further resources on this subject: Adding Graphics to the Map [article] Introduction to Mobile Web ArcGIS Development [article] Python functions – Avoid repeating code [article]
Read more
  • 0
  • 0
  • 7055

article-image-cross-platform-building
Packt
10 Aug 2015
11 min read
Save for later

Cross-platform Building

Packt
10 Aug 2015
11 min read
In this article by Karan Sequeira, author of the book Cocos2d-x Game Development Blueprints, we'll leverage the awesome aspect of Cocos2d-x to build one of our games on Android and Windows Phone 8! (For more resources related to this topic, see here.) Setting up the environment for Android At this point in the timeline of technological evolution, Android needs no introduction. This mobile operating system was acquired by Google, and it has reached far and wide across the globe. It is now one of the top choices for application developers and game developers. With octa-core CPUs and ever-powerful GPUs, the sheer power offered by Android devices is a motivating factor! While setting up the environment for Android, you have more choices than any other mobile development platform. Your workstation could be running any of the three major operating systems (Windows, Mac OS, or Linux) and you would be able to build to Android just fine. Since Android is not fussy about its build environment, developers mostly choose their work environment based on which other platforms they will be developing for. As such, you might choose to build for Android on a machine running Mac OS since you would be able to build for iOS and Android on the same machine. The same applies for a machine running Windows as well. You would be able to build for both Android and Windows Phone. Although building for Windows Phone 8 requires you to have at least Windows 8 installed. We will discuss more on that later. Let's begin listing down the various software required to set up the environment for Android. Java Development Kit 7+ Since you already know that Java is the programming language used within the Android SDK, you must ensure that you have the environment set up to compile and run Java files. So go ahead and download the Java Development Kit (JDK)version 6 or later. You can download and install a Standard Edition (SE) version from the page available at the following link: http://www.oracle.com/technetwork/java/javase/downloads/index.html Mac OS comes with JDK installed and as such, you won't have to follow this step if you're setting up your development environment on a Mac. The Android SDK Once you've downloaded JDK, it's time to download the Android SDK from the following URL: http://developer.android.com/sdk/index.html If you're installing the Android SDK on Windows, a custom installer is provided that will take care of downloading and setting up the required parts of the Android SDK for you. For other operating systems, you can choose to download the respective archive files and extract them at the location of your choice. Eclipse or the ADT bundle Eclipse is the most commonly used IDE when it comes to Android application development. You can choose to download a standard Eclipse IDE for Java developers and then install the ADT plugin into Eclipse, or you can download the ADT bundle, which is a specialized version of Eclipse with the ADT plugin preinstalled. At the time of writing this article, the Android developer site had already deprecated ADT in favor of Android Studio. As such, we will choose the former approach for setting up our environment in Eclipse. You can download and install the standard Eclipse IDE for Java Developers for your specific machine from the following URL: http://www.eclipse.org/downloads/ ADT plugin for Eclipse Once you've downloaded Eclipse, you must now install a custom plugin for Eclipse: Android Development Tools (ADT). Visit the following URL and follow the detailed instructions that will help you install the ADT plugin into Eclipse: http://developer.android.com/sdk/installing/installing-adt.html Once you've followed the instructions on the preceding page, you will need to inform Eclipse about the location of the Android SDK that you downloaded earlier. So, open up the Preferences page for Eclipse and go to the location where you've placed the Android SDK in the Android section. With that done, we can now fire up the SDK Manager to install a few more necessary pieces of software. To launch the Android SDK Manager, select Android SDK Manager from the Windows menu in Eclipse. The resultant window should look something like this: By default, you will see a whole lot of packages selected, out of which Android SDK Platform-tools and Android SDK Build-tools are necessary. From the rest, you must select at least one of the target Android platforms. An additional package will be required if you're target environment is Windows: Google USB Driver. It is located under the Extras list. I would suggest skipping downloading the documentation and samples. If you already have an Android device, I would go one step further and suggest you skip downloading the system images as well. However, if you don't have an Android device, you will need at least one system image so that you can at least test on an emulator. Once you've chosen from the various platforms needed, proceed to install the packages and you get a window like this: Now, you must select Accept License and click on the Install button to install the respective packages. Once these packages have been installed, you have to add their locations to the path variable on your respective machines. For Windows, modify your path variable (go to Properties | Advance Settings | Environment Variables) to include the following: ;E:Androidandroid-sdkplatform-tools For Mac OS, you can add the following line to the .bash_profile file found under the home directory: export PATH=$PATH:/Android/android-sdk/platform-tools/ The preceding line can also be added to the .bash_rc file found under the home directory on your Linux machine. At this point, you can use Eclipse for Android development. Installing Cygwin for Windows Developers working on Linux can skip this step as most Linux distributions come with the make utility. Also, developers working on Mac OS may download Xcode from the Mac App Store, which will install the make utility on their respective Macs. We need to install Cygwin on Windows specifically for the GNU make utility. So, go to the following URL and download the installer for Cygwin: http://www.cygwin.com/install.html Once you've run the .exe file that you downloaded and get a window like this, click on the Next button: The next window will ask how you would like to install the required packages. Here, select option Install from Internet and click on Next: The next window will ask where you would like to install Cygwin. I'd recommend leaving it at the default value unless you have a reason to change it. Proceed by clicking on Next. In the next window, you will be asked to specify a path where the installation can download the files it requires. You can fill in a suitable path of your choice in the box and click on Next. In the next window, you will be asked to specify your Internet connection. Leave it at the Direct Connection option and click on Next. In the next window, you will be asked to select a mirror location from where to download the installation files. Here, select the site that is geographically closest to you and click on Next. In the window that follows, expand the Devel section and search for make: The GNU version of the 'make' utility. Click on the Skip option to select this package. The version of the make utility that will be installed is now displayed in place of Skip. Your window should look something like this: You can now go ahead and click the Next button to begin the download and installation of the required packages. The window should look something like this: Once all the packages have been downloaded, click on Finish to close the installation. Now that we have the make utility installed, we can go ahead and download the Android NDK, which will actually build our entire C++ code base. The Android NDK To download the Android NDK for your respective development machine, navigate to the following URL: https://developer.android.com/tools/sdk/ndk/index.html Unzip the downloaded archive and place it in the same location as the Android SDK. We must now add an environment variable named NDK_ROOT that points to the root of the Android NDK. For Windows, add a new user variable NDK_ROOT with the location of the Android NDK on your filesystem as its value. You can do this by going to Properties | Advance Settings | Environment Variables. Once you've done that, the Environment Variables window should look something like this: I'm sure you noticed the value of the NDK_ROOT variable in the previous screenshot. The value of this variable is given in Unix style and depends on the Cygwin environment, since it will be accessed within a Cygwin bash shell while executing the build script for each Android project. Mac OS and Linux users can add the following line to their .bash_profile and .bashrc files, respectively: export NDK_ROOT=/Android/android-ndk-r10 We have now successfully completed setting up the environment to build our Cocos2d-x games on Android. To test this, open up a Cygwin bash terminal (for Windows) or a standard terminal (for Mac OS or Linux) and navigate to the Cocos2d-x test bed located inside the samples folder of your Cocos2d-x source. Now, navigate to the proj.android folder and run the build_native.sh file. This is what my Cygwin bash terminal looks like on a Windows 7 machine: If you've followed the aforementioned instructions correctly, the build_native.sh script will then go on to compile the C++ source files required by the TestCpp project and will result in a single shared object (.so) file in the libs folder within the proj.android folder. Creating an Android Virtual Device We're close to running the game, but we need to create an Android Virtual Device (AVD) before we proceed. Open up the Android Virtual Device Manager from the Windows menu and click on Create.   In the next window, fill in the required details as per your requirements and configuration and click OK. This is what my window looks like with everything filled in: From the Android Virtual Device Manager window, select the newly created AVD and click on Start to boot it. Building the tests on Android With an Android device that is ready to run our project, let's begin by first importing the project into Eclipse. Within Eclipse, select File | Import.... In the following window, select Existing Projects into Workspace under the General setting and click on Next: In the next window, browse to the proj.android folder under the cocos2d-x-2.2.5samplesCppTestCpp path and click on Finish: Once imported, you can find the TestCpp project under Package Explorer. It should look something like this: As you can see, there are a few errors with the project. If you look at the Problems view (Window | Show View | Problems) located on the bottom-half of Eclipse, you might see something like this: All these errors are due to the fact that the Android project for our game depends on Cocos2d-x's Android project for Android-specific functionality, things such as the actual OpenGL surface where everything is rendered, the music player, accelerometer functionality, and many more. So let's import the Android project for Cocos2d-x located inside the following path in your Cocos2d-x source bundle: cocos2d-x-2.2.5cocos2dxplatformandroid You can import it the same way you imported TestCpp. Once the project has been imported, it will be titled libcocos2dx in Package Explorer. Now, select Clean... from the Project menu; You will notice that when the clean operation has finished, the pumpkindefense dependency on libcocos2dx is taken care of and the project for pumpkindefense builds error-free. Running the tests on Android Running the tests is as simple as right-clicking on the TestCpp project in Package Explorer and selecting Run As | Android Application. It might take a bit more time running on an emulator as compared to an actual device, but ultimately you will have something like this: Summary In this article, you learned what necessary software components are needed to set up your workstation to build and run an Android native application. You had also set up an Android Virtual Device and ran the Cocos2d-x test bed application on it. Resources for Article: Further resources on this subject: Run Xcode Run [article] Creating Games with Cocos2d-x is Easy and 100 percent Free [article] Creating Cool Content [article]
Read more
  • 0
  • 0
  • 21598

article-image-introduction-wep
Packt
10 Aug 2015
4 min read
Save for later

An Introduction to WEP

Packt
10 Aug 2015
4 min read
In this article by Marco Alamanni, author of the book, Kali Linux Wireless Penetration Testing Essentials, has explained that the WEP protocol was introduced with the original 802.11 standard as a means to provide authentication and encryption to wireless LAN implementations. It is based on the RC4 (Rivest Cipher 4) stream cypher with a preshared secret key (PSK) of 40 or 104 bits, depending on the implementation. A 24 bit pseudo-random Initialization Vector (IV) is concatenated with the preshared key to generate the per-packet keystream used by RC4 for the actual encryption and decryption processes. Thus, the resulting keystream could be 64 or 128 bits long. (For more resources related to this topic, see here.) In the encryption phase, the keystream is XORed with the plaintext data to obtain the encrypted data, while in the decryption phase the encrypted data is XORed with the keystream to obtain the plaintext data. The encryption process is shown in the following diagram: Attacks against WEP First of all, we must say that WEP is an insecure protocol and has been deprecated by the Wi-Fi Alliance. It suffers from various vulnerabilities related to the generation of the keystreams, to the use of IVs and to the length of the keys. The IV is used to add randomness to the keystream, trying to avoid the reuse of the same keystream to encrypt different packets. This purpose has not been accomplished in the design of WEP, because the IV is only 24 bits long (with 2^24 = 16,777,216 possible values) and it is transmitted in clear-text within each frame. Thus, after a certain period of time (depending on the network traffic) the same IV, and consequently the same keystream, will be reused, allowing the attacker to collect the relative cypher texts and perform statistical attacks to recover the plain texts and the key. The first well-known attack against WEP was the Fluhrer, Mantin and Shamir (FMS) attack, back in 2001. The FMS attack relies on the way WEP generates the keystreams and on the fact that it also uses weak IVs to generate weak keystreams, making possible for an attacker to collect a sufficient number of packets encrypted with these keystreams, analyze them, and recover the key. The number of IVs to be collected to complete the FMS attack is about 250,000 for 40-bit keys and 1,500,000 for 104-bit keys. The FMS attack has been enhanced by Korek, improving its performances. Andreas Klein found more correlations between the RC4 keystream and the key than the ones discovered by Fluhrer, Mantin, and Shamir, that can used to crack the WEP key. In 2007, Pyshkin, Tews, and Weinmann (PTW) extended Andreas Klein's research and improved the FMS attack, significantly reducing the number of IVs needed to successfully recover the WEP key. Indeed, the PTW attack does not rely on weak IVs like the FMS attack does and is very fast and effective. It is able to recover a 104-bit WEP key with a success probability of 50 percent using less than 40,000 frames and with a probability of 95 percent with 85,000 frames. The PTW attack is the default method used by Aircrack-ng to crack WEP keys. Both the FMS and PTW attacks need to collect quite a large number of frames to succeed and can be conducted passively, sniffing the wireless traffic on the same channel of the target AP and capturing frames. The problem is that, in normal conditions, we will have to spend quite a long time to passively collect all the necessary packets for the attacks, especially with the FMS attack. To accelerate the process, the idea is to re-inject frames in the network to generate traffic in response so that we could collect the necessary IVs more quickly. A type of frame that is suitable for this purpose is the ARP request, because the AP broadcasts it and each time with a new IV. As we are not associated with the AP, if we send frames to it directly, they are discarded and a de-authentication frame is sent. Instead, we can capture ARP requests from associated clients and retransmit them to the AP. This technique is called the ARP Request Replay attack and is also adopted by Aircrack-ng for the implementation of the PTW attack. Summary In this article, we covered the WEP protocol, the attacks that have been developed to crack the keys. Resources for Article: Further resources on this subject: Kali Linux – Wireless Attacks [article] What is Kali Linux [article] Penetration Testing [article]
Read more
  • 0
  • 0
  • 9674

article-image-editing-uv-islands
Packt
10 Aug 2015
10 min read
Save for later

Editing the UV islands

Packt
10 Aug 2015
10 min read
In this article by Enrico Valenza, the author of Blender 3D Cookbook, we are going to join the two UV islands' halves together, in order to improve the final look of the texturing; we are also going to modify, if possible, a little of the island proportions in order to obtain a more regular flow of the UV vertices, and fix the distortions. We are going to the use the pin tool, which is normally used in conjunction with the Live Unwrap tool. (For more resources related to this topic, see here.) Getting ready First, we'll try to recalculate the unwrap of some of the islands by modifying the seams of the mesh. Before we start though, let's see if we can improve some of the visibility of the UV islands in the UV/Image Editor: Put the mouse cursor in the UV/Image Editor window and press the N key. In the Properties sidepanel that appears by pressing the N key on the right-hand side of the window, go to the Display subpanel and click on the Black or White button (depending on your preference) under the UV item. Check also the Smooth item box. Also, check the Stretch item, which even though it was made for a different purpose, can increase the visibility of the islands a lot. Press N again to get rid of the Properties sidepanel. All these options enabled should make the islands more easily readable in the UV/Image Editor window: The UV islands made more easily readable by the enabled items How to do it… Now we can start with the editing; initially, we are going to freeze the islands that we don't want to modify because their unwrap is either satisfactory, or we will deal with it later. So, perform the following steps: Press A to select all the islands, then by putting the mouse pointer on the two pelvis island halves and pressing Shift + L, multi-deselect them; press the P key to pin the remaining selected UV islands and then A to deselect everything: To the right-hand side, the pinned UV islands Zoom in on the islands of the pelvis, select both the left and right outer edge-loops, as shown in the following left image, and press P to pin them. Go to the 3D view and clear only the front part of the median seam on the pelvis. To do this, start to clear the seam from the front edges, go down and stop where it crosses the horizontal seam that passes the bottom part of the groin and legs, and leave the back part of the vertical median seam still marked: Pinning the extreme vertices in the UV/Image Editor, and editing the seam on the mesh Go into Face selection mode and select all the faces of the pelvis; put the mouse pointer in the 3D view and press U | Unwrap (alternatively, go into the UV/Image Editor and press E): Unwrapping again with the pinning and a different seam The island will keep the previous position because of the pinned edges, and is now unwrapped as one single piece (with the obvious exception of the seam on the back). We won't modify the pelvis island any further, so select all its vertices and press P to pin all of them and then deselect them. Press A in the 3D view to select all the faces of the mesh and make all the islands visible in the UV/Image Editor. Note that they are all pinned at the moment, so just select the vertices you want to unpin (Alt + P) in the islands of the tongue and inner mouth. Then, clear the median seam in the corresponding pieces on the mesh, and press E again: Re-unwrapping the tongue and inner mouth areas Select the UV vertices of the resulting islands and unpin them all; next, pin just one vertex at the top of the islands and one at the bottom, and unwrap again. This will result in a more organically distributed unwrap of the parts: Re-unwrapping again with a different pinning Select all the faces of the mesh, and then all the islands in the UV/Image Editor window. Press Ctrl + A to average their relative size and adjust their position in the default tile space: The rearranged UV islands Now, let's work on the head piece that, as in every character, should be the most important and well-finished piece. At the moment, the face is made using two separate islands; although this won't be visible in the final textured rendering of our character, it's always better, if possible, to join them in order to have a single piece, especially in the front mesh faces. Due to the elongated snout of the character, if we were to unwrap the head as a single piece simply without the median seam, we wouldn't get a nice evenly mapped result, so we must divide the whole head into more pieces. Actually, we can take advantage of the fact that the Gidiosaurus is wearing a helmet and that most of the head will be covered by it; this allows us to easily split the face from the rest of the mesh, hiding the seams under the helmet. Go into Edge selection mode and mark the seams, dividing the face from the cranium and neck as shown in the following screenshots. Select the crossing edge-loops, and then clear the unnecessary parts: New seams for the character's head part 1 Also clear the median seam in the upper face part, and under the seam on the bottom jaw, leaving it only on the front mandible and on the back of the cranium and neck: New seams for the character's head part 2 Go in the Face selection mode and select only the face section of the mesh, and then press E to unwrap. The new unwrap comes upside down, so select all the UV vertices and rotate the island by 180 degrees: The character's face unwrapped Select the cranium/neck section on the mesh and repeat the process: The rest of the head mesh unwrapped as a whole piece Now, select all the faces of the mesh and all the islands in the UV/Image Editor, and press Ctrl + A to average their reciprocal size. Once again, adjust the position of the islands inside the UV tile (Ctrl + P to automatically pack them inside the available space, and then tweak their position, rotation, and scale): The character's UV islands packed inside the default U0/V0 tile space How it works… Starting from the UV unwrap, we improved some of the islands by joining together the halves representing common mesh parts. When doing this, we tried to retain the already good parts of the unwrap by pinning the UV vertices that we didn't want to modify; this way, the new unwrap process was forced to calculate the position of the unpinned vertices using the constraints of the pinned ones (pelvis, tongue, and inner mouth). In other cases, we totally cleared the old seams on the model and marked new ones, in order to have a completely new unwrap of the mesh part (the head), we also used the character furniture (such as the armor) to hide the seams (which in any case, won't be visible at all). There's more… At this point, looking at the UV/Image Editor window containing the islands, it's evident that if we want to keep several parts in proportion to each other, some of the islands are a little too small to give a good amount of detail when texturing; for example, the Gidiosaurus's face. A technique for a good unwrap that is the current standard in the industry is UDIM UV Mapping, which means U-Dimension; basically, after the usual unwrap, the islands are scaled bigger and placed outside the default U0/V0 tile space. Look at the following screenshots, showing the Blender UV/Image Editor window:   The default U0/V0 tile space and the possible consecutive other tile spaces On the left-hand side, you can see, highlighted with red lines, the single UV tile that at present is the standard for Blender, which is identified by the UV coordinates 0 and 0: that is, U (horizontal) = 0 and V (vertical) = 0. Although not visible in the UV/Image Editor window, all the other possible consecutive tiles can be identified by the corresponding UV coordinates, as shown on the right-hand side of the preceding screenshot (again, highlighted with red lines). So, adjacent to the tile U0/V0, we can have the row with the tiles U1/V0, U2/V0, and so on, but we can also go upwards: U0/V1, U1/V1, U2/V1, and so on. To help you identify the tiles, Blender will show you the amount of pixels and also the number of tiles you are moving the islands in the toolbar of the UV/Image Editor window. In the following screenshot, the arm islands have been moved horizontally (on the negative x axis) by -3072.000 pixels; this is correct because that's exactly the X size of the grid image. In fact, in the toolbar of the UV/Image Editor window, while moving the islands we can read D: -3072.000 (pixels) and (inside brackets) 1.0000 (tile) along X; effectively, 3072 pixels = 1 tile.   Moving the arm islands to the U1/V0 tile space When moving UV islands from tile to tile, remember to check that the Constrain to Image Bounds item in the UVs menu on the toolbar of the UV/Image Editor window is disabled; also, enabling the Normalized item inside the Display subpanel under the N key Properties sidepanel of the same editor window will display the UV coordinates from 0.0 to 1.0, rather than in pixels. More, pressing the Ctrl key while moving the islands will constrain the movement to intervals, making it easy to translate them to exactly 1 tile space. Because at the moment Blender doesn't support the UDIM UV Mapping standard, simply moving an island outside the default U0/V0 tile, for example to U1/V0, will repeat the image you loaded in the U0/V0 tile and on the faces associated with the moved islands. To solve this, it's necessary, after moving the islands, to assign a different material, if necessary with its own different image textures, to each group of vertices/faces associated with each tile space. So, if you shared your islands over 4 tiles, you need to assign 4 different materials to your object, and each material must load the proper image texture. The goal of this process is obviously to obtain bigger islands mapped with bigger texture images, by selecting all the islands, scaling them bigger together using the largest ones as a guide, and then tweaking their position and distribution. One last thing: it is also better to unwrap the corneas and eyes (which are separate objects from the Gidiosaurus body mesh) and add their islands to the tiles where you put the face, mouth, teeth, and so on (use the Draw Other Objects tool in the View menu of the UV/Image Editor window to also show the UV islands of the other nonjoined unwrapped objects):   UV islands unwrapped, following the UDIM UV Mapping standard In our case, we assigned the Gidiosaurus body islands to 5 different tiles, U0/V0, U1/V0, U2/V0, U0/V1, and U1/V1, so we'll have to assign 5 different materials. Note that for exposition purposes only, in the preceding screenshot, you can see the cornea and eye islands together with the Gidiosaurus body islands because I temporarily joined the objects; however, it's usually better to maintain the eyes and corneas as separate objects from the main body. Summary In this article, we saw how we can work with UV islands. Resources for Article: Further resources on this subject: Working with Blender [article] Blender Engine : Characters [article] Blender 2.5: Rigging the Torso [article]
Read more
  • 0
  • 0
  • 12293

article-image-share-and-share-alike
Packt
10 Aug 2015
13 min read
Save for later

Share and Share Alike

Packt
10 Aug 2015
13 min read
In this article by Kevin Harvey, author of the book Test-Driven Development with Django, we'll expose the data in our application via a REST API. As we do, we'll learn: The importance of documentation in the API development process How to write functional tests for API endpoints API patterns and best practices (For more resources related to this topic, see here.) It's an API world, we're just coding in it It's very common nowadays to include a public REST API in your web project. Exposing your services or data to the world is generally done for one of two reasons: You've got interesting data, and other developers might want to integrate that information into a project they're working on You're building a secondary system that you expect your users to interact with, and that system needs to interact with your data (that is, a mobile or desktop app, or an AJAX-driven front end) We've got both reasons in our application. We're housing novel, interesting data in our database that someone might want to access programmatically. Also, it would make sense to build a desktop application that could interact with a user's own digital music collection so they could actually hear the solos we're storing in our system. Deceptive simplicity The good news is that there are some great options for third-party plugins for Django that allow you to build a REST API into an existing application. The bad news is that the simplicity of adding one of these packages can let you go off half-cocked, throwing an API on top of your project without a real plan for it. If you're lucky, you'll just wind up with a bird's nest of an API: inconsistent URLs, wildly varying payloads, and difficult authentication. In the worst-case scenario, your bolt-on API exposes data you didn't intend to make public and wind up with a self-inflicted security issue. Never forget that an API is sort of invisible. Unlike traditional web pages, where bugs are very public and easy to describe, API bugs are only visible to other developers. Take special care to make sure your API behaves exactly as intended by writing thorough documentation and tests to make sure you've implemented it correctly. Writing documentation first "Documentation is king." - Kenneth Reitz If you've spent any time at all working with Python or Django, you know what good documentation looks like. The Django folks in particular seem to understand this well: the key to getting developers to use your code is great documentation. In documenting an API, be explicit. Most of your API methods' docs should take the form of "if you send this, you will get back this", with real-world examples of input and output. A great side effect of prewriting documentation is that it makes the intention of your API crystal clear. You're allowing yourself to conjure up the API from thin air without getting bogged down in any of the details, so you can get a bird's-eye view of what you're trying to accomplish. Your documentation will keep you oriented throughout the development process. Documentation-Driven testing Once you've got your documentation done, testing is simply a matter of writing test cases that match up with what you've promised. The actions of the test methods exercise HTTP methods, and your assertions check the responses. Test-Driven Development really shines when it comes to API development. There are great tools for sending JSON over the wire, but properly formatting JSON can be a pain, and reading it can be worse. Enshrining test JSON in test methods and asserting they match the real responses will save you a ton of headache. More developers, more problems Good documentation and test coverage are exponentially more important when two groups are developing in tandem—one on the client application and one on the API. Changes to an API are hard for teams like this to deal with, and should come with a lot of warning (and apologies). If you have to make a change to an endpoint, it should break a lot of tests, and you should methodically go and fix them all. What's more, no one feels the pain of regression bugs like the developer of an API-consuming client. You really, really, really need to know that all the endpoints you've put out there are still going to work when you add features or refactor. Building an API with Django REST framework Now that you're properly terrified of developing an API, let's get started. What sort of capabilities should we add? Here are a couple possibilities: Exposing the Album, Track, and Solo information we have Creating new Solos or updating existing ones Initial documentation In the Python world it's very common for documentation to live in docstrings, as it keeps the description of how to use an object close to the implementation. We'll eventually do the same with our docs, but it's kind of hard to write a docstring for a method that doesn't exist yet. Let's open up a new Markdown file API.md, right in the root of the project, just to get us started. If you've never used Markdown before, you can read an introduction to GitHub's version of Markdown at https://help.github.com/articles/markdown-basics/. Here's a sample of what should go in API.md. Have a look at https://github.com/kevinharvey/jmad/blob/master/API.md for the full, rendered version. ...# Get a Track with Solos* URL: /api/tracks/<pk>/* HTTP Method: GET## Example Response{"name": "All Blues","slug": "all-blues","album": {"name": "Kind of Blue","url": "http://jmad.us/api/albums/2/"},"solos": [{"artist": "Cannonball Adderley","instrument": "saxophone","start_time": "4:05","end_time": "6:04","slug": "cannonball-adderley","url": "http://jmad.us/api/solos/281/"},...]}# Add a Solo to a Track* URL: /api/solos/* HTTP Method: POST## Example Request{"track": "/api/tracks/83/","artist": "Don Cherry","instrument": "cornet","start_time": "2:13","end_time": "3:54"}## Example Response{"url": "http://jmad.us/api/solos/64/","artist": "Don Cherry","slug": "don-cherry","instrument": "cornet","start_time": "2:13","end_time": "3:54","track": "http://jmad.us/api/tracks/83/"} There's not a lot of prose, and there needn't be. All we're trying to do is layout the ins and outs of our API. It's important at this point to step back and have a look at the endpoints in their totality. Is there enough of a pattern that you can sort of guess what the next one is going to look like? Does it look like a fairly straightforward API to interact with? Does anything about it feel clunky? Would you want to work with this API by yourself? Take time to think through any weirdness now before anything gets out in the wild. $ git commit -am 'Initial API Documentation'$ git tag -a ch7-1-init-api-docs Introducing Django REST framework Now that we've got some idea what we're building, let's actually get it going. We'll be using Django REST Framework (http://www.django-rest-framework.org/). Start by installing it in your environment: $ pip install djangorestframework Add rest_framework to your INSTALLED_APPS in jmad/settings.py: INSTALLED_APPS = (...'rest_framework') Now we're ready to start testing. Writing tests for API endpoints While there's no such thing as browser-based testing for an external API, it is important to write tests that cover its end-to-end processing. We need to be able to send in requests like the ones we've documented and confirm that we receive the responses our documentation promises. Django REST Framework (DRF from here on out) provides tools to help write tests for the application functionality it provides. We'll use rest_framework.tests.APITestCase to write functional tests. Let's kick off with the list of albums. Convert albums/tests.py to a package, and add a test_api.py file. Then add the following: from rest_framework.test import APITestCasefrom albums.models import Albumclass AlbumAPITestCase(APITestCase):def setUp(self):self.kind_of_blue = Album.objects.create(name='Kind of Blue')self.a_love_supreme = Album.objects.create(name='A Love Supreme')def test_list_albums(self):"""Test that we can get a list of albums"""response = self.client.get('/api/albums/')self.assertEqual(response.status_code, 200)self.assertEqual(response.data[0]['name'],'A Love Supreme')self.assertEqual(response.data[1]['url'],'http://testserver/api/albums/1/') Since much of this is very similar to other tests that we've seen before, let's talk about the important differences: We import and subclass APITestCase, which makes self.client an instance of rest_framework.test.APIClient. Both of these subclass their respective django.test counterparts add a few niceties that help in testing APIs (none of which are showcased yet). We test response.data, which we expect to be a list of Albums. response.data will be a Python dict or list that corresponds to the JSON payload of the response. During the course of the test, APIClient (a subclass of Client) will use http://testserver as the protocol and hostname for the server, and our API should return a host-specific URI. Run this test, and we get the following: $ python manage.py test albums.tests.test_apiCreating test database for alias 'default'...F=====================================================================FAIL: test_list_albums (albums.tests.test_api.AlbumAPITestCase)Test that we can get a list of albums---------------------------------------------------------------------Traceback (most recent call last):File "/Users/kevin/dev/jmad-project/jmad/albums/tests/test_api.py",line 17, in test_list_albumsself.assertEqual(response.status_code, 200)AssertionError: 404 != 200---------------------------------------------------------------------Ran 1 test in 0.019sFAILED (failures=1) We're failing because we're getting a 404 Not Found instead of a 200 OK status code. Proper HTTP communication is important in any web application, but it really comes in to play when you're using AJAX. Most frontend libraries will properly classify responses as successful or erroneous based on the status code: making sure the code are on point will save your frontend developers friends a lot of headache. We're getting a 404 because we don't have a URL defined yet. Before we set up the route, let's add a quick unit test for routing. Update the test case with one new import and method: from django.core.urlresolvers import resolve...def test_album_list_route(self):"""Test that we've got routing set up for Albums"""route = resolve('/api/albums/')self.assertEqual(route.func.__name__, 'AlbumViewSet') Here, we're just confirming that the URL routes to the correct view. Run it: $ python manage.py testalbums.tests.test_api.AlbumAPITestCase.test_album_list_route...django.core.urlresolvers.Resolver404: {'path': 'api/albums/','tried': [[<RegexURLResolver <RegexURLPattern list> (admin:admin)^admin/>], [<RegexURLPattern solo_detail_view^recordings/(?P<album>[w-]+)/(?P<track>[w-]+)/(?P<artist>[w-]+)/$>], [<RegexURLPattern None ^$>]]}---------------------------------------------------------------------Ran 1 test in 0.003sFAILED (errors=1) We get a Resolver404 error, which is expected since Django shouldn't return anything at that path. Now we're ready to set up our URLs. API routing with DRF's SimpleRouter Take a look at the documentation for routers at http://www.django-rest-framework.org/api-guide/routers/. They're a very clean way of setting up URLs for DRF-powered views. Update jmad/urls.py like so: ...from rest_framework import routersfrom albums.views import AlbumViewSetrouter = routers.SimpleRouter()router.register(r'albums', AlbumViewSet)urlpatterns = [# Adminurl(r'^admin/', include(admin.site.urls)),# APIurl(r'^api/', include(router.urls)),# Appsurl(r'^recordings/(?P<album>[w-]+)/(?P<track>[w-]+)/(?P<artist>[w-]+)/$','solos.views.solo_detail',name='solo_detail_view'),url(r'^$', 'solos.views.index'),] Here's what we changed: We created an instance of SimpleRouter and used the register method to set up a route. The register method has two required arguments: a prefix to build the route methods from, and something called a viewset. Here we've supplied a non-existent class AlbumViewSet, which we'll come back to later. We've added a few comments to break up our urls.py, which was starting to look a little like a rat's nest. The actual API URLs are registered under the '^api/' path using Django's include function. Run the URL test again, and we'll get ImportError for AlbumViewSet. Let's add a stub to albums/views.py: class AlbumViewSet():pass Run the test now, and we'll start to see some specific DRF error messages to help us build out our view: $ python manage.py testalbums.tests.test_api.AlbumAPITestCase.test_album_list_routeCreating test database for alias 'default'...F...File "/Users/kevin/.virtualenvs/jmad/lib/python3.4/sitepackages/rest_framework/routers.py", line 60, in registerbase_name = self.get_default_base_name(viewset)File "/Users/kevin/.virtualenvs/jmad/lib/python3.4/sitepackages/rest_framework/routers.py", line 135, inget_default_base_nameassert queryset is not None, ''base_name' argument not specified,and could ' AssertionError: 'base_name' argument not specified, and could notautomatically determine the name from the viewset, as it does nothave a '.queryset' attribute. After a fairly lengthy output, the test runner tells us that it was unable to get base_name for the URL, as we did not specify the base_name in the register method, and it couldn't guess the name because the viewset (AlbumViewSet) did not have a queryset attribute. In the router documentation, we came across the optional base_name argument for register (as well as the exact wording of this error). You can use that argument to control the name your URL gets. However, let's keep letting DRF do its default behavior. We haven't read the documentation for viewsets yet, but we know that a regular Django class-based view expects a queryset parameter. Let's stick one on AlbumViewSet and see what happens: from .models import Albumclass AlbumViewSet():queryset = Album.objects.all() Run the test again, and we get: django.core.urlresolvers.Resolver404: {'path': 'api/albums/','tried': [[<RegexURLResolver <RegexURLPattern list> (admin:admin)^admin/>], [<RegexURLPattern solo_detail_view^recordings/(?P<album>[w-]+)/(?P<track>[w-]+)/(?P<artist>[w-]+)/$>], [<RegexURLPattern None ^$>]]}---------------------------------------------------------------------Ran 1 test in 0.011sFAILED (errors=1) Huh? Another 404 is a step backwards. What did we do wrong? Maybe it's time to figure out what a viewset really is. Summary In this article, we covered basic API design and testing patterns, including the importance of documentation when developing an API. In doing so, we took a deep dive into Django REST Framework and the utilities and testing tools available in it. Resources for Article: Further resources on this subject: Test-driven API Development with Django REST Framework [Article] Adding a developer with Django forms [Article] Code Style in Django [Article]
Read more
  • 0
  • 0
  • 2211
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-securing-openstack-networking
Packt
10 Aug 2015
10 min read
Save for later

Securing OpenStack Networking

Packt
10 Aug 2015
10 min read
In this article by Fabio Alessandro Locati, author of the book OpenStack Cloud Security, you will learn about the importance of firewall, IDS, and IPS. You will also learn about Generic Routing Encapsulation, VXLAN. (For more resources related to this topic, see here.) The importance of firewall, IDS, and IPS The security of a network can and should be achieved in multiple ways. Three components that are critical to the security of a network are: Firewall Intrusion detection system (IDS) Intrusion prevention system (IPS) Firewall Firewalls are systems that control traffic passing through them based on rules. This can seem something like a router, but they are very different. The router allows communication between different networks while the firewall limits communication between networks and hosts. The root of this confusion may occur because very often the router will have the firewall functionality and vice versa. Firewalls need to be connected in a series to your infrastructure. The first paper on the firewall technology appeared in 1988 and designed the packet filter firewall. This kind of firewall is often known as first generation firewall. This kind of firewall analyzes the packages passing through and if the package matches a rule, the firewall will act accordingly to that rule. This firewall will analyze each package by itself and will not consider other aspects such as other packages. It works on the first three layers of the OSI model with very few features using layer 4 specifically to check port numbers and protocols (UDP/TCP). First generation firewalls are still in use, because in a lot of situations, to do the job properly and are cheap and secure. Examples of typical filtering those firewalls prohibit (or allow) to IPs of certain classes (or specific IPs), to access certain IPs, or allow traffic to a specific IP only on specific ports. There are no known attacks to those kind of firewalls, but specific models can have specific bugs that can be exploited. In 1990, a new generation of firewall appeared. The initial name was circuit-level gateway, but today it is far more commonly known as stateful firewalls or second generation firewall. These firewalls are able to understand when connections are being initialized and closed so that the firewall comes to know what is the current state of a connection when a package arrives. To do so, this kind of firewall uses the first four layers of the networking stack. This allows the firewall to drop all packages that are not establishing a new connection or are in an already established connection. These firewalls are very powerful with the TCP protocol because it has states, while they have very small advantages compared to first generation firewalls handling UDP or ICMP packages, since those packages travel with no connection. In these cases, the firewall sets the connection as established; only the first valid package passes through and closes it after the connection times out. Performance-wise, stateful firewall can be faster than packet firewall because if the package is part of an active connection, no further test will be performed against that package. These kinds of firewalls are more susceptible to bugs in their code since reading more about the package makes it easier to exploit. Also, on many devices, it is possible to open connections (with SYN packages) until the firewall is saturated. In such cases, the firewall usually downgrades itself as a simple router allowing all traffic to pass through it. In 1991, improvements were made to the stateful firewall allowing it to understand more about the protocol of the package it was evaluating. The firewalls of this kind before 1994 had major problems, such as working as a proxy that the user had to interact with. In 1994, the first application firewall, as we know it, was born doing all its job completely transparently. To be able to understand the protocol, this kind of firewall requires an understanding of all seven layers of the OSI model. As for security, the same as the stateful firewall does apply to the application firewall as well. Intrusion detection system (IDS) IDSs are systems that monitor the network traffic looking for policy violation and malicious traffic. The goal of the IDS is not to block malicious activity, but instead to log and report them. These systems act in a passive mode, so you'll not see any traffic coming from them. This is very important because it makes them invisible to attackers so you can gain information about the attack, without the attacker knowing. IDSs need to be connected in parallel to your infrastructure. Intrusion prevention system (IPS) IPSs are sometimes referred to as Intrusion Detection and Prevention Systems (IDPS), since they are IDS that are also able to fight back malicious activities. IPSs have greater possibility to act than IDSs. Other than reporting, like IDS, they can also drop malicious packages, reset the connection, and block the traffic from the offending IP address. IPSs need to be connected in series to your infrastructure. Generic Routing Encapsulation (GRE) GRE is a Cisco tuning protocol that is difficult to position in the OSI model. The best place for it to be is between layers 2 and 3. Being above layer 2 (where VLANs are), we can use GRE inside VLAN. We will not go deep into the technicalities of this protocol. I'd like to focus more on the advantages and disadvantages it has over VLAN. The first advantage of (extended) GRE over VLAN is scalability. In fact, VLAN is limited to 4,096, while GRE tunnels do not have this limitation. If you are running a private cloud and you are working in a small corporation, 4,096 networks could be enough, but will definitely not be enough if you work for a big corporation or if you are running a public cloud. Also, unless you use VTP for your VLANs, you'll have to add VLANs to each network device, while GREs don't need this. You cannot have more than 4,096 VLANs in an environment. The second advantage is security. Since you can deploy multiple GRE tunnels in a single VLAN, you can connect a machine to a single VLAN and multiple GRE networks without the risks that come with putting a port in trunking that is needed to bring more VLANs in the same physical port. For these reasons, GRE has been a very common choice in a lot of OpenStack clusters deployed up to OpenStack Havana. The current preferred networking choice (since Icehouse) is Virtual Extensible LAN (VXLAN). VXLAN VXLAN is a network virtualization technology whose specifications have been originally created by Arista Networks, Cisco, and VMWare, and many other companies have backed the project. Its goal is to offer a standardized overlay encapsulation protocol and it was created because the standard VLAN were too limited for the current cloud needs and the GRE protocol was a Cisco protocol. It works using layer 2 Ethernet frames within layer 4 UDP packages on port 4789. As for the maximum number of networks, the limit is 16 million logical networks. Since the Icehouse release, the suggested standard for networking is VXLAN. Flat network versus VLAN versus GRE in OpenStack Quantum In OpenStack Quantum, you can decide to use multiple technologies for your networks: flat network, VLAN, GRE, and the most recent, VXLAN. Let's discuss them in detail: Flat network: It is often used in private clouds since it is very easy to set up. The downside is that any virtual machine will see any other virtual machines in our cloud. I strongly discourage people from using this network design because it's unsafe, and in the long run, it will have problems, as we have seen earlier. VLAN: It is sometimes used in bigger private clouds and sometimes even in small public clouds. The advantage is that many times you already have a VLAN-based installation in your company. The major disadvantages are the need to trunk ports for each physical host and the possible problems in propagation. I discourage this approach, since in my opinion, the advantages are very limited while the disadvantages are pretty strong. VXLAN: It should be used in any kind of cloud due to its technical advantages. It allows a huge number of networks, its way more secure, and often eases debugging. GRE: Until the Havana release, it was the suggested protocol, but since the Icehouse release, the suggestion has been to move toward VXLAN, where the majority of the development is focused. Design a secure network for your OpenStack deployment As for the physical infrastructure, we have to design it securely. We have seen that the network security is critical and that there a lot of possible attacks in this realm. Is it possible to design a secure environment to run OpenStack? Yes it is, if you remember a few rules: Create different networks, at the very least for management and external data (this network usually already exists in your organization and is the one where all your clients are) Never put ports on trunking mode if you use VLANs in your infrastructure, otherwise physically separated networks will be needed The following diagram is an example of how to implement it: Here, the management, tenant external networks could be either VLAN or real networks. Remember that to not use VLAN trunking, you need at least the same amount of physical ports as of VLAN, and the machine has to be subscribed to avoid port trunking that can be a huge security hole. A management network is needed for the administrator to administer the machines and for the OpenStack services to speak to each other. This network is critical, since it may contain sensible data, and for this reason, it has to be disconnected from other networks, or if not possible, have very limited connectivity. The external network is used by virtual machines to access the Internet (and vice versa). In this network, all machines will need an IP address reachable from the Web. The tenant network, sometimes even called internal or guest network is the network where the virtual machines can communicate with other virtual machines in the same cloud. This network, in some deployment cases, can be merged with the external network, but this choice has some security drawbacks. The API network is used to expose OpenStack APIs to the users. This network requires IP addresses reachable from the Web, and for this reason, is often merged into the external network. There are cases where provider networks are needed to connect tenant networks to existing networks outside the OpenStack cluster. Those networks are created by the OpenStack administrator and map directly to an existing physical network in the data center. Summary In this article, we have seen how networking works, which attacks we can expect, and how we can counter them. Also, we have seen how to implement a secure deployment of OpenStack Networking. Resources for Article: Further resources on this subject: Cloud distribution points [Article] Photo Stream with iCloud [Article] Integrating Accumulo into Various Cloud Platforms [Article]
Read more
  • 0
  • 0
  • 12711

article-image-sending-and-syncing-data
Packt
10 Aug 2015
4 min read
Save for later

Sending and Syncing Data

Packt
10 Aug 2015
4 min read
This article, by Steven F. Daniel, author of the book, Android Wearable Programming, will provide you with the background and understanding of how you can effectively build applications that communicate between the Android handheld device and the Android wearable. Android Wear comes with a number of APIs that will help to make communicating between the handheld and the wearable a breeze. We will be learning the differences between using MessageAPI, which is sometimes referred to as a "fire and forget" type of message, and DataLayerAPI that supports syncing of data between a handheld and a wearable, and NodeAPI that handles events related to each of the local and connected device nodes. (For more resources related to this topic, see here.) Creating a wearable send and receive application In this section, we will take a look at how to create an Android wearable application that will send an image and a message, and display this on our wearable device. In the next sections, we will take a look at the steps required to send data to the Android wearable using DataAPI, NodeAPI, and MessageAPIs. Firstly, create a new project in Android Studio by following these simple steps: Launch Android Studio, and then click on the File | New Project menu option. Next, enter SendReceiveData for the Application name field. Then, provide the name for the Company Domain field. Now, choose Project location and select where you would like to save your application code: Click on the Next button to proceed to the next step. Next, we will need to specify the form factors for our phone/tablet and Android Wear devices using which our application will run. On this screen, we will need to choose the minimum SDK version for our phone/tablet and Android Wear. Click on the Phone and Tablet option and choose API 19: Android 4.4 (KitKat) for Minimum SDK. Click on the Wear option and choose API 21: Android 5.0 (Lollipop) for Minimum SDK: Click on the Next button to proceed to the next step. In our next step, we will need to add Blank Activity to our application project for the mobile section of our app. From the Add an activity to Mobile screen, choose the Add Blank Activity option from the list of activities shown and click on the Next button to proceed to the next step: Next, we need to customize the properties for Blank Activity so that it can be used by our application. Here we will need to specify the name of our activity, layout information, title, and menu resource file. From the Customize the Activity screen, enter MobileActivity for Activity Name shown and click on the Next button to proceed to the next step in the wizard: In the next step, we will need to add Blank Activity to our application project for the Android wearable section of our app. From the Add an activity to Wear screen, choose the Blank Wear Activity option from the list of activities shown and click on the Next button to proceed to the next step: Next, we need to customize the properties for Blank Wear Activity so that our Android wearable can use it. Here we will need to specify the name of our activity and the layout information. From the Customize the Activity screen, enter WearActivity for Activity Name shown and click on the Next button to proceed to the next step in the wizard:   Finally, click on the Finish button and the wizard will generate your project and after a few moments, the Android Studio window will appear with your project displayed. Summary In this article, we learned about three new APIs, DataAPI, NodeAPI, and MessageAPIs, and how we can use them and their associated methods to transmit information between the handheld mobile and the wearable. If, for whatever reason, the connected wearable node gets disconnected from the paired handheld device, the DataApi class is smart enough to try sending again automatically once the connection is reestablished. Resources for Article: Further resources on this subject: Speeding up Gradle builds for Android [article] Saying Hello to Unity and Android [article] Testing with the Android SDK [article]
Read more
  • 0
  • 0
  • 7490

article-image-bootstrap-box
Packt
07 Aug 2015
6 min read
Save for later

Bootstrap in a Box

Packt
07 Aug 2015
6 min read
In this article written by Snig Bhaumik, author of the book Bootstrap Essentails, we explain the concept of Bootstrap, responsive design patterns, navigation patterns, and the different components that are included in Bootstrap. (For more resources related to this topic, see here.) Responsive design patterns Here are the few established and well-adopted patterns in Responsive Web Design: Fluid design: This is the most popular and easiest option for responsive design. In this pattern, larger screen multiple columns layout renders as a single column in a smaller screen in absolutely same sequence. Column drop: In this pattern also, the page gets rendered in a single column; however, the order of blocks gets altered. That means, if a content block is visible first in order in case of a larger screen, that might be rendered as second or third in case of a smaller screen. Layout shifter: This is a complex but powerful pattern where the whole layout of the screen contents gets altered in case of a smaller screen. This means that you need to develop different page layouts for large, medium, and small screens. Navigation patterns You should take care of the following things while designing a responsive web page. These are essentially the major navigational elements that you would concentrate on while developing a mobile friendly and responsive website: Menu bar Navigation/app bar Footer Main container shell Images Tabs HTML forms and elements Alerts and popups Embedded audios and videos, and so on You can see that there are lots of elements and aspects you need to take care of to create a fully responsive design. While all of these are achieved by using various features and technologies in CSS3, it is of course not an easy problem to solve without a framework that could help you do so. Precisely, you need a frontend framework that takes care of all the pains of technical responsive design implementation and releases you only for your brand and application design. Now, we introduce Bootstrap that would help you design and develop a responsive web design in a much optimized and efficient way. Introducing Bootstrap Simply put, Bootstrap is a frontend framework for faster and easier web development in the new standard of mobile-first philosophy. It uses HTML, CSS, and JavaScript. In August 2010, Twitter released Bootstrap as Open Source. There are quite a few similar frontend frameworks available in the industry, but Bootstrap is arguably the most popular framework in the lot. It is evident when we see Bootstrap is the most starred project in GitHub since 2012. Until now, you must be in a position to fathom why and where we need to use Bootstrap for web development; however, just to recap, here are the points in short. The mobile-first approach A responsive design Automatic browser support and handling Easy to adapt and get going What Bootstrap includes The following diagram demonstrates the overall structure of Bootstrap: CSS Bootstrap comes with fundamental HTML elements styled, global CSS classes, classes for advanced grid patterns, and lots of enhanced and extended CSS classes. For example, this is how the HTML global element is configured in Bootstrap CSS: html { font-family: sans-serif; -webkit-text-size-adjust: 100%; -ms-text-size-adjust: 100%; } This is how a standard HR HTML element is styled: hr { height: 0; -webkit-box-sizing: content-box; -moz-box-sizing: content-box; box-sizing: content-box; } Here is an example of new classes introduced in Bootstrap: .glyphicon { position: relative; top: 1px; display: inline-block; font-family: 'Glyphicons Halflings'; font-style: normal; font-weight: normal; line-height: 1; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } Components Bootstrap offers a rich set of reusable and built-in components, such as breadcrumbs, progress bars, alerts, and navigation bars. The components are technically custom CSS classes specially crafted for the specific purpose. For example, if you want to create a breadcrumb in your page, you simply add a DIV tag in your HTML using Bootstrap’s breadcrumb class: <ol class="breadcrumb"> <li><a href="#">Home</a></li> <li><a href="#">The Store</a></li> <li class="active">Offer Zone</li> </ol> In the background (stylesheet), this Bootstrap class is used to create your breadcrumb: .breadcrumb { padding: 8px 15px; margin-bottom: 20px; list-style: none; background-color: #f5f5f5; border-radius: 4px; } .breadcrumb > li { display: inline-block; } .breadcrumb > li + li:before { padding: 0 5px; color: #ccc; content: "/ 0a0"; } .breadcrumb > .active { color: #777; } Please note that these set of code blocks are simply snippets. JavaScript Bootstrap framework comes with a number of ready-to-use JavaScript plugins. Thus, when you need to create Popup windows, Tabs, Carousels or Tooltips, and so on, you just use one of the prepackaged JavaScript plugins. For example, if you need to create a tab control in your page, you use this: <div role="tabpanel"> <ul class="nav nav-tabs" role="tablist"> <li role="presentation" class="active"><a href="#recent" aria-controls="recent" role="tab" data-toggle="tab">Recent Orders</a></li> <li role="presentation"><a href="#all" aria-controls="al" role="tab" data-toggle="tab">All Orders</a></li> <li role="presentation"><a href="#redeem" aria-controls="redeem" role="tab" data-toggle="tab">Redemptions</a></li> </ul>   <div class="tab-content"> <div role="tabpanel" class="tab-pane active" id="recent"> Recent Orders</div> <div role="tabpanel" class="tab-pane" id="all">All Orders</div> <div role="tabpanel" class="tab-pane" id="redeem">Redemption History</div> </div> </div> To activate (open) a tab, you write this JavaScript code: $('#profileTab li:eq(1) a').tab('show'); As you could guess by looking at the syntax of this JavaScript line that the Bootstrap JS plugins are built on top of jQuery. Thus, the JS code you would write for Bootstrap are also all based on jQuery. Customization Even though Bootstrap offers most (if not all) standard features and functionalities for Responsive Web Design, there might be several cases when you would want to customize and extend the framework. One of the very basic requirements for customization would be to deploy your own branding and color combinations (themes) instead of the Bootstrap default ones. There can be several such use cases where you would want to change the default behavior of the framework. Bootstrap offers very easy and stable ways to customize the platform. When you use the Bootstrap CSS, all the global and fundamental HTML elements automatically become responsive and would properly behave as the client device on which the web page is browsed. The built-in components are also designed to be responsive. As the developer, you shouldn’t be worried about how these advanced components would behave in different devices and client agents. Summary In this article we have discussed the basics of Bootstarp along with a brief explanation on the design patterns and the navigation patterns. Resources for Article: Further resources on this subject: Deep Customization of Bootstrap [article] The Bootstrap grid system [article] Creating a Responsive Magento Theme with Bootstrap 3 [article]
Read more
  • 0
  • 0
  • 11725

article-image-nltk-hackers
Packt
07 Aug 2015
9 min read
Save for later

NLTK for hackers

Packt
07 Aug 2015
9 min read
In this article written by Nitin Hardeniya, author of the book NLTK Essentials, we will learn that "Life is short, we need Python" that's the mantra I follow and truly believe in. As fresh graduates, we learned and worked mostly with C/C++/JAVA. While these languages have amazing features, Python has a charm of its own. The day I started using Python I loved it. I really did. The big coincidence here is that I finally ended up working with Python during my initial projects on the job. I started to love the kind of datastructures, Libraries, and echo system Python has for beginners as well as for an expert programmer. (For more resources related to this topic, see here.) Python as a language has advanced very fast and spatially. If you are a Machine learning/ Natural language Processing enthusiast, then Python is 'the' go-to language these days. Python has some amazing ways of dealing with strings. It has a very easy and elegant coding style, and most importantly a long list of open libraries. I can go on and on about Python and my love for it. But here I want to talk about very specifically about NLTK (Natural Language Toolkit), one of the most popular Python libraries for Natural language processing. NLTK is simply awesome, and in my opinion,it's the best way to learn and implement some of the most complex NLP concepts. NLTK has variety of generic text preprocessing tool, such as Tokenization, Stop word removal, Stemming, and at the same time,has some very NLP-specific tools,such as Part of speech tagging, Chunking, Named Entity recognition, and dependency parsing. NLTK provides some of the easiest solutions to all the above stages of NLP and that's why it is the most preferred library for any text processing/ text mining application. NLTK not only provides some pretrained models that can be applied directly to your dataset, it also provides ways to customize and build your own taggers, tokenizers, and so on. NLTK is a big library that has many tools available for an NLP developer. I have provided a cheat-sheet of some of the most common steps and their solutions using NLTK. In our book, NLTK Essentials, I have tried to give you enough information to deal with all these processing steps using NLTK. To show you the power of NLTK, let's try to develop a very easy application of finding topics in the unstructured text in a word cloud. Word CloudNLTK Instead of going further into the theoretical aspects of natural language processing, let's start with a quick dive into NLTK. I am going to start with some basic example use cases of NLTK. There is a good chance that you have already done something similar. First, I will give a typical Python programmer approach and then move on to NLTK for a much more efficient, robust, and clean solution. We will start analyzing with some example text content: >>>import urllib2>>># urllib2 is use to download the html content of the web link>>>response = urllib2.urlopen('http://python.org/')>>># You can read the entire content of a file using read() method>>>html = response.read()>>>print len(html)47020 For the current example, I have taken the content from Python's home page: https://www.python.org/. We don't have any clue about the kind of topics that are discussed in this URL, so let's say that we want to start an exploratory data analysis (EDA). Typically in a text domain, EDA can have many meanings, but will go with a simple case of what kinds of terms dominate the documents. What are the topics? How frequent are they? The process will involve some level of preprocessing we will try to do this in a pure Python wayand then we will do it using NLTK. Let's start with cleaning the html tags. One way to do this is to select just tokens, including numbers and character. Anybody who has worked with regular expression should be able to convert html string into a list of tokens: >>># regular expression based split the string>>>tokens = [tok for tok in html.split()]>>>print "Total no of tokens :"+ str(len(tokens))>>># first 100 tokens>>>print tokens[0:100]Total no of tokens :2860['<!doctype', 'html>', '<!--[if', 'lt', 'IE', '7]>', '<html', 'class="no-js', 'ie6', 'lt-ie7', 'lt-ie8', 'lt-ie9">', '<![endif]-->', '<!--[if', 'IE', '7]>', '<html', 'class="no-js', 'ie7', 'lt-ie8', 'lt-ie9">', '<![endif]-->', ''type="text/css"', 'media="not', 'print,', 'braille,' ...] As you can see, there is an excess of html tags and other unwanted characters when we use the preceding method. A cleaner version of the same task will look something like this: >>>import re>>># using the split function https://docs.python.org/2/library/re.html>>>tokens = re.split('W+',html)>>>print len(tokens)>>>print tokens[0:100]5787['', 'doctype', 'html', 'if', 'lt', 'IE', '7', 'html', 'class', 'no', 'js', 'ie6', 'lt', 'ie7', 'lt', 'ie8', 'lt', 'ie9', 'endif', 'if', 'IE', '7', 'html', 'class', 'no', 'js', 'ie7', 'lt', 'ie8', 'lt', 'ie9', 'endif', 'if', 'IE', '8', 'msapplication', 'tooltip', 'content', 'The', 'official', 'home', 'of', 'the', 'Python', 'Programming', 'Language', 'meta', 'name', 'apple' ...] This looks much cleaner now. But still you can do more; I leave it to you to try to remove as much noise as you can. You can still look for word length as a criteria and remove words that have a length one—it will remove elements,such as 7, 8, and so on, which are just noise in this case. Now let's go to NLTK for the same task. There is a function called clean_html() that can do all the work we were looking for: >>>import nltk>>># http://www.nltk.org/api/nltk.html#nltk.util.clean_html>>>clean = nltk.clean_html(html)>>># clean will have entire string removing all the html noise>>>tokens = [tok for tok in clean.split()]>>>print tokens[:100]['Welcome', 'to', 'Python.org', 'Skip', 'to', 'content', '&#9660;', 'Close', 'Python', 'PSF', 'Docs', 'PyPI', 'Jobs', 'Community', '&#9650;', 'The', 'Python', 'Network', '&equiv;', 'Menu', 'Arts', 'Business' ...] Cool, right? This definitely is much cleaner and easier to do. No analysis in any EDA can start without distribution. Let's try to get the frequency distribution. First, let's do it the Python way, then I will tell you the NLTK recipe. >>>import operator>>>freq_dis={}>>>for tok in tokens:>>>    if tok in freq_dis:>>>        freq_dis[tok]+=1>>>    else:>>>        freq_dis[tok]=1>>># We want to sort this dictionary on values ( freq in this case )>>>sorted_freq_dist= sorted(freq_dis.items(), key=operator.itemgetter(1), reverse=True)>>> print sorted_freq_dist[:25][('Python', 55), ('>>>', 23), ('and', 21), ('to', 18), (',', 18), ('the', 14), ('of', 13), ('for', 12), ('a', 11), ('Events', 11), ('News', 11), ('is', 10), ('2014-', 10), ('More', 9), ('#', 9), ('3', 9), ('=', 8), ('in', 8), ('with', 8), ('Community', 7), ('The', 7), ('Docs', 6), ('Software', 6), (':', 6),  ('3:', 5), ('that', 5), ('sum', 5)] Naturally, as this is Python's home page, Python and the >>> interpreters are the most common terms, also giving a sense about the website. A better and efficient approach is to use NLTK's FreqDist() function. For this, we will take a look at the same code we developed before: >>>import nltk>>>Freq_dist_nltk=nltk.FreqDist(tokens)>>>print Freq_dist_nltk>>>for k,v in Freq_dist_nltk.items():>>>    print str(k)+':'+str(v)<FreqDist: 'Python': 55, '>>>': 23, 'and': 21, ',': 18, 'to': 18, 'the': 14, 'of': 13, 'for': 12, 'Events': 11, 'News': 11, ...>Python:55>>>:23and:21,:18to:18the:14of:13for:12Events:11News:11 Let's now do some more funky things. Let's plot this: >>>Freq_dist_nltk.plot(50, cumulative=False)>>># below is the plot for the frequency distributions We can see that the cumulative frequency is growing, and at words such as other and frequency 400, the curve is going into long tail. Still, there is some noise, and there are words such asthe, of, for, and =. These are useless words, and there is a terminology for these words. These words are stop words,such asthe, a, and an. Article pronouns are generally present in most of the documents; hence, they are not discriminative enough to be informative. In most of the NLP and information retrieval tasks, people generally remove stop words. Let's go back again to our running example: >>>stopwords=[word.strip().lower() for word in open("PATH/english.stop.txt")]>>>clean_tokens=[tok for tok in tokens if len(tok.lower())>1 and (tok.lower() not in stopwords)]>>>Freq_dist_nltk=nltk.FreqDist(clean_tokens)>>>Freq_dist_nltk.plot(50, cumulative=False) This looks much cleaner now! After finishing this much, you should be able to get something like this using word cloud: Please go to http://www.wordle.net/advanced for more word clouds. Summary To summarize, this article was intended to give you a brief introduction toNatural Language Processing. The book does assume some background in NLP andprogramming in Python, but we have tried to give a very quick head start to Pythonand NLP. Resources for Article: Further resources on this subject: Hadoop Monitoring and its aspects [Article] Big Data Analysis (R and Hadoop) [Article] SciPy for Signal Processing [Article]
Read more
  • 0
  • 0
  • 2823
article-image-camera-api
Packt
07 Aug 2015
4 min read
Save for later

The Camera API

Packt
07 Aug 2015
4 min read
In this article by Purusothaman Ramanujam, the author of PhoneGap Beginner's Guide Third Edition, we will look at the Camera API. The Camera API provides access to the device's camera application using the Camera plugin identified by the cordova-plugin-camera key. With this plugin installed, an app can take a picture or gain access to a media file stored in the photo library and albums that the user created on the device. The Camera API exposes the following two methods defined in the navigator.camera object: getPicture: This opens the default camera application or allows the user to browse the media library, depending on the options specified in the configuration object that the method accepts as an argument cleanup: This cleans up any intermediate photo file available in the temporary storage location (supported only on iOS) (For more resources related to this topic, see here.) As arguments, the getPicture method accepts a success handler, failure handler, and optionally an object used to specify several camera options through its properties as follows: quality: This is a number between 0 and 100 used to specify the quality of the saved image. destinationType: This is a number used to define the format of the value returned in the success handler. The possible values are stored in the following Camera.DestinationType pseudo constants: DATA_URL(0): This indicates that the getPicture method will return the image as a Base64-encoded string FILE_URI(1): This indicates that the method will return the file URI NATIVE_URI(2): This indicates that the method will return a platform-dependent file URI (for example, assets-library:// on iOS or content:// on Android) sourceType: This is a number used to specify where the getPicture method can access an image. The following possible values are stored in the Camera.PictureSourceType pseudo constants: PHOTOLIBRARY (0), CAMERA (1), and SAVEDPHOTOALBUM (2): PHOTOLIBRARY: This indicates that the method will get an image from the device's library CAMERA: This indicates that the method will grab a picture from the camera SAVEDPHOTOALBUM: This indicates that the user will be prompted to select an album before picking an image allowEdit: This is a Boolean value (the value is true by default) used to indicate that the user can make small edits to the image before confirming the selection; it works only in iOS. encodingType: This is a number used to specify the encoding of the returned file. The possible values are stored in the Camera.EncodingType pseudo constants: JPEG (0) and PNG (1). targetWidth and targetHeight: These are the width and height in pixels, to which you want the captured image to be scaled; it's possible to specify only one of the two options. When both are specified, the image will be scaled to the value that results in the smallest aspect ratio (the aspect ratio of an image describes the proportional relationship between its width and height). mediaType: This is a number used to specify what kind of media files have to be returned when the getPicture method is called using the Camera.PictureSourceType.PHOTOLIBRARY or Camera.PictureSourceType.SAVEDPHOTOALBUM pseudo constants as sourceType; the possible values are stored in the Camera.MediaType object as pseudo constants and are PICTURE (0), VIDEO (1), and ALLMEDIA (2). correctOrientation: This is a Boolean value that forces the device camera to correct the device orientation during the capture. cameraDirection: This is a number used to specify which device camera has to be used during the capture. The values are stored in the Camera.Direction object as pseudo constants and are BACK (0) and FRONT (1). popoverOptions: This is an object supported on iOS to specify the anchor element location and arrow direction of the popover used on iPad when selecting images from the library or album. saveToPhotoAlbum: This is a Boolean value (the value is false by default) used in order to save the captured image in the device's default photo album. The success handler receives an argument that contains the URI to the file or data stored in the file's Base64-encoded string, depending on the value stored in the encodingType property of the options object. The failure handler receives a string containing the device's native code error message as an argument. Similarly, the cleanup method accepts a success handler and a failure handler. The only difference between the two is that the success handler doesn't receive any argument. The cleanup method is supported only on iOS and can be used when the sourceType property value is Camera.PictureSourceType.CAMERA and the destinationType property value is Camera.DestinationType.FILE_URI. Summary In this article, we looked at the various properties available with the Camera API. Resources for Article: Further resources on this subject: Geolocation – using PhoneGap features to improve an app's functionality, write once use everywhere [article] Using Location Data with PhoneGap [article] iPhone JavaScript: Installing Frameworks [article]
Read more
  • 0
  • 0
  • 4574

article-image-storage-ergonomics
Packt
07 Aug 2015
19 min read
Save for later

Storage Ergonomics

Packt
07 Aug 2015
19 min read
In this article by Saurabh Grover, author of the book Designing Hyper-V Solutions, we will be discussing the last of the basics to get you equipped to create and manage a simple Hyper-V structure. No server environment, physical or virtual, is complete without a clear consideration and consensus over the underlying storage. In this article, you will learn about the details of virtual storage, how to differentiate one from the other, and how to convert one to the other and vice versa. We will also see how Windows Server 2012 R2 removes dependencies on raw device mappings by way of pass-through or iSCSI LUN, which were required for guest clustering. VHDX can now be shared and delivers better results than pass-through disks. There are more merits to VHDX than the former, as it allows you to extend the size even if the virtual machine is alive. Previously, Windows Server 2012 added a very interesting facet for storage virtualization in Hyper-V when it introduced virtual SAN, which adds a virtual host bus adapter (HBA) capability to a virtual machine. This allows a VM to directly view the fibre channel SAN. This in turn allows FC LUN accessibility to VMs and provides you with one more alternative for shared storage for guest clustering. Windows Server 2012 also introduced the ability to utilize the SMI-S capability, which was initially tested on System Center VMM 2012. Windows 2012 R2 carries the torch forward, with the addition of new capabilities. We will discuss this feature briefly in this article. In this article, you will cover the following: Two types of virtual disks, namely VHD and VHDX Merits of using VHDX from Windows 2012 R2 onwards Virtual SAN storage Implementing guest clustering using shared VHDX Getting an insight into SMI-S (For more resources related to this topic, see here.) Virtual storage A virtual machine is a replica of a physical machine in all rights and with respect to the building components, regardless of the fact that it is emulated, resembles, and delivers the same performance as a physical machine. Every computer ought to have storage for the OS or application loading. This condition applies to virtual machines as well. If VMs are serving as independent servers for roles such as domain controller or file server, where the server needs to maintain additional storage apart from the OS, the extended storage can be extended for domain user access without any performance degradation. Virtual machines can benefit from multiple forms of storage, namely VHD/VHDX, which are file-based storage; iSCSI LUNs; pass-through LUNs, which are raw device mappings; and of late, virtual-fibre-channel-assigned LUNs. There have been enhancements to each of these, and all of these options have a straightforward implementation procedure. However, before you make a selection, you should identify the use case according to your design strategy and planned expenditure. In the following section, we will look at the storage choices more closely. VHD and VHDX VHD is the old flag bearer for Microsoft virtualization ever since the days of virtual PC and virtual server. The same was enhanced and employed in early Hyper-V releases. However, as a file-based storage that gets mounted as a normal storage for a virtual machine, VHD had its limitations. VHDX, a new feature addition to Windows Server 2012, was built further upon the limitations of its predecessor and provides greater storage capacity, support for large sector disks, and better protection against corruption. In the current release of Windows Server 2012 R2, VHDX has been bundled with more ammo. VHDX packed a volley of feature enhancements when it was initially launched, and with Windows Server 2012 R2, Microsoft only made it better. If we compare the older, friendlier version of VHD with VHDX, we can draw the following inferences: Size factor: VHD had an upper size limit of 2 TB, while VHDX gives you a humungous maximum capacity of 64 TB. Large disk support: With the storage industry progressing towards 4 KB sector disks from the 512 bytes sector, for applications that still may depend on the older sector format, there are two offerings from the disk alignment perspective: native 4 KB disk and 512e (or 512 byte emulation disks). The operating system, depending on whether it supports native 4 KB disk or not, will either write 4 KB chunks of data or inject 512 bytes of data into a 4 KB sector. The process of injecting 512 bytes into a 4 KB sector is called RMW, or Read-Write-Modify. VHDs are generically supported on 512e disks. Windows Server 2012 and R2 both support native 4 KB disks. However, the VHD driver has a limitation; it cannot open VHD files on physical 4 KB disks. This limitation is checked by enabling VHD to be aligned to 4 KB and RMW ready, but if you are migrating from the older Hyper-V platform, you will need to convert it accordingly. VHDX, on the other hand, is the "superkid". It can be used on all disk forms, namely 512, 512e, and the native 4 KB disk as well, without any RMW dependency. Data corruption safety: In the event of power outages or failures, the possibility of data corruption is reduced with VHDX. Metadata inside the VHDX is updated via a logging process that ensures that the allocations inside VHDX are committed successfully. Offloaded data transfers (ODX): With Windows Server 2012 Hyper-V supporting this feature, data transfer and moving and sizing of virtual disks can be achieved at the drop of a hat, without host server intervention. The basic prerequisite for utilizing this feature is to host the virtual machines on ODX-capable hardware. Thereafter, Windows Server 2012 self-detects and enables the feature. Another important clause is that virtual disks (VHDX) should be attached to the SCSI, not IDE. TRIM/UNMAP: Termed by Microsoft in its documentation as efficiency in representing data, this feature works in tandem with thin provisioning. It adds the ability to allow the underlying storage to reclaim space and maintain it optimally small. Shared VHDX: This is the most interesting feature in the collection released with Windows Server 2012 R2. It made guest clustering (failover clustering in virtual machines) in Hyper-V a lot simpler. With Windows Server 2012, you could set up a guest cluster using virtual fibre channel or iSCSI LUN. However, the downside was that the LUN was exposed to the user of the virtual machine. Shared VHDX proves to be the ideal shared storage. It gives you the benefit of storage abstraction, flexibility, and faster deployment of guest clusters, and it can be stored on an SMB share or a cluster-shared volume (CSV). Now that we know the merits of using VHDX over VHD, it is important to realize that either of the formats can be converted into the other and can be used under various types of virtual disks, allowing users to decide a trade-off between performance and space utilization. Virtual disk types Beyond the two formats of virtual hard disks, let's talk about the different types of virtual hard disks and their utility as per the virtualization design. There are three types of virtual hard disks, namely dynamically expanding, fixed-size, and differencing virtual hard disks: Dynamically expanding: Also called a dynamic virtual hard disk, this is the default type. It gets created when you create a new VM or a new VHD/VHDX. This is Hyper-V's take on thin provisioning. The VHD/VHDX file will start off from a small size and gradually grow up to the maximum defined size for the file as and when chunks of data get appended or created inside the OSE (short for operating system environment) hosted by the virtual disk. This disk type is quite beneficial, as it prevents storage overhead and utilizes as much as required, rather than committing the entire block. However, due to the nature of the virtual storage, as it spawns in size, the actual file gets written in fragments across the Hyper-V CSV or LUN (physical storage). Hence, it affects the performance of the disk I/O operations of the VM. Fixed size: As the name indicates, the virtual disk type commits the same block size on the physical storage as its defined size. In other words, if you have specified a fixed size 1 TB, it will create a 1 TB VHDX file in the storage. The creation of a fixed size takes a considerable amount of time, commits space on the underlying storage, and does allow SAN thin provisioning to reclaim it, somewhat like whitespaces in a database. The advantage of using this type is that it delivers amazing read performance and heavy workloads from SQL, and exchange can benefit from it. Differencing: This is the last of the lot, but quite handy as an option when it comes to quick deployment of virtual machines. This is by far an unsuitable option, unless employed for VMs with a short lifespan, namely pooled VDI (short for virtual desktop infrastructure) or lab testing. The idea behind the design is to have a generic virtual operating system environment (VOSE) in a shut down state at a shared location. The VHDX of the VOSE is used as a parent or root, and thereafter, multiple VMs can be spawned with differencing or child virtual disks that use the generalized OS from the parent and append changes or modifications to the child disk. So, the parent stays unaltered and serves as a generic image. It does not grow in size; on the contrary, the child disk keeps on growing as and when data is added to the particular VM. Unless used for short-lived VMs, the long-running VMs could enter an outage state or may be performance-stricken soon due to the unpredictable growth pattern of a differencing disk. Hence, these should be avoided for server virtual machines without even a second thought. Virtual disk operations Now we will apply all of the knowledge gained about virtual hard disks, and check out what actions and customizations we can perform on them. Creating virtual hard disks This goal can be achieved in different ways: You can create a new VHD when you are creating a new VM, using the New Virtual Machine Wizard. It picks up the VHDX as the default option. You can also launch the New Virtual Hard Disk Wizard from a virtual machine's settings. This can be achieved by PowerShell cmdlets as well:New-VHD You may employ the Disk Management snap-in to create a new VHD as well. The steps to create a VHD here are pretty simple: In the Disk Management snap-in, select the Action menu and select Create VHD, like this: Figure 5-1: Disk Management – Create VHD This opens the Create and Attach Virtual Hard Disk applet. Specify the location to save the VHD at, and fill in Virtual hard disk format and Virtual hard disk type as depicted here in figure 5-2: Figure 5-2: Disk Management – Create and Attach Virtual Hard Disk The most obvious way to create a new VHD/VHDX for a VM is by launching New Virtual Hard Disk Wizard from the Actions pane in the Hyper-V Manager console. Click on New and then select the Hard Disk option. It will take you to the following set of screens: On the Before You Begin screen, click on Next, as shown in this screenshot: Figure 5-3: New Virtual Hard Disk Wizard – Create VHD The next screen is Choose Disk Format, as shown in figure 5-4. Select the relevant virtual hard disk format, namely VHD or VHDX, and click on Next. Figure 5-4: New Virtual Hard Disk Wizard – Virtual Hard Disk Format In the screen for Choose Disk Type, select the relevant virtual hard disk type and click on Next, as shown in the following screenshot: Figure 5-5: New Virtual Hard Disk Wizard– Virtual Hard Disk Type The next screen, as shown in figure 5-6, is Specify Name and Location. Update the Name and Location fields to store the virtual hard disk and click on Next. Figure 5-6: New Virtual Hard Disk Wizard – File Location The Configure Disk screen, shown in figure 5-7, is an interesting one. If needs be, you can convert or copy the content of a physical storage (local, LUN, or something else) to the new virtual hard disk. Similarly, you can copy the content from an older VHD file to the Windows Server 2012 or R2 VHDX format. Then click on Next. Figure 5-7: New Virtual Hard Disk Wizard – Configure Disk On the Summary screen, as shown in the following screenshot, click on Finish to create the virtual hard disk: Figure 5-8: New Virtual Hard Disk Wizard – Summary Editing virtual hard disks There may be one or more reasons for you to feel the need to modify a previously created virtual hard disk to suit a purpose. There are many available options that you may put to use, given a particular virtual disk type. Before you edit a VHDX, it's a good practice to inspect the VHDX or VHD. The Inspect Disk option can be invoked from two locations: from the VM settings under the IDE or SCSI controller, or from the Actions pane of the Hyper-V Manager console. Also, don't forget how to do this via PowerShell: Get-VHD -Path "E:Hyper-VVirtual hard disks1.vhdx" You may now proceed with editing a virtual disk. Again, the Edit Disk option can be invoked in exactly the same fashion as Inspect Disk. When you edit a VHDX, you are presented with four options, as shown in figure 5-9. It may sound obvious, but not all the options are for all the disk types: Compact: This operation is used to reduce or compact the size of a virtual hard disk, though the preset capacity remains the same. A dynamic disk, or differencing disk, grows as data elements are added, though deletion of the content does not automatically reclaim the storage capacity. Hence, a manual compact operation becomes imperative reduce the file size. PowerShell cmdlet can also do this trick, as follows: Optimize-VHD Convert: This is an interesting one, and it almost makes you change your faith. As the name indicates, this operation allows you to convert one virtual disk type to another and vice versa. You can also create a new virtual disk of the desired format and type at your preferred location. The PowerShell construct used to help you achieve the same goal is as follows: Convert-VHD Expand: This operation comes in handy, similar to Extend a LUN. You end up increasing the size of a virtual hard disk, which happens visibly fast for a dynamic disk and a bit slower for its fixed-size cousins. After this action, you have to perform the follow-up action inside the virtual machine to increase the volume size from disk management. Now, for the PowerShell code: Resize-VHD Merge: This operation is disk-type-specific—differencing virtual disks. It allows two different actions. You can either merge the differencing disk with the original parent, or create a new merged VHD out of all the contributing VHDs, namely the parent and the child or the differencing disk. The latter is the preferred way of doing it, as in utmost probability, there would be more than differencing to a parent. In PowerShell, the alternative the cmdlet is this: Merge-VHD Figure 5-9: Edit Virtual Hard Disk Wizard – Choose Action Pass-through disks As the name indicates, these are physical LUNs or hard drives passed on from the Hyper-V hosts, and can be assigned to a virtual machine as a standard disk. A once popular method on older Hyper-V platforms, this allowed the VM to harness the full potential of the raw device bypassing the Hyper-V host filesystem and also not getting restricted by the 2 TB limit of VHDs. A lot has changed over the years, as Hyper-V has matured into a superior virtualization platform and introduced VHDX, which went past the size limitation. with Windows Server 2012 R2 can be used as a shared storage for Hyper-V guest clusters. There are, however, demerits to this virtual storage. When you employ a pass-through disk, the virtual machine configuration file is stored separately. Hence, the snapshotting becomes unknown to this setup. You would not be able to utilize the dynamic disk's or differential disk's abilities here too. Another challenge of using this form of virtual storage is that when using a VSS-based backup, the VSS writer ignores the pass-through and iSCSI LUN. Hence, a complex backup plan has to be implemented by involving a running backup within VM and on the virtualization host separately. The following are steps, along with a few snapshots, that show you how to set up a pass-through disk: Present a LUN to the Hyper-V host. Confirm the LUN in Disk Management and ensure that it stays in the Offline State and as Not Initialized. Figure 5-10: Hyper-V Host Disk Management In Hyper-V Manager, right-click on the VM you wish to assign the pass-through to and select Settings. Figure 5-11: VM Settings – Pass-through disk placement Select SCSI Controller (or IDE in the case of Gen-1 VM) and then select the Physical hard disk option, as shown in the preceding screenshot. In the drop-down menu, you will see the raw device or LUN you wish to assign. Select the appropriate option and click on OK. Check Disk Management within the virtual machine to confirm that the disk has visibility. Figure 5-12: VM Disk Management – Pass-through Assignment Bring it online and initialize. Figure 5-13: VM Disk Management – Pass-through Initialization As always the preceding path can be chalked out with the help of a PowerShell cmdlet: Add-VMHardDiskDrive -VMName VM5 –ControllerType SCSI – ControllerNumber 0 –ControllerLocation 2 –DiskNumber 3 Virtual fibre channel Let's move on to the next big offering in Windows Server 2012 and R2 Hyper-V Server. There was pretty much a clamor for direct FC connectivity to virtual machines, as pass-through disks were supported only via iSCSI LUNs (with some major drawbacks not with FC). Also, needless to say, FC is faster. Enterprises with high-performance workloads relying on the FC SAN refrained from virtualizing or migrating to the cloud. Windows Server 2012 introduced the virtual fibre channel SAN ability in Hyper-V, which extended the HBA (short for host bus adapter) abilities to a virtual machine, granting them a WWN (short for world wide node name) and allowing access to a fibre channel SAN over a virtual SAN. The fundamental principle behind the virtual SAN is the same as the Hyper-V virtual switch, wherein you create a virtual SAN that hooks up to the SAN fabric over the physical HBA of the Hyper-V host. The virtual machine has new synthetic hardware for the last piece. It is called a virtual host bus adapter or vHBA, which gets its own set of WWNs, namely WWNN (node name) and WWPN (port name). The WWN is to the FC protocol as MAC is to the Ethernet. Once the WWNs are identified at the fabric and the virtual SAN, the storage admins can set up zoning and present the LUN to the specific virtual machine. The concept is straightforward, but there are prerequisites that you will need to ensure are in place before you can get down to the nitty-gritty of the setup: One or more Windows Server 2012 or R2 Hyper-V hosts. Hosts should have one or more FC HBAs with the latest drivers, and should support the virtual fibre channel and NPIV. NPIV may be disabled at the HBA level (refer to the vendor documentation prior to deployment). The same can be enabled using command-line utilities or GUI-based such as OneCommand manager, SANSurfer, and so on. NPIV should be enabled on the SAN fabric or actual ports. Storage arrays are transparent to NPIV, but they should support devices that present LUNs. Supported guest operating systems for virtual SAN are Windows 2008, Windows 2008 R2, Windows 2012, and Windows 2012 R2. The virtual fibre channel does not allow boot from SAN, unlike pass-through disks. We are now done with the prerequisites! Now, let's look at two important aspects of SAN infrastructure, namely NPIV and MPIO. N_Port ID virtualization (NPIV) An ANSI T11 standard extension, this feature allows virtualization of the N_Port (WWPN) of an HBA, allowing multiple FC initiators to share a single HBA port. The concept is popular and is widely accepted and promoted by different vendors. Windows Server 2012 and R2 Hyper-V utilizes this feature to the best, wherein each virtual machine partaking in the virtual SAN gets assigned a unique WWPN and access to the SAN over a physical HBA spawning its own N_Port. Zoning follows next, wherein the fabric can have the zone directed to the VM WWPN. This attribute leads to a very small footprint, and thereby, easier management and operational and capital expenditure. Summary It is going to be quite a realization that we have covered almost all the basic attributes and aspects required for a simple Windows Server 2012 R2 Hyper-V infrastructure setup. If we revise the contents, we will notice this: we started off in this article by understanding and defining the purpose of virtual storage, and what the available options are for storage to be used with a virtual machine. We reviewed various virtual hard disk types, formats, and associated operations that may be required to customize a particular type or modify it accordingly. We recounted how the VHDX format is superior to its predecessor VHD and which features were added with the latest Window Server releases, namely 2012 and 2012 R2. We discussed shared VHDX and how it can be used as an alternative to the old-school iSCSI or FC LUN as a shared storage for Windows guest clustering. Pass-through disks are on their way out, and we all know the reason why. The advent of the virtual fibre channel with Windows Server 2012 has opened the doors for virtualization of high-performance workloads relying heavily on FC connectivity, which until now was a single reason and enough of a reason to decline consolidation of these workloads. Resources for Article: Further resources on this subject: Hyper-V Basics [article] Getting Started with Hyper-V Architecture and Components [article] Hyper-V building blocks for creating your Microsoft virtualization platform [article]
Read more
  • 0
  • 0
  • 7174

article-image-simplify-deployment-infrastructure-manifest-part-2
Cody A.
06 Aug 2015
8 min read
Save for later

Simplify Deployment with an Infrastructure Manifest, Part 2

Cody A.
06 Aug 2015
8 min read
This is the second part of a post on using a Manifest of your infrastructure for automation. The first part described how to use your Cloud API to transform Application Definitions into an Infrastructure Manifest. This post will show examples of automation tools built using an Infrastructure Manifest. In particular, we'll explore application deployment and load balancer configuration management. Recall our example Infrastructure Manifest from Part 1: { "prod": { "us-east-1": { "appserve01ea1": { "applications": [ "appserve" ], "zone": "us-east-1a", "fqdn": "ec2-1-2-3-4.compute-1.amazonaws.com", "private ip": "10.9.8.7", "public ip": "1.2.3.4", "id": "i-a1234bc5" }, ... }, ... } As I mentioned previously, this Manifest can form the basis for numerous automations. Some tools my team at Signal has built on top of this concept are automated deployments, load balancing, security group management, and DNS. Application Deployment Let's see how an Infrastructure Manifest can simplify application deployment. Although we'll use Fabric as the basis for our deployment system, the concept should work with Chef and many other push-based deployment systems as well. from json import load as json_decode from urllib2 import urlopen MANIFEST = json_decode(urlopen(env.manifest)) for hostname, meta in MANIFEST.iteritems(): for role in meta['roles']: env.roledefs[role].append(hostname) Note: For this to work, you must set the manifest URL in Fabric's environment as env.manifest. For example, you can set this in the ~/.fabricrc file or pass it on the command-line. manifest=http://manifest:5000/api/prod/us-east-1/manifest That's all Fabric really requires to know where to deploy each application! Given the manifest above, this would add the "appserve" role so that you can run tasks on these instances simultaneously. For example, to deploy the "appserve" application to all the hosts with this role: @task @roles('appserve') def deploy_appserve(): # standard Fabric deploy logic here Now calling fab deploy_appserve will run the commands to deploy the "appserve application on each host with the "appserve" role. Easy, right? You might want to deploy some applications to every host in your infrastructure. Instead of adding these special roles to every Application Definition, you can include them here. For example, if you have a custom monitoring application ("mymon"), then you can read the list of all hosts from the Manifest and add them to the "mymon" role. # set up special cases for roledefs: env.roledefs = defaultdict(list, { 'mymon': list(MANIFEST.keys()), }) Now, after adding a deploy_mymon task, you'll be able to easily deploy "mymon" to all hosts in your infrastructure. Even if you auto-deploy using a specialized git receiver, Jenkins hooks, or similar, this approach will enable you to make your deployments cloud-aware, to deploy each application to the appropriate hosts in your cloud. That's it! Deployments can't be much simpler than this. Load Balancer Configuration Management A common challenge in cloud environments is maintaining the list of all hosts for load balancer configurations. If you don't want to lock in to a vendor or cloud-specific solution such as Amazon ELB, you may choose an open source software load balancer such as HAProxy. However, this leaves you with the challenge of maintaining the configurations as hosts appear and disappear in your cloud-based infrastructure. This problem is amplified when you use software-based load balancers between each set of services (or each tier) in your application. Using the Infrastructure Manifest, a first-pass solution can be quite simple. You can revision-control the configuration templates and interpolate the application ports and host information from the Manifest. Then periodically update the generated configuration files and distribute them using your existing configuration management software (such as Puppet or Chef). Let's say you want to generate a HAProxy configuration for your load balancer. The complete configuration file might look like this: global user haproxy group haproxy daemon frontend main_vip bind *:80 # ACLs for basic name-based virtual-hosts acl appserve_host hdr_beg(host) -i app.example.com acl uiserve_host hdr_beg(host) -i portal.example.com use_backend appserve if appserve_host use_backend uiserve if uiserve_host default_backend uiserve backend appserve balance roundrobin option httpclose option httpchk GET /hc http-check disable-on-404 server appserve01ea1 10.42.1.91:8080 check server appserve02ea1 10.42.1.92:8080 check server appserve03ea1 10.42.1.93:8080 check backend uiserve balance roundrobin option httpclose option httpchk GET /hc server uiserve01ea1 10.42.1.111:8082 check server uiserve02ea1 10.42.1.112:8082 check The simplest way to produce this configuration file is to generate it from a template. There are many templating solutions from which to choose. I'm fond of Jinja2, so we'll use that for exploring this solution in Python. We want to load the template from a file located in a "templates" directory, so we start by creating a Jinja2 loader and environment: from jinja2 import Environment, FileSystemLoader import os loader = FileSystemLoader(os.path.join(os.path.dirname(__file__), 'templates')) environment = Environment(loader=loader, lstrip_blocks=True) The template corresponding to this output could look like this. We'll call it 'lb.txt' since it's for the lb server group. global user haproxy group haproxy daemon frontend main_vip bind *:80 # ACLs for basic name-based virtual-hosts acl appserve_host hdr_beg(host) -i app.example.com acl uiserve_host hdr_beg(host) -i portal.example.com use_backend appserve if appserve_host use_backend uiserve if uiserve_host default_backend uiserve backend appserve balance roundrobin option httpclose option httpchk GET {{ vips.appserve.healthcheck_resource }} http-check disable-on-404 {%- for server in vips.appserve.servers %} server {{ server['name'] }} {{ server.details['private_ip'] }}:{{ vips.appserve.backend_port }} check {%- endfor %} backend uiserve balance roundrobin option httpclose option httpchk GET {{ vips.uiserve.healthcheck_resource }} {%- for server in vips.uiserve.servers %} server {{ server['name'] }} {{ server.details['private_ip'] }}:{{ vips.uiserve.backend_port }} check {%- endfor %} You can see by examining the template that it only expects a single variable: vips. This is a map of application names to their load balancer configuration. Specifically, each vip contains a backend port, a healthcheck resource (i.e., HTTP path), and a list of servers (with server name and private IP address for each). Coincidentally, all of this information is available in the Infrastructure Manifest and Application Definitions we developed in Part 1. We can easily fetch this information from the webapp. from requests import get def main(manifest_host, env, region, server_group): manifest = get('http://%s/api/%s/%s/manifest' % (manifest_host, env, region)).json() applications = get('http://%s/api/applications' % manifest_host).json() print generate_haproxy(manifest, applications, server_group) Note: we didn't actually add the /api/applications endpoint last week, so its left as an exercise for the reader; hint: jsonify(config()['APPLICATIONS']). Now we can dive into the meat of this tool, the generate_haproxy function. As you might guess, this uses the Jinja2 environment to render the template. But first it must merge the Application Definitions and Manifest into the vips variable that the template expects. def generate_haproxy(manifest, applications, server_group): apps = {} for application, meta in applications.iteritems(): app_object = { 'servers': [], 'frontend_port': meta['frontend'], 'backend_port': meta['backend'], 'healthcheck_resource': meta['healthcheck']['resource'] } for server in manifest: if application in manifest[server]['applications']: app_object['servers'].append({'name': server, 'details': manifest[server]}) app_object['servers'].sort(key=lambda e: e['name']) apps[application] = app_object return environment.get_template("%s.txt" % server_group).render(vips=apps) There's not much going on here. We iterate through all the applications and create a vip (app_object) with all the needed variables for each one. Then we render the server_group's template with Jinja2. Finally, we can call the main we created above to see this in action: main('localhost:5000', 'prod', 'us-east-1', 'lb') This will print the HAProxy configuration for the lb load balancer group for your production us-east-1 region. (It assumes that the Manifest webapp is running on the same host.) Depending on what hosts you have in your cloud infrastructure, this should print something like the complete HAProxy configuration file shown at the top. To easily keep your load balancer configurations up-to-date, you could run this regularly for each environment and region. Then the generated files could be distributed using your existing configuration management system. Alternatively, if your load balancers support programmatic rule updates, that would be even cleaner than this simple first-pass approach which relies on configuration file updates. I hope this spurs your imagination and shows the benefit of using an Infrastructure Manifest to automate all the things. About the author Cody A. Ray is an inquisitive, tech-savvy, entrepreneurially-spirited dude. Currently, he is a software engineer at Signal, an amazing startup in downtown Chicago, where he gets to work with a dream team that's changing the service model underlying the Internet.
Read more
  • 0
  • 0
  • 2574
article-image-rundown-example
Packt
06 Aug 2015
10 min read
Save for later

Rundown Example

Packt
06 Aug 2015
10 min read
In this article by Miguel Oliveira, author of the book Microsoft System Center Orchestrator 2012 R2 Essentials, we will learn to get started on the creational process. We will be able to driven on how to address and connect all the pieces together in order to successfully create a Runbook. (For more resources related to this topic, see here.) Runbook for Active Directory User Account Provisioning Now, for this Runbook, we've been challenged by our HR department to come out with a solution for them to be able to create new user accounts for recently joined employees. The request was specifically drawn with the target for them (HR) to be able to: Provide the first and last name Provide the department name Get that user added to the proper department group and get all the information of the user Send the newly created account to the IT department to provide a machine, a phone, and an e-mail address With these requirements at the back of our heads, let's see which activities we need to get into our Runbook. I'll place these in steps for this example, so it's easy to follow: Data input: So, we definitely need an activity to allow the HR to feed the information into the Runbook. For this, we can use the Initialize Data activity (Runbook control category), or we could work along with a monitored file and read the data from a line, or even from a SharePoint list. But to keep it simple for now, let's use the Initialize Data. Data processing: In here, the idea would be to retrieve the Department given by the HR and process it to retrieve the group (the Get Group activity from the Active Directory category) and include our user (the Add User To Group activity from the Active Directory category) into the group we've retrieved; but in between, we'll need to create the user account (Create User activity from the Active Directory category), and generate a password (the Generate Random Text activity from the Utilities category). Data output: At the very end of all this, send an e-mail (the Send Email activity from the Email category) back to HR with the account information and status of its creation and inform our IT department (for security reasons) too about the account that has been created. We're also going to closely watch for errors with a few activities that will show us whether an error occurs. Let's see the look of this Runbook from a structured point (and actually almost how it's going to look in the end) and we'll detail the activities and options within them step by step from there. Here's the aspect of the Runbook structured with the activities properly linked between them allowing the data bus to flow and transport the published data from the beginning to the end: As described in the steps, we start with an Initialize Data activity in which we're going to request some inputs from the person executing the Runbook. To create a user, we'll need his First Name and Last Name and also the Department. For that, we'll fill in the following information in the Fetch User Details activity seen in the previous screenshot. For the sake of avoiding errors, the HR department should have a proper list of departments that we know will translate into a proper group in the upcoming activities. After filling the information, the processing of the information begins and with it, our automation process that will find the group for that department, create our user account, set a password, change password on the first login, add the user to the group, and enable the account. For that, we'll start with the Get Group activity in which we'll fill in the following: Set up the proper configuration in the Get Group Properties window for the Active Directory Domain in which you'll want this to execute, and in the Filters options, set to filter Sam Account Name of the group as the Department filled by the HR department. Now we'll set another prerequisite to create the account—the password! For this, we'll get the Generate Random Text activity and set it with the following parameters: These values should be set to accordingly accommodate your existing security policy and minimum password requirements for your domain. These previous activities are all we need to have the necessary values to proceed with the account creation by using the Create User activity. These should be the parameters filled in. All of these parameters are actually being retrieved from the Published Data from the last activities. As the list is long, we'll detail them here for your better understanding. Everything that's between {} is Published Data: Common Name: {First Name from "Fetch User Details"} {Last Name from "Fetch User Details"} Department: {Display Name from "Get Group"} Display Name: {First Name from "Fetch User Details"} {Last Name from "Fetch User Details"} First Name: {First Name from "Fetch User Details"} Last Name: {Last Name from "Fetch User Details"} Password: {Random text from "Generate Random Text"} User Must Change Password: True SAM Account Name: {First Name from "Fetch User Details"}.{Last Name from "Fetch User Details"} User Principal Name: {First Name from "Fetch User Details"}.{Last Name from "Fetch User Details"}@test.local Email: {First Name from "Fetch User Details"}.{Last Name from "Fetch User Details"}@test.com Manager: {Managed By from "Get Group"} As said previously, most of the data comes from the Published Data and we've created subscriptions in all these fields to retrieve it. The only two fields that have data different from Published Data are the User Must Change Password, User Principal Name (UPN), and Email. The User Must Change Password is a Boolean field that will display only Yes or No, and in the UPN and e-mail we've added the domain information (@test.local and @test.com) to the Published Data. Depending on the Create User activity's output, it will trigger a different activity. For now, let's assume that the activity returns a success on the execution, this will make the Runbook follow the smart link that goes on with the Get User activity. The Get User activity will retrieve all the information concerning the newly created user account that will be useful for the next activities down the line. In order to retrieve the proper information, we'll need to configure the following in the Filters area within the activity: You'll need to add a filter, selecting Sam Account Name and Relation as Equals for Value under the subscribed data from Sam Account Name that comes out of the Create User activity. From here, we'll link with the activity that Add User to Group (here renamed to Add User to Department) and within that activity we're going to specify the group and the user so the activity can add the user into the group. It should look exactly like the screenshot that follows: We'll once again assume that everything's running as expected and prepare our next activity that is to enable user account and for this one, we'll use the Enable User activity. The configuration of the activity can be seen in the next screenshot: Once again, we'll get the information out of the Published Data and feed it into the activity. After this activity is completed, we're going to log the execution and information output into the platform with the Send Platform Event activity so we can see any necessary information available from the execution. Here is a sample of the configuration for the message output: To get the Details text box expanded this way, right-click on it and select Expand… from the menu, then you can format and include the data that you feel is more important for you to see it. Then we'll send an e-mail for the HR team with the account creation details so they can communicate to the newly arrived employee and another e-mail for the IT department only with the account name and the department (plus the group name) for security reasons. Here are the samples of these two activities, starting with the HR e-mail: Let's go point by point on this configuration sample. In the Details section, we've settled the following: Subject: Account {Sam Account Name from "Get User"} Created Recipients: to: hr.dept@test.com Message: The message description is given in the following screenshot: Message option that consists of choosing the Priority of the message (high, normal, or low), and set the necessary SMTP authentication parameters (account, password, and domain) so you can send the message through your e-mail service. If you have an application e-mail service relay, you can leave the SMTP authentication without any configuration. In connect Connect option, you'll find the place to configure the e-mail address that you want the user to see and the SMTP connection (server, port, and SSL) through which you'll send your messages. Now our Send Email IT activity will be more or less the same, with the exception for the destination and the message itself. It should be something a little more or less like the following screenshot: By now you've got the idea and you're pumped to create new Runbooks, but we still have to do some error control on some of these tasks; although they're chained, if one fails, everything fails. So for this Runbook, we'll create error control on two tasks that if we observe well, are more or less the only two that can fail! One is the Create User Account activity, which can fail due to the user account existing or by some issue with privileges on its creation. The other is Add User To Department that might fail to add the user into the group for some reason. So for this, we'll create two notification activities called Send Event and Log Message that we'll rename to User Account Error and Group Error respectively. If we look into the User Account Error activity, we'll set something more or less like the following screenshot: A quick explanation of the settings is as follows: Computer: This is the computer to which Windows Event Viewer we're going to write the event into. In this case, we'll concentrate over our Management Server, but you might have a logging server for this. Message: The message gets logged into the windows event viewer. Here, we can subscribe for the error data coming out of the last activity executed. Severity: This is usually an Error. You can set Information or Warning if you are deploying these activities to keep a track on each given step. So for our Group Error Properties the philosophy will be the same. Now that we are all set, we'll need to work our smart links so that they can direct the Runbook execution flow into the following activity depending on the previous activity output (success or error). In the end, your Runbook should look a little bit more like this: That's it for the Runbook for Active Directory User Account Provisioning. We'll now speed up a little bit more on the other Runbooks as you'll have a much clearer understanding after this first sample. Summary We've seen one of the Runbook samples these Runbooks should serve as the base for real case scenarios in the environment and help us in the creativity process and also to better understand the configurations necessary on each activity in order to proceed successfully. Resources for Article: Further resources on this subject: Unpacking System Center 2012 Orchestrator [article] Working with VMware Infrastructure [article] Unboxing Docker [article]
Read more
  • 0
  • 0
  • 1783

article-image-detecting-touchscreen-gestures
Packt
06 Aug 2015
18 min read
Save for later

Detecting Touchscreen Gestures

Packt
06 Aug 2015
18 min read
In this article by Kyle Mew author of the book, Android 5 Programming by Example, we will learn how to: Add a GestureDetector to a view Add an OnTouchListener and an OnGestureListener Detect and refine fling gestures Use the DDMS Logcat to observe the MotionEvent class Edit the Logcat filter configuration Simplify code with a SimpleOnGestureListener Add a GestureDetector to an Activity Edit the Manifest to control launch behavior Hide UI elements Create a splash screen Lock screen orientation (For more resources related to this topic, see here.) Adding a GestureDetector to a view Together, view.GestureDetector and view.View.OnTouchListener are all that are required to provide our ImageView with gesture functionality. The listener contains an onTouch() callback that relays each MotionEvent to the detector. We are going to program the large ImageView so that it can display a small gallery of related pictures that can be accessed by swiping left or right on the image. There are two steps to this task as, before we implement our gesture detector, we need to provide the data for it to work on. Adding the gallery data As this app is for demonstration and learning purposes, and so we can progress as quickly as possible, we will only provide extra images for one or two of the ancient sites in the project. Here is how it's done: Open the Ancient Britain project. Open the MainData.java file. Add the following arrays: static Integer[] hengeArray = {R.drawable.henge_large, R.drawable.henge_2, R.drawable.henge_3, R.drawable.henge_4}; static Integer[] horseArray = {}; static Integer[] wallArray = {R.drawable.wall_large, R.drawable.wall_2}; static Integer[] skaraArray = {}; static Integer[] towerArray = {}; static Integer[][] galleryArray = {hengeArray, horseArray, wallArray, skaraArray, towerArray}; Either download the project files from the Packt website or find four of your own images (around 640 x 480 px). Name them henge_2, henge_3, henge_4, and wall_2 and place them in your res/drawable directory. This is all very straightforward, and the code that will accompany it allows you to have individual arrays of any length. This is all we need to add to our gallery data. Now, we need to code our GestureDetector and OnTouchListener. Adding the GestureDetector Along with the OnTouchListener that we will define for our ImageView, the GestureDetector has its own listeners. Here we will use GestureDetector.OnGestureListener to detect a fling gesture and collect the MotionEvent that describe it. Follow these steps to program your ImageView to respond to fling gestures: Open the DetailActivity.java file. Declare the following class fields: private static final int MIN_DISTANCE = 150; private static final int OFF_PATH = 100; private static final int VELOCITY_THRESHOLD = 75; private GestureDetector detector; View.OnTouchListener listener; private int ImageIndex; In the onCreate() method assigns both the detector and listener like this: detector = new GestureDetector(this, new GalleryGestureDetector()); listener = new View.OnTouchListener() { @Override public boolean onTouch(View v, MotionEvent event) { return detector.onTouchEvent(event); } }; Beneath this, add the following line: ImageIndex = 0; Beneath the line detailImage = (ImageView) findViewById(R.id.detail_image);, add the following line: detailImage.setOnTouchListener(listener); Create the following inner class: class GalleryGestureDetector implements GestureDetector.OnGestureListener { } Before dealing with the errors this generates, add the following field to the class: private int item; { item = MainActivity.currentItem; } Click anywhere on the line registering the error and press Alt + Enter. Then select Implement Methods, making sure that you have the Copy JavaDoc and Insert @Override boxes checked. Complete the onDown() method like this: @Override public boolean onDown(MotionEvent e) { return true; } Fill in the onShowPress() method: @Override public void onShowPress(MotionEvent e) { detailImage.setElevation(4); } Then fill in the onFling() method: @Override public boolean onFling(MotionEvent event1, MotionEvent event2, float velocityX, float velocityY) { if (Math.abs(event1.getY() - event2.getY()) > OFF_PATH) return false; if (MainData.galleryArray[item].length != 0) { // Swipe left if (event1.getX() - event2.getX() > MIN_DISTANCE && Math.abs(velocityX) > VELOCITY_THRESHOLD) { ImageIndex++; if (ImageIndex == MainData.galleryArray[item].length) ImageIndex = 0; detailImage.setImageResource(MainData .galleryArray[item][ImageIndex]); } else { // Swipe right if (event2.getX() - event1.getX() > MIN_DISTANCE && Math.abs(velocityX) > VELOCITY_THRESHOLD) { ImageIndex--; if (ImageIndex < 0) ImageIndex = MainData.galleryArray[item].length - 1; detailImage.setImageResource(MainData .galleryArray[item][ImageIndex]); } } } detailImage.setElevation(0); return true; } Test the project on an emulator or handset. The process of gesture detection in the preceding code begins when the OnTouchListener listener's onTouch() method is called. It then passes that MotionEvent to our gesture detector class, GalleryGestureDetector, which monitors motion events, sometimes stringing them together and timing them until one of the recognized gestures is detected. At this point, we can enter our own code to control how our app responds as we did here with the onDown(), onShowPress(), and onFling() callbacks. It is worth taking a quick look at these methods in turn. It may seem, at the first glance, that the onDown() method is redundant; after all, it's the fling gesture that we are trying to catch. In fact, overriding the onDown() method and returning true from it is essential in all gesture detections as all the gestures begin with an onDown() event. The purpose of the onShowPress() method may also appear unclear as it seems to do a little more than onDown(). As the JavaDoc states, this method is handy for adding some form of feedback to the user, acknowledging that their touch has been received. The Material Design guidelines strongly recommend such feedback and here we have raised the view's elevation slightly. Without including our own code, the onFling() method will recognize almost any movement across the bounding view that ends in the user's finger being raised, regardless of direction or speed. We do not want very small or very slow motions to result in action; furthermore, we want to be able to differentiate between vertical and horizontal movement as well as left and right swipes. The MIN_DISTANCE and OFF_PATH constants are in pixels and VELOCITY_THRESHOLD is in pixels per second. These values will need tweaking according to the target device and personal preference. The first MotionEvent argument in onFling() refers to the preceding onDown() event and, like any MotionEvent, its coordinates are available through its getX() and getY() methods. The MotionEvent class contains dozens of useful classes for querying various event properties—for example, getDownTime(), which returns the time in milliseconds since the current onDown() event. In this example, we used GestureDetector.OnGestureListener to capture our gesture. However, the GestureDetector has three such nested classes, the other two being SimpleOnGestureListener and OnDoubleTapListener. SimpleOnGestureListener provides a more convenient way to detect gestures as we only need to implement those methods that relate to the gestures we are interested in capturing. We will shortly edit our Activity so that it implements the SimpleOnGestureListener instead, allowing us to tidy our code and remove the four callbacks that we do not need. The reason for taking this detour, rather than applying the simple listener to begin with, was to get to see all of the gestures available to us through a gesture listener and demonstrate how useful JavaDoc comments can be, particularly if we are new to the framework. For example, take a look at the following screenshot: Another very handy tool is the Dalvik Debug Monitor Server (DDMS), which allows us to see what is going on inside our apps while they are running. The workings of our gesture listener are a good place to do this as most of its methods operate invisibly. Viewing gesture activity with DDMS To view the workings of our OnGestureListener with DDMS, we need to first create a tag to identify our messages and then a filter to view them. The following steps demonstrate how to do this: Open the DetailActivity.java file. Declare the following constant: private static final String DEBUG_TAG = "tag"; Add the following line inside the onDown() method: Log.d(DEBUG_TAG, "onDown"); Add the line Log.d(DEBUG_TAG, "onShowPress"); to the onShowPress() method and do the same for each of our OnGestureDetector methods. Add the following lines to the appropriate clauses in onFling(): Log.d(DEBUG_TAG, "left"); Log.d(DEBUG_TAG, "right"); Open the Android DDMS pane from the Android tab at the bottom of the window or by pressing Alt + 6. If logcat is not visible, it can be opened with the icon to the right of the top-right drop-down menu. Click on this drop-down menu and select Edit Filter Configuration. Complete the dialog as shown in the following screenshot: You can now run the project on a handset or emulator and view, in the Logcat, which gestures are being triggered and how. Your output should resemble the one here: 02-17 14:39:00.990 1430- 1430/com.example.kyle.ancientbritain D/tag﹕ onDown 02-17 14:39:01.039 1430- 1430/com.example.kyle.ancientbritain D/tag﹕ onSingleTapUp 02-17 14:39:03.503 1430- 1430/com.example.kyle.ancientbritain D/tag﹕ onDown 02-17 14:39:03.601 1430- 1430/com.example.kyle.ancientbritain D/tag﹕ onShowPress 02-17 14:39:04.101 1430- 1430/com.example.kyle.ancientbritain D/tag﹕ onLongPress 02-17 14:39:10.484 1430- 1430/com.example.kyle.ancientbritain D/tag﹕ onDown 02-17 14:39:10.541 1430- 1430/com.example.kyle.ancientbritain D/tag﹕ onScroll 02-17 14:39:11.091 1430- 1430/com.example.kyle.ancientbritain D/tag﹕ onScroll 02-17 14:39:11.232 1430- 1430/com.example.kyle.ancientbritain D/tag﹕ onFling 02-17 14:39:11.680 1430- 1430/com.example.kyle.ancientbritain D/tag﹕ right 02-17 14:39:01.039   1430- 1430/com.example.kyle.ancientbritain D/tag﹕ onSingleTapUp DDMS is an invaluable tool when it comes to debugging our apps and seeing what is going on beneath the hood. Once a Log Tag has been defined in the code, we can then create a filter for it so that we see only the messages we are interested in. The Log class contains several methods to report information based on its level of importance. We used Log.d, which stands for debug. All these methods work with the same two parameters: Log.[method](String tag, String message). The full list of these methods is as follows: Log.v: Verbose Log.d: Debug Log.i: Information Log.w: Warning Log.e: Error Log.wtf: Unexpected error It is worth noting that most debug messages will be ignored during the packaging for distribution except for the verbose messages; thus, it is essential to remove them before your final build. Having seen a little more of the inner workings of our gesture detector and listener, we can now strip our code of unused methods by implementing GestureDetector.SimpleOnGestureListener. Implementing a SimpleOnGestureListener It is very simple to convert our gesture detector from one class of listener to another. All we need to do is change the class declaration and delete the unwanted methods. To do this, perform the following steps: Open the DetailActivity file. Change the class declaration for our gesture detector class to the following: class GalleryGestureDetector extends GestureDetector.SimpleOnGestureListener { Delete the onShowPress(), onSingleTapUp(), onScroll(), and onLongPress() methods. This is all you need to do to switch to the SimpleOnGestureListener. We have now successfully constructed and edited a gesture detector to allow the user to browse a series of images. You will have noticed that there is no onDoubleTap() method in the gesture listener. Double-taps can, in fact, be handled with the third GestureDetector listener, OnDoubleTapListener, which operates in a very similar way to the other two. However, Google, in its UI guidelines, recommends that a long press should be used instead, whenever possible. Before moving on to multitouch events, we will take a look at how to attach a GestureDetector listener to an entire Activity by adding a splash screen to our project. In the process, we will also see how to create a Full-Screen Activity and how to edit the Maniftest file so that our app launches with the splash screen. Adding a GestureDetector to an Activity The method we have employed so far allows us to attach a GestureDetector listener to any view or views and this, of course, applies to ViewGroups such as Layouts. There are times when we may want to detect gestures to be applied to the whole screen. For this purpose, we will create a splash screen that can be dismissed with a long press. There are two things we need to do before implementing the gesture detector: creating a layout and editing the Manifest file so that the app launches with our splash screen. Designing the splash screen layout The main difference between processing gestures for a whole Activity and an individual widget, is that we do not need an OnTouchListener as we can override the Activity's own onTouchEvent(). Here is how it is done: Create a new Blank Activity from the Project Explorer context menu called SplashActivity.java. The Activity wizard should have created an associated XML layout called activity_splash.xml. Open this and view it using the Text tab. Remove all the padding properties from the root layout so that it looks similar to this: <RelativeLayout android:layout_width="match_parent" android:layout_height="match_parent" tools:context="com.example.kyle.ancientbritain .SplashActivity"> Here we will need an image to act as the background for our splash screen. If you have not downloaded the project files from the Packt website, find an image, roughly of the size and aspect of your target device's screen, upload it to the project drawable folder, and call it splash. The file I used is 480 x 800 px. Remove the TextView that the wizard placed inside the layout and replace it with this ImageView: <ImageView android:id="@+id/splash_image" android:layout_width="wrap_content" android:layout_height="wrap_content" android:src="@drawable/splash"/> Create a TextView beneath this, such as the following: <TextView android:layout_width="match_parent" android:layout_height="wrap_content" android:layout_alignParentBottom="true" android:layout_centerHorizontal="true" android:layout_marginBottom="40dp" android:gravity="center_horizontal" android:textAppearance="?android:attr/ textAppearanceLarge" android:textColor="#fffcfcbd"/> Add the following text property: android:text="Welcome to <b>Ancient Britain</b>npress and hold nanywhere on the screennto start" To save time adding string resources to the strings.xml file, enter a hardcoded string such as the preceding one and heed the warning from the editor to have the string extracted for you like this: There is nothing in this layout that we have not encountered before. We removed all the padding so that our splash image will fill the layout; however, you will see from the preview that this does not appear to be the case. We will deal with this next in our Java code, but we need to edit our Manifest first so that the app gets launched with our SplashActivity. Editing the Manifest It is very simple to configure the AndroidManifest file so that an app will get launched with whichever Activity we choose; the way it does so is with an intent. While we are editing the Manifest, we will also configure the display to fill the screen. Simply follow these steps: Open the res/values-v21/styles.xml file and add the following style: <style name="SplashTheme" parent="android:Theme.Material. NoActionBar.Fullscreen"> </style> Open the AndroidManifest.xml file. Cut-and-paste the <intent-filter> element from MainActivity to SplashActivity. Include the following properties so that the entire <activity> node looks similar to this: <activity android:name=".SplashActivity" android:theme="@style/SplashTheme" android:screenOrientation="portrait" android:configChanges="orientation|screenSize" android:label="Old UK" > <intent-filter> <action android_name="android.intent.action.MAIN" /> <category android_name="android.intent.category.LAUNCHER" /> </intent-filter> </activity> We have encountered themes and styles before and, here, we took advantage of a built-in theme designed for full screen activities. In many cases, we might have designed a landscape layout here but, as is often the case with splash screens, we locked the orientation with the android:screenOrientation property. The android:configChanges line is not actually needed here, but is included as it is useful to know about it. Configuring any attribute such as this prevents the system from automatically reloading the Activity whenever the device is rotated or the screen size changed. Instead of the Activity restarting, the onConfigurationChanged() method is called. This was not needed here as the screen size and orientation were taken care of in the previous lines of code and this line was only included as a point of interest. Finally, we changed the value of android:label. You may have noticed that, depending on the screen size of the device you are using, the name of our app is not displayed in full on the home screen or apps drawer. In such cases, when you want to use a shortened name for your app, it can be inserted here. With everything else in place, we can get on with adding our gesture detector. This is not dissimilar to the way we did this before but, this time, we will apply the detector to the whole screen and will be listening for a long press, rather than a fling. Adding the GestureDetector Along with implementing a gesture detector for the entire Activity here, we will also take the final step in configuring our splash screen so that the image fills the screen, but maintains its aspect ratio. Follow these steps to complete the app splash screen. Open the SplashActivity file. Declare a GestureDetector as we did in the earlier exercise: private GestureDetector detector; In the onCreate() method, assign and configure our splash image and gesture detector like this: ImageView imageView = (ImageView) findViewById(R.id.splash_image); imageView.setScaleType(ImageView.ScaleType.CENTER_CROP); detector = new GestureDetector(this, new SplashListener()); Now, override the Activity's onTouchEvent() like this: @Override public boolean onTouchEvent(MotionEvent event) { this.detector.onTouchEvent(event); return super.onTouchEvent(event); } Create the following SimpleOnGestureListener class: private class SplashListener extends GestureDetector. SimpleOnGestureListener { @Override public boolean onDown(MotionEvent e) { return true; } @Override public void onLongPress(MotionEvent e) { startActivity(new Intent(getApplicationContext(), MainActivity.class)); } } Build and run the app on your phone or an emulator. The way a gesture detector is implemented across an entire Activity should be familiar by this point, as should the capturing of the long press event. The ImageView.setScaleType(ImageView.ScaleType) method is essential here; it is a very useful method in general. The CENTER_CROP constant scales the image to fill the view while maintaining the aspect ratio, cropping the edges when necessary. There are several similar ScaleTypes, such as CENTER_INSIDE, which scales the image to the maximum size possible without cropping it, and CENTER, which does not scale the image at all. The beauty of CENTER_CROP is that it means that we don't have to design a separate image for every possible aspect ratio on the numerous devices our apps will end up running on. Provided that we make allowances for very wide or very narrow screens by not including essential information too close to the edges, we only need to provide a handful of images of varying pixel densities to maintain the image quality on large, high-resolution devices. The scale type of ImageView can be set from within XML with android:scaleType="centerCrop", for example. You may have wondered why we did not use the built-in Full-Screen Activity from the wizard; we could easily have done so. The template code the wizard creates for a Full-Screen Activity provides far more features than we needed for this exercise. Nevertheless, the template is worth taking a look at, especially if you want a fullscreen that brings the status bar and other components into view when the user interacts with the Activity. That brings us to the end of this article. Not only have we seen how to make our apps interact with touch events and gestures, but also how to send debug messages to the IDE and make a Full-Screen Activity. Summary We began this article by adding a GestureDetector to our project. We then edited it so that we could filter out meaningful touch events (swipe right and left, in this case). We went on to see how the SimpleOnGestureListener can save us a lot of time when we are only interested in catching a subset of the recognized gestures. We also saw how to use DDMS to pass debug messages during runtime and how, through a combination of XML and Java, the status and action bars can be hidden and the entire screen be filled with a single view or view group. Resources for Article: Further resources on this subject: Speeding up Gradle builds for Android [Article] Saying Hello to Unity and Android [Article] Testing with the Android SDK [Article]
Read more
  • 0
  • 0
  • 7934
Modal Close icon
Modal Close icon