Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7018 Articles
article-image-defining-data-model-spatial-data-storage
Packt
04 Oct 2013
26 min read
Save for later

Defining a Data Model for Spatial Data Storage

Packt
04 Oct 2013
26 min read
(For more resources related to this topic, see here.) The spatial component of a real-world feature is the geometric representation of its shape in some coordinate space (either in 2D or 3D), and in vector space, this is referred to as its geometry. Oracle Spatial is designed to make spatial data management easier and more natural to users of location-enabled business applications and geographic information system (GIS) applications. Oracle allows the storage of spatial data in a table using the SDO_GEOMETRY data type that is just like any other data type in the database. Once the spatial data is stored in the Oracle database, it can be easily manipulated, retrieved, and related to all other data stored in the database. A spatial database should be designed just like any other database with a fully specified model. A fully specified model that is application independent should control the spatial data storage. A good data model supports and enhances application access without compromising the quality. In addition to these features, database features can be used to support applications that have limited functionality when it comes to table and column design. For example, some applications mandate a single spatial column per table or only a single homogeneous geometry type per spatial column. These limitations can be accommodated quite easily using database features such as views and triggers. In addition, there are a number of issues that arise when designing a data model that directly affects the data quality, performance, and access. The goal of this article is to give readers an understanding of how to model spatial data as SDO_GEOMETRY columns within tables, how to support spatial constraints for improved data quality, how to use synchronous and asynchronous triggers for implementing topological constraint checking, and to present methods for coping with multiple representations for faster web service access. All these issues, with solutions, are covered in this article: Defining a sample schema Using spatial metadata Using Oracle metadata views Using OGC metadata views Using different types of geometric representations Implementing tables with homogeneous and heterogeneous columns Implementing multiple representations for a single object Implementing multiple instances of a single column, for example, pre-thinned data for different scales and reprojection for faster web service access Restricting data access via views Using views to expose a single geometry type when multiple geometry types are present in the table Using views to expose tables with single geometry columns when multiple geometry columns are present in the table Implementing spatial constraints at the database level Restricting geometry types Spatial topological constraints Implementation of synchronous triggers Implementation of asynchronous triggers Defining a sample schema We will first define a sample schema that will be used for all the examples in this article. The schema is intended to model typical spatial assets maintained in a city-level GIS. Oracle Spatial provides all the functionality needed to model or describe the spatial properties of an asset (in modeling, it is often called an entity). This spatial description of an asset should not be treated differently from any other descriptive attribute. In addition, a data model should describe all assets/entities within it independently of any application. This should include, to the best of the ability of SQL, all business rules that define or control these assets/entities within the database, and these rules should be implemented using standard database practices. Defining the data model We use a schema with 12 tables to represent a spatial database for a city. This schema has tables to represent administrative areas managed at the city level, such as land parcels and neighborhoods, along with tables to manage natural features such as water boundaries. The LAND_PARCELS table has information about land at the lowest administrative level of the city. Buildings have to be fully contained in these land parcels. A table called BUILDING_FOOTPRINTS has information about all the buildings in the city. This table has the footprint of each building along with other information, such as name, height, and other attributes. Sets of neighborhoods are defined as a collection of land parcels to create more granular administrative areas. These neighborhoods are stored in the PLANNING_NEIGHBORHOODS table. There is a master table, BASE_ADDRESSES, to store information about all the valid street addresses in the city. Every record in the BUILDING_FOOTPRINTS table must have one parent record in this master address table. Note that the master address table does not list all the addresses of the apartments in a building. Rather, it stores one record for each street level address. So, each record in the BUILDING_FOOTPRINTS table has only one corresponding record in the master address table. There is also a master table, ROADS, that is used to store information about all the roads in the city. ROADS stores one record for each named road in the city so that all common information for the road can be stored together in one table. This is the only table in the schema without any geometry information. Each road in turn maps to a set of road segments that are stored in the ROAD_CLINES table. This table is used to store the geometric representation of center lines of road segments. This table also stores information about address ranges on these road segments. Road segments typically have different address ranges on the left side of the road and on the right side of the road. Each road segment also has a parent ROAD_ID associated with it from the ROADS table. A city usually manages sidewalks and other assets, such as street lamps, trashcans, and benches that are placed on these sidewalks. The SIDEWALKS table stores the information for all the sidewalks managed by the city. The CITY_FURNITURE table stores all the data corresponding to the assets, such as benches, streetlights, and trashcans. The ORTHO_PHOTOS table stores the collected information using aerial photography. The raster information stored in this table can be used to look for changes over time for the built-in features of the city. The water features of the city are stored in two different tables: the WATER_LINE table is used to store the water line features, such as creeks, rivers, and canals. The WATER_AREA table is used to store area features, such as lakes, rivers, and bays. The following figure shows the entity-relationship (E-R) diagram for this data model: The following figure shows the further E-R diagram for same data model: Creating tables in the schema Create a user called BOOK and assign it a password. Load the script <schema_load.sql> and it will create the tables required for running the examples described in this article. It will also create the Oracle Spatial metadata required for these tables. The following privileges are granted to the BOOK user: grant connect,resource to Book identified by <password>; grant connect, resource to book; grant create table to book; grant create view to book; grant create sequence to book; grant create synonym to book; grant create any directory to book; grant query rewrite to book; grant unlimited tablespace to book; Understanding spatial metadata Oracle Spatial requires certain metadata before the spatial data can be meaningfully used by applications. The database views that contain this metadata also act as a catalog for all the spatial data in the database. There are two basic views defined to store this metadata information: USER_SDO_GEOM_METADATA and ALL_SDO_GEOM_METADATA. The USER_ view is used to create a metadata entry for a single SDO_GEOMETRY column within a database table or view. An entry must be created for each SDO_GEOMETRY column within a table; entries for SDO_GEOMETRY columns in views are optional. If a table has more than one column of type SDO_GEOMETRY, then there is one metadata entry for each column of spatial data in that table. The ALL_ view shows all of the spatial layers that can be accessed by the current user. If a user has the Select grant on another user’s table with SDO_GEOMETRY columns, the first user can see the metadata entries for those tables in the ALL_ view. The views are set up so that owners of the spatial tables or views can create the metadata for them. And, the owner of a layer can grant read access to a layer to other users in the system. Granting a Select privilege on the table or view to other users will let them see the metadata for these tables and views. The ALL_ view displays all the spatial tables owned by the user along with other spatial tables for which the current user has read access. Spatial Reference System Each SDO_GEOMETRY object has a Spatial Reference System (SRS) associated with it, and all the SDO_GEOMETRY objects in a column should have the same SRS. In Oracle Spatial, a Spatial Reference ID (SRID) is used to associate an SRS with SDO_GEOMETRY objects. There are cases (for example, engineering drawings) where there is no SRS associated with an SDO_GEOMETRY object. In such cases, a NULL SRID is used to denote that the spatial data has no spatial reference information. An SRS can be geographic or non-geographic. A geographic SRS is used when the spatial data is used to represent features on the surface of the Earth. These types of SRS usually have a reference system that can relate the coordinates of the spatial data to locations on Earth. A unit of measurement is also associated with an SRS so that measurements can be done using a well-defined system. A non-geographic SRS is used when the spatial data is not directly related to locations on Earth. But these systems usually have a unit of measurement associated with them. Building floor plans is a good example of spatial data that is often not directly related to locations on Earth. A geographic system can be either geodetic or projected. Coordinates in a geodetic system are often described using longitude and latitude. In Oracle Spatial, the convention is to use longitude as the first coordinate and latitude as the second coordinate. A projected system is a Cartesian system that is defined as a planar projection based on the datum and projection parameters. Before an entry is created for a layer of data, the SRID associated with the data should be identified along with the tolerance to be used for the spatial layer. All the spatial data has an inherent accuracy associated with it. Hence, the tolerance value used for a spatial layer is very important and should be determined based on the accuracy of the data. Once these two values are identified, you are ready to create the metadata for the spatial layer. More on Spatial Reference Systems Oracle Spatial supports hundreds of SRSs, and it is very important to choose the right SRS for any given data set. The definition of an SRS can be easily obtained by looking at the well-known text (WKT) for that SRS. The WKTs for all the SRSs supplied as part of Oracle Spatial are available from the MDSYS.CS_SRS view. In addition to this view, there are several other metadata tables under MDSYS that contain more details on how these SRSs are defined. Oracle Spatial also supports the EPSG standard-based SRSs. SRS in Oracle Spatial is flexible and allows users to define new reference systems if they are not present in the supplied SRSs. Creating spatial metadata The tables used in our sample schema contain data that is geographically referenced. Spatial metadata can be created using Insert statements into the USER_SDO_GEOM_METADATA view. This view is defined as a public-writable view on top of MDSYS.SDO_GEOM_METADATA_TABLE that is used to store metadata for all the spatial columns in the database. Let us look at the metadata creation process for some of the spatial tables. Most of the tables used in the current schema have spatial data in the California State Plane Zone 3 Coordinate System. In Oracle Spatial, the corresponding SRID for this SRS is 2872. This coordinate system has foot as the unit of measurement and we will use 0.05 as our tolerance (that is, five-hundredths of a foot). The metadata is created using the Insert statement as shown in the following code: Insert Into USER_SDO_GEOM_METDATA Values (‘LAND_PARCELS’, ‘GEOM’, SDO_DIM_ARRAY(SDO_DIM_ARRAY(SDO_DIM_ELEMENT(‘X’, 5900000, 6100000, 0.05), SDO_DIM_ELEMENT(‘Y’,2000000, 2200000, 0.05)), 2872); The SDO_DIM_ELEMENT object is used to specify the lower and upper bounds for each dimension of the coordinate system along with the tolerance value. The metadata allows one entry for each dimension, even though it is very common to use the same tolerance value for the X and Y dimensions. When storing 3D data, it is very common to use a different tolerance value for the Z dimension. The BASE_ADDRESSES table has geometries stored in two columns: GEOMETRY and GEOD_GEOMETRY. The GEOMETRY column has data in the 2872 SRID, while the GEOD_GEOMETRY column has data in longitude and latitude. As this is a geodetic system, the tolerance for such systems is required to be in meters. So, a tolerance of 0.05 means a tolerance of 5cm. For geodetic data, it is recommended that the tolerance should not be less than 5cm for all of the topology and distance-based operations. Insert Into USER_SDO_GEOM_METDATA Values(‘BASE_ADDRESSES’, ‘GEOD_GEOM’, SDO_DIM_ARRAY((SDO_DIM_ELEMENT(‘Longitude’, -122.51436, -122.36638, .05), SDO_DIM_ELEMENT(‘Latitude’, 37.7081463, 37.8309382, .05)), 8307); As this is a geodetic system, the longitude range goes from -180 to 180 and the latitude range goes from -90 to 90. Even though it is normal practice to use these ranges for the metadata entry, many developers use the actual ranges spanned by the SDO_GEOMETRY object. Mapping tools and applications typically use this extent from the metadata to compute the initial extent of the data in each column of the spatial data. Any application looking for all the spatial columns in the database should select the data from the ALL_SDO_GEOM_METADATA view. This will return one row for each column of spatial data in the database that is visible to the current user. OGC-defined metadata views Open Geospatial Consortium (OGC) defines a different set of standardized metadata views. OGC standard metadata can be defined using a new set of tables or views in Oracle Spatial. For a simple solution for the OGC metadata schema, we will show a view-based implementation using the Oracle Spatial metadata table. All Oracle supplied packages, functions, and types of Oracle Spatial are in the MDSYS schema. It is generally not recommended to create any user objects under this schema as it might cause problems during database upgrades. Oracle also supplies another predefined schema called MDDATA that can be used for Oracle Spatial-related user objects that are general purpose in nature. We use this MDDATA schema to create the OGC metadata views. This user comes locked and it is recommended that you do not unlock this user. But, it does require a few privileges to make the following code work, so grant those privileges as required. Connect to the database as a user with SYSDBA privileges and execute all the following steps as the MDDATA user by changing the current schema to MDDATA. We need to grant an explicit Select privilege on SDO_GEOM_METADATA_TABLE to MDDATA. Alter session set current_schema=MDDATA; GRANT Select on MDSYS.SDO_GEOM_METADATA_TABLE to MDDATA; The OGC standard requires the geometry type as part of the metadata view. But, this is not part of the MDSYS owned metadata view and has to be computed based on the geometry table information stored in the MDSYS table. So, first define a function that can compute the geometry type based on the rows in the spatial tables. Note that this function just looks at the first non-NULL geometry and returns the type of that geometry. Users can modify this to make it look at the whole table to decide on the geometry type, but it can be a very expensive operation. Create Or Replace Function MDDATA.GET_GEOMETRY_TYPE (tsname varchar2, tname varchar2, cname varchar2) Return Number IS gtype number; Begin Begin execute immediate ‘ Select a.’|| SYS.DBMS_ASSERT.ENQUOTE_NAME(cname, false)|| ‘.sdo_gtype From ‘|| SYS.DBMS_ASSERT.ENQUOTE_NAME(tsname, false)||’.’|| SYS.DBMS_ASSERT.ENQUOTE_NAME(tname, false)|| ‘ a Where a.’|| SYS.DBMS_ASSERT.ENQUOTE_NAME(cname, false)|| ‘ is not null and rownum < 2’ Into gtype; Return gtype MOD 100; EXCEPTION When OTHERS Then Return 4; End; End; Notice all the uses of the ENQUOTE_NAME function from the SYS.DBMS_ASSERT package. This is used to avoid any possible SQL injection issues typically associated with functions that create SQL statements using the user supplied SQL. As we are creating a general purpose function that can be invoked by any user directly or indirectly, it is a good idea to protect the function from any possible SQL injection. Next, we define an OGC metadata view to see all the rows from the MDSYS owned metadata table. Create Or Replace View MDDATA.OGC_GEOMETRY_COLUMNS As Select GM.SDO_OWNER As F_TABLE_SCHEMA, GM.SDO_TABLE_NAME As F_TABLE_NAME, GM.SDO_COLUMN_NAME As F_GEOMETRY_COLUMN, Get_Geometry_Type(GM.sdo_owner, GM.sdo_table_name, GM.sdo_column_name) As GEOMETRY_TYPE, (Select count(*) From Table(GM.SDO_DIMINFO) ) As COORD_DIMENSION, GM.SDO_SRID As SRID From MDSYS.SDO_GEOM_METADATA_TABLE GM; And finally, we define a user view that will show all the geometry columns that are visible to the current user. Create Or Replace View GEOMETRY_COLUMNS As Select b.F_TABLE_SCHEMA , b.F_TABLE_NAME , b.F_GEOMETRY_COLUMN, b.COORD_DIMENSION, b.SRID, b.GEOMETRY_TYPE From MDDATA.OGC_GEOMETRY_COLUMNS b, ALL_OBJECTS a Where b.F_TABLE_NAME = a.OBJECT_NAME And b.F_TABLE_SCHEMA = a.OWNER And a.OBJECT_TYPE in (‘TABLE’, ‘SYNONYM’, ‘VIEW’); Grant Select On MDDATA.GEOMETRY_COLUMNS to public; Create PUBLIC SYNONYM GEOMETRY_COLUMNS FOR MDDATA.GEOMETRY_COLUMNS; Tolerance in Oracle Spatial Tolerance is used in Oracle Spatial to associate a level of precision with the data and to check the validity of geometries among other things. Tolerance should be derived based on the resolution and accuracy of the data. If the devices or methods used to collect the spatial data are correct up to a five-meter resolution, the tolerance for that layer should be set to 5 meters. The actual tolerance value, inserted into the metadata view depends on the real-world tolerance value and the unit of measurement used in the coordinate system is used for the column of spatial data. For example, let the tolerance for the spatial data be 5 centimeters and the unit of measurement of the coordinate system used for the spatial column is feet. Then, the five-centimeter value should first be converted to feet (1 centimeter is 0.032 feet)—this comes out to be 0.164 feet. So, you use a value of 0.164 for tolerance in the metadata. In practice, Oracle Spatial uses the following rules based on tolerance to determine if the geometry is valid or not. These are in addition to other topological consistency rules (as described by the OGC Simple Feature Specification) used to check the validity of geometries: If the distance between two consecutive vertices in the geometry is less than the tolerance value, the geometry is invalid. This rule applies to line-string and polygon type geometries. If the distance between a vertex and the nearest edge to that vertex in a polygon is less than the tolerance value, the geometry is invalid. This rule only applies to the polygon type geometries. Managing homogeneous and heterogeneous data If a spatial column in a table contains data of one single geometry type (for example, polygon or line-string, but not both), we can say that spatial data in that column is homogeneous. In other situations, a column may contain data from one or more geometry types (heterogeneous representation). For example, the spatial description of a rare and endangered flora object may normally be a single plant (a plant via a point), but in other situations, it may be an area (a patch via a polygon). Consider the CITY_FURNITURE table that is used for storing city assets like benches, trashcans, and streetlights. The geometry for the benches is represented using line-strings, while streetlights and trashcans are represented with points. It is perfectly correct, semantically, to store different types of observations within a single geometry column. However, while some mapping software systems can cope with multiple geometry types per SDO_GEOMETRY column, others, such as some traditional GIS packages, require homogeneity. We will describe how to achieve this next, when a column in the table has heterogeneous data. We define three views on top of the CITY_FURNITURE table corresponding to each of the types of data stored in the table. This table has three classes of objects: benches, trashcans, and streetlights. After the views are defined, we also need to create metadata entries for these views in USER_SDO_GEOM_METADATA so that any GIS tool can discover these views as if they are tables. Create the database views corresponding to each of the three types of data stored in the CITY_FURNITURE table. -- DDL for View CITY_FURN_BENCHES Create Or Replace FORCE VIEW CITY_FURN_BENCHES (FID, FEATTYPE, GEOM) As Select FID, FEATTYPE, GEOM From CITY_FURNITURE Where FEATTYPE=’BENCH’; -- DDL for View CITY_FURN_LIGHTS Create Or Replace FORCE VIEW CITY_FURN_LIGHTS (FID, FEATTYPE, GEOM) As Select FID, FEATTYPE, GEOM From CITY_FURNITURE Where FEATTYPE=’LIGHT’; -- DDL for View CITY_FURN_TRASHCANS Create Or Replace FORCE VIEW CITY_FURN_TRASHCANS (FID, FEATTYPE, GEOM) As Select FID, FEATTYPE, GEOM From CITY_FURNITURE Where FEATTYPE=’TRASHCAN’; The preceding examples show how to use other relational attributes to create the required views. Another way to do this is to constrain based on the SDO_GTYPE attribute of the SDO_GEOMETRY column. The following example shows how to do this for one of the preceding views, as the rest can be done with similar SQL: -- DDL for View CITY__FURN_BENCHES Create Or Replace FORCE VIEW CITY_FURN_BENCHES (FID, FEATTYPE, GEOM) AS Select FID, FEATTYPE, GEOM From CITY_FURNITURE A Where A.GEOM.SDO_GTYPE = 2002; Now create the metadata for each of these views so that any GIS can access this as if it is stored in a separate table. Note that these additional metadata entries are not required for the correct usage of Oracle Spatial. They are created only to facilitate the GIS tools that don’t support heterogeneous data in spatial columns. Insert Into USER_SDO_GEOM_METADATA Values ( ‘CITY_FURN_BENCHES’, ‘GEOM’, SDO_DIM_ARRAY(SDO_DIM_ELEMENT(‘X’, 5900000, 6100000, .05), SDO_DIM_ELEMENT(‘Y’, 2000000, 2200000, .05)), 2872); Insert Into USER_SDO_GEOM_METADATA Values( ‘CITY_FURN_LIGHTS’, ‘GEOM’, SDO_DIM_ARRAY(SDO_DIM_ELEMENT(‘X’, 5900000, 6100000, .05), SDO_DIM_ELEMENT(‘Y’, 2000000, 2200000, .05)), 2872); Insert Into USER_SDO_GEOM_METADATA Values( ‘CITY_FURN_TRASHCANS’, ‘GEOM’, SDO_DIM_ARRAY(SDO_DIM_ELEMENT(‘X’, 5900000, 6100000, .05), SDO_DIM_ELEMENT(‘Y’, 2000000, 2200000, .05)), 2872); How metadata is used Applications typically look at the ALL_SDO_GEOM_METADATA view to see the spatial tables available in the database for a given user. If you select the data from this view, now you will see 11 rows returned: 8 rows corresponding to tables and 3 rows corresponding to the views defined in the CITY_FURNITURE table. From an application point of view, it does not make any difference whether this data is stored in a view or a table. It will all look the same to the application. Sometimes it is useful to constrain the type of spatial data stored in the table to be homogeneous. For example, the ROAD_CLINES table should contain only linear geometries, as the roads are usually geometries of line type. This can be done by constraints that can be imposed by the spatial index defined in the ROAD_CLINES table. While creating the spatial index, provide the LAYER_GTYPE keyword and specify the type of data that will be stored in this table. -- DDL for Index ROAD_CLINES_SIDX Create Index ROAD_CLINES_SIDX ON ROAD_CLINES (GEOM) INDEXTYPE IS MDSYS.SPATIAL_INDEX PARAMETERS (‘LAYER_GTYPE=LINE’); Now, if you try to insert a row with a geometry that has a different SDO_GTYPE attribute than 2002, it will raise an error. Insert Into ROAD_CLINES Values ( 198999, 402, 0, 0, 2300, 2498, 190201, 23564000, 23555000, 94107, 10, ‘Y’, ‘DPW’, ‘Potrero Hill’, ‘190201’, ‘03RD ST’, ‘3’, ‘3RD’, SDO_GEOMETRY(2001, 2872, NULL, SDO_ELEM_INFO_ARRAY(1, 1, 1), SDO_ORDINATE_ARRAY( 6015763.86, 2104882.29))); * ERROR at line 1: ORA-29875: failed in the execution of the ODCIINDEXInsert routine ORA-13375: the layer is of type [2002] while geometry inserted has type [2001] ORA-06512: at “MDSYS.SDO_INDEX_METHOD_10I”, line 720 ORA-06512: at “MDSYS.SDO_INDEX_METHOD_10I”, line 225 The error message clearly indicates that the row that is currently being inserted has geometry with the wrong SDO_GTYPE attribute. This is the easiest way to strictly enforce the GTYPE constraints on the spatial data. However, this has the problem of rejecting the whole row when the geometry type does not match the LAYER_GTYPE keyword. And it is also not easy to log these cases as the error is thrown and the database moves on to process the next Insert statement. In some cases, the user might still want to insert the row into the table, but record cthe fact that there is invalid data in the row. Users can then come back and look at all the invalid entries and fix the issues. We will describe a few methods to do this logging and error processing later in this article. Using database check constraints Specifying the layer_gtype keyword is not the only way to constrain the type in a spatial layer. One can also use table level constraints to achieve the same result. With constraints, users get the additional benefit of specifying more complex constraints, such as allowing only points and lines in a layer. However, if only a single geometry type constraint is required, it is better to implement that constraint using the LAYER_GTYPE method as this is more efficient than the check constraint. These constraints can also be enforced with database triggers, and these trigger-based constraints are discussed in a later section. Alter Table CITY_FURNITURE ADD Constraint city_furniture_gtype_ck CHECK ( geom.sdo_gtype in (2002, 2001) ); Insert Into CITY_FURNITURE Values (432432, 'BENCH', SDO_GEOMETRY(2003,2872,NULL, SDO_ELEM_INFO_ARRAY(1, 1003, 3), SDO_ORDINATE_ARRAY(6010548.53, 2091896.34, 6010550.45,2091890.11))); This will fail with the following error: ERROR at line 1: ORA-02290: check Constraint (BOOK.CITY_FURNITURE_GTYPE_CK) violated Similarly, this approach can be useful for an OBJTYPE column check in the CITY_FURNITURE table. Alter Table CITY_FURNITURE ADD Constraint city_furniture_type_ck CHECK ( feattype in (‘BENCH’,’LIGHT’,’TRASHCAN’) ); Insert Into CITY_FURNITURE Values (432432, ‘LIGHTS’, SDO_GEOMETRY(2001,2872, SDO_POINT_TYPE(6010548.53, 2091896.34, NULL), NULL, NULL)); ERROR at line 1: ORA-02290: check Constraint (BOOK.CITY_FURNITURE_TYPE_CK) violated Now these two constraints are checking two independent columns, but what we really need is a more complex check to ensure each value of OBJTYPE has the corresponding SDO_GEOMETRY with the right type. That is, we want to make sure that TRASHCAN and LIGHT types have a point geometry and BENCH has a line geometry. Alter Table CITY_FURNITURE Drop Constraint city_furniture_gtype_ck; Alter Table CITY_FURNITURE Drop Constraint city_furniture_type_ck; Alter Table CITY_FURNITURE ADD Constraint city_furniture_objtype_geom_ck CHECK ( ( (“FEATTYPE”=’TRASHCAN’ Or “FEATTYPE”=’LIGHT’) AND “GEOM”.”SDO_GTYPE”=2001 ) Or (“FEATTYPE”=’BENCH’ AND “GEOM”.”SDO_GTYPE”=2002) /* Else Invalid combination */ ) ; Insert Into CITY_FURNITURE Values (432432, ‘BENCH’, SDO_GEOMETRY(2001,2872, SDO_POINT_TYPE(6010548.53, 2091896.34, NULL), NULL, NULL)); ERROR at line 1: ORA-02290: check Constraint (BOOK.CITY_FURNITURE_TYPE_CK) violated Multiple representations for the same objects In some situations, it is beneficial to have multiple representations for the same geometric feature. For example, an address usually has a point representation for its location. If a footprint of a building is associated with the address, then that footprint will be represented as a polygon. In some cases, a building might have many different point locations associated with it. One point may allow a GIS application to draw an icon for the building depending on its function (for example, a fire station). Another point may allow the building to be labeled with its street address, and finally another one may show an alternate location that is used for main delivery or emergency services entry at the back of the building. Similarly, a land parcel table can have an interior point of the parcel represented as point geometry in addition to the polygon representation. For a visualization application, it is sometimes useful to represent the land parcel as a point. When a map is displayed at a smaller scale (city level), the map will be cluttered if each land parcel is displayed as a polygon. In such cases, if land parcels are displayed as points with a suitable icon, the map will be less cluttered. When the map is displayed at a larger scale (street level), the same land parcel can be displayed as a polygon. Oracle Spatial allows such multiple representations by allowing multiple SDO_GEOMETRY columns in the same table. We first start with the BUILDING_FOOTPRINTS table and alter it to add an additional SDO_GEOMETRY column to allow the street address to be represented at the center of the building via a point feature. We can use a spatial function that can compute a point inside a polygon automatically to populate this column. Summary A spatial database application should be designed just like any other database application. It should follow the same standard data model practices like any other database application. In this article, we introduced a data model that can be used in a city-wide spatial data management system. This data model is used throughout the article to illustrate different aspects of spatial database application development. We discussed the use of triggers and queues to manage spatial data in the database. We also showed how to design database level constraints for spatial data management. We introduced the concepts of database triggers and queues and showed how they can be used for spatial data management in the database. Resources for Article: Further resources on this subject: Remote Job Agent in Oracle 11g Database with Oracle Scheduler [Article] Introduction to Oracle Service Bus & Oracle Service Registry [Article] Configuration, Release and Change Management with Oracle [Article]
Read more
  • 0
  • 0
  • 4982

article-image-let-there-be-light
Packt
04 Oct 2013
8 min read
Save for later

Let There be Light!

Packt
04 Oct 2013
8 min read
(For more resources related to this topic, see here.) Basic light sources You use lights to give a scene brightness, ambience, and depth. Without light, everything looks flat and dull. Use additional light sources to even-out lighting and to set up interior scenes. In Unity, lights are components of GameObjects. The different kinds of light sources are as follows: Directional lights: These lights are commonly used to mimic the sun. Their position is irrelevant, as only orientation matters. Every architectural scene should at least have one main Directional light. When you only need to lighten up an interior room, they are more tricky to use, as they tend to brighten up the whole scene, but they help getting some light through the windows, inside the project. We'll see a few use cases in the next few sections. Point lights: These lights are easy to use, as they emit light in any direction. Try to minimize their Range, so they don't spill light in other places. In most scenes, you'll need several of them to balance out dark spots and corners and to even-out the overall lighting. Spot lights: These lights only emit light into a cone and are good to simulate interior light fixtures. They cast a distinct bright circular light spot so use them to highlight something. Area lights: These are the most advanced lights, as they allow a light source to be given an actual rectangular size. This results in smoother lights and shadows, but their effect is only visible when baking and they require a pro-license. They are good to simulate light panels or the effect of light coming in through a window. In the free version, you can simulate them using multiple Spot or Directional Lights. Shadows Most current games support some form of shadows. They can be pre-calculated or rendered in real-time. Pre-calculation implies that the effect of shadows and lighting is calculated in advance and rendered onto an additional material layer. It only makes sense for objects that don't move in the scene. Real-time shadows are rendered using the GPU, but can be computationally expensive and should only be used for dynamic lighting. You might be familiar with real-time shadows from applications such as SketchUp and recent versions of ArchiCAD or Revit. Ideally, both techniques are combined. The overall lighting of the scene (for example, buildings, street, interiors, and so on) is pre-calculated and baked in texture maps. Additional real-time shadows are used on the moving characters. Unity can blend both types of shadows to simulate dynamic lighting in large scenes. Some of these techniques, however, are only supported in the pro-version. Real-time shadows Imagine we want to create a sun or shadow study of a building. This is best appreciated in real-time and by looking from the outside. We will use the same model as we did in the previous article, but load it in a separate scene. We want to have a light object acting as a sun, a spherical object to act as a visual clue where the sun is positioned and link them together to control the rotations in an easy way. The steps to be followed to achieve this are as follows: Add a Directional light, name it SunLight and choose the Shadow Type. Hard shadows are more sharply defined and are the best choice in this example, whereas Soft shadows look smoother and are better suited for a subtle, mood lighting. Add an empty GameObject by navigating to GameObject | Create Empty that is positioned in the center of the scene and name it ORIGIN. Create a Sphere GameObject by navigating to GameObject | Create Other | Sphere, name it VisualSun. Make it a child of the ORIGIN by dragging the VisualSun name in the Hierarchy Tab onto the ORIGIN name. Give it a bright, yellow material, using a Self-Illumin/Diffuse Shader. Deactivate Cast Shadows and Receive Shadows on the Mesh Renderer component. After you have placed the VisualSun as a child of the origin object, reset the position of the Sphere to be 0 for X, Y and Z. It now sits in the same place as its parent. Even if you move the parent, its local position stays at X=0, Y=0 and Z=0, which makes it convenient for a placement relative to its parent object. Alter the Z-position to define an offset from the origin, for example 20 units. The negative Z will facilitate the SunLight orientation in the next step. The SunLight can be dragged onto the VisualSun and its local position reset to zero as well. When all rotations are also zero, it emits light down the Z-axis and thus straight to the origin. If you want to have a nice glow effect, you can add a Halo by navigating to Components | Effects | Halo and then to SunLight and setting a suitable Size. We now have a hierarchic structure of the origin, the visible sphere and the Directional light, that is accentuated by the halo. We can adjust this assembly by rotating the origin around. Rotating around the Y-axis defines the orientation of the sun, whereas a rotation around the X-axis defines the azimuth. With these two rotations, we can position the sun wherever we want. Lightmapping Real-time lighting is computationally very expensive. If you don't have the latest hardware, it might not even be supported. Or you might avoid it for a mobile app, where hardware resources are limited. It is possible to pre-calculate the lighting of a scene and bake it onto the geometry as textures. This process is called Lightmapping, for more information on it visit: http://docs.unity3d.com/Documentation/Manual/Lightmapping.html While actual calculations are rather complex, the process in Unity is made easy, thanks to the integrated Beast Lightmapping. There are a few things you need to set up properly. These are given as follows: First, ensure that any object that needs to be baked is set to Static. Each GameObject has a static-toggle, right next to the Name property. Activate this for all models and light objects that will not move in the Scene. Secondly, ensure that all geometry has a second set of texture coordinates, called UV2 coordinates in Unity. Default GameObjects have those set up, but for imported models, they usually need to be added. Luckily for us, this is automated when Generate Lightmap UVs is activated on the model import settings given earlier in Quick Walk Around Your Design, in the section entitled, Controlling the import settings. If all lights and meshes are static and UV2 coordinates are calculated, you are ready to go. Open the Lightmapping dialog by navigating to Window | Lightmapping and dock it somewhere conveniently. There are several settings, but we start with a basic setup that consists of the following steps: Usually a Single Lightmap suffices. Dual Lightmaps can look better, but require the deferred rendering method that is only supported in Unity Pro. Choose the Quality High modus. Quality Low gives jagged edges and is only used for quick testing. Activate Ambient Occlusion as a quick additional rendering step that darkens corners and occluded areas, such as where objects touch. This adds a real sense of depth and is highly recommended. Set both sliders somewhere in the middle and leave the distance at 0.1, to control how far the system will look to detect occlusions. Start with a fairly low Resolution, such as 5 or 10 texels per world unit. This defines how detailed the calculated Lightmap texture is, when compared to the geometry. Look at the Scene view, to get a checkered overlay visible, when adjusting Lightmapping settings. For final results, increase this to 40 or 50, to give more detail to the shadows, at the cost of longer rendering times. There are additional settings for which Unity Pro is required, such as Sky Light and Bounced Lighting. They both add to the realism of the lighting, so they are actually highly recommended for architectural visualization, if you have the pro-version. On the Object sub-tab, you can also tweak the shadow calculation settings for individual lights. By increasing the radius, you get a smoother shadow edge, at the cost of longer rendering times. If you increase the radius, you should also increase the amount of samples, which helps reduce the noise that gets added with sampling. This is shown in the following screenshot: Now you can go on and click Bake Scene. It can take quite some time for large models and fine resolutions. Check the blue time indicator on the right side of the status bar (but you can continue working in Unity). After the calculations are finished, the model is reloaded with the new texture and baked shadows are visible in Scene and Game views, as shown in the following screenshot: Beware that to bake a Scene, it needs to be saved and given a name, as Unity places the calculated Lightmap textures in a subfolder with the same name as the Scene. Summary In this article we learned about the use of different light sources and shadow. To avoid the heavy burden of real-time shadows, we discussed the use Lightmapping technique to bake lights and shadows on the model, from within Unity Resources for Article: Further resources on this subject: Unity Game Development: Welcome to the 3D world [Article] Introduction to Game Development Using Unity 3D [Article] Unity 3-0 Enter the Third Dimension [Article]    
Read more
  • 0
  • 0
  • 12586

article-image-cql-client-applications
Packt
04 Oct 2013
7 min read
Save for later

CQL for client applications

Packt
04 Oct 2013
7 min read
(For more resources related to this topic, see here.) Using the Thrift API The Thrift library is based on the Thrift RPC protocol. High-level clients built over it have been a standard way of building an application for a long time. In this section, we'll explain how to write a client application using CQL as the query language and thrift as the Java API. When we start Cassandra, by default it listens to Thrift clients (start_rpc: true property in the CASSANDRA_HOME/conf/cassandra.yaml file enables this). Let's build a small program that connects to Cassandra using the Thrift API, and runs CQL 3 queries for reading/writing data in the UserProfiles table we created for the facebook application. The program can be built by performing the following steps: For downloading the Thrift Library, you need to enter apache-assandra-thrift-1.2.x.jar (which is to be found in the CASSANDRA_HOME/lib folder) into your classpath. If your Java project is mavenized, you need to insert the following entry in pom.xml under the dependency section (version will vary depending upon your Cassandra server installation): <dependency> <groupId>org.apache.cassandra</groupId> <artifactId>cassandra-thrift</artifactId> <version>1.2.5</version> </dependency> For connecting to the Cassandra server on a given host and port, you need to open org.apache.thrift.transport.TTransport to the Cassandra node and create an instance of org.apache.cassandra.thrift.Cassandra.Client as follows: TTransport transport = new TFramedTransport(new TSocket("localhost", 9160)); TProtocol protocol = new TBinaryProtocol(transport); Cassandra.Client client = new Cassandra.Client(protocol); transport.open(); client.set_cql_version("3.0.0"); The default CQL version for Thrift is 2.0.0. You must set it to 3.0.0 if you are writing CQL 3 queries and don't want to see any version related errors. After you are done with transport, close it gracefully (usually at the end of read/write operations) as follows: transport.close(); Creating a schema : The executeQuery() utility method accepts String CQL 3 query and runs it: CqlResult executeQuery(String query) throws Exception { return client.execute_cql3_query(ByteBuffer.wrap(query.getBytes("UTF-8")), Compression.NONE, ConsistencyLevel.ONE); } Now, create keyspace and the table by directly executing CQL 3 query: //Create keyspace executeQuery("CREATE KEYSPACE facebook WITH replication = "{'class':'SimpleStrategy','replication_factor':3};"); executeQuery("USE facebook;"); //Create table executeQuery("CREATE TABLE UserProfiles(" +"email_id text," + "password text,"+ "name text," + "age int," + "profile_picture blob," + "PRIMARY KEY(email_id)" + ");" ); Reading/writing data: A couple of records can be inserted as follows: executeQuery("USE facebook;"); executeQuery("INSERT INTO UserProfiles(email_id, password, name, age, profile_picture) VALUES('john.smith@example.com','p4ssw0rd',' John Smith',32,0x8e37);"); executeQuery("INSERT INTO UserProfiles(email_id, password, name, age, profile_picture) VALUES('david.bergin@example.com','guess1t',' David Bergin',42,0xc9f1);"); Executing the SELECT query returns CQLResult, on which we can iterate easily to fetch records: CqlResult result = executeQuery("SELECT * FROM facebook.UserProfiles " + "WHERE email_id = 'john.smith@example.com';"); for (CqlRow row : result.getRows()) { System.out.println(row.getKey(); } Using the Datastax Java driver The Datastax Java driver is based on the Cassandra binary protocol that was introduced in Cassandra 1.2, and works only with CQL 3. The Cassandra binary protocol is specifically made for Cassandra in contrast to Thrift, which is a generic framework and has many limitations. Now, we are going to write a Java program that uses the Datastax Java driver for reading/writing data into Cassandra, by performing the following steps: Downloading the driver library : This driver library JAR file must be in your classpath in order to build an application using it. If you have a maven-based Java project, you need to insert the following entry into the pom.xml file under the dependeny section: <dependency> <groupId>com.datastax.cassandra</groupId> <artifactId>cassandra-driver-core</artifactId> <version>1.0.1</version> </dependency> This driver project is hosted on Github: (https://github.com/datastax/java-driver). It makes sense to check and download the latest version. Configuring Cassandra to listen to native clients : In the newer version of Cassandra, this would be enabled by default and Cassandra will listen to clients using binary protocol. But the earlier Cassandra installations may require enabling this. All you have to do is to check and enable the start_native_transport property into the CASSANDRA_HOME/conf/Cassandra.yaml file by inserting/uncommenting the following line: start_native_transport: true The port that Cassandra will use for listening to native clients is determined by the native_transport_port property. It is possible for Cassandra to listen to both Thrift and native clients simultaneously. If you want to disable Thrift, just set the start_rpc property to false in CASSANDRA_HOME/conf/Cassandra.yaml. Connecting to Cassandra : The com.datastax.driver.core.Cluster class is the entry point for clients to connect to the Cassandra cluster: Cluster cluster = Cluster.builder().addContactPoint("127.0.0.1").build(); After you are done with using it (usually when application shuts down), close it gracefully: cluster.shutdown(); Creating a session : An object of com.datastax.driver.core.Session allows you to execute a CQL 3 statement. The following line creates a Session instance: Session session = cluster.connect(); Creating a schema : Before reading/writing data, let's create a keyspace and a table similar to UserProfiles in the facebook application we built earlier: // Create Keyspace session.execute("CREATE KEYSPACE facebook WITH replication = " + "{'class':'SimpleStrategy','replication_factor':1};"); session.execute("USE facebook"); // Create table session.execute("CREATE TABLE UserProfiles(" + "email_id text," + "password text,"+ "name text," + "age int," + "profile_picture blob," + "PRIMARY KEY(email_id)" + ");" ); Reading/writing data : We can insert a couple of records as follows: session.execute("USE facebook"); session.execute("INSERT INTO UserProfiles(email_id, password, name, age, profile_picture) VALUES('john.smith@example.com','p4ssw0rd','John Smith',32,0x8e37);"); session.execute("INSERT INTO UserProfiles(email_id, password, name, age, profile_picture) VALUES('david.bergin@example.com','guess1t','David Bergin',42,0xc9f1);"); Finding and printing records : A SELECT query returns an instance of com.datastax.driver.core.ResultSet. You can fetch individual rows by iterating over it using the com.datastax.driver.core.Row object: ResultSet results = session.execute ("SELECT * FROM facebook.UserProfiles " + "WHERE email_id = 'john.smith@example.com';"); for (Row row : results) { System.out.println ("Email: " + row.getString("email_id") + "tName: " + row.getString("name")+ "t Age : " + row.getInt("age")); } Deleting records : We can delete a record as follows: session.execute("DELETE FROM facebook.UserProfiles WHERE email_id='john.smith@example.com';"); Using high-level clients In addition to the libraries based on Thrift and binary protocols, some high-level clients are built with the purpose to ease development and provide additional services, such as connection pooling, load balancing, failover, secondary indexing, and so on. Some of them are listed here: Astyanax (https://github.com/Netflix/astyanax): Astyanax is a high-level Java client for Cassandra. It allows you to run both simple and prepared CQL queries. Hector (https://github.com/hector-client/hector): Hector is a high-level client for Cassandra. At the time of writing this book, it supported CQL 2 only (not CQL 3). Kundera (https://github.com/impetus-opensource/Kundera): Kundera is a JPA 2.0-based object datastore mapping library for Cassandra and many other NoSQL datastores. CQL 3 queries are run with Kundera using the native queries as described in JPA specification. Summary From this article, we basically learn about using CQL in queries using three different preceding methods. Resources for Article : Further resources on this subject: Quick start – Creating your first Java application [Article] Apache Cassandra: Libraries and Applications [Article] Getting Started with Apache Cassandra [Article]
Read more
  • 0
  • 0
  • 2129

article-image-using-events-interceptors-and-logging-services
Packt
03 Oct 2013
19 min read
Save for later

Using Events, Interceptors, and Logging Services

Packt
03 Oct 2013
19 min read
(For more resources related to this topic, see here.) Understanding interceptors Interceptors are defined as part of the EJB 3.1 specification (JSR 318), and are used to intercept Java method invocations and lifecycle events that may occur in Enterprise Java Beans (EJB) or Named Beans from Context Dependency Injection (CDI). The three main components of interceptors are as follows: The Target class: This class will be monitored or watched by the interceptor. The target class can hold the interceptor methods for itself. The Interceptor class: This interceptor class groups interceptor methods. The Interceptor method: This method will be invoked according to the lifecycle events. As an example, a logging interceptor will be developed and integrated into the Store application. Following the hands-on approach of this article, we will see how to apply the main concepts through the given examples without going into a lot of details. Check the Web Resources section to find more documentation about interceptors. Creating a log interceptor A log interceptor is a common requirement in most Java EE projects as it's a simple yet very powerful solution because of its decoupled implementation and easy distribution among other projects if necessary. Here's a diagram that illustrates this solution: Log and LogInterceptor are the core of the log interceptor functionality; the former can be thought of as the interface of the interceptor, it being the annotation that will decorate the elements of SearchManager that must be logged, and the latter carries the actual implementation of our interceptor. The business rule is to simply call a method of class LogService, which will be responsible for creating the log entry. Here's how to implement the log interceptor mechanism: Create a new Java package named com.packt.store.log in the project Store. Create a new enumeration named LogLevel inside this package. This enumeration will be responsible to match the level assigned to the annotation and the logging framework: package com.packt.store.log; public enum LogLevel { // As defined at java.util.logging.Level SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST; public String toString() { return super.toString(); } } We're going to create all objects of this section—LogLevel, Log, LogService, and LogInterceptor—into the same package, com.packt.store.log. This decision makes it easier to extract the logging functionality from the project and build an independent library in the future, if required. Create a new annotation named Log. This annotation will be used to mark every method that must be logged, and it accepts the log level as a parameter according to the LogLevel enumeration created in the previous step: package com.packt.store.log; @Inherited @InterceptorBinding @Retention(RetentionPolicy.RUNTIME) @Target({ElementType.METHOD, ElementType.TYPE}) public @interface Log { @Nonbinding LogLevel value() default LogLevel.FINEST; } As this annotation will be attached to an interceptor, we have to add the @InterceptorBinding decoration here. When creating the interceptor, we will add a reference that points back to the Log annotation, creating the necessary relationship between them. Also, we can attach an annotation virtually to any Java element. This is dictated by the @Target decoration, where we can set any combination of the ElementType values such as ANNOTATION_TYPE, CONSTRUCTOR, FIELD, LOCAL_VARIABLE, METHOD, PACKAGE, PARAMETER, and TYPE (mapping classes, interfaces, and enums), each representing a specific element. The annotation being created can be attached to methods and classes or interface definitions. Now we must create a new stateless session bean named LogService that is going to execute the actual logging: @Stateless public class LogService { // Receives the class name decorated with @Log public void log(final String clazz, final LogLevel level, final String message) { // Logger from package java.util.logging Logger log = Logger.getLogger(clazz); log.log(Level.parse(level.toString()), message); } } Create a new class, LogInterceptor, to trap calls from classes or methods decorated with @Log and invoke the LogService class we just created—the main method must be marked with @AroundInvoke—and it is mandatory that it receives an InvocationContext instance and returns an Object element: @Log @Interceptor public class LogInterceptor implements Serializable { private static final long serialVersionUID = 1L; @Inject LogService logger; @AroundInvoke public Object logMethod(InvocationContext ic) throws Exception { final Method method = ic.getMethod(); // check if annotation is on class or method LogLevel logLevel = method.getAnnotation(Log.class)!= null ? method.getAnnotation(Log.class).value() : method.getDeclaringClass().getAnnotation(Log.class).value(); // invoke LogService logger.log(ic.getClass().getCanonicalName(),logLevel, method.toString()); return ic.proceed(); } } As we defined earlier, the Log annotation can be attached to methods and classes or interfaces by its @Target decoration; we need to discover which one raised the interceptor to retrieve the correct LogLevel value. When trying to get the annotation from the class shown in the method.getDeclaringClass().getAnnotation(Log.class) line, the engine will traverse through the class' hierarchy searching for the annotation, up to the Object class if necessary. This happens because we marked the Log annotation with @Inherited. Remember that this behavior only applies to the class's inheritance, not interfaces. Finally, as we marked the value attribute of the Log annotation as @Nonbinding in step 3, all log levels will be handled by the same LogInterceptor function. If you remove the @Nonbinding line, the interceptor should be further qualified to handle a specific log level, for example @Log(LogLevel.INFO), so you would need to code several interceptors, one for each existing log level. Modify the beans.xml (under /WEB-INF/) file to tell the container that our class must be loaded as an interceptor—currently, the file is empty, so add all the following lines: <beans xsi_schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/beans_1_0.xsd"> <interceptors> <class>com.packt.store.log.LogInterceptor</class> </interceptors> </beans> Now decorate a business class or method with @Log in order to test what we've done. For example, apply it to the getTheaters() method in SearchManager from the project Store. Remember that it will be called every time you refresh the query page: @Log(LogLevel.INFO) public List<Theater> getTheaters() { ... } Make sure you have no errors in the project and deploy it to the current server by right-clicking on the server name and then clicking on the Publish entry. Access the theater's page, http://localhost:7001/theater/theaters.jsf, refresh it a couple of times, and check the server output. If you have started your server from Eclipse, it should be under the Console tab: Nov 12, 2012 4:53:13 PM com.packt.store.log.LogService log INFO: public java.util.List com.packt.store.search.SearchManager.getTheaters() Let's take a quick overview of what we've accomplished so far; we created an interceptor and an annotation that will perform all common logging operations for any method or class marked with such an annotation. All log entries generated from the annotation will follow WebLogic's logging services configuration. Interceptors and Aspect Oriented Programming There are some equivalent concepts on these topics, but at the same time, they provide some critical functionalities, and these can make a completely different overall solution. In a sense, interceptors work like an event mechanism, but in reality, it's based on a paradigm called Aspect Oriented Programming (AOP). Although AOP is a huge and complex topic and has several books that cover it in great detail, the examples shown in this article make a quick introduction to an important AOP concept: method interception. Consider AOP as a paradigm that makes it easier to apply crosscutting concerns (such as logging or auditing) as services to one or multiple objects. Of course, it's almost impossible to define the multiple contexts that AOP can help in just one phrase, but for the context of this article and for most real-world scenarios, this is good enough. Using asynchronous methods A basic programming concept called synchronous execution defines the way our code is processed by the computer, that is, line-by-line, one at a time, in a sequential fashion. So, when the main execution flow of a class calls a method, it must wait until its completion so that the next line can be processed. Of course, there are structures capable of processing different portions of a program in parallel, but from an external viewpoint, the execution happens in a sequential way, and that's how we think about it when writing code. When you know that a specific portion of your code is going to take a little while to complete, and there are other things that could be done instead of just sitting and waiting for it, there are a few strategies that you could resort to in order to optimize the code. For example, starting a thread to run things in parallel, or posting a message to a JMS queue and breaking the flow into independent units are two possible solutions. If your code is running on an application server, you should know by now that thread spawning is a bad practice—only the server itself must create threads, so this solution doesn't apply to this specific scenario. Another way to deal with such a requirement when using Java EE 6 is to create one or more asynchronous methods inside a stateless session bean by annotating either the whole class or specific methods with javax.ejb.Asynchronous. If the class is decorated with @Asynchronous, all its methods inherit the behavior. When a method marked as asynchronous is called, the server usually spawns a thread to execute the called method—there are cases where the same thread can be used, for instance, if the calling method happens to end right after emitting the command to run the asynchronous method. Either way, the general idea is that things are explicitly going to be processed in parallel, which is a departure from the synchronous execution paradigm. To see how it works, let's change the LogService method to be an asynchronous one; all we need to do is decorate the class or the method with @Asynchronous: @Stateless @Asynchronous public class LogService { … As the call to its log method is the last step executed by the interceptor, and its processing is really quick, there is no real benefit in doing so. To make things more interesting, let's force a longer execution cycle by inserting a sleep method into the method of LogService: public void log(final String clazz,final LogLevel level,final String message) { Logger log = Logger.getLogger(clazz); log.log(Level.parse(level.toString()), message); try { Thread.sleep(5000); log.log(Level.parse(level.toString()), "reached end of method"); } catch (InterruptedException e) { e.printStackTrace(); } } Using Thread.sleep() when running inside an application server is another classic example of a bad practice, so keep away from this when creating real-world solutions. Save all files, publish the Store project, and load the query page a couple of times. You will notice that the page is rendered without delay, as usual, and that the reached end of method message is displayed after a few seconds in the Console view. This is a pretty subtle scenario, so you can make it harsher by commenting out the @Asynchronous line and deploying the project again—this time when you refresh the browser, you will have to wait for 5 seconds before the page gets rendered. Our example didn't need a return value from the asynchronous method, making it pretty simple to implement. If you need to get a value back from such methods, you must declare it using the java.util.concurrent.Future interface: @Asynchronous public Future<String> doSomething() { … } The returned value must be changed to reflect the following: return new AsyncResult<String>("ok"); The javax.ejb.AsyncResult function is an implementation of the Future interface that can be used to return asynchronous results. There are other features and considerations around asynchronous methods, such as ways to cancel a request being executed and to check if the asynchronous processing has finished, so the resulting value can be accessed. For more details, check the Creating Asynchronous methods in EJB 3.1 reference at the end of this article. Understanding WebLogic's logging service Before we advance to the event system introduced in Java EE 6, let's take a look at the logging services provided by Oracle WebLogic Server. By default, WebLogic Server creates two log files for each managed server: access.log: This is a standard HTTP access log, where requests to web resources of a specific server instance are registered with details such as HTTP return code, the resource path, response time, among others <ServerName.log>: This contains the log messages generated by the WebLogic services and deployed applications of that specific server instance These files are generated in a default directory structure that follows the pattern $DOMAIN_NAME/servers/<SERVER_NAME>/logs/. If you are running a WebLogic domain that spawns over more than one machine, you will find another log file named <DomainName>.log in the machine where the administration server is running. This file aggregates messages from all managed servers of that specific domain, creating a single point of observation for the whole domain. As a best practice, only messages with a higher level should be transferred to the domain log, avoiding overhead to access this file. Keep in mind that the messages written to the domain log are also found at the managed server's specific log file that generated them, so there's no need to redirect everything to the domain log. Anatomy of a log message Here's a typical entry of a log file: ####<Jul 15, 2013 8:32:54 PM BRT> <Alert> <WebLogicServer> <sandbox-lap> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1373931174624> <BEA-000396> <Server shutdown has been requested by weblogic.> The description of each field is given in the following table: Text Description #### Fixed, every log message starts with this sequence <Jul 15, 2013 8:32:54 PM BRT> Locale-formatted timestamp <Alert> Message severity <WebLogicServer> WebLogic subsystem-other examples are WorkManager, Security, EJB, and Management <sandbox-lap> Physical machine name <AdminServer> WebLogic Server name <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> Thread ID <weblogic> User ID <>  Transaction ID, or empty if not in a transaction context <>  Diagnostic context ID, or empty if not applicable; it is used by the Diagnostics Framework to correlate messages of a specific request <1373931174624> Raw time in milliseconds <BEA-000396> Message ID <Server shutdown has been requested by weblogic.> Description of the event The Diagnostics Framework presents functionalities to monitor, collect, and analyze data from several components of WebLogic Server. Redirecting standard output to a log file The logging solution we've just created is currently using the Java SE logging engine—we can see our messages on the console's screen, but they aren't being written to any log file managed by WebLogic Server. It is this way because of the default configuration of Java SE, as we can see from the following snippet, taken from the logging.properties file used to run the server: # "handlers" specifies a comma separated list of log Handler # classes. These handlers will be installed during VM startup. # Note that these classes must be on the system classpath. # By default we only configure a ConsoleHandler, which will only # show messages at the INFO and above levels. handlers= java.util.logging.ConsoleHandler You can find this file at $JAVA_HOME/jre/lib/logging.properties. So, as stated here, the default output destination used by Java SE is the console. There are a few ways to change this aspect: If you're using this Java SE installation solely to run WebLogic Server instances, you may go ahead and change this file, adding a specific WebLogic handler to the handlers line as follows: handlers= java.util.logging.ConsoleHandler,weblogic.logging.ServerLoggingHandler Tampering with Java SE files is not an option (it may be shared among other software, for instance); you can duplicate the default logging.properties file into another folder $DOMAIN_HOME being a suitable candidate, add the new handler, and instruct WebLogic to use this file at startup by adding this argument to the following command line: -Djava.util.logging.config.file=$DOMAIN_HOME/logging.properties You can use the administration console to set the redirection of the standard output (and error) to the log files. To do so, perform the following steps: In the left-hand side panel, expand Environment and select Servers. In the Servers table, click on the name of the server instance you want to configure. Select Logging and then General. Find the Advanced section, expand it, and tick the Redirect stdout logging enabled checkbox: Click on Save to apply your changes. If necessary, the console will show a message stating that the server must be restarted to acquire the new configuration. If you get no warnings asking to restart the server, then the configuration is already in use. This means that both WebLogic subsystems and any application deployed to that server is automatically using the new values, which is a very powerful feature for troubleshooting applications without intrusive actions such as modifying the application itself—just change the log level to start capturing more detailed messages! Notice that there are a lot of other logging parameters that can be configured, and three of them are worth mentioning here: The Rotation group (found in the inner General tab): The rotation feature instructs WebLogic to create new log files based on the rules set on this group of parameters. It can be set to check for a size limit or create new files from time to time. By doing so, the server creates smaller files that we can easily handle. We can also limit the number of files retained in the machine to reduce the disk usage. If the partition where the log files are being written to reaches 100 percent of utilization, WebLogic Server will start behaving erratically. Always remember to check the disk usage; if possible, set up a monitoring solution such as Nagios to keep track of this and alert you when a critical level is reached. Minimum severity to log (also in the inner General tab): This entry sets the lower severity that should be logged by all destinations. This means that even if you set the domain level to debug, the messages will be actually written to the domain log only if this parameter is set to the same or lower level. It will work as a gatekeeper to avoid an overload of messages being sent to the loggers. HTTP access log enabled (found in the inner HTTP tab): When WebLogic Server is configured in a clustered environment, usually a load-balancing solution is set up to distribute requests between the WebLogic managed servers; the most common options are Oracle HTTP Server (OHS) or Apache Web Server. Both are standard web servers, and as such, they already register the requests sent to WebLogic in their own access logs. If this is the case, disable the WebLogic HTTP access log generation, saving processing power and I/O requests to more important tasks. Integrating Log4J to WebLogic's logging services If you already have an application that uses Log4J and want it to write messages to WebLogic's log files, you must add a new weblogic.logging.log4j.ServerLoggingAppender appender to your lo4j.properties configuration file. This class works like a bridge between Log4J and WebLogic's logging framework, allowing the messages captured by the appender to be written to the server log files. As WebLogic doesn't package a Log4J implementation, you must add its JAR to the domain by copying it to $DOMAIN_HOME/tickets/lib, along with another file, wllog4j.jar, which contains the WebLogic appender. This file can be found inside $MW_HOME/wlserver/server/lib. Restart the server, and it's done! If you're using a *nix system, you can create a symbolic link instead of copying the files—this is great to keep it consistent when a path changing these specific files must be applied to the server. Remember that having a file inside $MW_HOME/wlserver/server/lib doesn't mean that the file is being loaded by the server when it starts up; it is just a central place to hold the libraries. To be loaded by a server, a library must be added to the classpath parameter of that server, or you can add it to the domain-wide lib folder, which guarantees that it will be available to all nodes of the domain on a specific machine. Accessing and reading log files If you have direct access to the server files, you can open and search them using a command-line tool such as tail or less, or even use a graphical viewer such as Notepad. But when you don't have direct access to them, you may use WebLogic's administration console to read their content by following the steps given here: In the left-hand side pane of the administration console, expand Diagnostics and select Log Files. In the Log Files table, select the option button next to the name of the log you want to check and click on View: The types displayed on this screen, which are mentioned at the start of the section, are Domain Log, Server Log, and HTTP Access. The others are resource-specific or linked to the diagnostics framework. Check the Web resources section at the end of this article for further reference. The page displays the latest contents of the log file; the default setting shows up to 500 messages in reverse chronological order. The messages at the top of the window are the most recent messages that the server has generated. Keep in mind that the log viewer does not display messages that have been converted into archived log files.
Read more
  • 0
  • 0
  • 7331

article-image-getting-familiar-story-features
Packt
03 Oct 2013
11 min read
Save for later

Getting Familiar with the Story Features

Packt
03 Oct 2013
11 min read
(For more resources related to this topic, see here.) Manga Studio 5 is a great all-around drawing software. But it's primarily a comic creation software, and as such the Multiple Pages feature is its bread and butter. The Page Manager tab functions like a book, allowing you to see zoomed-out versions of all the pages in your comic at once, open them individually, drag-and-drop them to new locations, and add or delete pages from your story at will. It's the thumbnail view of your story—the digital equivalent of having your pages laid out on index cards and shuffling them around as you please. It's a CMC file that points to individual image files of your pages in one convenient location so that you no longer have to keep track of separate files, like in other graphics programs. Creating a new file In order to make a new story, navigate to File | New (or press Ctrl + N), or click on the new icon on your toolbar. Once you've done that, the New dialog box will open. Make sure that the Manga draft settings(O) and Multiple pages checkboxes are selected. As you can see, we have quite a lot of options to play with here, so let's go over them. The Preset drop-down menu has common paper sizes used in the production of comics. It also has some other sizes, such as postcard and common web resolution sizes. What size and dpi setting you use is up to you and the specifications of your book printer. My rule is to always plan to print, even if you're just thinking of doing a webcomic and have no ambitions to do so when you first start off. You never know when you might have a huge hit on your hands and suddenly get thousands of people screaming for you to make books. Or you might just decide that you really want to see your work in print! Set up correctly from the start and you'll save yourself a lot of work—and headaches later. Print resolution is at least 300 dpi. The higher you can go on the dpi though, the crisper your work will look when it's printed. Most of the preset sizes in Manga Studio are already set to 350 dpi. If your computer can handle going higher, to about 600 dpi or so, then it's suggested that you do so, especially if you will be printing your work in black and white. High resolution is great for black-and-white work and produces nice, crisp lines. Creating and saving a custom page template The standard size for a newspaper strip is 4 1/16 by 13 inches (34.29 cm x 11.43 cm). We're making the width and height slightly larger than our finished size because we want our active drawing area to be 4 1/16 by 13 inches, and we need to account for the binding size and default border. Want to have a long strip down instead of across? Or set a regular page size to landscape format instead of portrait? Simply click on the arrows to the left of the Width and Height fields to flip them! The Binding (finish) size is the final size of our page once the bleed is accounted for. A bleed in printing is when the image goes off the page. In order to achieve this bleed, you must produce your artwork at a size larger than the final printed size, and draw to the edge of the canvas. When your work is printed, the excess at the edge will then be cut off, so don't put anything important near the bleed area! Most printers require a bleed of at least one-eighth of an inch all the way around. So if you're going to be printing your comics in a 6 by 9 inch format, you will want to set your page size to 6.25 by 9.25 inches to compensate for the bleed. If you already have a printer in mind, be sure to get their printing specifications before you start drawing so that you can set your pages up correctly. It will save you time and a lot of headache in the end! The Default border (inner) is the interior margin of the page. For most printers you should leave at least a quarter of an inch all the way around the edge inside which the important elements (such as text) stay, so that they're not cut off in the printing process. The X offset and Y offset options for the basic frame will move the guides horizontally when the X offset value is adjusted and vertically for the Y offset value. Handy if you need a larger margin in the center of pages to account for a book's gutter, or if you wish to have a larger margin at the bottom or top of a page—say for text at the bottom or top with the comic title and author information. Since we wanted our finished paper size to be 13 by 4 1/16 inches, under the Binding (finish) size field we're going to enter 13 and 4.06 inches (33.02 cm x 10.31 cm). The bleed will be an eighth of an inch on each side, so 16 should be close enough. Now, to achieve a bit of a gutter between the finished edges of our pages and the sides of our panels, set the basic frame to 12.5 by 3.75 inches (31.75 by 9.53 cm) and we have the dimensions of our drawing all set up. There are still a few more options on this screen to address though. The Basic expression color option allows you to set the color mode for your entire file. It defaults to Color, but you can also select Monochrome and Gray from the drop-down menu. If you are creating a comic that will be in color, leave the setting on Color. For grayscale comics, use the Gray option. If you are creating a pure black-and-white comic, with no shades of gray included, then select the Monochrome option. The Paper color option allows you to change the color of your base canvas. Since this can also be changed any time during the drawing process, I usually leave this white when I'm doing basic sketching, and then change it when I'm coloring. The Template checkbox allows you to pick a frame template material from the Materials library to automatically place in your new file. There are many pre-set templates to choose from in Manga Studio 5. Checking this box will bring up the Template dialog box, where you can scroll through the options or do keywords search. Up at the top of our dialog box, underneath the OK and Cancel buttons, there is a button for Register to preset. This will allow us to save the dimensions that we have just created and use them later on in other files. Click on the Register to preset button and then enter a descriptive name. We'll call our current settings Comic Strip—easy to remember and to the point. You can choose what options will be saved in your preset. Resolution is automatically checked. You can also save the template, if you are using one. This is handy if you are creating a strip, or something that usually has the same number and layout of panels. Page settings will save anything under the Multiple pages options, which is very handy if you have a comic that is always the same number of pages per chapter or storyline. And of course the Default expression color option can also be saved, so that your saved preset will always be whichever option you selected. Now, click on OK in the dialog box and our Comic Strip template will be saved in to the Preset drop-down menu for us to use at any time. At the bottom of our New dialog box are the Multiple pages options. This allows us to create many page files all at once and be able to organize them. The Number of pages dropdown gives some common number of pages, but you can also enter a custom number simply by clicking in the box and typing a number. The Spread corresponding page(M) option makes it possible to create two-page spreads with our files. Binding point is the option for what side of the finished book the spine will be located on. I'm an English speaker, so I click on the Left binding option. If your book will be bound with the spine on the right, click on Right binding. The Start page indicates which side the first page will be on. In books that are bound on the left, this is usually the right-hand page (because the back of the cover is usually blank, or filled with an advertisement in the case of mainstream comics publishing in the U.S.). The Save folder is the folder on your hard drive where this file will be stored. The Reference... button beneath it allows you to change the folder by browsing your hard drive. Click on Reference… and choose where you want to store your file. Then the Management folder text is the new folder that Manga Studio will create to store the image files. This can be a storyline name, an issue number, anything that you desire, so long as you know what it is. Let's name our Management folder Chapter One for these exercises. The Management file text under the Management folder name changes as we change the folder name, to show us how our file is going to be set up on our hard drive. Now, click on the OK button to create your new file. Navigating and rearranging pages Once you click on OK and Manga Studio creates the pages, the Page Manager window will open. This window is the command center for our story file. We can see thumbnails of our pages, zoom in and out on them, and open pages to work on. As you can see, in this view we can see thumbnails of all of our files in this story. This is great for a number of things, including being able to check the flow of your story. I often catch myself putting too many splash pages in a chapter of my comic, and I see that when I look over the pages in story mode. We can also rearrange pages in this tab. To rearrange the pages, simply left-click on a page and drag-and-drop it in its new position. A red line will appear where the page will be dropped so that you can be sure you have it in the spot where you want it. When you let go of the page in its new spot in the story, Manga Studio will need a moment to update all of the page information. Once that's done, you can continue working on your project. At the bottom of your Manga Studio window, there are several options for zooming in and out of your page thumbnails. Three of these are the same options that are in your bottom toolbar when working on a canvas. The first option is a slider that allows you to zoom in and out by dragging the light gray rectangle to the left or right. Click on it and drag to the right to zoom in. Dragging to the left will zoom back out on the thumbnails. If you're having trouble telling what pages in your story are what, you can zoom in to see them better, and then zoom out again to get a look at the visual flow of the entire chapter or a section of it all at once. The next two icons in the lower bar are also for zooming in and out, but they do so by pre-set percentages. The right-most icon is the Fitting control, which can be toggled back and forth between the pages being laid out horizontally or vertically. Click on it and the pages will rearrange so that they're two across on the screen and fit inside the Manga Studio workspace. Alright, now you know how to zoom in and out of your page thumbnails, so we can explore some of the other things you can do in the story mode of Manga Studio.
Read more
  • 0
  • 0
  • 2195

article-image-ninja-topics
Packt
03 Oct 2013
16 min read
Save for later

Ninja Topics

Packt
03 Oct 2013
16 min read
(For more resources related to this topic, see here.) Using Capybara outside of Cucumber Capybara is by no means coupled to Cucumber, and can be used in any setting you wish, within Test::Unit, RSpec, or just from vanilla Ruby code. In fact, if you are using Cucumber but want to abstract some logic out of your step definitions and into Page Objects, then you will still need to consider how to use Capybara outside of Cucumber's world (https://github.com/cucumber/cucumber/wiki/A-Whole-New-World). Including the modules The first option you have for using Capybara outside of Cucumber is to include the DSL modules into your own modules or classes: require 'capybara/dsl' require 'rspec/expectations' Capybara.default_driver = :selenium module MyModule include Capybara::DSL include RSpec::Matchers def play_song visit 'http://localhost/html/html5.html' click_on 'Play' find('#log').should have_text 'playing' end end class Runner include MyModule def run play_song end end Runner.new.run It is important to require the Capybara DSL file, as this contains all the Capybara methods that need to be "mixed in". In this example, we have our own Ruby module and crucially within this, the relevant Capybara module Capybara::DSL is included. In addition, the RSpec::Matchers module has also been included, which allows us to utilize standard RSpec Matchers (obviously you do not have to use RSpec; you could choose a different way to assert behavior). If you follow this pattern in your own code, you can now mix in any of the standard Capybara methods into your own module or class methods. It is worth remembering that if you include modules in a base class, then all subclasses will inherit the ability to use those module methods. This would come in handy, for example, if you were using a Page Object pattern, where the only class that would have to include the DSL module would be the base page. Using the session directly The other option for mixing Capybara into your code is to use the session directly, that is to say, you instantiate a new instance of the session object and then call the DSL methods on it. The following example implements the same test as before, but this time by using a session instance and raising a simple exception if the expected content is not found: require 'capybara' session = Capybara::Session.new :selenium session.visit('http://localhost/html/html5.html') session.click_on 'Play' raise 'song not playing' unless session.find('#log') == 'playing' If you are using an object-oriented model for building your tests, you will need to pass the session instance around or find an appropriate strategy to deal with this, as you do not have the benefit of the modules mixing in the DSL methods globally. Capybara and popular test frameworks Capybara provides out of the box integration with a number of popular test frameworks; this is not a subject we will cover in depth, simply because they are covered very well in the Capybara README, which can be found at https://github.com/jniklas/capybara. Cucumber Throughout this article, examples have been set in the context of Cucumber, so you should be happy with how to implement Capybara's API in step definitions and do some simple setup in the env.rb file, such as setting the default driver. There are a couple of additional pieces of functionality that Capybara adds when using in conjunction with Cucumber, which are worth examining. The first is that Capybara hooks into Cucumber's Before do block as follows: Before do Capybara.reset_sessions! Capybara.use_default_driver end Apart from setting the default driver, this crucially makes a call to reset_sessions!, and this in turn will invoke some code in the underlying driver. In the case of Selenium, this deletes all cookies to ensure that you start each scenario with no pollution from the previous one. For Rack::Test, this will destroy the browser instance, so a new one gets created each time; for any other drivers, you will need to check the implementation of the reset! method to see what they do. Finally, Capybara also hooks into any Cucumber scenarios you have tagged with @javascript and automatically switches you to the driver you have set up to handle JavaScript. For example, you may use Rack::Test as your default driver, but have set the following in your env.rb file: Capybara.javascript_dirver = :selenium In this case, any scenarios tagged with @javascript will result in Capybara starting a browser using Selenium as the driver. RSpec Outside of Cucumber, RSpec is one of the most popular Ruby test frameworks, lending itself well to unit, integration, and acceptance tests, and used inside and outside of Rails. Capybara adds the following features to RSpec: Lets you mix in the DSL to your specs by adding require 'capybara/rspec' into your spec_helper.rb file Lets you use :js => true to invoke the JavaScript driver Adds a DSL for writing descriptive "feature style" acceptance tests using RSpec For more details on these features, check out the README, which is available at https://github.com/jniklas/capybara. Test::Unit If you are using Ruby's basic unit test library outside of Rails, then using Capybara simply means mixing in the DSL module via include Capybara::DSL, as you saw earlier. As noted in the README, it makes a lot of sense to reset the browser session in your teardown method, for example: def teardown Capybara.reset_sessions! Capybara.use_default_driver end If you are using Rails, then there will be other considerations, such as turning off transactional fixtures, as these will not work with Selenium; again, the Capybara README details this behavior fully. MiniTest::Spec MiniTest is a new unit test framework introduced in Ruby 1.9, which also has the ability to support BDD style tests. Capybara does not have built-in support for MiniTest, because MiniTest does not use RSpec but rather uses its own matchers. There is another gem named capybara_minitest_spec (https://github.com/ordinaryzelig/capybara_minitest_spec), which adds support for these matchers to Capybara. Advanced interactions and accessing the driver directly Although we have covered a great deal of Capybara's API, there are still a few interactions that we have not addressed, for example, hovering over an element or dragging elements around. Capybara does provide support for a lot of these more advanced interactions; for example, a recent addition (Capybara 2.1) to methods you can call on an element is hover: find('#box1').hover In Selenium, this results in a call to the mouse.move_to method and works for both elements using CSS's hover property or JavaScript's mouseenter / mouseleave methods. Other drivers may implement this differently, and obviously in some it may not be supported at all, either where JavaScript support is non-existent (Rack::Test) or rudimentary (Celerity). You can also emulate drag-and-drop using the following line of code: find('#mydiv').drag_to '#droplocation' Again, driver support is likely to be patchy, but of course this will work fine in Selenium WebDriver. Despite all the bells and whistles offered by Capybara, there may still be occasions where you need to access the API that exists in the underlying driver, but that has not been mapped in Capybara. In this instance, you have two options: Call Capybara's native method on any Caybara::Element, and then call the driver method Use page.driver.browser.manage to call the driver methods that are not called on elements Using the native method If you wish to call a method in the underlying driver, and that method is one that is called on a retrieved DOM element, then you can use the native method. A good example is retrieving a computed CSS style value. Elements in the DOM can obtain CSS properties in a couple of ways; firstly, there is the inline style: <p style="font-weight:bold;">Bold Paragraph Text</p> This information could be easily retrieved by accessing the style attribute via the methods. The other way in which an element obtains CSS properties is via style elements or stylesheets that are referenced via link tags in the page. When the browser loads the stylesheets and applies styles to the specified DOM elements, these are known as computed styles. Capybara has no direct API for retrieving the computed style of an element, which was most likely a deliberate design decision as only a few drivers would ever support this. However, Selenium WebDriver does have this capability, and it is possible that you would want to access this information. Consider the following code, where we apply a CSS hover property to a div element, so that when the user hovers over the element, it changes color. <html> <head> <title>Hover Examples</title> <style> .box { height: 200px; width: 200px; margin: 10px; background-color: blue; } .box:hover { background-color: green; } </style> </head> <body> <div id="main"> <div id="box1" class="box"> </div> </div> </body> </html> The Cucumber step definitions that follow use Capybara and Selenium WebDriver to assert that the color has changed: When(/^I hover over an element whose color changes on hover using CSS$/) do visit 'http://capybara.local/html/chapter5/hover.html' find('#box1').hover end Then(/^I see the color change$/) do find('#box1').native.style('background-color').should == 'rgba(0, 128, 0, 1)' end Here, the style method within Selenium WebDriver is accessed via Capybara's native method, and the value can then be validated. Accessing driver methods using browser.manage The other use case is when we wish to access functionality provided in the underlying driver that is neither mapped by Capybara nor related to a specific element on the page. A good example of this is accessing cookie information. There has been a long running debate on the Capybara forums and GitHub issue tracker, about whether Capybara should expose an API for cookie getters and setters, but the overriding feeling has always been that this should not be exposed (arguably, you should not set cookies in your tests, as this is changing the state of the application as a user never would). Nevertheless, it is something you may well need to do, and given that Selenium WebDriver and a number of other drivers support this functionality, you will need to access the driver methods directly. Consider this page that sets a cookie using JavaScript: <html> <head> <title>Cookie Examples</title> <script> document.cookie = 'mycookie=foobar'; </script> </head> <body></body> </html> The following steps simply visits the page and then outputs to the console all the cookies that are available on the current page: When(/^I visit a page that sets a Cookie$/) do visit 'http://localhost/html/cookie.html' end Then(/^I can access the cookie using Selenium$/) do puts page.driver.browser.manage.all_cookies end We can access any method in the underlying driver by using page.driver.browser.manage, and then the method we wish to call; in this instance, we call Selenium WebDriver's all_cookies method, and the output will be as follows: [{:name=>"mycookie", :value=>"foobar", :path=>"/html/chapter5", :domain=>"localhost", :expires=>nil, :secure=>false}] Selenium WebDriver exposes a full API for accessing and setting cookies. The documentation can be found at http://rubydoc.info/gems/selenium-webdriver/0.0.28/Selenium/WebDriver/Driver. Advanced driver configuration So far, we have only set the default driver or the JavaScript driver using a symbol: Capybara.default_driver = :selenium It is quite likely that you will need to fine-tune the configuration of your driver or register multiple configurations, which you can select from at run time. An example of this might be that you are running tests from your office, and the corporate network sits behind an HTTP Proxy (the bane of a tester's life). If you are using Selenium WebDriver with Firefox, you could register a custom driver configuration in Capybara as follows: Capybara.register_driver :selenium_proxy do |app| profile = Selenium::WebDriver::Firefox::Profile.new profile["network.proxy.type"] = 1 profile["network.proxy.no_proxies_on"] = "capybara.local" profile["network.proxy.http"] = "cache-mycompany.com" profile["network.proxy.ssl"] = 'securecache-mycompany.com' profile["network.proxy.http_port"] = 9999 profile["network.proxy.ssl_port"] = 9999 profile.native_events = true Capybara::Selenium::Driver.new(app, :browser => :firefox, :profile => profile) end Capybara.default_driver = :selenium_proxy This configuration uses the Selenium WebDriver API to construct a custom Firefox profile, set the proxy details programmatically, register the driver with the name :selenium-proxy, and then make it the default driver. Obviously the possibilities here are endless, and in browsers such as Firefox and Chrome, the amount of options you can customize runs into hundreds, so knowing how to set these is important. For example, you could use this technique to create profiles with JavaScript disabled or Cookies disabled to ensure your application behaves correctly in these cases. The driver ecosystem Capybara bundles two drivers, which as you know are Rack::Test and Selenium WebDriver. However, Capybara is architected in such a way to make it easy for developers to implement other drivers, and indeed, there is a healthy ecosystem of pluggable drivers, which offer interesting alternatives to the two built-in options. Capybara-WebKit Pretty much every developer I know who is passionate about test automation is desperate for one thing—a headless browser with great JavaScript support. A headless browser is one that runs without a UI. Headless browsers typically run tests faster than opening a real browser, and make it easy to run in Continuous Integration (CI) environments, where often, a windowing environment (either MS Windows or X11 on Linux) may not be available. You can refer to the following links to find out more about Capybara-WebKit: https://github.com/thoughtbot/capybara-webkit http://qt.digia.com/ Capybara-WebKit is a driver that wraps QtWebKit and is maintained by the guys at Thoughtbot. The Qt project provides a cross-platform framework for building native GUI applications, and as part of this, it provides a WebKit browser implementation that lends itself to headless implementations. There is a slight catch with this particular implantation of QtWebKit; you will need to install the Qt system libraries separately, and there is still a reliance on X11 being present on Linux distributions despite the browser being headless. Poltergeist Poltergeist is another driver, which, in the background, will use QtWebKit. The difference here is that it wraps PhantomJS, which brings a couple of potential advantages over Capybara-WebKit. You can refer to the following links to find out more about Poltergeist: https://github.com/jonleighton/poltergeist http://phantomjs.org/ You will still need to install PhantomJS as a system dependency, but the PhantomJS project has tackled some of the issues around making QtWebKit purely headless, so you will not need X11, and you will not need to install the entire Qt framework, because PhantomJS bundles this for you. Capybara-Mechanize Mechanize is a headless browser implemented purely in Ruby, and has long been a stalwart for the Ruby community; it uses Nokogiri at its core to provide DOM access, and then builds browser capabilities around this. You can refer to the following links to find out more about Capybara-Mechanize: https://github.com/jeroenvandijk/capybara-mechanize https://github.com/sparklemotion/mechanize Capybara-Mechanize is a driver that allows you to use Mechanize to run your tests. It is a very powerful option, but there are some important aspects to consider, such as: No JavaScript support: Mechanize does not contain a JavaScript engine; therefore, none of the JavaScript on any of your pages will be run No rendering engine: Mechanize does not contain a rendering engine; therefore, no computed styles will be evaluated Fast: Because it's not attempting to evaluate JavaScript and do graphical rendering, it will be super fast The benefits of using Mechanize may not be obvious at first sight, and a lot will depend on how your site is implemented. For sites whose functionality is wholly dependent on JavaScript, this option is clearly not feasible; however, if your site follows the principles of "progressive enhancement", where JavaScript is simply used to enrich the user's experience, Mechanize is a great option. You can implement all the core functional tests using Mechanize, which will be ensure they run with minimal latency and will be far less fragile than using Selenium, and then just implement a sprinkling of Selenium or WebKit tests to test the JavaScript dependent features. Capybara-Celerity The final driver that is worth considering is Capybara-Celerity, which wraps the Ruby library Celerity. This is especially worth a look if you are running your test code on JRuby (Ruby running on the JVM) as Celerity itself wraps the Java library HtmlUnit. You can refer to the following links to find out more about Capybara-Celerity: https://github.com/sobrinho/capybara-celerity https://github.com/jarib/celerity http://htmlunit.sourceforge.net/ Going back a few years, HtmlUnit would have been a serious consideration for anybody wanting a headless browser, indeed it was the default headless browser for the Selenium WebDriver project. HtmlUnit is a pure Java implementation of a browser and uses Rhino (https://developer.mozilla.org/en/docs/Rhino) to run JavaScript. Unfortunately, you may find that the JavaScript support still fails when attempting to evaluate heavy-weight JS libraries, as the project is not actively maintained, and keeping pace with the demands of modern browsers is unrealistic for a small project. If you are looking for a headless alternative to Mechanize, Celerity is worth a look, but don't rely on it for JavaScript support. Summary This article has taken you from confidently implementing Capybara tests against your application to being a Capybara ninja. By understanding how to use Capybara outside the comfort zone of Cucumber, you can now use it in almost any setting and even write your own custom framework. This is important even if you use Cucumber, because as your Cucumber tests grow, you will probably want to implement some Page Objects and mix Capybara::DSL into these. Another important aspect of writing effective tests is being able to configure the underlying driver and access it directly when required; there is nothing magical about this, and it means you can harness the full power of your chosen driver. Finally, we introduced some alternative drivers that should whet your appetite and prove why Capybara is such a powerful framework. You write your tests once and then run them in multitude of compatible drivers. Win! Resources for Article: Further resources on this subject: First Steps with Selenium RC [Article] Behavior-driven Development with Selenium WebDriver [Article] Setting up environment for Cucumber BDD Rails [Article]
Read more
  • 0
  • 0
  • 1548
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at €18.99/month. Cancel anytime
article-image-using-sprites-animation
Packt
03 Oct 2013
6 min read
Save for later

Using Sprites for Animation

Packt
03 Oct 2013
6 min read
(For more resources related to this topic, see here.) Sprites Let's briefly discuss sprites. In gaming, sprites are usually used for animation sequences; a sprite is a single image in which individual frames of a character animation are stored. We are going use sprites in our animations. If you already have knowledge of graphics design, it's good for you because it is an edge for you to define how you want your game to look like and how you want to define animation sequences in sprites. You can try out tools such as Sprite Maker for making your own sprites with ease; you can get a copy of Sprite Maker at http://www.spriteland.com/sprites/sprite-maker.zip. The following is an sample animation sprite by Marc Russell, which is available for free at http://opengameart.org/content/gfxlib-fuzed you can find other open source sprites at http://opengameart.org/content/platformersidescroller-tiles: The preceding sprite will play the animation of the character moving to the right. The character sequence is well organized using an invisible grid, as shown in the following screenshot: The grid is 32 x 32; the size of our grid is very important in setting up the quads for our game. A quad in LÖVE is a specific part of an image. Because our sprite is a single image file, quads will be used to specify each of the sequences we want to draw per unit time and will be the largest part part of our animation algorithm. Animation The animation algorithm will simply play the sprite like a tape of film; we'll be using a basic technique here as LÖVE doesn't have an official module for that. Some members of the LÖVE forum have come up with different libraries to ease the way we play animations. First of all let us load our file: function love.load() sprite = love.graphics.newImage "sprite.png" end Then we create quads for each part of the sprite by using love.graphics. newQuad(x, y, width, height, sw, sh), where x is the top-left position of the quad along the x axis, y is the top-left position of the quad along the y axis, width is the width of the quad, height is the height of the quad, sw is the sprite's width, and sh is the sprite's height: love.graphics.newQuad(0, 0, 32, 32, 256, 32) --- first quad love.graphics.newQuad(32, 0, 32, 32, 256, 32) --- second quad love.graphics.newQuad(64, 0, 32, 32, 256, 32) --- Third quad love.graphics.newQuad(96, 0, 32, 32, 256, 32) --- Fourth quad love.graphics.newQuad(128, 0, 32, 32, 256, 32) --- Fifth quad love.graphics.newQuad(160, 0, 32, 32, 256, 32) --- Sixth quad love.graphics.newQuad(192, 0, 32, 32, 256, 32) --- Seventh quad love.graphics.newQuad(224, 0, 32, 32, 256, 32) --- Eighth quad The preceding code can be rewritten in a more concise loop as shown in the following code snippet: for i=1,8 do love.graphics.newQuad((i-1)*32, 0, 32, 32, 256, 32) end As advised by LÖVE, we shouldn't state our quads in the draw() or update() functions, because it will cause the quad data to be repeatedly loaded into memory with every frame, which is a bad practice. So what we'll do is pretty simple; we'll load our quad parameters in a table, while love.graphics.newQuad will be referenced locally outside the functions. So the new code will look like the following for the animation in the right direction: local Quad = love.graphics.newQuad function love.load() sprite = love.graphics.newImage "sprite.png" quads = {} quads['right'] ={} quads['left'] = {} for j=1,8 do quads['right'][j] = Quad((j-1)*32, 0, 32, 32, 256, 32); quads['left'][j] = Quad((j-1)*32, 0, 32, 32, 256, 32); -- for the character to face the opposite direction, the quad need to be flipped by using the Quad:flip(x, y) method, where x and why are Boolean. quads.left[j]:flip(true, false) --flip horizontally x = true, y = false end end Now that our animation table is set, it is important that we set a Boolean value for the state of our character. At the start of the game our character is idle, so we set idle to true. Also, there are a number of quads the algorithm should read in order to play our animation. In our case, we have eight quads, so we need a maximum of eight iterations, as shown in the following code snippet: local Quad = love.graphics.newQuad function love.load() character= {} character.player = love.graphics.newImage("sprite.png") character.x = 50 character.y = 50 direction = "right" iteration = 1 max = 8 idle = true timer = 0.1 quads = {} quads['right'] ={} quads['left'] = {} for j=1,8 do quads['right'][j] = Quad((j-1)*32, 0, 32, 32, 256, 32); quads['left'][j] = Quad((j-1)*32, 0, 32, 32, 256, 32); -- for the character to face the opposite direction, the quad need to be flipped by using the Quad:flip(x, y) method, where x and why are Boolean. quads.left[j]:flip(true, false) --flip horizontally x = true, y = false end end Now let us update our motion; if a certain key is pressed, the animation should play; if the key is released, the animation should stop. Also, if the key is pressed, the character should change position. We'll be using the love.keypressed callback function here, as shown in the following code snippet: function love.update(dt) if idle == false then timer = timer + dt if timer > 0.2 then timer = 0.1 -- The animation will play as the iteration increases, so we just write iteration = iteration + 1, also we'll stop reset our iteration at the maximum of 8 with a timer update to keep the animation smooth. iteration = iteration + 1 if love.keyboard.isDown('right') then sprite.x = sprite.x + 5 end if love.keyboard.isDown('left') then sprite.x = sprite.x - 5 end if iteration > max then iteration = 1 end end end end function love.keypressed(key) if quads[key] then direction = key idle = false end end function love.keyreleased(key) if quads[key] and direction == key then idle = true iteration = 1 direction = "right" end end Finally, we can draw our character on the screen. Here we'll be using love.graphics.drawq(image, quad, x, y), where image is the image data, quad will load our quads table, x is the position in x axis and y is the position in the y axis: function love.draw() love.graphics.drawq(sprite.player, quads[direction][iteration], sprite.x, sprite.y) end So let's package our game and run it to see the magic in action by pressing the left or right navigation key: Summary That is all for this article. We have learned how to draw 2D objects on the screen and move the objects in four directions. We have delved into the usage of sprites for animations and how to play these animations with code. Resources for Article: Further resources on this subject: Panda3D Game Development: Scene Effects and Shaders [Article] Microsoft XNA 4.0 Game Development: Receiving Player Input [Article] Introduction to Game Development Using Unity 3D [Article]
Read more
  • 0
  • 0
  • 5818

Packt
03 Oct 2013
8 min read
Save for later

Quick start – using Foundation 4 components for your first website

Packt
03 Oct 2013
8 min read
(For more resources related to this topic, see here.)   Step 1 – using the Grid The base building block that Foundation 4 provides is the Grid. This component allows us to easily put the rest of elements in the page. The Grid also avoids the temptation of using tables to put elements in their places. Tables should be only used to show tabular data. Don't use them with any other meaning. Web design using tables is considered a really terrible practice. Defining a grid, intuitively, consists of defining rows and columns. There are basically two ways to do this, depending on which kind of layout you want to create. They are as follows: If you want a simple layout which evenly splits contents of the page, you should use Block Grid . To use Block Grid, we must have the default CSS package or be sure to have selected Block Grid from a custom package. If you want a more complex layout, with different sized elements and not necessarily evenly distributed, you should use the normal Grid. This normal Grid contains up to 12 grid columns, to put elements into. After picking the general layout of your page, you should decide if you want your grid structure to be the same for small devices, such as smartphones or tablets, as it will be on large devices, such as desktop computers or laptops. So, our first task is to define a grid structure for our page as follows: Select how we want to distribute our elements. We choose Block Grid, the simpler one. Consequently, we define a <ul> element with several <li> elements inside it. Select if we want different structure for large and small screens. We Yes .This is important to determine which Foundation 4 CSS class our elements will belong to. As result, we have the following code: <ul class="large-block-grid-4"> <li><img src ="demo1.jpg"></li> <li><img src ="demo2.jpg"></li> <li><img src ="demo3.jpg"></li> <li><img src ="demo4.jpg"></li> </ul> The key concept here is the class large-block-grid-4. There are two important classes related to Block Grid: Small-block-grid : If the element belongs to this class, the resulting Block Grid will keep its spacing and configuration for any screen size Large-block-grid : With this, the behavior will change between large and small screen sizes. So, large forces the responsive behavior. You can also use both classes together. In that case, large overrides the behavior of small class. The number 4 at the end of the class name is just the number of grid columns. The complete code of our page, so far is as follows: <!DOCTYPE html> <!--[if IE 8]><html class="no-js lt-ie9" lang="en" ><![endif]--> <!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> <head> <meta charset="utf-8" /> <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" /> <title>My First Foundation Web</title> <link rel="stylesheet" href="css/foundation.css" /> <script src ="js/vendor/custom.modernizr.js"></script> <style> img { width: 300px; border: 1px solid #ddd; } </style> </head> <body> <!-- Grid --> <ul class="large-block-grid-4"> <li><img src ="demo1.jpg"></li> <li><img src ="demo2.jpg"></li> <li><img src ="demo3.jpg"></li> <li><img src ="demo4.jpg"></li> </ul> <!-- end grid --> <script> document.write('<script src =' + ('__proto__' in {} ? 'js/vendor/zepto' : 'js/vendor/jquery') + .js><\/script>') </script> <script src ="js/foundation.min.js"></script> <script> $(document).foundation(); </script> </body> </html> We have created a simple HTML file with a basic grid that contains 4 images in a list, using the Block Grid facility. Each image spans 4 grid columns. The following screenshot shows how our page looks: Not very fancy, uh? Don't worry, we will add some nice features in the following steps. We have way more options to choose from, for the grid arrangements of our pages. Visit http://foundation.zurb.com/docs/components/grid.html for more information. Step 2 – the navigation bar Adding a basic top navigation bar to our website with Foundation 4 is really easy. We just follow steps: Create an HTML nav element by inserting the following code: <nav class="top-bar"></nav> Add a title for the navigation bar (optional) by inserting the following code: <nav class="top-bar"> <ul class="title-area"> <li class="name"> <h1><a href="#">My First Foundation Web</h1> </li> <li class="toggle-topbar"> <a href="#"><span>Menu</span></a> </li> </ul> </nav> Add some navigation elements inside the nav element <nav class="top-bar"> <!--Title area --> <ul class="title-area"> <li class="name"> <h1><a href="#">My First Foundation Web</h1> </li> <li class="toggle-topbar"> <a href="#"><span>Menu</span></a> </li> </ul> <!-- Here starts nav Section --> <section class="top-bar-section"> <!-- Left Nav Section --> <ul class="left"> <li class="divider"></li> <li class="has-dropdown"><a href="#">Options</a> <!-- First submenu --> <ul class="dropdown"> <li class="has-dropdown"><a href="#">Option 1a</a> <!-- Second submenu --> <ul class="dropdown"> <li><label>2nd Options list</label></li> <li><a href="#">Option 2a</a></li> <li><a href="#">Option 2b</a></li> <li class="has-dropdown"> <a href="#">Option 2c</a> <!-- Third submenu --> <ul class="dropdown"> <li><label>3rd Options list</label></li> <li><a href="#">Option 3a</a></li> <li><a href="#">Option 3b</a></li> <li><a href="#">Option 3c</a></li> </ul> </li> </ul> </li> <!-- Visual separation between elements --> <li class="divider"></li> <li><a href="#">Option 2b</a></li> <li><a href="#">Option 2c</a></li> </ul> </li> <li class="divider"></li> </ul> </section> </nav> Interesting parts in the preceding code are as follows: <li class="divider">: It creates a visual separation between the elements of a list <li class="has-dropdown">: It shows a drop-down element when is clicked. <ul class="dropdown">: It indicates that the list is a drop-down menu. Apart from that, the left class, used to specify where we want the buttons on the bar. We would use right class to put them on the right side of the screen, or both classes, if we want several buttons in different places. Our navigation bar looks like the following screenshot: Of course, this navigation bar shows responsive behavior, thanks to the class toggle-topbar-menu-icon. This class forces the buttons to collapse in narrower screens. And it looks like the following screenshot: Now we know how to add a top navigation bar to our page. We just add the navigation bar's code before the grid, and the result is shown in the following screenshot: Summary In this article we learnt how to use 2 of the many UI elements provided to us by Foundation 4, the grid and the navigation bar. The screenshots provided help in giving a better idea of how the elements look, when incorporated in our website. Resources for Article : Further resources on this subject: HTML5: Generic Containers [Article] Building HTML5 Pages from Scratch [Article] Video conversion into the required HTML5 Video playback [Article]
Read more
  • 0
  • 0
  • 4659

article-image-simple-graphs-d3js
Packt
03 Oct 2013
8 min read
Save for later

Simple graphs with d3.js

Packt
03 Oct 2013
8 min read
(For more resources related to this topic, see here.) There are many component vendors selling graphing controls. Frequently, these graph libraries are complicated to work with and expensive. When using them, you need to consider what would happen if the vendor went out of business or refused to fix an issue: For simple graphs, such as the one shown in the preceding screenshot, d3.js (http://d3js.org/) brings a number of functions and a coding style that makes creating graphs a snap. Let's create a using d3. The first thing to do is introduce an SVG element to the page. In d3, we'll append an SVG element explicitly: var graph = d3.select("#visualization") .append("svg") .attr("width", 500) .attr("height", 500); d3 relies heavily on the use of method chaining. If you're new to this concept, it is quick to pick up. Each call to a method performs some action and then returns an object, and the next method call operates on this object. So, in this case, the select method returns the div with the id of visualization. Calling append on the selected div adds an SVG element and then returns it. Finally, the attr methods set a property inside the object and then return the object. At first, method chaining may seem odd, but as we move on you'll see that it is a very powerful technique and cleans up the code considerably. Without method chaining, we end up with a lot of temporary variables. Next, we need to find the maximum element in the data array. Previously we might have used a jQuery each loop to find that element. d3 has built-in array functions that make this much cleaner: var maximumValue = d3.max(data, function(item){ return item.value;}); There are similar functions for finding minimums and means. None of the functions are anything you couldn't get by using a JavaScript utility library such as underscore.js (http://underscorejs.org/) or lodash (http://lodash.com/), however, it is convenient to make use of the built-in versions. In the next piece, we make us of are d3's scaling functions: var yScale = d3.scale.linear() .domain([maximumValue, 0]) .range([0, 300]); Scaling functions serve to map from one dataset to another. In our case, we're mapping from the values in our data array to the coordinates in our SVG. We use two different scales here: linear and ordinal. The linear scale is used to map a continuous domain to a continuous range. The mapping will be done linearly, so if our domain contained values between 0 and 10, and our range had values between 0 and 100, a value of 6 would map to 60, 3 to 30, and so forth. It seems trivial, but with more complicated domains and ranges, scales are very helpful. Apart from linear scales, there are power and logarithmic scales that may fit your data better. In our example data, our y values are not continuous, they're not even numeric. For this case we can make use of an ordinal scale: var xScale = d3.scale.ordinal() .domain(data.map(function(item){return item.month;})) .rangeBands([0,500], .1); ordinal scales map a discrete domain into a continuous range. Here, the domain is the list of months and the range the width of our SVG. You'll note that instead of using range we use rangeBands. rangebands splits the range up into chunks each to which each range item is assigned. So, if our domain was {May, June} and the range 0 to 100, from May we would receive a band from 0 to 49 and 50 to 100 from June. You'll also note that rangeBands takes an additional parameter, in our case 0.1. This is a padding value that generates a sort of no man's land between each band. This is ideal for creating a bar or column graph, as we may not want the columns touching each other. The padding parameter can take a value between 0 and 1 as a decimal representation of how much of the range should be reserved for padding. 0.25 would reserve 25% of the range for padding. There are also a family of built-in scales that deal with providing colors. Selecting colors for your visualization can be challenging, as the colors have to be far enough apart to be discernible. If you're color-challenged like me, the scales category10, category20, category20b, and category20c may be for you. You can declare a color scale in the the following manner: var colorScale = d3.scale.category10() .domain(data.map(function(item){return item.month;})); The preceding code will assign a different color to each month out of a set of 10 pre-calculated possible colors. Finally, we need to actually draw our graph: var graphData = graph.selectAll(".bar") .data(data); We select all the .bar elements inside the graph using selectAll. Hang on! There aren't any elements inside the graph, let alone elements that match the .bar selector. Typically, selectAll will return a collection of elements matching the selector just as the $ function does in jQuery. In this case, we're using selectAll as a short-hand method of creating an empty d3 collection that has a data method and can be chained. We next specify a set of data to union with the data from the existing selection of elements. d3 operates on collections objects without using looping techniques. This allows for a more declarative style of programming, but can be difficult to grasp immediately. In effect, we're unioning two sets of data: the currently existing data (found using selectAll) and the new data (provided by the data function). This method of dealing with data allows for easy updates to the data elements, should further elements be added or removed later. When new data elements are added, you can select just those elements by using enter(). This prevents repeating actions on existing data. You don't need to redraw the entire image, just the portions that have been updated with new data. Equally, if there are elements in the new dataset that didn't appear in the old one, they can be selected with exit(). Typically, you want to just remove those elements that can be done by running the following: graphData.exit().remove() When we create elements using the newly generated dataset, the data elements will actually be attached to the newly created DOM elements. Creating the elements involves calling append: graphData.enter() .append("rect") .attr("x", function(item){ return xScale(item.month);}) .attr("y", function(item){ return yScale(item.value);}) .attr("height", function(item){ return 300 - yScale(item.value);}) .attr("width", xScale.rangeBand()) .attr("fill", function(item, index){return colorScale(item.month);}); The following diagram shows how data() works with new and existing dataset: You can see in this chunk of code how useful method chaining has become. It makes the code much shorter and more readable than assigning a series of temporary variables or passing the results into standalone methods. The scales also come on their own here. The x coordinate is found simply by scaling the month we have using the ordinal scale. Because that scale takes into account the number of elements as well as the padding, nothing more complicated is needed. The y coordinate is similarly found using previously defined yScale. Because the origin in an SVG is in the top-left, we have to take the inverse of the scale to calculate the height. Again, this is a place where we wouldn't generally be using a constant except for the brevity of our example. The width of the column is found by asking the xScale for the width of the bands. Finally, we set the color based on the color scale so it appears as follows: Transitions Being able to animate your visualization is a powerful technique for adding that "wow" factor. People are far more likely to stop and pay attention to your visualization if it looks cool. Spending time to make your visualization look nifty can actually have a payback other than getting mad cred from other developers. d3 makes transitions simple by doing most of the heavy lifting for you. The transitions work by creating gradual changes in values of properties: graph.selectAll(".bar") .attr("height", 0) .transition() .attr("height", 50) This code will gradually change the height on a .bar from 0 to 50 px. By default, the transition will take 250ms, but this can be changed by chaining a call to duration and delayed with a call to delay: graph.selectAll(".bar") .attr("height", 0) .transition() .duration(400) .delay(100) .attr("height", 50) This transition will wait 100ms then grow the bar height over a course of 400ms. Transitions can also be used to change non-numeric attributes such as color. I like to use a few transitions during loading to get attention, but they can also be useful when changing the state of the visualization. Even such things as adding new elements using .data can have a transition attached to them. If there is a problem with transitions, it is that they are too easy to use. Be careful that you don't overuse them and overwhelm the user. You should also pay attention to the duration of the transition: if it takes too long to execute, users will lose interest. Summary This article shed light on the features of d3.js that can be used to create simple graphs. It also covered how to add that "wow" factor to your visualizations using transitions. Resources for Article: Further resources on this subject: Introducing QlikView elements [Article] Data sources for the Charts [Article] HTML5 Presentations - creating our initial presentation [Article]
Read more
  • 0
  • 0
  • 2730

article-image-creating-our-first-bot-webbot
Packt
03 Oct 2013
9 min read
Save for later

Creating our first bot, WebBot

Packt
03 Oct 2013
9 min read
(For more resources related to this topic, see here.)   With the knowledge you have gained, we are now ready to develop our first bot, which will be a simple bot that gathers data (documents) based on a list of URLs and datasets (field and field values) that we will require. First, let's start by creating our bot package directory. So, create a directory called WebBot so that the files in our project_directory/lib directory look like the following: '-- project_directory|-- lib | |-- HTTP (our existing HTTP package) | | '-- (HTTP package files here) | '-- WebBot | |-- bootstrap.php| |-- Document.php | '-- WebBot.php |-- (our other files)'-- 03_webbot.php As you can see, we have a very clean and simple directory and file structure that any programmer should be able to easily follow and understand. The WebBot class Next, open the file WebBot.php file and add the code from the project_directory/lib/WebBot/WebBot.php file: In our WebBot class, we first use the __construct() method to pass the array of URLs (or documents) we want to fetch, and the array of document fields are used to define the datasets and regular expression patterns. Regular expression patterns are used to populate the dataset values (or document field values). If you are unfamiliar with regular expressions, now would be a good time to study them. Then, in the __construct() method, we verify whether there are URLs to fetch or not. If there , we set an error message stating this problem. Next, we use the __formatUrl() method to properly format URLs we fetch data. This method will also set the correct protocol: either HTTP or HTTPS ( Hypertext Transfer Protocol Secure ). If the protocol is already set for the URL, for example http://www.[dom].com, we ignore setting the protocol. Also, if the class configuration setting conf_force_https is set to true, we force the HTTPS protocol again unless the protocol is already set for the URL. We then use the execute() method to fetch data for each URL, set and add the Document objects to the array of documents, and track document statistics. This method also implements fetchdelay logic that will delay each fetch by x number of seconds if set in the class configuration settings conf_delay_between_fetches. We also include the logic that only allows distinct URL fetches, meaning that, if we have already fetched data for a URL we won't fetch it again; this eliminates duplicate URL data fetches. The Document object is used as a container for the URL data, and we can use the Document object to use the URL data, the data fields, and their corresponding data field values. In the execute() method, you can see that we have performed a HTTPRequest::get() request using the URL and our default timeout value—which is set with the class configuration settings conf_default_timeout. We then pass the HTTPResponse object that is returned by the HTTPRequest::get() method to the Document object. Then, the Document object uses the data from the HTTPResponse object to build the document data. Finally, we include the getDocuments() method, which simply returns all the Document objects in an array that we can use for our own purposes as we desire. The WebBot Document class Next, we need to create a class called Document that can be used to store document data and field names with their values. To do this we will carry out the following steps: We first pass the data retrieved by our WebBot class to the Document class. Then, we define our document's fields and values using regular expression patterns. Next, add the code from the project_directory/lib/WebBot/Document.php file. Our Document class accepts the HTTPResponse object that is set in WebBot class's execute() method, and the document fields and document ID. In the Document __construct() method, we set our class properties: the HTTP Response object, the fields (and regular expression patterns), the document ID, and the URL that we use to fetch the HTTP response. We then check if the HTTP response successful (status code 200), and if it isn't, we set the error with the status code and message. Lastly, we call the __setFields() method. The __setFields() method parses out and sets the field values from the HTTP response body. For example, if in our fields we have a title field defined as $fields = ['title' => '<title>(.*)</title>'];, the __setFields() method will add the title field and pull all values inside the <title>*</title> tags into the HTML response body. So, if there were two title tags in the URL data, the __setField() method would add the field and its values to the document as follows: ['title'] => [ 0 => 'title x', 1 => 'title y' ] If we have the WebBot class configuration variable—conf_include_document_field_raw_values—set to true, the method will also add the raw values (it will include the tags or other strings as defined in the field's regular expression patterns) as a separate element, for example: ['title'] => [ 0 => 'title x', 1 => 'title y', 'raw' => [ 0 => '<title>title x</title>', 1 => '<title>title y</title>' ] ] The preceding code is very useful when we want to extract specific data (or field values) from URL data. To conclude the Document class, we have two more methods as follows: getFields(): This method simply returns the fields and field values getHttpResponse(): This method can be used to get the HTTPResponse object that was originally set by the WebBot execute() method This will allow us to perform logical requests to internal objects if we wish. The WebBot bootstrap file Now we will create a bootstrap.php file (at project_directory/lib/WebBot/) to load the HTTP package and our WebBot package classes, and set our WebBot class configuration settings: <?php namespace WebBot; /** * Bootstrap file * * @package WebBot */ // load our HTTP package require_once './lib/HTTP/bootstrap.php'; // load our WebBot package classes require_once './lib/WebBot/Document.php'; require_once './lib/WebBot/WebBot.php'; // set unlimited execution time set_time_limit(0); // set default timeout to 30 seconds WebBotWebBot::$conf_default_timeout = 30; // set delay between fetches to 1 seconds WebBotWebBot::$conf_delay_between_fetches = 1; // do not use HTTPS protocol (we'll use HTTP protocol) WebBotWebBot::$conf_force_https = false; // do not include document field raw values WebBotWebBot::$conf_include_document_field_raw_values = false; We use our HTTP package to handle HTTP requests and responses. You have seen in our WebBot class how we use HTTP requests to fetch the data, and then use the HTTP Response object to store the fetched data in the previous two sections. That is why we need to include the bootstrap file to load the HTTP package properly. Then, we load our WebBot package files. Because our WebBot class uses the Document class, we load that class file first. Next, we use the built-in PHP function set_time_limit() to tell the PHP interpreter that we want to allow unlimited execution time for our script. You don't necessarily have to use unlimited execute time. However, for testing reasons, we will use unlimited execution time for this example. Finally, we set the WebBot class configuration settings. These settings are used by the WebBot object internally to make our bot work as we desire. We should always make the configuration settings as simple as possible to help other developers understand. This means we should also include detailed comments in our code to ensure easy usage of package configuration settings. We have set up four configuration settings in our WebBot class. These are static and public variables, meaning that we can set them from anywhere after we have included the WebBot class, and once we set them they will remain the same for all WebBot objects unless we change the configuration variables. If you do not understand the PHP keyword static, now would be a good time to research this subject. The first configuration variable is conf_default_timeout. This variable is used to globally set the default timeout (in seconds) for all WebBot objects we create. The timeout value tells the HTTPRequest class how long it continue trying to send a request before stopping and deeming it as a bad request, or a timed-out request. By default, this configuration setting value is set to 30 (seconds). The second configuration variable—conf_delay_between_fetches—is used to set a time delay (in seconds) between fetches (or HTTP requests). This can be very useful when gathering a lot of data from a website or web service. For example, say, you had to fetch one million documents from a website. You wouldn't want to unleash your bot with that type of mission without fetch delays because you could inevitably cause—to that website—problems due to massive requests. By default, this value is set to 0, or no delay. The third WebBot class configuration variable—conf_force_https—when set to true, can be used to force the HTTPS protocol. As mentioned earlier, this will not override any protocol that is already set in the URL. If the conf_force_https variable is set to false, the HTTP protocol will be used. By default, this value is set to false. The fourth and final configuration variable—conf_include_document_field_raw_values—when set to true, will force the Document object to include the raw values gathered from the ' regular expression patterns. We've discussed configuration settings in detail in the WebBot Document Class section earlier in this article. By default, this value is set to false. Summary In this article you have learned how to get started with building your first bot using HTTP requests and responses. Resources for Article : Further resources on this subject: Installing and Configuring Jobs! and Managing Sections, Categories, and Articles using Joomla! [Article] Search Engine Optimization in Joomla! [Article] Adding a Random Background Image to your Joomla! Template [Article]
Read more
  • 0
  • 0
  • 2621
article-image-working-databases
Packt
01 Oct 2013
19 min read
Save for later

Working with Databases

Packt
01 Oct 2013
19 min read
(For more resources related to this topic, see here.) We have learned how to create snippets, how to work with forms, and Ajax, how to test your code, and how to create a REST API, and all this is awesome! However, we did not learn how to persist data to make it durable. In this article, we will see how to use Mapper, an object-relational mapping ( ORM ) system for relational databases, included with Lift. The idea is to create a map of database tables into a well-organized structure of objects; for example, if you have a table that holds users, you will have a class that represents the user table. Let's say that a user can have several phone numbers. Probably, you will have a table containing a column for the phone numbers and a column containing the ID of the user owning that particular phone number. This means that you will have a class to represent the table that holds phone numbers, and the user class will have an attribute to hold the list of its phone numbers; this is known as a one-to-many relationship . Mapper is a system that helps us build such mappings by providing useful features and abstractions that make working with databases a simple task. For example, Mapper provides several types of fields such as MappedString, MappedInt, and MappedDate which we can use to map the attributes of the class versus the columns in the table being mapped. It also provides useful methods such as findAll that is used to get a list of records or save, to persist the data. There is a lot more that Mapper can do, and we'll see what this is through the course of this article. Configuring a connection to database The first thing we need to learn while working with databases is how to connect the application that we will build with the database itself. In this recipe, we will show you how to configure Lift to connect with the database of our choice. For this recipe, we will use PostgreSQL; however, other databases can also be used: Getting ready Start a new blank project. Edit the build.sbt file to add the lift-mapper and PostgreSQL driver dependencies: "net.liftweb" %% "lift-mapper" % liftVersion % "compile", "org.postgresql" % "postgresql" % "9.2-1003-jdbc4" % "compile" Create a new database. Create a new user. How to do it... Now carry out the following steps to configure a connection with the database: Add the following lines into the default.props file: db.driver=org.postgresql.Driver db.url=jdbc:postgresql:liftbook db.user=<place here the user you've created> db.password=<place here the user password> Add the following import statement in the Boot.scala file: import net.liftweb.mapper._ Create a new method named configureDB() in the Boot.scala file with the following code: def configureDB() { for { driver <- Props.get("db.driver") url <- Props.get("db.url") } yield { val standardVendor = new StandardDBVendor(driver, url, Props.get("db.user"), Props.get("db.password")) LiftRules.unloadHooks.append(standardVendor.closeAllConnections_! _) DB.defineConnectionManager(DefaultConnectionIdentifier, standardVendor) } } Then, invoke configureDB() from inside the boot method. How it works... Lift offers the net.liftweb.mapper.StandardDBVendor class, which we can use to create connections to the database easily. This class takes four arguments: driver, URL, user, and password. These are described as follows: driver : The driver argument is the JDBC driver that we will use, that is, org.postgresql.Driver URL : The URL argument is the JDBC URL that the driver will use, that is, jdbc:postgresql:liftbook user and password : The user and password arguments are the values you set when you created the user in the database. After creating the default vendor, we need to bind it to a Java Naming and Directory Interface (JNDI) name that will be used by Lift to manage the connection with the database. To create this bind, we invoked the defineConnectionManager method from the DB object. This method adds the connection identifier and the database vendor into a HashMap, using the connection identifier as the key and the database vendor as the value. The DefaultConnectionIdentifier object provides a default JNDI name that we can use without having to worry about creating our own connection identifier. You can create your own connection identifier if you want. You just need to create an object that extends ConnectionIdentifier with a method called jndiName that should return a string. Finally, we told Lift to close all the connections while shutting down the application by appending a function to the unloadHooks variable. We did this to avoid locking connections while shutting the application down. There's more... It is possible to configure Lift to use a JNDI datasource instead of using the JDBC driver directly. In this way, we can allow the container to create a pool of connections and then tell Lift to use this pool. To use a JNDI datasource, we will need to perform the following steps: Create a file called jetty-env.xml in the WEB-INF folder under src/main/webapp/ with the following content: <!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure.dtd"> <Configure class="org.eclipse.jetty.webapp.WebAppContext"> <New id="dsliftbook" class="org.eclipse.jetty.plus.jndi.Resource"> <Arg>jdbc/dsliftbook</Arg> <Arg> <New class="org.postgresql.ds.PGSimpleDataSource"> <Set name="User">place here the user you've created</Set> <Set name="Password">place here the user password</Set> <Set name="DatabaseName">liftbook</Set> <Set name="ServerName">localhost</Set> <Set name="PortNumber">5432</Set> </New> </Arg> </New> </Configure> Add the following line into the build .sbt file: env in Compile := Some(file("./src/main/webapp/WEB-INF/jetty-env.xml") asFile) Remove all the jetty dependencies and add the following: "org.eclipse.jetty" % "jetty-webapp" % "8.0.4.v20111024" % "container", "org.eclipse.jetty" % "jetty-plus" % "8.0.4.v20111024" % "container", Add the following code into the web.xml file. <resource-ref> <res-ref-name>jdbc/dsliftbook</res-ref-name> <res-type>javax.sql.DataSource</res-type> <res-auth>Container</res-auth> </resource-ref> Remove the configureDB method. Replace the invocation of configureDB method with the following line of code: DefaultConnectionIdentifier.jndiName = "jdbc/dsliftbook" The creation of the file jetty-envy.xml and the change we made in the web.xml file were to create the datasource in jetty and make it available to Lift. Since the connections will be managed by jetty now, we don't need to append hooks so that Lift can close the connections or any other configuration when shutting down the application. All we need to do is tell Lift to get the connections from the JNDI datasource that we have configured into the jetty. We do this by setting the jndiName variable of the default connection identifier, DefaultConnectionIdentifier as follows: DefaultConnectionIdentifier.jndiName = "jdbc/dsliftbook" The change we've made to the build.sbt file was to make the jetty-env.xml file available to the embedded jetty. So, we can use it when we get the application started by using the container:start command. See also... You can learn more about how to configure a JNDI datasource in Jetty at the following address: http://wiki.eclipse.org/Jetty/Howto/Configure_JNDI_Datasource Mapping a table to a Scala class Now that we know how to connect Lift applications to the database, the next step is to learn how to create mappings between a database table and a Scala object using Mapper. Getting ready We will re-use the project we created in the previous recipe since it already has the connection configured. How to do it... Carry out the following steps to map a table into a Scala object using Mapper: Create a new file named Contact.scala inside the model package under src/main/scala/code/ with the following code: package code.model import net.liftweb.mapper.{MappedString, LongKeyedMetaMapper, LongKeyedMapper, IdPK} class Contact extends LongKeyedMapper[Contact] with IdPK { def getSingleton = Contact object name extends MappedString(this, 100) } object Contact extends Contact with LongKeyedMetaMapper[Contact] { override def dbTableName = "contacts" } Add the following import statement in the Boot.scala file: import code.model.Contact Add the following code into the boot method of the Boot class: Schemifier.schemify( true, Schemifier.infoF _, Contact ) Create a new snippet named Contacts with the following code: package code.snippet import code.model.Contact import scala.xml.Text import net.liftweb.util.BindHelpers._ class Contacts { def prepareContacts_!() { Contact.findAll().map(_.delete_!) val contactsNames = "John" :: "Joe" :: "Lisa" :: Nil contactsNames.foreach(Contact.create.name(_).save()) } def list = { prepareContacts_!() "li *" #> Contact.findAll().map { c => { c.name.get } } } } Edit the index.html file by replacing the content of the div element with main as the value of id using the following code : <div data-list="Contacts.list"> <ul> <li></li> </ul> </div> Start the application. Access your local host (http://localhost:8080), and you will see a page with three names, as shown in the following screenshot: How it works... To work with Mapper, we use a class and an object. The first is the class that is a representation of the table; in this class we define the attributes the object will have based on the columns of the table we are mapping. The second one is the class' companion object where we define some metadata and helper methods. The Contact class extends the LongKeyedMapper[Contact] trait and mixes the trait IdPK. This means that we are defining a class that has an attribute called id (the primary key), and this attribute has the type Long. We are also saying that the type of this Mapper is Contact. To define the attributes that our class will have, we need to create objects that extend "something". This "something" is the type of the column. So, when we say an object name extends MappedString(this, 100), we are telling Lift that our Contact class will have an attribute called name, which will be a string that can be 100 characters long. After defining the basics, we need to tell our Mapper where it can get the metadata about the database table. This is done by defining the getSingleton method. The Contact object is the object that Mapper uses to get the database table metadata. By default, Lift will use the name of this object as the table name. Since we don't want our table to be called contact but contacts, we've overridden the method dbTableName. What we have done here is created an object called Contact, which is a representation of a table in the database called contacts that has two columns: id and name. Here, id is the primary key and of the type Long, and name is of the type String, which will be mapped to a varchar datatype. This is all we need to map a database table to a Scala class, and now that we've got the mapping done, we can use it. To demonstrate how to use the mapping, we have created the snippet Contacts. This snippet has two methods. The list method does two things; it first invokes the prepareContacts_!() method, and then invokes the findAll method from the Contact companion object. The prepareContacts_!() method also does two things: it first deletes all the contacts from the database and then creates three contacts: John, Joe, and Lisa. To delete all the contacts from the database, it first fetches all of them using the findAll method, which executes a select * from contacts query and returns a list of Contact objects, one for each existing row in the table. Then, it iterates over the collection using the foreach method and for each contact, it invokes the delete_! method which as you can imagine will execute a delete from contacts where id = contactId query is valid. After deleting all the contacts from the database table, it iterates the contactsNames list, and for each element it invokes the create method of the Contact companion object, which then creates an instance of the Contact class. Once we have the instance of the Contact class, we can set the value of the name attribute by passing the value instance.name(value). You can chain commands while working with Mapper objects because they return themselves. For example, let's say our Contact class has firstName and lastName as attributes. Then, we could do something like this to create and save a new contact: Contact.create.firstName("John").lastName("Doe").save() Finally, we invoke the save() method of the instance, which will make Lift execute an insert query, thus saving the data into the database by creating a new record. Getting back to the list method, we fetch all the contacts again by invoking the findAll method, and then create a li tag for each contact we have fetched from the database. The content of the li tags is the contact name, which we get by calling the get method of the attribute we want the value of. So, when you say contact.name.get, you are telling Lift that you want to get the value of the name attribute from the contact object, which is an instance of the Contact class. There's more... Lift comes with a variety of built-in field types that we can use; MappedString is just one of them. The others include, MappedInt, MappedLong, and MappedBoolean. All these fields come with some built-in features such as the toForm method, which returns the HTML needed to generate a form field, and the validate method that validates the value of the field. By default, Lift will use the name of the object as the name of the table's column; for example, if you define your object as name—as we did—Lift will assume that the column name is name. Lift comes with a built-in list of database-reserved words such as limit, order, user, and so on. If the attribute you are mapping is a database-reserved word, Lift will append _c at the end of the column's name when using Schemifier. For example, if you create an attribute called user, Lift will create a database column called user_c. You can change this behavior by overriding the dbColumnName method, as shown in the following code: object name extends MappedString(this, 100) { override def dbColumnName = "some_new_name" } In this case, we are telling Lift that the name of the column is some_new_name. We have seen that you can fetch data from the database using the findAll method. However, this method will fetch every single row from the database. To avoid this, you can filter the result using the By object; for example, let's say you want to get only the contacts with the name Joe. To accomplish this, you would add a By object as the parameter of the findAll method as follows: Contact.findAll(By(Contact.name, "Joe")). There are also other filters such as ByList and NotBy. And if for some reason the features offered by Mapper to build select queries aren't enough, you can use methods such as findAllByPreparedStatement and findAllByInsecureSQL where you can use raw SQL to build the queries. The last thing left to talk about here is how this example would work if we didn't create any table in the database. Well, I hope you remember that we added the following lines of code to the Boot.scala file: Schemifier.schemify( true, Schemifier.infoF _, Contact ) As it turns out, the Schemifier object is a helper object that assures that the database has the correct schema based on a list of MetaMappers. This means that for each MetaMapper we pass to the Schemifier object, the object will compare the MetaMapper with the database schema and act accordingly so that both the MetaMapper and the database schema match. So, in our example, Schemifier created the table for us. If you change MetaMapper by adding attributes, Schemifier will create the proper columns in the table. See also... To learn more about query parameters, you can find more at: https://www.assembla.com/spaces/liftweb/wiki/Mapper#querying_the_database Creating one-to-many relationships In the previous recipe, we learned how to map a Scala class to a database table using Mapper. However, we have mapped a simple class with only one attribute, and of course, we will not face this while dealing with real-world applications. We will probably need to work with more complex data such as the one having one-to-many or many-to-many relationships. An example of this kind of relationship would be an application for a store where you'll have customers and orders, and we need to associate each customer with the orders he or she placed. This means that one customer can have many orders. In this recipe, we will learn how to create such relationships using Mapper. Getting ready We will modify the code from the last section by adding a one-to-many relationship into the Contact class. You can use the same project from before or duplicate it and create a new project. How to do it... Carry out the following steps: Create a new class named Phone into the model package under src/main/scala/code/ using the following code: package code.model import net.liftweb.mapper._ class Phone extends LongKeyedMapper[Phone] with IdPK { def getSingleton = Phone object number extends MappedString(this, 20) object contact extends MappedLongForeignKey(this, Contact) } object Phone extends Phone with LongKeyedMetaMapper[Phone] { override def dbTableName = "phones" } Change the Contact class declaration from: class Contact extends LongKeyedMapper[Contact] with IdPK To: class Contact extends LongKeyedMapper[Contact] with IdPK with OneToMany[Long, Contact] Add the attribute that will hold the phone numbers to the Contact class as follows: object phones extends MappedOneToMany(Phone, Phone.contact, OrderBy(Phone.id, Ascending)) with Owned[Phone] with Cascade[Phone] Insert the new class into the Schemifier list of parameters. It should look like the following code: Schemifier.schemify( true, Schemifier.infoF _, Contact, Phone ) In the Contacts snippet, replace the import statement from the following code: import code.model.Contact To: import code.model._ Modify the Contacts.prepareContacts_! method to associate a phone number to each contact. Your method should be similar to the one in the following lines of code: def prepareContacts_!() { Contact.findAll().map(_.delete_!) val contactsNames = "John" :: "Joe" :: "Lisa" :: Nil val phones = "5555-5555" :: "5555-4444" :: "5555-3333" :: "5555-2222" :: "5555-1111" :: Nil contactsNames.map(name => { val contact = Contact.create.name(name) val phone = Phone.create.number(phones((new Random()).nextInt(5))).saveMe() contact.phones.append(phone) contact.save() }) } Replace the list method's code with the following code: def list = { prepareContacts_!() ".contact *" #> Contact.findAll().map { contact => { ".name *" #> contact.name.get & ".phone *" #> contact.phones.map(_.number.get) } } } In the index.html file, replace the ul tag with the one in the following code snippet: <ul> <li class="contact"> <span class="name"></span> <ul> <li class="phone"></li> </ul> </li> </ul> Start the application. Access http://localhost:8080. Now you will see a web page with three names and their phone numbers, as shown in the following screenshot: How it works... In order to create a one-to-many relationship, we have mapped a second table called phone in the same way we've created the Contact class. The difference here is that we have added a new attribute called contact. This attribute extends MappedLongForeignKe y, which is the class that we need to use to tell Lift that this attribute is a foreign key from another table. In this case, we are telling Lift that contact is a foreign key from the contacts table in the phones table. The first parameter is the owner, which is the class that owns the foreign key, and the second parameter is MetaMapper, which maps the parent table. After mapping the phones table and telling Lift that it has a foreign key, we need to tell Lift what the "one" side of the one-to-many relationship will be. To do this, we need to mix the OneToMany trait in the Contact class. This trait will add features to manage the one-to-many relationship in the Contact class. Then, we need to add one attribute to hold the collection of children records; in other words, we need to add an attribute to hold the contact's phone numbers. Note that the phones attribute extends MappedOneToMany, and that the first two parameters of the MappedOneToMany constructor are Phone and Phone.contact. This means that we are telling Lift that this attribute will hold records from the phones table and that it should use the contact attribute from the Phone MetaMapper to do the join. The last parameter, which is optional, is the OrderBy object. We've added it to tell Lift that it should order, in an ascending order, the list of phone numbers by their id. We also added a few more traits into the phones attribute to show some nice features. One of the traits is Owned; this trait tells Lift to delete all orphan fields before saving the record. This means that if, for some reason, there are any records in the child table that has no owner in the parent table, Lift will delete them when you invoke the save method, helping you keep your databases consistent and clean. Another trait we've added is the Cascade trait. As its name implies, it tells Lift to delete all the child records while deleting the parent record; for example, if you invoke contact.delete_! for a given contact instance, Lift will delete all phone records associated with this contact instance. As you can see, Lift offers an easy and handy way to deal with one-to-many relationships. See also... You can read more about one-to-many relationships at the following URL: https://www.assembla.com/wiki/show/liftweb/Mapper#onetomany
Read more
  • 0
  • 0
  • 2526

article-image-ar-experience-using-vuforia-and-features-definition
Packt
01 Oct 2013
4 min read
Save for later

AR experience using Vuforia and features definition

Packt
01 Oct 2013
4 min read
(For more resources related to this topic, see here.) What decides trackable score? Trackables are the foundation of the AR experience using Vuforia. It is paramount to understand and create a suitable trackable for the experience to be robust and useful. The score attributed to the trackable in the target manager is our indication of how robust the target image is going to perform, but what decides that score? Best way of understanding this, is by understanding how Vuforia tracks the images. The idea is simple, it looks for position of contrasting edges in clusters all around the image. Those edges are tracked, and based on the map of positions that are stored in the dataset, Vuforia can tell the relative position of the trackable in the real world and accordingly render the 3D content on top of it. This particularly means that tracking the image is not a function of its color or what really is in it, as much as how many contrasting edges are there in the image, and how well they are distributed on the image. To better understand this, we can look on the current edges that are recognizable in the image we have just uploaded. To do that, simply click on the Show Features link on the top left of the webpage. The following image shows features in image target stones: Once the Show Features link has been clicked, the image target manager layers over the target image an overlay of where it detects a recognizable edge that it can track in a Vuforia image target. Notice that it is only tracking the dark edges between the Stones and nothing else in the image. It is even tracking only the high contrast edges between the Stones, while ignoring some of the lighter ones. Also notice that the number of edges found in the image is large, and evenly distributed all around the image. This is a great factor in what made this image great for tracking. To contrast this image's result, let's try an image that will yield a 1-star score when tried on the target manager. The following image shows landscape image added to target image: Before adding this image, intuitively we might think that this image is suitable for tracking. It certainly has a lot of details of a wide-angle landscape. But this image yielded a shocking 1-star result when added to the Target Manager. The main reason for the low score for this image is the fact that the entire image is a shade of green. This greatly diminishes contrasting edges in the image. If we are to click on the Show Features link on the top, we will be able to see what the target manager detected from the image. The following image shows features in the mountain landscape image: Immediately, we notice the considerably lower number of features detected in the image compared to the stones one. It only detected the edges created by the shadows of the objects in the image, which is clearly not enough to award it any score above 1 star. Features definition To help us get a higher score, we must understand what are the features that the target manager is looking for. We do know now that the main thing that the target manager is looking for in an image is edges, but what kind of edges specifically? To understand that, we need the definition of features. A feature is a sharp and spiked detail in the image, like the corner of an edge. Features must be very contrasting to be found and it has to be distributed evenly across the image and in a random manner. The following image shows shapes and features recognized in them: In the shapes illustrated above, we can see the yellow crosses representation of the features recognizable in the shape. The representation is as follows: Shape 1: It is a perfect circle without any corners at all, and such no features are recognizable in it. Shape 2: It has an edge to the left with two recognizable corners. That yields two features recognizable in the shape. Shape 3: It is a square with four edges and four corners. This yields four recognizable features in the shape. This means that any curved object yields little to none features at all. Primarily, humans and animals make very poor trackables due to their curved nature. Summary Thus in this article, we learned about how to track an image and which features are recognizable in an image. Resources for Article: Further resources on this subject: Interface Designing for Games in iOS [Article] Unity Game Development: Welcome to the 3D world [Article] Unity Game Development: Interactions (Part 1) [Article]
Read more
  • 0
  • 0
  • 17585

article-image-cross-platform-development-build-once-deploy-anywhere
Packt
01 Oct 2013
19 min read
Save for later

Cross-platform Development - Build Once, Deploy Anywhere

Packt
01 Oct 2013
19 min read
(For more resources related to this topic, see here.) The demo application – how the projects work together Take a look at the following diagram to understand and familiarize yourself with the configuration pattern that all of your Libgdx applications will have in common: What you see here is a compact view of four projects. The demo project to the very left contains the shared code that is referenced (that is, added to the build path) by all the other platform-specific projects. The main class of the demo application is MyDemo.java. However, looking at it from a more technical view, the main class where an application gets started by the operating system, which will be referred to as Starter Classes from now on. Notice that Libgdx uses the term "Starter Class" to distinguish between these two types of main classes in order to avoid confusion. We will cover everything related to the topic of Starter Classes in a moment. While taking a closer look at all these directories in the preceding screenshot, you may have spotted that there are two assets folders: one in the demo-desktop project and another one in demo-android. This brings us to the question, where should you put all the application's assets? The demo-android project plays a special role in this case. In the preceding screenshot, you see a subfolder called data, which contains an image named libgdx.png, and it also appears in the demo-desktop project in the same place. Just remember to always put all of your assets into the assets folder under the demo-android project. The reason behind this is that the Android build process requires direct access to the application's assets folder. During its build process, a Java source file, R.java, will automatically be generated under the gen folder. It contains special information for Android about the available assets. It would be the usual way to access assets through Java code if you were explicitly writing an Android application. However, in Libgdx, you will want to stay platform-independent as much as possible and access any resource such as assets only through methods provided by Libgdx. You may wonder how the other platform-specific projects will be able to access the very same assets without having to maintain several copies per project. Needless to say that this would require you to keep all copies manually synchronized each time the assets change. Luckily, this problem has already been taken care of by the generator as follows: The demo-desktop project uses a linked resource, a feature by Eclipse, to add existing files or folders to other places in a workspace. You can check this out by right-clicking on the demo-desktop project then navigating to Properties | Resource | Linked Resources and clicking on the Linked Resources tab. The demo-html project requires another approach since Google Web Toolkit ( GWT ) has a different build process compared to the other projects. There is a special file GwtDefinition.gwt.xml that allows you to set the asset path by setting the configuration property gdx.assetpath, to the assets folder of the Android project. Notice that it is good practice to use relative paths such as ../demo-android/assets so that the reference does not get broken in case the workspace is moved from its original location. Take this advice as a precaution to protect you and maybe your fellow developers too from wasting precious time on something that can be easily avoided by using the right setup right from the beginning. The following is the code listing for GwtDefinition.gwt.xml from demo-html : <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit trunk//EN" "http://google-web-toolkit.googlecode.com/svn/trunk/ distro-source/core/src/gwt-module.dtd"> <module> <inherits name='com.badlogic.gdx.backends.gdx_backends_gwt' /> <inherits name='MyDemo' /> <entry-point class='com.packtpub.libgdx.demo.client.GwtLauncher' /> <set-configuration-property name="gdx.assetpath" value="../demo-android/assets" /> </module> Backends Libgdx makes use of several other libraries to interface the specifics of each platform in order to provide cross-platform support for your applications. Generally, a backend is what enables Libgdx to access the corresponding platform functionalities when one of the abstracted (platform-independent) Libgdx methods is called. For example, drawing an image to the upper-left corner of the screen, playing a sound file at a volume of 80 percent, or reading and writing from/to a file. Libgdx currently provides the following three backends: LWJGL (Lightweight Java Game Library) Android JavaScript/WebGL As already mentioned in Introduction to Libgdx and Project Setup , there will also be an iOS backend in the near future. LWJGL (Lightweight Java Game Library) LWJGL ( Lightweight Java Game Library ) is an open source Java library originally started by Caspian Rychlik-Prince to ease game development in terms of accessing the hardware resources on desktop systems. In Libgdx, it is used for the desktop backend to support all the major desktop operating systems, such as Windows, Linux, and Mac OS X. For more details, check out the official LWJGL website at http://www.lwjgl.org/. Android Google frequently releases and updates their official Android SDK. This represents the foundation for Libgdx to support Android in the form of a backend. There is an API Guide available which explains everything the Android SDK has to offer for Android developers. You can find it at http://developer.android.com/guide/components/index.html. WebGL WebGL support is one of the latest additions to the Libgdx framework. This backend uses the GWT to translate Java code into JavaScript and SoundManager2 ( SM2 ), among others, to add a combined support for HTML5, WebGL, and audio playback. Note that this backend requires a WebGL-capable web browser to run the application. You might want to check out the official website of GWT: https://developers.google.com/web-toolkit/. You might want to check out the official website of SM2: http://www.schillmania.com/projects/soundmanager2/. You might want to check out the official website of WebGL: http://www.khronos.org/webgl/. There is also a list of unresolved issues you might want to check out at https://github.com/libgdx/libgdx/blob/master/backends/gdx-backends-gwt/issues.txt. Modules Libgdx provides six core modules that allow you to access the various parts of the system your application will run on. What makes these modules so great for you as a developer is that they provide you with a single Application Programming Interface ( API ) to achieve the same effect on more than just one platform. This is extremely powerful because you can now focus on your own application and you do not have to bother with the specialties that each platform inevitably brings, including the nasty little bugs that may require tricky workarounds. This is all going to be transparently handled in a straightforward API which is categorized into logic modules and is globally available anywhere in your code, since every module is accessible as a static field in the Gdx class. Naturally, Libgdx does always allow you to create multiple code paths for per-platform decisions. For example, you could conditionally increase the level of detail in a game when run on the desktop platform, since desktops usually have a lot more computing power than mobile devices. The application module The application module can be accessed through Gdx.app. It gives you access to the logging facility, a method to shutdown gracefully, persist data, query the Android API version, query the platform type, and query the memory usage. Logging Libgdx employs its own logging facility. You can choose a log level to filter what should be printed to the platform's console. The default log level is LOG_INFO. You can use a settings file and/or change the log level dynamically at runtime using the following code line: Gdx.app.setLogLevel(Application.LOG_DEBUG); The available log levels are: LOG_NONE: This prints no logs. The logging is completely disabled. LOG_ERROR: This prints error logs only. LOG_INFO: This prints error and info logs. LOG_DEBUG: This prints error, info, and debug logs. To write an info, debug, or an error log to the console, use the following listings: Gdx.app.log("MyDemoTag", "This is an info log."); Gdx.app.debug("MyDemoTag", "This is a debug log."); Gdx.app.error("MyDemoTag", "This is an error log."); Shutting down gracefully You can tell Libgdx to shutdown the running application. The framework will then stop the execution in the correct order as soon as possible and completely de-allocate any memory that is still in use, freeing both Java and the native heap. Use the following listing to initiate a graceful shutdown of your application: Gdx.app.exit(); You should always do a graceful shutdown when you want to terminate your application. Otherwise, you will risk creating memory leaks, which is a really bad thing. On mobile devices, memory leaks will probably have the biggest negative impact due to their limited resources. Persisting data If you want to persist your data, you should use the Preferences class. It is merely a dictionary or a hash map data type which stores multiple key-value pairs in a file. Libgdx will create a new preferences file on the fly if it does not exist yet. You can have several preference files using unique names in order to split up data into categories. To get access to a preference file, you need to request a Preferences instance by its filename as follows: Preferences prefs = Gdx.app.getPreferences("settings.prefs"); To write a (new) value, you have to choose a key under which the value should be stored. If this key already exists in a preferences file, it will be overwritten. Do not forget to call flush() afterwards to persist the data, or else all the changes will be lost. prefs.putInteger("sound_volume", 100); // volume @ 100% prefs.flush(); Persisting data needs a lot more time than just modifying values in memory (without flushing). Therefore, it is always better to modify as many values as possible before a final flush() method is executed. To read back a certain value from a preferences file, you need to know the corresponding key. If this key does not exist, it will be set to the default value. You can optionally pass your own default value as the second argument (for example, in the following listing, 50 is for the default sound volume): int soundVolume = prefs.getInteger("sound_volume", 50); Querying the Android API Level On Android, you can query the Android API Level, which allows you to handle things differently for certain versions of the Android OS. Use the following listing to find out the version: Gdx.app.getVersion(); On platforms other than Android, the version returned is always 0. Querying the platform type You may want to write a platform-specific code where it is necessary to know the current platform type. The following example shows how it can be done: switch (Gdx.app.getType()) { case Desktop: // Code for Desktop application break; case Android: // Code for Android application break; case WebGL: // Code for WebGL application break; default: // Unhandled (new?) platform application break; } Querying memory usage You can query the system to find out its current memory footprint of your application. This may help you find excessive memory allocations that could lead to application crashes. The following functions return the amount of memory (in bytes) that is in use by the corresponding heap: long memUsageJavaHeap = Gdx.app.getJavaHeap(); long memUsageNativeHeap = Gdx.app.getNativeHeap(); Graphics module The graphics module can be accessed either through Gdx.getGraphics() or by using the shortcut variable Gdx.graphics. Querying delta time Query Libgdx for the time span between the current and the last frame in seconds by calling Gdx.graphics.getDeltaTime(). Querying display size Query the device's display size returned in pixels by calling Gdx.graphics.getWidth() and Gdx.graphics.getHeight(). Querying the FPS (frames per second) counter Query a built-in frame counter provided by Libgdx to find the average number of frames per second by calling Gdx.graphics.getFramesPerSecond(). Audio module The audio module can be accessed either through Gdx.getAudio() or by using the shortcut variable Gdx.audio. Sound playback To load sounds for playback, call Gdx.audio.newSound(). The supported file formats are WAV, MP3, and OGG. There is an upper limit of 1 MB for decoded audio data. Consider the sounds to be short effects like bullets or explosions so that the size limitation is not really an issue. Music streaming To stream music for playback, call Gdx.audio.newMusic(). The supported file formats are WAV, MP3, and OGG. Input module The input module can be accessed either through Gdx.getInput() or by using the shortcut variable Gdx.input. In order to receive and handle input properly, you should always implement the InputProcessor interface and set it as the global handler for input in Libgdx by calling Gdx.input.setInputProcessor(). Reading the keyboard/touch/mouse input Query the system for the last x or y coordinate in the screen coordinates where the screen origin is at the top-left corner by calling either Gdx.input.getX() or Gdx.input.getY(). To find out if the screen is touched either by a finger or by mouse, call Gdx.input.isTouched() To find out if the mouse button is pressed, call Gdx.input.isButtonPressed() To find out if the keyboard is pressed, call Gdx.input.isKeyPressed() Reading the accelerometer Query the accelerometer for its value on the x axis by calling Gdx.input.getAccelerometerX(). Replace the X in the method's name with Y or Z to query the other two axes. Be aware that there will be no accelerometer present on a desktop, so Libgdx always returns 0. Starting and canceling vibrator On Android, you can let the device vibrate by calling Gdx.input.vibrate(). A running vibration can be cancelled by calling Gdx.input.cancelVibrate(). Catching Android soft keys You might want to catch Android's soft keys to add an extra handling code for them. If you want to catch the back button, call Gdx.input.setCatchBackKey(true). If you want to catch the menu button, call Gdx.input.setCatchMenuKey(true). On a desktop where you have a mouse pointer, you can tell Libgdx to catch it so that you get a permanent mouse input without having the mouse ever leave the application window. To catch the mouse cursor, call Gdx.input.setCursorCatched(true). The files module The files module can be accessed either through Gdx.getFiles() or by using the shortcut variable Gdx.files. Getting an internal file handle You can get a file handle for an internal file by calling Gdx.files.internal(). An internal file is relative to the assets folder on the Android and WebGL platforms. On a desktop, it is relative to the root folder of the application. Getting an external file handle You can get a file handle for an external file by calling Gdx.files.external(). An external file is relative to the SD card on the Android platform. On a desktop, it is relative to the user's home folder. Note that this is not available for WebGL applications. The network module The network module can be accessed either through Gdx.getNet() or by using the shortcut variable Gdx.net. HTTP GET and HTTP POST You can make HTTP GET and POST requests by calling either Gdx.net.httpGet() or Gdx.net.httpPost(). Client/server sockets You can create client/server sockets by calling either Gdx.net.newClientSocket() or Gdx.net.newServerSocket(). Opening a URI in a web browser To open a Uniform Resource Identifier ( URI ) in the default web browser, call Gdx.net.openURI(URI). Libgdx's Application Life-Cycle and Interface The Application Life-Cycle in Libgdx is a well-defined set of distinct system states. The list of these states is pretty short: create, resize, render, pause, resume, and dispose. Libgdx defines an ApplicationListener interface that contains six methods, one for each system state. The following code listing is a copy that is directly taken from Libgdx's sources. For the sake of readability, all comments have been stripped. public interface ApplicationListener { public void create (); public void resize (int width, int height); public void render (); public void pause (); public void resume (); public void dispose (); } All you need to do is implement these methods in your main class of the shared game code project. Libgdx will then call each of these methods at the right time. The following diagram visualizes the Libgdx's Application Life-Cycle: Note that a full and dotted line basically has the same meaning in the preceding figure. They both connect two consecutive states and have a direction of flow indicated by a little arrowhead on one end of the line. A dotted line additionally denotes a system event. When an application starts, it will always begin with create(). This is where the initialization of the application should happen, such as loading assets into memory and creating an initial state of the game world. Subsequently, the next state that follows is resize(). This is the first opportunity for an application to adjust itself to the available display size (width and height) given in pixels. Next, Libgdx will handle system events. If no event has occurred in the meanwhile, it is assumed that the application is (still) running. The next state would be render(). This is where a game application will mainly do two things: Update the game world model Draw the scene on the screen using the updated game world model Afterwards, a decision is made upon which the platform type is detected by Libgdx. On a desktop or in a web browser, the displaying application window can be resized virtually at any time. Libgdx compares the last and current sizes on every cycle so that resize() is only called if the display size has changed. This makes sure that the running application is able to accommodate a changed display size. Now the cycle starts over by handling (new) system events once again. Another system event that can occur during runtime is the exit event. When it occurs, Libgdx will first change to the pause() state, which is a very good place to save any data that would be lost otherwise after the application has terminated. Subsequently, Libgdx changes to the dispose() state where an application should do its final clean-up to free all the resources that it is still using. This is also almost true for Android, except that pause() is an intermediate state that is not directly followed by a dispose() state at first. Be aware that this event may occur anytime during application runtime while the user has pressed the Home button or if there is an incoming phone call in the meanwhile. In fact, as long as the Android operating system does not need the occupied memory of the paused application, its state will not be changed to dispose(). Moreover, it is possible that a paused application might receive a resume system event, which in this case would change its state to resume(), and it would eventually arrive at the system event handler again. Starter Classes A Starter Class defines the entry point (starting point) of a Libgdx application. They are specifically written for a certain platform. Usually, these kinds of classes are very simple and mostly consist of not more than a few lines of code to set certain parameters that apply to the corresponding platform. Think of them as a kind of boot-up sequence for each platform. Once booting has finished, the Libgdx framework hands over control from the Starter Class (for example, the demo-desktop project) to your shared application code (for example, the demo project) by calling the different methods from the ApplicationListener interface that the MyDemo class implements. Remember that the MyDemo class is where the shared application code begins. We will now take a look at each of the Starter Classes that were generated during the project setup. Running the demo application on a desktop The Starter Class for the desktop application is called Main.java. The following listing is Main.java from demo-desktop : package com.packtpub.libgdx.demo; import com.badlogic.gdx.backends.lwjgl.LwjglApplication; import com.badlogic.gdx.backends.lwjgl.LwjglApplicationConfiguration; public class Main { public static void main(String[] args) { LwjglApplicationConfiguration cfg = new LwjglApplicationConfiguration(); cfg.title = "demo"; cfg.useGL20 = false; cfg.width = 480; cfg.height = 320; new LwjglApplication(new MyDemo(), cfg); } } In the preceding code listing, you see the Main class, a plain Java class without the need to implement an interface or inherit from another class. Instead, a new instance of the LwjglApplication class is created. This class provides a couple of overloaded constructors to choose from. Here, we pass a new instance of the MyDemo class as the first argument to the constructor. Optionally, an instance of the LwjglApplicationConfiguration class can be passed as the second argument. The configuration class allows you to set every parameter that is configurable for a Libgdx desktop application. In this case, the window title is set to demo and the window's width and height is set to 480 by 320 pixels. This is all you need to write and configure a Starter Class for a desktop. Let us try to run the application now. To do this, right-click on the demo-desktop project in Project Explorer in Eclipse and then navigate to Run As | Java Application. Eclipse may ask you to select the Main class when you do this for the first time. Simply select the Main class and also check that the correct package name ( com.packtpub.libgdx.demo ) is displayed next to it. The desktop application should now be up and running on your computer. If you are working on Windows, you should see a window that looks as follows: Summary In this article, we learned about Libgdx and how all the projects of an application work together. We covered Libgdx's backends, modules, and Starter Classes. Additionally, we covered what the Application Life Cycle and corresponding interface are, and how they are meant to work. Resources for Article: Further resources on this subject: Panda3D Game Development: Scene Effects and Shaders [Article] Microsoft XNA 4.0 Game Development: Receiving Player Input [Article] Introduction to Game Development Using Unity 3D [Article]
Read more
  • 0
  • 0
  • 5715
article-image-routes-and-model-binding-intermediate
Packt
01 Oct 2013
6 min read
Save for later

Routes and model binding (Intermediate)

Packt
01 Oct 2013
6 min read
(For more resources related to this topic, see here.) Getting ready This section builds on the previous section and assumes you have the TodoNancy and TodoNancyTests projects all set up. How to do it... The following steps will help you to handle the other HTTP verbs and work with dynamic routes: Open the TodoNancy Visual Studio solution. Add a new class to the NancyTodoTests project, call it TodosModulesTests, and fill this test code for a GET and a POST route into it: public class TodosModuleTests { private Browser sut; private Todo aTodo; private Todo anEditedTodo; public TodosModuleTests() { TodosModule.store.Clear(); sut = new Browser(new DefaultNancyBootstrapper()); aTodo = new Todo { title = "task 1", order = 0, completed = false }; anEditedTodo = new Todo() { id = 42, title = "edited name", order = 0, completed = false }; } [Fact] public void Should_return_empty_list_on_get_when_no_todos_have_been_posted() { var actual = sut.Get("/todos/"); Assert.Equal(HttpStatusCode.OK, actual.StatusCode); Assert.Empty(actual.Body.DeserializeJson<Todo[]>()); } [Fact] public void Should_return_201_create_when_a_todo_is_posted() { var actual = sut.Post("/todos/", with => with.JsonBody(aTodo)); Assert.Equal(HttpStatusCode.Created, actual.StatusCode); } [Fact] public void Should_not_accept_posting_to_with_duplicate_id() { var actual = sut.Post("/todos/", with => with.JsonBody(anEditedTodo)) .Then .Post("/todos/", with => with.JsonBody(anEditedTodo)); Assert.Equal(HttpStatusCode.NotAcceptable, actual.StatusCode); } [Fact] public void Should_be_able_to_get_posted_todo() { var actual = sut.Post("/todos/", with => with.JsonBody(aTodo) ) .Then .Get("/todos/"); var actualBody = actual.Body.DeserializeJson<Todo[]>(); Assert.Equal(1, actualBody.Length); AssertAreSame(aTodo, actualBody[0]); } private void AssertAreSame(Todo expected, Todo actual) { Assert.Equal(expected.title, actual.title); Assert.Equal(expected.order, actual.order); Assert.Equal(expected.completed, actual.completed); } } The main thing to notice new in these tests is the use of actual.Body.DesrializeJson<Todo[]>(), which takes the Body property of the BrowserResponse type, assumes it contains JSON formatted text, and then deserializes that string into an array of Todo objects. At the moment, these tests will not compile. To fix this, add this Todo class to the TodoNancy project as follows: public class Todo { public long id { get; set; } public string title { get; set; } public int order { get; set; } public bool completed { get; set; } } Then, go to the TodoNancy project, and add a new C# file, call it TodosModule, and add the following code to body of the new class: public static Dictionary<long, Todo> store = new Dictionary<long, Todo>(); Run the tests and watch them fail. Then add the following code to TodosModule: public TodosModule() : base("todos") { Get["/"] = _ => Response.AsJson(store.Values); Post["/"] = _ => { var newTodo = this.Bind<Todo>(); if (newTodo.id == 0) newTodo.id = store.Count + 1; if (store.ContainsKey(newTodo.id)) return HttpStatusCode.NotAcceptable; store.Add(newTodo.id, newTodo); return Response.AsJson(newTodo) .WithStatusCode(HttpStatusCode.Created); }; } The previous code adds two new handlers to our application. One handler for the GET "/todos/" HTTP and the other handler for the POST "/todos/" HTTP. The GET handler returns a list of todo items as a JSON array. The POST handler allows for creating new todos. Re-run the tests and watch them succeed. Now let's take a closer look at the code. Firstly, note how adding a handler for the POST HTTP is similar to adding handlers for the GET HTTP. This consistency extends to the other HTTP verbs too. Secondly, note that we pass the "todos"string to the base constructor. This tells Nancy that all routes in this module are related to /todos. Thirdly, notice the this.Bind<Todo>() call, which is Nancy's data binding in action; it deserializes the body of the POST HTTP into a Todo object. Now go back to the TodosModuleTests class and add these tests for the PUT and DELETE HTTP as follows: [Fact] public void Should_be_able_to_edit_todo_with_put() { var actual = sut.Post("/todos/", with => with.JsonBody(aTodo)) .Then .Put("/todos/1", with => with.JsonBody(anEditedTodo)) .Then .Get("/todos/"); var actualBody = actual.Body.DeserializeJson<Todo[]>(); Assert.Equal(1, actualBody.Length); AssertAreSame(anEditedTodo, actualBody[0]); } [Fact] public void Should_be_able_to_delete_todo_with_delete() { var actual = sut.Post("/todos/", with => with.Body(aTodo.ToJSON())) .Then .Delete("/todos/1") .Then .Get("/todos/"); Assert.Equal(HttpStatusCode.OK, actual.StatusCode); Assert.Empty(actual.Body.DeserializeJson<Todo[]>()); } After watching these tests fail, make them pass by adding this code to the constructor of TodosModule: Put["/{id}"] = p => { if (!store.ContainsKey(p.id)) return HttpStatusCode.NotFound; var updatedTodo = this.Bind<Todo>(); store[p.id] = updatedTodo; return Response.AsJson(updatedTodo); }; Delete["/{id}"] = p => { if (!store.ContainsKey(p.id)) return HttpStatusCode.NotFound; store.Remove(p.id); return HttpStatusCode.OK; }; All tests should now pass. Take a look at the routes to the new handlers for the PUT and DELETE HTTP. Both are defined as "/{id}". This will match any route that starts with /todos/ and then something more that appears after the trailing /, such as /todos/42 and the {id} part of the route definition is 42. Notice that both these new handlers use their p argument to get the ID from the route in the p.id expression. Nancy lets you define very flexible routes. You can use any regular expression to define a route. All named parts of such regular expressions are put into the argument for the handler. The type of this argument is DynamicDictionary, which is a special Nancy type that lets you look up parts via either indexers (for example, p["id"]) like a dictionary, or dot notation (for example, p.id) like other dynamic C# objects. There's more... In addition to the handlers for GET, POST, PUT, and DELETE, which we added in this recipe, we can go ahead and add handler for PATCH and OPTIONS by following the exact same pattern. Out of the box, Nancy automatically supports HEAD and OPTIONS for you. To handle the HEAD HTTP request, Nancy will run the corresponding GET handler but only return the headers. To handle OPTIONS, Nancy will inspect which routes you have defined and respond accordingly. Summary In this article we saw how to handle the other HTTP verbs apart from GET and how to work with dynamic routes. We will also saw how to work with JSON data and how to do model binding. Resources for Article: Further resources on this subject: Displaying MySQL data on an ASP.NET Web Page [Article] Layout with Ext.NET [Article] ASP.Net Site Performance: Speeding up Database Access [Article]
Read more
  • 0
  • 0
  • 1443

article-image-what-new-12c
Packt
30 Sep 2013
23 min read
Save for later

What is New in 12c

Packt
30 Sep 2013
23 min read
(For more resources related to this topic, see here.) Oracle Database 12c has introduced many new features and enhancements for backup and recovery. This article will introduce you to some of them and you will have the opportunity to learn in more detail how they could be used in real life situations. But I cannot start talking about Oracle 12 c without talking first about a revolutionary whole new concept that was introduced with this new version of the database product, called Multitenant Container Database( CDB ) that will contain two or more pluggable databases ( PDB ). When a container database only contains one PDB it is called Single Tenant Container Database. You can also have your database on Oracle 12c using the same format as before 12c, it will be called non-CDB database and will not allow the use of PDBs. Pluggable database We are now able to have multiple databases sharing a single instance and Oracle binaries. Each of the databases will be configurable to a degree and will allow some parameters to be set specifically for themselves (due that they will share the same initialization parameter file) and what is better, each database will be completely isolated from each other without either knowing that the other exists. A CDB is a single physical database that contains a root container with the main Oracle data dictionary and at least one PDB with specific application data. A PDB is a portable container with its own data dictionary, including metadata and internal links to the system-supplied objects in the root container, and this PDB will appear to an Oracle Net client as a traditional Oracle database. The CDB also contains a PDB called SEED, which is used as a template when an empty PDB needs to be created. The following figure shows an example of a CDB with five PDBs: When creating a database on Oracle 12 c , you can now create a CDB with one or more PDBs, and what is even better is that you can easily clone a PDB, or unplug it and plug it into a different server with a preinstalled CDB, if your target server is running out of resources such as CPU or memory. Many years ago, the introduction of external storage gave us the possibility to store data on external devices and the flexibility to plug and unplug them to any system independent of their OS. For example, you can connect an external device to a system using Windows XP and read your data without any problems. Later you can unplug it and connect it to a laptop running Windows 7 and you will still be able to read your data. Now with the introduction of Oracle pluggable databases, we will be able to do something similar with Oracle when upgrading a PDB, making this process simple and easy. All you will need to do to upgrade a PDB, as per example, is: Unplug your PDB (step 1 in the following figure) that is using a CDB running 12.1.0.1. Copy the PDB to the destination location with a CDB that is using a later version such as 12.2.0.1 (step 2 in the following figure). Plug the PDB to the CDB (step 3 in the following figure), and your PDB is now upgraded to 12.2.0.1. This new concept is a great solution for database consolidation and is very useful for multitenant SaaS (Software as a Service) providers, improving resource utilization, manageability, integration, and service management. Some key points about pluggable databases are: You can have many PDBs if you want inside a single container (a CDB can contain a maximum of 253 PDBs) A PDB is fully backwards compatible with an ordinary pre-12.1 database in an applications perspective, meaning that an application built for example to run on Oracle 11.1 will have no need to be changed to run on Oracle 12c A system administrator can connect to a CDB as a whole and see a single system image If you are not ready to make use of this new concept, you can still be able to create a database on Oracle 12c as before, called non-CDB (non-Container Database) Each instance in RAC opens the CDB as a whole. A foreground session will see only the single PDB it is connected to and sees it just as a non-CDB The Resource Manager is extended with some new between-PDB capabilities Fully integrated with Oracle Enterprise Manager 12c and SQL Developer Fast provisioning of new databases (empty or as a copy/clone of an existing PDB) On Clone triggers can be used to scrub or mask data during a clone process Fast unplug and plug between CDBs Fast path or upgrade by unplugging a PDB and plugging it into a different CDB already patched or with a later database version Separation of duties between DBA and application administrators Communication between PDBs is allowed via intra-CDB dblinks Every PDB has a default service with its name in one Listener An unplugged PDB carries its lineage, Opatch, encryption key info, and much more All PDBs in a CDB should use the same character set All PDBs share the same control files, SPFILE, redo log files, flashback log files, and undo Flashback PDB is not available on 12.1, it expected to be available with 12.2 Allows multitenancy of Oracle Databases, very useful for centralization, especially if using Exadata Multitenant Container Database is only available for Oracle Enterprise Edition as a payable option, all other editions of the Oracle database can only deploy non-CDB or Single Tenant Pluggable databases. RMAN new features and enhancements Now we can continue and take a fast and closer look at some of the new features and enhancements introduced in this database version for RMAN. Container and pluggable database backup and restore As we saw earlier, the introduction of Oracle 12c and the new pluggable database concept made it possible to easily centralize multiple databases maintaining the individuality of each one when using a single instance. The introduction of this new concept also forced Oracle to introduce some new enhancements to the already existent BACKUP, RESTORE, and RECOVERY commands to enable us to be able to make an efficient backup or restore of the complete CDB. This includes all PDBs or just one of more PDBs, or if you want to be more specific, you can also just backup or restore one or more tablespaces from a PDB. Some examples of how to use the RMAN commands when performing a backup on Oracle 12c are: RMAN> BACKUP DATABASE; (To backup the CBD + all PDBs) RMAN> BACKUP DATABASE root; (To backup only the CBD) RMAN> BACKUP PLUGGABLE DATABASE pdb1,pdb2; (To backup all specified PDBs) RMAN> BACKUP TABLESPACE pdb1:example; (To backup a specific tablespace in a PDB) Some examples when performing RESTORE operations are: RMAN> RESTORE DATABASE; (To restore an entire CDB, including all PDBs) RMAN> RESTORE DATABASE root; (To restore only the root container) RMAN> RESTORE PLUGGABLE DATABASE pdb1; (To restore a specific PDB) RMAN> RESTORE TABLESPACE pdb1:example; (To restore a tablespace in a PDB) Finally, some example of RECOVERY operations are: RMAN> RECOVER DATABASE; (Root plus all PDBs) RMAN> RUN { SET UNTIL SCN 1428; RESTORE DATABASE; RECOVER DATABASE; ALTER DATABASE OPEN RESETLOGS; } RMAN> RUN } RESTORE PLUGGABLE DATABASE pdb1 TO RESTORE POINT one; RECOVER PLUGGABLE DATABASE pdb1 TO RESTORE POINT one; ALTER PLUGGABLE DATABASE pdb1 OPEN RESETLOGS;} Enterprise Manager Database Express The Oracle Enterprise Manager Database Console or Database Control that many of us used to manage an entire database is now deprecated and replaced by the new Oracle Enterprise Manager Database Express. This new tool uses Flash technology and allows the DBA to easily manage the configurations, storage, security, and performance of a database. Note that RMAN, Data Pump, and the Oracle Enterprise Manager Cloud Control are now the only tools able to perform backup and recovery operations in a pluggable database environment, in other words, you cannot use the Enterprise Manager Database Express for database backup/recovery operations. Backup privileges Oracle Database 12c provides separation support for the separation of DBA duties for the Oracle Database by introducing task-specific and least privileged administrative privileges for backups that do not require the SYSDBA privilege. The new system privilege introduced with this new release is SYSBACKUP. Avoid the use of the SYSDBA privilege for backups unless it is strictly necessary. When connecting to the database using the AS SYSDBA system privilege, you are able to see any object structure and all the data within the object, whereas if you are connecting using the new system privilege AS SYSBACKUP, you will still be able to see the structure of an object but not the object data. If you try to see any data using the SYSBACKUP privilege, the ORA-01031: insufficient privileges message will be raised. Tighter security policies require a separation of duties. The new SYSBACKUP privilege facilitates the implementation of the separation of duties, allowing backup and recovery operations to be performed without implicit access to the data, so if access to the data is required for one specific user, it will need to be granted explicitly to this user. RMAN has introduced some changes when connecting to a database such as: TARGET: It will require the user to have the SYSBACKUP administrative privilege to be able to connect to the TARGET database CATALOG: As in the earlier versions a user was required to have the RECOVERY_CATALOG_OWNER role assigned to be able to connect to the RMAN catalog, now it will need to have assigned the SYSBACKUP privilege to be able to connect to the catalog AUXILIARY: It will require the SYSBACKUP administrative privilege to connect to the AUXILIARY database Some important points about the SYSBACKUP administrative privilege are: It includes permissions for backup and recovery operations It does not include data access privileges such as SELECT ANY TABLE that the SYSDBA privilege has It can be granted to the SYSBACKUP user that is created during the database installation process It's the default privilege when a RMAN connection string is issued and does not contain the AS SYSBACKUP clause: $ RMAN TARGET / Before connecting as the SYSBACKUP user created during the database creation process, you will need to unlock the account and grant the SYSBACKUP privilege to the user. When you use the GRANT command to give the SYSBACKUP privilege to a user, the username and privilege information will be automatically added to the database password file. The v$pwfile_users view contains all information regarding users within the database password file and indicates whether a user has been granted any privileged system privilege. Let's take a closer look to this view: SQL> DESC v$pwfile_users Name Null? Type ----------------------------- -------- ----------------- USERNAME VARCHAR2(30) SYSDBA VARCHAR2(5) SYSOPER VARCHAR2(5) SYSASM VARCHAR2(5) SYSBACKUP VARCHAR2(5) SYSDG VARCHAR2(5) SYSKM VARCHAR2(5) CON_ID NUMBER As you can see, this view now contains some new columns, such as: SYSBACKUP: It indicates if the user is able to connect using the SYSBACKUP privileges SYSDG: It indicates if the user is able to connect using the SYSDG (new for Data Guard) privileges SYSKM: It indicates if the user is able to connect using the SYSKM (new for Advanced Security) privileges. CON_ID: It is the ID of the current container. If 0, it will indicate that it is related to the entire CDB or to an entire traditional database (non-CDB): if the value is 1, then this user has the access only to root; if other value, then the view will identify a specific container ID. To help you clearly understand the use of the SYSBACKUP privilege, let's run a few examples to make it completely clear. Let's connect to our newly created database as SYSDBA and take a closer look at the SYSBACKUP privilege: $ sqlplus / as sysdbaSQL> SET PAGES 999SQL> SET LINES 99SQL> COL USERNAME FORMAT A21SQL> COL ACCOUNT_STATUS FORMAT A20SQL> COL LAST_LOGIN FORMAT A41 SQL> SELECT username, account_status, last_login 2 FROM dba_users 3 WHERE username = 'SYSBACKUP';USERNAME ACCOUNT_STATUS LAST_LOGIN------------ -------------------- -----------------------SYSBACKUP EXPIRED & LOCKED As you can see, the SYSBACKUP account created during the database creation is currently EXPIRED & LOCKED, you will need to unlock this account and grant the SYSBACKUP privilege to it if you want to use this user for any backup and recovery purposes: For this demo I will use the original SYSBACKUP account, but in a production environment never use the SYSBACKUP account, instead grant the SYSBACKUP privilege to the user(s) that will be responsible for the backup and recovery operations. SQL> ALTER USER sysbackup IDENTIFIED BY "demo" ACCOUNT UNLOCK; User altered. SQL> GRANT sysbackup TO sysbackup; Grant succeeded. SQL> SQL> SELECT username, account_status 2 FROM dba_users 3 WHERE account_status NOT LIKE '%LOCKED'; USERNAME ACCOUNT_STATUS --------------------- -------------------- SYS OPEN SYSTEM OPEN SYSBACKUP OPEN We can also easily identify what system privileges and roles are assigned to SYSBACKUP by executing the following SQLs: SQL> COL grantee FORMAT A20 SQL> SELECT * 2 FROM dba_sys_privs 3 WHERE grantee = 'SYSBACKUP'; GRANTEE PRIVILEGE ADM COM ------------- ----------------------------------- --- --- SYSBACKUP ALTER SYSTEM NO YES SYSBACKUP AUDIT ANY NO YES SYSBACKUP SELECT ANY TRANSACTION NO YES SYSBACKUP SELECT ANY DICTIONARY NO YES SYSBACKUP RESUMABLE NO YES SYSBACKUP CREATE ANY DIRECTORY NO YES SYSBACKUP UNLIMITED TABLESPACE NO YES SYSBACKUP ALTER TABLESPACE NO YES SYSBACKUP ALTER SESSION NO YES SYSBACKUP ALTER DATABASE NO YES SYSBACKUP CREATE ANY TABLE NO YES SYSBACKUP DROP TABLESPACE NO YES SYSBACKUP CREATE ANY CLUSTER NO YES 13 rows selected. SQL> COL granted_role FORMAT A30 SQL> SELECT * 2 FROM dba_role_privs 3 WHERE grantee = 'SYSBACKUP'; GRANTEE GRANTED_ROLE ADM DEF COM -------------- ------------------------------ --- --- --- SYSBACKUP SELECT_CATALOG_ROLE NO YES YES Where the column ADMIN_OPTION refers to if the user has or not, the ADMIN_OPTION privilege, the column DEFAULT_ROLE indicates whether or not ROLE is designated as a default role for the user, and the column COMMON refers to if it's common to all the containers and pluggable databases available. SQL and DESCRIBE As you know well, you are able to execute the SQL commands, and the PL/SQL procedures from the RMAN command line starting with Oracle 12.1, do not require the use of the SQL prefix or quotes for most SQL commands in RMAN. You can now run some simple SQL commands in RMAN such as: RMAN> SELECT TO_CHAR(sysdate,'dd/mm/yy - hh24:mi:ss') 2> FROM dual; TO_CHAR(SYSDATE,'DD) ------------------- 17/09/12 - 02:58:40 RMAN> DESC v$datafile Name Null? Type --------------------------- -------- ------------------- FILE# NUMBER CREATION_CHANGE# NUMBER CREATION_TIME DATE TS# NUMBER RFILE# NUMBER STATUS VARCHAR2(7) ENABLED VARCHAR2(10) CHECKPOINT_CHANGE# NUMBER CHECKPOINT_TIME DATE UNRECOVERABLE_CHANGE# NUMBER UNRECOVERABLE_TIME DATE LAST_CHANGE# NUMBER LAST_TIME DATE OFFLINE_CHANGE# NUMBER ONLINE_CHANGE# NUMBER ONLINE_TIME DATE BYTES NUMBER BLOCKS NUMBER CREATE_BYTES NUMBER BLOCK_SIZE NUMBER NAME VARCHAR2(513) PLUGGED_IN NUMBER BLOCK1_OFFSET NUMBER AUX_NAME VARCHAR2(513) FIRST_NONLOGGED_SCN NUMBER FIRST_NONLOGGED_TIME DATE FOREIGN_DBID NUMBER FOREIGN_CREATION_CHANGE# NUMBER FOREIGN_CREATION_TIME DATE PLUGGED_READONLY VARCHAR2(3) PLUGIN_CHANGE# NUMBER PLUGIN_RESETLOGS_CHANGE# NUMBER PLUGIN_RESETLOGS_TIME DATE CON_ID NUMBER RMAN> ALTER TABLESPACE users 2> ADD DATAFILE '/u01/app/oracle/oradata/cdb1/pdb1/user02.dbf' size 50M; Statement processed Remember that the SYSBACKUP privilege does not grant access to the user tables or views, but the SYSDBA privilege does. Multi-section backups for incremental backups Oracle Database 11g introduced multi-section backups to allow us to backup and restore very large files using backup sets (remember that Oracle datafiles can be up to 128 TB in size). Now with Oracle Database 12c , we are able to make use of image copies when creating multi-section backups as a complement of the previous backup set functionality. This helps us to reduce image copy creation time for backups, transporting tablespaces, cloning, and doing a TSPITR (tablespace point-in-time recovery), it also improves backups when using Exadata. The main restrictions to make use of this enhancement are: The COMPATIBLE initialization parameter needs to be set to 12.0 or higher to make use of the new image copy multi-section backup feature This is only available for datafiles and cannot be used to backup control or password files Not to be used with a large number of parallelisms when a file resides on a small number of disks, to avoid each process to compete with each other when accessing the same device Another new feature introduced with multi-section backups is the ability to create multi-section backups for incremental backups. This will allow RMAN to only backup the data that has changed since the last backup, consequently enhancing the performance of multi-section backups due that they are processed independently, either serially or in parallel. Network-based recovery Restoring and recovering files over the network is supported starting with Oracle Database 12c . We can now recover a standby database and synchronize it with its primary database via the network without the need to ship the archive log files. When the RECOVER command is executed, an incremental backup is created on the primary database. It is then transferred over the network to the physical standby database and applied to the standby database to synchronize it within the primary database. RMAN uses the SCN from the standby datafile header and creates the incremental backup starting from this SCN on the primary database, in other words, only bringing the information necessary to the synchronization process. If block change tracking is enabled for the primary database, it will be used while creating the incremental backup making it faster. A network-based recovery can also be used to replace any missing datafiles, control files, SPFILE, or tablespaces on the primary database using the corresponding entity from the physical standby to the recovery operation. You can also use multi-section backup sets, encryption, or even compression within a network-based recovery. Active Duplicate The Active Duplicate feature generates an online backup on the TARGET database and directly transmits it via an inter-instance network connection to the AUXILIARY database for duplication (not written to disk in the source server). Consequently, this reduces the impact on the TARGET database by offloading the data transfer operation to the AUXILIARY database, also reducing the duplication time. This very useful feature has now received some important enhancements. In Oracle 11 g when this feature was initially introduced, it only allowed us to use a push process based on the image copies. Now it allows us to make use of the already known push process or to make use of the newly introduced pull process from the AUXILIARY database that is based on backup sets (the pull process is now the new default and automatically copies across all datafiles, control files, SPFILE and archive log files). Then it performs the restore of all files and uses a memory script to complete the recovery operation and open the AUXILIARY database. RMAN will dynamically determine, based on your DUPLICATE clauses, which process will be used (push or pull). It is very possible that soon Oracle will end deprecating the push process on the future releases of the database. You can now choose your choice of compression, section size, and encryption to be used during the Active Duplication process. For example, if you specify the SET ENCRYPTION option before the DUPLICATE command, all the backups sent from the target to the auxiliary database will be encrypted. For an effective use of parallelism, allocate more AUXILIARY channels instead of TARGET channels as in the earlier releases. Finally, another important new enhancement is the possibility to finish the duplication process with the AUXILIARY database in not open state (the default is to open the AUXILIARY database after the duplication is completed). This option is very useful when you are required to: Modify the block change tracking Configure fast incremental backups or flashback database settings Move the location of the database, for example, to ASM Upgrade the AUXILIARY database (due that the database must not be open with reset logs prior to applying the upgrade scripts) Or when you know that the attempt to open the database would produce errors To make it clearer, let's take a closer look at what operations RMAN will perform when a DUPLICATE command is used: Create an SPFILE string for the AUXILIARY instance. Mount the backup control file. Restore the TARGET datafiles on the AUXILIARY database. Perform incomplete recovery using all the available incremental backups and archived redo log files. Shut down and restart the AUXILIARY instance in the NOMOUNT mode. Create a new control file, create and store the new database ID in the datafiles (it will not happen if the FOR STANDBY clause is in use). Mount and opens the duplicate database using the RESETLOGS option, and create the online redo log files by default. If the NOOPEN option is used, the duplicated database will not be opened with RESETLOGS and will remain in the MOUNT state. Here are some examples of how to use the DUPLICATE command with PDBs: RMAN> DUPLICATE TARGET DATABASE TO <CDB1>; RMAN> DUPLICATE TARGET DATABASE TO <CDB1> PLUGGABLE DATABASE <PDB1>, <PDB2>, <PDB3>; Support for the third-party snapshot In the past when using a third-party snapshot technology to make a backup or clone of a database, you were forced to change the database to the backup mode (BEGIN BACKUP) before executing the storage snapshot. This requirement is no longer necessary if the following conditions are met: The database crash is consistent at the point of the snapshot Write ordering is preserved for each file within the snapshot The snapshot stores the time at which the snapshot is completed If a storage vendor cannot guarantee compliance with the conditions discussed, then you must place your database in backup mode before starting with the snapshot. The RECOVER command now has a newly introduced option called SNAPSHOT TIME that allows RMAN to recover a snapshot that was taken without being in backup mode to a consistent point-in-time. Some examples of how to use this new option are: RMAN> RECOVER DATABASE UNTIL TIME '10/12/2012 10:30:00' SNAPSHOT TIME '10/12/2012 10:00:00'; RMAN> RECOVER DATABASE UNTIL CANCEL SNAPSHOT TIME '10/12/2012 10:00:00'; Only trust your backups after you ensure that they are usable for recovery. In other words, always test your backup methodology first, ensuring that it can be used in the future in case of a disaster. Cross-platform data transport Starting with Oracle 12c, transporting data across platforms can be done making use of backup sets and also create cross-platform inconsistent tablespace backups (when the tablespace is not in the read-only mode) using image copies and backup sets. When using backup sets, you are able to make use of the compression and multi-section options, reducing downtime for the tablespace and the database platform migrations. RMAN does not catalog backup sets created for cross-platform transport in the control file, and always takes into consideration the endian format of the platforms and the database open mode. Before creating a backup set that will be used for a cross-platform data transport, the following prerequisites should be met: The compatible parameter in the SPFILE string should be 12.0 or greater The source database must be open in read-only mode when transporting an entire database due that the SYS and SYSAUX tablespaces will participate in the transport process If using Data Pump, the database must be open in read-write mode You can easily check the current compatible value and open_mode of your database by running the following SQL commands: SQL> SHOW PARAMETER compatible NAME TYPE VALUE ---------------------- ----------- ---------------------- compatible string 12.0.0.0.0 SQL> SELECT open_mode FROM v$database; OPEN_MODE -------------------- READ WRITE When making use of the FOR TRANSPORT or the TO PLATFORM clauses in the BACKUP command, you cannot make use of the following clauses: CUMULATIVE forRecoveryOfSpec INCREMENTAL LEVEL n keepOption notBackedUpSpec PROXY SECTION SIZE TAG VALIDATE Table recovery In previous versions of Oracle Database, the process to recover a table to a specific point-in-time was never easy. Oracle has now solved this major issue by introducing the possibility to do a point-in-time recovery of a table, group of tables or even table partitions without affecting the remaining database objects using RMAN. This makes the process easier and faster than ever before. Remember that Oracle has previously introduced features such as database point-in-time recovery ( DBPITR ), tablespace point-in-time recovery ( TSPITR ) and Flashback database; this is an evolution of the same technology and principles. The recovery of tables and table partitions is useful in the following situations: To recover a very small set of tables to a particular point-in-time To recover a tablespace that is not self-contained to a particular point-in-time, remember that TSPITR can only be used if the tablespace is self-contained To recover tables that are corrupted or dropped with the PURGE option, so the FLASHBACK DROP functionality is not possible to be used When logging for a Flashback table is enabled but the flashback target time or SCN is beyond the available undo To recover data that was lost after a data definition language ( DDL ) operation that changed the structure of a table To recover tables and table partitions from a RMAN backup, the TARGET database should be (prerequisites): At the READ/WRITE mode In the ARCHIVELOG mode The COMPATIBLE parameter should be set to 12.0 or higher You cannot recover tables or table partitions from the SYS, SYSTEM and SYSAUX schemas, or even from a standby database. Now let's take a closer look at the steps to do a table or table partitions recovery using RMAN: First check if all the prerequisites to do a table recovery are met. Start a RMAN session with the CONNECT TARGET command. Use the RECOVER TABLE command with all the required clauses. RMAN will determine which backup contains the data that needs to be recovered based on the point-in-time specified. RMAN creates an AUXILIARY instance, you can also specify the location of the AUXILIARY instance files using the AUXILIARY DESTINATION or SET NEWNAME clause. RMAN recovers the specified objects into the AUXILIARY instance. RMAN creates a Data Pump export dump file that contains the objects. RMAN imports the recovered objects from the dump file previously created into the TARGET database. If you want to manually import the objects to the TARGET database, you can make use of the NOTABLEIMPORT clause in the RECOVER command to achieve this goal. RMAN optionally offers the possibility to rename the recovered objects in the TARGET database using the REMAP TABLE clause, or to import the recovered objects to a different tablespace using the REMAP TABLESPACE clause. An example of how to use the new RECOVER TABLE command is: RMAN> RECOVER TABLE SCOTT.test UNTIL SEQUENCE 5481 THREAD 2 AUXILARY DESTINATION '/tmp/recover' REMAP TABLE SCOTT.test:my_test;
Read more
  • 0
  • 0
  • 3755
Modal Close icon
Modal Close icon