Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials - Programming

1081 Articles
article-image-resource-manager-centos-6
Packt
27 Apr 2015
19 min read
Save for later

Resource Manager on CentOS 6

Packt
27 Apr 2015
19 min read
In this article is written by Mitja Resman, author of the book CentOS High Availability, we will learn cluster resource management on CentOS 6 with the RGManager cluster resource manager. We will learn how and where to find the information you require about the cluster resources that are supported by RGManager, and all the details about cluster resource configuration. We will also learn how to add, delete, and reconfigure resources and services in your cluster. Then we will learn how to start, stop, and migrate resources from one cluster node to another. When we are done with this article, your cluster will be configured to run and provide end users with a service. (For more resources related to this topic, see here.) Working with RGManager When we work with RGManager, the cluster resources are configured within the /etc/cluster/cluster.conf CMAN configuration file. RGManager has a dedicated section in the CMAN configuration file defined by the <rm> tag. Part of configuration within the <rm> tag belongs to RGManager. The RGManager section begins with the <rm> tag and ends with the </rm> tag. This syntax is common for XML files. The RGManager section must be defined within the <cluster> section of the CMAN configuration file but not within the <clusternodes> or <fencedevices> sections. We will be able to review the exact configuration syntax from the example configuration file provided in the next paragraphs. The following elements can be used within the <rm> RGManager tag: Failover Domain: (tag: <failoverdomains></failoverdomains>): A failover domain is a set of cluster nodes that are eligible to run a specific cluster service in the event of a cluster node failure. More than one failure domain can be configured with different rules applied for different cluster services. Global Resources: (tag: <resources></resources>): Global cluster resources are globally configured resources that can be related when configuring cluster services. Global cluster resources simplify the process of cluster service configuration by global resource name reference. Cluster Service: (tag: <service></service>): A cluster service usually defines more than one resource combined to provide a cluster service. The order of resources provided within a cluster service is important because it defines the resource start and stop order. The most used and important RGManager command-line expressions are as follows: clustat: The clustat command provides cluster status information. It also provides information about the cluster, cluster nodes, and cluster services. clusvcadm: The clusvcadm command provides cluster service management commands such as start, stop, disable, enable, relocate, and others. By default, RGManager logging is configured to log information related to RGManager to the syslog/var/log/messages file. If the logfile parameter in the Corosync configuration file's logging section is configured, information related to RGManager will be logged in the location specified by the logfile parameter. The default RGManager log file is named rgmanager.log. Let's start with the details of RGManager configuration. Configuring failover domains The <rm> tag in the CMAN configuration file usually begins with the definition of a failover domain, but configuring a failover domain is not required for normal operation of the cluster. A failover domain is a set of cluster nodes with configured failover rules. The failover domain is attached to the cluster service configuration; in the event of a cluster node failure, the configured cluster service's failover domain rules are applied. Failover domains are configured within the <rm> RGManager tag. The failover domain configuration begins with the <failoverdomains> tag and ends with the </failoverdomains> tag. Within the <failoverdomains> tag, you can specify one or more failover domains in the following form: <failoverdomain failoverdomainname failoverdomain_options> </failoverdomain> The failoverdomainname parameter is a unique name provided for the failover domain in the form of name="desired_name". The failoverdomain_options options are the rules that we apply to the failover domain. The following rules can be configured for a failover domain: Unrestricted: (parameter: restricted="0"): This failover domain configuration allows you to run a cluster service on any of the configured cluster nodes. Restricted: (parameter: restricted="1"): This failover domain configuration allows you to restrict a cluster service to run on the members you configure. Ordered: (parameter: ordered="1"): This failover domain configuration allows you to configure a preference order for cluster nodes. In the event of cluster node failure, the preference order is taken into account. The order of the listed cluster nodes is important because it is also the priority order. Unordered: (parameter: ordered="0"): This failover domain configuration allows any of the configured cluster nodes to run a specific cluster service. Failback: (parameter: nofailback="0"): This failover domain configuration allows you to configure failback for the cluster service. This means the cluster service will fail back to the originating cluster node once the cluster node is operational. Nofailback: (parameter: nofailback="1"): This failover domain configuration allows you to disable the failback of the cluster service back to the originating cluster node once it is operational. Within the <failoverdomain> tag, the desired cluster nodes are configured with a <failoverdomainnode> tag in the following form: <failoverdomainnode nodename/> The nodename parameter is the cluster node name as provided in the <clusternode> tag of the CMAN configuration file. You can add the following simple failover domain configuration to your existing CMAN configuration file. In the following screenshot, you can see the CMAN configuration file with a simple failover domain configuration. The previous example shows a failover domain named simple with no failback, no ordering, and no restrictions configured. All three cluster nodes are listed as failover domain nodes. Note that it is important to change the config_version parameter in the second line on every CMAN cluster configuration file. Once you have configured the failover domain, you need to validate the cluster configuration file. A valid CMAN configuration is required for normal operation of the cluster. If the validation of the cluster configuration file fails, recheck the configuration file for common typo errors. In the following screenshot, you can see the command used to check the CMAN configuration file for errors: Note that, if a specific cluster node is not online, the configuration file will have to be transferred manually and the cluster stack software will have to be restarted to catch up once it comes back online. Once your configuration is validated, you can propagate it to other cluster nodes. In this screenshot, you can see the CMAN configuration file propagation command used on the node-1 cluster node: For successful CMAN configuration file distribution to the other cluster nodes, the CMAN configuration file's config_version parameter number must be increased. You can confirm that the configuration file was successfully distributed by issuing the ccs_config_dump command on any of the other cluster nodes and comparing the XML output. Adding cluster resources and services The difference between cluster resources and cluster services is that a cluster service is a service built from one or more cluster resources. A configured cluster resource is prepared to be used within a cluster service. When you are configuring a cluster service, you reference a configured cluster resource by its unique name. Resources Cluster resources are defined within the <rm> RGManager tag of the CMAN configuration file. They begin with the <resources> tag and end with the </resources> tag. Within the <resources> tag, all cluster resources supported by RGManager can be configured. Cluster resources are configured with resource scripts, and all RGManager-supported resource scripts are located in the /usr/share/cluster directory along with the cluster resource metadata information required to configure a cluster resource. For some cluster resources, the metadata information is listed within the cluster resource scripts, while others have separate cluster resource metadata files. RGManager reads metadata from the scripts while validating the CMAN configuration file. Therefore, knowing the metadata information is the best way to correctly define and configure a cluster resource. The basic syntax used to configure a cluster resource is as follows: <resource_agent_name resource_options"/> The resource_agent_name parameter is provided in the cluster resource metadata information and is defined as name. The resource_options option is cluster resource-configurable options as provided in the cluster resource metadata information. If you want to configure an IP address cluster resource, you should first review the IP address of the cluster resource metadata information, which is available in the /usr/share/cluster/ip.sh script file. The syntax used to define an IP address cluster resource is as follows: <ip ip_address_options/> We can configure a simple IPv4 IP address, such as 192.168.88.50, and bind it to the eth1 network interface by adding the following line to the CMAN configuration: <ip address="192.168.88.50" family="IPv4" prefer_interface="eth1"/> The address option is the IP address you want to configure. The family option is the address protocol family. The prefer_interface option binds the IP address to the specific network interface. By reviewing the IP address of resource metadata information we can see that a few additional options are configurable and well explained: monitor_link nfslock sleeptime disable_rdisc If you want to configure an Apache web server cluster resource, you should first review the Apache web server resource's metadata information in the /usr/share/cluster/apache.metadata metadata file. The syntax used to define an Apache web server cluster resource is as follows: <apache apache_web_server_options/> We can configure a simple Apache web server cluster resource by adding the following line to the CMAN configuration file: <apache name="apache" server_root="/etc/httpd" config_file="conf/httpd.conf" shutdown_wait="60"/> The name parameter is the unique name provided for the apache cluster resource. The server_root option provides the Apache installation location. If no server_root option is provided, the default value is /etc/httpd. The config_file option is the path to the main Apache web server configuration file from the server_root file. If no config_file option is provided, the default value is conf/httpd.conf. The shutdown_wait option is the number of seconds to wait before the correct end-of-service shutdown. By reviewing the Apache web server resource metadata, you can see that a few additional options are configurable and well explained: httpd httpd_options service_name We can add the IP address and Apache web server cluster resources to the example configuration we are building, as follows. <resources> <ip address="192.168.10.50" family="IPv4"   prefer_interface="eth1"/> <apache name="apache" server_root="/etc/httpd"   config_file="conf/httpd.conf" shutdown_wait="60"/> </resources> Do not forget to increase the config_version parameter number. Make sure that you the validate cluster configuration file with every change. In the following screenshot, you can see the command used to validate the CMAN configuration: After we've validated our configuration, we can distribute the cluster configuration file to other nodes. In this screenshot, you can see the command used to distribute the CMAN configuration file from the node-1 cluster node to other cluster nodes: Services The cluster services are defined within the <rm> RGManager tag of the CMAN configuration file after the cluster resources tag. They begin with the <service> tag and end with the </service> tag. The syntax used to define a service is as follows: <service service_options> </service> The resources within the cluster services are referenced to the globally configured cluster resources. The order of the cluster resources configured within the cluster service is important. This is because it is also a resource start order. The syntax for cluster resource configuration within the cluster service is as follows: <service service_options> <resource_agent_name ref="referenced_cluster_resource_name"/> </service> The service options can be the following: Autostart: (parameter: autostart="1"): This parameter starts services when RGManager starts. By default, RGManager starts all services when it is started and Quorum is present. Noautostart (parameter: autostart="0"): This parameter disables the start of all services when RGManager starts. Restart recovery (parameter: recovery="restart"): This is RGManager's default recovery policy. On failure, RGManager will restart the service on the same cluster node. If the service restart fails, RGManager will relocate the service to another operational cluster node. Relocate recovery (parameter: recovery="relocate"): On failure, RGManager will try to start the service on other operational cluster nodes. Disable recovery (parameter: recovery="disable"): On failure RGManager, will place the service in the disabled state. Restart disable recovery (parameter: recovery="restart-disable"): On failure, RGManager will try to restart the service on the same cluster node. If the restart fails, it will place the service in the disabled state. Additional restart policy extensions are available, as follows: Maximum restarts (parameter: max_restarts="N"; where N is the desired integer value): the maximum restarts parameter is defined by an integer that specifies the maximum number of service restarts before taking additional recovery policy actions Restart expire time (parameter: restart_expire_time="N"; where N is the desired integer value in seconds): The restart expire time parameter is defined by an integer value in seconds, and configures the time to remember a restart event We can configure a web server cluster service with respect to the configured IP address and Apache web server resources with the following CMAN configuration file syntax: <service name="webserver" autostart="1" recovery="relocate"> <ip ref="192.168.88.50"/> <apache ref="apache"/> </service> A minimal configuration of a web server cluster service requires a cluster IP address and an Apache web server resource. The name parameter defines a unique name for the web server cluster service. The autostart parameter defines an automatic start of the webserver cluster service on RGManager startup. The recovery parameter configures the restart of the web server cluster service on other cluster nodes in the event of failure. We can add the web server cluster service to the example CMAN configuration file we are building, as follows. <resources> <ip address="192.168.10.50" family="IPv4"   prefer_interface="eth1"/> <apache name="apache" server_root="/etc/httpd"   config_file="conf/httpd.conf" shutdown_wait="60"/> </resources> <service name="webserver" autostart="1" recovery="relocate"> <ip ref="192.168.10.50"/> <apache ref="apache"/> </service> Do not forget to increase the config_version parameter. Make sure you validate the cluster configuration file with every change. In the following screenshot, we can see the command used to validate the CMAN configuration: After you've validated your configuration, you can distribute the cluster configuration file to other nodes. In this screenshot, we can see the command used to distribute the CMAN configuration file from the node-1 cluster node to other cluster nodes: With the final distribution of the cluster configuration, a cluster service is configured and RGManager starts the cluster service called webserver. You can use the clustat command to check whether the web server cluster service was successfully started and also which cluster node it is running on. In the following screenshot, you can see the clustat command issued on the node-1 cluster node: Let's take a look at the following terms: Service Name: This column defines the name of the service as configured in the CMAN configuration file. Owner: This column lists the node the service is running on or was last running on. State: This column provides information about the status of the service. Managing cluster services Once you have configured the cluster services as you like, you must learn how to manage them. We can manage cluster services with the clusvcadm command and additional parameters. The syntax of the clusvcadm command is as follows: clusvcadm [parameter] With the clusvcadm command, you can perform the following actions: Disable service (syntax: clusvcadm -d <service_name>): This stops the cluster service and puts it into the disabled state. This is the only permitted operation if the service in question is in the failed state. Start service (syntax: clusvcadm -e <service_name> -m <cluster_node>): This starts a non-running cluster service. It optionally provides the cluster node name you would like to start the service on. Relocate service (syntax: clusvcadm -r <service_name> -m <cluster_node>): This stops the cluster service and starts it on a different cluster node as provided with the -m parameter. Migrate service (syntax: clusvcadm -M <service_name> -m <cluster_node>): Note that this applies only to virtual machine live migrations. Restart service (syntax: clusvcadm -R <service_name>): This stops and starts a cluster service on the same cluster node. Stop service (syntax: clusvcadm -s <service_name>): This stops the cluster service and keeps it on the current cluster node in the stopped state. Freeze service (syntax: clusvcadm -Z <service_name>): This keeps the cluster service running on the current cluster node but disables service status checks and service failover in the event of a cluster node failure. Unfreeze service (syntax: clusvcadm -U <service_name>): This takes the cluster service out of the frozen state and enables service status checks and failover. We can continue with the previous example and migrate the webserver cluster service from the currently running node-1 cluster node to the node-3 cluster node. To achieve cluster service relocation, the clusvcadm command with the relocate service parameter must be used, as follows. In the following screenshot, we can see the command issued to migrate the webserver cluster service to the node-3 cluster node: The clusvcadm command is the cluster service command used to administer and manage cluster services. The -r webserver parameter provides information that we need to relocate a cluster service named webserver. The -m node-3 command provides information on where we want to relocate the cluster service. Once the cluster service migration command completes, the webserver cluster service will be relocated to the node-3 cluster node. The clustat command shows that the webserver service is now running on the node-3 cluster node. In this screenshot, we can see that the webserver cluster service was successfully relocated to the node-3 cluster node: We can easily stop the webserver cluster service by issuing the appropriate command. In the following screenshot, we can see the command used to stop the webserver cluster service: The clusvcadm command is the cluster service command used to administer and manage cluster services. The -s webserver parameter provides the information that you require to stop a cluster service named webserver. Another take at the clustat command should show that the webserver cluster service has stopped; it also provides the information that the last owner of the running webserver cluster service is the node-3 cluster node. In this screenshot, we can see the output of the clustat command, showing that the webserver cluster service is running on the node-3 cluster node: If we want to start the webserver cluster service on the node-1 cluster node, we can do this by issuing the appropriate command. In the following screenshot, we can see the command used to start the webserver cluster service on the node-1 cluster node: clusvcadm is the cluster service command used to administer and manage cluster services. The -e webserver parameter provides the information that you need to start a webserver cluster service. The -m node-1 parameter provides the information that you need to start the webserver cluster service on the node-1 cluster node. As expected, another look at the clustat command should make it clear that the webserver cluster service has started on the node-1 cluster node, as follows. In this screenshot, you can see the output of the clustat command, showing that the webserver cluster service is running on the node -1 cluster node: Removing cluster resources and services Removing cluster resources and services is the reverse of adding them. Resources and services are removed by editing the CMAN configuration file and removing the lines that define the resources or services you would like to remove. When removing cluster resources, it is important to verify that the resources are not being used within any of the configured or running cluster services. As always, when editing the CMAN configuration file, the config_version parameter must be increased. Once the CMAN configuration file is edited, you must run the CMAN configuration validation check for errors. When the CMAN configuration file validation succeeds, you can distribute it to all other cluster nodes. The procedure for removing cluster resources and services is as follows: Remove the desired cluster resources and services and increase the config_version number. Validate the CMAN configuration file. Distribute the CMAN configuration file to all other nodes. We can proceed to remove the webserver cluster service from our example cluster configuration. Edit the CMAN configuration file and remove the webserver cluster service definition. Remember to increase the config_version number. Validate your cluster configuration with every CMAN configuration file change. In this screenshot, we can see the command used to validate the CMAN configuration: When your cluster configuration is valid, you can distribute the CMAN configuration file to all other cluster nodes. In the following screenshot, we can see the command used to distribute the CMAN configuration file from the node-1 cluster node to other cluster nodes: Once the cluster configuration is distributed to all cluster nodes, the webserver cluster service will be stopped and removed. The clustat command shows no service configured and running. In the following screenshot, we can see that the output of the clustat command shows no cluster service called webserver existing in the cluster: Summary In this article, you learned how to add and remove cluster failover domains, cluster resources, and cluster services. We also learned how to start, stop, and migrate cluster services from one cluster node to another, and how to remove cluster resources and services from a running cluster configuration. Resources for Article: Further resources on this subject: Replication [article] Managing public and private groups [article] Installing CentOS [article]
Read more
  • 0
  • 0
  • 4162

article-image-moving-your-application-production
Packt
16 Dec 2013
9 min read
Save for later

Moving Your Application to Production

Packt
16 Dec 2013
9 min read
(For more resources related to this topic, see here.) We will start by examining the Sencha Cmd compiler. Compiling with Sencha Cmd This section will focus on using Sencha Cmd to compile our Ext JS 4 application for deployment within a Web Archive (WAR) file. The goal of the compilation process is to create a single JavaScript file that contains all of the code needed for the application, including all the Ext JS 4 dependencies. The index.html file that was created during the application skeleton generation is structured as follows: <!DOCTYPE HTML> <html> <head> <meta charset="UTF-8"> <title>TTT</title> <!-- <x-compile> --> <!-- <x-bootstrap> --> <link rel="stylesheet" href="bootstrap.css"> <script src = "ext/ext-dev.js"></script> <script src = "bootstrap.js"></script> <!-- </x-bootstrap> --> <script src = "app.js"></script> <!-- </x-compile> --> </head> <body></body> </html> The open and close tags of the x-compile directive enclose the part of the index.html file where the Sencha Cmd compiler will operate. The only declarations that should be contained in this block are the script tags. The compiler will process all of the scripts within the x-compile directive, searching for dependencies based on the Ext.define, requires, or uses directives. An exception to this is the ext-dev.js file. This file is considered to be a "bootstrap" file for the framework and will not be processed in the same way. The compiler ignores the files in the x-bootstrap block and the declarations are removed from the final compiler-generated page. The first step in the compilation process is to examine and parse all the JavaScript source code and analyze any dependencies. To do this the compiler needs to identify all the source folders in the application. Our application has two source folders: Ext JS 4 sources in webapp/ext/src and 3T application sources in webapp/app. These folder locations are specified using the -sdk and -classpath arguments in the compile command: sencha –sdk {path-to-sdk} compile -classpath={app-sources-folder} page -yui -in{index-page-to-compile}-out {output-file-location} For our 3T application the compile command is as follows: sencha –sdk ext compile -classpath=app page -yui -in index.html-out build/index.html This command performs the following actions: The Sencha Cmd compiler examines all the folders specified by the -classpath argument. The -sdk directory is automatically included for scanning. The page command then includes all of the script tags in index.html that are contained in the x-compile block. After identifying the content of the app directory and the index.html page, the compiler analyzes the JavaScript code and determines what is ultimately needed for inclusion in a single JavaScript file representing the application. A modified version of the original index.html file is written to build/index.html. All of the JavaScript files needed by the new index.html file are concatenated and compressed using the YUI Compressor, and written to the build/all-classes.js file. The sencha compile command must be executed from within the webapp directory, which is the root of the application and is the directory containing the index.html file. All the arguments supplied to the sencha compile command can then be relative to the webapp directory. Open a command prompt (or terminal window in Mac) and navigate to the webapp directory of the 3T project. Executing the sencha compile command as shown earlier in this section will result in the following output: Opening the webapp/build folder in NetBeans should now show the two newly generated files: index.html and all-classes.js. The all-classes.js file will contain all the required Ext JS 4 classes in addition to all the 3T application classes. Attempting to open this file in NetBeans will result in the following warning: "The file seems to be too large to open safely...", but you can open the file in a text editor to see the following concatenated and minified content: Opening the build/index.html page in NetBeans will display the following screenshot: You can now open the build/index.html file in the browser after running the application, but the result may surprise you: The layout that is presented will depend on the browser, but regardless, you will see that the CSS styling is missing. The CSS files required by our application need to be moved outside the <!-- <x-compile> --> directive. But where are the styles coming from? It is now time to briefly delve into Ext JS 4 themes and the bootstrap.css file. Ext JS 4 theming Ext JS 4 themes leverage Syntactically Awesome StyleSheets (SASS) and Compass (http://compass-style.org/) to enable the use of variables and mixins in stylesheets. Almost all of the styles for Ext JS 4 components can be customized, including colors, fonts, borders, and backgrounds, by simply changing the SASS variables. SASS is an extension of CSS that allows you to keep large stylesheets well-organized; a very good overview and reference can be found at http://sass-lang.com/documentation/file.SASS_REFERENCE.html. Theming an Ext JS 4 application using Compass and SASS is beyond the scope of this book. Sencha Cmd allows easy integration with these technologies to build SASS projects; however, the SASS language and syntax is a steep learning curve in its own right. Ext JS 4 theming is very powerful and minor changes to the existing themes can quickly change the appearance of your application. You can find out more about Ext JS 4 theming at http://docs.sencha.com/extjs/4.2.2/#!/guide/theming. The bootstrap.css file was created with the default theme definition during the generation of the application skeleton. The content of the bootstrap.css file is as follows: @import 'ext/packages/ext-theme-classic/build/resources/ext-theme-classic-all.css'; This file imports the ext-theme-classic-all.css stylesheet, which is the default "classic" Ext JS theme. All of the available themes can be found in the ext/packages directory of the Ext JS 4 SDK: Changing to a different theme is as simple as changing the bootstrap.css import. Switching to the neptune theme would require the following bootstrap.css definition: @import 'ext/packages/ext-theme-neptune/build/resources/ext-theme-neptune-all.css'; This modification will change the appearance of the application to the Ext JS "neptune" theme as shown in the following screenshot: We will change the bootstrap.css file definition to use the gray theme: @import 'ext /packages/ext-theme-gray/build/resources/ext-theme-gray-all.css'; This will result in the following appearance: You may experiment with different themes but should note that not all of the themes may be as complete as the classic theme; minor changes may be required to fully utilize the styling for some components. We will keep the gray theme for our index.html page. This will allow us to differentiate the (original) index.html page from the new ones that will be created in the following section using the classic theme. Compiling for production use Until now we have only worked with the Sencha Cmd-generated index.html file. We will now create a new index-dev.html file for our development environment. The development file will be a copy of the index.html file without the bootstrap.css file. We will reference the default classic theme in the index-dev.html file as follows: <!DOCTYPE HTML> <html> <head> <meta charset="UTF-8"> <title>TTT</title> <link rel="stylesh eet" href="ext/packages/ext-theme-classic/build/resources/ext-theme-classic-all. css"> <link rel="stylesheet" href="resources/styles.css"> <!-- <x-compile> --> <!-- <x-bootstrap> --> <script src = "ext/ext-dev.js"></script> <script src = "bootstrap.js"></script> <!-- </x-bootstrap> --> <script src = "app.js"></script> <!-- </x-compile> --> </head> <body></body> </html> Note that we have moved the stylesheet definition out of the <!-- <x-compile> --> directive. If you are using the downloaded source code for the book, you will have the resources/styles.css file and the resources directory structure available. The stylesheet and associated images in the resources directory contain the 3T logos and icons. We recommend you download the full source code now for completeness. We can now modify the Sencha Cmd compile command to use the index-dev.html file and output the generated compile file to index-prod.html in the webapp directory: sencha –sdk ext compile -classpath=app page -yui -in index-dev.html-out index-prod.html This command will generate the index-prod.html file and the all-classes.js files in the webapp directory as shown in the following screenshot: The index-prod.html file references the stylesheets directly and uses the single compiled and minified all-classes.js file. You can now run the application and browse the index-prod.html file as shown in the following screenshot: You should notice a significant increase in the speed with which the logon window is displayed as all the JavaScript classes are loaded from the single all-classes.js file. The index-prod.html file will be used by developers to test the compiled all-classes.js file. Accessing the individual pages will now allow us to differentiate between environments: The logon window as displayed in the browser Page description The index.html page was generated by Sencha Cmd and has been configured to use the gray theme in bootstrap.css. This page is no longer needed for development; use index-dev.html instead. You can access this page at http://localhost:8080/index.html The index-dev.html page uses the classic theme stylesheet included outside the <!-- <x-compile> --> directive. Use this file for application development. Ext JS 4 will dynamically load source files as required. You can access this page at http://localhost:8080/index-dev.html The index-prod.html file is dynamically generated by the Sencha Cmd compile command. This page uses the all-classes.js all-in-one compiled JavaScript file with the classic theme stylesheet. You can access this page at http://localhost:8080/index-prod.html
Read more
  • 0
  • 0
  • 4159

article-image-business-rules-management-bpm-and-soa
Packt
27 Nov 2009
13 min read
Save for later

Business Rules Management, BPM, and SOA

Packt
27 Nov 2009
13 min read
Introduction to Business Rules Management Let us start by understanding some key concepts around business rules. What are Business Rules? Business rules can be defined as the key decisions and policies of the business. Rules are virtually everywhere in an organization; an example is the rule in a bank to deny a loan for a customer if his or her annual income is less than $15,000. We can generally categorize business rules under the following categories: Business Policies: These are rules associated with general business policies of a company, for example, loan approval policies, escalation policies, and so on. Constraints: These are the rules which business has to keep in mind, and work within the scope of while going about their operations. Rules associated with regulatory requirements will fall under this category. Computation: These are the rules associated with decisions involving any calculations, for example, discounting rules, premium adjustments, and so on. Reasoning capabilities: These are the rules that apply logic and inference course of actions based on multiple criteria. For example, rules associated with the up-sell or cross-sell of products and services to customers based on their profile. Allocation Rules: There are some rules that are applicable in terms of determining the course of action for the process, based on information from the previous tasks. They also include rules that manage the receiving, assignment, routing, and tracking of work. Business Rules Anatomy To understand the anatomy of a business rule, we can divide a business rule primarily into the following four blocks: Definitions of Terms: This helps in providing a vocabulary for expressing the rules. Defining a term acts as the category for the rules. For example, customer, car, claims, and so on define the entities for the business. Facts: These are used to relate terms in definitions with each other. For example, a customer may apply for a claim. Constraints: These are the constraints, limitations, or controls on how an organization wants to use and update the data. For example, for opening an account, a customer's passport details or social security details are required. Inference: This basically applies to logical assertions such as 'if X, then Y' to a fact, and infers new facts. For example, if we have a single account validation rule (if an applicant is a defaulter, then the applicant is high-risk), and we know that Harry (the applicant) has defaulted earlier on his payments for other bank services, we can infer that Harry is a high-risk customer. Automating Business Rules As we discuss the externalization and automation of business rules, it's important to understand the distinction between implicit and explicit rules. An implicit rule can be viewed as a rule that is a part of a larger context within the system. It's like multiple rules that are implemented in traditional applications to implement decision logic, for example, assessing the risk level for a loan. Its implementation is usually part of the application it is being developed for, and is never considered beyond the scope of the application, perhaps to be re-used. So Typically, in the IT world, these implicit rules are embedded within the complex application code and spread across multiple systems, making it extremely difficult to introduce changes quickly, and without creating a domino effect across systems. Some of these issues can be resolved by implementing a Business Rules Management System (BRMS) in collaboration with the BPM system in place. This allows the decision logic, which is being used by the process during its execution, to be driven by a central repository where all the rules are stored and managed. This repository provides a way to abstract the decision logic from the applications, and helps in managing this logic centrally, allowing for better management and flexibility for change and re-use. Hence, these rules are explicit in nature. For the loan approval example, business rules such as these would traditionally be embedded in application code, and might appear in an application as follows: public boolean checkAnnualIncome(Customer customer){boolean declineLoan = false;int income = customer.getincome();if( income < 10000 ){declineLoan = true;}return declineLoan;} The above example shows that this rule is obviously difficult for the business users to understand. In today's world, with the need for an organization to be agile, (considering our previous example) the business has to wait for weeks before a small change can be implemented by IT. What is required is the ability of the business users to define and control their own rules, and to be able to get the changes out in the market faster. Business Rules Management and related technology tries to solve this problem. Automating Business Rules for Business Issues Automation of business rules via BRMS is ideal for use, where the following issues are being faced by an organization: Dynamism and Volatility: Companies need to repeatedly change business policies, procedures, and products to meet the market needs. In this case, the rules change very dynamically, and having a BRMS can help in implementing these changes faster, and reducing the time to market and cost of implementation. Time to Market: In this case, the organization might want a particular set of changes to be released quickly due to market pressure, or to gain a competitive advantage. So, Even though the rules are not changed very often, a delay in their implementation could lead to a serious business loss. In this case, the organization needs to have the ability to get these changes in quickly, without roadblocks, which can be addressed by a BRMS. Regulatory Compliance: Failure to comply with regulatory requirements such as Anti-Money Laundering (AML) laws can result in millions of dollars in fines, and legal issues for the organizations. To solve these issues, institutions can combine business rules with SOA to create an effective strategy for enforcing compliance. Business rules technology helps in implementing these rules quickly, and helps them to be kept up–to-date across an enterprise. Business Participation: There could be rules which might be better off being controlled and owned by the business users. In this case, a BRMS can expose certain rules to be managed and edited by selected business users, providing an easy to use interface. Rules related to product configuration, customer eligibility, discounts and so on, are some examples where business users can manage the rules, and change them as required by changing scenarios. Complexity: Some scenarios, such as complex product and service pricing, require extremely complex dependencies between several rules to implement the scenario logic. These kinds of rules are best suited for implementation inside a BRMS rather than a procedural language, as is being done traditionally. Telecom Fraud Management, for example, is an area where rules management is being used along with BAM to identify potential frauds. There are similar applications in credit card and banking industries. Consistency: Rules managed centrally provides a more consistent way of managing certain policies requiring re-use and consistency across the enterprise. This is especially true in cases where inconsistency was an issue due to multiple applications, databases, and different lines of businesses. Business Rules Management, BPM, and SOA Business Rules Management, BPM, and SOA share a synergistic relationship, especially, when used together to provide agility to an organization. The term 'Agility' can be defined as "the ability of an enterprise to sense and predict change in their environment and respond quickly, efficiently, and effectively to that change". Agility, requires the organization to be flexible enough in introducing change and in modifying their current operations, to achieve higher levels of performance or output. A process-driven approach to SOA allows business users to introduce changes to the process for faster execution, and with less cost. This value is amplified by using a Business Rules platform alongside process orchestration. If we look at the BPM reference architecture again, rules functionality features in various layers of the architecture, in the initial rules discovery phase, during process mapping, and in its orchestration in the SOA environment. Business Rules-related technologies have been in the market for a number of years now. However, with the acceptance of BPM and SOA as enablers for increasing an organization's agility, today's enterprises are increasingly looking at using rules management to externalize their rules. Business rules management helps automate decisions and apply policies within processes. Automation of these decisions requires determining the meaning of a given situation, and applying a business policy in response to this. Business rules platforms provide tools to define this 'reasoning' logic for use by either developers or business analysts, and business stakeholders. Organizations are looking at Business Rules Management to deploy rules related to policy decisions, work allocation, compliance and control, business exception management, and even data validation. For example, a major financial services company uses business rules to apply privacy and anti-fraud policies to all of its transactions. Even more, these Business Rules are being considered as an asset for an organization that should be managed centrally and re-used across departments and systems, instead of being hard-coded into an application. So, it is important to ensure that business rules have a place in your SOA. Carefully defining and exposing your rules as services will enable all of the applications and services within your architecture to have simple access to a common rules repository. From an SOA perspective, before beginning a business rules implementation, you should: Incorporate a business rules platform into your SOA: This would be a service-enabled repository of your business rules, where instead of data you would maintain and execute rulesets using a business rules engine. Create standards and best practices for developing business rules: To maximize benefits from your rules implementation, you should focus on developing common standards and best practices for discovery, design, development, and interfacing of your rules. Some of the best practices for writing and designing business rules are: Declarative: Business rules should be declared, and not stated as procedures as in coding. How a rule will be enforced should not be part of a rule definition. For example, "If the customer is a premium customer, offer him further 5% discount." Precise: It's easier for business rules definitions to be misinterpreted due to the use of natural language syntax by business. One business rule should be open to only one interpretation, and would need rephrasing if it was found to be ambiguous. Consistency and non-redundancy: Business rules should be consistent and not conflict other rules. Similarly, you should look out for business rules that are redundant. Business Focused and Owned: Business rules should be declared using the business vocabulary so that they can understood by relevant business stakeholders. Avoid using technical jargons in business rules. Also business rules are best left under the ownership of the business, community, as that is the source for the rules. Key Considerations for Selecting a BRMS The following are some key considerations when selecting a BRMS to work with BPM and SOA: Standards-based Integration capability: The ability to integrate with the SOA landscape using a service layer. Business User Interface: The ability to provide the capability for business users to access and modify business rules through a user-friendly interface. Rule Language: The ability to provide support for natural languages for easily expressing a complex set of rules. Performance: The ability to provide support for high-volume transactions for mission-critical applications, which is normally measured in terms of the number of rules processed per second. Rules Monitoring and Reporting: The ability to feature support for rules debugging, rules reporting, and real time monitoring of rules. Rules Repository: The ability to provide a centralized repository for storing all rule-specific artifacts. The repository should also support change management by storing different versions of rules, and providing audit capabilities. Key components of a BRMS—A Brief Look into Oracle Business Rules Typically, a BRMS will comprise four main components: Business UI: This is a user interface component for writing and editing business rules. Typically, it will be a web-based interface for business users to log in and access existing business rules, create new ones, and so on. Rules Development Environment: Developers will be working in this environment to convert business rules defined by business users into code that can be implemented in the business rules engine. This will be also an environment where the service layer for the rules will be defined and implemented for integration with other applications and SOA components. Rules Repository: This will be a centralized repository where all rules-related information will be stored. Rules Execution Engine: This is the heart of the rules management system and will be responsible for executing the business rules in the run time environment. In SOA terms, this component will receive request for rules processing from the business process orchestration environment, based on which, it will run appropriate rules and provide decision information that will be sent back to the orchestration layer. Oracle also provides a suite of components under its Oracle Business Rules product to support rules management and execution, which are as follows: Oracle Rule Author: Rule Author provides a web-based graphical authoring environment that enables the easy creation of business rules via a web browser. The application developer uses Rule Author to define a data model and an initial set of rules. The business analyst uses Rule Author either to work with the initial set of rules, or to modify and customize the initial set of rules according to business needs. Using Rule Author, a business analyst can create and customize rules with little or no assistance from a programmer. Rules Engine: This is the heart of the rules system and executes and manages rules in a proper and efficient manner. This allows inference-based rule execution, based on the very popular Rete algorithm. The Rete algorithm is an efficient pattern-matching algorithm used for rules and facts, and stores partially-matched results in a single network of nodes in current working memory, allowing the rules engine to avoid unnecessary rechecking when facts are deleted, added, or modified. Oracle's rules engine provides a data-driven forward-changing system. This means that the facts will determine which rules can be triggered. When a particular rule is triggered, based on pattern matching within a set of facts, the rule may further add new facts. The new facts are again run against the rules as an iterative process untill it reaches an end state. This allows rules to be interlinked and triggered in a cycle, also referred to as an inference cycle.The rules engine also provides a web service interface with its SOA environment using 'Decision Services', which is available in a JDeveloper environment during the coding of business processes in BPEL. This can also be used to make a web service call to rules running in the rules engine. It also exposes a Rules API, which is based on JSR 94, a runtime specification for rules engines to integrate business rules application with other applications in an organization. Rule Repository: A rule repository is the database that stores business rules. The Oracle rules repository allows rules to be grouped as rulesets, and make it part of the rules dictionary in a central repository. These dictionaries can be versioned for better governance. Oracle's rules repository supports a WebDAV (Web Distributed Authoring and Versioning) repository and a file repository. Rules SDK: This allows users to develop and integrate the Rules Repository in to a custom authoring environment. This component also allows the development of a customized UI for business users to access and update the Rules repository, if required.
Read more
  • 0
  • 0
  • 4156

article-image-managing-user-accounts-oracle-siebel-crm-8
Packt
06 Aug 2010
5 min read
Save for later

Managing User Accounts in Oracle Siebel CRM 8

Packt
06 Aug 2010
5 min read
Understanding divisions and organizations Mapping real world entities to technical concepts is one of the key factors for successful software products. It is therefore quite interesting to see how the Siebel design team faced the challenge of bringing the complex hierarchical relationships of large corporations into the Siebel data model. Early in the analysis phase of a Siebel CRM implementation project, the divisional or departmental hierarchy of the customer is analyzed and documented. The typical diagram type to document the divisions of a company is an organization chart. The following is the organization chart for an example company: The Sales, Service, and Marketing divisions are subordinate to the Headquarter division. The Sales department has subdivisions, which define the territories (North and South) that the sales force operates in. Setting up divisions Once the divisional hierarchy of a company is documented, it must be translated to Siebel administrative data. An administrator uses the Administration - Group screen, Internal Divisions view—shown in the following screenshot—to enter and manage the division information: This view allows administrators to enter and maintain information about a company's divisions. The example company visible in the screenshot (Vision Corporation) is part of the Siebel sample database and has four subordinate divisions. In order to create a new divisional hierarchy, we can follow the steps in the task list below: Log in to the Siebel application using an administrative user account. Navigate to the Administration - Group screen, Internal Divisions view. Create a new record for the top-level division first. Enter address and other information if required. Save the division record. Create a new division for each subordinate division and use the Parent Division field to select the appropriate parent division. Use the explorer applet to verify that the divisional hierarchy represents the organization chart. The following screenshot shows the explorer applet in the Internal Divisions view after the entry of the data from the example organization chart: In order to distinguish the example company from other divisions in the database, the acronym "AHA " was used. From an administrative perspective, we must be aware of the fact that organizational changes might occur frequently. These changes can include one department becoming subordinate of another, or other departments being detached from the hierarchy in order to become separate companies. Siebel administrators must be informed of these changes in order to be able to adjust the division data in a timely manner. In your demonstration environment, use the instructions in the above section to create a division hierarchy. You might want to use the example or create your own divisions. Setting up organizations When a division or an entire partner company wants to use the Siebel CRM infrastructure, this is typically accompanied by the requirement to associate data with the division or partner company in order to provide data security. Siebel administrators can declare a division as an organization. This is done by simply checking the Organization Flag of a division. However, this change cannot be undone. Once the division is flagged as an organization and the record is saved, the flag becomes read only, as shown in the following screenshot: We can decide which divisions within the organization chart should be flagged as organizations depending on the data security requirements defined by the project team. The result is typically a second hierarchy of organizations within the division hierarchy. Once an organization is created, Siebel data such as customer accounts, service requests, and so on can be associated with the organization. The following diagram shows how the divisions named Headquarter and Sales have been flagged as organizations. They are now part of the organization hierarchy. By default, each new organization becomes subordinate to the "Default Organization", which the position of the Siebel Administrator (SADMIN) is assigned to. If data security policies mandate, we must set the Parent Organization field to an empty value in the Organizations view of the Administration - Group screen. Even if a division cannot be associated to Siebel data, employees who have a position within that division are automatically associated with the nearest organization that can be located upwards in the division hierarchy. In the above example, an employee who has a position in the AHA Sales North division will be associated with the Sales organization. The following screenshot shows the AHA Sales North division (note that the Organization Flag is unchecked) in the Internal Divisions view: The Organization Name field displays the name of the nearest organization (AHA Sales) above the AHA Sales North division. Employees who are associated with a position in the AHA Sales North division will automatically be associated with the AHA Sales organization. They will therefore be able to see data associated with the AHA Sales organization and each record they create will be automatically associated with the AHA Sales organization. Similar to divisions, organizations cannot be deleted. When organizational changes require it, a Siebel administrator must detach the non-existing organization or division from all parent records by emptying the parent division field and change the name to indicate the state of the organization or division. For example, the name can be prefixed with "NOT USED" to indicate that the division or organization no longer exists. Furthermore, records that are associated with an organization that no longer exists must be re-assigned to other organizations. This is typically achieved by using the Siebel Assignment Manager. Mark at least one of the sample divisions you created earlier as an organization by setting the Organization Flag and saving the record.
Read more
  • 0
  • 0
  • 4149

article-image-distributed-rails-applications-with-chef
Rahmal Conda
31 Oct 2014
4 min read
Save for later

Configuring Distributed Rails Applications with Chef: Part 1

Rahmal Conda
31 Oct 2014
4 min read
Since the Advent of Rails (and Ruby by extension), in the period between 2005 and 2010, Rails went from a niche Web Application Framework to being the center of a robust web application platform. To do this it needed more than Ruby and a few complementary gems. Anyone who has ever tried to deploy a Rails application into a production environment knows that Rails doesn’t run in a vacuum. Rails still needs a web server in front of it to help manage requests like Apache or Nginx. Oops, you’ll need unicorn or Passenger too. Almost all of the Rails apps are backed by some sort of data persistence layer. Usually that is some sort of relational database. More and more it’s a NoSQL DB like MongoDB or depending on the application, you’re probably going to deploy a caching strategy at some point: Memcached, Redis, the list goes on. What about background jobs? You’ll need another server instance for that too, and not just one either. High availability systems need to be redundant. If you’re lucky enough to get a lot of traffic, you’ll need a way to scale all of this. Why Chef? Chances are that you’re managing all of this traffic manually. Don’t feel bad, everyone starts out that way. But as you grow, how do you manage all of this without going insane? Most Rails developers start off with Capistrano, which is a great choice. Capistrano is a remote server automation tool. It’s used most often as a deployment tool for Rails. For the most part it’s a great solution for managing multiple servers that make up your Rails stack. It’s only when your architecture reaches a certain size that I’d recommend choosing Chef over Capistrano. But really, there’s no reason to choose one over the other since they actually work pretty well together, and they are both similar regarding deployment. Where Chef excels, however, is when you need to provision multiple servers with different roles, and changing software stacks. This is what I’m going to focus on in this post. But let’s introduce Chef first. What is Chef anyway? Basically, Chef is a Ruby-based configuration management engine. It is a software configuration management tool, used for provisioning servers for certain roles within a platform stack, and deploying applications to those servers. It is used to automate server configuration and integration into your infrastructure. You define your infrastructure in configuration files written in Chef’s Ruby DSL and Chef takes care of setting up individual machines and linking them together. Chef server You set up one of your server instances (virtual or otherwise) as the server and all your other instances are clients that communicate with the Chef "server" via REST over HTTPS. The server is an application that stores cookbooks for your nodes. Recipes and cookbooks Recipes are files that contain sets of instructions written in Chef’s Ruby DSL. These instructions perform some kind of procedure, usually installing software and configuring some service. These recipes are bound together along with configuration file templates, resources, and helper scripts as cookbooks. Cookbooks generally correspond to a specific server configuration. For instance, a Postgres cookbook might contain a recipe for Postgres Server, Postgres Client, maybe PostGIS, and some configuration files for how the DB instance should be provisioned. Chef Solo For stacks that don’t necessarily need a full Chef server setup, but use cookbooks to set up Rails and DB servers, there’s Chef Solo. Chef Solo is a local standalone Chef application that can be used to remotely deploy servers and applications. Wait, where is the code? In Part 2 of this post I’m going to walk you through the setting up of a Rails application with Chef Solo, then I’ll expand to show a full Chef server configuration management engine. While Chef can be used for many different application stacks, I’m going to focus on Rails configuration and deployment, provisioning and deploying the entire stack. See you next time! About the Author Rahmal Conda is a Software Development Professional and Ruby aficionado from Chicago. After 10 years working in web and application development, he moved out to the Bay Area, eager to join the startup scene. He had a taste of the startup life in Chicago working at a small personal finance company. After that he knew it was the life he had been looking for. So he moved his family out west. Since then he's made a name for himself in the social space at some high profile Silicon Valley startups. Right now he's the one of the Co-founders and Platform Architect of Boxes, a mobile marketplace for the world's hidden treasures.
Read more
  • 0
  • 0
  • 4141

article-image-fast-data-manipulation-r
Packt
14 Oct 2016
28 min read
Save for later

Fast Data Manipulation with R

Packt
14 Oct 2016
28 min read
Data analysis is a combination of art and science. The art part consists of data exploration and visualization, which is usually done best with better intuition and understanding of the data. The science part consists of statistical analysis, which relies on concrete knowledge of statistics and analytic skills. However, both parts of a serious research require proper tools and good skills to work with them. R is exactly the proper tool to do data analysis with. In this article by Kun Ren, author of the book Learning R Programming, we will discuss how R and data.table package make it easy to transform data and, thus, greatly unleash our productivity. (For more resources related to this topic, see here.) Loading data as data frames The most basic data structures in R are atomic vectors, such as. numeric, logical, character, and complex vector, and list. An atomic vector stores elements of the same type while list is allowed to store different types of elements. The most commonly used data structure in R to store real-world data is data frame. A data frame stores data in tabular form. In essence, a data frame is a list of vectors with equal length but maybe different types. Most of the code in this article is based on a group of fictitious data about some products (you can download the data at https://gist.github.com/renkun-ken/ba2d33f21efded23db66a68240c20c92). We will use the readr package to load the data for better handling of column types. If you don't have this package installed, please run install.packages("readr"). library(readr) product_info <- read_csv("data/product-info.csv") product_info ##    id      name  type   class released ## 1 T01    SupCar   toy vehicle      yes ## 2 T02  SupPlane   toy vehicle       no ## 3 M01     JeepX model vehicle      yes ## 4 M02 AircraftX model vehicle      yes ## 5 M03    Runner model  people      yes ## 6 M04    Dancer model  people       no Once the data is loaded into memory as a data frame, we can take a look at its column types, shown as follows: sapply(product_info, class) ##          id        name        type       class    released ## "character" "character" "character" "character" "character" Using built-in functions to manipulate data frames Although a data frame is essentially a list of vectors, we can access it like a matrix due to all column vectors being the same length. To select rows that meet certain conditions, we will supply a logical vector as the first argument of [] while the second is left empty. For example, we can take out all rows of toy type, shown as follows: product_info[product_info$type == "toy", ] ##    id     name type   class released ## 1 T01   SupCar  toy vehicle      yes ## 2 T02 SupPlane  toy vehicle       no Or, we can take out all rows that are not released. product_info[product_info$released == "no", ] ##    id     name  type   class released ## 2 T02 SupPlane   toy vehicle       no ## 6 M04   Dancer model  people       no To filter columns, we can supply a character vector as the second argument while the first is left empty, which is exactly the same with how we subset a matrix. product_info[1:3, c("id", "name", "type")] ##    id     name  type ## 1 T01   SupCar   toy ## 2 T02 SupPlane   toy ## 3 M01    JeepX model Alternatively, we can filter the data frame by regarding it as a list. We can supply only one character vector of column names in []. product_info[c("id", "name", "class")] ##    id      name   class ## 1 T01    SupCar vehicle ## 2 T02  SupPlane vehicle ## 3 M01     JeepX vehicle ## 4 M02 AircraftX vehicle ## 5 M03    Runner  people ## 6 M04    Dancer  people To filter a data frame by both row and column, we can supply a vector as the first argument to select rows and a vector as the second to select columns. product_info[product_info$type == "toy", c("name", "class", "released")] ##       name   class released ## 1   SupCar vehicle      yes ## 2 SupPlane vehicle       no If the row filtering condition is based on values of certain columns, the preceding code can be very redundant, especially when the condition gets more complicated. Another built-in function to simplify code is subset, as introduced previously. subset(product_info,   subset = type == "model" & released == "yes",   select = name:class) ##        name  type   class ## 3     JeepX model vehicle ## 4 AircraftX model vehicle ## 5    Runner model  people The subset function uses nonstandard evaluation so that we can directly use the columns of the data frame without typing product_info many times because the expressions are meant to be evaluated in the context of the data frame. Similarly, we can use with to evaluate an expression in the context of the data frame, that is, the columns of the data frame can be used as symbols in the expression without repeatedly specifying the data frame. with(product_info, name[released == "no"]) ## [1] "SupPlane" "Dancer" The expression can be more than a simple subsetting. We can summarize the data by counting the occurrences of each possible value of a vector. For example, we can create a table of occurrences of types of records that are released. with(product_info, table(type[released == "yes"])) ## ## model   toy ##     3     1 In addition to the table of product information, we also have a table of product statistics that describe some properties of each product. product_stats <- read_csv("data/product-stats.csv") product_stats ##    id material size weight ## 1 T01    Metal  120   10.0 ## 2 T02    Metal  350   45.0 ## 3 M01 Plastics   50     NA ## 4 M02 Plastics   85    3.0 ## 5 M03     Wood   15     NA ## 6 M04     Wood   16    0.6 Now, think of how we can get the names of products with the top three largest sizes? One way is to sort the records in product_stats by size in descending order, select id values of the top three records, and use these values to filter rows of product_info by id. top_3_id <- product_stats[order(product_stats$size, decreasing = TRUE), "id"][1:3] product_info[product_info$id %in% top_3_id, ] ##    id      name  type   class released ## 1 T01    SupCar   toy vehicle      yes ## 2 T02  SupPlane   toy vehicle       no ## 4 M02 AircraftX model vehicle      yes This approach looks quite redundant. Note that product_info and product_stats actually describe the same set of products in different perspectives. The connection between these two tables is the id column. Each id is unique and means the same product. To access both sets of information, we can put the two tables together into one data frame. The simplest way to do this is use merge: product_table <- merge(product_info, product_stats, by = "id") product_table ##    id      name  type   class released material size weight ## 1 M01     JeepX model vehicle      yes Plastics   50     NA ## 2 M02 AircraftX model vehicle      yes Plastics   85    3.0 ## 3 M03    Runner model  people      yes     Wood   15     NA ## 4 M04    Dancer model  people       no     Wood   16    0.6 ## 5 T01    SupCar   toy vehicle      yes    Metal  120   10.0 ## 6 T02  SupPlane   toy vehicle       no    Metal  350   45.0 Now, we can create a new data frame that is a combined version of product_table and product_info with a shared id column. In fact, if you reorder the records in the second table, the two tables still can be correctly merged. With the combined version, we can do things more easily. For example, with the merged version, we can sort the data frame with any column in one table we loaded without having to manually work with the other. product_table[order(product_table$size), ] ##    id      name  type   class released material size weight ## 3 M03    Runner model  people      yes     Wood   15     NA ## 4 M04    Dancer model  people       no     Wood   16    0.6 ## 1 M01     JeepX model vehicle      yes Plastics   50     NA ## 2 M02 AircraftX model vehicle      yes Plastics   85    3.0 ## 5 T01    SupCar   toy vehicle      yes    Metal  120   10.0 ## 6 T02  SupPlane   toy vehicle       no    Metal  350   45.0 To solve the problem, we can directly use the merged table and get the same answer. product_table[order(product_table$size, decreasing = TRUE), "name"][1:3] ## [1] "SupPlane"  "SupCar"    "AircraftX" The merged data frame allows us to sort the records by a column in one data frame and filter the records by a column in the other. For example, we can first sort the product records by weight in descending order and select all records of model type. product_table[order(product_table$weight, decreasing = TRUE), ][   product_table$type == "model",] ##    id      name  type   class released material size weight ## 6 T02  SupPlane   toy vehicle       no    Metal  350   45.0 ## 5 T01    SupCar   toy vehicle      yes    Metal  120   10.0 ## 2 M02 AircraftX model vehicle      yes Plastics   85    3.0 ## 4 M04    Dancer model  people       no     Wood   16    0.6 Sometimes, the column values are literal but can be converted to standard R data structures to better represent the data. For example, released column in product_info only takes yes and no, which can be better represented with a logical vector. We can use <- to modify the column values, as we learned previously. However, it is usually better to create a new data frame with the existing columns properly adjusted and new columns added without polluting the original data. To do this, we can use transform: transform(product_table,   released = ifelse(released == "yes", TRUE, FALSE),   density = weight / size) ##    id      name  type   class released material size weight ## 1 M01     JeepX model vehicle     TRUE Plastics   50     NA ## 2 M02 AircraftX model vehicle     TRUE Plastics   85    3.0 ## 3 M03    Runner model  people     TRUE     Wood   15     NA ## 4 M04    Dancer model  people    FALSE     Wood   16    0.6 ## 5 T01    SupCar   toy vehicle     TRUE    Metal  120   10.0 ## 6 T02  SupPlane   toy vehicle    FALSE    Metal  350   45.0 ##      density ## 1         NA ## 2 0.03529412 ## 3         NA ## 4 0.03750000 ## 5 0.08333333 ## 6 0.12857143 The result is a new data frame with released converted to a logical vector and a new density column added. You can easily verify that product_table is not modified at all. Additionally, note that transform is like subset, as both functions use nonstandard evaluation to allow direct use of data frame columns as symbols in the arguments so that we don't have to type product_table$ all the time. Now, we will load another table into R. It is the test results of the quality, and durability of each product. We store the data in product_tests. product_tests <- read_csv("data/product-tests.csv") product_tests ##    id quality durability waterproof ## 1 T01      NA         10         no ## 2 T02      10          9         no ## 3 M01       6          4        yes ## 4 M02       6          5        yes ## 5 M03       5         NA        yes ## 6 M04       6          6        yes Note that the values in both quality and durability contain missing values (NA). To exclude all rows with missing values, we can use na.omit(): na.omit(product_tests) ##    id quality durability waterproof ## 2 T02      10          9         no ## 3 M01       6          4        yes ## 4 M02       6          5        yes ## 6 M04       6          6        yes Another way is to use complete.cases() to get a logical vector indicating all complete rows, without any missing value,: complete.cases(product_tests) ## [1] FALSE  TRUE  TRUE  TRUE FALSE  TRUE Then, we can use this logical vector to filter the data frame. For example, we can get the id  column of all complete rows as follows: product_tests[complete.cases(product_tests), "id"] ## [1] "T02" "M01" "M02" "M04" Or, we can get the id column of all incomplete rows: product_tests[!complete.cases(product_tests), "id"] ## [1] "T01" "M03" Note that product_info, product_stats and product_tests all share an id column, and we can merge them altogether. Unfortunately, there's no built-in function to merge an arbitrary number of data frames. We can only merge two existing data frames at a time, or we'll have to merge them recursively. merge(product_table, product_tests, by = "id") ##    id      name  type   class released material size weight ## 1 M01     JeepX model vehicle      yes Plastics   50     NA ## 2 M02 AircraftX model vehicle      yes Plastics   85    3.0 ## 3 M03    Runner model  people      yes     Wood   15     NA ## 4 M04    Dancer model  people       no     Wood   16    0.6 ## 5 T01    SupCar   toy vehicle      yes    Metal  120   10.0 ## 6 T02  SupPlane   toy vehicle       no    Metal  350   45.0 ##   quality durability waterproof ## 1       6          4        yes ## 2       6          5        yes ## 3       5         NA        yes ## 4       6          6        yes ## 5      NA         10         no ## 6      10          9         no Data wrangling with data.table In the previous section, we had an overview on how we can use built-in functions to work with data frames. Built-in functions work, but are usually verbose. In this section, let's use data.table, an enhanced version of data.frame, and see how it makes data manipulation much easier. Run install.packages("data.table") to install the package. As long as the package is ready, we can load the package and use fread() to read the data files as data.table objects. library(data.table) product_info <- fread("data/product-info.csv") product_stats <- fread("data/product-stats.csv") product_tests <- fread("data/product-tests.csv") toy_tests <- fread("data/product-toy-tests.csv") It is extremely easy to filter data in data.table. To select the first two rows, just use [1:2], which instead selects the first two columns for data.frame. product_info[1:2] ##     id     name type   class released ## 1: T01   SupCar  toy vehicle      yes ## 2: T02 SupPlane  toy vehicle       no To filter by logical conditions, just directly type columns names as variables without quotation as the expression is evaluated within the context of product_info: product_info[type == "model" & class == "people"] ##     id   name  type  class released ## 1: M03 Runner model people      yes ## 2: M04 Dancer model people       no It is easy to select or transform columns. product_stats[, .(id, material, density = size / weight)] ##     id material   density ## 1: T01    Metal 12.000000 ## 2: T02    Metal  7.777778 ## 3: M01 Plastics        NA ## 4: M02 Plastics 28.333333 ## 5: M03     Wood        NA ## 6: M04     Wood 26.666667 The data.table object also supports using key for subsetting, which can be much faster than using ==. We can set a column as key for each data.table: setkey(product_info, id) setkey(product_stats, id) setkey(product_tests, id) Then, we can use a value to directly select rows. product_info["M02"] ##     id      name  type   class released ## 1: M02 AircraftX model vehicle      yes We can also set multiple columns as key so as to use multiple values to subset it. setkey(toy_tests, id, date) toy_tests[.("T02", 20160303)] ##     id     date sample quality durability ## 1: T02 20160303     75       8          8 If two data.table objects share the same key, we can join them easily: product_info[product_tests] ##     id      name  type   class released quality durability ## 1: M01     JeepX model vehicle      yes       6          4 ## 2: M02 AircraftX model vehicle      yes       6          5 ## 3: M03    Runner model  people      yes       5         NA ## 4: M04    Dancer model  people       no       6          6 ## 5: T01    SupCar   toy vehicle      yes      NA         10 ## 6: T02  SupPlane   toy vehicle       no      10          9 ##    waterproof ## 1:        yes ## 2:        yes ## 3:        yes ## 4:        yes ## 5:         no ## 6:         no Instead of creating new data.table, in-place modification is also supported. The := sets the values of a column in place without the overhead of making copies and, thus, is much faster than using <-. product_info[, released := (released == "yes")] ##     id      name  type   class released ## 1: M01     JeepX model vehicle     TRUE ## 2: M02 AircraftX model vehicle     TRUE ## 3: M03    Runner model  people     TRUE ## 4: M04    Dancer model  people    FALSE ## 5: T01    SupCar   toy vehicle     TRUE ## 6: T02  SupPlane   toy vehicle    FALSE product_info ##     id      name  type   class released ## 1: M01     JeepX model vehicle     TRUE ## 2: M02 AircraftX model vehicle     TRUE ## 3: M03    Runner model  people     TRUE ## 4: M04    Dancer model  people    FALSE ## 5: T01    SupCar   toy vehicle     TRUE ## 6: T02  SupPlane   toy vehicle    FALSE Another important argument of subsetting a data.table is by, which is used to split the data into multiple parts and for each part the second argument (j) is evaluated. For example, the simplest usage of by is counting the records in each group. In the following code, we can count the number of both released and unreleased products: product_info[, .N, by = released] ##    released N ## 1:     TRUE 4 ## 2:    FALSE 2 The group can be defined by more than one variable. For example, a tuple of type and class can be a group, and for each group, we can count the number of records, as follows: product_info[, .N, by = .(type, class)] ##     type   class N ## 1: model vehicle 2 ## 2: model  people 2 ## 3:   toy vehicle 2 We can also perform the following statistical calculations for each group: product_tests[, .(mean_quality = mean(quality, na.rm = TRUE)),   by = .(waterproof)] ##    waterproof mean_quality ## 1:        yes         5.75 ## 2:         no        10.00 We can chain multiple [] in turn. In the following example, we will first join product_info and product_tests by a shared key id and then calculate the mean value of quality and durability for each group of type and class of released products. product_info[product_tests][released == TRUE,   .(mean_quality = mean(quality, na.rm = TRUE),     mean_durability = mean(durability, na.rm = TRUE)),   by = .(type, class)] ##     type   class mean_quality mean_durability ## 1: model vehicle            6             4.5 ## 2: model  people            5             NaN ## 3:   toy vehicle          NaN            10.0 Note that the values of the by columns will be unique in the resulted data.table; we can use keyby instead of by to ensure that it is automatically used as key by the resulted data.table. product_info[product_tests][released == TRUE,   .(mean_quality = mean(quality, na.rm = TRUE),     mean_durability = mean(durability, na.rm = TRUE)),   keyby = .(type, class)] ##     type   class mean_quality mean_durability ## 1: model  people            5             NaN ## 2: model vehicle            6             4.5 ## 3:   toy vehicle          NaN            10.0 The data.table package also provides functions to perform superfast reshaping of data. For example, we can use dcast() to spread id values along the x-axis as columns and align quality values to all possible date values along the y-axis. toy_quality <- dcast(toy_tests, date ~ id, value.var = "quality") toy_quality ##        date T01 T02 ## 1: 20160201   9   7 ## 2: 20160302  10  NA ## 3: 20160303  NA   8 ## 4: 20160403  NA   9 ## 5: 20160405   9  NA ## 6: 20160502   9  10 Although each month a test is conducted for each product, the dates may not exactly match with each other. This results in missing values if one product has a value on a day but the other has no corresponding value on exactly the same day. One way to fix this is to use year-month data instead of exact date. In the following code, we will create a new ym column that is the first 6 characters of toy_tests. For example, substr(20160101, 1, 6) will result in 201601. toy_tests[, ym := substr(toy_tests$date, 1, 6)] ##     id     date sample quality durability     ym ## 1: T01 20160201    100       9          9 201602 ## 2: T01 20160302    150      10          9 201603 ## 3: T01 20160405    180       9         10 201604 ## 4: T01 20160502    140       9          9 201605 ## 5: T02 20160201     70       7          9 201602 ## 6: T02 20160303     75       8          8 201603 ## 7: T02 20160403     90       9          8 201604 ## 8: T02 20160502     85      10          9 201605 toy_tests$ym ## [1] "201602" "201603" "201604" "201605" "201602" "201603" ## [7] "201604" "201605" This time, we will use ym for alignment instead of date: toy_quality <- dcast(toy_tests, ym ~ id, value.var = "quality") toy_quality ##        ym T01 T02 ## 1: 201602   9   7 ## 2: 201603  10   8 ## 3: 201604   9   9 ## 4: 201605   9  10 Now the missing values are gone, the quality scores of both products in each month are naturally presented. Sometimes, we will need to combine a number of columns into one that indicates the measure and another that stores the value. For example, the following code uses melt() to combine the two measures (quality and durability) of the original data into a column named measure and a column of the measured value. toy_tests2 <- melt(toy_tests, id.vars = c("id", "ym"),   measure.vars = c("quality", "durability"),   variable.name = "measure") toy_tests2 ##      id     ym    measure value ##  1: T01 201602    quality     9 ##  2: T01 201603    quality    10 ##  3: T01 201604    quality     9 ##  4: T01 201605    quality     9 ##  5: T02 201602    quality     7 ##  6: T02 201603    quality     8 ##  7: T02 201604    quality     9 ##  8: T02 201605    quality    10 ##  9: T01 201602 durability     9 ## 10: T01 201603 durability     9 ## 11: T01 201604 durability    10 ## 12: T01 201605 durability     9 ## 13: T02 201602 durability     9 ## 14: T02 201603 durability     8 ## 15: T02 201604 durability     8 ## 16: T02 201605 durability     9 The variable names are now contained in the data, which can be directly used by some packages. For example, we can use ggplot2 to plot data in such format. The following code is an example of a scatter plot with a facet grid of different combination of factors. library(ggplot2) ggplot(toy_tests2, aes(x = ym, y = value)) +   geom_point() +   facet_grid(id ~ measure) The graph generated is shown as follows: The plot can be easily manipulated because the grouping factor (measure) is contained as data rather than columns, which is easier to represent from the perspective of the ggplot2 package. ggplot(toy_tests2, aes(x = ym, y = value, color = id)) +   geom_point() +   facet_grid(. ~ measure) The graph generated is shown as follows: Summary In this article, we used both built-in functions and the data.table package to perform simple data manipulation tasks. Using built-in functions can be verbose while using data.table can be much easier and faster. However, the tasks in real-world data analysis can be much more complex than the examples we demonstrated, which also requires better R programming skills. It is helpful to have a good understanding on how nonstandard evaluation makes data.table so easy to work with, how environment works and scoping rules apply to make your code predictable, and so on. A universal and consistent understanding of how R basically works will certainly give you great confidence to write R code to work with data and enable you to learn packages very quickly. Resources for Article: Further resources on this subject: Supervised Machine Learning [article] Getting Started with Bootstrap [article] Basics of Classes and Objects [article]
Read more
  • 0
  • 0
  • 4136
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-moving-spatial-data-one-format-another
Packt
03 Nov 2015
29 min read
Save for later

Moving Spatial Data From One Format to Another

Packt
03 Nov 2015
29 min read
In this article by Michael Diener, author of the Python Geospatial Analysis Cookbook, we will cover the following topics: Converting a Shapefile to a PostGIS table using ogr2ogr Batch importing a folder of Shapefiles into PostGIS using ogr2ogr Batch exporting a list of tables from PostGIS to Shapefiles Converting an OpenStreetMap (OSM) XML to a Shapefile Converting a Shapefile (vector) to a GeoTiff (raster) Converting a GeoTiff (raster) to a Shapefile (vector) using GDAL (For more resources related to this topic, see here.) Introduction Geospatial data comes in hundreds of formats and massaging this data from one format to another is a simple task. The ability to convert data types, such as rasters or vectors, belongs to data wrangling tasks that are involved in geospatial analysis. Here is an example of a raster and vector dataset so you can see what I am talking about: Source: Michael Diener drawing The best practice is to run analysis functions or models on data stored in a common format, such as a Postgresql PostGIS database or a set of Shapefiles, in a common coordinate system. For example, running analysis on input data stored in multiple formats is also possible, but you can expect to find the devil in the details of your results if something goes wrong or your results are not what you expect. This article looks at some common data formats and demonstrates how to move these formats around from one to another with the most common tools. Converting a Shapefile to a PostGIS table using ogr2ogr The simplest way to transform data from one format to another is to directly use the ogr2ogr tool that comes with the installation of GDAL. This powerful tool can convert over 200 geospatial formats. In this solution, we will execute the ogr2ogr utility from within a Python script to execute generic vector data conversions. The python code is, therefore, used to execute this command-line tool and pass around variables that are needed to create your own scripts for data imports or exports. The use of this tool is also recommended if you are not really interested in coding too much and simply want to get the job done to move your data. A pure python solution is, of course, possible but it is definitely targeted more at developers (or python purists). Getting ready To run this script, you will need the GDAL utility application installed on your system. Windows users can visit OSGeo4W (http://trac.osgeo.org/osgeo4w) and download the 32-bit or 64-bit Windows installer as follows: Simply double-click on the installer to start it. Navigate to the bottommost option, Advanced Installation | Next. Click on Next to download from the Internet (this is the first default option). Click on Next to accept default location of path or change to your liking. Click on Next to accept the location of local saved downloads (default). Click on Next to accept the direct connection (default). Click on Next to select a default download site. Now, you should finally see the menu. Click on + to open the command-line utilities and you should see the following: Now, select gdal. The GDAL/OGR library and command line tools to install it. Click on Next to start downloading it, and then install it. For Ubuntu Linux users, use the following steps for installation: Execute this simple one-line command: $ sudo apt-get install gdal-bin This will get you up and running so that you can execute ogr2ogr directly from your terminal. Next, set up your Postgresql database using the PostGIS extension. First, we will create a new user to manage our new database and tables: Sudo su createuser  –U postgres –P pluto Enter a password for the new role. Enter the password again for the new role. Enter a password for postgres users since you're going to create a user with the help of the postgres user.The –P option prompts you to give the new user, called pluto, a password. For the following examples, our password is stars; I would recommend a much more secure password for your production database. Setting up your Postgresql database with the PostGIS extension in Windows is the same as setting it up in Ubuntu Linux. Perform the following steps to do this: Navigate to the c:Program FilesPostgreSQL9.3bin folder. Then, execute this command and follow the on-screen instructions as mentioned previously: Createuser.exe –U postgres –P pluto To create the database, we will use the command-line createdb command similar to the postgres user to create a database named py_geoan_cb. We will then assign the pluto user to be the database owner; here is the command to do this: $ sudo su createdb –O pluto –U postgres py_geoan_cb Windows users can visit the c:Program FilesPostgreSQL9.3bin and execute the createdb.exe command: createdb.exe –O pluto –U postgres py_geoan_cb Next, create the PostGIS extension for our newly created database: psql –U postgres -d py_geoan_cb -c "CREATE EXTENSION postgis;" Windows users can also execute psql from within the c:Program FilesPostgreSQL9.3bin folder: psql.exe –U postgres –d py_geoan_cb –c "CREATE EXTENSION postgis;" Lastly, create a schema called geodata to store the new spatial table. It is common to store spatial data in another schema outside the Postgresql default schema, public. Create the schema as follows: For Ubuntu Linux users: sudo -u postgres psql -d py_geoan_cb -c "CREATE SCHEMA geodata AUTHORIZATION pluto;" For Windows users: psql.exe –U postgres –d py_geoan_cb –c "CREATE SCHEMA geodata AUTHORIZATION pluto;" How to do it... Now let's get into the actual importing of our Shapefile into a PostGIS database that will automatically create a new table from our Shapefile: #!/usr/bin/env python # -*- coding: utf-8 -*- import subprocess # database options db_schema = "SCHEMA=geodata" overwrite_option = "OVERWRITE=YES" geom_type = "MULTILINESTRING" output_format = "PostgreSQL" # database connection string db_connection = """PG:host=localhost port=5432   user=pluto dbname=py_test password=stars""" # input shapefile input_shp = "../geodata/bikeways.shp" # call ogr2ogr from python subprocess.call(["ogr2ogr","-lco", db_schema, "-lco", overwrite_option, "-nlt", geom_type, "-f", output_format, db_connection,  input_shp]) Now we can call our script from the command line: $ python ch03-01_shp2pg.py How it works... We begin with importing the standard python module subprocess that will call the ogr2ogr command-line tool. Next, we'll set a range of variables that are used as input arguments and various options for ogr2ogr to execute. Starting with the SCHEMA=geodata Postgresql database, we'll set a nondefault database schema for the destination of our new table. It is best practice to store your spatial data tables in a separate schema outside the "public" schema, which is set as the default. This practice will make backups and restores much easier and keep your database better organized. Next, we'll create a overwrite_option variable that's set to "yes" so that we can overwrite any table with the same name when its created. This is helpful when you want to completely replace the table with new data, otherwise, it is recommended to use the -append option. We'll also specify the geometry type because, sometimes, ogr2ogr does not always guess the correct geometry type of our Shapefile so setting this value saves you any worry. Now, setting our output_format variable with the PostgreSQL keyword tells ogr2ogr that we want to output data into a Postgresql database. This is then followed by the db_connection variable, which specifies our database connection information. Do not forget that the database must already exist along with the "geodata" schema, otherwise, we will get an error. The last input_shp variable gives the full path to our Shapefile, including the .shp file ending. Now, we will call the subprocess module ,which will then call the ogr2ogr command-line tool and pass along the variable options required to run the tool. We'll pass this function an array of arguments, the first object in the array being the ogr2ogr command-line tool name. After this, we'll pass each option after another in the array to complete the call. Subprocess can be used to call any command-line tool directly. It takes a list of parameters that are separated by spaces. This passing of parameters is quite fussy, so make sure you follow along closely and don't add any extra spaces or commas. Last but not least, we need to execute our script from the command line to actually import our Shapefile by calling the python interpreter and passing the script. Then, head over to the PgAdmin Postgresql database viewer and see if it worked, or even better, open up Quantum GIS (www.qgis.org) and take a look at the newly created tables. See also If you would like to see the full list of options available with the ogr2ogr command, simple enter the following in the command line: $ ogr2ogr –help You will see the full list of options that are available. Also, visit http://gdal.org/ogr2ogr.html to read the required documentation. Batch importing a folder of Shapefiles into PostGIS using ogr2ogr We would like to extend our last script to loop over a folder full of Shapefiles and import them into PostGIS. Most importing tasks involve more than one file to import so this makes it a very practical task. How to do it... The following steps will batch import a folder of Shapefiles into PostGIS using ogr2ogr: Our script will reuse the previous code in the form of a function so that we can batch process a list of Shapefiles to import into the Postgresql PostGIS database. We will create our list of Shapefiles from a single folder for the sake of simplicity: #!/usr/bin/env python # -*- coding: utf-8 -*- import subprocess import os import ogr def discover_geom_name(ogr_type):     """     :param ogr_type: ogr GetGeomType()     :return: string geometry type name     """     return {ogr.wkbUnknown            : "UNKNOWN",             ogr.wkbPoint              : "POINT",             ogr.wkbLineString         : "LINESTRING",             ogr.wkbPolygon            : "POLYGON",             ogr.wkbMultiPoint         : "MULTIPOINT",             ogr.wkbMultiLineString    : "MULTILINESTRING",             ogr.wkbMultiPolygon       : "MULTIPOLYGON",             ogr.wkbGeometryCollection : "GEOMETRYCOLLECTION",             ogr.wkbNone               : "NONE",             ogr.wkbLinearRing         : "LINEARRING"}.get(ogr_type) def run_shp2pg(input_shp):     """     input_shp is full path to shapefile including file ending     usage:  run_shp2pg('/home/geodata/myshape.shp')     """     db_schema = "SCHEMA=geodata"     db_connection = """PG:host=localhost port=5432                     user=pluto dbname=py_geoan_cb password=stars"""     output_format = "PostgreSQL"     overwrite_option = "OVERWRITE=YES"     shp_dataset = shp_driver.Open(input_shp)     layer = shp_dataset.GetLayer(0)     geometry_type = layer.GetLayerDefn().GetGeomType()     geometry_name = discover_geom_name(geometry_type)     print (geometry_name)     subprocess.call(["ogr2ogr", "-lco", db_schema, "-lco", overwrite_option,                      "-nlt", geometry_name, "-skipfailures",                      "-f", output_format, db_connection, input_shp]) # directory full of shapefiles shapefile_dir = os.path.realpath('../geodata') # define the ogr spatial driver type shp_driver = ogr.GetDriverByName('ESRI Shapefile') # empty list to hold names of all shapefils in directory shapefile_list = [] for shp_file in os.listdir(shapefile_dir):     if shp_file.endswith(".shp"):         # apped join path to file name to outpout "../geodata/myshape.shp"         full_shapefile_path = os.path.join(shapefile_dir, shp_file)         shapefile_list.append(full_shapefile_path) # loop over list of Shapefiles running our import function for each_shapefile in shapefile_list:     run_shp2pg(each_shapefile)  print ("importing Shapefile: " + each_shapefile) Now, we can simply run our new script from the command line once again: $ python ch03-02_batch_shp2pg.py How it works... Here, we will reuse our code from the previous script but have converted it into a python function called run_shp2pg (input_shp), which takes exactly one argument to complete the path to the Shapefile we want to import. The input argument must include a Shapefile ending with .shp. We have a helper function that will get the geometry type as a string by reading in the Shapefile feature layer and outputting the geometry type so that the ogr commands know what to expect. This does not always work and some errors can occur. The skipfailures option will plow over any errors that are thrown during insert and will still populate our tables. To begin with, we need to define the folder that contains all our Shapefiles to be imported. Next up, we'll create an empty list object called shapefile_list that will hold a list of all the Shapefiles we want to import. The first for loop is used to get the list of all the Shapefiles in the directory specified using the standard python os.listdir() function. We do not want all the files in this folder; we only want files with ending with .shp, hence, the if statement will evaluate to True if the file ends with .shp. Once the .shp file is found, we need to append the file path to the file name to create a single string that holds the path plus the Shapefile name, and this is our variable called full_shapefile_path. The final part of this is to add each new file with its attached path to our shapefile_list list object. So, we now have our final list to loop through. It is time to loop through each Shapefile in our new list and run our run_shp2pg(input_shp) function for each Shapefile in the list by importing it into our Postgresql PostGIS database. See also If you have a lot of Shapefiles, and by this I mean mean hundred or more Shapefiles, performance will be one consideration and will, therefore, indicate that there are a lot of machines with free resources. Batch exporting a list of tables from PostGIS to Shapefiles We will now change directions and take a look at how we can batch export a list of tables from our PostGIS database into a folder of Shapefiles. We'll again use the ogr2ogr command-line tool from within a python script so that you can include it in your application programming workflow. Near the end, you can also see how this all of works in one single command line. How to do it... The script will fire the ogr2ogr command and loop over a list of tables to export the Shapefile format into an existing folder. So, let's take a look at how to do this using the following code: #!/usr/bin/env python # -*- coding: utf-8 -*- # import subprocess import os # folder to hold output Shapefiles destination_dir = os.path.realpath('../geodata/temp') # list of postGIS tables postgis_tables_list = ["bikeways", "highest_mountains"] # database connection parameters db_connection = """PG:host=localhost port=5432 user=pluto         dbname=py_geoan_cb password=stars active_schema=geodata""" output_format = "ESRI Shapefile" # check if destination directory exists if not os.path.isdir(destination_dir):     os.mkdir(destination_dir)     for table in postgis_tables_list:         subprocess.call(["ogr2ogr", "-f", output_format, destination_dir,                          db_connection, table])         print("running ogr2ogr on table: " + table) else:     print("oh no your destination directory " + destination_dir +           " already exist please remove it then run again") # commandline call without using python will look like this # ogr2ogr -f "ESRI Shapefile" mydatadump # PG:"host=myhost user=myloginname dbname=mydbname password=mypassword"   neighborhood parcels Now, we'll call our script from the command line as follows: $ python ch03-03_batch_postgis2shp.py How it works... Beginning with a simple import of our subprocess and os modules, we'll immediately define our destination directory where we want to store the exported Shapefiles. This variable is followed by the list of table names that we want to export. This list can only include files located in the same Postgresql schema. The schema is defined as the active_schema so that ogr2ogr knows where to find the tables to be exported. Once again, we'll define the output format as ESRI Shapefile. Now, we'll check whether the destination folder exists. If it does, continue and call our loop. Then, loop through the list of tables stored in our postgis_tables_list variable. If the destination folder does not exist, you will see an error printed on the screen. There's more... If you are programming an application, then executing the ogr2ogr command from inside your script is definitely quick and easy. On the other hand, for a one-off job, simply executing the command-line tool is what you want when you export your list of Shapefiles. To do this in a one-liner, please take a look at the following information box. A one line example of calling the ogr2ogr batch of the PostGIS table to Shapefiles is shown here if you simply want to execute this once and not in a scripting environment: ogr2ogr -f "ESRI Shapefile" /home/ch03/geodata/temp PG:"host=localhost user=pluto dbname=py_geoan_cb password=stars" bikeways highest_mountains The list of tables you want to export is located as a list separated by spaces. The destination location of the exported Shapefiles is ../geodata/temp. Note this this /temp directory must exist. Converting an OpenStreetMap (OSM) XML to a Shapefile OpenStreetMap (OSM) has a wealth of free data, but to use it with most other applications, we need to convert it to another format, such as a Shapefile or Postgresql PostGIS database. This recipe will use the ogr2ogr tool to do the conversion for us within a python script. The benefit derived here is simplicity. Getting ready To get started, you will need to download the OSM data at http://www.openstreetmap.org/export#map=17/37.80721/-122.47305 and saving the file (.osm) to your /ch03/geodata directory. The download button is located on the bar on the left-hand side, and when pressed, it should immediately start the download (refer to the following image). The area we are testing is from San Francisco, just before the golden gate bridge. If you choose to download another area from OSM feel free to, but make sure that you take a small area like preceding example link. If you select a larger area, the OSM web tool will give you a warning and disable the download button. The reason is simple: if the dataset is very large, it will most likely be better suited for another tool, such as osm2pgsql (http://wiki.openstreetmap.org/wiki/Osm2pgsql), for you conversion. If you need to get OSM data for a large area and want to export to Shapefile, it would be advisable to use another tool, such as osm2pgsql, which will first import your data to a Postgresql database. Then, export from the PostGIS database to Shapefile using the pgsql2shp tool.  A python tool to import OSM data into a PostGIS database is also available and is called imposm (located here at http://imposm.org/). Version 2 of it is written in python and version 3 is written in the "go" programming language if you want to give it a try. How to do it... Using the subprocess module, we will execute ogr2ogr to convert the OSM data that we downloaded into a new Shapefile: #!/usr/bin/env python # -*- coding: utf-8 -*- # convert / import osm xml .osm file into a Shapefile import subprocess import os import shutil # specify output format output_format = "ESRI Shapefile" # complete path to input OSM xml file .osm input_osm = '../geodata/OSM_san_francisco_westbluff.osm' # Windows users can uncomment these two lines if needed # ogr2ogr = r"c:/OSGeo4W/bin/ogr2ogr.exe" # ogr_info = r"c:/OSGeo4W/bin/ogrinfo.exe" # view what geometry types are available in our OSM file subprocess.call([ogr_info, input_osm]) destination_dir = os.path.realpath('../geodata/temp') if os.path.isdir(destination_dir):     # remove output folder if it exists     shutil.rmtree(destination_dir)     print("removing existing directory : " + destination_dir)     # create new output folder     os.mkdir(destination_dir)     print("creating new directory : " + destination_dir)     # list of geometry types to convert to Shapefile     geom_types = ["lines", "points", "multilinestrings", "multipolygons"]     # create a new Shapefile for each geometry type     for g_type in geom_types:         subprocess.call([ogr2ogr,                "-skipfailures", "-f", output_format,                  destination_dir, input_osm,                  "layer", g_type,                  "--config","OSM_USE_CUSTOM_INDEXING", "NO"])         print("done creating " + g_type) # if you like to export to SPATIALITE from .osm # subprocess.call([ogr2ogr, "-skipfailures", "-f", #         "SQLITE", "-dsco", "SPATIALITE=YES", #         "my2.sqlite", input_osm]) Now we can call our script from the command line: $ python ch03-04_osm2shp.py Go and have a look at your ../geodata folder to see the newly created Shapefiles, and try to open them up in Quantum GIS, which is a free GIS software (www.qgis.org) How it works... This script should be clear as we are using the subprocess module call to fire our ogr2ogr command-line tool. Specify our OSM dataset as an input file, including the full path to the file. The Shapefile name is not supplied as ogr2ogr and will output a set of Shapefiles, one for each geometry shape according to the geometry type it finds inside the OSM file. We only need to specify the name of the folder where we want ogr2ogr to export the Shapefiles to, automatically creating the folder if it does not exist. Windows users: if you do not have your ogr2ogr tool mapped to your environment variables, you can simply uncomment at lines 16 and 17 in the preceding code and replace the path shown with the path on your machine to the Windows executables. The first subprocess call prints out the screen that the geometry types have found inside the OSM file. This is helpful in most cases to help you identify what is available. Shapefiles can only support one geometry type per file, and this is why ogr2ogr outputs a folder full of Shapefiles, each one representing a separate geometry type. Lastly, we'll call subprocess to execute ogr2ogr, passing in the output "ESRI Shapefile" file type, output folder, and the name of the OSM dataset. Converting a Shapefile (vector) to a GeoTiff (raster) Moving data from format to format also includes moving from vector to raster or the other way around. In this recipe, we move from a vector (Shapefile) to a raster (GeoTiff) with the python gdal and ogr modules. Getting ready We need to be inside our virtual environment again, so fire it up so that we can access our gdal and ogr python modules. As usual, enter your python virtual environment with the workon pygeoan_cb command: $ source venvs/pygeoan_cb/bin/activate How to do it... Let's dive in and convert our golf course polygon Shapefile into a GeoTif; here is the code to do this: Import the ogr and gdal libraries, and then define the output pixel size along with the value that will be assigned to null: #!/usr/bin/env python # -*- coding: utf-8 -*- from osgeo import ogr from osgeo import gdal # set pixel size pixel_size = 1no_data_value = -9999 Set up the input Shapefile that we want to convert alongside the new GeoTiff raster that will be created when the script is executed: # Shapefile input name # input projection must be in cartesian system in meters # input wgs 84 or EPSG: 4326 will NOT work!!! input_shp = r'../geodata/ply_golfcourse-strasslach3857.shp' # TIF Raster file to be created output_raster = r'../geodata/ply_golfcourse-strasslach.tif' Now we need to create the input Shapefile object, so get the layer information and finally set the extent values: # Open the data source get the layer object # assign extent coordinates open_shp = ogr.Open(input_shp) shp_layer = open_shp.GetLayer() x_min, x_max, y_min, y_max = shp_layer.GetExtent() Here, we need to calculate the resolution distance to pixel value: # calculate raster resolution x_res = int((x_max - x_min) / pixel_size) y_res = int((y_max - y_min) / pixel_size) Our new raster type is a GeoTiff so we must explicitly tell this gdal to get the driver. The driver is then able to create a new GeoTiff by passing in the filename or the new raster we want to create. The x direction resolution is followed by the y direction resolution, and then our number of bands, which is, in this case, 1. Lastly, we'll set the new type of GDT_Byte raster: # set the image type for export image_type = 'GTiff' driver = gdal.GetDriverByName(image_type)   new_raster = driver.Create(output_raster, x_res, y_res, 1, gdal.GDT_Byte) new_raster.SetGeoTransform((x_min, pixel_size, 0, y_max, 0, -   pixel_size)) Now we can access the new raster band and assign the no data values and the inner data values for the new raster. All the inner values will receive a value of 255 similar to what we set for the burn_values variable: # get the raster band we want to export too raster_band = new_raster.GetRasterBand(1) # assign the no data value to empty cells raster_band.SetNoDataValue(no_data_value) # run vector to raster on new raster with input Shapefile gdal.RasterizeLayer(new_raster, [1], shp_layer, burn_values=[255]) Here we go! Lets run the script to see what our new raster looks like: $ python ch03-05_shp2raster.py The resulting raster should look like this if you open it  using QGIS (http://www.qgis.org): How it works... There are several steps involved in this code so please follow along as some points could lead to trouble if you are not sure what values to input. We start with the import of the gdal and ogr modules, respectively, since they will do the work for us by inputting a Shapefile (vector) and outputting a GeoTiff (raster). The pixel_size variable is very important since it will determine the size of the new raster we will create. In this example, we only have two polygons, so we'll set pixel_size = 1 to keep a fine border. If you have many polygons stretching across the globe in one Shapefile, it is wiser to set this value to 25 or more. Otherwise, you could end up with a 10 GB raster and your machine will run all night long! The no_data_value parameter is needed to tell GDAL what values to set in the empty space around our input polygons, and we set these to -9999 in order to be easily identified. Next, we'll simply set the input Shapefile stored in the EPSG:3857 web mercator and output GeoTiff. Check to make sure that you change the file names accordingly if you want to use some other dataset. We start by working with the OGR module to open the Shapefile and retrieve its layer information and the extent information. The extent is important because it is used to calculate the size of the output raster width and height values, which must be integers that are represented by the x_res and y_res variables. Note that the projection of your Shapefile must be in meters not degrees. This is very important since this will NOT work in EPSG:4326 or WGS 84, for example. The reason is that the coordinate units are LAT/LON. This means that WGS84 is not a flat plane projection and cannot be drawn as is. Our x_res and y_res values would evaluate to 0 since we cannot get a real ratio using degrees. This is a result of use not being able to simply subtract coordinate x from coordinate y because the units are in degrees and not in a flat plane meter projection. Now moving on to the raster setup, we'll define the type of raster we want to export as a Gtiff. Then, we can get the correct GDAL driver by the raster type. Once the raster type is set, we can create a new empty raster dataset, passing in the raster file name, width, and height of the raster in pixels, number of raster bands, and finally, the type of raster in GDAL terms, that is, the gdal.GDT_Byte. These five parameters are mandatory to create a new raster. Next, we'll call SetGeoTransform that handles transforming between pixel/line raster space and projection coordinates space. We want to activate the band 1 as it is the only band we have in our raster. Then, we'll assign the no data value for all our empty space around the polygon. The final step is to call the gdal.RasterizeLayer() function and pass in our new raster band Shapefile and the value to assign to the inside of our raster. The value of all the pixels inside our polygon will be 255. See also If you are interested, you can visit the command-line tool gdal_rasterize at http://www.gdal.org/gdal_rasterize.html. You can run this straight from the command line. Converting a raster (GeoTiff) to a vector (Shapefile) using GDAL We have now looked at how we can go from vector to raster, so it is time to go from raster to vector. This method is much more common because most of our vector data is derived from remotely sensed data such as satellite images, orthophotos, or some other remote sensing dataset such as lidar. Getting ready As usual, please enter your python virtual environment with the help of the workon pygeoan_cb command: $ source venvs/pygeoan_cb/bin/activate How to do it... Now let's begin: Import the ogr and gdal modules. Go straight ahead and open the raster that we want to convert by passing it the file name on disk. Then, get the raster band: #!/usr/bin/env python # -*- coding: utf-8 -*- from osgeo import ogr from osgeo import gdal #  get raster datasource open_image = gdal.Open( "../geodata/cadaster_borders-2tone-black-   white.png") input_band = open_image.GetRasterBand(3) Setup the output vector file as a Shapefile with output_shp, and then get the Shapefile driver. Now, we can create the output from our driver and create the layer: #  create output datasource output_shp = "../geodata/cadaster_raster" shp_driver = ogr.GetDriverByName("ESRI Shapefile") # create output file name output_shapefile = shp_driver.CreateDataSource( output_shp + ".shp" ) new_shapefile = output_shapefile.CreateLayer(output_shp, srs = None ) Our final step is to run the gdal.Polygonize function that does the heavy lifting by converting our raster to vector. gdal.Polygonize(input_band, None, new_shapefile, -1, [], callback=None) new_shapefile.SyncToDisk() Execute the new script. $ python ch03-06_raster2shp.py How it works... Working with ogr and gdal is similar in all our recipes; we must define the inputs and get the appropriate file driver to open the files. The GDAL library is very powerful and in only one line of code, we can convert a raster to a vector with the gdal. Polygonize function. All the preceding code is simply setup code to define which format we want to work with. We can then set up the appropriate driver to input and output our new file. Summary In this article we covered converting a Shapefile to a PostGIS table using ogr2ogr, batch importing a folder of Shapefiles into PostGIS using ogr2ogr, batch exporting a list of tables from PostGIS to Shapefiles, converting an OpenStreetMap (OSM) XML to a Shapefile, converting a Shapefile (vector) to a GeoTiff (raster), and converting a GeoTiff (raster) to a Shapefile (vector) using GDAL Resources for Article: Further resources on this subject: The Essentials of Working with Python Collections[article] Symbolizers[article] Preparing to Build Your Own GIS Application [article]
Read more
  • 0
  • 0
  • 4135

article-image-python-3-object-oriented-programming-managing-objects
Packt
12 Aug 2010
9 min read
Save for later

Python 3 Object Oriented Programming: Managing objects

Packt
12 Aug 2010
9 min read
(For more resources on Python 3, see here.) Managing objects The difference between these objects and most of the examples we've seen so far is that our examples tend to represent concrete ideas. Management objects are more like office managers; they don't do the actual "visible" work out on the floor, but without them, there would be no communication between departments and nobody would know what they are supposed to do. Analogously, the attributes on a management class tend to refer to other objects that do the "visible" work; the behaviors on such a class delegate to those other classes at the right time, and pass messages between them. As an example, we'll write a program that does a find and replace action for text files stored in a compressed ZIP file. We'll need objects to represent the ZIP file and each individual text file (luckily, we don't have to write these classes, they're available in the Python Standard Library). The manager object will be responsible for ensuring three steps occur in order: Unzipping the compressed file. Performing the find and replace action. Zipping up the new files. The class is initialized with the .zip filename and search and replace strings. We create a temporary directory to store the unzipped files in, so that the folder stays clean. We also add a useful helper method for internal use that helps identify an individual filename inside that directory: import sysimport osimport shutilimport zipfileclass ZipReplace: def __init__(self, filename, search_string, replace_string): self.filename = filename self.search_string = search_string self.replace_string = replace_string self.temp_directory = "unzipped-{}".format( filename) def _full_filename(self, filename): return os.path.join(self.temp_directory, filename) Then we create an overall "manager" method for each of the three steps. This method delegates responsibility to other methods. Obviously, we could do all three steps in one method, or indeed, in one script without ever creating an object. There are several advantages to separating the three steps: Readability: The code for each step is in a self-contained unit that is easy to read and understand. The method names describe what the method does, and no additional documentation is required to understand what is going on. Extensibility: If a subclass wanted to use compressed TAR files instead of ZIP files, it could override the zip and unzip methods without having to duplicate the find_replace method. Partitioning: An external class could create an instance of this class and call the find and replace method directly on some folder without having to zip the content. The delegation method is the first in the code below; the rest of the methods are included for completeness: def zip_find_replace(self): self.unzip_files() self.find_replace() self.zip_files() def unzip_files(self): os.mkdir(self.temp_directory) zip = zipfile.ZipFile(self.filename) try: zip.extractall(self.temp_directory) finally: zip.close() def find_replace(self): for filename in os.listdir(self.temp_directory): with open(self._full_filename(filename)) as file: contents = file.read() contents = contents.replace( self.search_string, self.replace_string) with open( self._full_filename(filename), "w") as file: file.write(contents) def zip_files(self): file = zipfile.ZipFile(self.filename, 'w') for filename in os.listdir(self.temp_directory): file.write( self._full_filename(filename), filename) shutil.rmtree(self.temp_directory)if __name__ == "__main__": ZipReplace(*sys.argv[1:4]).zip_find_replace() For brevity, the code for zipping and unzipping files is sparsely documented. Our current focus is on object-oriented design; if you are interested in the inner details of the zipfile module, refer to the documentation in the standard library, either online at http://docs.python.org/library/zipfile.html or by typing import zipfile ; help(zipfile) into your interactive interpreter. Note that this example only searches the top-level files in a ZIP file; if there are any folders in the unzipped content, they will not be scanned, nor will any files inside those folders. The last two lines in the code allow us to run the example from the command line by passing the zip filename, search string, and replace string as arguments: python zipsearch.py hello.zip hello hi Of course, this object does not have to be created from the command line; it could be imported from another module (to perform batch ZIP file processing) or accessed as part of a GUI interface or even a higher-level management object that knows what to do with ZIP files (for example to retrieve them from an FTP server or back them up to an external disk). As programs become more and more complex, the objects being modeled become less and less like physical objects. Properties are other abstract objects and methods are actions that change the state of those abstract objects. But at the heart of every object, no matter how complex, is a set of concrete properties and well-defined behaviors. Removing duplicate code Often the code in management style classes such as ZipReplace is quite generic and can be applied in many different ways. It is possible to use either composition or inheritance to help keep this code in one place, thus eliminating duplicate code. Before we look at any examples of this, let's discuss a tiny bit of theory. Specifically: why is duplicate code a bad thing? There are several reasons, but they all boil down to readability and maintainability. When we're writing a new piece of code that is similar to an earlier piece, the easiest thing to do is copy the old code and change whatever needs to change (variable names, logic, comments) to make it work in the new location. Alternatively, if we're writing new code that seems similar, but not identical to code elsewhere in the project, the easiest thing to do is write fresh code with similar behavior, rather than figure out how to extract the overlapping functionality. But as soon as someone has to read and understand the code and they come across duplicate blocks, they are faced with a dilemma. Code that might have made sense suddenly has to be understood. How is one section different from the other? How are they the same? Under what conditions is one section called? When do we call the other? You might argue that you're the only one reading your code, but if you don't touch that code for eight months it will be as incomprehensible to you as to a fresh coder. When we're trying to read two similar pieces of code, we have to understand why they're different, as well as how they're different. This wastes the reader's time; code should always be written to be readable first. I once had to try to understand someone's code that had three identical copies of the same three hundred lines of very poorly written code. I had been working with the code for a month before I realized that the three "identical" versions were actually performing slightly different tax calculations. Some of the subtle differences were intentional, but there were also obvious areas where someone had updated a calculation in one function without updating the other two. The number of subtle, incomprehensible bugs in the code could not be counted. Reading such duplicate code can be tiresome, but code maintenance is an even greater torment. As the preceding story suggests, keeping two similar pieces of code up to date can be a nightmare. We have to remember to update both sections whenever we update one of them, and we have to remember how the multiple sections differ so we can modify our changes when we are editing each of them. If we forget to update both sections, we will end up with extremely annoying bugs that usually manifest themselves as, "but I fixed that already, why is it still happening?" The result is that people who are reading or maintaining our code have to spend astronomical amounts of time understanding and testing it compared to if we had written the code in a non-repetitive manner in the first place. It's even more frustrating when we are the ones doing the maintenance. The time we save by copy-pasting existing code is lost the very first time we have to maintain it. Code is both read and maintained many more times and much more often than it is written. Comprehensible code should always be paramount. This is why programmers, especially Python programmers (who tend to value elegant code more than average), follow what is known as the Don't Repeat Yourself, or DRY principle. DRY code is maintainable code. My advice to beginning programmers is to never use the copy and paste feature of their editor. To intermediate programmers, I suggest they think thrice before they hit Ctrl + C. But what should we do instead of code duplication? The simplest solution is often to move the code into a function that accepts parameters to account for whatever sections are different. This isn't a terribly object-oriented solution, but it is frequently sufficient. For example, if we have two pieces of code that unzip a ZIP file into two different directories, we can easily write a function that accepts a parameter for the directory to which it should be unzipped instead. This may make the function itself slightly more difficult to read, but a good function name and docstring can easily make up for that, and any code that invokes the function will be easier to read. That's certainly enough theory! The moral of the story is: always make the effort to refactor your code to be easier to read instead of writing bad code that is only easier to write.
Read more
  • 0
  • 0
  • 4132

article-image-primitive-data-types-variables-and-operators-object-oriented-javascript
Packt
20 Oct 2009
10 min read
Save for later

Primitive Data Types, Variables, and Operators in Object-Oriented JavaScript

Packt
20 Oct 2009
10 min read
Let's get started. Variables Variables are used to store data. When writing programs, it is convenient to use variables instead of the actual data, as it's much easier to write pi instead of 3.141592653589793 especially when it happens several times inside your program. The data stored in a variable can be changed after it was initially assigned, hence the name "variable". Variables are also useful for storing data that is unknown to the programmer when the code is written, such as the result of later operations. There are two steps required in order to use a variable. You need to: Declare the variable Initialize it, that is, give it a value In order to declare a variable, you use the var statement, like this: var a;var thisIsAVariable;var _and_this_too;var mix12three; For the names of the variables, you can use any combination of letters, numbers, and the underscore character. However, you can't start with a number, which means that this is invalid: var 2three4five; To initialize a variable means to give it a value for the first (initial) time. You have two ways to do so: Declare the variable first, then initialize it, or Declare and initialize with a single statement An example of the latter is: var a = 1; Now the variable named a contains the value 1. You can declare (and optionally initialize) several variables with a single var statement; just separate the declarations with a comma: var v1, v2, v3 = 'hello', v4 = 4, v5; Variables are Case Sensitive Variable names are case-sensitive. You can verify this statement using the Firebug console. Try typing this, pressing Enter after each line: var case_matters = 'lower';var CASE_MATTERS = 'upper';case_mattersCASE_MATTERS To save keystrokes, when you enter the third line, you can only type ca and press the Tab key. The console will auto-complete the variable name to case_matters. Similarly, for the last line—type CA and press Tab. The end result is shown on the following figure. Throughout the rest of this article series, only the code for the examples will be given, instead of a screenshot: >>> var case_matters = 'lower';>>> var CASE_MATTERS = 'upper';>>> case_matters"lower">>> CASE_MATTERS"upper" The three consecutive greater-than signs (>>>) show the code that you type, the rest is the result, as printed in the console. Again, remember that when you see such code examples, you're strongly encouraged to type in the code yourself and experiment tweaking it a little here and there, so that you get a better feeling of how it works exactly. Operators Operators take one or two values (or variables), perform an operation, and return a value. Let's check out a simple example of using an operator, just to clarify the terminology. >>> 1 + 23 In this code: + is the operator The operation is addition The input values are 1 and 2 (the input values are also called operands) The result value is 3 Instead of using the values 1 and 2 directly in the operation, you can use variables. You can also use a variable to store the result of the operation, as the following example demonstrates: >>> var a = 1;>>> var b = 2;>>> a + 12>>> b + 24>>> a + b3>>> var c = a + b;>>> c3 The following table lists the basic arithmetic operators: Operator symbol Operation Example + Addition >>> 1 + 2 3 - Subtraction >>> 99.99 - 11 88.99 * Multiplication >>> 2 * 3 6 / Division >>> 6 / 4 1.5 % Modulo, the reminder of a division >>> 6 % 3 0 >>> 5 % 3 2 It's sometimes useful to test if a number is even or odd. Using the modulo operator it's easy. All odd numbers will return 1 when divided by 2, while all even numbers will return 0. >>> 4 % 2 0 >>> 5 % 2 1 ++ Increment a value by 1 Post-increment is when the input value is incremented after it's returned. >>> var a = 123; var b = a++; >>> b 123 >>> a 124 The opposite is pre-increment; the input value is first incremented by 1 and then returned. >>> var a = 123; var b = ++a; >>> b 124 >>> a 124 -- Decrement a value by 1 Post-decrement >>> var a = 123; var b = a--; >>> b 123 >>> a 122 Pre-decrement >>> var a = 123; var b = --a; >>> b 122 >>> a 122 When you type var a = 1; this is also an operation; it's the simple assignment operation and = is the simple assignment operator. There is also a family of operators that are a combination of an assignment and an arithmetic operator. These are called compound operators. They can make your code more compact. Let's see some of them with examples. >>> var a = 5;>>> a += 3;8 In this example a += 3; is just a shorter way of doing a = a + 3; >>> a -= 3;5 Here a -= 3; is the same as a = a - 3; Similarly: >>> a *= 2;10>>> a /= 5;2>>> a %= 2;0 In addition to the arithmetic and assignment operators discussed above, there are other types of operators, as you'll see later in this article series.   Primitive Data Types Any value that you use is of a certain type. In JavaScript, there are the following primitive data types: Number—this includes floating point numbers as well as integers, for example 1, 100, 3.14. String—any number of characters, for example "a", "one", "one 2 three". Boolean—can be either true or false. Undefined—when you try to access a variable that doesn't exist, you get the special value undefined. The same will happen when you have declared a variable, but not given it a value yet. JavaScript will initialize it behind the scenes, with the value undefined. Null—this is another special data type that can have only one value, the null value. It means no value, an empty value, nothing. The difference with undefined is that if a variable has a value null, it is still defined, it only happens that its value is nothing. You'll see some examples shortly. Any value that doesn't belong to one of the five primitive types listed above is an object. Even null is considered an object, which is a little awkward—having an object (something) that is actually nothing. The data types in JavaScript the data types are either: Primitive (the five types listed above), or Non-primitive (objects) Finding out the Value Type —the typeof Operator If you want to know the data type of a variable or a value, you can use the special typeof operator. This operator returns a string that represents the data type. The return values of using typeof can be one of the following—"number", "string", "boolean", "undefined", "object", or "function". In the next few sections, you'll see typeof in action using examples of each of the five primitive data types. Numbers The simplest number is an integer. If you assign 1 to a variable and then use the typeof operator, it will return the string "number". In the following example you can also see that the second time we set a variable's value, we don't need the var statement. >>> var n = 1;>>> typeof n;"number">>> n = 1234;>>> typeof n;"number" Numbers can also be floating point (decimals): >>> var n2 = 1.23;>>> typeof n;"number" You can call typeof directly on the value, without assigning it to a variable first: >>> typeof 123;"number" Octal and Hexadecimal Numbers When a number starts with a 0, it's considered an octal number. For example, the octal 0377 is the decimal 255. >>> var n3 = 0377;>>> typeof n3;"number">>> n3;255 The last line in the example above prints the decimal representation of the octal value. While you may not be very familiar with octal numbers, you've probably used hexadecimal values to define, for example, colors in CSS stylesheets. In CSS, you have several options to define a color, two of them being: Using decimal values to specify the amount of R (red), G (green) and B (blue) ranging from 0 to 255. For example rgb(0, 0, 0) is black and rgb(255, 0, 0) is red (maximum amount of red and no green or blue). Using hexadecimals, specifying two characters for each R, G and B. For example, #000000 is black and #ff0000 is red. This is because ff is the hexadecimal for 255. In JavaScript, you put 0x before a hexadecimal value (also called hex for short). >>> var n4 = 0x00;>>> typeof n4;"number">>> n4;0>>> var n5 = 0xff;>>> typeof n5;"number">>> n5;255 Exponent Literals 1e1 (can also be written as 1e+1 or 1E1 or 1E+1) represents the number one with one zero after it, or in other words 10. Similarly, 2e+3 means the number 2 with 3 zeros after it, or 2000. >>> 1e110>>> 1e+110>>> 2e+32000>>> typeof 2e+3;"number" 2e+3 means moving the decimal point 3 digits to the right of the number 2. There's also 2e-3 meaning you move the decimal point 3 digits to the left of the number 2. >>> 2e-30.002>>> 123.456E-30.123456>>> typeof 2e-3"number" Infinity There is a special value in JavaScript called Infinity. It represents a number too big for JavaScript to handle. Infinity is indeed a number, as typing typeof Infinity in the console will confirm. You can also quickly check that a number with 308 zeros is ok, but 309 zeros is too much. To be precise, the biggest number JavaScript can handle is 1.7976931348623157e+308 while the smallest is 5e-324. >>> InfinityInfinity>>> typeof Infinity"number">>> 1e309Infinity>>> 1e3081e+308 Dividing by 0 will give you infinity. >>> var a = 6 / 0;>>> aInfinity Infinity is the biggest number (or rather a little bigger than the biggest), but how about the smallest? It's infinity with a minus sign in front of it, minus infinity. >>> var i = -Infinity;>>> i-Infinity>>> typeof i"number" Does this mean you can have something that's exactly twice as big as Infinity—from 0 up to infinity and then from 0 down to minus infinity? Well, this is purely for amusement and there's no practical value to it. When you sum infinity and minus infinity, you don't get 0, but something that is called NaN (Not A Number). >>> Infinity - InfinityNaN>>> -Infinity + InfinityNaN Any other arithmetic operation with Infinity as one of the operands will give you Infinity: >>> Infinity - 20Infinity>>> -Infinity * 3-Infinity>>> Infinity / 2Infinity>>> Infinity - 99999999999999999Infinity NaN What was this NaN you saw in the example above? It turns out that despite its name, "Not A Number", NaN is a special value that is also a number. >>> typeof NaN"number">>> var a = NaN;>>> aNaN You get NaN when you try to perform an operation that assumes numbers but the operation fails. For example, if you try to multiply 10 by the character "f", the result is NaN, because "f" is obviously not a valid operand for a multiplication. >>> var a = 10 * "f";>>> aNaN NaN is contagious, so if you have even only one NaN in your arithmetic operation, the whole result goes down the drain. >>> 1 + 2 + NaNNaN
Read more
  • 0
  • 0
  • 4091

article-image-openlayers-overview-vector-layer
Packt
11 Apr 2011
7 min read
Save for later

OpenLayers: Overview of Vector Layer

Packt
11 Apr 2011
7 min read
OpenLayers 2.10 Beginner's Guide Create, optimize, and deploy stunning cross-browser web maps with the OpenLayers JavaScript web mapping library What is the Vector Layer? OpenLayers' Vector Class is generally used to display data on top of a map and allow real time interaction with the data. What does this mean? Basically, it means we can load in data from geospatial files, such as KML or GeoJSON files, and display the contents on a map, styling the data however we see fit. For example, take a look at this map: This shows a map with a Google layer as underlying base layer and a vector layer on top of it. The data (all the circles with numbers in them) are loaded in from a GeoJSON file, an open file format that many other applications support. In the vector layer, there are a bunch of data points throughout the map. Each dot on the map is an object in the vector layer, and these objects are referred to as Features. In this case, each feature is actually a cluster of data points—the numbers in each circle represent how many points belong to that cluster. This clustering behavior is something we can use out of the box with OpenLayers via the Strategy class. Before we get to that point, let's talk about one of the main things that separate a vector layer from other layers. What makes the Vector Layer special? With a raster image, what you see is what you get. If you were to look at some close up satellite imagery on your map and see a bunch of buildings clustered together, you wouldn't necessarily know any additional information about those buildings. You might not even know they are buildings. Since raster layers are made up of images, it is up to the user to interpret what they see. This isn't necessarily a bad thing, but vector layers provide much more. With a vector layer, you can show the actual geometry of the building and attach additional information to it—such as the value of it, who owns it, its square footage, etc. It's easy to put a vector layer on top of your existing raster layers and create features in a specific location. The Vector Layer is client side Another fundamental difference is that the vector layer is, generally, used as a client side layer. This means that, usually, interaction with the actual vector data happens only on the client side. When you navigate around the map, for instance, the vector layer does not send a request to a server to get more information about the layer. Once you get the initial data, it's in your browser and you do not have to request the same data again. Since, in most cases, the vector data is loaded on the client side, interaction with the vector layer usually happens nearly instantaneously. However, there are some limitations. The vector layer is dependent on the user's browser and computer. While most browsers other than Internet Explorer have been progressing exceptionally well and are becoming more powerful each day, limitations do exist. Due to browser limitations, too many features in a vector layer will start to slow things down. There is no hard number on the amount of features, but generally anything over a couple hundred of features will start to slow things down on most computers. However, there are many ways around this, such as deleting features when you don't need them, and we'll talk about performance issues in more depth later. Other uses With the vector layer, we can display any type of geometrical object we'd like—points, lines, polygons, squares, makers...any shape you can imagine. We can use the vector layer to draw lines or polygons and then calculate the distance between them. We can draw shapes and then export the data using a variety of formats, then import that data in other programs, such as Google Earth. What is a 'Vector'? In terms of graphics, there are essentially two types of images: raster and vector. Most images you see are raster images—meaning, basically, they are comprised of a grid of pixels and their quality degrades as you zoom in on them. A photograph, for example, would be a raster image. If you enlarge it, it tends to get blurry or stretched out. The majority of image files—.jpegs, .png, .gifs, any bitmap image—are raster images. A vector, on the other hand, uses geometrical shapes based on math equations to form an image. Meaning that when you zoom in, the quality is preserved. If you were to zoom in on a vector image of a circle, the lines would always appear curved—with raster image, the lines would appear straight, as raster images are made up of a grid of colors. Vector graphics are not constrained to a grid, so they preserve shape at all scales. Time for Action – creating a Vector Layer Let's begin by creating a basic vector layer. In this example, after you add some points and other feature types to your vector layer, try to zoom in. You'll notice that the points you added don't lose quality as you zoom in. We'll go over how it works afterwards. We'll start off by using a basic WMS layer: var wms_layer = new OpenLayers.Layer.WMS( 'OpenLayers WMS', 'http://vmap0.tiles.osgeo.org/wms/vmap0', {layers: 'basic'}, {} ); Now, let's create the vector layer itself. We'll use the default projection and default values for the vector layer, so to create the layer all we need to do is create it: var vector_layer = new OpenLayers.Layer.Vector('Basic Vector Layer') Add the layers to the map now: map.addLayers([wms_layer, vector_layer]); If we looked at the map now, we would just see a simple map—our vector layer does not have any data loaded into it, nor do we have any controls to let us add vector data. Let's add the EditingToolbar control to the map, which allows us to add points and draw polygons on a vector layer. To do so, we just need to instantiate an object from OpenLayers.Control.EditingToolbar and pass in a vector layer. We'll pass in the vector_layer object we previously created: map.addControl(new OpenLayers.Control.EditingToolbar(vector_ layer)); Take a look at the map now. You should see the EditingToolbar control (which is basically a panel control with control buttons). Selecting different controls will allow you to place vector objects (called features) on the vector layer. Play around with the EditingToolbar control and place a few different points / polygons on the map: Now, one more step. You've placed some features (points / polygons / lines / etc.) on the map, but if you were to refresh the page they would disappear. We can, however, get the information about those features and then export it to a geospatial file. We'll work with files later, but for now let's grab the information about the features we've created. To access the information about the vector layer's features, all we need to do is access its features array. In Firebug, type and run the following: map.layers[1].features You should see a bunch of objects listed, each object is a feature you placed on the map: [Object { layer=Object, more...}, Object { layer=Object, more...}, Object { layer=Object, more...}, ...] Now, if you expand one of those objects, you'll get the information about a feature. The geometry property is an anonymous object each feature has which contains geometry information. You can also see the methods of the feature objects—try playing around with different functions. You can access the individual features by using map.layers[1].features[x], where x is the index of the feature in the features array. For instance, to destroy the first feature which we added to the map we could use: map.layers[1].features[0].destroy(); What Just Happened? We just demonstrated how to create a basic vector layer and added features to it using the EditingToolbar control. Using the features array of the vector layer, we also destroyed some features. As you've just seen, it's not terribly difficult to start using the vector layer— pretty easy, in fact.
Read more
  • 0
  • 0
  • 4087
article-image-how-vector-features-are-displayed
Packt
30 Dec 2014
23 min read
Save for later

How Vector Features are Displayed

Packt
30 Dec 2014
23 min read
In this article by Erik Westra, author of the book Building Mapping Applications with QGIS, we will learn how QGIS symbols and renderers are used to control how vector features are displayed on a map. In addition to this, we will also learn saw how symbol layers work. The features within a vector map layer are displayed using a combination of renderer and symbol objects. The renderer chooses which symbol is to be used for a given feature, and the symbol does the actual drawing. There are three basic types of symbols defined by QGIS: Marker symbol: This displays a point as a filled circle Line symbol: This draws a line using a given line width and color Fill symbol: This draws the interior of a polygon with a given color These three types of symbols are implemented as subclasses of the qgis.core.QgsSymbolV2 class: qgis.core.QgsMarkerSymbolV2 qgis.core.QgsLineSymbolV2 qgis.core.QgsFillSymbolV2 You might be wondering why all these classes have "V2" in their name. This is a historical quirk of QGIS. Earlier versions of QGIS supported both an "old" and a "new" system of rendering, and the "V2" naming refers to the new rendering system. The old rendering system no longer exists, but the "V2" naming continues to maintain backward compatibility with existing code. Internally, symbols are rather complex, using "symbol layers" to draw multiple elements on top of each other. In most cases, however, you can make use of the "simple" version of the symbol. This makes it easier to create a new symbol without having to deal with the internal complexity of symbol layers. For example: symbol = QgsMarkerSymbolV2.createSimple({'width' : 1.0,                                        'color' : "255,0,0"}) While symbols draw the features onto the map, a renderer is used to choose which symbol to use to draw a particular feature. In the simplest case, the same symbol is used for every feature within a layer. This is called a single symbol renderer, and is represented by the qgis.core.QgsSingleSymbolRenderV2class. Other possibilities include: Categorized symbol renderer (qgis.core.QgsCategorizedSymbolRendererV2): This renderer chooses a symbol based on the value of an attribute. The categorized symbol renderer has a mapping from attribute values to symbols. Graduated symbol renderer (qgis.core.QgsGraduatedSymbolRendererV2): This type of renderer has a series of ranges of attribute values, and maps each range to an appropriate symbol. Using a single symbol renderer is very straightforward: symbol = ... renderer = QgsSingleSymbolRendererV2(symbol) layer.setRendererV2(renderer) To use a categorized symbol renderer, you first define a list of qgis.core.QgsRendererCategoryV2 objects, and then use that to create the renderer. For example: symbol_male = ... symbol_female = ...   categories = [] categories.append(QgsRendererCategoryV2("M", symbol_male, "Male")) categories.append(QgsRendererCategoryV2("F", symbol_female,                                        "Female"))   renderer = QgsCategorizedSymbolRendererV2("", categories) renderer.setClassAttribute("GENDER") layer.setRendererV2(renderer) Notice that the QgsRendererCategoryV2 constructor takes three parameters: the desired value, the symbol to use, and the label used to describe that category. Finally, to use a graduated symbol renderer, you define a list of qgis.core.QgsRendererRangeV2 objects and then use that to create your renderer. For example: symbol1 = ... symbol2 = ...   ranges = [] ranges.append(QgsRendererRangeV2(0, 10, symbol1, "Range 1")) ranges.append(QgsRendererRange(11, 20, symbol2, "Range 2"))   renderer = QgsGraduatedSymbolRendererV2("", ranges) renderer.setClassAttribute("FIELD") layer.setRendererV2(renderer) Working with symbol layers Internally, symbols consist of one or more symbol layers that are displayed one on top of the other to draw the vector feature: The symbol layers are drawn in the order in which they are added to the symbol. So, in this example, Symbol Layer 1 will be drawn before Symbol Layer 2. This has the effect of drawing the second symbol layer on top of the first. Make sure you get the order of your symbol layers correct, or you may find a symbol layer completely obscured by another layer. While the symbols we have been working with so far have had only one layer, there are some clever tricks you can perform with multilayer symbols. When you create a symbol, it will automatically be initialized with a default symbol layer. For example, a line symbol (an instance of QgsLineSymbolV2) will be created with a single layer of type QgsSimpleLineSymbolLayerV2. This layer is used to draw the line feature onto the map. To work with symbol layers, you need to remove this default layer and replace it with your own symbol layer or layers. For example: symbol = QgsSymbolV2.defaultSymbol(layer.geometryType()) symbol.deleteSymbolLayer(0) # Remove default symbol layer.   symbol_layer_1 = QgsSimpleFillSymbolLayerV2() symbol_layer_1.setFillColor(QColor("yellow"))   symbol_layer_2 = QgsLinePatternFillSymbolLayer() symbol_layer_2.setLineAngle(30) symbol_layer_2.setDistance(2.0) symbol_layer_2.setLineWidth(0.5) symbol_layer_2.setColor(QColor("green"))   symbol.appendSymbolLayer(symbol_layer_1) symbol.appendSymbolLayer(symbol_layer_2) The following methods can be used to manipulate the layers within a symbol: symbol.symbolLayerCount(): This returns the number of symbol layers within this symbol symbol.symbolLayer(index): This returns the given symbol layer within the symbol. Note that the first symbol layer has an index of zero. symbol.changeSymbolLayer(index, symbol_layer): This replaces a given symbol layer within the symbol symbol.appendSymbolLayer(symbol_layer): This appends a new symbol layer to the symbol symbol.insertSymbolLayer(index, symbol_layer): This inserts a symbol layer at a given index symbol.deleteSymbolLayer(index): This removes the symbol layer at the given index Remember that to use the symbol once you've created it, you create an appropriate renderer and then assign that renderer to your map layer. For example: renderer = QgsSingleSymbolRendererV2(symbol) layer.setRendererV2(renderer) The following symbol layer classes are available for you to use: PyQGIS class Description Example QgsSimpleMarkerSymbolLayerV2 This displays a point geometry as a small colored circle.   QgsEllipseSymbolLayerV2 This displays a point geometry as an ellipse.   QgsFontMarkerSymbolLayerV2 This displays a point geometry as a single character. You can choose the font and character to be displayed.   QgsSvgMarkerSymbolLayerV2 This displays a point geometry using a single SVG format image.   QgsVectorFieldSymbolLayer This displays a point geometry by drawing a displacement line. One end of the line is the coordinate of the point, while the other end is calculated using attributes of the feature.   QgsSimpleLineSymbolLayerV2 This displays a line geometry or the outline of a polygon geometry using a line of a given color, width, and style.   QgsMarkerLineSymbolLayerV2 This displays a line geometry or the outline of a polygon geometry by repeatedly drawing a marker symbol along the length of the line.   QgsSimpleFillSymbolLayerV2 This displays a polygon geometry by filling the interior with a given solid color and then drawing a line around the perimeter.   QgsGradientFillSymbolLayerV2 This fills the interior of a polygon geometry using a color or grayscale gradient.   QgsCentroidFillSymbolLayerV2 This draws a simple dot at the centroid of a polygon geometry.   QgsLinePatternFillSymbolLayer This draws the interior of a polygon geometry using a repeated line. You can choose the angle, width, and color to use for the line.   QgsPointPatternFillSymbolLayer This draws the interior of a polygon geometry using a repeated point.   QgsSVGFillSymbolLayer This draws the interior of a polygon geometry using a repeated SVG format image.   These predefined symbol layers, either individually or in various combinations, give you enormous flexibility in how features are to be displayed. However, if these aren't enough for you, you can also implement your own symbol layers using Python. We will look at how this can be done later in this article. Combining symbol layers By combining symbol layers, you can achieve a range of complex visual effects. For example, you could combine an instance of QgsSimpleMarkerSymbolLayerV2 with a QgsVectorFieldSymbolLayer to display a point geometry using two symbols at once: One of the main uses of symbol layers is to draw different LineString or PolyLine symbols to represent different types of roads. For example, you can draw a complex road symbol by combining multiple symbol layers, like this: This effect is achieved using three separate symbol layers: Here is the Python code used to generate the above map symbol: symbol = QgsLineSymbolV2.createSimple({}) symbol.deleteSymbolLayer(0) # Remove default symbol layer.   symbol_layer = QgsSimpleLineSymbolLayerV2() symbol_layer.setWidth(4) symbol_layer.setColor(QColor("light gray")) symbol_layer.setPenCapStyle(Qt.FlatCap) symbol.appendSymbolLayer(symbol_layer)   symbol_layer = QgsSimpleLineSymbolLayerV2() symbol_layer.setColor(QColor("black")) symbol_layer.setWidth(2) symbol_layer.setPenCapStyle(Qt.FlatCap) symbol.appendSymbolLayer(symbol_layer)   symbol_layer = QgsSimpleLineSymbolLayerV2() symbol_layer.setWidth(1) symbol_layer.setColor(QColor("white")) symbol_layer.setPenStyle(Qt.DotLine) symbol.appendSymbolLayer(symbol_layer) As you can see, you can set the line width, color, and style to create whatever effect you want. As always, you have to define the layers in the correct order, with the back-most symbol layer defined first. By combining line symbol layers in this way, you can create almost any type of road symbol that you want. You can also use symbol layers when displaying polygon geometries. For example, you can draw QgsPointPatternFillSymbolLayer on top of QgsSimpleFillSymbolLayerV2 to have repeated points on top of a simple filled polygon, like this: Finally, you can make use of transparency to allow the various symbol layers (or entire symbols) to blend into each other. For example, you can create a pinstripe effect by combining two symbol layers, like this: symbol = QgsFillSymbolV2.createSimple({}) symbol.deleteSymbolLayer(0) # Remove default symbol layer.   symbol_layer = QgsGradientFillSymbolLayerV2() symbol_layer.setColor2(QColor("dark gray")) symbol_layer.setColor(QColor("white")) symbol.appendSymbolLayer(symbol_layer)   symbol_layer = QgsLinePatternFillSymbolLayer() symbol_layer.setColor(QColor(0, 0, 0, 20)) symbol_layer.setLineWidth(2) symbol_layer.setDistance(4) symbol_layer.setLineAngle(70) symbol.appendSymbolLayer(symbol_layer) The result is quite subtle and visually pleasing: In addition to changing the transparency for a symbol layer, you can also change the transparency for the symbol as a whole. This is done by using the setAlpha() method, like this: symbol.setAlpha(0.3) The result looks like this: Note that setAlpha() takes a floating point number between 0.0 and 1.0, while the transparency of a QColor object, like the ones we used earlier, is specified using an alpha value between 0 and 255. Implementing symbol layers in Python If the built-in symbol layers aren't flexible enough for your needs, you can implement your own symbol layers using Python. To do this, you create a subclass of the appropriate type of symbol layer (QgsMarkerSymbolLayerV2, QgsLineSymbolV2, or QgsFillSymbolV2) and implement the various drawing methods yourself. For example, here is a simple marker symbol layer that draws a cross for a Point geometry: class CrossSymbolLayer(QgsMarkerSymbolLayerV2):    def __init__(self, length=10.0, width=2.0):        QgsMarkerSymbolLayerV2.__init__(self)        self.length = length        self.width = width   def layerType(self):        return "Cross"   def properties(self):        return {'length' : self.length,               'width' : self.width}      def clone(self): return CrossSymbolLayer(self.length, self.width)      def startRender(self, context):        self.pen = QPen()        self.pen.setColor(self.color()) self.pen.setWidth(self.width)      def stopRender(self, context): self.pen = None   def renderPoint(self, point, context):        left = point.x() - self.length        right = point.x() + self.length        bottom = point.y() - self.length        top = point.y() + self.length          painter = context.renderContext().painter()        painter.setPen(self.pen)        painter.drawLine(left, bottom, right, top)        painter.drawLine(right, bottom, left, top) Using this custom symbol layer in your code is straightforward: symbol = QgsMarkerSymbolV2.createSimple({}) symbol.deleteSymbolLayer(0)   symbol_layer = CrossSymbolLayer() symbol_layer.setColor(QColor("gray"))   symbol.appendSymbolLayer(symbol_layer) Running this code will draw a cross at the location of each point geometry, as follows: Of course, this is a simple example, but it shows you how to use custom symbol layers implemented in Python. Let's now take a closer look at the implementation of the CrossSymbolLayer class, and see what each method does: __init__(): Notice how the __init__ method accepts parameters that customize the way the symbol layer works. These parameters, which should always have default values assigned to them, are the properties associated with the symbol layer. If you want to make your custom symbol available within the QGIS Layer Properties window, you will need to register your custom symbol layer and tell QGIS how to edit the symbol layer's properties. We will look at this shortly. layerType(): This method returns a unique name for your symbol layer. properties(): This should return a dictionary that contains the various properties used by this symbol layer. The properties returned by this method will be stored in the QGIS project file, and used later to restore the symbol layer. clone(): This method should return a copy of the symbol layer. Since we have defined our properties as parameters to the __init__ method, implementing this method simply involves creating a new instance of the class and copying the properties from the current symbol layer to the new instance. startRender(): This method is called before the first feature in the map layer is rendered. This can be used to define any objects that will be required to draw the feature. Rather than creating these objects each time, it is more efficient (and therefore faster) to create them only once to render all the features. In this example, we create the QPen object that we will use to draw the Point geometries. stopRender(): This method is called after the last feature has been rendered. This can be used to release the objects created by the startRender() method. renderPoint(): This is where all the work is done for drawing point geometries. As you can see, this method takes two parameters: the point at which to draw the symbol, and the rendering context (an instance of QgsSymbolV2RenderContext) to use for drawing the symbol. The rendering context provides various methods for accessing the feature being displayed, as well as information about the rendering operation, the current scale factor, etc. Most importantly, it allows you to access the PyQt QPainter object needed to actually draw the symbol onto the screen. The renderPoint() method is only used for symbol layers that draw point geometries. For line geometries, you should implement the renderPolyline() method, which has the following signature: def renderPolyline(self, points, context): The points parameter will be a QPolygonF object containing the various points that make up the LineString, and context will be the rendering context to use for drawing the geometry. If your symbol layer is intended to work with polygons, you should implement the renderPolygon() method, which looks like this: def renderPolygon(self, outline, rings, context): Here, outline is a QPolygonF object that contains the points that make up the exterior of the polygon, and rings is a list of QPolygonF objects that define the interior rings or "holes" within the polygon. As always, context is the rendering context to use when drawing the geometry. A custom symbol layer created in this way will work fine if you just want to use it within your own external PyQGIS application. However, if you want to use a custom symbol layer within a running copy of QGIS, and in particular, if you want to allow end users to work with the symbol layer using the Layer Properties window, there are some extra steps you will have to take, which are as follows: If you want the symbol to be visually highlighted when the user clicks on it, you will need to change your symbol layer's renderXXX() method to see if the feature being drawn has been selected by the user, and if so, change the way it is drawn. The easiest way to do this is to change the geometry's color. For example: if context.selected():    color = context.selectionColor() else:    color = self.color To allow the user to edit the symbol layer's properties, you should create a subclass of QgsSymbolLayerV2Widget, which defines the user interface to edit the properties. For example, a simple widget for the purpose of editing the length and width of a CrossSymbolLayer can be defined as follows: class CrossSymbolLayerWidget(QgsSymbolLayerV2Widget):    def __init__(self, parent=None):        QgsSymbolLayerV2Widget.__init__(self, parent)        self.layer = None          self.lengthField = QSpinBox(self)        self.lengthField.setMinimum(1)        self.lengthField.setMaximum(100)        self.connect(self.lengthField,                      SIGNAL("valueChanged(int)"),                      self.lengthChanged)          self.widthField = QSpinBox(self)        self.widthField.setMinimum(1)        self.widthField.setMaximum(100)        self.connect(self.widthField,                      SIGNAL("valueChanged(int)"),                      self.widthChanged)          self.form = QFormLayout()        self.form.addRow('Length', self.lengthField)        self.form.addRow('Width', self.widthField)          self.setLayout(self.form)      def setSymbolLayer(self, layer):        if layer.layerType() == "Cross":            self.layer = layer            self.lengthField.setValue(layer.length)            self.widthField.setValue(layer.width)      def symbolLayer(self):        return self.layer      def lengthChanged(self, n):        self.layer.length = n        self.emit(SIGNAL("changed()"))      def widthChanged(self, n):        self.layer.width = n        self.emit(SIGNAL("changed()")) We define the contents of our widget using the standard __init__() initializer. As you can see, we define two fields, lengthField and widthField, which let the user change the length and width properties respectively, for our symbol layer. The setSymbolLayer() method tells the widget which QgsSymbolLayerV2 object to use, while the symbolLayer() method returns the QgsSymbolLayerV2 object this widget is editing. Finally, the two XXXChanged() methods are called when the user changes the value of the fields, allowing us to update the symbol layer's properties to match the value set by the user. Finally, you will need to register your symbol layer. To do this, you create a subclass of QgsSymbolLayerV2AbstractMetadata and pass it to the QgsSymbolLayerV2Registry object's addSymbolLayerType() method. Here is an example implementation of the metadata for our CrossSymbolLayer class, along with the code to register it within QGIS: class CrossSymbolLayerMetadata(QgsSymbolLayerV2AbstractMetadata):    def __init__(self):        QgsSymbolLayerV2AbstractMetadata.__init__(self, "Cross", "Cross marker", QgsSymbolV2.Marker)      def createSymbolLayer(self, properties):        if "length" in properties:            length = int(properties['length'])        else:            length = 10        if "width" in properties:            width = int(properties['width'])        else:            width = 2        return CrossSymbolLayer(length, width)      def createSymbolLayerWidget(self, layer):        return CrossSymbolLayerWidget()   registry = QgsSymbolLayerV2Registry.instance() registry.addSymbolLayerType(CrossSymbolLayerMetadata()) Note that the parameters of QgsSymbolLayerV2AbstractMetadata.__init__() are as follows: The unique name for the symbol layer, which must match the name returned by the symbol layer's layerType() method. A display name for this symbol layer, as shown to the user within the Layer Properties window. The type of symbol that this symbol layer will be used for. The createSymbolLayer() method is used to restore the symbol layer based on the properties stored in the QGIS project file when the project was saved. The createSymbolLayerWidget() method is called to create the user interface widget that lets the user view and edit the symbol layer's properties. Implementing renderers in Python If you need to choose symbols based on more complicated criteria than what the built-in renderers will provide, you can write your own custom QgsFeatureRendererV2 subclass using Python. For example, the following Python code implements a simple renderer that alternates between odd and even symbols as point features are displayed: class OddEvenRenderer(QgsFeatureRendererV2):    def __init__(self): QgsFeatureRendererV2.__init__(self, "OddEvenRenderer")        self.evenSymbol = QgsMarkerSymbolV2.createSimple({})        self.evenSymbol.setColor(QColor("light gray"))        self.oddSymbol = QgsMarkerSymbolV2.createSimple({})        self.oddSymbol.setColor(QColor("black"))        self.n = 0      def clone(self):        return OddEvenRenderer()      def symbolForFeature(self, feature):        self.n = self.n + 1        if self.n % 2 == 0:            return self.evenSymbol        else:            return self.oddSymbol      def startRender(self, context, layer):        self.n = 0        self.oddSymbol.startRender(context)        self.evenSymbol.startRender(context)      def stopRender(self, context):        self.oddSymbol.stopRender(context)        self.evenSymbol.stopRender(context)      def usedAttributes(self):        return [] Using this renderer will cause the various point geometries to be displayed in alternating colors, for example: Let's take a closer look at how this class was implemented, and what the various methods do: __init__(): This is your standard Python initializer. Notice how we have to provide a unique name for the renderer when calling the QgsFeatureRendererV2.__init__() method; this is used to keep track of the various renderers within QGIS itself. clone(): This creates a copy of this renderer. If your renderer uses properties to control how it works, this method should copy those properties into the new renderer object. symbolForFeature(): This returns the symbol to use for drawing the given feature. startRender(): This prepares to start rendering the features within the map layer. As the renderer can make use of multiple symbols, you need to implement this so that your symbols are also given a chance to prepare for rendering. stopRender(): This finishes rendering the features. Once again, you need to implement this so that your symbols can have a chance to clean up once the rendering process has finished. usedAttributes():This method should be implemented to return the list of attributes that the renderer requires if your renderer makes use of feature attributes to choose between the various symbols,. If you wish, you can also implement your own widget that lets the user change the way the renderer works. This is done by subclassing QgsRendererV2Widget and setting up the widget to edit the renderer's various properties in the same way that we implemented a subclass of QgsSymbolLayerV2Widget to edit the properties for a symbol layer. You will also need to provide metadata about your new renderer (by subclassing QgsRendererV2AbstractMetadata) and use the QgsRendererV2Registry object to register your new renderer. If you do this, the user will be able to select your custom renderer for new map layers, and change the way your renderer works by editing the renderer's properties. Summary In this article, we learned how QGIS symbols and renderers are used to control how vector features are displayed on a map. We saw that there are three standard types of symbols: marker symbols for drawing points, line symbols for drawing lines, and fill symbols for drawing the interior of a polygon. We then learned how to instantiate a "simple" version of each of these symbols for use in your programs. We next looked at the built-in renderers, and how these can be used to choose the same symbol for every feature (using the QgsSingleSymbolRenderV2 class), to select a symbol based on the exact value of an attribute (using QgsCategorizedSymbolRendererV2), and to choose a symbol based on a range of attribute values (using the QgsGraduatedSymbolRendererV2 class). We then saw how symbol layers work, and how to manipulate the layers within a symbol. We looked at all the different types of symbol layers built into QGIS, and learned how they can be combined to produce sophisticated visual effects. Finally, we saw how to implement our own symbol layers using Python, and how to write your own renderer from scratch if one of the existing renderer classes doesn't meet your needs. Using these various PyQGIS classes, you have an extremely powerful set of tools at your disposal for displaying vector data within a map. While simple visual effects can be achieved with a minimum of fuss, you can produce practically any visual effect you want using an appropriate combination of built-in or custom-written QGIS symbols and renderers. Resources for Article: Further resources on this subject: Combining Vector and Raster Datasets [article] QGIS Feature Selection Tools [article] Creating a Map [article]
Read more
  • 0
  • 0
  • 4082

article-image-web-services-apache-ofbiz
Packt
14 Sep 2010
12 min read
Save for later

Web Services in Apache OFBiz

Packt
14 Sep 2010
12 min read
  Apache OfBiz Cookbook Over 60 simple but incredibly effective recipes for taking control of OFBiz Optimize your OFBiz experience and save hours of frustration with this timesaving collection of practical recipes covering a wide range of OFBiz topics. Get answers to the most commonly asked OFBiz questions in an easy-to-digest reference style of presentation. Discover insights into OFBiz design, implementation, and best practices by exploring real-life solutions. Each recipe in this Cookbook is crafted to describe not only "how" to accomplish a specific task, but also "why" the technique works to ensure you get the most out of your OFBiz implementation. Read more about this book (For more resources on Apache see here.) Introduction Ask five people what "web services" are and you will likely get at least six different opinions. Because the term evokes such ambiguity, we need to set ground rules for this article and define what we mean by "web services". Therefore, for the purposes of this book, "web services" are the interactive exchange of messages from one system to another using the Internet as the network transport and HTTP/HTTPS as the messaging protocol. Message exchange transpires without human intervention and may be one-way—that is, called without an immediate response expected—or two-way. Web services operate as producer/consumer systems where the producer—called the "service provider"—offers one or more "services" to the "consumer"—sometimes referred to as the "client". In the web services world, Internet-based service providers advertise and deliver service from locations throughout the Web. The Internet, and the Web in particular, serve as the highway over which potential web service clients travel to find service providers, make contact, and deliver products. Service-oriented by design, OFBiz is a natural for building and deploying both web service clients and web service providers. Any OFBiz web application (webapp) may both consume web services and act as a web service provider within the same application or in an unlimited number of OFBiz webapps. Within the web service producer/consumer model, service providers are responsible for accepting and validating requests for service and delivering the product. Consumers must find service providers, request service, and accept delivery of the product. There are a number of ad-hoc and formal standards that have evolved over the last few years to help facilitate the business of enabling web services, including, but not limited to, URL parameter passing, XML-RPC, and Simple Object Access Protocol (SOAP) based messaging. OFBiz provides builtin support with tools and integration points to make implementing both service provider and consumer web services a snap. Requesting web services using URL parameters There are many web service providers that require nothing more than URL parameter passing to request a service. These service providers take HTTP/HTTPS request parameters as appended to a prospective consumer's URL, and process requests according to the passed parameter values. Services are delivered back to the requestor through the client's HTTP/ HTTPS response message or as a separate HTTP/HTTPS request message exchange. An example of such a real world web service is the PayPal Payments Standard payment processing service . This web service expects requests to come in on an advertised URL with request particulars appended to the URL as request parameters. Prospective consuming systems send HTTP/HTTPS request messages to the PayPal URL asking for service. Once a request for service has been accepted by PayPal, the Payments Standard web service responds and delivers service using the HTTP/HTTPS response message. A separate web service, also involving PayPal, called the Instant Payment Notification (IPN) web service is an example of a web service in which parameters are passed on the URL from the service provider to a consumer such as OFBiz. In this case, OFBiz listens on a configured URL for an HTTP/HTTPS request message from PayPal. When one is received, OFBiz responds appropriately, taking delivery of PayPal service. Note: to implement a PayPal IPN client within an OFBiz web application, you must provide PayPal a destination URL that maps to a valid OFBiz request-map, and then process any incoming URL parameters according to the PayPal IPN directions. Getting ready To act as a web service client and pass parameters on the URL within an OFBiz Java Event or Service, make sure you include the following Java packages in your program: import java.io.IOException;import java.net.URL;import java.net.URLConnection;import javax.Servlet.http.HttpServletRequest;import javax.servlet.http.HttpServletResponse; How to do it... To request a service using URL parameters, follow these steps: Within an existing OFBiz Java program (either an Event, Service, or any other) method, create a Java HashMap containing the name/value pairs that you wish to pass on the URL. For example: Map parameters = UtilMisc.toMap("param1", "A", "param2", "B"); Use the OFBiz-provided UtilHttp.urlEncodeArgs method to properly encode the parameters directly from the HashMap into a Java String. For example: String encodedParameters = UtilHttp.urlEncodeArgs(parameters, false); Build a URL and add the encoded parameter string to the URL: String redirectUrl = "http://www.somehost.com";String redirectString = redirectUrl + "?" + encodedParameters; Send the request by way of the HttpServletResponse object's redirect method: try { response.sendRedirect(redirectString); } catch (IOException e) {// process errors here return "error";} How it works... In a very simple web service client scenario, OFBiz acts on behalf of a user or another OFBiz process and makes a web service consumer request of a remote service provider. The request is sent using a HTTP/HTTPS request message with one or more service parameters appended to the URL. The URL is the location on the web of the service provider. Because the scenario described above is a one-way message exchange—that is, the request message is delivered to the destination URL, but OFBiz does not wait around for a response message—there must be assumptions made about how the web service provider will deliver service. Many times, service is delivered through a totally separate message exchange, initiated by first making the one-way request as described earlier. To illustrate how this may play out, we consider the PayPal Payments Standard web service. This web service may be invoked from OFBiz in at least two different ways. For example, one approach is to include an HTML form on an OFBiz web page with a Submit button. The Submit button when clicked redirects the browser (with the request message) to the PayPal web service site passing the form contents as is. An alternative implementation is to have an HTML form on an OFBiz web page with a Submit button that, when clicked, forwards the browser's request of an OFBiz Service (or Event). In this case, the OFBiz Event or Service will take the form attribute values (from the request message) and create a URL for the PayPal web service location. The original form attribute values and any other information as provided by the context or through database reads is appended to the URL. OFBiz then redirects the user's original request message using the sendRedirect method on the HttpServletResponse object to effectively send the user, by way of a browser and an appropriately crafted URL, to the PayPal web service. Building URL request parameters out of plain Java strings can be tricky given the nature of the characters used to construct request parameters and delimit multiple parameter strings. For example, how do you pass a space character as part of a parameter when spaces are used as delimiters? Enter URL encoding. URL encoding takes certain characters, deemed "special" characters, and "escapes" them so that they may be used as part of a request parameter string. OFBiz provides an easy-to-use encoder (and decoder) method(s): the UrlEncodeArgs method on the UtilHttp utility that takes as an argument a Java Map and returns an encoded string that may then be appended to a URL as shown earlier. Note: an exhaustive treatment of URL encoding is beyond the scope of this article. For more information on HTML and ASCII character codes and encoding symbols, please see the HTML Codes page There's more... The UrlEncodeArgs method has two modes of operation: selecting the false method-parameter value will encode using only an ampersand (&) symbol while the true parameter value tells OFBiz to use &amp as the encoding string. Based on the web service provider's instructions, you will need to determine which mode is appropriate for your application. Many web services do not require encoded values. You will need to verify for each service whether or not it is necessary to encode and decode URL parameters. For example, the PayPal IPN web service sends request parameters without any encoding. When returning IPN messages, the client web service is, however, instructed to encode parameters. PayPal IPN is strictly a server-to-server (there is no human intervention) messaging system where message traffic is transported across the web from one server URL to another. PayPal is the web service provider while, in this scenario, OFBiz acts as the web service client. It works something like shown in the following diagram: Requesting web services using an HttpClient Many web services are implemented using HTTP/HTTPS request methods and parameters passed as part of the request's message body. These requests for service mimic a user sitting at a browser submitting HTML forms and waiting for server response. Service providers read HTTP/HTTPS request header information and name/value request message parameter pairs, and deliver service through the HTTP/HTTPS response message. Writing clients for these types of web services usually require a synchronous call to the service provider. From within your OFBiz client code, you initiate a call to a service provider and then wait until a response (or timeout) is received. Unlike the PayPal Payments Standard service described earlier, the OFBiz client program does not redirect HTTP/HTTPS request messages to another URL. There are a number of examples within the out-of-the-box OFBiz project of service providers that use the HTTP/HTTPS request message body to exchange information with OFBiz clients. They include, but are not limited to: Authorize.net Payment Services ClearCommerce Payment Services Go Software RiTA ValueLink Prepaid/Gift Card Payment Services DHL, FedEx, United Parcel Services (UPS), and United States Post Office Shipping Services CDYNE Web Based Services PayPal Payments Pro Getting ready The first step in writing any web service client is to gather the following information about how the target web service operates: The URL on the web for the service provider Any connection parameters and/or HTTP/HTTPS request message header settings that must be passed as required by the service provider The HTTP/HTTPS connection verb (get, post, or other) Within a Java program, to send and receive web service messages as a client and use the built-in HTTP client utility provided by OFBiz, make sure you have the following Java packages imported in your program: import org.ofbiz.base.util.HttpClient;import org.ofbiz.base.util.HttpClientException; How to do it... You can request a web service by following these steps: Create a connection string that includes the URL of the target web service and the type of request: String connectString = "http://www.some_web_service_url.com/serviceName"; Within your Java program, create an instance of the HttpClient object with the URL/connection string passed to the constructor as shown here: HttpClient http = new HttpClient(connectString); Create the content of your request as dictated by the target web service provider. For example, some web services expect XML documents; others, simple string parameters. A web service that expects a string of name/value pairs could be coded as follows: http.setParameter("Param1", "X");http.setParameter("Param2", "Y"); Send your request using the appropriate method on the HttpClient object. For "get" requests, use the get method. For "post" requests, use the post method as shown here: try { response = http.post();}catch (HttpClientException e) { // Process error conditions here} Handle any service response inline. Unlike the asynchronous nature of the PayPal IPN web service described earlier, HttpClient based web services process return calls inline with the initial web service call. Under the covers, the HttpClient utility handles all the network connection set up and lower-level message transmissions. There is no need to release or close the connection as OFBiz manages the handoff of connections. How it works... When using the HttpClient to access web services remote to OFBiz, you send the consumer-side call synchronously; that is, you wait for the return from the remote web service call within your program. The OFBiz integration of the HttpClient utility manages the details necessary to open the network connection, maintain direct request and response message exchanges, and close the connection upon completion of processing. There's more... The OFBiz implementation of the HttpClient object provides several convenience constructors, which may be useful depending on your processing needs. These include: // To create a new client object and connect using a URL object instead of a StringURL url = "https://www.some_host.com/";HttpClient http = new HttpClient(url);// To create a new client object using a Java Map containing request parametersHttpClient http = new HttpClient(url, UtilMisc.toMap("param1", "X", "param2", "Y");//To create a new client object with a parameter map and header settingsHttpClient http = new HttpClient(connectString, UtilMisc.toMap("param1", "X"), UtilMisc.toMap("User-Agent, "Mozilla/4.0")); See also OFBiz provides an integration of the Apache HttpClient software package: the Jakarta Commons HTTP Client that is accessed by creating a new HttpClient object. Any method you can call on, the original Apache HttpClient object is available in the OFBiz implementation. This includes full support for HTTPS (SSL) clients. For more information, please see the Jakarta Commons HTTP Client web page
Read more
  • 0
  • 0
  • 4081

article-image-how-to-build-a-koa-web-application-part-2
Christoffer Hallas
08 Feb 2015
5 min read
Save for later

How to Build a Koa Web Application - Part 2

Christoffer Hallas
08 Feb 2015
5 min read
In Part 1 of this series, we got everything in place for our Koa app using Jade and Mongel. In this post, we will cover Jade templates and how to use listing and viewing pages. Please note that this series requires that you use Node.js version 0.11+. Jade templates Rendering HTML is always an important part of any web application. Luckily, when using Node.js there are many great choices, and for this article we’ve chosen Jade. Keep in mind though that we will only touch on a tiny fraction of the Jade functionality. Let’s create our first Jade template. Create a file called create.jade and put in the following: create.jade doctype html html(lang='en') head title Create Page body h1 Create Page form(method='POST', action='/create') input(type='text', name='title', placeholder='Title') input(type='text', name='contents', placeholder='Contents') input(type='submit') For all the Jade questions you have that we won’t answer in this series, I refer you to the excellent official Jade website at http://jade-lang.com . If you add the following statement app.listen(3000); to the end of index.js, then you should be able to run the program from your terminal using the following command and by visiting http://localhost:3000 in your browser. $ node --harmony index.js The --harmony flag just tells the node program that we need support for generators in our program: Listing and viewing pages Now that we can create a page in our MongoDB database, it is time to actually list and view these pages. For this purpose we need to add another middleware to our index.js file after the first middleware: app.use(function* () { if (this.method != 'GET') { this.status = 405; this.body = 'Method Not Allowed'; return } … }); As you can probably already tell, this new middleware is very similar to the first one we added that handled the creation of pages. At first we make sure that the method of the request is GET, and if not, we respond appropriately and return the following: var params = this.path.split('/').slice(1); var id = params[0]; if (id.length == 0) { var pages = yield Page.find(); var html = jade.renderFile('list.jade', { pages: pages }); this.body = html; return } Then, we proceed to inspect the path attribute of the Koa context, looking for an ID that represents the page in the database. Remember how we redirected using the ID in the previous middleware. We inspect the path by splitting it into an array of strings separated by the forward slashes of a URL; this way the path /1234 becomes an array of ‘’ and ‘1234.’ Because the path starts with a forward slash, the first item in the array will always be the empty string, so we just discard that by default. Then we check the length of the ID parameter, and if it’s zero we know that there is in fact no ID in the path, and we should just look for the pages in the database and render our list.jade template with those pages made available to the template as the variable pages. Making data available in templates is also known as providing locals to the template. list.jade doctype html html(lang="en") head title Your Web Application body h1 Your Web Application ul - each page in pages li a(href='/#{page._id}')= page.title But if the length of id was not zero, we assume that it’s an id and we try to load that specific page from the database instead of all the pages, and we proceed to render our view.jade template with the: var page = yield Page.findById(id); var html = jade.renderFile('view.jade', page); this.body = html; view.jade doctype html html(lang="en") head title= title body h1= title p= contents That’s it You should now be able to run the app as previously described and create a page, list all of your pages, and view them. If you want to, you can continue and build a simple CMS system. Koa is very simple to use and doesn’t enforce a lot of functionality on you, allowing you to pick and choose between libraries that you need and want to use. There are many possibilities and that is one of Koa’s biggest strengths. Find even more Node.js content on our Node.js page. Featuring our latest titles and most popular tutorials, it's the perfect place to learn more about Node.js. About the author Christoffer Hallas is a software developer and entrepreneur from Copenhagen, Denmark. He is a computer polyglot and contributes to and maintains a number of open source projects. When not contemplating his next grand idea (which remains an idea), he enjoys music, sports, and design of all kinds. Christoffer can be found on GitHub as hallas and at Twitter as @hamderhallas.
Read more
  • 0
  • 0
  • 4080
article-image-wxpython-28-advanced-building-blocks-user-interface
Packt
30 Dec 2010
10 min read
Save for later

wxPython 2.8: Advanced Building Blocks of a User Interface

Packt
30 Dec 2010
10 min read
Displaying collections of data and managing complex window layouts are a task that most UI developers will be faced with at some point. wxPython provides a number of components to help developers meet the requirements of these more demanding interfaces. As the amount of controls and data that an application is required to display in its user interface increases, so does the task of efficiently managing available screen real estate. To fit this information into the available space requires the use of some more advanced controls and containers; so let's dive in and begin our exploration of some of the more advanced controls that wxPython has to offer. Listing data with a ListCtrl The ListCtrl is a versatile control for displaying collections of text and/or images. The control supports many different display formats, although typically its most often-used display mode is the report mode. Report mode has a visual representation that is very similar to a grid or spreadsheet in that it can have multiple rows and columns with column headings. This recipe shows how to populate and retrieve data from a ListCtrl that was created in report mode. How to do it... The ListCtrl takes a little more set up than most basic controls, so we will start by creating a subclass that sets up the columns that we wish to have in the control: class MyListCtrl(wx.ListCtrl): def __init__(self, parent): super(MyListCtrl, self).__init__(parent, style=wx.LC_REPORT) # Add three columns to the list self.InsertColumn(0, "Column 1") self.InsertColumn(1, "Column 2") self.InsertColumn(2, "Column 3") def PopulateList(self, data): """Populate the list with the set of data. Data should be a list of tuples that have a value for each column in the list. [('hello', 'list', 'control'),] """ for item in data: self.Append(item) Next we will create an instance of our ListCtrl and put it on a Panel, and then use our PopulateList method to put some sample data into the control: class MyPanel(wx.Panel): def __init__(self, parent): super(MyPanel, self).__init__(parent) # Attributes self.lst = MyListCtrl(self) # Setup data = [ ("row %d" % x, "value %d" % x, "data %d" % x) for x in range(10) ] self.lst.PopulateList(data) # Layout sizer = wx.BoxSizer(wx.VERTICAL) sizer.Add(self.lst, 1, wx.EXPAND) self.SetSizer(sizer) # Event Handlers self.Bind(wx.EVT_LIST_ITEM_SELECTED, self.OnItemSelected) def OnItemSelected(self, event): selected_row = event.GetIndex() val = list() for column in range(3): item = self.lst.GetItem(selected_row, column) val.append(item.GetText()) # Show what was selected in the frames status bar frame = self.GetTopLevelParent() frame.PushStatusText(",".join(val)) How it works... Usually there tends to be a fair amount of set up with the ListCtrl, and due to this it is good to encapsulate the usage of the control in a specialized subclass instead of using it directly. We kept things pretty basic here in our ListCtrl class. We just used the InsertColumn method to set our list up with three columns. Then the PopulateList method was added for convenience, to allow the population of the ListCtrl from a Python list of data. It simply wraps the Append method of ListCtrl, which just takes an iterable that has a string for each column in the list. The MyPanel class is there to show how to use the ListCtrl class that we created. First we populate it with some data by generating a list of tuples and calling our PopulateList method. To show how to retrieve data from the list, we created an event handler for EVT_LIST_ITEM_SELECTED which will be fired each time a new selection is made in the control. In order to retrieve a value from a ListCtrl, you need to know the row and column index of the cell that you wish to retrieve the data from, and then call GetItem with the row and column to get the ListItem object that represents that cell. Then the string value of the cell can be retrieved by calling the GetText method of ListItem. There's more... Depending on the style flags that are used to create a ListCtrl, it will behave in many different possible ways. Because of this, it is important to know some of the different style flags that can be used to create a ListCtr.   Style flags Description LC_LIST In List mode, the control will calculate the columns automatically, so there is no need to call InsertColumn. It can be used to display strings and, optionally, small icons LC_REPORT Single or multicolumn report view that can be shown with or without headers LC_ICON Large icon view that can optionally have labels LC_SMALL_ICON Small icon view that can optionally have labels LC_EDIT_LABELS Allow the item labels to be editable by users LC_NO_HEADER Hide the column headers (report mode) LC_SORT_ASCENDING Sort items in ascending order (must provide a SortItems callback method) LC_SORT_DESCENDING Sort items in descending order (must provide a SortItems callback method) LC_HRULE Draw a horizontal line between rows (report mode) LC_VRULE Draw a vertical line between columns (report mode) LC_SINGLE_SEL Only allow a single item to be selected at a time (Default is to allow for multiple selections) LC_VIRTUAL Fetch items to display in the list on demand (report mode) Virtual Mode When a ListCtrl is created in virtual mode (using the LC_VIRTUAL style flag), it does not store the data internally; instead it will instead ask for the data from a datasource when it needs to display it. This mode is useful when you have a very large set of data where preloading it in the control would present performance issues. To use a ListCtrl in virtual mode, you must call SetItemCount to tell the control how many rows of data there are, and override the OnGetItemText method to return the text for the ListItem when the control asks for it. Browsing files with the CustomTreeCtrl A TreeCtrl is a way of displaying hierarchical data in a user interface. The CustomTreeCtrl is a fully owner-drawn TreeCtrl that looks and functions much the same way as the default TreeCtrl, but that offers a number of additional features and customizability that the default native control cannot. This recipe shows how to make a custom file browser class by using the CustomTreeCtrl. How to do it... To create this custom FileBrowser control, we will use its constructor to set up the images to use for the folders and files in the tree: import os import wx import wx.lib.customtreectrl as customtree class FileBrowser(customtree.CustomTreeCtrl): FOLDER, ERROR, FILE = range(3) def __init__(self, parent, rootdir, *args, **kwargs): super(FileBrowser, self).__init__(parent, *args, **kwargs) assert os.path.exists(rootdir), "Invalid Root Directory!" assert os.path.isdir(rootdir), "rootdir must be a Directory!" # Attributes self._il = wx.ImageList(16, 16) self._root = rootdir self._rnode = None # Setup for art in (wx.ART_FOLDER, wx.ART_ERROR, wx.ART_NORMAL_FILE): bmp = wx.ArtProvider.GetBitmap(art, size=(16,16)) self._il.Add(bmp) self.SetImageList(self._il) self._rnode = self.AddRoot(os.path.basename(rootdir), image=FileBrowser.FOLDER, data=self._root) self.SetItemHasChildren(self._rnode, True) # use Windows-Vista-style selections self.EnableSelectionVista(True) # Event Handlers self.Bind(wx.EVT_TREE_ITEM_EXPANDING, self.OnExpanding) self.Bind(wx.EVT_TREE_ITEM_COLLAPSED, self.OnCollapsed) def _GetFiles(self, path): try: files = [fname for fname in os.listdir(path) if fname not in ('.', '..')] except OSError: files = None return files The following two event handlers are used to update which files are displayed when a node is expanded or collapsed in the tree: def OnCollapsed(self, event): item = event.GetItem() self.DeleteChildren(item) def OnExpanding(self, event): item = event.GetItem() path = self.GetPyData(item) files = self._GetFiles(path) # Handle Access Errors if files is None: self.SetItemImage(item, FileBrowser.ERROR) self.SetItemHasChildren(item, False) return for fname in files: fullpath = os.path.join(path, fname) if os.path.isdir(fullpath): self.AppendDir(item, fullpath) else: self.AppendFile(item, fullpath) The following methods are added as an API for working with the control to add items and retrieve their on-disk paths: def AppendDir(self, item, path): """Add a directory node""" assert os.path.isdir(path), "Not a valid directory!" name = os.path.basename(path) nitem = self.AppendItem(item, name, image=FileBrowser.FOLDER, data=path) self.SetItemHasChildren(nitem, True) def AppendFile(self, item, path): """Add a file to a node""" assert os.path.isfile(path), "Not a valid file!" name = os.path.basename(path) self.AppendItem(item, name, image=FileBrowser.FILE, data=path) def GetSelectedPath(self): """Get the selected path""" sel = self.GetSelection() path = self.GetItemPyData(sel) return path def GetSelectedPaths(self): """Get a list of selected paths""" sels = self.GetSelections() paths = [self.GetItemPyData(sel) for sel in sels ] return paths How it works... With just a few lines of code here we have created a pretty useful little widget for displaying and working with the file system. Let's take a quick look at how it works. In the classes constructor, we added a root node with the control's AddRoot method. A root node is a top-level node that has no other parent nodes above it. The first argument is the text that will be shown, the image argument specifies the default image for the TreeItem, and the data argument specifies any type of data associated with the item—in this case we are setting a string for the items path. We then called SetItemHasChildren for the item so that it will get a button next to it to allow it to be expanded. The last thing that we did in the constructor was to Bind the control to two events so that we can update the tree when one of its nodes is being expanded or collapsed. Immediately before the node is going to be expanded our handler for EVT_TREE_ITEM_ EXPANDING will be called. It is here where we find all the files and folders under a directory node, and then add them as children of that node by calling AppendItem, which works just like AddRoot but is used to add items to already-existing nodes in the tree. Conversely when a node in the tree is going to be collapsed, our EVT_TREE_ITEM_COLLAPED event handler will be called. Here we simply call DeleteChildren in order to remove the children items from the node so that we can update them more easily the next time that the node is expanded. Otherwise, we would have to find what was different the next time it was expanded, and then remove the items that have been deleted and insert new items that may have been added to the directory. The last two items in our class are for getting the file paths of the selected items, which—since we store the file path in each node—is simply just a matter of getting the data from each of the currently-selected TreeItems with a call to GetPyData. There's more... Most of what we did in this recipe could actually also be replicated with the standard TreeCtrl. The difference is in the amount of extra customizability that the CustomTreeCtrl provides. Since it is a fully owner-drawn control, nearly all of the visible attributes of it can be customized. Following is a list of some of the functions that can be used to customize its appearance:
Read more
  • 0
  • 0
  • 4076

article-image-developing-simple-workflow-within-sugarcrm
Packt
22 Oct 2009
4 min read
Save for later

Developing a Simple Workflow within SugarCRM

Packt
22 Oct 2009
4 min read
A Very Simple Workflow In our simple workflow we'll assume that each task is carried out by one person at a time, and that all tasks are done sequentially (i.e. none are done in parallel). So, we'll look at the PPI Preliminary Investigation which, as you remember, maps to the standard SugarCRM Opportunity. Also, in this example, we're going to have a different person carrying out each one of the Investigation stages. Setting up the Process Stages If you look at SugarCRM then you'll see that by default none of the stages are related to investigations—they're all named using standard CRM terms: Obviously the first thing to do is to decide what the preliminary investigation stages actually are, and then map these to the SugarCRM stages. You'll realize that you'll need to edit the custom/include/langauge/en_us.lang.php file: $app_list_strings['sales_stage_dom']=array ( 'Prospecting' => 'Fact Gathering', 'Qualification' => 'Witness and Subject Location', 'Needs Analysis' => 'Witness and Subject Interviews', 'Value Proposition' => 'Scene Investigation', 'Id. Decision Makers' => 'Financial and background Investigation', 'Perception Analysis' => 'Document and evidence retrieval', 'Proposal/Price Quote' => 'Covert Camera surveillance', 'Negotiation/Review' => 'Wiretapping', 'Closed Won' => 'Full Investigation required', 'Closed Lost' => 'Insufficient Evidence',); Don't forget that you can also do this via Studio. However, once you've added your mapping into custom/include/langauge/en_us.lang.php file, and refresh your browser, then you'll see the new stages: Now that our stages are set up we need to know who'll be carrying out each one. Deciding Who Does What In our simple workflow there may not be the need to do anything further. Each person just needs to know who does what next: For example, once Kurt finishes the 'Covert Camera surveillance' stage then he just needs to update the Preliminary Investigation so that the stage is set to 'Wiretapping' and the assigned user as 'dobbsm'. However, things are rarely as simple as that. It's much more likely that: Investigations may be based on geographical locations, so that the above table may only apply to investigations based in London. Investigations based in New York follow the same process but with a different set of staff. On Mondays Fran does 'Witness and Subject Location' and William does 'Fact Gathering'. This means, of course, that we need to be using some businesses rules. Introducing Business Rules There are six 'triggers' that will cause the logic hooks to fire: after_retrieve before_save before_delete after_delete before_undelete after_undelete And the logic hooks are stored in custom/modules/<module name>/logic_hook.php, so for 'Preliminary Inquiries' this will be custom/modules/Opportunities/logic_hook.php. You'll also remember, of course, that the logic hook file needs to contain: The priority of the business rule The name of the businesses rule The file containing the business rule The business rule class The business rule function So, custom/modules/Opportunities/logic_hook.php needs to contain something like: <?php#As always ensure that the file can only be accessed through SugarCRMif(!defined('sugarEntry') || !sugarEntry) die( 'Not A Valid Entry Point');$hook_array = Array(); #Create an array$hook_array['before_save'] = Array();$hook_array['before_save'][] = Array(1, 'ppi_workflow', 'custom/include/ppi_workflow.php', 'ppi_workflow', 'ppi_workflow');?> Next we'll need the file that logic hook will be calling, but to start with this can be very basic—so, custom/include/ppi_workflow.php just needs to contain something like: <?php#Define the entry pointif(!defined('sugarEntry') || !sugarEntry) die( 'Not A Valid Entry Point');#Load any required filesrequire_once('data/SugarBean.php');require_once('modules/Opportunities/Opportunity.php');#Define the classclass ppi_workflow{ function ppi_workflow (&$bean, $event, $arguments) { }}?> With those two files set up as above nothing obvious will change in the operation of SugarCRM—the logic hook will fire, but we haven't told it to do anything, and so that what we'll do now. When the logic hook does run (i.e. when any Primary Investigation is saved) we would want it to: Check to see what stage we're now at Define the assigned user accordingly  
Read more
  • 0
  • 0
  • 4067
Modal Close icon
Modal Close icon