How-To Tutorials - Programming

What is SOA? SOA is the acronym for Service Oriented Architecture. As it has come to be known, SOA is an architectural design pattern by which several guiding principles determine the nature of the design. Basically, SOA states that every component of a system should be a service, and the system should be composed of several loosely-coupled services. A service here means a unit of a program that serves a business process. "Loosely-coupled" here means that these services should be independent of each other, so that changing one of them should not affect any other services. SOA is not a specific technology, nor a specific language. It is just a blueprint, or a system design approach. It is an architecture model that aims to enhance the efficiency, agility, and productivity of an enterprise system. The key concepts of SOA are services, high interoperability and loose coupling. Several other architecture/technologies such as RPC, DCOM, and CORBA have existed for a long time, and attempted to address the client/server communication problems. The difference between SOA and these other approaches is that SOA is trying to address the problem from the client side, and not from the server side. It tries to decouple the client side from the server side, instead of bundling them, to make the client side application much easier to develop and maintain. This is exactly what happened when object-oriented programming (OOP) came into play 20 years ago. Prior to object-oriented programming, most designs were procedure-oriented, meaning the developer had to control the process of an application. Without OOP, in order to finish a block of work, the developer had to be aware of the sequence that the code would follow. This sequence was then hard-coded into the program, and any change to this sequence would result in a code change. With OOP, an object simply supplied certain operations; it was up to the caller of the object to decide the sequence of those operations. The caller could mash up all of the operations, and finish the job in whatever order needed. There was a paradigm shift from the object side to the caller side. This same paradigm shift is happening today. Without SOA, every application is a bundled, tightly coupled solution. The client-side application is often compiled and deployed along with the server-side applications, making it impossible to quickly change anything on the server side. DCOM and CORBA were on the right track to ease this problem by making the server-side components reside on remote machines. The client application could directly call a method on a remote object, without knowing that this object was actually far away, just like calling a method on a local object. However, the client-side applications continue to remain tightly coupled with these remote objects, and any change to the remote object will still result in a recompiling or redeploying of the client application. Now, with SOA, the remote objects are truly treated as remote objects. To the client applications, they are no longer objects; they are services. The client application is unaware of how the service is implemented, or of the signature that should be used when interacting with those services. The client application interacts with these services by exchanging messages. What a client application knows now is only the interfaces, or protocols of the services, such as the format of the messages to be passed in to the service, and the format of the expected returning messages from the service. Historically, there have been many other architectural design approaches, technologies, and methodologies to integrate existing applications. EAI (Enterprise Application Integration) is just one of them. Often, organizations have many different applications, such as order management systems, accounts receivable systems, and customer relationship management systems. Each application has been designed and developed by different people using different tools and technologies at different times, and to serve different purposes. However, between these applications, there are no standard common ways to communicate. EAI is the process of linking these applications and others in order to realize financial and operational competitive advantages. It may seem that SOA is just an extension of EAI. The similarity is that they are both designed to connect different pieces of applications in order to build an enterprise-level system for business. But fundamentally, they are quite different. EAI attempts to connect legacy applications without modifying any of the applications, while SOA is a fresh approach to solve the same problem. Why SOA? So why do we need SOA now? The answer is in one word—agility. Business requirements change frequently, as they always have. The IT department has to respond more quickly and cost-effectively to those changes. With a traditional architecture, all components are bundled together with each other. Thus, even a small change to one component will require a large number of other components to be recompiled and redeployed. Quality assurance (QA) effort is also huge for any code changes. The processes of gathering requirements, designing, development, QA, and deployment are too long for businesses to wait for, and become actual bottlenecks. To complicate matters further, some business processes are no longer static. Requirements change on an ad-hoc basis, and a business needs to be able to dynamically define its own processes whenever it wants. A business needs a system that is agile enough for its day-to-day work. This is very hard, if not impossible, with existing traditional infrastructure and systems. This is where SOA comes into play. SOA's basic unit is a service. These services are building blocks that business users can use to define their own processes. Services are designed and implemented so that they can serve different purposes or processes, and not just specific ones. No matter what new processes a business needs to build or what existing processes a business needs need to modify, the business users should always be able to use existing service blocks, in order to compete with others according to current marketing conditions. Also, if necessary, some new service blocks can be used. These services are also designed and implemented so that they are loosely coupled, and independent of one another. A change to one service does not affect any other service. Also, the deployment of a new service does not affect any existing service. This greatly eases release management and makes agility possible. For example, a GetBalance service can be designed to retrieve the balance for a loan. When a borrower calls in to query the status of a specific loan, this GetBalance service may be called by the application that is used by the customer service representatives. When a borrower makes a payment online, this service can also be called to get the balance of the loan, so that the borrower will know the balance of his or her loan after the payment. Yet in the payment posting process, this service can still be used to calculate the accrued interest for a loan, by multiplying the balance with the interest rate. Even further, a new process can be created by business users to utilize this service if a loan balance needs to be retrieved. The GetBalance service is developed and deployed independently from all of the above processes. Actually, the service exists without even knowing who the client will be or even how many clients there will be. All of the client applications communicate with this service through its interface, and its interface will remain stable once it is in production. If we have to change the implementation of this service, for example by fixing a bug, or changing an algorithm inside a method of the service, all of the client applications can still work without any change. When combined with the more mature Business Process Management (BPM) technology, SOA plays an even more important role in an organization's efforts to achieve agility. Business users can create and maintain processes within BPM, and through SOA they can plug a service into any of the processes. The front-end BPM application is loosely coupled to the back-end SOA system. This combination of BPM and SOA will give an organization much greater flexibility in order to achieve agility. How do we implement SOA? Now that we've established why SOA is needed by the business, the question becomes—how do we implement SOA? To implement SOA in an organization, three key elements have to be evaluated—people, process, and technology. Firstly, the people in the organization must be ready to adopt SOA. Secondly, the organization must know the processes that the SOA approach will include, including the definition, scope, and priority. Finally, the organization should choose the right technology to implement it. Note that people and processes take precedence over technology in an SOA implementation, but they are out of the scope of this article. In this article, we will assume people and processes are all ready for an organization to adopt SOA. Technically, there are many SOA approaches. At certain degrees, traditional technologies such as RPC, DCOM, CORBA, or some modern technologies such as IBM WebSphere MQ, Java RMI, and .NET Remoting could all be categorized as service-oriented, and can be used to implement SOA for one organization. However, all of these technologies have limitations, such as language or platform specifications, complexity of implementation, or the ability to support binary transports only. The most important shortcoming of these approaches is that the server-side applications are tightly coupled with the client-side applications, which is against the SOA principle. Today, with the emergence of web service technologies, SOA becomes a reality. Thanks to the dramatic increase in network bandwidth, and given the maturity of web service standards such as WS-Security, and WS-AtomicTransaction, an SOA back-end can now be implemented as a real system. SOA from different users' perspectives However, as we said earlier, SOA is not a technology, but only a style of architecture, or an approach to building software products. Different people view SOA in different ways. In fact, many companies now have their own definitions for SOA. Many companies claim they can offer an SOA solution, while they are really just trying to sell their products. The key point here is—SOA is not a solution. SOA alone can't solve any problem. It has to be implemented with a specific approach to become a real solution. You can't buy an SOA solution. You may be able to buy some kinds of products to help you realize your own SOA, but this SOA should be customized to your specific environment, for your specific needs. Even within the same organization, different players will think about SOA in quite different ways. What follows are just some examples of how different players in an organization judge the success of an SOA initiative using different criteria. [Gartner, Twelve Common SOA Mistakes and How to Avoid Them, Publication Date: 26 October 2007 ID Number: G00152446] To a programmer, SOA is a form of distributed computing in which the building blocks (services) may come from other applications or be offered to them. SOA increases the scope of a programmer's product and adds to his or her resources, while also closely resembling familiar modular software design principles. To a software architect, SOA translates to the disappearance of fences between applications. Architects turn to the design of business functions rather than to self-contained and isolated applications. The software architect becomes interested in collaboration with a business analyst to get a clear picture of the business functionality and scope of the application. SOA turns software architects into integration architects and business experts. For the Chief Investment Officers (CIOs), SOA is an investment in the future. Expensive in the short term, its long-term promises are lower costs, and greater flexibility in meeting new business requirements. Re-use is the primary benefit anticipated as a means to reduce the cost and time of new application development. For business analysts, SOA is the bridge between them and the IT organization. It carries the promise that IT designers will understand them better, because the services in SOA reflect the business functions in business process models. For CEOs, SOA is expected to help IT become more responsive to business needs and facilitate competitive business change. Complexities in SOA implementation Although SOA will make it possible for business parties to achieve agility, SOA itself is technically not simple to implement. In some cases, it even makes software development more complex than ever, because with SOA you are building for unknown problems. On one hand, you have to make sure that the SOA blocks you are building are useful blocks. On the other, you need a framework within which you can assemble those blocks to perform business activities. The technology issues associated with SOA are more challenging than vendors would like users to believe. Web services technology has turned SOA into an affordable proposition for most large organizations by providing a universally-accepted, standard foundation. However, web services play a technology role only for the SOA backplane, which is the software infrastructure that enables SOA-related interoperability and integration. The following figure shows the technical complexity of SOA. It has been taken from Gartner, Twelve Common SOA Mistakes and How to Avoid Them, Publication Date: 26 October 2007 ID Number: G00152446. As Gartner says, users must understand the complex world of middleware, and point-to-point web service connections only for small-scale, experimental SOA projects. If the number of services deployed grows to more than 20 or 30, then use a middleware-based intermediary—the SOA backplane. The SOA backplane could be an Enterprise Service Bus (ESB), a Message-Oriented Middleware (MOM), or an Object Request Broker (ORB). However, in this article, we will not cover it. We will build only point-to-point services using WCF. Web services There are many approaches to realizing SOA, but the most popular and practical one is—using web services. What is a web service? A web service is a software system designed to support interoperable machine-to-machine interaction over a network. A web service is typically hosted on a remote machine (provider), and called by a client application (consumer) over a network. After the provider of a web service publishes the service, the client can discover it and invoke it. The communications between a web service and a client application use XML messages. A web service is hosted within a web server and HTTP is used as the transport protocol between the server and the client applications. The following diagram shows the interaction of web services: Web services were invented to solve the interoperability problem between applications. In the early 90s, along with the LAN/WAN/Internet development, it became a big problem to integrate different applications. An application might have been developed using C++, or Java, and run on a Unix box, a Windows PC, or even a mainframe computer. There was no easy way for it to communicate with other applications. It was the development of XML that made it possible to share data between applications across hardware boundaries and networks, or even over the Internet. For example, a Windows application might need to display the price of a particular stock. With a web service, this application can make a request to a URL, and/or pass an XML string such as <QuoteRequest><GetPrice Symble='XYZ'/></QuoteRequest>. The requested URL is actually the Internet address of a web service, which, upon receiving the above quote request, gives a response, <QuoteResponse><QuotePrice Symble='XYZ'>51.22</QuotePrice></QuoteResponse/>. The Windows application then uses an XML parser to interpret the response package, and display the price on the screen. The reason it is called a web service is that it is designed to be hosted in a web server, such as Microsoft Internet Information Server, and called over the Internet, typically via the HTTP or HTTPS protocols. This is to ensure that a web service can be called by any application, using any programming language, and under any operating system, as long as there is an active Internet connection, and of course, an open HTTP/HTTPS port, which is true for almost every computer on the Internet. Each web service has a unique URL, and contains various methods. When calling a web service, you have to specify which method you want to call, and pass the required parameters to the web service method. Each web service method will also give a response package to tell the caller the execution results. Besides new applications being developed specifically as web services, legacy applications can also be wrapped up and exposed as web services. So, an IBM mainframe accounting system might be able to provide external customers with a link to check the balance of an account. Web service WSDL In order to be called by other applications, each web service has to supply a description of itself, so that other applications will know how to call it. This description is provided in a language called a WSDL. WSDL stands for Web Services Description Language. It is an XML format that defines and describes the functionalities of the web service, including the method names, parameter names, and types, and returning data types of the web service. For a Microsoft ASMX web service, you can get the WSDL by adding ?WSDL to the end of the web service URL, say http://localhost/MyService/MyService.asmx?WSDL. Web service proxy A client application calls a web service through a proxy. A web service proxy is a stub class between a web service and a client. It is normally auto-generated by a tool such as Visual Studio IDE, according to the WSDL of the web service. It can be re-used by any client application. The proxy contains stub methods mimicking all of methods of the web service so that a client application can call each method of the web service through these stub methods. It also contains other necessary information required by the client to call the web service such as custom exceptions, custom data and class types, and so on. The address of the web service can be embedded within the proxy class, or it can be placed inside a configuration file. A proxy class is always for a specific language. For each web service, there could be a proxy class for Java clients, a proxy class for C# clients, and yet another proxy class for COBOL clients. To call a web service from a client application, the proper proxy class first has to be added to the client project. Then, with an optional configuration file, the address of the web service can be defined. Within the client application, a web service object can be instantiated, and its methods can be called just as for any other normal method. SOAP There are many standards for web services. SOAP is one of them. SOAP was originally an acronym for Simple Object Access Protocol, and was designed by Microsoft. As this protocol became popular with the spread of web services, and its original meaning was misleading, the original acronym was dropped with version 1.2 of the standard. It is now merely a protocol, maintained by W3C. SOAP, now, is a protocol for exchanging XML-based messages over computer networks. It is widely-used by web services and has become its de-facto protocol. With SOAP, the client application can send a request in XML format to a server application, and the server application will send back a response in XML format. The transport for SOAP is normally HTTP / HTTPS, and the wide acceptance of HTTP is one of the reasons why SOAP is widely accepted today.

"The views expressed in this article are the author's own and do not necessarily reflect the views of Oracle." (For more resources on Oracle, see here.) Introduction This article is better understood by people who have some familiarity with Oracle database, SQL, PL/SQL, and of course Java (including JDBC). Beginners can also understand the article to some extent, because it does not contain many specifics/details. The article can be useful to software developers, designers and architects working with Java. Oracle database provides a Java runtime in its database server process. Because of this, it is possible not only to store Java sources and Java classes in an Oracle database, but also to run the Java classes within the database server. Such Java classes will be 'executed' by the Java Virtual Machine embedded in the database server. The Java platform provided is J2SE-compliant, and in addition to the JVM, it includes all the Java system classes. So, conceptually, whatever Java code that can be run using the JREs (like Sun's JRE) on the operating system, can be run within the Oracle database too. Java stored procedure The key unit of the Java support inside the Oracle database is the 'Java Stored Procedure' (that may be referred to as JSP, as long as it is not confused with JavaServer Pages). A Java stored procedure is an executable unit stored inside the Oracle database, and whose implementation is in Java. It is similar to PL/SQL stored procedures and functions. Creation Let us see an example of how to create a simple Java stored procedure. We will create a Java stored procedure that adds two given numbers and returns the sum. The first step is to create a Java class that looks like the following: public class Math{ public static int add(int x, int y) { return x + y; }} This is a very simple Java class that just contains one static method that returns the sum of two given numbers. Let us put this code in a file called Math.java, and compile it (say, by doing 'javac Math.java') to get Math.class file. The next step is to 'load' Math.class into the Oracle database. That is, we have to put the class file located in some directory into the database, so that the class file gets stored in the database. There are a few ways to do this, and one of them is to use the command-line tool called loadjava provided by Oracle, as follows: loadjava -v -u scott/tiger Math.class Generally, in Oracle database, things are always stored in some 'schema' (also known as 'user'). Java classes are no exception. So, while loading a Java class file into the database, we need to specify the schema where the Java class should be stored. Here, we have given 'scott' (along with the password). There are a lot of other things that can be done using loadjava, but we will not go into them here. Next, we have to create a 'PL/SQL wrapper' as follows: SQL> connect scott/tigerConnected.SQL>SQL> create or replace function addition(a IN number, b IN number) return number 2 as language java name 'Math.add(int, int) return int'; 3 /Function created.SQL> We have created the PL/SQL wrapper called 'addition', for the Java method Math.add(). The syntax is same as the one used to create a PL/SQL function/procedure, but here we have specified that the implementation of the function is in the Java method Math.add(). And that's it. We've created a Java stored procedure! Basically, what we have done is, implemented our requirement in Java, and then exposed the Java implementation via PL/SQL. Using Jdeveloper, an IDE from Oracle, all these steps (creating the Java source, compiling it, loading it into the database, and creating the PL/SQL wrapper) can be done easily from within the IDE. One thing to remember is that, we can create Java stored procedures for Java static methods only, but not for instance methods. This is not a big disadvantage, and in fact makes sense, because even the main() method, which is the entry point for a Java program, is also 'static'. Here, since Math.add() is the entry point, it has to be 'static'. So, we can write as many static methods in our Java code as needed and make them entry points by creating the PL/SQL wrappers for them. Invocation We can call the Java stored procedure we have just created, just like any PL/SQL procedure/function is called, either from SQL or PL/SQL: SQL> select addition(10, 20) from dual;ADDITION(10,20)--------------- 30SQL>SQL> declare 2 s number; 3 begin 4 s := addition(10, 20); 5 dbms_output.put_line('SUM = ' || s); 6 end; 7 /SUM = 30PL/SQL procedure successfully completed.SQL> Here, the 'select' query, as well as the PL/SQL block, invoked the PL/SQL function addition(), which in turn invoked the underlying Java method Math.add(). A main feature of the Java stored procedure is that, the caller (like the 'select' query above) has no idea that the procedure is indeed implemented in Java. Thus, the stored procedures implemented in PL/SQL and Java can be called alike, without requiring to know the language in which the underlying implementation is. So, in general, whatever Java code we have, can be seamlessly integrated into the PL/SQL code via the PL/SQL wrappers. Putting in other words, we now have more than one language option to implement a stored procedure - PL/SQL and Java. If we have any project where stored procedures are to be implemented, then Java is a good option, because today it is relatively easier to find a Java programmer.

JBoss Web Server currently uses the Apache Tomcat 6.0 release and it is ships as service archive (SAR) application in the deploy folder. The location of the embedded web server has changed at almost every new release of JBoss. The following table could be a useful reference if you are using different versions of JBoss: JBoss release Location of Tomcat 5.0.0 GA deploy/jbossweb.sar 4.2.2 GA deploy/jboss-web.deployer 4.0.5 GA deploy/jbossweb-tomcat55.sar 3.2.X deploy/jbossweb-tomcat50.sar The main configuration file is server.xml which, by default, has the following minimal configuration: <Server><Listener className="org.apache.catalina.core.AprLifecycleListener"SSLEngine="on" /><Listener className="org.apache.catalina.core.JasperListener" /><Service name="jboss.web"><Connector protocol="HTTP/1.1" port="8080"address="${jboss.bind.address}"connectionTimeout="20000" redirectPort="8443" /><Connector protocol="AJP/1.3" port="8009"address="${jboss.bind.address}"redirectPort="8443" /><Engine name="jboss.web" defaultHost="localhost"><Realm className="org.jboss.web.tomcat.security.JBossWebRealm"certificatePrincipal="org.jboss.security.auth.certs.SubjectDNMapping" allRolesMode="authOnly" /><Host name="localhost"><Valve className="org.jboss.web.tomcat.service.jca.CachedConnectionValve"cachedConnectionManagerObjectName="jboss.jca:service=CachedConnectionManager"transactionManagerObjectName="jboss:service=TransactionManager" /></Host></Engine></Service></Server> Following is a short description for the key elements of the configuration: Element Description Server The Server is Tomcat itself, that is, an instance of the web application server and is a top-level component. Service An Engine is a request-processing component that represents the Catalina servlet engine. It examines the HTTP headers to determine the virtual host or context to which requests should be passed. Connector It's the gateway to Tomcat Engine. It ensures that requests are received from clients and are assigned to the Engine. Engine Engine handles all requests. It examines the HTTP headers to determine the virtual host or context to which requests should be passed. Host One virtual host. Each virtual host is differentiated by a fully qualified hostname. Valve A component that will be inserted into the request processing pipeline for the associated Catalina container. Each Valve has distinct processing capabilities. Realm It contains a set of users and roles. As you can see, all the elements are organized in a hierarchical structure where the Server element acts as top-level container: The lowest elements in the configuration are Valve and Realm, which can be nested into Engine or Host elements to provide unique processing capabilities and role management. Customizing connectors Most of the time when you want to customize your web container, you will have to change some properties of the connector. <Connector protocol="HTTP/1.1" port="8080"address="${jboss.bind.address}"connectionTimeout="20000" redirectPort="8443" /> A complete list of the connector properties can be found on the Jakarta Tomcat site (http://tomcat.apache.org/). Here, we'll discuss the most useful connector properties: port: The TCP port number on which this connector will create a server socket and await incoming connections. Your operating system will allow only one server application to listen to a particular port number on a particular IP address. acceptCount: The maximum queue length for incoming connection requests, when all possible request processing threads are in use. Any requests received when the queue is full will be refused. The default value is 10. connectionTimeout: The number of milliseconds the connector will wait after accepting a connection for the request URI line to be presented. The default value is 60000 (that is, 60 seconds). address: For servers with more than one IP address, this attribute specifies which address will be used for listening on the specified port. By default, this port will be used on all IP addresses associated with the server. enableLookups: Set to true if you want to perform DNS lookups in order to return the actual hostname of the remote client and to false in order to skip the DNS lookup and return the IP address in string form instead (thereby improving performance). By default, DNS lookups are enabled. maxHttpHeaderSize: The maximum size of the request and response HTTP header, specified in bytes. If not specified, this attribute is set to 4096 (4 KB). maxPostSize: The maximum size in bytes of the POST, which will be handled by the container FORM URL parameter parsing. The limit can be disabled by setting this attribute to a value less than or equal to zero. If not specified, this attribute is set to 2097152 (2 megabytes). maxThreads: The maximum number of request processing threads to be created by this connector, which therefore determines the maximum number of simultaneous requests that can be handled. If not specified, this attribute is set to 200. The new Apache Portable Runtime connector Apache Portable Runtime (APR) is a core Apache 2.x library designed to provide superior scalability, performance, and better integration with native server technologies. The mission of the Apache Portable Runtime (APR) project is to create and maintain software libraries that provide a predictable and consistent interface to underlying platform-specific implementations. The primary goal is to provide an API to which software developers may code and be assured of predictable if not identical behaviour regardless of the platform on which their software is built, relieving them of the need to code special-case conditions to work around or take advantage of platform-specific deficiencies or features. The high-level performance of the new APR connector is made possible by the introduction of socket pollers for persistent connections (keepalive). This increases the scalability of the server, and by using sendfile system calls, static content is delivered faster and with lower CPU utilization. Once you have set up the APR connector, you are allowed to use the following additional properties in your connector: keepAliveTimeout: The number of milliseconds the APR connector will wait for another HTTP request, before closing the connection. If not set, this attribute will use the default value set for the connectionTimeout attribute. pollTime: The duration of a poll call; by default it is 2000 (5 ms). If you try to decrease this value, the connector will issue more poll calls, thus reducing latency of the connections. Be aware that this will put slightly more load on the CPU as well. pollerSize: The number of sockets that the poller kept alive connections can hold at a given time. The default value is 768, corresponding to 768 keepalive connections. useSendfile: Enables using kernel sendfile for sending certain static files. The default value is true. sendfileSize: The number of sockets that the poller thread dispatches for sending static files asynchronously. The default value is 1024. If you want to consult the full documentation of APR, you can visit http://apr.apache.org/. Installing the APR connector In order to install the APR connector, you need to add some native libraries to your JBoss server. The native libraries can be found at http://www.jboss.org/jbossweb/downloads/jboss-native/. Download the version that is appropriate for your OS. Once you are ready, you need to simply unzip the content of the archive into your JBOSS_HOME directory. As an example, Unix users (such as HP users) would need to perform the following steps: cd jboss-5.0.0.GAtar tvfz jboss-native-2.0.6-hpux-parisc2-ssl.tar.gz Now, restart JBoss and, from the console, verify that the connector is bound to Http11AprProtocol. A word of caution!At the time of writing, the APR library still has some open issues that prevent it from loading correctly on some platforms, particularly on the 32-bit Windows. Please consult the JBoss Issue Tracker (https://jira.jboss.org/jira/secure/IssueNavigator.jspa?) to verify that there are no open issues for your platform.

Cassandra High Performance Cookbook Over 150 recipes to design and optimize large scale Apache Cassandra deployments       Changing debugging to determine where read operations are being routed Cassandra replicates data to multiple nodes; because of this, a read operation can be served by multiple nodes. If a read at QUORUM or higher is submitted, a Read Repair is executed, and the read operation will involve more than a single server. In a simple flat network which nodes have chosen for digest reads, are not of much consequence. However, in multiple datacenter or multiple switch environments, having a read cross a switch or a slower WAN link between datacenters can add milliseconds of latency. This recipe shows how to debug the read path to see if reads are being routed as expected. How to do it... Edit <cassandra_home>/conf/log4j-server.properties and set the logger to debug, then restart the Cassandra process: log4j.rootLogger=DEBUG,stdout,R On one display, use the tail -f <cassandra_log_dir>/system.log to follow the Cassandra log: DEBUG 06:07:35,060 insert writing local RowMutation(keyspace='ks1', key='65', modifications=[cf1]) DEBUG 06:07:35,062 applying mutation of row 65 In another display, open an instance of the Cassandra CLI and use it to insert data. Remember, when using RandomPartitioner, try different keys until log events display on the node you are monitoring: [default@ks1] set cf1[‘e'][‘mycolumn']='value'; Value inserted. Fetch the column using the CLI: [default@ks1] get cf1[‘e'][‘mycolumn']; Debugging messages should be displayed in the log. DEBUG 06:08:35,917 weakread reading SliceByNamesReadComman d(table='ks1', key=65, columnParent='QueryPath(columnFami lyName='cf1', superColumnName='null', columnName='null')', columns=[6d79636f6c756d6e,]) locally ... DEBUG 06:08:35,919 weakreadlocal reading SliceByNamesReadCo mmand(table='ks1', key=65, columnParent='QueryPath(columnFa milyName='cf1', superColumnName='null', columnName='null')', columns=[6d79636f6c756d6e,]) How it works... Changing the logging property level to DEBUG causes Cassandra to print information as it is handling reads internally. This is helpful when troubleshooting a snitch or when using the consistency levels such as LOCAL_QUORUM or EACH_QUORUM, which route requests based on network topologies. Using IPTables to simulate complex network scenarios in a local environment While it is possible to simulate network failures by shutting down Cassandra instances, another failure you may wish to simulate is a failure that partitions your network. A failure in which multiple systems are UP but cannot communicate with each other is commonly referred to as a split brain scenario. This state could happen if the uplink between switches fails or the connectivity between two datacenters is lost. Getting ready When editing any firewall, it is important to have a backup copy. Testing on a remote machine is risky as an incorrect configuration could render your system unreachable. How to do it... Review your iptables configuration found in /etc/sysconfig/iptables. Typically, an IPTables configuration accepts loopback traffic: :RH-Firewall-1-INPUT - [0:0] -A INPUT -j RH-Firewall-1-INPUT -A FORWARD -j RH-Firewall-1-INPUT -A RH-Firewall-1-INPUT -i lo -j ACCEPT Remove the highlighted rule and restart IPTables. This should prevent instances of Cassandra on your machine from communicating with each other: #/etc/init.d/iptables restart Add a rule to allow a Cassandra instance running on 10.0.1.1 communicate to 10.0.1.2: -A RH-Firewall-1-INPUT -m state --state NEW -s 10.0.1.1 -d 10.0.1.2 -j ACCEPT How it works... IPTables is a complete firewall that is a standard part of current Linux kernel. It has extensible rules that can permit or deny traffic based on many attributes, including, but not limited to, source IP, destination IP, source port, and destination port. This recipe uses the traffic blocking features to simulate network failures, which can be used to test how Cassandra will operate with network failures. Choosing IP addresses to work with RackInferringSnitch A snitch is Cassandra's way of mapping a node to a physical location in the network. It helps determine the location of a node relative to another node in order to ensure efficient request routing. The RackInferringSnitch can only be used if your network IP allocation is divided along octets in your IP address. Getting ready The following network diagram demonstrates a network layout that would be ideal for RackInferringSnitch. How to do it... In the <cassandra_home>/conf/cassandra.yaml file: endpoint_snitch: org.apache.cassandra.locator.RackInferringSnitch Restart the Cassandra instance for this change to take effect. How it works... The RackInferringSnitch requires no extra configuration as long as your network adheres to a specific network subnetting scheme. In this scheme, the first octet, Y.X.X.X, is the private network number 10. The second octet, X.Y.X.X, represents the datacenter. The third octet, X.X.Y.X, represents the rack. The final octet represents the host, X.X.X.Y. Cassandra uses this information to determine which hosts are ‘closest'. It is assumed that ‘closer' nodes will have more bandwidth and less latency between them. Cassandra uses this information to send Digest Reads to the closest nodes and route requests efficiently. There's more... While it is ideal if the network conforms to what is required for RackInferringSnitch, it is not always practical or possible. It is also rigid in that if a single machine does not adhere to the convention, the snitch will fail to work properly.

To move data processing to the data, you can use stored procedures, stored functions, and triggers. All these components are implemented inside the underlying database, and can significantly improve performance of your application due to reducing network overhead associated with multiple calls to the database. It is important to realize, though, the decision to move any piece of processing logic into the database should be taken with care. In some situations, this may be simply inefficient. For example, if you decide to move some logic dealing with the data stored in a custom Python list into the database, while still keeping that list implemented in your Python code, this can be inefficient in such a case, since it only increases the number of calls to the underlying database, thus causing significant network overhead. To fix this situation, you could move the list from Python into the database as well, implementing it as a table. Starting with version 5.0, MySQL supports stored procedures, stored functions, and triggers, making it possible for you to enjoy programming on the underlying database side. In this article, you will look at triggers in action. Stored procedures and functions can be used similarly. Planning Changes for the Sample Application Assuming you have followed the instructions in Python Data Persistence using MySQL, you should already have the application structure to be reorganized here. To recap, what you should already have is: tags nested list of tags used to describe the posts obtained from the Packt Book Feed page. obtainPost function obtains the information about the most recent post on the Packt Book Feed page. determineTags function determines tags appropriate to the latest post obtained from the Packt Book Feed page. insertPost function inserts the information about the obtained post into the underlying database tables: posts and posttags. execPr function brings together the functionality of the described above functions. That’s what you should already have on the Python side. And on the database side, you should have the following components: posts table contains records representing posts obtained from the Packt Book Feed page. posttags table contains records each of which represents a tag associated with a certain post stored in the posts table. Let’s figure out how we can refactor the above structure, moving some data processing inside the database. The first thing you might want to do is to move the tags list from Python into the database, creating a new table tags for that. Then, you can move the logic implemented with the determineTags function inside the database, defining the AFTER INSERT trigger on the posts table. From within this trigger, you will also insert rows into the posttags table, thus eliminating the need to do it from within the insertPost function. Once you’ve done all that, you can refactor the Python code implemented in the appsample module. To summarize, here are the steps you need to perform in order to refactor the sample application discussed in the earlier article: Create tags table and populate it with the data currently stored in the  tags list implemented in Python. Define the AFTER INSERT trigger on the posts table. Refactor the insertPost function in the appsample.py module. Remove the tags list from the appsample.py module. Remove the determineTags function from the appsample.py module. Refactor the execPr function in the appsample.py module. Refactoring the Underlying Database To keep things simple, the tags table might contain a single column tag with the primary key constraint defined on it. So, you can create the tags table as follows: CREATE TABLE tags ( tag VARCHAR(20) PRIMARY KEY ) ENGINE = InnoDB; Then, you might want to modify the posttags table, adding a foreign key constraint to its tag column. Before you can do that, though, you will need to delete all the rows from this table. This can be done with the following query: DELETE FROM posttags; Now you can move on and alter posttags as follows: ALTER TABLE posttags ADD FOREIGN KEY (tag) REFERENCES tags(tag); The next step is to populate the tags table. You can automate this process with the help of the following Python script: >>> import MySQLdb >>> import appsample >>> db=MySQLdb.connect(host="localhost",user="usrsample",passwd="pswd",db=">>> dbsample") >>> c=db.cursor() >>> c.executemany("""INSERT INTO tags VALUES(%s)""", appsample.tags) >>> db.commit() >>> db.close() As a result, you should have the tags table populated with the data taken from the tags list discussed in Python Data Persistence using MySQL. To make sure it has done so, you can turn back to the mysql prompt and issue the following query against the tags table: SELECT * FROM tags; The above should output the list of tags you have in the tags list. Of course, you can always extend this list, adding new tags with the INSERT statement. For example, you could issue the following statement to add the Visual Studio tag: INSERT INTO tags VALUES('Visual Studio'); Now you can move on and define the AFTER INSERT trigger on the posts table: delimiter // CREATE TRIGGER insertPost AFTER INSERT ON posts FOR EACH ROW BEGIN INSERT INTO posttags(title, tag) SELECT NEW.title as title, tag FROM tags WHERE LOCATE(tag, NEW.title)>0; END // delimiter ; As you can see, the posttags table will be automatically populated with appropriate tags just after a new row is inserted into the posts table. Notice the use of the INSERT … SELECT statement in the body of the trigger. Using this syntax lets you insert several rows into the posttags table at once, without having to use an explicit loop. In the WHERE clause of SELECT, you use standard MySQL string function LOCATE returning the position of the first occurrence of the substring, passed in as the first argument, in the string, passed in as the second argument. In this particular example, though, you are not really interested in obtaining the position of an occurrence of the substring in the string. All you need to find out here is whether the substring appears in the string or not. If it is, it should appear in the posttags table as a separate row associated with the row just inserted into the posts table. Refactoring the Sample’s Python Code Now that you have moved some data and data processing from Python into the underlying database, it’s time to reorganize the appsample custom Python module created as discussed in Python Data Persistence using MySQL. As mentioned earlier, you need to rewrite the insertPost and execPr functions and remove the determineTags function and the tags list. This is what the appsample module should look like after revising: import MySQLdb import urllib2 import xml.dom.minidom def obtainPost(): addr = "http://feeds.feedburner.com/packtpub/sDsa?format=xml" xmldoc = xml.dom.minidom.parseString(urllib2.urlopen(addr).read()) item = xmldoc.getElementsByTagName("item")[0] title = item.getElementsByTagName("title")[0].firstChild.data guid = item.getElementsByTagName("guid")[0].firstChild.data pubDate = item.getElementsByTagName("pubDate")[0].firstChild.data post ={"title": title, "guid": guid, "pubDate": pubDate} return post def insertPost(title, guid, pubDate): db=MySQLdb.connect(host="localhost",user="usrsample",passwd="pswd",db="dbsample") c=db.cursor() c.execute("""INSERT INTO posts (title, guid, pubDate) VALUES(%s,%s,%s)""", (title, guid, pubDate)) db.commit() db.close() def execPr(): p = obtainPost() insertPost(p["title"], p["guid"], p["pubDate"]) If you compare it with appsample discussed in Part 1, you should notice that the revision is much shorter. It’s important to note, however, that nothing has changed from the user standpoint. So, if you now start the execPr function in your Python session: >>>import appsample >>>appsample.execPr() This should insert a new record into the posts table, inserting automatically corresponding tags records into the posttags table, if any. The difference lies in the way it’s going on behind the scenes. Now the Python code is responsible only for obtaining the latest post from the Packt Book Feed page and then inserting a record into the posts table. Dealing with tags is now responsibility of the logic implemented inside the database. In particular, the AFTER INSERT trigger defined on the posts table should take care of inserting the rows into the posttags table. To make sure that everything has worked smoothly, you can now check out the content of the posts and posttags tables. To look at the latest post stored in the posts table, you could issue the following query: SELECT title, str_to_date(pubDate,'%a, %e %b %Y') lastdate FROM posts ORDER BY lastdate DESC LIMIT 1; Then, you might want to look at the related tags stored in the posttags tables, by issuing the following query: SELECT p.title, t.tag, str_to_date(p.pubDate,'%a, %e %b %Y') lastdate FROM posts p, posttags t WHERE p.title=t.title ORDER BY lastdate DESC LIMIT 1; Conclusion In this article, you looked at how some business logic of a Python/MySQL application can be moved from Python into MySQL. For that, you continued with the sample application originally discussed in Python Data Persistence using MySQL.

(For more resources on Microsoft Dynamics CRM, see here.) Editing a user dashboard After creating a user dashboard or getting access to another user dashboard, you may still need to adjust the layout and settings of the dashboard. Getting ready Navigate to the Dashboards section in the Dynamics CRM 2011 Workplace area. How to do it... Carry out the following steps in order to complete this recipe: Select the Dashboards link from the Workplace area. Select one of your user dashboards, as shown in the following screenshot: From the Dashboards menu in the Dynamics CRM 2011 ribbon, click on the Edit button, as highlighted in the following screenshot: The dashboard editor screen will open, and the dashboard is now in Edit mode, as shown in the following screenshot: In order to edit the components on the dashboard, select a component by clicking on it with the mouse, and then click on the Edit Component ribbon button, as shown in the following screenshot: There's more... Dynamics CRM has a robust security system that combines roles-based security and user permissions. These security settings allow the administrator to control access to data and functionality in the Dynamics CRM system. Security roles for editing user dashboards In order for a Dynamics CRM user to edit user dashboards, they must have a security role that grants the Write privilege for the User Dashboard entity . If a user's security role does not have this privilege, then they will not see the Edit button on the dashboard ribbon: Editing a system dashboard The system dashboards are intended to be viewed by all users of Dynamics CRM. These dashboards are created and managed by users with the System Customizer or System Administrator security roles (by default these roles have the Write privilege for the System Forms entity). Edits made to these dashboards are seen by all users. Getting ready Editing a System dashboard requires you to first navigate to the Customization section in the Dynamics CRM 2011 Settings area. How to do it... Carry out the following steps in order to complete this recipe: From the Customization section, click on the Customize the System link, as shown in the following screenshot: This will launch the solution editor dialog showing the Default Solution for Dynamics CRM 2011. Click on the Dashboards link located in the left-hand side navigation section, as shown in the following screenshot: A listing of system dashboards will be shown. Double-click on the Microsoft Dynamics CRM Overview dashboard record. This will launch the dashboard editor screen. In order to edit the components on the dashboard, select a component by clicking on it with the mouse, and then click on the Edit Component ribbon button, as shown in the following screenshot: There's more... Dynamics CRM has a robust security system that combines roles-based security and user permissions. These security settings allow the administrator to control access to data and functionality in the Dynamics CRM system. Security roles for editing system dashboards In order for a Dynamics CRM user to edit system dashboards, they must have a security role which grants the Write privilege for the System Form entity. If a user's security role does not have this privilege, then they will not be able to edit the dashboard when customizing the system. By default, the System Forms are only editable by users with the System Customizer or System Administrator security roles as they both have full privileges to the System Form entity. Deleting a user dashboard Creating new dashboards in Dynamics CRM is an excellent feature; however the on-going management of dashboards may require you to remove or delete some dashboards that are no longer needed. Deleting dashboards in Dynamics CRM cannot be undone; users should understand that deleting a dashboard is permanent. Getting ready Navigate to the Dashboards section in the Dynamics CRM 2011 Workplace area. How to do it... Carry out the following steps in order to complete this recipe: Select the Dashboards link from the Workplace area, as shown in the following screenshot: The user dashboards will be in the My Dashboards section of this list. Once you have selected a user dashboard, the Delete button in the Dashboards bar will be enabled. Click on the Delete button, as shown in the following screenshot: You will be prompted with a Confirm Deletion dialog . As the message in this dialog states, deleting a dashboard cannot be undone. If you want to continue and delete this dashboard from your system, click on the OK button. When the operation is finished, the screen will refresh and that dashboard will no longer be available. How it works... The layouts and settings used to generate user dashboards are stored as records in the Dynamics CRM database. Deleting the dashboard will remove this record from the CRM database and cannot be reversed. Deleting the dashboard will only remove the dashboard layout and settings, not the associated data.

In this article by Peter Verhas author of the book Java 9 Programming By Example, we will develop a simple sort program. Using this code as an example, we will look at different build tools, which are frequently used for Java projects, and learn the basic features of the Java language. (For more resources related to this topic, see here.) The problem we will solve The sorting problem is one of the oldest programming tasks that an engineer solves. We will have a set of records and we know that we will want to find a specific one sometime later, and we will want to find that one fast. To find it, we will sort the records in a specific order that helps finding the record we want fast. As an example, we can have the names of the students with some marks on cards. When students will come to the office asking for the result, we can turn all pages one after the other to find the name of the enquiring student. However, it is better if we sort the papers by the name of the students lexicographically. When a student comes, we can search the mark attached to the name much faster. We can look at the middle card; if it shows the name of the student, then we are happy to have found the name and the mark. If the card precedes the name of the student lexicographically, then we will continue searching in the second half, otherwise the first half. Following that approach, we can find the name of the student in no more steps than as many times the pack of cards can be halved. If we have two cards, then it is two steps at most. If it is four, then we will need three steps at most. If there are eight cards, then we may need four steps, but not more. If there are 1000 cards, then we may need at most 11 steps, while the original, non-sorted set will need 1000 steps, worst case. That is, approximately, it speeds up the search 100 times, so this is worth sorting the cards, unless the sorting itself takes too much time. In many cases, it is worth sorting the dataset and there are many sorting algorithms to do that. There are simpler and more complex algorithms, and as in many cases, more complex algorithms are the one that run faster. As we are focusing on the Java programming part and not the algorithm forging, in this article, we will develop a Java code that implements a simple and not-that-fast algorithm. Bubble sort The algorithm that we will implement in this article is well known as bubble sort. The approach is very simple. Begin at the start of the cards and compare the first and the second card. If the first card is later in lexicographic order than the second one, then swap the two cards. Then, repeat this for the card that is at the second place now, then the third, and so on. There is a card that is lexicographically the latest, say Wilson, and sometime later, we will get to this card as we go on swapping, the cards going from start to end. When we get this card and start to compare it with the next one, we will always swap them; this way, Wilson's card will travel to the last place where it has to be after the sort. All we have to do is repeat this travelling from the start and the occasional swapping of cards again, but this time only to the last but one element. This time, the second latest element will get to its place—say Wilkinson will be right before Wilson. If we have n cards, and we repeat this n-1 times, all cards will get to their place. Project structure and build tools When a project is more complex than a single class, and it usually is, then it is wise to define a project structure. We will have to decide where we store the source files, where the resource files (those that contain some resource for the program, but are not Java source) are, where should the .class files be written by the compiler, and so on. Generally, the structure is mainly the directory setup and configuring the tools that perform the build that use these tools. The compilation of complex programs cannot be feasibly done using the command line issuing javac commands. If we have a 100 Java source files, the compilation will require that many javac commands to be issued. We can write a simple bash script that does that. First, it will be just 100 lines, each compiling one source Java file to class file. Then, we will realize that this is only time, CPU, and power consuming to compile the files that are not changed since the last compilation. So, we can add some bash programming that checks the time stamp on the source and generated files. Then, we will probably realize that… whatever. At the end, we will end up with a tool that is essentially a build tool. And, this is already done. Instead of creating one, we will use a build tool that is ready. There are a few of them that can be found at https://en.wikipedia.org/wiki/List_of_build_automation_software Make The Make program was originally created in April 1976, so this is not a new tool. It is included in the Unix system so this tool is available without any extra installation on Linux, Mac OS X, or any other Unix-based system. Additionally, there are numerous ports of the tool on Windows and some version is/was included in the Visual C compiler toolset. The Make is not tied to Java. It was created when the major programming language was C, but it is not tied to C or any other language. Make is a dependency description language that has a very simple syntax. The Make, just like any other build tool, works controlled by a project description file. In case of make, this file contains a rule set. The description file is usually named Makefile, but in case the name of the description file is different, it can be specified as a command-line option to the make command. Rules in Makefile follow each other and a it is one or more lines. The first line starts at the first position (there is no tab or space at the start of the line) and the following lines start with a tab character. Thus, Makefile may look something like the following code: run : hello.jar java -cp hello.jar HelloWorld hello.jar : HelloWorld.class jar -cf hello.jar HelloWorld.class HelloWorld.class : HelloWorld.java javac HelloWorld.java The file defines three so-called targets: run, hello.jar, and HelloWorld.class. To create HelloWorld.class, type the following line at the Command Prompt: make HelloWorld.class The make will look at the rule and see that it depends on HelloWorld.java. If the HelloWorld.class file does not exist, or HelloWorld.java is newer than the Java source file, make will execute the command that is written on the next line and it will compile the Java source file. If the class file was created following the last modification of HelloWorld.java, then make knows that there is no need to run the command. In case of creating HelloWorld.class,the make program has an easy task. The source file was already there. If you issue the make hello.jar command, the procedure is more complex. The make command sees that in order to create hello.jar, it needs HelloWorld.class, which itself is also a target on another rule. Thus, it may need to be created. First, it starts the problem the same way as before. If HelloWorld.class is there, and is older than hello.jar, there is nothing to do. If it is not there, or is newer than hello.jar, then the jar -cf hello.jar HelloWorld.class command needs to be executed, but not yet. It remembers that this command has to be executed sometime in the future when all the commands that are needed to create HelloWorld.class are already executed successfully. Thus, it continues to create the class file exactly the same way as I already described earlier. In general, a rule can have the following format: target : dependencies command The make command can create any target using the make target command by first calculating which commands to execute and then executing them one by one. The commands are shell commands executing in a different process and may pose problems under Windows, which may render the Makefile files operating system dependent. Note that the run target is not an actual file that make creates. A target can be a file name or just a name for the target. In the latter case, make will never consider the readily available target. As we do not use make for Java project, there is no room to get into more details. Additionally, I cheated a bit by making the description of a rule simpler than it should be. The make tool has many powerful features out of the scope of this book. There are also several implementations that differ a little from each other. You will most probably meet the one made by the Free Software Foundation—the GNU make. And, of course, just in case of any Unix command-line tool, man is your friend. The man make command will display the documentation of the tool on the screen. The main points that you should remember about make are as follows: It defines the dependencies of the individual artifacts (targets) in a declarative way It defines the actions to create the missing artifacts in an imperative way. Summary In this article, we have developed a very basic sort algorithm. It was made purposefully simple so that we could reiterate on the basic and most important Java language elements, classes, packages, variables, methods, and so on. Resources for Article: Further resources on this subject: Algorithm Analysis [article] Introduction to C# and .NET [article] Parallel Computing [article]

(For more resources related to this topic, see here.) Map-reduce is a technique that is used to take large quantities of data and farm it out for processing. A somewhat trivial example might be: given 1TB of HTTP log data, count the number of hits that come from a given country, and report those numbers. For example, if you have the log entries: 204.12.226.2 - - [09/Jun/2013:09:12:24 -0700] "GET /who-we-are HTTP/1.0"404 471 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)"174.129.187.73 - - [09/Jun/2013:10:58:22 -0700] "GET /robots.txtHTTP/1.1" 404 452 "-" "CybEye.com/2.0 (compatible; MSIE 9.0; Windows NT5.1; Trident/4.0; GTB6.4)"157.55.35.37 - - [02/Jun/2013:23:31:01 -0700] "GET / HTTP/1.1" 200 483"-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"206.183.1.74 - - [02/Jun/2013:18:24:35 -0700] "GET / HTTP/1.1" 200 482"-" "Mozilla/4.0 (compatible; http://search.thunderstone.com/texis/websearch/about.html)"1.202.218.21 - - [02/Jun/2013:17:38:20 -0700] "GET /robots.txt HTTP/1.1"404 471 "-" "Mozilla/5.0 (compatible; JikeSpider; +http://shoulu.jike.com/spider.html)" Then the answer to the question would be as follows: US: 4China: 1 Clearly this example dataset does not warrant distributing the data processing among multiple machines, but imagine if instead of five rows of log data we had twenty-five billion rows. If your program took a single computer a half a second to process five records, it would take a little short of eighty years to process twenty-five billion records. To solve for this, we could break up the data into smaller chunks and then process those smaller chunks, rejoining them when we were finished. To apply this to a slightly larger dataset, imagine you extrapolated these five records to one hundred records and then split those one hundred records into five groups, each containing twenty records. From those five groups we might compute the following results: Group 1   Group 2   Group 3   Group 4   Group 5   US 5 Mexico 2 US 15 Italy 1 Finland 5 Greece 4 Scotland 6 China 2 Greece 4 China 5 Ireland 8 Canada 9 Finland 3 Scotland 10 US 10 Canada 3 Ireland 3     US 5     If we were to combine these data points by using the country name as a key and store them in a map, adding the value to any existing value, we would get the count per country across all one hundred records. Using Ruby, we can write a simple program to do this, first without using Gearman, and then with it. To demonstrate this, we will write the following: A simple library that we can use in our non-distributed program and in our Gearman-enabled programs An example program that demonstrates using the library A client that uses the library to split up our data and submit jobs to our manager A worker that uses the library to process the job requests and return the results The shared library First we will develop a library that we can reuse. This will demonstrate that you can reuse existing logic to quickly take advantage of Gearman because it ensures the following things: The program, client, and worker are much simpler so we can see what's going on in them The behavior between our program, client, and worker is guaranteed to be consistent The shared library will have two methods, map_data and reduce_data. The map_data method will be responsible for splitting up the data into chunks to be processed, and the reduce_data method will process those chunks of data and return something that can be merged together into an accurate answer. Take the following example, and save it to a file named functions.rb for later use: #!/bin/env ruby# Generate sub-lists of the data# each sub-list has size = blocksizedef map_data(lines, blocksize)blocks = []counter = 0block = []lines.each do |line|if (counter >= blocksize)blocks << blockblock = []counter = 0endblock << linecounter += 1endblocks << block if block.size> 0blocksend# Extract the number of times we see a unique line# Result is a hash with key = line, value = countdef reduce_data(lines)results = {}lines.each do |line|results[line] ||= 0results[line] += 1endresultsend A simple program To use this library, we can write a very simple program that demonstrates the functionality: require './functions.rb'countries = ["china", "us", "greece", "italy"]lines = []results = {}(1..100).each { |i| lines << countries[i % 4] }blocks = map_data(lines, 20)blocks.each do |block|reduce_data(block).each do |k,v|results[k] ||= 0results[k] += vendendputs results.inspect Put the contents of this example into a Ruby source file, named mapreduce.rb in the same directory as you placed your functions.rb file, and execute it with the following: [user@host:$] ruby ./mapreduce.rb This script will generate a list with one hundred elements in it. Since there are four distinct elements, each will appear 25 times as the following output shows: {"us"=>25, "greece"=>25, "italy"=>25, "china"=>25} Following in this vein, we can add in Gearman to extend our example to operate using a client that submits jobs and a single worker that will process the results serially to generate the same results. The reason we wrote these methods in a separate module from the driver application was to make them reusable in this fashion. The client The following code for the client in this example will be responsible for the mapping phase, it will split apart the results and submit jobs for the blocks of data it needs processed. In this example worker/client setup, we are using JSON as a simple way to serialize/deserialize data being sent back and forth: require 'rubygems'require 'gearman'require 'json'require './functions.rb'client = Gearman::Client.new('localhost:4730')taskset = Gearman::TaskSet.new(client)countries = ["china", "us", "greece", "italy"]jobcount = 1lines = []results = {}(1..100).each { |i| lines << countries[i % 4] }blocks = map_data(lines, 20)blocks.each do |block|# Generate a task with a unique iduniq = rand(36**8).to_s(36)task = Gearman::Task.new('count_countries',JSON.dump(block),:uniq =>uniq)# When the task is complete, add its results into ourstask.on_complete do |d|# We are passing data back and forth as JSON, so# decode it to a hash and then iterate over the# k=>v pairsJSON.parse(d).each do |k,v|results[k] ||= 0results[k] += vendendtaskset.add_task(task)puts "Submitted job #{jobcount}"jobcount += 1endputs "Submitted all jobs, waiting for results."start_time = Time.nowtaskset.wait(100)time_diff = (Time.now - start_time).to_iputs "Took #{time_diff} seconds: #{results.inspect}" This client uses a few new concepts that were not used in the introductory examples, that is, task sets and unique identifiers. In the Ruby client, a task set is a group of tasks that are submitted together and can be waited upon collectively. To generate a task set, you construct it by giving it the client that you want to submit the task set with: taskset = Gearman::TaskSet.new(client) Then you can create and add tasks to the task set: task = Gearman::Task.new('count_countries',JSON.dump(block), :uniq =>uniq)taskset.add_task(task) Finally, you tell the task set how long you want to wait for the results: taskset.wait(100) This will block the program until the timeout passes, or all the tasks in the task set complete hold true (again, complete does necessarily mean that the worker succeeded at the task, but that it saw it to completion). In this example, it will wait 100 seconds for all the tasks to complete before giving up on them. This doesn't mean that the jobs won't complete if the client disconnects, just that the client won't see the end results (which may or may not be acceptable). The worker To complete the distributed MapReduce example, we need to implement the worker that is responsible for performing the actual data processing. The worker will perform the following tasks: Receive a list of countries serialized as JSON from the manager Decode that JSON data into a Ruby structure Perform the reduce operation on the data converting the list of countries into a corresponding hash of counts Serialize the hash of counts as a JSON string Return the JSON string to the manager (to be passed on to the client) require 'rubygems'require 'gearman'require 'json'require './functions.rb'Gearman::Util.logger.level = Logger::DEBUG@servers = ['localhost:4730']w = Gearman::Worker.new(@servers)w.add_ability('count_countries') do |json_data,job|puts "Received: #{json_data}"data = JSON.parse(json_data)result = reduce_data(data)puts "Result: #{result.inspect}"returndata = JSON.dump(result)puts "Returning #{returndata}"sleep 4returndataendloop { w.work } Notice that we have introduced a slight delay in returning the results by instructing our worker to sleep for four seconds before returning the data. This is here in order to simulate a job that takes a while to process. To run this example, we will repeat the exercise from the first section. Save the contents of the client to a file called mapreduce_client.rb, and then contents of the worker to a file named mapreduce_worker.rb in the same directory as the functions.rb file. Then, start the worker first by running the following: ruby mapreduce_worker.rb And then start the client by running the following: ruby mapreduce_client.rb When you run these scripts, the worker will be waiting to pick up jobs, and then the client will generate five jobs, each with a block containing a list of countries to be counted, and submit them to the manager. These jobs will be picked up by the worker and then processed, one at a time, until they are all complete. As a result there will be a twenty second difference between when the jobs are submitted and when they are completed. Parallelizing the pipeline Implementing the solution this way clearly doesn't gain us much performance from the original example. In fact, it is going to be slower (even ignoring the four second sleep inside each job execution) than the original because there is time involved in serialization and deserialization of the data, transmitting the data between the actors, and transmitting the results between the actors. The goal of this exercise is to demonstrate building a system that can increase the number of workers and parallelize the processing of data, which we will see in the following exercise. To demonstrate the power of parallel processing, we can now run two copies of the worker. Simply open a new shell and execute the worker via ruby mapreduce_worker.rb and this will spin up a second copy of the worker that is ready to process jobs. Now, run the client a second time and observe the behavior. You will see that the client has completed in twelve seconds instead of twenty. Why not ten? Remember that we submitted five jobs, and each will take four seconds. Five jobs do not get divided evenly between two workers and so one worker will acquire three jobs instead of two, which will take it an additional four seconds to complete: [user@host]% ruby mapreduce_client.rbSubmitted job 1Submitted job 2Submitted job 3Submitted job 4Submitted job 5Submitted all jobs, waiting for results.Took 12 seconds: {"us"=>25, "greece"=>25, "italy"=>25, "china"=>25} Feel free to experiment with the various parameters of the system such as running more workers, increasing the number of records that are being processed, or adjusting the amount of time that the worker sleeps during a job. While this example does not involve processing enormous quantities of data, hopefully you can see how this can be expanded for future growth. Summary In this article, we have discussed MapReduce technique. Hope this article gives you a glimpse of how the book flows. Resources for Article : Further resources on this subject: BPMN 2.0 Concepts and The Sales Quote Process [Article] Simplifying Parallelism Complexity in C# [Article] Oracle BPM Suite 11gR1: Creating a BPM Application [Article]

(For more resources related to this topic, see here.) Drawing a bar chart with Flex The Flex framework offers some charting components that are fairly easy to use. It is not ActionScript per say, but it still compiles to the SWF format. Because the resulting charts look good and are pretty customizable, we decided to cover it in one recipe. There is a downside though to using this: the Flex framework will be included in your SWF, which will increase its size. Future recipes will explain how to do the same thing using just ActionScript. Getting ready Open FlashDevelop and create a new Flex Project. How to do it... The following are the steps required to build a bar chart using the Flex framework. Copy and paste the following code in the Main.mxml file. When you run it, it will show you a bar chart. <?xml version="1.0" encoding="utf-8"?> <s:Application minWidth="955" minHeight="600"> <fx:Script> <![CDATA[ import mx.collections.ArrayCollection; [Bindable] private var monthsAmount:ArrayCollection = new ArrayCollection( [ { Month: "January", Amount: 35}, { Month: "February", Amount: 32 }, { Month: "March", Amount: 27 } ]); ]]> </fx:Script> <mx:BarChart id="barchart" x="30" y="30" dataProvider="{monthsAmount}"> <mx:verticalAxis> <mx:CategoryAxis categoryField="Month"/> </mx:verticalAxis> <mx:horizontalAxis> <mx:LinearAxis minimum="10"/> </mx:horizontalAxis> <mx:series> <mx:BarSeries yField="Month" xField="Amount" /> </mx:series> </mx:BarChart> </s:Application> How it works... When you create a new Flex project, Flash Builder will generate for you the XML file and the Application tag. After that, in the script tag we created the data we will need to show in the chart. We do so by creating an ArrayCollection data structure, which is an array encapsulated to be used as DataProvider for multiple components of the Flex framework, in this case mx:BarChart. Once we have the data part done, we can start creating the chart. Everything is done in the BarChart tag. Inside that tag you can see we linked it with ArrayCollection, which we previously created using this code: dataProvider = "{monthsAmount}". Inside the BarChart tag we added the verticalAxis tag. This tag is used to associate values in the ArrayCollection to an axis. In this case we say that the values of the month will be displayed on the vertical axis. Next comes the horizontalAxis tag, we added it to tell the chart to use 10 as a minimum value for the horizontal axis. It's optional, but if you were to remove the tag it would use the smallest value in ArrayCollection as the minimum for the axis, so one month, in this case, March, would have no bar and the bar chart wouldn't look as good. Finally, the series tag will tell for a column, what data to use in ArrayCollection. You can basically think of the series as representing the bars in the chart. There's more... As we mentioned earlier, this component of the Flex framework is pretty customizable and you can use it to display multiple kinds of bar charts. Showing data tips Multiple options are available using this component; if you want to display the numbers that the bar represents in the chart while the user moves the mouse over the bar, simply add showDataTips = "true" inside the BarChart tag and it is done. Displaying vertical bars If you would like to use vertical bars instead of horizontal bars in the graph, Flex provides the ColumnChart charts to do so. In the previous code, change the BarChart tag to ColumnChart, and change BarSeries to ColumnSeries. Also, since the vertical axis and horizontal axis will be inverted, you will need verticalAxis by horizontalAxis and horizontalAxis by verticalAxis (switch them, but keep their internal tags) and in the ColumnSeries tag, xField should be Month and yField should be Amount. When you run that code it will show vertical bars. Adding more bars By adding more data in the ArrayCollection data structure and by adding another BarSeries tag, you can display multiple bars for each month. See the Adobe documentation at the following link to learn how to do it: http://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/mx/charts/BarChart.html. Building vertical bar charts Now that we have built a bar chart using Flex, we are ready to do the same in pure ActionScript. This bar chart version will allow you to expand it in multiple ways and will remove the weight that the Flex framework adds to the file size. Now a bit about bar charts; Bar charts are good when you don't have too much data (more than 20 bars starts to make a big chart), or when you've averaged it. It is a quick way to compare data visually. Getting ready All we will need for this is to start a new project in FlashDevelop. Also, it would help to read about preparing data and about axes in the book ActionScript Graphing Cookbook. How to do it... This section will refer a lot to the code provided with the book. You will notice that we divided all the elements in the charts into their own classes. It all starts in the Main.as file, where we create the data that we will use to display in the chart after that we just create the chart and add it to the display list. var data:Vector.<BarData> = new Vector.<BarData>(); data.push(new BarData("January", 60)); data.push(new BarData("February", 100)); data.push(new BarData("March", 30)); var chart:BarChart = new BarChart(data, 400, 410); chart.x = 30; chart.y = 30; addChild(chart); From here you can look into the BarData class, which it is just two variables, a string and a number that represents the data that we are going to show. We now need to create a class for all the elements that comprise a bar chart. They are: the bars, the vertical axis, and the horizontal axis. Now this recipe is building a vertical bar chart so the vertical axis is the one that will have numerical marks and the horizontal axis will have labels on the marks. First the Bar class: This class will only draw a rectangle with the height representing the data for a certain label. The following is its constructor: public function Bar(width:int, height:int) { graphics.beginFill(0xfca25a); graphics.drawRect(-width/2, 0, width, -height); graphics.endFill(); } The horizontal axis will take the x coordinate of the created bars and will place a label under it. public function HorizontalAxis(listOfMark:Vector.<Number>, data:Vector.<BarData>, width:Number) { drawAxisLine(new Point(0, 0), new Point(width, 0)); for (var i:int = 0; i < listOfMark.length; i++) { drawAxisLine(new Point(listOfMark[i], -3), new Point(listOfMark[i], 3)); var textField:TextField = new TextField(); textField.text = data[i].label; textField.width = textField.textWidth + 5; textField.height = textField.textHeight + 3; textField.x = listOfMark[i] - textField.width / 2; textField.y = 5; addChild(textField); } } Now the vertical axis will make 10 marks at regular interval and will add a label with the associated value in it: for (var i:int = 0; i < _numberOfMarks; i++) { drawAxisLine(new Point( -3, (i + 1) * -heightOfAxis / _ numberOfMarks ), new Point(3, (i + 1) * -heightOfAxis / _ numberOfMarks)); var textField:TextField = new TextField(); textField.text = String(((i + 1) / (_numberOfMarks)) * maximumValue ); textField.width = textField.textWidth + 5; textField.height = textField.textHeight + 3; textField.x = -textField.width - 3; textField.y = (i + 1) * -heightOfAxis / _numberOfMarks - textField.height / 2; addChild(textField); } Finally, the BarChart class will take the three classes we just created and put it all together. By iterating through all the data, it will find the maximum value, so that we know what range of values to put on the vertical axis. var i:int; var maximumValue:Number = data[0].data; for (i = 1; i < data.length; i++) { if (data[i].data > maximumValue) { maximumValue = data[i].data; } } After that we create each bar, notice that we also keep the position of each bar to give it to the horizontal axis thereafter: var listOfMarks:Vector.<Number> = new Vector.<Number>(); var bar:Bar; for (i = 0; i < data.length; i++) { bar = new Bar(_barWidth, data[i].data * scaleHeight); bar.x = MARGIN + _barSpacing + _barWidth / 2 + i * (_barWidth + _barSpacing); listOfMarks.push(bar.x - MARGIN); bar.y = height - MARGIN; addChild(bar); } Now all we have left to do is create the axes and then we are done; this is done really easily as shown in the following code: _horizontalAxis = new HorizontalAxis(listOfMarks, data, width - MARGIN); _horizontalAxis.x = MARGIN; _horizontalAxis.y = height - MARGIN; addChild(_horizontalAxis); _verticalAxis = new VerticalAxis(height - MARGIN, maximumValue); _verticalAxis.x = MARGIN; _verticalAxis.y = height -MARGIN; addChild(_verticalAxis); How it works... So we divided all the elements into their own classes because this will permit us to extend and modify them more easily in the future. So let's begin where it all starts, the data. Well, our BarChart class accepts a vector of BarData as an argument. We did this so that you could add as many bars as you want and the chart would still work. Be aware that if you add many bars, you might have to give more width to the chart so that it can accommodate them. You can see in the code, that the width of the bar of determined by the width of the graph divided by the number bars. We decided that 85 percent of that value would be given to the bars and 15 percent would be given to the space between the bars. Those values are arbitrary and you can play with them to give different styles to the chart. Also the other important step is to determine what our data range is. We do so by finding what the maximum value is. For simplicity, we assume that the values will start at 0, but the validity of a chart is always relative to the data, so if there are negative values it wouldn't work, but you could always fix this. So when we found our maximum value, we can decide for a scale for the rest of the values. You can use the following formula for it: var scaleHeight:Number = (height - 10) / maximumValue; Here, height is the height of the chart and 10 is just a margin we leave to the graph to place the labels. After that, if we multiply that scale by the value of the data, it will give us the height of each bar and there you have it, a completed bar chart. There's more... We created a very simple version of a bar chart but there are numerous things we could do to improve it. Styling, interactivity, and the possibility of accommodating a wider range of data are just some examples. Styling This basic chart could use a little bit of styling. By modifying the color of the bars, the font of the labels, and by adding a drop shadow to the bars, it could be greatly enhanced. You could also make all of them dynamic so that you could specify them when you create a new chart. Interactivity It would be really good to show the values for the bars when you move the mouse over them. Right now you can kind of get an idea of which one is the biggest bar but that is all. If this feature is implemented, you can get the exact value. Accommodating a wider data range As we explained earlier, we didn't account for all the data range. Values could be very different; some could be negative, some could be very small (between 0 and 1), or you would want to set the minimum and maximum value of the vertical axes. The good thing here is that you can modify the code to better fit your data.

  JavaFX 1.2 Application Development Cookbook Over 60 recipes to create rich Internet applications with many exciting features Easily develop feature-rich internet applications to interact with the user using various built-in components of JavaFX Make your application visually appealing by using various JavaFX classes—ListView, Slider, ProgressBar—to display your content and enhance its look with the help of CSS styling Enhance the look and feel of your application by embedding multimedia components such as images, audio, and video Part of Packt's Cookbook series: Each recipe is a carefully organized sequence of instructions to complete the task as efficiently as possible Read more about this book (For more resources on JavaFX, see here.) Using javafxc to compile JavaFX code While it certainly makes it easier to build JavaFX with the support of an IDE (see the NetBeans and Eclipse recipes), it is not a requirement. In some situations, having direct access to the SDK tools is preferred (automated build for instance). This recipe explores the build tools that are shipped with the JavaFX SDK and provides steps to show you how to manually compile your applications. Getting ready To use the SDK tools, you will need to download and install the JavaFX SDK. See the recipe Installing the JavaFX SDK, for instructions on how to do it. How to do it... Open your favorite text/code editor and type the following code. The full code is available from ch01/source-code/src/hello/HelloJavaFX.fx. package hello;import javafx.stage.Stage;import javafx.scene.Sceneimport javafx.scene.text.Text;import javafx.scene.text.Font;Stage { title: "Hello JavaFX" width: 250 height: 80 scene: Scene { content: [ Text { font : Font {size : 16} x: 10 y: 30 content: "Hello World!" } ] }} Save the file at location hello/Main.fx. To compile the file, invoke the JavaFX compiler from the command line from a directory up from the where the file is stored (for this example, it would be executed from the src directory): javafxc hello/Main.fx If your compilation command works properly, you will not get any messages back from the compiler. You will, however, see the file HelloJavaFX.class created by the compiler in the hello directory. If, however, you get a "file not found" error during compilation, ensure that you have properly specified the path to the HelloJavaFX.fx file. How it works... The javafxc compiler works in similar ways as your regular Java compiler. It parses and compiles the JavaFX script into Java byte code with the .class extension. javafxc accepts numerous command-line arguments to control how and what sources get compiled, as shown in the following command: javafxc [options] [sourcefiles] [@argfiles] where options are your command-line options, followed by one or more source files, which can be followed by list of argument files. Below are some of the more commonly javafxc arguments: classpath (-cp)—the classpath option specifies the locations (separated by a path separator character) where the compiler can find class files and/or library jar files that are required for building the application. javafxc -cp .:lib/mylibrary.jar MyClass.fx sourcepath—in more complicated project structure, you can use this option to specify one or more locations where the compiler should search for source file and satisfy source dependencies. javafxc -cp . -sourcepath .:src:src1:src2 MyClass.fx -d—with this option, you can set the target directory where compiled class files are to be stored. The compiler will create the package structure of the class under this directory and place the compiled JavaFX classes accordingly. javafxc -cp . -d build MyClass.fx The @argfiles option lets you specify a file which can contain javafxc command-line arguments. When the compiler is invoked and a @argfile is found, it uses the content of the file as an argument for javafxc. This can help shorten tediously long arguments into short, succinct commands. Assume file cmdargs has the following content: -d build-cp .:lib/api1.jar:lib/api2.jar:lib/api3.jar-sourcepath core/src:components/src:tools/src Then you can invoke javafxc as: $> javafxc @cmdargs See also Installing the JavaFX SDK Creating and using JavaFX classes JavaFX is an object-oriented scripting language. As such, object types, represented as classes, are part of the basic constructs of the language. This section shows how to declare, initialize, and use JavaFX classes. Getting ready If you have used other scripting languages such as ActionScript, JavaScript, Python, or PHP, the concepts presented in this section should be familiar. If you have no idea what a class is or what it should be, just remember this: a class is code that represents a logical entity (tree, person, organization, and so on) that you can manipulate programmatically or while using your application. A class usually exposes properties and operations to access the state or behavior of the class. How to do it... Let's assume we are building an application for a dealership. You may have a class called Vehicle to represent cars and other type of vehicles processed in the application. The next code example creates the Vehicle class. Refer to ch01/source-code/src/javafx/Vehicle.fx for full listing of the code presented here. Open your favorite text editor (or fire up your favorite IDE). Type the following class declaration: class Vehicle { var make; var model; var color; var year; function drive () : Void { println("You are driving a " "{year} {color} {make} {model}!") }} Once your class is properly declared, it is now ready to be used. To use the class, add the following (highlighted code) to the file: class Vehicle {...}var vehicle = Vehicle { year:2010 color: "Grey" make:"Mini" model:"Cooper"};vehicle.drive(); Save the file as Vehicle.fx. Now, from the command-line, compile it with: $> javafxc Vehicle.fx If you are using an IDE, you can simply right, click on the file to run it. When the code executes, you should see: $> You are driving a 2010 Grey Mini Cooper! How it works... The previous snippet shows how to declare a class in JavaFX. Albeit a simple class, it shows the basic structure of a JavaFX class. It has properties represented by variables declarations: var make;var model;var color;var year; and it has a function: function drive () : Void { println("You are driving a " "{year} {color} {make} {model}!")} which can update the properties and/or modify the behavior (for details on JavaFX functions, see the recipe Creating and Using JavaFX functions). In this example, when the function is invoked on a vehicle object, it causes the object to display information about the vehicle on the console prompt. Object literal initialization Another aspect of JavaFX class usage is object declaration. JavaFX supports object literal declaration to initialize a new instance of the class. This format lets developers declaratively create a new instance of a class using the class's literal representation and pass in property literal values directly into the initialization block to the object's named public properties. var vehicle = Vehicle { year:2010 color: "Grey" make:"Mini" model:"Cooper"}; The previous snippet declares variable vehicle and assigns to it a new instance of the Vehicle class with year = 2010, color = Grey, make = Mini, and model = Cooper. The values that are passed in the literal block overwrite the default values of the named public properties. There's more... JavaFX class definition mechanism does not support a constructor as in languages such as Java and C#. However, to allow developers to hook into the life cycle of the object's instance creation phase, JavaFX exposes a specialized code block called init{} to let developers provide custom code which is executed during object initialization. Initialization block Code in the init block is executed as one of the final steps of object creation after properties declared in the object literal are initialized. Developers can use this facility to initialize values and initialize resources that the new object will need. To illustrate how this works, the previous code snippet has been modified with an init block. You can get the full listing of the code at ch01/source-code/src/javafx/Vehicle2.fx. class Vehicle {... init { color = "Black"; } function drive () : Void { println("You are driving a " "{year} {color} {make} {model}!"); }}var vehicle = Vehicle { year:2010 make:"Mini" model:"Cooper"};vehicle.drive(); Notice that the object literal declaration of object vehicle no longer includes the color declaration. Nevertheless, the value of property color will be initialized to Black in the init{} code block during the object's initialization. When you run the application, it should display: You are driving a 2010 Black Mini Cooper! See also Declaring and using variables in JavaFX Creating and using JavaFX functions Creating and using variables in JavaFX JavaFX is a statically type-safe and type-strict scripting language. Therefore, variables (and anything which can be assigned to a variable, including functions and expressions) in JavaFX, must be associated with a type, which indicates the expected behavior and representation of the variable. This sections explores how to create, initialize, and update JavaFX variables. Getting ready Before we look at creating and using variables, it is beneficial to have an understanding of what is meant by data type and be familiar with some common data types such as String, Integer, Float, and Boolean. If you have written code in other scripting languages such as ActionScript, Python, and Ruby, you will find the concepts in this recipe easy to understand. How to do it... JavaFX provides two ways of declaring variables including the def and the var keywords. def X_STEP = 50;prntln (X_STEP);X_STEP++; // causes errorvar x : Number;x = 100;...x = x + X_LOC; How it works... In JavaFX, there are two ways of declaring a variable: def—The def keyword is used to declare and assign constant values. Once a variable is declared with the def keyword and assigned a value, it is not allowed be reassigned a new value. var—The var keyword declares variables which are able to be updated at any point after their declaration. There's more... All variables must have an associated type. The type can be declared explicitly or be automatically coerced by the compiler. Unlike Java (similar to ActionScript and Scala), the type of the variable follows the variable's name separated by a colon. var location:String; Explicit type declaration The following code specifies the type (class) that the variable will receive at runtime: var location:String;location = "New York"; The compiler also supports a short-hand notation that combines declaration and initialization. var location:String = "New York"; Implicit coercion In this format, the type is left out of the declaration. The compiler automatically converts the variable to the proper type based on the assignment. var location;location = "New York"; Variable location will automatically receive a type of String during compilation because the first assignment is a string literal. Or, the short-hand version: var location = "New York"; JavaFX types Similar to other languages, JavaFX supports a complete set of primitive types as listed: :String—this type represents a collection of characters contained within within quotes (double or single, see following). Unlike Java, the default value for String is empty (""). "The quick brown fox jumps over the lazy dog" or 'The quick brown fox jumps over the lazy dog' :Number—this is a numeric type that represents all numbers with decimal points. It is backed by the 64-bit double precision floating point Java type. The default value of Number is 0.0. 0.01234100.01.24e12 :Integer—this is a numeric type that represents all integral numbers. It is backed by the 32-bit integer Java type. The default value of an Integer is 0. -44700xFF :Boolean—as the name implies, this type represents the binary value of either true or false. :Duration—this type represent a unit of time. You will encounter its use heavily in animation and other instances where temporal values are needed. The supported units include ms, s, m, and h for millisecond, second, minute, and hour respectively. 12ms4s12h0.5m :Void—this type indicates that an expression or a function returns no value. Literal representation of Void is null. Variable scope Variables can have three distinct scopes, which implicitly indicates the access level of the variable when it is being used. Script level Script variables are defined at any point within the JavaFX script file outside of any code block (including class definition). When a script-level variable is declared, by default it is globally visible within the script and is not accessible from outside the script (without additional access modifiers). Instance level A variable that is defined at the top-level of a class is referred to as an instance variable. An instance level is visible within the class by the class members and can be accessed by creating an instance of the class. Local level The least visible scope are local variables. They are declared within code blocks such as functions. They are visible only to members within the block.

(For more resources on Python, see here.) So let's get on with it! Installation prerequisites Before we jump in to the main topic, it is necessary to install the following packages. Python In this article, we will use Python Version 2.6, or to be more specific, Version 2.6.4. It can be downloaded from the following location: http://python.org/download/releases/ Windows platform For Windows, just download and install the platform-specific binary distribution of Python 2.6.4. Other platforms For other platforms, such as Linux, Python is probably already installed on your machine. If the installed version is not 2.6, build and install it from the source distribution. If you are using a package manager on a Linux system, search for Python 2.6. It is likely that you will find the Python distribution there. Then, for instance, Ubuntu users can install Python from the command prompt as: $sudo apt-get python2.6 Note that for this, you must have administrative permission on the machine on which you are installing Python. Python Imaging Library (PIL) We will learn image-processing techniques by making extensive use of the Python Imaging Library (PIL) throughout this article. PIL is an open source library. You can download it from http://www.pythonware.com/products/pil/. Install the PIL Version 1.1.6 or later. Windows platform For Windows users, installation is straightforward—use the binary distribution PIL 1.1.6 for Python 2.6. Other platforms For other platforms, install PIL 1.1.6 from the source. Carefully review the README file in the source distribution for the platform-specific instructions. Libraries listed in the following table are required to be installed before installing PIL from the source. For some platforms like Linux, the libraries provided in the OS should work fine. However, if those do not work, install a pre-built "libraryName-devel" version of the library. For example, for JPEG support, the name will contain "jpeg-devel-", and something similar for the others. This is generally applicable to rpm-based distributions. For Linux flavors like Ubuntu, you can use the following command in a shell window. $sudo apt-get install python-imaging However, you should make sure that this installs Version 1.1.6 or later. Check PIL documentation for further platform-specific instructions. For Mac OSX, see if you can use fink to install these libraries. See http://www.finkproject.org/ for more details. You can also check the website http://pythonmac.org or Darwin ports website http://darwinports.com/ to see if a binary package installer is available. If such a pre-built version is not available for any library, install it from the source. The PIL prerequisites for installing PIL from source are listed in the following table: Library URL Version Installation options (a) or (b) libjpeg (JPEG support) http://www.ijg.org/files 7 or 6a or 6b (a) Pre-built version. For example: jpeg-devel-7 Check if you can do: sudo apt-install libjpeg (works on some flavors of Linux) (b) Source tarball. For example: jpegsrc.v7.tar.gz zib (PNG support) http://www.gzip.org/zlib/ 1.2.3 or later (a) Pre-built version. For example: zlib-devel-1.2.3.. (b) Install from the source. freetype2 (OpenType /TrueType support) http://www.freetype.org 2.1.3 or later (a) Pre-built version. For example: freetype2-devel-2.1.3.. (b) Install from the source. PyQt4 This package provides Python bindings for Qt libraries. We will use PyQt4 to generate GUI for the image-processing application that we will develop later in this article. The GPL version is available at: http://www.riverbankcomputing.co.uk/software/pyqt/download. Windows platform Download and install the binary distribution pertaining to Python 2.6. For example, the executable file's name could be 'PyQt-Py2.6-gpl-4.6.2-2.exe'. Other than Python, it includes everything needed for GUI development using PyQt. Other platforms Before building PyQt, you must install SIP Python binding generator. For further details, refer to the SIP homepage: http://www.riverbankcomputing.com/software/sip/. After installing SIP, download and install PyQt 4.6.2 or later, from the source tarball. For Linux/Unix source, the filename will start with PyQt-x11-gpl-.. and for Mac OS X, PyQt-mac-gpl-... Linux users should also check if PyQt4 distribution is already available through the package manager. Summary of installation prerequisites   Package Download location Version Windows platform Linux/Unix/OS X platforms Python http://python.org/download/releases/ 2.6.4 (or any 2.6.x) Install using binary distribution (a) Install from binary; Also install additional developer packages (For example, with python-devel in the package name in the rpm systems) OR (b) Build and install from the source tarball. (c) MAC users can also check websites such as http://darwinports.com/ or http://pythonmac.org/. PIL http://www.pythonware.com/products/pil/ 1.1.6 or later Install PIL 1.1.6 (binary) for Python 2.6 (a) Install prerequisites if needed. Refer to Table #1 and the README file in PIL source distribution. (b) Install PIL from source. (c) MAC users can also check websites like http://darwinports.com/ or http://pythonmac.org/. PyQt4 http://www.riverbankcomputing.co.uk/software/pyqt/download 4.6.2 or later Install using binary pertaining to Python 2.6 (a) First install SIP 4.9 or later. (b) Then install PyQt4.

(For more resources related to this topic, see here.) Relationship queries Relationship queries are mainly used to query the records from one or more objects in a single SOQL statement in Salesforce.com. We cannot query the records from more than one object without having a relationship between the objects. Filtering multiselect picklist values The INCLUDES and EXCLUDES operators are used to filter the multiselect picklist field. The multiselect picklist field in Salesforce allows the user to select more than one value from the list of values provided. Sorting in both the ascending and descending orders Sometimes, we may get a chance to sort the records when we fetch these using the SOQL statements based on two fields, one field in the ascending order and another field in the descending order. The following sample query will help us to achieve this easily: SELECT Name, Industry FROM Account ORDER By Name ASC, Industry DESC Using the preceding SOQL query, the accounts will first be sorted by Name in the ascending order and then by Industry in the descending order. The following screenshot shows the output of the SOQL execution: First, the records are arranged in the ascending order of the account's Name, and then it is sorted by Industry in the descending order. Using the GROUP BY ROLLUP clause The GROUP BY ROLLUP clause is used to add subtotals for aggregated data in query results. A query with a GROUP BY ROLLUP clause returns the same aggregated data as an equivalent query with a GROUP BY clause. It also returns multiple levels of subtotal rows. You can include up to three fields in a comma-separated list in a GROUP BY ROLLUP clause. Using the FOR REFERENCE clause The FOR REFERENCE clause is used to find the date/time when a record has been referenced. The LastReferencedDate field is updated for any retrieved records. The FOR REFERENCE clause is used to track the date/time when a record has been referenced last while executing a SOQL query. Using the FOR VIEW clause The FOR VIEW clause is used to find the date when a record has been last viewed. The LastViewedDate field is updated for any retrieved records. The FOR VIEW clause is used to track the date when the record was viewed last while executing a SOQL query. Using the GROUP BY CUBE clause The GROUP BY CUBE clause is used to add subtotals for every possible combination of the grouped field in the query results. The GROUP BY CUBE clause can be used with aggregate functions such as SUM() and COUNT(fieldName). A SOQL query with a GROUP BY CUBE clause retrieves the same aggregated records as an equivalent query with a GROUP BY clause. It also retrieves additional subtotal rows for each combination of fields specified in the comma-separated grouping list as well as the grand total. Using the OFFSET clause The OFFSET clause is used to specify the starting row number from which the records will be fetched. The OFFSET clause will be very useful when we implement pagination in the Visualforce page. The OFFSET clause along with Limits very useful in retrieving a subset of the records. The OFFSET usage in SOQL has many limitations and restrictions. Summary In this article, we saw how to query the records from more than one object using the relationship queries. The steps to get the relationship name among objects were also provided. Querying the records using both standard relationship and custom relationship was also discussed. Resources for Article: Further resources on this subject: Learning to Fly with Force.com [Article] Working with Home Page Components and Custom Links [Article] Salesforce CRM Functions [Article]

Setting up Liferay Portal As an administrator at the enterprise, you need to undertake a lot of administration tasks, such as installing Liferay portal, installing and setting up databases, and so on. You can install Liferay Portal through different ways, based on your specific needs. Normally, there are three main installation options: Using an open source bundle—It is the easiest and fastest installation method to install Liferay portal as a bundle. By using a Java SE runtime environment with an embedded database, you simply unzip and run the bundle. Detailed installation procedure—You can install the portal in an existing application server. This option is available for all the supported application servers. Using the extension environment—You can use a full development environment to extend the functionality. We will take up the third installation option "Using the extension environment" in the coming section. Using Liferay Portal Bundled with Tomcat 5.5 in Windows First let's consider one scenario when you, as an administrator, need to install Liferay portal in Windows with MySQL database, and your local Java version is JavaSE 5.0. Let's install Liferay portal bundled with Tomcat 5.5 in Windows as follows: Download Liferay Portal bundled with Tomcat for JDK 5.0 from Liferay official web site. Unzip the bundled file. Set up MySQL database as follows:create database liferay;grant all on liferay.* to 'liferay'@'localhost' identified by'liferay' with grant option;grant all on liferay.* to 'liferay'@'localhost.localdomain'identified by 'liferay' with grant option; Create a database and account in MySQL: Copy the MySQL JDBC driver mysql.jar to $TOMCAT_DIR/lib/ext; Comment the Hypersonic data source (HSQL) configuration and uncomment MySQL configuration ($TOMCAT_DIR/conf/Catalina/localhost/ROOT.xml):<!-- Hypersonic --><!--<Resource name="jdbc/LiferayPool" auth="Container"type="javax.sql.DataSource" driverClassName="org.hsqldb.jdbcDriver"url="jdbc:hsqldb:lportal"username="sa"password=""maxActive="20" /> --><!-- MySQL --><Resource name="jdbc/LiferayPool" auth="Container"type="javax.sql.DataSource" driverClassName="com.mysql.jdbc.Driver"url="jdbc:mysql://localhost/liferay?useUnicode=true&amp;characterEncoding=UTF-8"username="liferay"password="liferay"maxActive="20" /> Run $TOMCAT_DIR /bin/startup.bat. Open your browser and go to http://localhost:8080 (here we assume that it is a local installation, otherwise use the real host name or IP). Login as an administrator—User: test@liferay.com and Password: test. Note that the bundle comes with an embedded HSQL database loaded with sample data from the public website of Liferay. Do not use the Hypersonic in production. Using Liferay Portal Bundled with Tomcat 6.x in Linux Let's consider another scenario when you, as an administrator, need to install Liferay portal in Linux with MySQL database, and your local Java version is Java 6.0. Let's install Liferay portal bundled with Tomcat 6.0 in Linux as follows: Download Liferay Portal bundled with Tomcat 6.0 from Liferay official web site. Unzip the bundled file. Create a database and account in MySQL (as stated before). Run $TOMCAT_DIR/bin/startup.sh. Open your browser and go to http://localhost:8080 (assuming local installation; otherwise use the real host name or IP). Log in as an administrator—User: test@liferay.com and Password: test. Note that, Liferay Portal creates the tables it needs along with example data, the first time it starts. Furthermore, it is necessary to make the script executable by running chmod +x filename.sh. It is often necessary to run the executable from the directory where it resides. Using More Options for Liferay Portal Installation You can use one of the following options for Servlet containers and full Java EE application servers to install Liferay Portal: Geronimo + Tomcat Glassfish for AIX Glassfish for Linux Glassfish for OSX Glassfish for Solaris Glassfish for Solaris (x86) Glassfish for Windows JBoss + Jetty 4.0 JBoss + Tomcat 4.0 JBoss + Tomcat 4.2 Jetty JOnAS + Jetty JOnAS + Tomcat Pramati Resin Tomcat 5.5 for JDK 1.4 Tomcat 5.5 for JDK 5.0 Tomcat 6.0 You can choose a preferred bundle according to your requirements and download it from the official download page directly. Simply go to the website http://www.liferay.com and click on Downloads page. Flexible Deployment Matrix As an administrator, you can install Liferay Portals on all major application servers, databases, and operating systems. There are over 700 ways to deploy Liferay Portal. Thus, you can reuse your existing resources, stick to your budget and get an immediate return on you investment that everyone can be happy with. In general, you can install Liferay portal in Linux, UNIX and Windows with any one of the following application servers (or Servlet containers) and by selecting any one of the following database systems. The applications servers (or Servlet containers) that Liferay Portal can run on, include: Borland ES 6.5 Apache Geronimo 2.x Sun GlassFish 2 UR1 JBoss 4.0.x, 4.2.x JOnAS 4.8.x JRun 4 Updater 3 OracleAS 10.1.3.x Orion 2.0.7 Pramati 5.0 RexIP 2.5 SUN JSAS 9.1 WebLogic 8.1 SP4, 9.2, 10 WebSphere 5.1, 6.0.x, 6.1.x Jetty 5.1.10 Resin 3.0.19 Tomcat 5.0.x/5.5.x/6.0.x Databases that Liferay portal can run on include: Apache Derby IBM DB2 Firebird Hypersonic Informix InterBase JDataStore MySQL Oracle PostgresSQL SAP SQL Server Sybase Operating systems that Liferay portal can run on include: LINUX (Debian, RedHat, SUSE, Ubuntu, and so on.) UNIX (AIX, FreeBSD, HP-UX, OS X, Solaris, and so on.) WINDOWS MAC OS X

What is an RSS Feed? First of all, let us understand what a web feed is. Basically, it is a data format that provides frequently updated content to users. Content distributors syndicate the web feed, allowing users to subscribe, by using feed aggregator. RSS feeds contain data in an XML format. RSS is the term used for describing Really Simple Syndication, RDF Site Summary, or Rich Site Summary, depending upon the different versions. RDF (Resource Description Framework), a family of W3C specification, is a data model format for modelling various information such as title, author, modified date, content etc through variety of syntax format. RDF is basically designed to be read by computers for exchanging information. Since, RSS is an XML format for data representation, different authorities defined different formats of RSS across different versions like 0.90, 0.91, 0.92, 0.93, 0.94, 1.0 and 2.0. The following table shows when and by whom were the different RSS versions proposed. RSS Version Year Developer's Name RSS 0.90 1999 Netscape introduced RSS 0.90. RSS 0.91 1999 Netscape proposed the simpler format of RSS 0.91. 1999 UserLand Software proposed the RSS specification. RSS 1.0 2000 O'Reilly released RSS 1.0. RSS 2.0 2000 UserLand Software proposed the further RSS specification in this version and it is the most popular RSS format being used these days. Meanwhile, Harvard Law school is responsible for the further development of the RSS specification. There had been a competition like scenario for developing the different versions of RSS between UserLand, Netscape and O'Reilly before the official RSS 2.0 specification was released. For a detailed history of these different versions of RSS you can check http://www.rss-specifications.com/history-rss.htm The current version RSS is 2.0 and it is the common format for publishing RSS feeds these days. Like RSS, there is another format that uses the XML language for publishing web feeds. It is known as ATOM feed, and is most commonly used in Wiki and blogging software. Please refer to http://en.wikipedia.org/wiki/ATOM for detail. The following is the RSS icon that denotes links with RSS feeds. If you're using Mozilla's Firefox web browser then you're likely to see the above image in the address bar of the browser for subscribing to an RSS feed link available in any given page. Web browsers like Firefox and Safari discover available RSS feeds in web pages by looking at the Internet media type application/rss+xml. The following tag specifies that this web page is linked with the RSS feed URL: http://www.example.com/rss.xml<link href="http://www.example.com/rss.xml" rel="alternate" type="application/rss+xml" title="Sitewide RSS Feed" /> Example of RSS 2.0 format First of all, let’s look at a simple example of the RSS format. <?xml version="1.0" encoding="UTF-8" ?><rss version="2.0"><channel> <title>Title of the feed</title> <link>http://www.examples.com</link> <description>Description of feed</description> <item> <title>News1 heading</title> <link>http://www.example.com/news-1</link> <description>detail of news1 </description> </item> <item> <title>News2 heading</title> <link>http://www.example.com/news-2</link> <description>detail of news2 </description> </item></channel></rss> The first line is the XML declaration that indicates its version is 1.0. The character encoding is UTF-8. UTF-8 characters support many European and Asian characters so it is widely used as character encoding in web. The next line is the rss declaration, which declares that it is a RSS document of version 2.0 The next line contains the <channel> element which is used for describing the detail of the RSS feed. The <channel> element must have three required elements <title>, <link> and <description>. The title tag contains the title of that particular feed. Similarly, the link element contains the hyperlink of the channel and the description tag describes or carries the main information of the channel. This tag usually contains the information in detail. Furthermore, each <channel> element may have one or more <item> elements which contain the story of the feed. Each <item> element must have the three elements <title>, <link> and <description> whose use is similar to those of channel elements, but they describe the details of each individual items. Finally, the last two lines are the closing tags for the <channel> and <rss> elements. Creating RSS Web Widget The RSS widget we're going to build is a simple one which displays the headlines from the RSS feed, along with the title of the RSS feed. This is another widget which uses some JavaScript, PHP CSS and HTML. The content of the widget is displayed within an Iframe so when you set up the widget, you've to adjust the height and width. To parse the RSS feed in XML format, I've used the popular PHP RSS Parser – Magpie RSS. The homepage of Magpie RSS is located at http://magpierss.sourceforge.net/. Introduction to Magpie RSS Before writing the code, let's understand what the benefits of using the Magpie framework are, and how it works. It is easy to use. While other RSS parsers are normally limited for parsing certain RSS versions, this parser parses most RSS formats i.e. RSS 0.90 to 2.0, as well as ATOM feed. Magpie RSS supports Integrated Object Cache which means that the second request to parse the same RSS feed is fast— it will be fetched from the cache. Now, let's quickly understand how Magpie RSS is used to parse the RSS feed. I'm going to pick the example from their homepage for demonstration. require_once 'rss_fetch.inc';$url = 'http://www.getacoder.com/rss.xml';$rss = fetch_rss($url);echo "Site: ", $rss->channel['title'], "<br>";foreach ($rss->items as $item ) { $title = $item[title]; $url = $item[link]; echo "<a href=$url>$title</a></li><br>";} If you're more interested in trying other PHP RSS parsers then you might like to check out SimplePie RSS Parser (http://simplepie.org/) and LastRSS (http://lastrss.oslab.net/). You can see in the first line how the rss_fetch.inc file is included in the working file. After that, the URL of the RSS feed from getacoder.com is assigned to the $url variable. The fetch_rss() function of Magpie is used for fetching data and converting this data into RSS Objects. In the next line, the title of RSS feed is displayed using the code $rss->channel['title']. The other lines are used for displaying each of the RSS feed's items. Each feed item is stored within an $rss->items array, and the foreach() loop is used to loop through each element of the array. Writing Code for our RSS Widget As I've already discussed, this widget is going to use Iframe for displaying the content of the widget, so let's look at the JavaScript code for embedding Iframe within the HTML code. var widget_string = '<iframe src="http://www.yourserver.com/rsswidget/rss_parse_handler.php?rss_url=';widget_string += encodeURIComponent(rss_widget_url);widget_string += '&maxlinks='+rss_widget_max_links;widget_string += '" height="'+rss_widget_height+'" width="'+rss_widget_width+'"';widget_string += ' style="border:1px solid #FF0000;"';widget_string +=' scrolling="no" frameborder="0"></iframe>';document.write(widget_string); In the above code, the widget string variable contains the string for displaying the widget. The source of Iframe is assigned to rss_parse_handler.php. The URL of the RSS feed, and the headlines from the feed are passed to rss_parse_handler.php via the GET method, using rss_url and maxlinks parameters respectively. The values to these parameters are assigned from the Javascript variables rss_widget_url and rss_widget_max_links. The width and height of the Iframe are also assigned from JavaScript variables, namely rss_widget_width and rss_widget_height. The red border on the widget is displayed by assigning 1px solid #FF0000 to the border attribute using the inline styling of CSS. Since, Inline CSS is used, the frameborder property is set to 0 (i.e. the border of the frame is zero). Displaying borders from the CSS has some benefit over employing the frameborder property. While using CSS code, 1px dashed #FF0000 (border-width border-style border-color) means you can display a dashed border (you can't using frameborder), and you can use the border-right, border-left, border-top, border-bottom attributes of CSS to display borders at specified positions of the object. The scrolling property is set to no here, which means that the scroll bar will not be displayed in the widget if the widget content overflows. If you want to show a scroll bar, then you can set this property to yes. The values of JavaScript variables like rss_widget_url, rss_widget_max_links etc come from the page where we'll be using this widget. You'll see how the values of these variables will be assigned from the section at the end where we'll look at how to use this RSS widget.  

 In this article by Mahender Pal, the author of the book Microsoft Dynamics CRM 2015 Application Design, we will see how Microsoft Dynamics CRM provides different components that can be highly extended to map our custom business requirements. Although CRM provides a rich set of features that help us execute different business operations without any modification. However, we can still extend its behavior and capabilities with the supported customization. (For more resources related to this topic, see here.) The following is the extendibility architecture of CRM 2015, where we can see how different components interact with each other and what are the components that can be extended with the help of CRM APIs: Extendibility Architecture Let's discuss these components one by one and the possible extendibility options for them. CRM databases During installation of CRM, two databases, organization and configuration, are created. The organization database is created with the name of organization_MSCRM and the configuration database is created with the name of MSCRM_CONFIG. The organization database contains complete organization-related data stored on different entities. For every entity in CRM, there is a corresponding table with the name of Entityname+Base. Although technically it is possible but any direct data modification in these tables are not supported. Any changes to CRM data should be done by using CRM APIs only. Adding indexes to the CRM database is supported, you can refer to https://msdn.microsoft.com/en-us/library/gg328350.aspx for more details on supported customizations. Apart from table, CRM also creates a special view for every entity with the name of Filtered+Entityname. These fields view provide data based on the user security role; so for example, if you are a sales person you will only get data while querying filtered views based on the sales person role. We use filtered views for writing custom reports for CRM. You can refer to more details on filtered views at https://technet.microsoft.com/en-us/library/dn531182.aspx. Entity relationship diagram can be downloaded from https://msdn.microsoft.com/en-us/library/jj602918.aspx for CRM 2015. The Platform Layer Platform layer works as middleware between CRM UI and database, it is responsible for executing inbuilt and custom business logic and moving data back and forth. When we browse a CRM application, the platform layer presents data that is available based on the current user security roles. When we develop and deploy custom component on the top of platform layer. Process Process is a way of implementing automation in CRM. We can set up process using process designer and also develop custom assemblies to enhance the capability of workflow designer and include custom steps. CRM web services CRM provides Windows Communication Foundation (WCF) based web services, which help us interact with organization data and metadata; so whenever we want to create or modify an entity's data or want to customize a CRM component's metadata, we need to utilize these web services. We can also develop our custom web services with the help of CRM web services if required. We will be discussing more about CRM web services in details in a later topic. Plugins Plugins are another way of extending the CRM capability. These are .NET assemblies that help us implement our custom business logic in the CRM platform. It helps us to execute our business logic before or after the main platform operation. We can also run our plugin on a transaction that is similar to a SQL transaction, which means if any operation failed, all the changes under transaction will rollback. We can setup asynchronous and synchronous plugins. Reporting CRM provides rich reporting capabilities. We have many out of box reports for every module such as sales, marketing, and service. We can also create new reports and customize existing reports in Visual Studio. While working with reports, we always utilize an entity-specific filtered view so that data can be exposed based on the user security role. We should never use a CRM table while writing reports. Custom reports can be developed using out of box report wizard or using Visual Studio. The report wizard helps us create reports by following a couple of screens where we can select an entity and filter the criteria for our report with different rendering and formatting options. We can create two types of reports in Visual Studio SSRS and FetchXML. Custom SSRS reports are supported on CRM on premise deployment whereas CRM online only FetchXML. You can refer to https://technet.microsoft.com/en-us/library/dn531183.aspx for more details on report development. Client extensions We can also extend the CRM application from the Web and Outlook client. We can also develop custom utility tools for it. Sitemap and Command bar editor add-ons are example of such applications. We can modify different CRM components such as entity structure, web resources, business rules, different type of web resources, and other components. CRM web services can be utilized to map custom requirements. We can make navigational changes from CRM clients by modifying Sitemap and Command Bar definition. Integrated extensions We can also develop custom extensions in terms of custom utility and middle layer to interact with CRM using APIs. It can be a portal application or any .NET or non .NET utility. CRM SDK comes with many tools that help us to develop these integrated applications. We will be discussing more on custom integration with CRM in a later topic. Introduction to Microsoft Dynamics CRM SDK Microsoft Dynamics CRM SDK contains resources that help us develop code for CRM. It includes different CRM APIs and helpful resources such as sample codes (both server side and client side) and a list of tools to facilitate CRM development. It provides a complete documentation of the APIs, methods, and their uses, so if you are a CRM developer, technical consultant, or solution architect, the first thing you need to make sure is to download the latest CRM SDK. You can download the latest version of CRM SDK from http://www.microsoft.com/en-us/download/details.aspx?id=44567. The following table talks about the different resources that come with CRM SDK: Name Descriptions Bin This folder contains all the assemblies of CRM. Resources This folder contains different resources such as data import maps, default entity ribbon XML definition, and images icons of CRM applications. SampleCode This folder contains all the server side and client side sample code that can help you get started with the CRM development. This folder also contains sample PowerShell commands. Schemas This folder contains XML schemas for CRM entities, command bars, and sitemap. These schemas can be imported in Visual Studio while editing the customization of the .xml file manually. Solutions This folder contains the CRM 2015 solution compatibility chart and one portal solution. Templates This folder contains the visual studio templates that can be used to develop components for a unified service desk and the CRM package deployment. Tools This folder contains tools that are shipped with CRM SDK such as the metadata browser that can used to get CRM entity metadata, plugin registration tool, web resource utility, and others. Walkthroughts This folder contains console and web portal applications. CrmSdk2015 This is the .chm help file. EntityMetadata This file contains entity metadata information. Message-entity support for plugins This is a very important file that will help you understand events available for entities to write custom business logic (plug-ins) Learning about CRM assemblies CRM SDK ships with different assemblies under the bin folder that we can use to write CRM application extension. We can utilize them to interact with CRM metadata and organization data. The following table provides details about the most common CRM assemblies: Name Details Microsoft.Xrm.Sdk.Deployment This assembly is used to work with the CRM organization. We can create, update, and delete organization assembly methods. Microsoft.Xrm.Sdk This is very important assembly as it contains the core methods and their details, this assembly is used for every CRM extension. This assembly contains different namespaces for different functionality, for example Query, which contains different classes to query CRM DB; Metadata, which help us interact with the metadata of the CRM application; Discovery, which help us interact with the discover service (we will be discussing the discovery services in a later topic); Messages, which provide classes for all CURD operation requests and responses with metadata classes. Microsoft.Xrm.Sdk.Workflow This assembly helps us extend the CRM workflows' capability. It contains methods and types which are required for writing custom workflow activity. This assembly contains the activities namespace, which is used by the CRM workflow designer. Microsoft.Crm.Sdk.Proxy This assembly contains all noncore requests and response messages. Microsoft.Xrm.Tooling This is a new assembly added in SDK. This assembly helps to write Windows client applications for CRM Microsoft.Xrm.Portal This assembly provides methods for portal development, which includes security management, cache management, and content management. Microsoft.Xrm.Client This is another assembly that is used in the CRM client application to communicate with CRM from the application. It contains connection classes that we can use to setup the connection using different CRM authentication methods. We will be working with these APIs in later topics. Understanding CRM web services Microsoft Dynamics CRM provides web service support, which can be used to work with CRM data or metadata. CRM web services are mentioned here. The deployment service The deployment service helps us work with organizations. Using this web service, we can create a new organization, delete, or update existing organizations. The discovery service Discovery services help us identify correct web service endpoints based on the user. Let's take an example where we have multiple CRM organizations, and we want to get a list of the organization where current users have access, so we can utilize discovery service to find out unique organization ID, endpoint URL and other details. We will be working with discovery service in a later topic. The organization service The organization service is used to work with CRM organization data and metadata. It has the CRUD method and other request and response messages. For example, if we want to create or modify any existing entity record, we can use organization service methods. The organization data service The organization data service is a RESTful service that we can use to get data from CRM. We can use this service's CRUD methods to work with data, but we can't use this service to work with CRM metadata. To work with CRM web services, we can use the following two programming models: Late bound Early bound Early bound In early bound classes, we use proxy classes which are generated by CrmSvcUtil.exe. This utility is included in CRM SDK under the SDKBin path. This utility generates classes for every entity available in the CRM system. In this programming model, a schema name is used to refer to an entity and its attributes. This provides intelligence support, so we don't need to remember the entity and attributes name; as soon as we type the first letter of the entity name, it will display all the entities with that name. We can use the following syntax to generate proxy class for CRM on premise: CrmSvcUtil.exe /url:http://<ServerName>/<organizationName>/XRMServices/2011/ Organization.svc /out:proxyfilename.cs /username:<username> /password:<password> /domain:<domainName> /namespace:<outputNamespace> /serviceContextName:<serviceContextName> The following is the code to generate proxy for CRM online: CrmSvcUtil.exe /url:https://orgname.api.crm.dynamics.com/XRMServices/2011/ Organization.svc /out:proxyfilename.cs /username:"myname@myorg.onmicrosoft.com" /password:"myp@ssword! Organization service URLs can be obtained by navigating to Settings | Customization | Developer Resources. We are using CRM online for our demo. In case of CRM online, the organization service URL is dependent on the region where your organization is hosted. You can refer to https://msdn.microsoft.com/en-us/library/gg328127.aspx to get details about different CRM online regions. We can follow these steps to generate the proxy class for CRM online: Navigate to Developer Command Prompt under Visual Studio Tools in your development machine where visual studio is installed. Go to the Bin folder under CRM SDK and paste the preceding command: CrmSvcUtil.exe /url:https://ORGName.api.crm5.dynamics.com/XRMServices/2011/ Organization.svc /out:Xrm.cs /username:"user@ORGName.onmicrosoft.com" /password:"password" CrmSVCUtil Once this file is generated, we can add this file to our visual studio solution. Late bound In the late bound programming model, we use the generic Entity object to refer to entities, which means that we can also refer an entity which is not part of the CRM yet. In this programming mode, we need to use logical names to refer to an entity and its attribute. No intelligence support is available during code development in case of late bound. The following is an example of using the Entity class: Entity AccountObj = new Entity("account"); Using Client APIs for a CRM connection CRM client API helps us connect with CRM easily from .NET applications. It simplifies the developer's task to setup connection with CRM using a simplified connection string. We can use this connection string to create a organization service object. The following is the setup to console applications for our demo: Connect to Visual Studio and go to File | New | Project. Select Visual C# | Console Application and fill CRMConnectiondemo under the Name textbox as shown in the following screenshot: Console app Make sure you have installed the .NET 4.5.2 and .NET 4.5.2 developer packs before creating sample applications. Right-click on References and add the following CRM SDK: Microsoft.Xrm.SDK Microsoft.Xrm.Client We also need to add the following .NET assemblies System.Runtime.Serialization System.Configuration Make sure to add the App.config file if not available under project. We need to right-click on Project Name | Add Item and add Application Configuration File as shown here: app.configfile We need to add a connection string to our app.config file; we are using CRM online for our demo application, so we will be using following connection string: <?xml version="1.0" encoding="UTF-8"?> <configuration> <connectionStrings> <add name="OrganizationService" connectionString="Url=https://CRMOnlineServerURL; Username=User@ORGNAME.onmicrosoft.com; Password=Password;" /> </connectionStrings> </configuration> Right-click on Project, select Add Existing File, and browse our file that we generated earlier to add to our console application. Now we can add two classes in our application—one for early bound and another for late bound and let's name them Earlybound.cs and Latebound.cs You can refer to https://msdn.microsoft.com/en-us/library/jj602970.aspx to connection string for other deployment type, if not using CRM online After adding the preceding classes, our solution structure should look like this: Working with organization web services Whenever we need to interact with CRM SDK, we need to use the CRM web services. Most of the time, we will be working with the Organization service to create and modify data. Organization services contains the following methods to interact with metadata and organization data, we will add these methods to our corresponding Earlybound.cs and Latebound.cs files in our console application. Create This method is used to create system or custom entity records. We can use this method when we want to create entity records using CRM SDK, for example, if we need to develop one utility for data import, we can use this method or we want to create lead record in dynamics from a custom website. This methods takes an entity object as a parameter and returns GUID of the record created. The following is an example of creating an account record with early and late bound. With different data types, we are setting some of the basic account entity fields in our code: Early bound: private void CreateAccount() { using (OrganizationService crmService = new OrganizationService("OrganizationService")) { Account accountObject = new Account { Name = "HIMBAP Early Bound Example", Address1_City = "Delhi", CustomerTypeCode = new OptionSetValue(3), DoNotEMail = false, Revenue = new Money(5000), NumberOfEmployees = 50, LastUsedInCampaign = new DateTime(2015, 3, 2) }; crmService.Create(accountObject); } } Late bound: private void Create() { using (OrganizationService crmService = new OrganizationService("OrganizationService")) { Entity accountObj = new Entity("account"); //setting string value accountObj["name"] = "HIMBAP"; accountObj["address1_city"] = "Delhi"; accountObj["accountnumber"] = "101"; //setting optionsetvalue accountObj["customertypecode"] = new OptionSetValue(3); //setting boolean accountObj["donotemail"] = false; //setting money accountObj["revenue"] = new Money(5000); //setting entity reference/lookup accountObj["primarycontactid"] = new EntityReference("contact", new Guid("F6954457- 6005-E511-80F4-C4346BADC5F4")); //setting integer accountObj["numberofemployees"] = 50; //Date Time accountObj["lastusedincampaign"] = new DateTime(2015, 05, 13); Guid AccountID = crmService.Create(accountObj); } } We can also use the create method to create primary and related entity in a single call, for example in the following call, we are creating an account and the related contact record in a single call: private void CreateRecordwithRelatedEntity() { using (OrganizationService crmService = new OrganizationService("OrganizationService")) { Entity accountEntity = new Entity("account"); accountEntity["name"] = "HIMBAP Technology"; Entity relatedContact = new Entity("contact"); relatedContact["firstname"] = "Vikram"; relatedContact["lastname"] = "Singh"; EntityCollection Related = new EntityCollection(); Related.Entities.Add(relatedContact); Relationship accountcontactRel = new Relationship("contact_customer_accounts"); accountEntity.RelatedEntities.Add(accountcontactRel, Related); crmService.Create(accountEntity); } } In the preceding code, first we created account entity objects, and then we created an object of related contact entity and added it to entity collection. After that, we added a related entity collection to the primary entity with the entity relationship name; in this case, it is contact_customer_accounts. After that, we passed our account entity object to create a method to create an account and the related contact records. When we will run this code, it will create the account as shown here: relatedrecord Update This method is used to update existing record properties, for example, we might want to change the account city or any other address information. This methods takes the entity object as the parameter, but we need to make sure to update the primary key field to update any record. The following are the examples of updating the account city and setting the state property: Early bound: private void Update() { using (OrganizationService crmService = new OrganizationService("OrganizationService")) { Account accountUpdate = new Account { AccountId = new Guid("85A882EE-A500- E511-80F9-C4346BAC0E7C"), Address1_City = "Lad Bharol", Address1_StateOrProvince = "Himachal Pradesh" }; crmService.Update(accountUpdate); } } Late bound: private void Update() { using (OrganizationService crmService = new OrganizationService("OrganizationService")) { Entity accountUpdate = new Entity("account"); accountUpdate["accountid"] = new Guid("85A882EE-A500- E511-80F9-C4346BAC0E7C"); accountUpdate["address1_city"] = " Lad Bharol"; accountUpdate["address1_stateorprovince"] = "Himachal Pradesh"; crmService.Update(accountUpdate); } } Similarly, to create method, we can also use the update method to update the primary entity and the related entity in a single call as follows: private void Updateprimaryentitywithrelatedentity() { using (OrganizationService crmService = new OrganizationService("OrganizationService")) { Entity accountToUpdate = new Entity("account"); accountToUpdate["name"] = "HIMBAP Technology"; accountToUpdate["websiteurl"] = "www.himbap.com"; accountToUpdate["accountid"] = new Guid("29FC3E74- B30B-E511-80FC-C4346BAD26CC");//replace it with actual account id Entity relatedContact = new Entity("contact"); relatedContact["firstname"] = "Vikram"; relatedContact["lastname"] = "Singh"; relatedContact["jobtitle"] = "Sr Consultant"; relatedContact["contactid"] = new Guid("2AFC3E74- B30B-E511-80FC-C4346BAD26CC");//replace it with actual contact id EntityCollection Related = new EntityCollection(); Related.Entities.Add(relatedContact); Relationship accountcontactRel = new Relationship("contact_customer_accounts"); accountToUpdate.RelatedEntities.Add (accountcontactRel, Related); crmService.Update(accountToUpdate); } } Retrieve This method is used to get data from the CRM based on the primary field, which means that this will only return one record at a time. This method has the following three parameter: Entity: This is needed to pass the logical name of the entity as fist parameter ID: This is needed to pass the primary ID of the record that we want to query Columnset: This is needed to specify the fields list that we want to fetch The following are examples of using the retrieve method Early bound: private void Retrieve() { using (OrganizationService crmService = new OrganizationService("OrganizationService")) { Account retrievedAccount = (Account)crmService.Retrieve (Account.EntityLogicalName, new Guid("7D5E187C-9344-4267- 9EAC-DD32A0AB1A30"), new ColumnSet(new string[] { "name" })); //replace with actual account id } } Late bound: private void Retrieve() { using (OrganizationService crmService = new OrganizationService("OrganizationService")) { Entity retrievedAccount = (Entity)crmService.Retrieve("account", new Guid("7D5E187C- 9344-4267-9EAC-DD32A0AB1A30"), new ColumnSet(new string[] { "name"})); } RetrieveMultiple The RetrieveMultiple method provides options to define our query object where we can define criteria to fetch records from primary and related entities. This method takes the query object as a parameter and returns the entity collection as a response. The following are examples of using retrievemulitple with the early and late bounds: Late Bound: private void RetrieveMultiple() { using (OrganizationService crmService = new OrganizationService("OrganizationService")) { QueryExpression query = new QueryExpression { EntityName = "account", ColumnSet = new ColumnSet("name", "accountnumber"), Criteria = { FilterOperator = LogicalOperator.Or, Conditions = { new ConditionExpression { AttributeName = "address1_city", Operator = ConditionOperator.Equal, Values={"Delhi"} }, new ConditionExpression { AttributeName="accountnumber", Operator=ConditionOperator.NotNull } } } }; EntityCollection entityCollection = crmService.RetrieveMultiple(query); foreach (Entity result in entityCollection.Entities) { if (result.Contains("name")) { Console.WriteLine("name ->" + result.GetAttributeValue<string>("name").ToString()); } } } Early Bound: private void RetrieveMultiple() { using (OrganizationService crmService = new OrganizationService("OrganizationService")) { QueryExpression RetrieveAccountsQuery = new QueryExpression { EntityName = Account.EntityLogicalName, ColumnSet = new ColumnSet("name", "accountnumber"), Criteria = new FilterExpression { Conditions = { new ConditionExpression { AttributeName = "address1_city", Operator = ConditionOperator.Equal, Values = { "Delhi" } } } } }; EntityCollection entityCollection = crmService.RetrieveMultiple(RetrieveAccountsQuery); foreach (Entity result in entityCollection.Entities) { if (result.Contains("name")) { Console.WriteLine("name ->" + result.GetAttributeValue<string> ("name").ToString()); } } } } Delete This method is used to delete entity records from the CRM database. This methods takes the entityname and primaryid fields as parameters: private void Delete() { using (OrganizationService crmService = new OrganizationService("OrganizationService")) { crmService.Delete("account", new Guid("85A882EE-A500-E511- 80F9-C4346BAC0E7C")); } } Associate This method is used to setup a link between two related entities. It has the following parameters: Entity Name: This is the logical name of the primary entity Entity Id: This is the primary entity records it field. Relationship: This is the name of the relationship between two entities Related Entities: This is the correction of references The following is an example of using this method with an early bound: private void Associate() { using (OrganizationService crmService = new OrganizationService("OrganizationService")) { EntityReferenceCollection referenceEntities = new EntityReferenceCollection(); referenceEntities.Add(new EntityReference("account", new Guid("38FC3E74-B30B-E511-80FC-C4346BAD26CC"))); // Create an object that defines the relationship between the contact and account (we want to setup primary contact) Relationship relationship = new Relationship("account_primary_contact"); //Associate the contact with the accounts. crmService.Associate("contact", new Guid("38FC3E74-B30B- E511-80FC-C4346BAD26CC "), relationship, referenceEntities); } } Disassociate This method is the reverse of the associate. It is used to remove a link between two entity records. This method takes the same setup of parameter as associate method takes. The following is an example of a disassociate account and contact record: private void Disassociate() { using (OrganizationService crmService = new OrganizationService("OrganizationService")) { EntityReferenceCollection referenceEntities = new EntityReferenceCollection(); referenceEntities.Add(new EntityReference("account", new Guid("38FC3E74-B30B-E511-80FC-C4346BAD26CC "))); // Create an object that defines the relationship between the contact and account. Relationship relationship = new Relationship("account_primary_contact"); //Disassociate the records. crmService.Disassociate("contact", new Guid("15FC3E74- B30B-E511-80FC-C4346BAD26CC "), relationship, referenceEntities); } } Execute Apart from the common method that we discussed, the execute method helps to execute requests that is not available as a direct method. This method takes a request as a parameter and returns the response as a result. All the common methods that we used previously can also be used as a request with the execute method. The following is an example of working with metadata and creating a custom event entity using the execute method: private void Usingmetadata() { using (OrganizationService crmService = new OrganizationService("OrganizationService")) { CreateEntityRequest createRequest = new CreateEntityRequest { Entity = new EntityMetadata { SchemaName = "him_event", DisplayName = new Label("Event", 1033), DisplayCollectionName = new Label("Events", 1033), Description = new Label("Custom entity demo", 1033), OwnershipType = OwnershipTypes.UserOwned, IsActivity = false, }, PrimaryAttribute = new StringAttributeMetadata { SchemaName = "him_eventname", RequiredLevel = new AttributeRequiredLevelManagedProperty(AttributeRequiredLevel.None), MaxLength = 100, FormatName = StringFormatName.Text, DisplayName = new Label("Event Name", 1033), Description = new Label("Primary attribute demo", 1033) } }; crmService.Execute(createRequest); } } In the preceding code, we have utilized the CreateEntityRequest class, which is used to create a custom entity. After executing the preceding code, we can check out the entity under the default solution by navigating to Settings | Customizations | Customize the System. You can refer to https://msdn.microsoft.com/en-us/library/gg309553.aspx to see other requests that we can use with the execute method. Testing the console application After adding the preceding methods, we can test our console application by writing a simple test method where we can call our CRUD methods, for example, in the following example, we have added method in our Earlybound.cs. public void EarlyboundTesting() { Console.WriteLine("Creating Account Record....."); CreateAccount(); Console.WriteLine("Updating Account Record....."); Update(); Console.WriteLine("Retriving Account Record....."); Retrieve(); Console.WriteLine("Deleting Account Record....."); Delete(); } After that we can call this method in Main method of Program.cs file like below: static void Main(string[] args) { Earlybound obj = new Earlybound(); Console.WriteLine("Testing Early bound"); obj.EarlyboundTesting(); } Press F5 to run your console application. Summary In this article, you learned about the Microsoft Dynamics CRM 2015 SDK feature. We discussed various options that are available in CRM SDK. You learned about the different CRM APIs and their uses. You learned about different programming models in CRM to work with CRM SDK using different methods of CRM web services, and we created a sample console application. Resources for Article: Further resources on this subject: Attracting Leads and Building Your List [article] PostgreSQL in Action [article] Auto updating child records in Process Builder [article]