How-To Tutorials

What is AJAX? AJAX (Asynchronous JavaScript and XML) is a technique rather than a new technology for developing highly interactive web applications. Traditionally, when JavaScript is written, it uses the browser's XMLHttp DOM API class to make asynchronous calls to a server-side component, for example, servlets. The server-side component generates a resulting XML package and returns this to the client browser, which can then update the browser page without having to re-render the entire page. The result of using AJAX technologies (many different technologies can be used to develop AJAX functionality, for example, PHP, Microsoft .NET, Servlets, and Seam) is to provide an appearance similar to a desktop, for web applications. AJAX and the Seam Framework The Seam Framework provides built-in support for AJAX via its direct integration with libraries such as RichFaces and AJAX4JSF. Discussing the AJAX support of RichFaces and AJAX4JSF could fill an entire book, if not two books, so we'll discuss these technologies briefly, towards the end of this article, where we'll give an overview of how they can be used in a Seam application. However, Seam provides a separate technology called Seam Remoting that we'll discuss in detail in this article. Seam Remoting allows a method on Seam components to be executed directly from JavaScript code running within a browser, allowing us to easily build AJAX-style applications. Seam Remoting uses annotations and is conversation-aware, so that we still get all of the benefits of writing conversationally-aware components, except that we can now access them via JavaScript as well as through other view technologies, such as Facelets. Seam Remoting provides a ready-to-use framework, making AJAX applications easier to develop. For example, it provides debugging facilities and logging facilities similar to the ones that we use everyday when writing Java components. Configuring Seam applications for Seam Remoting To use Seam Remoting, we need to configure the Seam web application to support JavaScript code that is making asynchronous calls to the server back end. In a traditional servlet-based system this would require writing complex servlets that could read, parse, and return XML as part of an HTTP GET or POST request. With Seam Remoting, we don't need to worry about managing XML data and its transport mechanism. We don't even need to worry about writing servlets that can handle the communication for us—all of this is a part of the framework. To configure a web application to use Seam Remoting, all we need to do is declare the Seam Resource servlet within our application's WEB-INF/web.xml file. We do this as follows. <servlet> <servlet-name>Seam Resource Servlet</servlet-name> <servlet-class> org.jboss.seam.servlet.SeamResourceServlet </servlet-class></servlet><servlet-mapping> <servlet-name>Seam Resource Servlet</servlet-name> <url-pattern>/seam/resource/*</url-pattern></servlet-mapping> That's all we need to do to make a Seam web application work with Seam Remoting. To make things even easier, this configuration is automatically done when applications are created with SeamGen, so you would have to worry about this configuration only if you are using non-SeamGen created projects. Configuring Seam Remoting server side To declare that a Seam component can be used via Seam Remoting, the methods that are to be exposed need to be annotated with the @WebRemote annotation. For simple POJO components, this annotation is applied directly on the POJO itself, as shown in the following code snippet. @Name("helloWorld")public class HelloWorld implements HelloWorldAction { @WebRemote public String sayHello() { return "Hello world !!";} For Session Beans, the annotation must be applied on the Session Beans business interface rather than on the implementation class itself. A Session Bean interface would be declared as follows. import javax.ejb.Local;import org.jboss.seam.annotations.remoting.WebRemote;@Localpublic interface HelloWorldAction { @WebRemote public String sayHello(); @WebRemote public String sayHelloWithArgs(String name);} The implementation class is defined as follows: import javax.ejb.Stateless;import org.jboss.seam.annotations.Name;@Stateless@Name("helloWorld")public class HelloWorld implements HelloWorldAction { public String sayHello() { return "Hello world !!"; } public String sayHelloWithArgs(String name) { return "Hello "+name; }} Note that, to make a method available to Seam Remoting, all we need to do is to annotate the method with @WebRemote and then import the relevant class. As we can see in the preceding code, it doesn't matter how many parameters our methods take. Configuring Seam Remoting client side In the previous sections, we've seen that minimal configuration is required to enable Seam Remoting and to declare Seam components as Remoting-aware. Similarly in this section, we'll see that minimal work is required within a Facelets file to enable Remoting. The Seam Framework provides built-in JavaScript to enable Seam Remoting. To use this JavaScript, we first need to define it within a Facelets file in the following way: <script type="text/javascript" src="/HelloWorld/seam/resource/ remoting/resource/remote.js"></script><script type="text/javascript" src="/HelloWorld/seam/resource/ remoting/interface.js?helloWorld"> To include the relevant JavaScript into a Facelets page, we need to import the /seam/resource/remoting/resource/remote.js and /seam/resource/remoting/interface.js JavaScript files. These files are served via the Seam resource servlet that we defined earlier in this article. You can see that the interface.js file takes an argument defining the name of the Seam component that we will be accessing (this is the name of the component for which we have defined methods with the @WebRemote annotation). If we wish to use two or more different Seam components from a Remoting interface, we would specify their names as parameters to the interface.js file separated by using an "&", for example: <script type="text/javascript" src="/HelloWorld/seam/resource/ remoting/interface.js?helloWorld&mySecondComponent& myThirdComponent"> To specify that we will use Seam components from the web tier is straight-forward, however, the Seam tag library makes this even easier. Instead of specifying the JavaScript shown in the preceding examples, we can simply insert the <s:remote /> tag into Facelets, passing the name of the Seam component to use within the include parameter. <ui:compositiontemplate="layout/template.xhtml"> <ui:define name="body"> <h1>Hello World</h1> <s:remote include="helloWorld"/> To use the <s:remote /> tag, we need to import the Seam tag library, as shown in this example. When the web page is rendered, Seam will automatically generate the relevant JavaScript. If we are using the <s:remote /> tag and we want to invoke methods on multiple Seam components, we need to place the component names as comma-separated values within the include parameter of the tag instead, for example: <s:remote include="helloWorld, mySecondComponent, myThirdComponent" /> Invoking Seam components via Remoting Now that we have configured our web application, defined the services to be exposed from the server, and imported the JavaScript to perform the AJAX calls, we can execute our remote methods. To get an instance of a Seam component within JavaScript, we use the Seam.Component.getInstance() method. This method takes one parameter, which specifies the name of the Seam component that we wish to interact with. Seam.Component.getInstance("helloWorld") This method returns a reference to Seam Remoting JavaScript to allow our exposed @WebReference methods to be invoked. When invoking a method via JavaScript, we must specify any arguments to the method (possibly there will be none) and a callback function. The callback function will be invoked asynchronously when the server component's method has finished executing. Within the callback function we can perform any JavaScript processing (such as DOM processing) to give our required AJAX-style functionality. For example, to execute a simple Hello World client, passing no parameters to the server, we could define the following code within a Facelets file. <ui:define name="body"> <h1>Hello World</h1> <s:remote include="helloWorld"/> <p> <button onclick="javascript:sayHello()">Say Hello</button> </p> <p> <div id="helloResult"></div> </p> <script type="text/javascript"> function sayHello() { var callback = function(result) { document.getElementById("helloResult").innerHTML=result; }; Seam.Component.getInstance("helloWorld"). sayHello(callback); } </script></ui:define> Let's take a look at this code, one piece at a time, to see exactly what is happening. <s:remote include="helloWorld"/> <p> <button onclick="javascript:sayHello()">Say Hello</button> </p> In this part of the code, we have specified that we want to invoke methods on the helloWorld Seam component by using the <s:remote /> tag. We've then declared a button and specified that the sayHello() JavaScript method will be invoked when the button is clicked. <div id="helloResult"></div> Next we've defined an empty <div /> called helloResult. This <div /> will be populated via the JavaScript DOM API with the results from out server side method invocation. <script type="text/javascript"> function sayHello() { var callback = function(result) { document.getElementById("helloResult").innerHTML=result; }; Seam.Component.getInstance("helloWorld"). sayHello(callback); }</script> Next, we've defined our JavaScript function sayHello(), which is invoked when the button is clicked. This method declares a callback function that takes one parameter. The JavaScript DOM API uses this parameter to set the contents of the helloResult <div /> that we have defined earlier. So far, everything that we've done here has been simple JavaScript and hasn't used any Seam APIs. Finally, we invoke the Seam component using the Seam.Component.getInstance().sayHello() method, passing the callback function as the final parameter. When we open the page, the following flow of events occurs: The page is displayed with appropriate JavaScript created via the<s:remote /> tag. The user clicks on the button. The Seam JavaScript is invoked, which causes the sayHello() method on the helloWorld component to be invoked. The server side component completes execution, causing the JavaScript callback function to be invoked. The JavaScript DOM API uses the results from the server method to change the contents of the <div /> in the browser, without causing the entire page to be refreshed. This process shows how we've developed some AJAX functionality by writing a minimal amount of JavaScript, but more importantly, by not dealing with XML or the JavaScript XMLHttp class. The preceding code shows how we can easily invoke server side methods without passing any parameters. This code can easily be expanded to pass parameters, as shown in the following code snippet: <s:remote include="helloWorld"/><p> <button onclick="javascript:sayHelloWithArgs()"> Say Hello with Args </button></p><p> <div id="helloResult"></div></p><script type="text/javascript"> function sayHelloWithArgs() { var name = "David"; var callback = function(result) { document.getElementById("helloResult").innerHTML=result; }; Seam.Component.getInstance("helloWorld"). sayHelloWithArgs(name, callback); }</script> The preceding code shows that the process for invoking remote methods with parameters is similar to the process for invoking remote methods with no parameters. The important point to note is that the callback function is specified as the last parameter. When our simple application is run, we get the following screenshot. Clicking on either of the buttons on the page causes our AJAX code to be executed, and the text of the <div /> component to be changed. If we want to invoke a server side method via Seam Remoting and we want the method to be invoked as a part of a Seam conversation, we can use the Seam.Remoting.getcontext.setConversationId() method to set the conversation ID. This ID will then by used by the Seam Framework to ensure that the AJAX request is a part of the appropriate conversation. Seam.Remoting.getContext().setConversationId(#{conversation.id});

Data item Sections Earlier in our discussion on reports, we referred to the primary components of a report. The Triggers and Properties we have reviewed so far are the data processing components. Next in the report processing sequence are Sections. In Classic RD reports, Sections are the output layout and formatting components. In RTC reports, Sections have a much more limited, but still critically important, role. In the process of creating the initial report design, you may be entering data either completely manually as we've done in our example work, or you may use the Classic Report Wizard. If you use the Wizard, you will end up with Sections defined suitable for Classic Client Report processing. Those Sections may be only rough draft equivalents of what you may want your final report to look like, but they are a suitable starting place for the Classic RD layout work, if that were the tool you were going to use. If you are creating your report completely manually, that is by not using the Wizard, you may also find it appropriate to define Sections to the point that the Classic Client could print a basic, readable report. In our case, we are focusing our production report development effort on the RoleTailored Client, so we will invest minimal effort on Classic Client compatible report layouts. We might do just enough to allow test report runs for data examination purposes and logic flow debugging. However, creating basic Section layouts provides us with another benefit relative to our VS RD layout work, especially if we can create them using the Report Wizard, because all the fields to be used by VS RD must be specified in the Sections. Creating RTC reports via the Classic Report Wizard Let's look at the RTC report development flow again. The preceding image is very similar to the one we studied earlier in this articles, but this flowchart only shows the steps that are pertinent to VS RD. In Step 6 of this flow, there is an option to Create Layout Suggestion in the Visual Studio Report Designer as shown in the following screenshot: When you choose Create Layout Suggestion, the C/SIDE Report Designer will invoke a process that transforms the layout in Sections to a layout in the Visual Studio Report Designer. If a VS RD layout previously existed, the newly created layout will overwrite it. Therefore, this option will normally be used only once, in the initial stages of report design. Let's experiment by using the Report Wizard to create a simple report listing the gifts received by ICAN. We will access the Report Wizard in the Object Designer. Click on Reports | New, then fill in the Wizard screen as shown in the following image. Then click on OK and choose fields to display in the report as shown in the following screenshot. Click on Next, then choose the sorting order (that is index or key) that starts with Donor ID. Click on Next again and choose to Group the data by Donor ID. Click on Next again and choose to create totals for the Estimated Value field. One more, click on Next and choose the List Style for the report, then click on Finish. At this point, you will have generated a Classic Client report using the Report Wizard. If you View | Sections, you should see a C/SIDE report layout that looks much like the following screenshot. Let's save our newly generated report so that, if we need to, we can come back to this point as a checkpoint. Click on File | Save As and assign the report to ID 50002 with the Name of Gifts by Donor. Now click on Tools | Create Layout Suggestion. The process of transforming the Classic Report Layout to a Visual Studio Report Designer Layout will take a few seconds. When the report layout transformation process completes, you should see a screen that looks very similar to the following screenshot. The primary data layout portion of the same VS RD screen is shown in the next image. Compare this to the Classic RD data layout we just looked at a couple of steps ago. You will see some similarities and some considerable differences. Without doing anything else, let's save the VS RD layout we just created for the RoleTailored Client, then run both versions of the report to see the differences in the generated results. To save the VS RD layout, start by simply exiting the VS Report Designer. Once the VS RD screen closes, you will see the following question. Respond by clicking Yes. Then, when you exit the Classic Report Designer, you will see this question. Respond by clicking Yes. You will then be presented with the following message. Again, click on Yes. If there were an error in the RDLC created within the VS RD (such as an incorrect variable name used), an error message similar to the following would display. Since, hopefully, we didn't get such an error message, we can proceed to test both the Classic Client and the RoleTailored Client versions of our generated report. We can test the Classic Client (or C/SIDE RD) version of Report 50002 from the same Object Designer screen where we did our initial design work. Highlight the line for Report 50002 and click on the Run button. You should see the following screen: If we were running this as users, we might want to make a selection of specific Donors here on which to report. As we are just testing, simply click on Preview to see our report onscreen. The report will then appear, looking like the following: As you can see, with minimum development effort (and a minimum of technical knowledge), we have designed and created a report listing Gifts by Donor with subtotals by Donor. The report has proper page and column headings. Not only that, but the report was initiated withs a Request Form allowing application of filters. Close the Classic Client report; now let's run the RTC version. Just like we could do with Pages, we will run our Report test from the Windows Run option. Click on Run and enter the command to run Report 50002, as shown in the next screenshot. Click on OK. If the RoleTailored Client is not active, after a short pause, it will be activated. Then the Request Page will appear. Compare the look and contents of this Request Page with the one we saw previously for the Classic Client. As before, click on Preview and view the report. Of course, this time we're looking at the RTC version. This method of automatic transformation is very useful for getting an initial base for a new report or, obviously, for the complete generation process for a simple report where the requirements for layout are not too restrictive.

In this article by Pat Myers, author of the book Intelligent Document Capture with Ephesoft- Second Edition will explain about the following: Different classification types Other techniques for exporting your documents and metadata (For more resources related to this topic, see here.) As we know how to configure search classification to enable Ephesoft to recognize an invoice document. There are several other classification types available; we will explain these alternatives now. Classification types You can select the process that Ephesoft will use to classify documents by editing your batch class, editing the Document Assembly module, editing the Document Assembler plugin module within that module, and then selecting a value for DA Classification Type. Search Search classification (also sometimes called Lucene classification) is the default classification method and is recommended for most content. When configured to perform search classification, Ephesoft compares the text on each input page to the text on training documents to determine its confidence that a document is of a certain type. Image Image classification is the best option when classification cannot be made based on content. This occurs on forms that do not have a lot of text, or where the textual content is unpredictable but the physical appearance (such as layout, graphics, and formatting) is consistent. Credit card applications that are red dropout forms (where only the user-entered text is visible to the OCR engine) are candidates for this classification technique. Barcodes Barcodes can be used for documents that vary in content and layout, like white mail (unformatted correspondence received in the mail). If a barcode is found on the page with a name that matches an Ephesoft document type, Ephesoft will set the current document's type to that type. Automatic The automatic classification type tells Ephesoft to use the scores of every classification plugin that is enabled. This may be necessary when no single classification technique will suffice for your batch class, but configuring multiple classification plugins will have a negative impact on Ephesoft's performance. One document classification One document classification is a variant of automatic classification. It assembles all the pages in the batch into a single document. Confidence Ephesoft calculates confidence scores for each page in a batch. The page scores represent Ephesoft's certainty that the page being considered is the first, middle, or last page within each document type. They are used to classify and assemble the pages into documents. Ephesoft also uses these page scores to create an aggregate score for each document. This score is compared to the confidence threshold for each document type in the batch class definition. Any document that receives a confidence score below the minimum threshold will be flagged for review. A batch with one or more flagged documents will be placed in a queue for review by an operator. Confidence scores are calculated differently for each classification type. Search classification The default classification type is search classification. Search classification separates and classifies documents by using a two-step process. The first step is to collect information about the pages. The Search Classification plugin of the Page Processing module performs this function. The second step is to separate documents and determine their type. This is the responsibility of the Document Assembler plugin. The Search Classification plugin calculates the initial page scores by comparing the text on the page to the text on the training documents. Multiple scores are generated for each page as Ephesoft finds several matches from samples for any given page. The page scores are then adjusted using weighted values that can be modified in the administrative interface by editing the Search Classification plugin of the Page Processing module. Pages can be weighted on the basis of the page type (first, middle, or last). By default, Ephesoft is configured to reduce the scores for the middle and last pages by 10 percent and 20 percent, respectively, as the first pages are more important when it comes to the separation of documents. This effectively biases Ephesoft in favor of using a page to create a new document (over using it as the middle or last page of a document). The plugin properties of search classification Using the page scores calculated in the previous step (and adjusted using the weighted values from the Search Classification plugin), Ephesoft calculates all possible document assemblies and selects the result with the highest score. The score is calculated as follows: First, the scores of each page in the assembly are averaged. Ephesoft then adjusts the average by using a multiplier in the Document Assembler plugin. You will notice, looking at the following plugin settings screen, that there are several multipliers available. If the assembly has a first and a last page, for example, the DA Rule first-last Page multiplier will be chosen. An assembly with the first, last, and middle pages will use the "DA Rule First-middle-last Page" multiplier. The plugin properties of Document Assembler Suppose, for example, that you have trained a batch class to recognize the first and middle pages of an invoice. If you run a three-page batch through Ephesoft, you might get the following results: Page 1 is determined to be the first page of an invoice because Invoice_First_Page received the highest score: Page 1 compared to Invoice_First_Page receives a score of 30.2 Page 1 compared to Invoice_Middle_Page receives a score of 4.2 Page 2 is determined to be the second page of an invoice because Invoice_Middle_Page received the highest score. Because of the order of this page in the batch, it is determined to be the second page of the invoice found in page 1. Page 2 compared to Invoice_First_Page receives a score of 2.6 Page 2 compared to Invoice_Middle_Page receives a score of 12.2 Page 3 was determined to be the first page of an invoice because Invoice_First_Page received the highest score. Since, it was determined to be a first page, it is the first page of a new document. Page 3 compared to Invoice_First_Page receives a score of 31.6 Page 3 compared to Invoice_Middle_Page receives a score of 3.8 In this case, there is no score for Invoice_Last_Page as there were no last page samples used to train this Ephesoft instance. When using the drag and drop classification training in Batch Class Management, Ephesoft will automatically place a last page for any document having more than one page. If that is not the only possible last page of the document type, you will have to go into Folder Management and move all samples and files from the last page training for the document type into the middle pages. Once the files are moved, go back into Batch Class Management and click on the Learn Files button to retrain the system. The first document assembled will be a two-page invoice because Ephesoft found a first page of an invoice followed by a middle page of an invoice. The second document assembled will be a one-page invoice since only the first page of an invoice was found. The confidence scores that each of these documents received are calculated as follows: Document 1 (page 1 and 2): (30.2 + 12.2)/2 = 21.2 × 50% = 10.6 Average score of pages times the page weight factor, DA Rule First-middle Page Document 2 (page 3): (31.6)/1 = 31.6 × 50%= 15.8 Average score of pages times the page weight factor, DA Rule First Page If the Minimum Confidence Scores setting of the Invoice document type is set to 10, then this batch will skip the review step and move directly to extraction. If the Minimum Confidence Scores for the Invoice document type is set to 15, then this batch will stop in review with the first document requiring review. Barcode classification Barcode classification is also a two-step process similar to search classification. In the Page Processing module, pages with barcodes are processed using either the Recostar plugin or the Barcode Reader plugin. In the Document Assembler plugin, Ephesoft creates documents when the first barcode is found and all the other pages are appended to the document until a new page with a barcode is found. The barcode value found by the barcode or the RecoStar plugin has to match one of the document type names. On Linux, Ephesoft will always use the Barcode Reader plugin. Image classification Image classification compares the pixels on the provided documents to the pixels on the trained documents. The more pixels that match the trained document, the higher is the confidence score that the document will attain. This is in contrast to search classification which OCRs the pages and then compares the text. When image classification is selected, the Document Assembler plugin uses the image confidence scores to separate and classify documents. The assembly is done using the same algorithm explained in the search classification section. Automatic classification Automatic classification uses all enabled classification types. The scores will be combined to come up with an aggregate score per page. This value will be used for assembly and then classification scoring. Export We use the Copy Batch XML plugin to export content to the Ephesoft server's file system. There are a number of additional export options. The CMIS and DB export plugins use standard-based interfaces to allow export to a large number of enterprise content management systems and relational databases. Let's take a look at how to configure these two plugins and then, review the other plugins that are available. CMIS export The Content Management Interoperability Services (CMIS) API is an open standard for interacting with enterprise document repositories. You can use the CMIS Export plugin to export your scanned content (and associated metadata) to any repository that supports the CMIS standard, such as Alfresco, Documentum, FileNet, or SharePoint. Let's look at how to configure the CMIS Export plugin to send content to Alfresco, a popular open source enterprise content management system. Ephesoft 4.0 supports CMIS 1.0 and 1.1 Establish a content model in your CMIS Suppose that you have an Invoice document type in Ephesoft that has fields for Vendor Name, Invoice Date, and Invoice Total. The first thing that you will want to do is define a custom content model in Alfresco to represent your scanned content. Alfresco defines custom content models in XML files that look like the following: <type name="acme:invoice"> <parent>cm:content</parent> <properties> <property name="acme:vendorName"> <title>Vendor Name</title> <type>d:text</type> <mandatory enforced="false">false</mandatory> <index enabled="true"> <atomic>true</atomic> <stored>false</stored> <tokenised>false</tokenised> </index> </property> Alfresco document type and property name have prefixes to prevent namespace collisions in the content models. We have used an acme prefix in our examples, as would be the case if this implementation were for Acme Corporation. The example above shows a document type acme:invoice that extends Alfresco's base document type cm:content. This custom type has a text property called acme:vendorName. Not shown here are the date property called acme:invoiceDate and the float property called acme:invoiceTotal. Configure the CMIS Export plugin After creating the content model, you will need to configure Ephesoft to use CMIS to send the processed content to Alfresco. There are three places in Ephesoft where you need to configure the CMIS export: The plugin settings in the administrative user interface The mapping files, in your batch class cmis-plugin-mapping folder The global configuration file, located in your Ephesoft installation folder here: Application/WEB-INF/classes/META-INF/dcma-cmis/dcma-cmis.properties Let's start with the plugin settings. From the batch class management interface, select and edit your batch class, the export module, and then the CMIS Export plugin. This comes configured by default with a disabled sample connection to Alfresco's public CMIS server. The plugin properties of CMIS Export The CMIS plugin can be configured as follows: Root Folder Name: This is the name of the destination folder in the document repository where Ephesoft should load the exported documents. In Alfresco, this folder will be created underneath the root folder (which is typically named Company Home). Upload File Extension: This setting controls whether the documents are uploaded to your document management system as PDF or TIF images. Server URL: The services provided by CMIS are defined in an XML service document; this is the location of that document. Alfresco 4.0 hosts this file at /alfresco/service/cmis. Alfresco 5.0 hosts this file at /alfresco/api/-default-/public/cmis/versions/1.1/atom. User Name and Password: This is the authentication information required to connect to the document management system. Repository Id: Some document management systems are capable of hosting multiple repositories. When this is the case, each repository is listed in the service document with an associated identifier. You should examine the service document to find the identifier for your repository. Server Switch: This can be used to enable and disable export to your document management system. Aspect Switch: Alfresco manages dynamically assignable groups of properties called aspects. This switch enables support for aspects. Export File Name: Naming convention for the documents exported. Export Client Key, Secret Key, Refresh Token, Redirect URL, and Export Network: These properties are used to implement OAuth authentication. Document type and property mapping Next, you need to associate Ephesoft document types with Alfresco document types. Ephesoft's fields also need to be mapped to the properties of Alfresco documents. Edit this file in your batch class configuration area: cmis-plugin-mapping/DLF-Attribute-mapping.properties. This file contains some examples of content mapping. Delete the examples and set up your own mapping, as follows: Invoice=D:acme:invoice Invoice.VendorName=acme:vendorName Invoice.InvoiceDate=acme:invoiceDate Invoice.InvoiceTotal=acme:invoiceTotal The first line of this property file associates the document types, and the last three lines associate the fields. When mapping document types, you will need to prepend D: to the beginning of your document repository's type name. This is the CMIS syntax for representing a document (as opposed to, for example, a folder) in Alfresco. Aspects are configured in the following batch class configuration file: cmis-plugin-mapping/aspects-mapping.properties. Global CMIS configuration The final area where CMIS is configured in Ephesoft is the following file: Application/WEB-INF/classes/META-INF/dcma-cmis/dcma-cmis.properties. This file affects the CMIS configuration of all batch classes. The most commonly modified setting in this file is the date format. When you map a date field, Ephesoft needs to parse the date in order to reformat the information to match the CMIS specification. The cmis.date_format parameter specifies how Ephesoft fields that will be exported using CMIS will be formatted. See the JavaDoc for the SimpleDateFormat class to learn how to specify date formats. If your content management system uses Web Service Security (WSS) to secure its CMIS web services, you will need to adjust the value of the cmis.security.mode property. This specifies the security mode to use when attempting to connect to the CMIS web services. There are two possible values: basic and wssecurity. HTTP Basic Authentication is the default setting for the Ephesoft CMIS connection. This corresponds to the basic setting for the cmis.security.mode property. The cmis.security.mode property is set to wssecurity in order to have the CMIS credentials that are configured in the CMIS_EXPORT plugin included in the WS-Security SOAP header of the CMIS web service requests. If your CMIS web services are not addressable from a single URL, you can configure the location of each service used by Ephesoft. You will see a set of properties that begin with cmis.url. These can be edited to specify where your content management system hosts this service's WDSL. Database Export DB Export allows document level fields values and metadata to be exported to relational databases using JDBC. Administrators can map the Ephesoft document fields to the database table columns. First, go to the system configuration area to create a new connection in Connection Manager: Connection Manager with connection properties for database export Next, return to the batch class management area and configure your batch class. If the DB Export plugin is configured into this batch class' workflow, then you will be able to configure the plugin from the Modules section. The configuration of the plugin is simple; there is simply a switch to enable the plugin. Plugin properties of database export In Batch Class Management under the document type, you can configure DB Export Configuration. Select the correct database connection, and then, map the document type fields to the table and column. Click on Apply to save your changes. Database export mapping When the DB Export plugin runs, it will export the extracted field data for each document in the batch. Sample results of database export Other export plugins Thus far, we have shown you how to export to the local file system or use CMIS and JDBC. These are general-purpose plugins that can be used in a variety of situations. Ephesoft comes with a few other general-purpose plugins such as the CSV plugin and the tabbed PDF plugin. Ephesoft also provides a handful of plugins to facilitate export into specific content management systems such as Docushare, HPII FileNet, and IBM CM. To see the list of available plugins, you should edit your batch class and then edit the export module. Summary In this article, you have learned how to process forms with many different layouts, additional extraction techniques. At this point, you should be able to use Ephesoft to implement intelligent document capture for a wide variety of organizations. Resources for Article: Further resources on this subject: A Quick Tour Of Ephesoft[article] Loading, Submitting, and Validating Forms using Ext JS 4[article] Mastering the Newer Prezi Features [article]

One of the largest changes to Oracle is the recent acquisition of several other software lines and technologies. Oracle has combined all of these technologies and customers under a single support site called My Oracle Support at http://support.oracle. com, effective from Fall 2009. Along the way, Oracle also completely redesigned the interface, making it flash-based in order to provide a personalized GUI. To take full advantage of the personalization features, you will need to install a free utility on each node and each ORACLE_HOME you would like to monitor. The following paragraphs outline several reasons for use and suggestions for getting started. Configuration management Are you the only Oracle DBA in your company? How do you provide disaster recovery and redundancy for personnel in that situation? MOS has a tool that provides an Automatic Document Repository (my words) called Oracle Configuration Manager (OCM). The real purpose of this tool is to manage all of your configurations (different systems, servers, databases, application servers) when dealing with Oracle support. It is automatic in the sense that if you are out of the office, temporarily or permanently, the system configurations are available for viewing by anyone with the same Oracle Customer Support Identifier (CSI) number . The information is also available to Oracle support personnel. The repository is located on My Oracle Support. The systems are for you to choose, whether you want to only include production and/or non-production systems. What information does OCM collect and upload? It contains extensive hardware details, software installs (not just Oracle products), databases, and Oracle application servers. There is enough information to help in recreating your site if there is a complete disaster. The GUI interface allows managers and other IT personnel to see how nodes and applications are related and how they fit into your architectural framework. The information can only be updated by the upload process. Using OCM in disconnected mode with masking There is sensitive information being collected from the OCM tool. If you are employed by an organization that doesn't allow you to reveal such information or allow direct access by the servers to the Internet, there are steps to improve the security of this upload process. This section is highly recommended to be reviewed before enabling OCM. You must know what types of information are there and how that information is used before enabling uploading capabilities to a support website. To disable the collection of IP and MAC addresses, you add the following entries to the $ORACLE_HOME/ccr/config/collector.properties file. To disable the collection of network addresses, add the following entry: ccr.metric.host.ecm_hw_nic.inet_address=false To disable the collection of the MAC address, add the following entry: ccr.metric.host.ecm_hw_nic.mac_address=false The OCM collector collects the schema usernames for databases configured for configuration collections. The collection of this information is filtered or masked when ccr.metric.oracle_database.db_users.username is assigned the value of 'mask' in the $ORACLE_HOME/ccr/config/collector.properties file. The default behavior of the collector is to not mask this data. MOS customers may request deletion of their configuration information by logging a Service Request (SR) indicating the specific configuration information and scope of the deletion request. Disconnected mode is carried out with something called Oracle Support Hub, which is installed at your site. This hub is configured as a local secure site for direct uploads from your nodes, which the hub can then upload to MOS through the Internet. This protects each of your nodes from any type of direct Internet access. Finally, there is a way to do a manual upload of a single node using the method outlined in the MOS document 763142.1: How to upload the collection file ocmconfig.jar to My Oracle Support for Oracle Configuration Manager (OCM) running in Disconnected Mode. This is probably the safest method to use for OCM. Run it for a specific purpose with appropriate masking built-in and then request the information to be deleted by entering a SR request. These tips came from these locations as well as the OCM licensing agreement found on MOS: http://www.oracle.com/support/collateral/customersupport- security-practices.pdf http://download.oracle.com/docs/html/E12881_01/toc.htm The Oracle Support Hub can by found on the OCM Companion Distribution Disk at: http://www.oracle.com/technology/ documentation/ocm.html. Each node with an installed OCM collector can be automated to upload any changes on a daily basis or interval of your choice. OCM is now an optional part of any of the 10.2.0.4+ Oracle Product GUI installs. The OCM collector is also found by logging into MOS and selecting the collector tab. It is recommended to use at least the 3.2 version for ease of installation across the enterprise. Be aware! The collector install actually creates the Unix cron entry to automatically schedule the uploads. Mass deployment utility The OCM collector utility has been out for over a year, but a recent enhancement makes installation easier with a mass deployment utility. On the MOS collector tab, find Configuration Manager Repeater & Mass Deployment Tools and the OCM Companion Distribution Guide. The template file required to install the collector on multiple servers is in csv format, which you may find difficult to edit using vi or vim. The template doesn't have an initial entry and the length is wider than the average session window. Once the first entry is filed out (try using desktop spreadsheet software), editing this file with a command-line tool is easier. It has a secure password feature so that no password is stored in clear text. You can enter a password at the prompt or allow the password utility to encrypt the open text passwords in the template file during the install run. Running the utility runs very quickly from a single node that has SSH access to all entries in the template. It auto detects if OCM was already installed and bypasses any of those entries. You may encounter an issue where the required JAVA version is higher than what is installed. Other prerequisites include SSH on Linux or CYGWIN for Windows. A downside is that all configuration information is available to everyone with the same CSI number. In a small IT shop, this isn't a problem as long as MOS access is maintained properly when personnel changes. Providing granular group access within a CSI number to your uploaded configurations is a highly anticipated feature. Release management As a DBA you must be consistent in the different aspects of administration. This takes dedication to keep all of your installed Oracle products up-to-date on critical patches. Most DBAs keep up-to-date with production down issues that require a patch install. But what about the quarterly security fixes? The operating systems that your system admin is in charge of will probably be patched more regularly than Oracle. Why is that the case? It seems to take an inordinate amount of effort to accomplish what appears to be a small task. Newer versions of Oracle are associated with major enhancements—as shown by the differences between versions 11.1 and 11.2. Patch sets contain at least all the cumulative bug fixes for a particular version of Oracle and an occasional enhancement as shown in the version difference between 11.1.0.6 and 11.1.0.7. Oracle will stop supporting certain versions, indicating which is the most stable version (labeling it as the terminal release). For example, the terminal release of Oracle 10.1.x is 10.1.0.5, as that was the last patch set released. See the following document on MOS for further information on releases—Oracle Server (RDBMS) Releases Support Status Summary [Doc ID: 161818.1]. In addition to applying patch sets on a regular basis (usually an annual event) to keep current with bug fixes, there are other types of patches released on a regular basis. Consider these to be post-patch set patches. There is some confusing information from MOS, with two different methods of patching on a quarterly basis (Jan, April, July, Oct.)—Patch Set Updates and Critical Patch Updates. CPUs only contain security bug fixes. The newer method of patching—PSU—includes not only the security fixes but other major bugs. These are tested as a single unit and contain bug fixes that have been applied in customers' production environments. See the following for help in identifying a database version in relationship to PSUs: MOS Doc ID 850471.1 1st digit-Major release number 2nd digit-Maintenance release 3rd digit-Application server release 4th digit-Release component specific 5th digit-Platform specific release First PSU for Oracle Database Version-10.2.0.4.1 Second PSU for Oracle Database Version-10.2.0.4.2 While either PSUs or CPUs can be applied to a new or existing system, Oracle recommends that you stick to one type. If you have applied CPUs in the past and want to continue—that is one path. If you have applied CPUs in the past and now want to apply a PSU, you must now only apply PSUs from this point to prevent conflicts. Switching back and forth will cause problems and ongoing issues with further installs, and it requires significant effort to start down this path. You may need a merge patch when migrating from a current CPU environment, called a Merge Request on MOS. Important information on differences between CPUs and PSUs can be found in the following locations. If there is a document number, then that is found on the MOS support site: http://blogs.oracle.com/gridautomation/ http://www.oracle/technology/deploy/security/alerts. htm Doc 864316.1 Application of PSU can be automated through Deployment Procedures Doc 854428.1 Intro to Patch Set Updates Doc 756388.1 Recommended Patches Upgrade Companions 466181.1, 601807.1 Error Correction Policy 209768.1 Now to make things even more complicated for someone new to Oracle; let's discuss recommended patches. These are released between the quarterly PSUs and CPUs with common issues for targeted configurations . The following are targeted configurations: Generic—General database use Real Application Clusters and CRS—For running multiple instances on a single database with accompanying Oracle Clusterware software DataGuard (and/or Streams)—Oracle Redo Apply technology for moving data to a standby database or another read/write database Exadata—Vendor-specific HP hardware storage solution for Oracle Ebusiness Suite Certification—Oracle's version of Business Applications, which runs on an Oracle Database Recommended patches are tested as a single combined unit, reducing some of the risk involved with multiple patches. They are meant to stabilize production environments, hopefully saving time and cost with known issues starting with Oracle Database Release 10.2.0.3—see Doc ID: 756671.1.

Mock AJAX and your Facebook profile I'm sure that you've heard of AJAX (Asynchronous JavaScript and XML) with which you can build interactive web pages. Well, Facebook has Mock AJAX, and with this you can create interactive elements within a profile page. Mock AJAX has three attributes that you need to be aware of: clickwriteform: The form to be used to process any data. clickwriteid: The id of a component to be used to display our data. clickwriteurl: The URL of the application that will process the data. When using Mock AJAX, our application must do two things: Return the output of any processed data (and we can do that by using either echo or print). Define a form with which we'll enter any data, and a div to receive the processed data Using a form on your profile Since we want to make our application more interactive, one simple way is to add a form. So, for our first example we can add a function (or in this case a set of functions) to appinclude.php that will create a form containing a simple combo-box: function country_combo () {/*You use this function to display a combo-box containing a list of countries. It's in its own function so that we can use it in other forms without having to add any extra code*/$country_combo = <<<EndOfText<select name=sel_country><option>England</option><option>India</option></select>EndOfText;return $country_combo;}function country_form () {/*Like country_combo-box we can use this form where ever needed because we've encapsulated it in its own function */global $appcallbackurl;$country_form = "<form>";$country_form .= country_combo ();$country_form .= <<<EndOfText<input type="submit" clickrewriteurl="$appcallbackurl" clickrewriteid="info_display" value="View Country"/><div id="info_display" style="border-style: solid; border-color: black; border-width: 1px; padding: 5px;">No country selected</div></form>EndOfText;return $country_form;}function display_simple_form () {/*This function displays the country form with a nice subtitle (on the Profile page)*/global $facebook, $_REQUEST;#Return any processed dataif (isset($_REQUEST['sel_country'])) { echo $_REQUEST['sel_country'] . " selected"; exit;}#Define the form and the div$fbml_text = <<<EndOfText<fb:subtitle><fb:name useyou=false uid=$user firstnameonly=true possessive=true> </fb:name> Suspect List</fb:subtitle>EndOfText;$fbml_text .= country_form ();$facebook->api_client->profile_setFBML($fbml_text, $user);echo $fbml_text;} And, of course, you'll need to edit index.php: display_simple_form (); You'll notice from the code that we need to create a div with the id info_display, and that this is what we use for the clickrewriteid of the submit button. You'll also notice that we're using $appcallbackurl for the clickrewriteurl ($appcallbackurl is defined in appinclude.php). Now, it's just a matter of viewing the new FMBL (by clicking on the application URL in the left-navigation panel): If you select a country, and then click on View Country, you'll see: I'm sure that you can see where we're going with this. The next stage is to incorporate this form into our Suspect Tracker application. And the great thing now is that because of the functions that we've already added to appinclude.php, this is now a very easy job: function first_suspect_tracker () {global $facebook, $_REQUEST;if (isset($_REQUEST['sel_country'])) { $friend_details = get_friends_details_ by_country ($_REQUEST['sel_country']); foreach ($friend_details as $friend) { $div_text .= "<fb:name uid=" . $friend['uid'] . " firstnameonly=false></fb:name>, "; } echo $div_text; exit;}$fbml_text .= country_form ();$facebook->api_client->profile_setFBML($fbml_text, $user);$facebook->redirect($facebook->get_facebook_url() . '/profile.php');} You may also want to change the country_form function, so that the submit button reads View Suspects. And, of course, we'll also need to update index.php. Just to call our new function: <?phprequire_once 'appinclude.php';first_suspect_tracker ();?> This time, we'll see the list of friends in the selected country: or: OK, I know what you're thinking, this is fine if all of your friends are in England and India, but what if they're not? And you don't want to enter the list of countries manually, do you? And what happens if someone from a country not in the list becomes your friend? Obviously, the answer to all of these questions is to create the combo-box dynamically. Creating a dynamic combo-box I'm sure that from what we've done so far, you can work out how to extract a list of countries from Facebook: function country_list_sql () {/*We're going to be using this piece of SQL quite often so it deserves its own function*/global $user;$country_list_sql = <<<EndSQLSELECT hometown_location.countryFROM userWHERE uid IN (SELECT uid1FROM friendWHERE uid2=$user)EndSQL;return $country_list_sql;}function full_country_list () {/*With the SQL in a separate function this one is very short and simple*/global $facebook;$sql = country_list_sql ();$full_country_list = $facebook-> api_client->fql_query($sql);print_r ($full_country_list);} However, from the output, you can see that there's a problem with the data: If you look through the contents of the array, you'll notice that some of the countries are listed more than once—you can see this even more clearly if we simulate building the combo-box: function options_country_list () {global $facebook;$sql = country_list_sql ();$country_list = $facebook->api_client->fql_query($sql);foreach ($country_list as $country){ echo "option:" . $country['hometown_location']['country'] ."<br>";}} From which, we'd get the output: This is obviously not what we want in the combo-box. Fortunately, we can solve the problem by making use of the array_unique method, and we can also order the list by using the sort function: function filtered_country_list () {global $facebook;$sql = country_list_sql ();$country_list = $facebook->api_client->fql_query($sql);$combo_full = array();foreach ($country_list as $country){ array_push($combo_full, $country['hometown_location']['country']);}$combo_list = array_unique($combo_full);sort($combo_list);foreach ($combo_list as $combo){ echo "option:" . $combo ."<br>";}} And now, we can produce a usable combo-box: Once we've added our code to include the dynamic combo-box, we've got the workings for a complete application, and all we have to do is update the country_combo function: function country_combo () {/*The function now produces a combo-box derived from the friends' countries */global $facebook;$country_combo = "<select name=sel_country>";$sql = country_list_sql ();$country_list = $facebook->api_client->fql_query($sql);$combo_full = array();foreach ($country_list as $country){ array_push($combo_full, $country['hometown_location']['country']);}$combo_list = array_unique($combo_full);sort($combo_list);foreach ($combo_list as $combo){ $country_combo .= "<option>" . $combo ."</option>";}$country_combo .= "</select>";return $country_combo;} Of course, you'll need to reload the application via the left-hand navigation panel for the result: Limiting access to the form You may have spotted a little fly in the ointment at this point. Anyone who can view your profile will also be able to access your form and you may not want that (if they want a form of their own they should install the application!). However, FBML has a number of if (then) else statements, and one of them is <fb:if-is-own-profile>: <?phprequire_once 'appinclude.php';$fbml_text = <<<EndOfText<fb:if-is-own-profile>Hi <fb:name useyou=false uid=$user firstnameonly=true></fb:name>, welcome to your Facebook Profile page.<fb:else>Sorry, but this is not your Facebook Profile page - it belongs to <fb:name useyou=false uid=$user firstnameonly=false> </fb:name>,</fb:else></fb:if-is-own-profile>EndOfText;$facebook->api_client->profile_setFBML($fbml_text, $user);echo "Profile updated";?> So, in this example, if you were logged on to Facebook, you'd see the following on your profile page: But anyone else viewing your profile page would see: And remember that the FBML is cached when you run: $facebook->api_client->profile_setFBML($fbml_text, $user); Also, don't forget, it is not dynamic that is it's not run every time that you view your profile page. You couldn't, for example, produce the following for a user called Fred Bloggs: Sorry Fred, but this is not Your Facebook Profile page - it belongs to Mark Bain That said, you are now able to alter what's seen on the screen, according to who is logged on. Storing data—keeping files on your server From what we've looked at so far, you already know that you not only have, but need, files stored on your server (the API libraries and your application files). However, there are other instances when it is useful to store files there. Storing FBML on your server In all of the examples that we've worked on so far, you've seen how to use FBML mixed into your code. However, you may be wondering if it's possible to separate the two. After all, much of the FBML is static—the only reason that we include it in the code is so that we can produce an output. As well as there may be times when you want to change the FBML, but you don't want to have to change your code every time you do that (working on the principle that the more times you edit the code the more opportunity there is to mess it up). And, of course, there is a simple solution. Let's look at a typical form: <form><div id="info_display" style="border-style: solid; border-color: black; border-width: 1px; padding: 5px;"></div><input name=input_text><input type="submit" clickrewriteurl="http://213.123.183.16/f8/penguin_pi/" clickrewriteid="info_display" value="Write Result"></form> Rather than enclosing this in $fbml_text = <<<EndOfText ... EndOfText; as we have done before, you can save the FBML into a file on your server, in a subdirectory of your application. For example /www/htdocs/f8/penguin_pi/fbml/form_input_text.fbml. "Aha" I hear your say, "won't this invalidate the caching of FBML, and cause Facebook to access my server more often than it needs?" Well, no, it won't. It's just that we need to tell Facebook to update the cache from our FBML file. So, first we need to inform FBML that some external text needs to be included, by making use of the <fb:ref> tag, and then we need to tell Facebook to update the cache by using the fbml_refreshRefUrl method: function form_from_server () {global $facebook, $_REQUEST, $appcallbackurl, $user;$fbml_file = $appcallbackurl . "fbml/form_input_text.fbml";if (isset($_REQUEST['input_text'])) { echo $_REQUEST['input_text']; exit;}$fbml_text .= "<fb:ref url='" . $fbml_file . "' />";$facebook->api_client->profile_setFBML($fbml_text, $user);$facebook->api_client->fbml_refreshRefUrl($fbml_file);echo $fbml_text;} As far as your users are concerned, there is no difference. They'll just see another form on their profile page: Even if your users don't appreciate this leap forward, it will make a big difference to your coding—you're now able to isolate any static FBML from your PHP (if you want). And now, we can turn our attention to one of the key advantages of having your own server—your data. Storing data on your server So far, we've concentrated on how to extract data from Facebook and display it on the profile page. You've seen, for example, how to list all of your friends from a given country. However, that's not how Pygoscelis' list would work in reality. In reality, you should be able to select one of your friends and add them to your suspect list. We will, therefore, spend just a little time on looking at creating and using our own data. We're going to be saving our data in files, and so your first job must be to create a directory in which to save those files. Your new directory needs to be a subdirectory of the one containing your application. So, for example, on my Linux server I would do: cd /www/htdocs/f8/penguin_pi       #Move to the application directory mkdir data #Create a new directory chgrp www-data data                          #Change the group of the directory chmod g+w data                                  #Ensure that the group can write to data

Working with Chat Portlet For the purpose of this article, we will use an intranet website called book.com  which is created  for a fictions company named "Palm Tree Publications". In order to let employees enjoy chatting and instant messaging with others, we should use the Liferay Chat portlet. Let's experience how to enjoy chatting and instant messaging first. As an administrator of "Palm Tree Publications", you need to create a Page called "Instant Messaging" under the Page, "Community", at the Book Lovers Community and also add the Chat portlet in the Page, "Instant Messaging". Adding a Participant First of all, log in as "Palm Tree" and do the following: Add a Page called "Instant Messaging" under the Page, "Community" at the Book Lovers Community Public Pages, if the Page is not already present. Add the Chat portlet in the Page, "Instant Messaging" of the Book Lovers Community where you want to set up chatting and an instant messaging environment, if Chat portlet is not already present. After adding the Chat portlet, you can view it as shown in the following figure. Then, we need to add a participant in the Chat portlet. As an editor at the Editorial department, "Lotti Stein" wants to ping the manager, "David Berger", online and further share some comments about Liferay books. Let's do it as follows: Login as "Lotti Stein" first. Go to the Page, "Instant Messaging", " under the Page, "Community", at the Book Lovers Community Public Pages. Click on the Add icon. Input a participant's email address, such as "david@book.com". Press the Enter key. You will see the participant's full name appear, such as "David Berger". After adding more participants, such as "John Stuckia" and "Rolf Hess", you can view all participants as shown in the following figure. Managing Participants All the Users you've invited will appear as a list of participants. If the User "David Berger" is online ,then the icon to the left of the User name becomes light blue. Otherwise, it remains light gray; for example, User "John Stuckia". As shown in the following figure, only two Users ("David Berger" and "Lotti Stein") are online in the server. For details about OpenFire, refer to the forthcoming section. The participants are removable. For example, "Lotti Stein" wants to remove a participant "Rolf Hess", from the list of participants. Let's do it as follows: Locate the participant, such as "Rolf Hess". Click on the icon to the left of the User name, such as "Rolf Hess". You will see that the participant "Rolf Hess" is highlighted. Click the Remove icon. This participant will be removed from the list of participants. In short, to remove a User from the list of participants, simply locate the User you want to remove by clicking on the icon to the left of the User name. Then, click the Remove icon. The selected User name will be removed from the list of participants. Starting Chatting Irrespective of whether the participants are online or not, you can begin to Chat with them. For example, as an editor of editorial department, "Lotti Stein" wants to start chatting with the manager, "David Berger". Let's do it as follows: Locate the participant, "David Berger". Click the User name, "David Berger". A Chat box will appear. Input the message, "David, how are you?" Press the Enter key. Your messages will appear starting with the keyword, Me, in the message box (as shown in the following figure). As a manager of the editorial department, "David Berger" will have to do the following, to receive the messages from "Lotti Stein": Login as "David Berger" in new browser. Go to the Page, "Instant Messaging" under the Page, "Community" at the Book Lovers Community Public Pages. Locate the participant, "Lotti Stein". Click the User name, "Lotti Stein". A chat box will appear with the messages from "Lotti Stein". Input the message, "I am fine, and you?" Press the Enter key. Your messages will appear starting with the keyword "Me:" in the message box and the messages sent by the other User, "Lotti Stein" here, will appear starting with the User's name as shown in the following figure: Generally, to start chat, locate the User you want to chat with, by the User name first. Click on the User name link. A Chat box will appear. You can chat with many Users at the same time. To do this, just click on the Users' name link. Each Chat box is only for one unique User. The Chat box contains the User's name on the upper left. You can close the current Chat box by clicking on the mark X to the upper right. Note that the Chat box is hidden in your current Page initiatively. Whenever a new message comes from the User, you are chatting with, the Chat box will pop up with the new message and possible previous messages. To send messages, simply input your messages in the message input box first, and then press the Enter key. Your messages will appear starting with the keyword, Me, in the message box, and the messages sent by the other Users will appear starting with their User names.

 In this article by Maxwell Dayvson Da Silva and Hugo Lopes Tavares, the authors of Redis Essentials, we will get an overview of many different Redis commands and features, from techniques to reduce network latency to extending Redis with Lua scripting. At the end of this article, we will explain optimizations further. (For more resources related to this topic, see here.) Pub/Sub Pub/Sub stands for Publish-Subscribe, which is a pattern where messages are not sent directly to specific receivers. Publishers send messages to channels, and subscribers receive these messages if they are listening to a given channel. Redis supports the Pub/Sub pattern and provides commands to publish messages and subscribe to channels. Here are some examples of Pub/Sub applications: News and weather dashboards Chat applications Push notifications, such as subway delay alerts Remote code execution, similar to what the SaltStack tool supports The following examples implement a remote command execution system, where a command is sent to a channel and the server that is subscribed to that channel executes the command. The command PUBLISH sends a message to the Redis channel, and it returns the number of clients that received that message. A message gets lost if there are no clients subscribed to the channel when it comes in. Create a file called publisher.js and save the following code into it: var redis = require("redis"); var client = redis.createClient(); var channel = process.argv[2]; // 1 var command = process.argv[3]; // 2 client.publish(channel, command); // 3 client.quit(); Assign the third argument from the command line to the variable channel (the first argument is node and the second is publisher.js). Assign the fourth argument from the command line to the variable command. Execute the command PUBLISH, passing the variables channel and command. The command SUBSCRIBE subscribes a client to one or many channels. The command UNSUBSCRIBE unsubscribes a client from one or many channels. The commands PSUBSCRIBE and PUNSUBSCRIBE work the same way as the SUBSCRIBE and UNSUBSCRIBE commands, but they accept glob-style patterns as channel names. Once a Redis client executes the command SUBSCRIBE or PSUBSCRIBE, it enters the subscribe mode and stops accepting commands, except for the commands SUBSCRIBE, PSUBSCRIBE, UNSUBSCRIBE, and PUNSUBSCRIBE. Create a file called subscriber.js and save the following: var os = require("os"); // 1 var redis = require("redis"); var client = redis.createClient(); var COMMANDS = {}; // 2 COMMANDS.DATE = function() { // 3 var now = new Date(); console.log("DATE " + now.toISOString()); }; COMMANDS.PING = function() { // 4 console.log("PONG"); }; COMMANDS.HOSTNAME = function() { // 5 console.log("HOSTNAME " + os.hostname()); }; client.on("message", function(channel, commandName) { // 6 if (COMMANDS.hasOwnProperty(commandName)) { // 7 var commandFunction = COMMANDS[commandName]; // 8 commandFunction(); // 9 } else { // 10 console.log("Unknown command: " + commandName); } }); client.subscribe("global", process.argv[2]); // 11 Require the Node.js module os. Create the variable COMMANDS, which is a JavaScript object. All command functions in this module will be added to this object. This object is intended to act as a namespace. Create the function DATE, which displays the current date. Then create the function PING, which displays PONG. Create the function HOSTNAME, which displays the server hostname. Register a channel listener, which is a function that executes commands based on the channel message. Check whether the variable commandName is a valid command. Create the variable commandFunction and assign the function to it. Execute commandFunction. Display an error message if the variable commandName contains a command that is not available. Execute the command SUBSCRIBE, passing "global", which is the channel that all clients subscribe to, and a channel name from the command line. Open three terminal windows and run the previous files, as shown the following screenshot (from left to right and top to bottom): terminal-1: A subscriber that listens to the global channel and channel-1 terminal-2: A subscriber that listens to the global channel and channel-2 terminal-3: A publisher that publishes the message PING to the global channel (both subscribers receive the message), the message DATE to channel-1 (the first subscriber receives it), and the message HOSTNAME to channel-2 (the second subscriber receives it) The command PUBSUB introspects the state of the Redis Pub/Sub system. This command accepts three subcommands: CHANNELS, NUMSUB, and NUMPAT. The CHANNELS subcommand returns all active channels (channels with at least one subscriber). This command accepts an optional parameter, which is a glob-style pattern. If the pattern is specified, all channel names that match the pattern are returned; if no pattern is specified, all channel names are returned. The command syntax is as follows: PUBSUB CHANNELS [pattern] The NUMSUB subcommand returns the number of clients connected to channels via the SUBSCRIBE command. This command accepts many channel names as arguments. Its syntax is as follows: PUBSUB NUMSUB [channel-1 … channel-N] The NUMPAT subcommand returns the number of clients connected to channels via the PSUBSCRIBE command. This command does not accept channel patterns as arguments. Its syntax is as follows: PUBSUB NUMPAT Redis contributor Pieter Noordhuis created a web chat implementation in Ruby using Redis and Pub/Sub. It can be found at https://gist.github.com/pietern/348262. Transactions A transaction in Redis is a sequence of commands executed in order and atomically. The command MULTI marks the beginning of a transaction, and the command EXEC marks its end. Any commands between the MULTI and EXEC commands are serialized and executed as an atomic operation. Redis does not serve any other client in the middle of a transaction. All commands in a transaction are queued in the client and are only sent to the server when the EXEC command is executed. It is possible to prevent a transaction from being executed by using the DISCARD command instead of EXEC. Usually, Redis clients prevent a transaction from being sent to Redis if it contains command syntax errors. Unlike in traditional SQL databases, transactions in Redis are not rolled back if they produce failures. Redis executes the commands in order, and if any of them fail, it proceeds to the next command. Another downside of Redis transactions is that it is not possible to make any decisions inside the transaction, since all the commands are queued. For example, the following code simulates a bank transfer. Here, money is transferred from a source account to a destination account inside a Redis transaction. If the source account has enough funds, the transaction is executed. Otherwise, it is discarded. Save the following code in a file called bank-transaction.js: var redis = require("redis"); var client = redis.createClient(); function transfer(from, to, value, callback) { // 1 client.get(from, function(err, balance) { // 2 var multi = client.multi(); // 3 multi.decrby(from, value); // 4 multi.incrby(to, value); // 5 if (balance >= value) { // 6 multi.exec(function(err, reply) { // 7 callback(null, reply[0]); // 8 }); } else { multi.discard(); // 9 callback(new Error("Insufficient funds"), null); // 10 } }); } Create the function transfer, which receives an account ID from which to withdraw money, another account ID from which to receive money, the monetary value to transfer, and a callback function to call after the transfer. Retrieve the current balance of the source account. Create a Multi object, which represents the transaction. All commands sent to it are queued and executed after the EXEC command is issued. Enqueue the command DECRBY into the Multi object. Then enqueue the command INCRBY into the Multi object. Check whether the source account has sufficient funds. Execute the EXEC command, which triggers sequential execution of the queued transaction commands. Execute the callback function and pass the value null as an error, and the balance of the source account after the command DECRBY is executed. Execute the DISCARD command to discard the transaction. No commands from the transaction will be executed in Redis. Execute the function callback and pass an error object if the source account has insufficient funds. The following code uses the previous example, transferring $40 from Max's account to Hugo's account (both accounts had $100 before the transfer). Append the following to the file bank-transaction.js: client.mset("max:checkings", 100, "hugo:checkings", 100, function(err, reply) { // 1 console.log("Max checkings: 100"); console.log("Hugo checkings: 100"); transfer("max:checkings", "hugo:checkings", 40, function(err, balance) { // 2 if (err) { console.log(err); } else { console.log("Transferred 40 from Max to Hugo") console.log("Max balance:", balance); } client.quit(); }); }); Set the initial balance of each account to $100. Execute the function transfer to transfer $40 from max:checkings to hugo:checkings. Then execute the file using the following command: $ node bank-transaction.js Max checkings: 100 Hugo checkings: 100 Transferred 40 from Max to Hugo Max balance: 60 It is possible to make the execution of a transactionconditional using the WATCH command, which implements an optimistic lock on a group of keys. The WATCH command marks keys as being watched so that EXEC executes the transaction only if the keys being watched were not changed. Otherwise, it returns a null reply and the operation needs to be repeated; this is the reason it is called an optimistic lock. The command UNWATCH removes keys from the watch list. The following code implements a zpop function, which removes the first element of a Sorted Set and passes it to a callback function, using a transaction with WATCH. A race condition could exist if the WATCH command is not used. Create a file called watch-transaction.js with the following code: var redis = require("redis"); var client = redis.createClient(); function zpop(key, callback) { // 1 client.watch(key, function(watchErr, watchReply) { // 2 client.zrange(key, 0, 0, function(zrangeErr, zrangeReply) { // 3 var multi = client.multi(); // 4 multi.zrem(key, zrangeReply); // 5 multi.exec(function(transactionErr, transactionReply) { // 6 if (transactionReply) { callback(zrangeReply[0]); // 7 } else { zpop(key, callback); // 8 } }); }); }); } client.zadd("presidents", 1732, "George Washington"); client.zadd("presidents", 1809, "Abraham Lincoln"); client.zadd("presidents", 1858, "Theodore Roosevelt"); zpop("presidents", function(member) { console.log("The first president in the group is:", member); client.quit(); }); Create the function zpop, which receives a key and a callback function as arguments. Execute the WATCH command on the key passed as an argument. Then execute the ZRANGE command to retrieve the first element of the Sorted Set. Create a multi object. Enqueue the ZREM command in the transaction. Execute the transaction. Execute the callback function if the key being watched has not been changed. Execute the function zpop with the same parameters as before if the key being watched has not been changed. Then execute the file using the following command: $ node watch-transaction.js The first president in the group is: George Washington Pipelines In Redis, a pipeline is a way to send multiple commands together to the Redis server without waiting for individual replies. The replies are read all at once by the client. The time taken for a Redis client to send a command and obtain a reply from the Redis server is called Round Trip Time (RTT). When multiple commands are sent, there are multiple RTTs. Pipelines can decrease the number of RTTs because commands are grouped, so a pipeline with 10 commands will have only one RTT. This can improve the network's performance significantly. For instance, if the network link between a client and server has an RTT of 100 ms, the maximum number of commands that can be sent per second is 10, no matter how many commands can be handled by the Redis server. Usually, a Redis server can handle hundreds of thousands of commands per second, and not using pipelines may be a waste of resources. When Redis is used without pipelines, each command needs to wait for a reply. Assume the following: var redis = require("redis"); var client = redis.createClient(); client.set("key1", "value1"); client.set("key2", "value2"); client.set("key3", "value3"); Three separate commands are sent to Redis, and each command waits for its reply. The following diagram shows what happens when Redis is used without pipelines: Redis commands sent in a pipeline must be independent. They run sequentially in the server (the order is preserved), but they do not run as a transaction. Even though pipelines are neither transactional nor atomic (this means that different Redis commands may occur between the ones in the pipeline), they are still useful because they can save a lot of network time, preventing the network from becoming a bottleneck as it often does with heavy load applications. By default, node_redis, the Node.js library we are using, sends commands in pipelines and automatically chooses how many commands will go into each pipeline. Therefore, you don't need to worry about this. However, other Redis clients may not use pipelines by default; you will need to check out the client documentation to see how to take advantage of pipelines. The PHP, Python, and Ruby clients do not use pipelines by default. This is what happens when commands are sent to Redis in a pipeline: When sending many commands, it might be a good idea to use multiple pipelines rather than one big pipeline. Pipelines are not a new idea or an exclusive feature or command in Redis; they are just a technique of sending a group of commands to a server at once. Commands inside a transaction may not be sent as a pipeline by default. This will depend on the Redis client you are using. For example, node_redis sends everything automatically in pipelines (as we mentioned before), but different clients may require additional configuration. It is a good idea to send transactions in a pipeline to avoid an extra round trip. Scripting Redis 2.6 introduced the scripting feature, and the language that was chosen to extend Redis was Lua. Before Redis 2.6, there was only one way to extend Redis—changing its source code, which was written in C. Lua was chosen because it is very small and simple, and its C API is very easy to integrate with other libraries. Although it is lightweight, Lua is a very powerful language (it is commonly used in game development). Lua scripts are atomically executed, which means that the Redis server is blocked during script execution. Because of this, Redis has a default timeout of 5 seconds to run any script, although this value can be changed through the configuration lua-time-limit. Redis will not automatically terminate a Lua script when it times out. Instead, it will start to reply with a BUSY message to every command, stating that a script is running. The only way to make the server return to normalcy is by aborting the script execution with the command SCRIPT KILL or SHUTDOWN NOSAVE. Ideally, scripts should be simple, have a single responsibility, and run fast. The popular games Civilization V, Angry Birds, and World of Warcraft use Lua as their scripting language. Lua syntax basics Lua is built around basic types such as booleans, numbers, strings, tables (the only composite data type), and functions. Let's see some basics of Lua's syntax: Comments: -- this is a comment Global variable declaration: x = 123 Local variable declaration: local y = 456 Function definition: function hello_world() return "Hello World" end Iteration: for i = 1, 10 do print(i) end Conditionals: if x == 123 then print("x is the magic number") else print("I have no idea what x is") end String concatenation: print("Hello" .. " World") Using a table as an array — arrays in Lua start indexing at 1, not at 0 (as in most languages): data_types = {1.0, 123, "redis", true, false, hello_world} print(data_types[3]) -- the output is "redis" Using a table as a hash: languages = {lua = 1993, javascript = 1995, python = 1991, ruby = 1995} print("Lua was created in " .. languages["lua"]) print("JavaScript was created in " .. languages.javascript) Redis meets Lua A Redis client must send Lua scripts as strings to the Redis server. Therefore, this section will have JavaScript strings that contain Lua code. Redis can evaluate any valid Lua code, and a few libraries are available (for example, bitop, cjson, math, and string). There are also two functions that execute Redis commands: redis.call and redis.pcall. The function redis.call requires the command name and all its parameters, and it returns the result of the executed command. If there are errors, redis.call aborts the script. The function redis.pcall is similar to redis.call, but in the event of an error, it returns the error as a Lua table and continues the script execution. Every script can return a value through the keyword return, and if there is no explicit return, the value nil is returned. It is possible to pass Redis key names and parameters to a Lua script, and they will be available inside the Lua script through the variables KEYS and ARGV, respectively. Both redis.call and redis.pcall automatically convert the result of a Redis command to a Lua type, which means that if the Redis command returns an integer, it will be converted into a Lua number. The same thing happens to commands that return a string or an array. Since every script will return a value, this value will be converted from a Lua type to a Redis type. There are two commands for running Lua scripts: EVAL and EVALSHA. The next example will use EVAL, and its syntax is the following: EVAL script numkeys key [key ...] arg [arg ...] The parameters are as follows: script: The Lua script itself, as a string numkeys: The number of Redis keys being passed as parameters to the script key: The key name that will be available through the variable KEYS inside the script arg: An additional argument that will be available through the variable ARGV inside the script The following code uses Lua to run the command GET and retrieve a key value. Create a file called intro-lua.js with the following code: var redis = require("redis"); var client = redis.createClient(); client.set("mykey", "myvalue"); // 1 var luaScript = 'return redis.call("GET", KEYS[1])'; // 2 client.eval(luaScript, 1, "mykey", function(err, reply) { // 3 console.log(reply); // 4 client.quit(); }); Execute the command SET to create a key called mykey. Create the variable luaScript and assign the Lua code to it. This Lua code uses the redis.call function to execute the Redis command GET, passing a parameter. The KEYS variable is an array with all key names passed to the script. Execute the command EVAL to execute a Lua script. Display the return of the Lua script execution. Then execute it: $ node intro-lua.js myvalue Avoid using hardcoded key names inside a Lua script; pass all key names as parameters to the commands EVAL/EVALSHA. Previously in this article, in the Transactions section, we presented an implementation of a zpop function using WATCH/MULTI/EXEC. That implementation was based on an optimistic lock, which meant that the entire operation had to be retried if a client changed the Sorted Set before the MULTI/EXEC was executed. The same zpop function can be implemented as a Lua script, and it will be simpler and atomic, which means that retries will not be necessary. Redis will always guarantee that there are no parallel changes to the Sorted Set during script execution. Create a file called zpop-lua.js and save the following code into it: var redis = require("redis"); var client = redis.createClient(); client.zadd("presidents", 1732, "George Washington"); client.zadd("presidents", 1809, "Abraham Lincoln"); client.zadd("presidents", 1858, "Theodore Roosevelt"); var luaScript = [ 'local elements = redis.call("ZRANGE", KEYS[1], 0, 0)', 'redis.call("ZREM", KEYS[1], elements[1])', 'return elements[1]' ].join('n'); // 1 client.eval(luaScript, 1, "presidents", function(err, reply){ // 2 console.log("The first president in the group is:", reply); client.quit(); }); Create the variable luaScript and assign the Lua code to it. This Lua code uses the redis.call function to execute the Redis command ZRANGE to retrieve an array with only the first element in the Sorted Set. Then, it executes the command ZREM to remove the first element of the Sorted Set, before returning the removed element. Execute the command EVAL to execute a Lua script. Then, execute the file using the following command: $ node zpop-lua.js The first president in the group is: George Washington Many Redis users have replaced their transactional code in the form of WATCH/MULTI/EXEC with Lua scripts. It is possible to save network bandwidth usage by using the commands SCRIPT LOAD and EVALSHA instead of EVAL when executing the same script multiple times. The command SCRIPT LOAD caches a Lua script and returns an identifier (which is the SHA1 hash of the script). The command EVALSHA executes a Lua script based on an identifier returned by SCRIPT LOAD. With EVALSHA, only a small identifier is transferred over the network, rather than a Lua code snippet: var redis = require("redis"); var client = redis.createClient(); var luaScript = 'return "Lua script using EVALSHA"'; client.script("load", luaScript, function(err, reply) { var scriptId = reply; client.evalsha(scriptId, 0, function(err, reply) { console.log(reply); client.quit(); }) }); Then execute the script: $ node zpop-lua-evalsha.js Lua script using EVALSHA In order to make scripts play nicely with Redis replication, you should write scripts that do not change Redis keys in non-deterministic ways (that is, do not use random values). Well-written scripts behave the same way when they are re-executed with the same data. Miscellaneous commands This section covers the most important Redis commands that we have not previously explained. These commands are very helpful in a variety of situations, including obtaining a list of clients connected to the server, monitoring the health of a Redis server, expiring keys, and migrating keys to a remote server. All the examples in this section use redis-cli. INFO The INFO command returns all Redis server statistics, including information about the Redis version, operating system, connected clients, memory usage, persistence, replication, and keyspace. By default, the INFO command shows all available sections: memory, persistence, CPU, command, cluster, clients, and replication. You can also restrict the output by specifying the section name as a parameter: 127.0.0.1:6379> INFO memory # Memory used_memory:354923856 used_memory_human:338.48M used_memory_rss:468979712 used_memory_peak:423014496 used_memory_peak_human:403.42M used_memory_lua:33792 mem_fragmentation_ratio:1.32 mem_allocator:libc 127.0.0.1:6379> INFO cpu # CPU used_cpu_sys:3.71 used_cpu_user:40.36 used_cpu_sys_children:0.00 used_cpu_user_children:0.00 DBSIZE The DBSIZE command returns the number of existing keys in a Redis server: 127.0.0.1:6379> DBSIZE (integer) 50 DEBUG SEGFAULT The DEBUG SEGFAULT command crashes the Redis server process by performing an invalid memory access. It can be quite interesting to simulate bugs during the development of your application: 127.0.0.1:6379> DEBUG SEGFAULT MONITOR The command MONITOR shows all the commands processed by the Redis server in real time. It can be helpful for seeing how busy a Redis server is: 127.0.0.1:6379> MONITOR The following screenshot shows the MONITOR command output (left side) after running the leaderboard.js example (right side): While the MONITOR command is very helpful for debugging, it has a cost. In the Redis documentation page for MONITOR, an unscientific benchmark test says that MONITOR could reduce Redis's throughput by over 50%. CLIENT LIST and CLIENT SET NAME The CLIENT LIST command returns a list of all clients connected to the server, as well as relevant information and statistics about the clients (for example, IP address, name, and idle time). The CLIENT SETNAME command changes a client name; it is only useful for debugging purposes. CLIENT KILL The CLIENT KILL command terminates a client connection. It is possible to terminate client connections by IP, port, ID, or type: 127.0.0.1:6379> CLIENT KILL ADDR 127.0.0.1:51167 (integer) 1 127.0.0.1:6379> CLIENT KILL ID 22 (integer) 1 127.0.0.1:6379> CLIENT KILL TYPE slave (integer) 0 FLUSHALL The FLUSHALL command deletes all keys from Redis—this cannot be undone: 127.0.0.1:6379> FLUSHALL OK RANDOMKEY The command RANDOMKEY returns a random existing key name. This may help you get an overview of the available keys in Redis. The alternative would be to run the KEYS command, but it analyzes all the existing keys in Redis. If the keyspace is large, it may block the Redis server entirely during its execution: 127.0.0.1:6379> RANDOMKEY "mykey" EXPIRE and EXPIREAT The command EXPIRE sets a timeout in seconds for a given key. The key will be deleted after the specified amount of seconds. A negative timeout will delete the key instantaneously (just like running the command DEL). The command EXPIREAT sets a timeout for a given key based on a Unix timestamp. A timestamp of the past will delete the key instantaneously. These commands return 1 if the key timeout is set successfully or 0 if the key does not exist: 127.0.0.1:6379> MSET key1 value1 key2 value2 OK 127.0.0.1:6379> EXPIRE key1 30 (integer) 1 127.0.0.1:6379> EXPIREAT key2 1435717600 (integer) 1 TTL and PTTL The TTL command returns the remaining time to live (in seconds) of a key that has an associated timeout. If the key does not have an associated TTL, it returns -1, and if the key does not exist, it returns -2. The PTTL command does the same thing, but the return value is in milliseconds rather than seconds: 127.0.0.1:6379> SET redis-essentials:authors "By Maxwell Dayvson da Silva, Hugo Lopes Tavares" EX 30 OK 127.0.0.1:6379> TTL redis-essentials:authors (integer) 18 127.0.0.1:6379> PTTL redis-essentials:authors (integer) 13547 The SET command has optional parameters, and these were not shown before. The complete command syntax is as follows:   SET key value [EX seconds|PX milliseconds] [NX|XX] The parameters are explained as follows: EX: Set an expiration time in seconds PX: Set an expiration time in milliseconds NX: Only set the key if it does not exist XX: Only set the key if it already exists PERSIST The PERSIST command removes the existing timeout of a given key. Such a key will never expire, unless a new timeout is set. It returns 1 if the timeout is removed or 0 if the key does not have an associated timeout: 127.0.0.1:6379> SET mykey value OK 127.0.0.1:6379> EXPIRE mykey 30 (integer) 1 127.0.0.1:6379> PERSIST mykey (integer) 1 127.0.0.1:6379> TTL mykey (integer) -1 SETEX The SETEX command sets a value to a given key and also sets an expiration atomically. It is a combination of the commands, SET and EXPIRE: 127.0.0.1:6379> SETEX mykey 30 value OK 127.0.0.1:6379> GET mykey "value" 127.0.0.1:6379> TTL mykey (integer) 29 DEL The DEL command removes one or many keys from Redis and returns the number of removed keys—this command cannot be undone: 127.0.0.1:6379> MSET key1 value1 key2 value2 OK 127.0.0.1:6379> DEL key1 key2 (integer) 2 EXISTS The EXISTS command returns 1 if a certain key exists and 0 if it does not: 127.0.0.1:6379> SET mykey myvalue OK 127.0.0.1:6379> EXISTS mykey (integer) 1 PING The PING command returns the string PONG. It is useful for testing a server/client connection and verifying that Redis is able to exchange data: 127.0.0.1:6379> PING PONG MIGRATE The MIGRATE command moves a given key to a destination Redis server. This is an atomic command, and during the key migration, both Redis servers are blocked. If the key already exists in the destination, this command fails (unless the REPLACE parameter is specified). The command syntax is as follows: MIGRATE host port key destination-db timeout [COPY] [REPLACE] There are two optional parameters for the command MIGRATE, which can be used separately or combined: COPY: Keep the key in the local Redis server and create a copy in the destination Redis server REPLACE: Replace the existing key in the destination server SELECT Redis has a concept of multiple databases, each of which is identified by a number from 0 to 15 (there are 16 databases by default). It is not recommended to use multiple databases with Redis. A better approach would be to use multiple redis-server processes rather than a single one, because multiple processes are able to use multiple CPU cores and give better insights into bottlenecks. The SELECT command changes the current database that the client is connected to. The default database is 0: 127.0.0.1:6379> SELECT 7 OK 127.0.0.1:6379[7]> AUTH The AUTH command is used to authorize a client to connect to Redis. If authorization is enabled on the Redis server, clients are allowed to run commands only after executing the AUTH command with the right authorization key: 127.0.0.1:6379> GET mykey (error) NOAUTH Authentication required. 127.0.0.1:6379> AUTH mysecret OK 127.0.0.1:6379> GET mykey "value" SCRIPT KILL The SCRIPT KILL command terminates the running Lua script if no write operations have been performed by the script. If the script has performed any write operations, the SCRIPT KILL command will not be able to terminate it; in that case, the SHUTDOWN NOSAVE command must be executed. There are three possible return values for this command: OK NOTBUSY No scripts in execution right now. UNKILLABLE Sorry the script already executed write commands against the dataset. You can either wait the script termination or kill the server in a hard way using the SHUTDOWN NOSAVE command. 127.0.0.1:6379> SCRIPT KILL OK SHUTDOWN The SHUTDOWN command stops all clients, causes data to persist if enabled, and shuts down the Redis server. This command accepts one of the following optional parameters: SAVE: Forces Redis to save all of the data to a file called dump.rdb, even if persistence is not enabled NOSAVE: Prevents Redis from persisting data to the disk, even if persistence is enabled 127.0.0.1:6379> SHUTDOWN SAVE not connected> 127.0.0.1:6379> SHUTDOWN NOSAVE not connected> OBJECT ENCODING The OBJECT ENCODING command returns the encoding used by a given key: 127.0.0.1:6379> HSET myhash field value (integer) 1 127.0.0.1:6379> OBJECT ENCODING myhash "ziplist" Data type optimizations In Redis, all data types can use different encodings to save memory or improve performance. For instance, a String that has only digits (for example, 12345) uses less memory than a string of letters (for example, abcde) because they use different encodings. Data types will use different encodings based on thresholds defined in the Redis server configuration. The redis-cli will be used in this section to inspect the encodings of each data type and to demonstrate how configurations can be tweaked to optimize for memory. When Redis is downloaded, it comes with a file called redis.conf. This file is well documented and has all the Redis configuration directives, although some of them are commented out. Usually, the default values in this file are sufficient for most applications. The Redis configurations can also be specified via the command-line option or the CONFIG command; the most common approach is to use a configuration file. For this section, we have decided to not use a Redis configuration file. The configurations are passed via the command line for simplicity. Start redis-server with low values for all configurations: $ redis-server --hash-max-ziplist-entries 3 --hash-max-ziplist-value 5 --list-max-ziplist-entries 3 --list-max-ziplist-value 5 --set-max-intset-entries 3 --zset-max-ziplist-entries 3 --zset-max-ziplist-value 5 The default redis.conf file is well documented, and we recommend that you read it and discover new directive configurations. String The following are the available encoding for Strings: int: This is used when the string is represented by a 64-bit signed integer embstr: This is used for strings with fewer than 40 bytes raw: This is used for strings with more than 40 bytes These encodings are not configurable. The following redis-cli examples show how the different encodings are chosen: 127.0.0.1:6379> SET str1 12345 OK 127.0.0.1:6379> OBJECT ENCODING str1 "int" 127.0.0.1:6379> SET str2 "An embstr is small" OK 127.0.0.1:6379> OBJECT ENCODING str2 "embstr" 127.0.0.1:6379> SET str3 "A raw encoded String is anything greater than 39 bytes" OK 127.0.0.1:6379> OBJECT ENCODING str3 "raw" List These are the available encodings for Lists: ziplist: This is used when the List size has fewer elements than the configuration list-max-ziplist-entries and each List element has fewer bytes than the configuration list-max-ziplist-value linkedlist: This is used when the previous limits are exceeded 127.0.0.1:6379> LPUSH list1 a b (integer) 2 127.0.0.1:6379> OBJECT ENCODING list1 "ziplist" 127.0.0.1:6379> LPUSH list2 a b c d (integer) 4 127.0.0.1:6379> OBJECT ENCODING list2 "linkedlist" 127.0.0.1:6379> LPUSH list3 "only one element" (integer) 1 127.0.0.1:6379> OBJECT ENCODING list3 "linkedlist" Set The following are the available encodings for Sets: intset: This is used when all elements of a Set are integers and the Set cardinality is smaller than the configuration set-max-intset-entries hashtable: This is used when any element of a Set is not an integer or the Set cardinality exceeds the configuration set-max-intset-entries 127.0.0.1:6379> SADD set1 1 2 (integer) 2 127.0.0.1:6379> OBJECT ENCODING set1 "intset" 127.0.0.1:6379> SADD set2 1 2 3 4 5 (integer) 5 127.0.0.1:6379> OBJECT ENCODING set2 "hashtable" 127.0.0.1:6379> SADD set3 a (integer) 1 127.0.0.1:6379> OBJECT ENCODING set3 "hashtable" Hash The following are the available encodings for Hashes: ziplist: Used when the number of fields in the Hash does not exceed the configuration hash-max-ziplist-entries and each field name and value of the Hash is less than the configuration hash-max-ziplist-value (in bytes). hashtable: Used when a Hash size or any of its values exceed the configurations hash-max-ziplist-entries and hash-max-ziplist-value, respectively: 127.0.0.1:6379> HMSET myhash1 a 1 b 2 OK 127.0.0.1:6379> OBJECT ENCODING myhash1 "ziplist" 127.0.0.1:6379> HMSET myhash2 a 1 b 2 c 3 d 4 e 5 f 6 OK 127.0.0.1:6379> OBJECT ENCODING myhash2 "hashtable" 127.0.0.1:6379> HMSET myhash3 a 1 b 2 c 3 d 4 e 5 f 6 OK 127.0.0.1:6379> OBJECT ENCODING myhash3 "hashtable" Sorted Set The following are the available encodings: ziplist: Used when a Sorted Set has fewer entries than the configuration set-max-ziplist-entries and each of its values are smaller than zset-max-ziplist-value (in bytes) skiplist and hashtable: These are used when the Sorted Set number of entries or size of any of its values exceed the configurations set-max-ziplist-entries and zset-max-ziplist-value 127.0.0.1:6379> ZADD zset1 1 a (integer) 1 127.0.0.1:6379> OBJECT ENCODING zset1 "ziplist" 127.0.0.1:6379> ZADD zset2 1 abcdefghij (integer) 1 127.0.0.1:6379> OBJECT ENCODING zset2 "skiplist" 127.0.0.1:6379> ZADD zset3 1 a 2 b 3 c 4 d (integer) 4 127.0.0.1:6379> OBJECT ENCODING zset3 "skiplist" Measuring memory usage Previously, redis-server was configured to use a ziplist for Hashes with a maximum of three elements, in which each element was smaller than 5 bytes. With that configuration, it was possible to check how much memory Redis would use to store 500 field-value pairs: The total used memory was approximately 68 kB (1,076,864 – 1,008,576 = 68,288 bytes). If redis-server was started with its default configuration of 512 elements and 64 bytes for hash-max-ziplist-entries and hash-max-ziplist-value, respectively, the same 500 field-value pairs would use less memory, as shown here: The total used memory is approximately 16 kB (1,025,104 – 1,008,624 = 16,480 bytes). The default configuration in this case was more than four times more memory-efficient. Forcing a Hash to be a ziplist has a trade-off—the more elements a Hash has, the slower the performance. A ziplist is a dually linked list designed to be memory-efficient, and lookups are performed in linear time (O(n), where n is the number of fields in a Hash). On the other hand, a hashtable's lookup runs in constant time (O(1)), no matter how many elements exist. If you have a large dataset and need to optimize for memory, tweak these configurations until you find a good trade-off between memory and performance. Instagram tweaked their Hash configurations and found that 1,000 elements per Hash was a good trade-off for them. You can learn more about the Instagram solution in the blog post at http://instagram-engineering.tumblr.com/post/12202313862/storing-hundreds-of-millions-of-simple-key-value. The same logic for tweaking configurations and trade-offs applies to all other data type encodings presented previously. Algorithms that run in linear time (O(n)) are not always bad. If the input size is very small, they can run in near-constant time. Summary This article introduced the concepts behind Pub/Sub, transactions, and pipelines. It also showed the basics of the Lua language syntax, along with explanations on how to extend Redis with Lua. A good variety of Redis commands was presented, such as commands that are used to monitor and debug a Redis server. This article also showed how to perform data type optimizations by tweaking the redis-server configuration. Resources for Article: Further resources on this subject: Transactions in Redis[article] Redis in Autosuggest[article] Using Redis in a hostile environment (Advanced) [article]

Now that SpamAssassin is installed, we need to configure the system to use it. SpamAssassin can be used in many ways. It can be integrated into the MTA for maximum performance; it can run as a daemon or a simple script to avoid complexity; it can use separate settings for each user or use a single set of settings for all users; and it can be used for all accounts or just for the chosen ones. In this article, we will discuss using SpamAssassin in three ways. The first method is with Procmail. This is the simplest method to configure and is suitable for low-volume sites, for example, less than 10,000 e-mails a day. The second method is to use SpamAssassin as a daemon. This is more efficient, and can still be used with Procmail, if desired. The third method is to integrate SpamAssassin with a content filter such as amavisd. This offers performance advantages, but occasionally the content filter does not work with the latest release of SpamAssassin. Problems, if any, are usually resolved quickly. To help you get the most out of SpamAssassin, Packt Publishing has published SpamAssassin: A practical guide to integration and configuration, (ISBN 1-904811-12-4) by Alistair McDonald. Using SpamAssassin with Procmail Before we configure the system to use SpamAssassin, let's consider what SpamAssassin does. SpamAssassin is not an e-mail filter. A filter is something that changes the destination of an e-mail. SpamAssassin adds e-mail headers to an e-mail message to indicate if it is spam or not. Consider an e-mail with headers like this: Return-Path: <user@domain.com>X-Original-To: jdoe@localhostDelivered-To: jdoe@host.domain.comReceived: from localhost (localhost [127.0.0.1])by domain.com (Postfix) with ESMTP id 52A2CF2948for <jdoe@localhost>; Thu, 11 Nov 2004 03:39:42 +0000 (GMT)Received: from pop.ntlworld.com [62.253.162.50]by localhost with POP3 (fetchmail-6.2.5)for jdoe@localhost (single-drop); Thu, 11 Nov 2004 03:39:42 +0000(GMT)Message-ID: <D8F7B41C.4DDAFE7@anotherdomain.com>Date: Wed, 10 Nov 2004 17:54:14 -0800From: "stephen mellors" <gregory@anotherdomain.com>User-Agent: MIME-tools 5.503 (Entity 5.501)X-Accept-Language: en-usMIME-Version: 1.0To: "Jane Doe" <jdoe@domain.com>Subject: nearest pharmacy onlineContent-Type: text/plain;charset="us-ascii"Content-Transfer-Encoding: 7bit SpamAssassin will add header lines. X-Spam-Flag: YESX-Spam-Checker-Version: SpamAssassin 3.1.0-r54722 (2004-10-13) onhost.domain.comX-Spam-Level: *****X-Spam-Status: Yes, score=5.8 required=5.0 tests=BAYES_05,HTML_00_10,HTML_MESSAGE,MPART_ALT_DIFF autolearn=noversion=3.1.0-r54722 SpamAssassin doesn't change the destination of the e-mail, all it does is add headers that enable something else to change the destination of the e-mail. The best indication that an e-mail is spam is the X-Spam-Flag. If this is YES, SpamAssassin considers the mail to be spam and it can be filtered by Procmail. SpamAssassin also assigns a score to each e-mail—the higher the score, the more likely that the e-mail is spam. The threshold that determines if an e-mail is spam can be configured on a system-wide or per-user basis. The default of 5.0 is a sensible default if you are using an unmodified installation of SpamAssassin without any custom rulesets. Global procmailrc file Let's suppose that we want to check all incoming e-mail for spam using SpamAssassin. Commands in the /etc/procmailrc file are run for all users, so executing SpamAssassin here is ideal. The following simple recipe will run SpamAssassin for all users when placed in /etc/procmailrc: :0fw| /usr/bin/spamassassin To place all spam in an individual spam folder, ensure that the global/etc/procmailrc file has a line specifying a default destination. For example: DEFAULT=$HOME/.maildir/ If not, then add a line specifying DEFAULT. To filter spam into a folder, add a recipe similar to the following: * ^X-Spam-Flag: Yes.SPAM/new This assumes that each user has a folder called SPAM already configured. To place all the spam in a single, central folder, use an absolute path to the destination in the recipe: * ^X-Spam-Flag: Yes/var/spool/poss_spam This will place all spam in a single folder, which can be reviewed by the system administrator. As times, regular e-mail may occasionally be wrongly detected as spam, the folder should not be world-readable, which leads to a more generalized statement. SpamAssassin will be run under the system account used by Postfix. This means that the Bayesian database and the auto-whitelists and blacklists will be shared by all users. From a security point of view, it's important that the various databases that SpamAssassin creates are not world-writable. SpamAssassin stores user-specific files in the ~/.spamassassin/ directory. Here is a list of fi les that may be present for a user: //===INSERT TABLE 01=== Some of them may contain confidential data—for example, regular contacts will appear in the auto-whitelist files. Careful use of permissions will ensure that the files are not readable by regular user accounts. Using SpamAssassin on a per-user basis Perhaps some users don't receive spam, or there may be issues with users sharing whitelists and Bayesian databases. SpamAssassin can be run on an individual basis by moving the recipes to the ~/.procmailrc of specific users. This should increase the filtering performance for each user, but increases disk space usage for each user and requires setting up each individual user account by modifying its ~/.procmailrc. A typical user's .procmailrc might look like this: MAILDIR=$HOME/.maildir:0fw| /usr/bin/spamassassin:0* ^X-Spam-Flag: Yes.SPAM/cur As suggested, e-mail may sometimes be wrongly detected as spam. It's worthwhile reviewing spam to ensure that legitimate e-mails have not been wrongly classified. If the user receives a lot of spam, then wading through it all is time consuming, tedious, and error prone. Procmail can filter spam by checking the spam score written in the e-mail headers by SpamAssassin. The low-scoring spam (for example, scoring up to 9) can be placed in one folder called Probable_Spam, while higher scoring e-mails (which are more likely to be spam) can be placed a folder called Certain_Spam. To do this, we use the X-Spam-Level header, which SpamAssassin creates. This is simply the number of asterisks, related to the X-Spam-Level value. By moving e-mail with more than a certain number of asterisks to the Certain_Spam folder, the remaining spam is "Probable Spam". E-mail that is marked with X-Spam-Flag: NO, is obviously not spam. The following .procmailrc file will filter high scoring spam separately from low scoring spam and non spam: MAILDIR=$HOME/.maildir:0fw| /usr/bin/spamassassin:0* ^X-Spam-Level: **************.Certain_Spam/cur:0* ^X-Spam-FLAG: YES.Probable_Spam/cur Using SpamAssassin as a daemon with Postfix A daemon is a background process; one that waits for work, processes it, and then waits for more work. Using this approach actually improves performance (as long as there is sufficient memory) because responsiveness is improved—the program is always ready and waiting and does not have to be loaded each time spam tagging is required. To use SpamAssassin as a daemon, a user account should be added—it's dangerous to run any service as root. As root, enter the following commands to make a user and a group called spam: # groupadd spam# useradd -m -d /home/spam -g spam -s /bin/false spam# chmod 0700 /home/spam To configure Postfix to run SpamAssassin, use SpamAssassin as a daemon. The Postfix master.cf file must be changed. Edit the file and locate the line that begins with 'smtp inet'. Amend the line to add -o content_filter=spamd to the end. smtp inet n - n - - smtpd -o content_filter=spamd Add the following lines to the end of the file: spamd unix - n n - - pipeflags=R user=spam argv=/usr/bin/spamc-e /usr/sbin/sendmail -oi -f ${sender} ${recipient} If the text is spread across several lines, any continuing line must begin with spaces as shown. The changes to the file define a filter called spamd that runs the spamc client for each message and also specifies that the filter should be run whenever an e-mail is received via SMTP. On this line, spamd is the name of the filter and matches the name used in the content_filter line. The user= portion specifies the user context that should be used to run the command. The argv= portion describes the program that should be run. The other flags are used by Procmail and their presence is important. Using SpamAssassin with amavisd-new amavisd-new is an interface between MTAs and content checkers. Despite its name, amavisd-new is a well-established open source package that is well maintained. Content checkers scan e-mail for viruses and/or spam. amavisd-new is slightly different. Just like like spamd, it is written in Perl and runs as a daemon, but instead of accessing SpamAssassin via the spamc or spamassassin clients, it loads SpamAssassin into memory and accesses the SpamAssassin functions directly. It is therefore closely coupled to SpamAssassin and may need to be upgraded at the same time as SpamAssassin. Unlike other Perl-based applications and utilities, amavisd-new is not available from CPAN. However, it is available in source form and RPM form for many distributions of Linux, and is also available for debian-based repositories. Details of versions available are listed on http://www.ijs.si/software/amavisd/#download. We recommend that if the version of SpamAssassin that your distributor offers is up-to-date, then you should use their package of both SpamAssassin and amavisd.

Providing wish lists Wish lists allow customers to maintain a list of products that they would like to purchase at some point, or that they would like others to purchase for them as a gift. Creating the structure To effectively maintain wish lists for customers, we need to keep a record of: The product the customer desires The quantity of the product If they are a logged-in customer, their user ID If they are not a logged-in customer, some way to identify their wish-list products for the duration of their visit to the site The date they added the products to their wish list The priority of the product in their wish lists; that is, if they really want the product, or if it is something they wouldn't mind having Let's translate that into a suitable database table that our framework can interact with: Field Type Description ID Integer (Primary Key, Auto Increment) A reference for the database Product Integer The product the user wishes to purchase Quantity Integer The number of them the user would like Date added Datetime The date they added the product to their wish list Priority Integer Relative to other products in their wish list, and how important is this one Session ID Varcharr The user's session id(so they don't need to be logged in) IP Address Varchar The user's IP address (so they don't need to be logged in) By combining the session ID and IP address of the customer, along with the timestamp of when they added the product to their wish list, we can maintain a record of their wish list for the duration of their visit. Of course, they would need to register, or log in, before leaving the site, for their wish list to be permanently saved. This also introduces an element of maintenance to this feature, as once a customer who has not logged in closes their session, their wish-list data cannot be retrieved, so we would need to implement some garbage collection functions to prune this table. The following SQL represents this table: CREATE TABLE `wish_list_products` (`ID` INT NOT NULL AUTO_INCREMENT PRIMARY KEY,`product` INT NOT NULL,`quantity` INT NOT NULL,`user` INT NOT NULL,`dateadded` TIMESTAMP NOT NULLDEFAULT CURRENT_TIMESTAMP,`priority` INT NOT NULL,`sessionID` VARCHAR( 50 ) NOT NULL,`IPAddress` VARCHAR( 50 ) NOT NULL,INDEX ( `product` )) ENGINE = INNODB COMMENT = 'Wish list products'ALTER TABLE `wish_list_products` ADD FOREIGN KEY ( `product` ) REFERENCES `book4`.`content` (`ID`)ON DELETE CASCADE ON UPDATE CASCADE; Saving wishes Now that we have a structure in place for storing wish-list products, we need to have a process available to save them into the database. This involves a link or button placed on the product view, and either some modifications to our product controller, or a wish-list controller, to save the wish. As wish lists will have their own controller and model for viewing and managing the lists, we may as well add the functionality into the wish-list controller. So we will need: a controller a link in our product view Wish-list controller The controller needs to detect if the user is logged in or not; if they are, then it should add products to the user's wish list; otherwise, it should be added to a session-based wish list, which lasts for the duration of the user's session. The controller also needs to detect if the product is valid; we can do this by linking it up with the products model, and if it isn't a valid product, the customer should be informed of this. Let's look through a potential addProduct() method for our wish-list controller. /** * Add a product to a user's wish list * @param String $productPath the product path * @return void */ We first check if the product is valid, by creating a new product model object, which informs us if the product is valid. private function addProduct( $productPath ){// check product path is a valid and active product$pathToRemove = 'wishlist/add/';$productPath = str_replace( $pathToRemove, '',$this->registry->getURLPath() );require_once( FRAMEWORK_PATH . 'models/products/model.php');$this->product = new Product( $this->registry, $productPath );if( $this->product->isValid(){// check if user is logged in or notif( $this->registry->getObject('authenticate')->loggedIn() == true ){//Assuming the user is logged in, we can also store their ID,// so the insert data is slightly different. Here we insert the// wish into the database.$wish = array();$pdata = $this->product->getData();$wish['product'] = $pdata['ID'];$wish['quantity'] = 1;$wish['user'] = $this->registry->getObject('authenticate')->getUserID();$this->registry->getObject('db')->insertRecords('wish_list_products', $wish );// inform the user$this->registry->getObject('template')->getPage()->addTag('message_heading', 'Product added to your wish list');$this->registry->getObject('template')->getPage()->addTag('message_heading', 'A ' . $pdata['name'].' has been added to your wish list');$this->registry->getObject('template')->buildFromTemplates('header.tpl.php', 'message.tpl.php','footer.tpl.php');} The customer isn't logged into the website, so we add the wish to the database, using session and IP address data to tie the wish to the customer. else{// insert the wish$wish = array();$wish['sessionID'] = session_id();$wish['user'] = 0;$wish['IPAddress'] = $_SERVER['REMOTE_ADDR'];$pdata = $this->product->getData();$wish['product'] = $pdata['ID'];$wish['quantity'] = 1;$this->registry->getObject('db')->insertRecords('wish_list_products', $wish );// inform the user$this->registry->getObject('template')->getPage()->addTag('message_heading','Product added to your wish list');$this->registry->getObject('template')->getPage()->addTag('message_heading', 'A ' . $pdata['name'].' has been added to your wish list');$this->registry->getObject('template')->buildFromTemplates('header.tpl.php', 'message.tpl.php','footer.tpl.php');}} The product wasn't valid, so we can't insert the wish, so we need to inform the customer of this. else{// we can't insert the wish, so inform the user$this->registry->getObject('template')->getPage()->addTag('message_heading', 'Invalid product');$this->registry->getObject('template')->getPage()->addTag('message_heading', 'Unfortunately, the product youtried to add to your wish list was invalid, and was notadded, please try again');$this->registry->getObject('template')->buildFromTemplates('header.tpl.php', 'message.tpl.php','footer.tpl.php');}} Add to wish list To actually add a product to our wish list, we need a simple link within our products view. This should be /wishlist/add/product-path. <p><a href="wishlist/add/{product_path}"title="Add {product_name} to your wishlist">Add to wishlist.</a></p> We could encase this link around a nice image if we wanted, making it more user friendly. When the user clicks on this link, the product will be added to their wish list and they will be informed of that.

  Magento 1.4 Themes Design Magento terminology Before you look at Magento themes, it's beneficial to know the difference between what Magento calls interfaces and what Magento calls themes, and the distinguishing factors of websites and stores. Magento websites and Magento stores To add to this, the terms websites and stores have a slightly different meaning in Magento than in general and in other systems. For example, if your business is called M2, you might have three Magento stores (managed through the same installation of Magento) called: Blue Store Red Store Yellow Store In this case, Magento refers to M2 as the website and the stores are Blue Store, Red Store, and Yellow Store. Each store then has one or more store views associated with it too. The simplest Magento website consists of a store and store view (usually of the same name): A slightly more complex Magento store may just have one store view for each store. This is a useful technique if you want to manage more than one store in the same Magento installation, with each store selling different products (for example, the Blue Store sells blue products and the Yellow Store sells yellow products). If a store were to make use of more than one Magento store view, it might be, to present customers with a bi-lingual website. For example, our Blue Store may have an English, French, and Japanese store view associated with it: Magento interfaces An interface consists of one or more Magento themes that comprise how your stores look and function for your customers. Interfaces can be assigned at two levels in Magento: At the website level At the store view level If you assign an interface at the website level of your Magento installation, all stores associated with the interface inherit the interface. For example, imagine your website is known as M2 in Magento and it contains three stores called: Blue Store Red Store Yellow Store If you assign an interface at the website level (that is, M2), then the subsequent stores, Blue Store, Red Store, and Yellow Store, inherit this interface: If you assigned the interface at the store view level of Magento, then each store view can retain a different interface: Magento packages A Magento package typically contains a base theme, which contains all of the templates, and other files that Magento needs to run successfully, and a custom theme. Let's take a typical example of a Magento store, M2. This may have two packages: the base package, located in the app/design/frontend/base/ directory and another package which itself consists of two themes: The base theme is in the app/design/frontend/base/ directory. The second package contains the custom theme's default theme in the app/design/frontend/ default/ directory, which acts as a base theme within the package. The custom theme itself, which is the non-default theme, is in the app/design/frontend/our-custom- theme/default/ and app/design/frontend/our-custom-theme/custom-theme/ directories. By default, Magento will look for a required file in the following order: Custom theme directory: app/design/frontend/our-custom-theme/ custom-theme/ Custom theme's default directory: app/design/frontend/our-custom-theme/ default/ Base directory: app/design/frontend/base/ Magento themes A Magento theme fits in to the Magento hierarchy in a number of positions: it can act as an interface or as a store view. There's more to discover about Magento themes yet, though there are two types of Magento theme: a base theme (this was called a default theme in Magento 1.3) and a non-default theme. Base theme A base theme provides all conceivable files that a Magento store requires to run without error, so that non-default themes built to customize a Magento store will not cause errors if a file does not exist within it. The base theme does not contain all of the CSS and images required to style your store, as you'll be doing this with our non-default theme. Don't change the base package! It is important that you do not edit any files in the base package and that you do not attempt to create a custom theme in the base package, as this will make upgrading Magento fully difficult. Make sure any custom themes you are working on are within their own design package; for example, your theme's files should be located at app/design/ frontend/your-package-name/default and skin/frontend/ your-package-name/default. Default themes A default theme in Magento 1.4 changes aspects of your store but does not need to include every file required by Magento as a base theme does, though it must just contain at least one file for at least one aspect of a theme (that is, locales, skins, templates, layout): Default themes in Magento 1.3 In Magento 1.3, the default theme acted the way the base theme did in Magento 1.4, providing every file that your Magento store required to operate. Non-default themes A non-default theme changes aspects of a Magento store but does not need to include every file required by Magento as the base theme does; it must just contain at least one file for at least one aspect of a theme (that is, locales, skins, templates, layout): In this way, non-default themes are similar to a default theme in Magento. Non-default themes can be used to alter your Magento store for different seasonal events such as Christmas, Easter, Eid, Passover, and other religious festivals, as well as events in your industry's corporate calendar such as annual exhibitions and conferences. Blocks in Magento Magento uses blocks to differentiate between the various components of its functionality, with the idea that this makes it easier for Magento developers and Magento theme designers to customize the functionality of Magento and the look and feel of Magento respectively. There are two types of blocks in Magento: Content blocks Structural blocks Content blocks A content block displays the generated XHTML provided by Magento for any given feature. Content blocks are used within Magento structural blocks. Examples of content blocks in Magento include the following: The search feature Product listings The mini cart Category listings Site navigation links Callouts (advertising blocks) The following diagram illustrates how a Magento store might have content blocks positioned within its structural blocks: Simply, content blocks are the what of a Magento theme: they define what type of content appears within any given page or view within Magento. Structural blocks In Magento, a structural block exists only to maintain a visual hierarchy to a page. Typical structural blocks in a Magento theme include: Header Primary area Left column Right column Footer

Clustering plays an important role in Enterprise applications as it lets you split the load of your application across several nodes, granting robustness to your applications. As we discussed earlier, for optimal results it's better to limit the size of your JVM to a maximum of 2-2.5GB, otherwise the dynamics of the garbage collector will decrease your application's performance. Combining relatively smaller Java heaps with a solid clustering configuration can lead to a better, scalable configuration plus significant hardware savings. The only drawback to scaling out your applications is an increased complexity in the programming model, which needs to be correctly understood by aspiring architects. JBoss AS comes out of the box with clustering support. There is no all-in-one library that deals with clustering but rather a set of libraries, which cover different kinds of aspects. The following picture shows how these libraries are arranged: The backbone of JBoss Clustering is the JGroups library, which provides the communication between members of the cluster. Built upon JGroups we meet two building blocks, the JBoss Cache framework and the HAPartition service. JBoss Cache handles the consistency of your application across the cluster by means of a replicated and transactional cache. On the other hand, HAPartition is an abstraction built on top of a JGroups Channel that provides support for making and receiving RPC invocations from one or more cluster members. For example HA-JNDI (High Availability JNDI) or HA Singleton (High Availability Singleton) both use HAPartition to share a single Channel and multiplex RPC invocations over it, eliminating the configuration complexity and runtime overhead of having each service create its own Channel. If you need more information about the HAPartition service you can consult the JBoss AS documentation https://developer.jboss.org/wiki/jBossAS5ClusteringGuide. In the next section we will learn more about the JGroups library and how to configure it to reach the best performance for clustering communication. Configuring JGroups transport Clustering requires communication between nodes to synchronize the state of running applications or to notify changes in the cluster definition. JGroups (http://jgroups.org/manual/html/index.html) is a reliable group communication toolkit written entirely in Java. It is based on IP multicast, but extends by providing reliability and group membership. Member processes of a group can be located on the same host, within the same Local Area Network (LAN), or across a Wide Area Network (WAN). A member can be in turn part of multiple groups. The following picture illustrates a detailed view of JGroups architecture: A JGroups process consists basically of three parts, namely the Channel, Building blocks, and the Protocol stack. The Channel is a simple socket-like interface used by application programmers to build reliable group communication applications. Building blocks are an abstraction interface layered on top of Channels, which can be used instead of Channels whenever a higher-level interface is required. Finally we have the Protocol stack, which implements the properties specified for a given channel. In theory, you could configure every service to bind to a different Channel. However this would require a complex thread infrastructure with too many thread context switches. For this reason, JBoss AS is configured by default to use a single Channel to multiplex all the traffic across the cluster. The Protocol stack contains a number of layers in a bi-directional list. All messages sent and received over the channel have to pass through all protocols. Every layer may modify, reorder, pass or drop a message, or add a header to a message. A fragmentation layer might break up a message into several smaller messages, adding a header with an ID to each fragment, and re-assemble the fragments on the receiver's side. The composition of the Protocol stack (that is, its layers) is determined by the creator of the channel: an XML file defines the layers to be used (and the parameters for each layer). Knowledge about the Protocol stack is not necessary when just using Channels in an application. However, when an application wishes to ignore the default properties for a Protocol stack, and configure their own stack, then knowledge about what the individual layers are supposed to do is needed. In JBoss AS, the configuration of the Protocol stack is located in the file, <server> deployclusterjgroups-channelfactory.sarMETA-INFjgroupschannelfactory- stacks.xml. The file is quite large to fit here, however, in a nutshell, it contains the following basic elements: The first part of the file includes the UDP transport configuration. UDP is the default protocol for JGroups and uses multicast (or, if not available, multiple unicast messages) to send and receive messages. A multicast UDP socket can send and receive datagrams from multiple clients. The interesting and useful feature of multicast is that a client can contact multiple servers with a single packet, without knowing the specific IP address of any of the hosts. Next to the UDP transport configuration, three protocol stacks are defined: udp: The default IP multicast based stack, with flow control udp-async: The protocol stack optimized for high-volume asynchronous RPCs udp-sync: The stack optimized for low-volume synchronous RPCs Thereafter, the TCP transport configuration is defined . TCP stacks are typically used when IP multicasting cannot be used in a network (for example, because it is disabled) or because you want to create a network over a WAN (that's conceivably possible but sharing data across remote geographical sites is a scary option from the performance point of view). You can opt for two TCP protocol stacks: tcp: Addresses the default TCP Protocol stack which is best suited to high-volume asynchronous calls. tcp-async: Addresses the TCP Protocol stack which can be used for low-volume synchronous calls. If you need to switch to TCP stack, you can simply include the following in your command line args that you pass to JBoss: -Djboss.default.jgroups.stack=tcp Since you are not using multicast in your TCP communication, this requires configuring the addresses/ports of all the possible nodes in the cluster. You can do this by using the property -Djgroups.tcpping. initial_hosts. For example: -Djgroups.tcpping.initial_hosts=host1[7600],host2[7600] Ultimately, the configuration file contains two stacks which can be used for optimising JBoss Messaging Control Channel (jbm-control) and Data Channel (jbm-data).

(For more resources on silverlight, see here.) Introduction Looking at the namespaces and classes in the Silverlight assemblies, it's easy to see that there are no ADO.NET-related classes available in Silverlight. Silverlight does not contain a DataReader, a DataSet, or any option to connect to a database directly. Thus, it's not possible to simply define a connection string for a database and let Silverlight applications connect with that database directly. The solution adds a layer on top of the database in the form of services. The services that talk directly to a database (or, more preferably, to a business and data access layer) can expose the data so that Silverlight can work with it. However, the data that is exposed in this way does not always have to come from a database. It can come from a third-party service, by reading a file, or be the result of an intensive calculation executed on the server. Silverlight has a wide range of options to connect with services. This is important as it's the main way of getting data into our applications. In this article, we'll look at the concepts of connecting with several types of services and external data. We'll start our journey by looking at how Silverlight connects and works with a regular service. We'll see the concepts that we use here recur for other types of service communications as well. One of these concepts is cross-domain service access. In other words, this means accessing a service on a domain that is different from the one where the Silverlight application is hosted. We'll see why Microsoft has implemented cross-domain restrictions in Silverlight and what we need to do to access externally hosted services. Next, we'll talk about working with the Windows Azure Platform. More specifically, we'll talk about how we can get our Silverlight application to get data from a SQL Azure database, how to communicate with a service in the cloud, and even how to host the Silverlight application in the cloud, using a hosted service or serving it from Azure Storage. Finally, we'll finish this chapter by looking at socket communication. This type of communication is rare and chances are that you'll never have to use it. However, if your application needs the fastest possible access to data, sockets may provide the answer. Connecting and reading from a standardized service Applies to Silverlight 3, 4 and 5 If we need data inside a Silverlight application, chances are that this data resides in a database or another data store on the server. Silverlight is a client-side technology, so when we need to connect to data sources, we need to rely on services. Silverlight has a broad spectrum of services to which it can connect. In this recipe, we'll look at the concepts of connecting with services, which are usually very similar for all types of services Silverlight can connect with. We'll start by creating an ASMX webservice—in other words, a regular web service. We'll then connect to this service from the Silverlight application and invoke and read its response after connecting to it. Getting ready In this recipe, we'll build the application from scratch. However, the completed code for this recipe can be found in the Chapter07/SilverlightJackpot_Read_Completed folder in the code bundle that is available on the Packt website. How to do it... We'll start to explore the usage of services with Silverlight using the following scenario. Imagine we are building a small game application in which a unique code belonging to a user needs to be checked to find out whether or not it is a winning code for some online lottery. The collection of winning codes is present on the server, perhaps in a database or an XML file. We'll create and invoke a service that will allow us to validate the user's code with the collection on the server. The following are the steps we need to follow: We'll build this application from scratch. Our first step is creating a new Silverlight application called SilverlightJackpot. As always, let Visual Studio create a hosting website for the Silverlight client by selecting the Host the Silverlight application in a new Web site checkbox in the New Silverlight Application dialog box. This will ensure that we have a website created for us, in which we can create the service as well. We need to start by creating a service. For the sake of simplicity, we'll create a basic ASMX web service. To do so, right-click on the project node in the SilverlightJackpot. Web project and select Add | New Item... in the menu. In the Add New Item dialog, select the Web Service item. We'll call the new service as JackpotService. Visual Studio creates an ASMX file (JackpotService.asmx) and a code-behind file (JackpotService.asmx.cs). To keep things simple, we'll mock the data retrieval by hardcoding the winning numbers. We'll do so by creating a new class called CodesRepository.cs in the web project. This class returns a list of winning codes. In real-world scenarios, this code would go out to a database and get the list of winning codes from there. The code in this class is very easy. The following is the code for this class: public class CodesRepository{ private List<string> winningCodes; public CodesRepository() { FillWinningCodes(); } private void FillWinningCodes() { if (winningCodes == null) { winningCodes = new List<string>(); winningCodes.Add("12345abc"); winningCodes.Add("azertyse"); winningCodes.Add("abcdefgh"); winningCodes.Add("helloall"); winningCodes.Add("ohnice11"); winningCodes.Add("yesigot1"); winningCodes.Add("superwin"); } } public List<string> WinningCodes { get { return winningCodes; } }} At this point, we need only one method in our JackpotService. This method should accept the code sent from the Silverlight application, check it with the list of winning codes, and return whether or not the user is lucky to have a winning code. Only the methods that are marked with the WebMethod attribute are made available over the service. The following is the code for our service: [WebService(Namespace = "http://tempuri.org/")][WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)][System.ComponentModel.ToolboxItem(false)]public class JackpotService : System.Web.Services.WebService{ List<string> winningCodes; public JackpotService() { winningCodes = new CodesRepository().WinningCodes; } [WebMethod] public bool IsWinningCode(string code) { if(winningCodes.Contains(code)) return true; return false; }} Build the solution at this point to ensure that our service will compile and can be connected from the client side. Now that the service is ready and waiting to be invoked, let's focus on the Silverlight application. To make the service known to our application, we need to add a reference to it. This is done by right-clicking on the SilverlightJackpot project node, and selecting the Add Service Reference... item. In the dialog that appears, we have the option to enter the address of the service ourselves. However, we can click on the Discover button as the service lives in the same solution as the Silverlight application. Visual Studio will search the solution for the available services. If there are no errors, our freshly created service should show up in the list. Select it and rename the Namespace: as JackpotService, as shown in the following screenshot. Visual Studio will now create a proxy class: The UI for the application is kept quite simple. An image of the UI can be seen a little further ahead. It contains a TextBox, where the user can enter a code, a Button that will invoke a check, and a TextBlock that will display the result. This can be seen in the following code: <StackPanel> <TextBox x_Name="CodeTextBox" Width="100" Height="20"> </TextBox> <Button x_Name="CheckForWinButton" Content="Check if I'm a winner!" Click="CheckForWinButton_Click"> </Button> <TextBlock x_Name="ResultTextBlock"> </TextBlock></StackPanel> In the Click event handler, we'll create an instance of the proxy class that was created by Visual Studio as shown in the following code: private void CheckForWinButton_Click(object sender, RoutedEventArgs e){ JackpotService.JackpotServiceSoapClient client = new SilverlightJackpot.JackpotService.JackpotServiceSoapClient();} All service communications in Silverlight happen asynchronously. Therefore, we need to provide a callback method that will be invoked when the service returns: client.IsWinningCodeCompleted += new EventHandler <SilverlightJackpot.JackpotService. IsWinningCodeCompletedEventArgs> (client_IsWinningCodeCompleted); To actually invoke the service, we need to call the IsWinningCodeAsync method as shown in the following line of code. This method will make the actual call to the service. We pass in the value that the user entered: client.IsWinningCodeAsync(CodeTextBox.Text); Finally, in the callback method, we can work with the result of the service via the Result property of the IsWinningCodeCompletedEventArgs instance. Based on the value, we display another message as shown in the following code: void client_IsWinningCodeCompleted(object sender, SilverlightJackpot.JackpotService. IsWinningCodeCompletedEventArgs e){ bool result = e.Result; if (result) ResultTextBlock.Text = "You are a winner! Enter your data below and we will contact you!"; else ResultTextBlock.Text = "You lose... Better luck next time!";} We now have a fully working Silverlight application that uses a service for its data needs. The following screenshot shows the result from entering a valid code: How it works... As it stands, the current version of Silverlight does not have support for using a local database. Silverlight thus needs to rely on external services for getting external data. Even if we had local database support, we would still need to use services in many scenarios. The sample used in this recipe is a good example of data that would need to reside in a secure location (meaning on the server). In any case, we should never store the winning codes in a local database that would be downloaded to the client side. Silverlight has the necessary plumbing on board to connect with the most common types of services. Services such as ASMX, WCF, REST, RSS, and so on, don't pose a problem for Silverlight. While the implementation of connecting with different types of services differs, the concepts are similar. In this recipe, we used a plain old web service. Only the methods that are attributed with the WebMethodAttribute are made available over the service. This means that even if we create a public method on the service, it won't be available to clients if it's not marked as a WebMethod. In this case, we only create a single method called IsWinningCode, which retrieves a list of winning codes from a class called CodesRepository. In real-world applications, this data could be read from a database or an XML file. Thus, this service is the entry point to the data. For Silverlight to work with the service, we need to add a reference to it. When doing so, Visual Studio will create a proxy class. Visual Studio can do this for us because the service exposes a Web Service Description Language (WSDL) file. This file contains an overview of the methods supported by the service. A proxy can be considered a copy of the server-side service class, but without the implementations. Instead, each copied method contains a call to the actual service method. The proxy creation process carried out by Visual Studio is the same as adding a service reference in a regular .NET application. However, invoking the service is somewhat different. All communication with services in Silverlight is carried out asynchronously. If this wasn't the case, Silverlight would have had to wait for the service to return its result. In the meantime, the UI thread would be blocked and no interaction with the rest of the application would be possible. To support the asynchronous service call inside the proxy, the IsWinningCodeAsync method as well as the IsWinningCodeCompleted event is generated. The IsWinningCodeAsync method is used to make the actual call to the service. To get access to the results of a service call, we need to define a callback method. This is where the IsWinningCodeCompleted event comes in. Using this event, we define which method should be called when the service returns (in our case, the client_IsWinningCodeCompleted method). Inside this method, we have access to the results through the Result parameter, which is always of the same type as the return type of the service method. See also Apart from reading data, we also have to persist data. In the next recipe, Persisting data using a standardized service, we'll do exactly that.

First Normal Form Violation in Dimension table Let’s revisit the author problem in the book dimension. The AUTHOR column contains multiple authors, a first-normal-form violation, which prevents us from querying book or sales by author. BOOK dimension table BOOK_SK TITLE AUTHOR PUBLISHER CATEGORY SUB-CATEGORY 1 Programming in Java King, Chan Pac Programming Java 2 Learning Python Simpson Pac Programming Python 3 Introduction to BIRT Chan, Gupta, Simpson (Editor) Apes Reporting BIRT 4 Advanced Java King, Chan Apes Programming Java Normalizing and spinning off the authors into a dimension and adding an artificial BOOK AUTHOR fact solves the problem; it is an artificial fact as it does not contain any real business measure.  Note that the Editor which is an author’s role is also “normalized” into its column in the AUTHOR table (It is related to author, but not actually an author’s name). AUTHOR table AUTHOR_SK AUTHOR_NAME 1 King 2 Chan 3 Simpson 4 Gupta BOOK AUTHOR table BOOK_SK AUTHOR_SK ROLE COUNT 1 1 Co-author 1 1 2 Co-author 1 2 3 Author 1 3 2 Co-author 1 3 3 Editor 1 3 4 Co-author 1 4 1 Co-author 1 4 2 Co-author 1 Note the artificial COUNT measure which facilitates aggregation always has a value of numeric 1. SELECT name, SUM(COUNT) FROM book_author ba, book_dim b, author_dim aWHERE ba.book_sk = b.book_sk AND ba.author_sk = a.author_skGROUP BY name You might need to query sales by author, which you can do so by combining the queries of each of the two stars (the two facts) on their common dimension (BOOK dimension), producing daily book sales by author. SELECT dt, title, name, role, sales_amt FROM (SELECT book_sk, dt, title, sales_amt FROM sales_fact s, date_dim d, book_dim b WHERE s.book_sk = b.book_sk AND s.date_sk = d.date_sk) sales, (SELECT b.book_sk, name, role FROM book_author ba, book_dim b, author_dim a WHERE ba.book_sk = b.book_sk AND ba.author_sk = a.author_sk) authorWHERE sales.book_sk = author.book_sk Single Column with Repeating Value in Dimension table Columns like the PUBLISHER, though not violating any normal form, is also good to get normalized, which we accomplish by adding an artificial fact, PUBLISHED BOOK fact, and its own dimension, PUBLISHER dimension. This normalization is not exactly the same as that in normalizing first-normal-form violation; the PUBLISHER dimension can correctly be linked to the SALES fact, the publisher surrogate key must be added though in the SALES fact. PUBLISHER table   PUBLISHER_SK PUBLISHER 1 Pac 2 Apes BOOK PUBLISHER table BOOK_SK PUBLISHER_SK COUNT 1 1 1 1 2 1 2 3 1 3 2 1 3 3 1 3 4 1 4 1 1 4 2 1 Related Columns with Repeating Value in Dimension table CATEGORY and SUB-CATEGORY columns are related, they form a hierarchy. Each of them can be normalized into its own dimension, but they need to be all linked into one artificial fact. Non-Measure Column in Fact table The ROLE column inside the BOOK AUTHOR fact is not a measure; it violates the dimensional modeling norm; to resolve we just need to spin it off into its own dimension, effectively normalizing the fact table. ROLE dimension table and sample rows   ROLE_SK ROLE 1 Author 2 Co-Author 3 Editor BOOK AUTHOR table with normalized ROLE BOOK_SK AUTHOR_SK ROLE_SK COUNT 1 1 2 1 1 2 2 1 2 3 1 1 3 2 2 1 3 3 3 1 3 4 2 1 4 1 2 1 4 2 2 1   Summary This article shows that both dimensional table and fact table in a dimensional model can be normalized without violating its modeling norm. If you have read this article, you may be interested to view : Solving Many-to-Many Relationship in Dimensional Modeling

In this article by Rafał Kuć and Marek Rogozinski the authors of this book Elasticsearch Server Third Edition, we dived into Elasticsearch indexing. We learned a lot when it comes to data handling. We saw how to tune Elasticsearch schema-less mechanism and we now know how to create our own mappings. We also saw the core types of Elasticsearch and we used analyzers – both the one that comes out of the box with Elasticsearch and the one we define ourselves. We used bulk indexing, and we added additional internal information to our indices. Finally, we learned what segment merging is, how we can fine tune it, and how to use routing in Elasticsearch and what it gives us. This article is fully dedicated to querying. By the end of this article, you will have learned the following topics: How to query Elasticsearch Using the script process Understanding the querying process (For more resources related to this topic, see here.) Querying Elasticsearch So far, when we searched our data, we used the REST API and a simple query or the GET request. Similarly, when we were changing the index, we also used the REST API and sent the JSON-structured data to Elasticsearch. Regardless of the type of operation we wanted to perform, whether it was a mapping change or document indexation, we used JSON structured request body to inform Elasticsearch about the operation details. A similar situation happens when we want to send more than a simple query to Elasticsearch we structure it using the JSON objects and send it to Elasticsearch in the request body. This is called the query DSL. In a broader view, Elasticsearch supports two kinds of queries: basic ones and compound ones. Basic queries, such as the term query, are used for querying the actual data. The second type of query is the compound query, such as the bool query, which can combine multiple queries. However, this is not the whole picture. In addition to these two types of queries, certain queries can have filters that are used to narrow down your results with certain criteria. Filter queries don't affect scoring and are usually very efficient and easily cached. To make it even more complicated, queries can contain other queries (don't worry; we will try to explain all this!). Furthermore, some queries can contain filters and others can contain both queries and filters. Although this is not everything, we will stick with this working explanation for now. The example data If not stated otherwise, the following mappings will be used for the rest of the article: { "book" : { "properties" : { "author" : { "type" : "string" }, "characters" : { "type" : "string" }, "copies" : { "type" : "long", "ignore_malformed" : false }, "otitle" : { "type" : "string" }, "tags" : { "type" : "string", "index" : "not_analyzed" }, "title" : { "type" : "string" }, "year" : { "type" : "long", "ignore_malformed" : false, "index" : "analyzed" }, "available" : { "type" : "boolean" } } } } The preceding mappings represent a simple library and were used to create the library index. One thing to remember is that Elasticsearch will analyze the string based fields if we don't configure it differently. The preceding mappings were stored in the mapping.json file and in order to create the mentioned library index we can use the following commands: curl -XPOST 'localhost:9200/library' curl -XPUT 'localhost:9200/library/book/_mapping' -d @mapping.json We also used the following sample data as the example ones for this article: { "index": {"_index": "library", "_type": "book", "_id": "1"}} { "title": "All Quiet on the Western Front","otitle": "Im Westen nichts Neues","author": "Erich Maria Remarque","year": 1929,"characters": ["Paul Bäumer", "Albert Kropp", "Haie Westhus", "Fredrich Müller", "Stanislaus Katczinsky", "Tjaden"],"tags": ["novel"],"copies": 1, "available": true, "section" : 3} { "index": {"_index": "library", "_type": "book", "_id": "2"}} { "title": "Catch-22","author": "Joseph Heller","year": 1961,"characters": ["John Yossarian", "Captain Aardvark", "Chaplain Tappman", "Colonel Cathcart", "Doctor Daneeka"],"tags": ["novel"],"copies": 6, "available" : false, "section" : 1} { "index": {"_index": "library", "_type": "book", "_id": "3"}} { "title": "The Complete Sherlock Holmes","author": "Arthur Conan Doyle","year": 1936,"characters": ["Sherlock Holmes","Dr. Watson", "G. Lestrade"],"tags": [],"copies": 0, "available" : false, "section" : 12} { "index": {"_index": "library", "_type": "book", "_id": "4"}} { "title": "Crime and Punishment","otitle": "Преступлéние и наказáние","author": "Fyodor Dostoevsky","year": 1886,"characters": ["Raskolnikov", "Sofia Semyonovna Marmeladova"],"tags": [],"copies": 0, "available" : true} We stored our sample data in the documents.json file, and we use the following command to index it: curl -s -XPOST 'localhost:9200/_bulk' --data-binary @documents.json A simple query The simplest way to query Elasticsearch is to use the URI request query. For example, to search for the word crime in the title field, you could send a query using the following command: curl -XGET 'localhost:9200/library/book/_search?q=title:crime&pretty' This is a very simple, but limited, way of submitting queries to Elasticsearch. If we look from the point of view of the Elasticsearch query DSL, the preceding query is the query_string query. It searches for the documents that have the term crime in the title field and can be rewritten as follows: { "query" : { "query_string" : { "query" : "title:crime" } } } Sending a query using the query DSL is a bit different, but still not rocket science. We send the GET (POST is also accepted in case your tool or library doesn't allow sending request body in HTTP GET requests) HTTP request to the _search REST endpoint as earlier and include the query in the request body. Let's take a look at the following command: curl -XGET 'localhost:9200/library/book/_search?pretty' -d '{ "query" : { "query_string" : { "query" : "title:crime" } } }' As you can see, we used the request body (the -d switch) to send the whole JSON-structured query to Elasticsearch. The pretty request parameter tells Elasticsearch to structure the response in such a way that we humans can read it more easily. In response to the preceding command, we get the following output: { "took" : 4, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.5, "hits" : [ { "_index" : "library", "_type" : "book", "_id" : "4", "_score" : 0.5, "_source" : { "title" : "Crime and Punishment", "otitle" : "Преступлéние и наказáние", "author" : "Fyodor Dostoevsky", "year" : 1886, "characters" : [ "Raskolnikov", "Sofia Semyonovna Marmeladova" ], "tags" : [ ], "copies" : 0, "available" : true } } ] } } Nice! We got our first search results with the query DSL. Paging and result size Elasticsearch allows us to control how many results we want to get (at most) and from which result we want to start. The following are the two additional properties that can be set in the request body: from: This property specifies the document that we want to have our results from. Its default value is 0, which means that we want to get our results from the first document. size: This property specifies the maximum number of documents we want as the result of a single query (which defaults to 10). For example, if weare only interested in aggregations results and don't care about the documents returned by the query, we can set this parameter to 0. If we want our query to get documents starting from the tenth item on the list and get 20 of items from there on, we send the following query: curl -XGET 'localhost:9200/library/book/_search?pretty' -d '{ "from" : 9, "size" : 20, "query" : { "query_string" : { "query" : "title:crime" } } }' Returning the version value In addition to all the information returned, Elasticsearch can return the version of the document. To do this, we need to add the version property with the value of true to the top level of our JSON object. So, the final query, which requests for version information, will look as follows: curl -XGET 'localhost:9200/library/book/_search?pretty' -d '{ "version" : true, "query" : { "query_string" : { "query" : "title:crime" } } }' After running the preceding query, we get the following results: { "took" : 4, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.5, "hits" : [ { "_index" : "library", "_type" : "book", "_id" : "4", "_version" : 1, "_score" : 0.5, "_source" : { "title" : "Crime and Punishment", "otitle" : "Преступлéние и наказáние", "author" : "Fyodor Dostoevsky", "year" : 1886, "characters" : [ "Raskolnikov", "Sofia Semyonovna Marmeladova" ], "tags" : [ ], "copies" : 0, "available" : true } } ] } } As you can see, the _version section is present for the single hit we got. Limiting the score For nonstandard use cases, Elasticsearch provides a feature that lets us filter the results on the basis of a minimum score value that the document must have to be considered a match. In order to use this feature, we must provide the min_score value at the top level of our JSON object with the value of the minimum score. For example, if we want our query to only return documents with a score higher than 0.75, we send the following query: curl -XGET 'localhost:9200/library/book/_search?pretty' -d '{ "min_score" : 0.75, "query" : { "query_string" : { "query" : "title:crime" } } }' We get the following response after running the preceding query: { "took" : 3, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 0, "max_score" : null, "hits" : [ ] } } If you look at the previous examples, the score of our document was 0.5, which is lower than 0.75, and thus we didn't get any documents in response. Limiting the score usually doesn't make much sense because comparing scores between the queries is quite hard. However, maybe in your case, this functionality will be needed. Choosing the fields that we want to return With the use of the fields array in the request body, Elasticsearch allows us to define which fields to include in the response. Remember that you can only return these fields if they are marked as stored in the mappings used to create the index, or if the _source field was used (Elasticsearch uses the _source field to provide the stored values and the _source field is turned on by default). So, for example, to return only the title and the year fields in the results (for each document), send the following query to Elasticsearch: curl -XGET 'localhost:9200/library/book/_search?pretty' -d '{ "fields" : [ "title", "year" ], "query" : { "query_string" : { "query" : "title:crime" } } }' In response, we get the following output: { "took" : 5, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.5, "hits" : [ { "_index" : "library", "_type" : "book", "_id" : "4", "_score" : 0.5, "fields" : { "title" : [ "Crime and Punishment" ], "year" : [ 1886 ] } } ] } } As you can see, everything worked as we wanted to. There are four things we will like to share with you, which are as follows: If we don't define the fields array, it will use the default value and return the _source field if available. If we use the _source field and request a field that is not stored, then that field will be extracted from the _source field (however, this requires additional processing). If we want to return all the stored fields, we just pass an asterisk (*) as the field name. From a performance point of view, it's better to return the _source field instead of multiple stored fields. This is because getting multiple stored fields may be slower compared to retrieving a single _source field. Source filtering In addition to choosing which fields are returned, Elasticsearch allows us to use the so-called source filtering. This functionality allows us to control which fields are returned from the _source field. Elasticsearch exposes several ways to do this. The simplest source filtering allows us to decide whether a document should be returned or not. Consider the following query: curl -XGET 'localhost:9200/library/book/_search?pretty' -d '{ "_source" : false, "query" : { "query_string" : { "query" : "title:crime" } } }' The result retuned by Elasticsearch should be similar to the following one: { "took" : 12, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.5, "hits" : [ { "_index" : "library", "_type" : "book", "_id" : "4", "_score" : 0.5 } ] } } Note that the response is limited to base information about a document and the _source field was not included. If you use Elasticsearch as a second source of data and content of the document is served from SQL database or cache, the document identifier is all you need. The second way is similar to as described in the preceding fields, although we define which fields should be returned in the document source itself. Let's see that using the following example query: curl -XGET 'localhost:9200/library/book/_search?pretty' -d '{ "_source" : ["title", "otitle"], "query" : { "query_string" : { "query" : "title:crime" } } }' We wanted to get the title and the otitle document fields in the returned _source field. Elasticsearch extracted those values from the original _source value and included the _source field only with the requested fields. The whole response returned by Elasticsearch looked as follows: { "took" : 2, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.5, "hits" : [ { "_index" : "library", "_type" : "book", "_id" : "4", "_score" : 0.5, "_source" : { "otitle" : "Преступлéние и наказáние", "title" : "Crime and Punishment" } } ] } } We can also use asterisk to select which fields should be returned in the _source field; for example, title* will return value for the title field and for title10 (if we have such field in our data). If we have more extended document with nested part, we can use notation with dot; for example, title.* to select all the fields nested under the title object. Finally, we can also specify explicitly which fields we want to include and which to exclude from the _source field. We can include fields using the include property and we can exclude fields using the exclude property (both of them are arrays of values). For example, if we want the returned _source field to include all the fields starting with the letter t but not the title field, we will run the following query: curl -XGET 'localhost:9200/library/book/_search?pretty' -d '{ "_source" : { "include" : [ "t*"], "exclude" : ["title"] }, "query" : { "query_string" : { "query" : "title:crime" } } }' Using the script fields Elasticsearch allows us to use script-evaluated values that will be returned with the result documents. To use the script fields functionality, we add the script_fields section to our JSON query object and an object with a name of our choice for each scripted value that we want to return. For example, to return a value named correctYear, which is calculated as the year field minus 1800, we run the following query: curl -XGET 'localhost:9200/library/book/_search?pretty' -d '{ "script_fields" : { "correctYear" : { "script" : "doc["year"].value - 1800" } }, "query" : { "query_string" : { "query" : "title:crime" } } }' By default, Elasticsearch doesn't allow us to use dynamic scripting. If you tried the preceding query, you probably got an error with information stating that the scripts of type [inline] with operation [search] and language [groovy] are disabled. To make this example work, you should add the script.inline: on property to the elasticsearch.yml file. However, this exposes a security threat. Using the doc notation, like we did in the preceding example, allows us to catch the results returned and speed up script execution at the cost of higher memory consumption. We also get limited to single-valued and single term fields. If we care about memory usage, or if we are using more complicated field values, we can always use the _source field. The same query using the _source field looks as follows: curl -XGET 'localhost:9200/library/book/_search?pretty' -d '{ "script_fields" : { "correctYear" : { "script" : "_source.year - 1800" } }, "query" : { "query_string" : { "query" : "title:crime" } } }' The following response is returned by Elasticsearch with dynamic scripting enabled: { "took" : 76, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.5, "hits" : [ { "_index" : "library", "_type" : "book", "_id" : "4", "_score" : 0.5, "fields" : { "correctYear" : [ 86 ] } } ] } } As you can see, we got the calculated correctYear field in response. Passing parameters to the script fields Let's take a look at one more feature of the script fields - passing of additional parameters. Instead of having the value 1800 in the equation, we can usea variable name and pass its value in the params section. If we do this, our query will look as follows: curl -XGET 'localhost:9200/library/book/_search?pretty' -d '{ "script_fields" : { "correctYear" : { "script" : "_source.year - paramYear", "params" : { "paramYear" : 1800 } } }, "query" : { "query_string" : { "query" : "title:crime" } } }' As you can see, we added the paramYear variable as part of the scripted equation and provided its value in the params section. This allows Elasticsearch to execute the same script with different parameter values in a slightly more efficient way. Understanding the querying process After reading the previous section, we now know how querying works in Elasticsearch. You know that Elasticsearch, in most cases, needs to scatter the query across multiple nodes, get the results, merge them, fetch the relevant documents from one or more shards, and return the final results to the client requesting the documents. What we didn't talk about are two additional things that define how queries behave: search type and query execution preference. We will now concentrate on these functionalities of Elasticsearch. Query logic Elasticsearch is a distributed search engine and so all functionality provided must be distributed in its nature. It is exactly the same with querying. Because we would like to discuss some more advanced topics on how to control the query process, we first need to know how it works. Let's now get back to how querying works. By default, if we don't alter anything, the query process will consist of two phases: the scatter and the gather phase. The aggregator node (the one that receivesthe request) will run the scatter phase first. During that phase, the query is distributed to all the shards that our index is built of (of course if routing is not used). For example, if it is built of 5 shards and 1 replica then 5 physical shards will be queried (we don't need to query a shard and its replica as they contain the same data). Each of the queried shards will only return the document identifier and the score of the document. The node that sent the scatter query will wait for all the shards to complete their task, gather the results, and sort them appropriately (in this case, from top scoring to the lowest scoring ones). After that, a new request will be sent to build the search results. However, now only to those shards that held the documents to build the response. In most cases, Elasticsearch won't send the request to all the shards but to its subset. That's because we usually don't get the complete result of the query but only a portion of it. This phase is called the gather phase. After all the documents are gathered, the final response is built and returned as the query result. This is the basic and default Elasticsearch behavior but we can change it. Search type Elasticsearch allows us to choose how we want our query to be processed internally. We can do that by specifying the search type. There are different situations where different search type are appropriate: sometimes one can care only about the performance while sometimes query relevance is the most important factor. You should remember that each shard is a small Lucene index and in order to return more relevant results, some information, such as frequencies, needs to be transferred between the shards. To control how the queries are executed, we can pass the search_type request parameter and set it to one of the following values: query_then_fetch: In the first step, the query is executed to get the information needed to sort and rank the documents. This step is executed against all the shards. Then only the relevant shards are queried for the actual content of the documents. Different from query_and_fetch, the maximum number of results returned by this query type will be equal to the size parameter. This is the search type used by default if no search type is provided with the query, and this is the query type we described previously. dfs_query_then_fetch: Again, as with the previous dfs_query_and_fetch, dfs_query_then_fetch is similar to its counterpart query_then_fetch. However, it contains an additional phase comparing which calculates distributed term frequencies. There are also two deprecated search types: count and scan. The first one is deprecated starting from Elasticsearch 2.0 and the second one starting with Elasticsearch 2.1. The first search type used to provide benefits where only aggregations or the number of documents was relevant, but now it is enough to add size equal to 0 to your queries. The scan request was used for scrolling functionality. So if we would like to use the simplest search type, we would run the following command: curl -XGET 'localhost:9200/library/book/_search?pretty&search_type=query_then_fetch' -d '{ "query" : { "term" : { "title" : "crime" } } }' Search execution preference In addition to the possibility of controlling how the query is executed, we can also control on which shards to execute the query. By default, Elasticsearch uses shards and replicas, both the ones available on the node we've sent the request and on the other nodes in the cluster. The default behavior is mostly the proper method of shard preference of queries. But there may be times when we want to change the default behavior. For example, you may want the search to be only executed on the primary shards. To do that, we can set the preference request parameter to one of the following values: _primary: The operation will be only executed on the primary shards, so the replicas won't be used. This can be useful when we need to use the latest information from the index but our data is not replicated right away. _primary_first: The operation will be executed on the primary shards if they are available. If not, it will be executed on the other shards. _replica: The operation will be executed only on the replica shards. _replica_first: This operation is similar to _primary_first, but uses replica shards. The operation will be executed on the replica shards if possible, and on the primary shards if the replicas are not available. _local: The operation will be executed on the shards available on the node which the request was sent and if such shards are not present, the request will be forwarded to the appropriate nodes. _only_node:node_id: This operation will be executed on the node with the provided node identifier. _only_nodes:nodes_spec: This operation will be executed on the nodes that are defined in nodes_spec. This can be an IP address, a name, a name or IP address using wildcards, and so on. For example, if nodes_spec is set to 192.168.1.*, the operation will be run on the nodes with IP address starting with 192.168.1. _prefer_node:node_id: Elasticsearch will try to execute the operation on the node with the provided identifier. However, if the node is not available, it will be executed on the nodes that are available. _shards:1,2: Elasticsearch will execute the operation on the shards with the given identifiers; in this case, on shards with identifiers 1 and 2. The _shards parameter can be combined with other preferences, but the shards identifiers need to be provided first. For example, _shards:1,2;_local. Custom value: Any custom, string value may be passed. Requests with the same values provided will be executed on the same shards. For example, if we would like to execute a query only on the local shards, we would run the following command: curl -XGET 'localhost:9200/library/_search?pretty&preference=_local' -d '{ "query" : { "term" : { "title" : "crime" } } }' Search shards API When discussing the search preference, we will also like to mention the search shards API exposed by Elasticsearch. This API allows us to check which shards the query will be executed at. In order to use this API, run a request against the search_shards rest end point. For example, to see how the query will be executed, we run the following command: curl -XGET 'localhost:9200/library/_search_shards?pretty' -d '{"query":"match_all":{}}' The response to the preceding command will be as follows: { "nodes" : { "my0DcA_MTImm4NE3cG3ZIg" : { "name" : "Cloud 9", "transport_address" : "127.0.0.1:9300", "attributes" : { } } }, "shards" : [ [ { "state" : "STARTED", "primary" : true, "node" : "my0DcA_MTImm4NE3cG3ZIg", "relocating_node" : null, "shard" : 0, "index" : "library", "version" : 4, "allocation_id" : { "id" : "9ayLDbL1RVSyJRYIJkuAxg" } } ], [ { "state" : "STARTED", "primary" : true, "node" : "my0DcA_MTImm4NE3cG3ZIg", "relocating_node" : null, "shard" : 1, "index" : "library", "version" : 4, "allocation_id" : { "id" : "wfpvtaLER-KVyOsuD46Yqg" } } ], [ { "state" : "STARTED", "primary" : true, "node" : "my0DcA_MTImm4NE3cG3ZIg", "relocating_node" : null, "shard" : 2, "index" : "library", "version" : 4, "allocation_id" : { "id" : "zrLPWhCOSTmjlb8TY5rYQA" } } ], [ { "state" : "STARTED", "primary" : true, "node" : "my0DcA_MTImm4NE3cG3ZIg", "relocating_node" : null, "shard" : 3, "index" : "library", "version" : 4, "allocation_id" : { "id" : "efnvY7YcSz6X8X8USacA7g" } } ], [ { "state" : "STARTED", "primary" : true, "node" : "my0DcA_MTImm4NE3cG3ZIg", "relocating_node" : null, "shard" : 4, "index" : "library", "version" : 4, "allocation_id" : { "id" : "XJHW2J63QUKdh3bK3T2nzA" } } ] ] } As you can see, in the response returned by Elasticsearch, we have the information about the shards that will be used during the query process. Of course, with the search shards API, we can use additional parameters that control the querying process. These properties are routing, preference, and local. We are already familiar with the first two. The local parameter is a Boolean (values true or false) one that allows us to tell Elasticsearch to use the cluster state information stored on the local node (setting local to true) instead of the one from the master node (setting local to false). This allows us to diagnose problems with cluster state synchronization. Summary This article has been all about the querying Elasticsearch. We started by looking at how to query Elasticsearch and what Elasticsearch does when it needs to handle the query. We also learned about the basic and compound queries, so we are now able to use both simple queries as well as the ones that group multiple small queries together. Finally, we discussed how to choose the right query for a given use case. Resources for Article: Further resources on this subject: Extending ElasticSearch with Scripting [article] Integrating Elasticsearch with the Hadoop ecosystem [article] Elasticsearch Administration [article]

  WordPress 3 Search Engine Optimization Optimize your website for popularity with search engines         Read more about this book       (For more resources on WordPress, see here.) Typical black hat techniques There is a wide range of black hat techniques fully available to all webmasters. Some techniques can improve rankings in short term, but generally not to the extent that legitimate web development would, if pursued with the same effort. The risk of black hat techniques is that they are routinely detected and punished. Black hat is never the way to go for a legitimate business, and pursuing black hat techniques can get your site (or sites) permanently banned and will also require you to build an entirely new website with an entirely new domain name. We will examine a few black hat techniques to help you avoid them. Hidden text on web pages Hidden text is the text that through either coding or coloring does not appear to users, but appears to search engines. Hidden text is a commonly-used technique, and would be better described as gray hat. It tends not to be severely punished when detected. One technique relies on the coloring of elements. When the color of a text element is set to the same color as the background (either through CSS or HTML coding), then the text disappears from human readers while still visible to search spiders. Unfortunately, for webmasters employing this technique, it's entirely detectible by Google. More easily detectible is the use of the CSS property display: none. In the language of CSS, this directs browsers to not display the text that is defined by that element. This technique is easily detectible by search engines. There is an obvious alternative to employing hidden text: Simply use your desired keywords in the text of your content and display the text to both users and search spiders. Spider detection, cloaking, redirection, and doorway pages Cloaking and spider detection are related techniques. Cloaking is a black hat SEO technique whereby the content presented to search engine spiders (via search spider detection) differs from the content presented to users. Who would employ such a technique? Cloaking is employed principally by sellers of products typically promoted by spam, such as pharmaceutics, adult sites, and gambling sites. Since legitimate search traffic is difficult to obtain in these niches, the purveyors of these products employ cloaking to gain visitors. Traditional cloaking relies upon spider detection. When a search spider visits a website, the headers accompanying a page view request identify the spider by names such as Goolgebot (Google's spider) or Slurp (Inktomi's spider). Conversely, an ordinary web browser (presumably with a human operator) will identify itself as Mozilla, Internet Explorer, or Safari, as the case may be. With simple JavaScript or with server configuration, it is quite easy to identify the requesting browser and deliver one version of a page to search spiders and another version of the page to human browsers. All you really need is to know the names of the spiders, which are publicly known. A variation of cloaking is a doorway page. A doorway page is a page through which human visitors are quickly redirected (through a meta refresh or JavaScript) to a destination page. Search spiders, however, index the doorway page, and not the destination page. Although the technique differs in execution, the effect is the same: Human visitors see one page, and the search engines see another. The potential harm from cloaking goes beyond search engine manipulation. More often than not, the true destination pages in a cloaking scheme are used for the transmission of malware, viruses, and Trojans. Because the search engines aren't necessarily reading the true destination pages, the malicious code isn't detected. Any type of cloaking, when reported or detected, is almost certain to result in a severe Google penalty, such as removal of a site from the search engine indexes. Linking to bad neighborhoods and link farms A bad neighborhood is a website or a network of websites that either earns inbound links through illegitimate means or employs other "black hat on-page" techniques such as cloaking, and redirects them. A link farm is a website that offers almost no content, but serves solely for the purpose of listing links. Link farms, in turn, offer links to other websites to increase the rankings of these sites. A wide range of black hat techniques can get a website labeled as a bad neighborhood. A quick test you can employ to determine if a site is a bad neighborhood is by entering the domain name as a part of the specialized Google search query, "site:the-website-domain.com" to see if Google displays any pages of that website in its index. If Google returns no results, the website is either brand new or has been removed from Google's index—a possible indicator that it has been labeled a bad neighborhood. Another quick test is to check the site's PageRank and compare the figure to the number of inbound links pointing to the site. If a site has a large number of backlinks but has a PageRank of zero, which would tend to indicate that its PageRank has been manually adjusted downwards due to a violation of Google's Webmaster Guidelines. If both of the previous tests are either positive or inconclusive, you would still be wise to give the site a "smell test". Here are some questions to ask when determining if a site might be deemed as a bad neighborhood: Does the site offer meaningful content? Did you detect any redirection while visiting the site? Did you get any virus warning while visiting the site? Is the site a little more than lists of links or text polluted with high numbers of links? Check the website's backlink profile. Are the links solely low-value inbound links? If it isn't a site you would engage with when visiting, don't link to it. Google Webmaster Guidelines Google Webmaster Guidelines are a set of written rules and prohibitions that outline recommended and forbidden website practices. You can find these webmaster guidelines at: http://www.google.com/support/webmasters/bin/ answer.py?hl=en&answer=35769, though you'll find it easier to search for "Google Webmaster Guidelines" and click on the top search result. You should read through the Google Webmaster Guidelines and refer to them occasionally. The guidelines are divided into design and content guidelines, technical guidelines, and quality guidelines. Google Webmaster Guidelines in a nutshell At their core, Google Webmaster Guidelines aim for quality in the technology underlying websites in their index, high-quality content, and also discourage manipulation of search results through deceptive techniques. All search engines have webmaster guidelines, but if you follow Google's dictates, you will not run afoul of any of the other search engines. Here, we'll discuss only the Google's rules. Google's design and content guidelines instruct that your site should have a clear navigational hierarchy with text links rather than image links. The guidelines specifically note that each page "should be reachable from at least one static text link". Because WordPress builds text-based, hierarchical navigation naturally, your site will also meet that rule naturally. The guidelines continue by instructing that your site should load quickly and display consistently among different browsers. The warnings come in Google's quality guidelines; in this section, you'll see how Google warns against a wide range of black hat techniques such as the following: Using hidden text or hidden links, elements that through coloring, font size, or CSS display properties to show to the search engines but do not show them to the users. The use of cloaking or "sneaky redirects". Cloaking means a script that detects search engine spiders and displays one version of a website to users and displays an alternate version to the search engines. The use of repetitive, automated queries to Google. Some unscrupulous software vendors (Google mentions one by name, WebPosition Gold, which is still in the market, luring unsuspecting webmasters) sell software and services that repeatedly query Google to determine website rankings. Google does allow such queries in some instances through their AJAX Search API Key—but you need to apply for one and abide by the terms of its use. The creation of multiple sites or pages that consist solely of duplicate content that appears on other web properties. The posting or installation of scripts that behave maliciously towards users, such as with viruses, trojans, browser interceptors, or other badware. Participation in link schemes. Google is quite public that it values inbound links as a measure of site quality, so it is ever vigilant to detect and punish illegitimate link programs. Linking to bad neighborhoods. A bad neighborhood means a website that uses illegitimate, forbidden techniques to earn inbound links or traffic. Stuffing keywords onto pages in order to fool search spiders. Keyword stuffing is "the oldest trick in the book". It's not only forbidden, but also highly ineffective at influencing search results and highly annoying to visitors. When Google detects violations of its guidelines Google, which is nearly an entirely automated system, is surprisingly capable of detecting violations of its guidelines. Google encourages user-reporting of spam websites, cloaked pages, and hidden text (through their page here: https://www. google.com/webmasters/tools/spamreport). They maintain an active antispam department that is fully engaged in an ongoing improvement in both, manual punishments for offending sites, and algorithmic improvements for detecting violations. When paid link abuses are detected, Google will nearly always punish the linking site, not necessarily the site receiving the link—even though the receiving site is the one earning a ranking benefit. At first glance, this may seem counter-intuitive, but there is a reason. If Google punished the site receiving a forbidden paid link, then any site owner could knock a competitor's website by buying a forbidden link, pointing to the competitor, and then reporting the link as spam. When an on-page black hat or gray hat element is detected, the penalty will be imposed upon the offending site. The penalties range from a ranking adjustment to an outright ban from search engine results. Generally, the penalty matches the crime; the more egregious penalties flow from more egregious violations. We need to draw a distinction, however, between a Google ban, penalty, and algorithmic filtering. Algorithmic filtering is simply an adjustment to the rankings or indexing of a site. If you publish content that is a duplicate of the other content on the Web, and Google doesn't rank or index that page, that's not a penalty, it's simply the search engine algorithm operating properly. If all of your pages are removed from the search index, that is most likely a ban. If the highest ranking you can achieve is position 40 for any search phrase, that could potentially be a penalty called a "-40 penalty". All search engines can impose discipline upon websites, but Google is the most strict and imposes far more penalties than the other search engines, so we will largely discuss Google here. Filtering is not a penalty; it is an adjustment that can be remedied by undoing the condition that led to the it. Filtering can occur for a variety of reasons but is often imposed following over optimization. For example, if your backlink profile comprises links of which 80% use the same anchor text, you might trigger a filter. The effect of a penalty or filter is the same: decreased rankings and traffic. In the following section, we'll look at a wide variety of known Google filters and penalties, and learn how to address them.