How-To Tutorials - Programming

This article by Aleksandar Prokopec, author of the book Learning Concurrent Programming in Scala - Second Edition, explains the concepts of asynchronous programming in Scala. Asynchronous programming helps you eliminate blocking; instead of suspending the thread whenever a resource is not available, a separate computation is scheduled to proceed once the resource becomes available. In a way, many of the concurrency patterns seen so far support asynchronous programming; thread creation and scheduling execution context tasks can be used to start executing a computation concurrent to the main program flow. Still, it is not straightforward to use these facilities directly when avoiding blocking or composing asynchronous computations. In this article, we will focus on two abstractions in Scala that are specifically tailored for this task—futures and promises. More specifically, we will study the following topics: Starting asynchronous computations and using Future objects Using Promise objects to interface Blocking threads inside asynchronous computations Alternative future frameworks (For more resources related to this topic, see here.) Futures The parallel executions in a concurrent program proceed on entities called threads. At any point, the execution of a thread can be temporarily suspended until a specific condition is fulfilled. When this happens, we say that the thread is blocked. Why do we block threads in the first place in concurrent programming? One of the reasons is that we have a finite amount of resources; so, multiple computations that share these resources sometimes need to wait. In other situations, a computation needs specific data to proceed, and if that data is not yet available, threads responsible for producing the data could be slow or the source of the data could be external to the program. A classic example is waiting for the data to arrive over the network. Let's assume that we have a getWebpage method that returns that webpage's contents when given a url string with the location of the webpage: def getWebpage(url: String): String The return type of the getWebpage method is String; the method must return a string with the webpage's contents. Upon sending an HTTP request, though, the webpage's contents are not available immediately. It takes some time for the request to travel over the network to the server and back before the program can access the document. The only way for the method to return the contents of the webpage as a string value is to wait for the HTTP response to arrive. However, this can take a relatively long period of time from the program's point of view even with a high-speed Internet connection, the getWebpage method needs to wait. Since the thread that called the getWebpage method cannot proceed without the contents of the webpage, it needs to pause its execution; therefore, the only way to correctly implement the getWebpage method is for blocking. We already know that blocking can have negative side effects, so can we change the return value of the getWebpage method to some special value that can be returned immediately? The answer is yes. In Scala, this special value is called a future. The future is a placeholder, that is, a memory location for the value. This placeholder does not need to contain a value when the future is created; the value can be placed into the future eventually by getWebpage. We can change the signature of the getWebpage method to return a future as follows: def getWebpage(url: String): Future[String] Here, the Future[String] type means that the future object can eventually contain a String value. We can now implement getWebpage without blocking—we can start the HTTP request asynchronously and place the webpage's contents into the future when they become available. When this happens, we can say that the getWebpage method completes the future. Importantly, after the future is completed with some value, that value can no longer change. The Future[T] type encodes latency in the program—use it to encode values that will become available later during execution. This removes blocking from the getWebpage method, but it is not clear how the calling thread can extract the content of the future. Polling is one non-blocking way of extracting the content. In the polling approach, the calling thread calls a special method for blocking until the value becomes available. While this approach does not eliminate blocking, it transfers the responsibility of blocking from the getWebpage method to the caller thread. Java defines its own Future type to encode values that will become available later. However, as a Scala developer, you should use Scala's futures instead; they allow additional ways of handling future values and avoid blocking, as we will soon see. When programming with futures in Scala, we need to distinguish between future values and future computations. A future value of the Future[T] type denotes some value of the T type in the program that might not be currently available, but could become available later. Usually, when we say a future, we really mean a future value. In the scala.concurrent package, futures are represented with the Future[T] trait: trait Future[T] By contrast, a future computation is an asynchronous computation that produces a future value. A future computation can be started by calling the apply method on the Future companion object. This method has the following signature in the scala.concurrent package: def apply[T](b: =>T)(implicit e: ExecutionContext): Future[T] This method takes a by-name parameter of the T type. This is the body of the asynchronous computation that results in some value of type T. It also takes an implicit ExecutionContext parameter, which abstracts over where and when the thread gets executed. Recall that Scala's implicit parameters can either be specified when calling a method, in the same way as normal parameters, or they can be left out; in this case, the Scala compiler searches for a value of the ExecutionContext type in the surrounding scope. Most Future methods take an implicit execution context. Finally, the Future.apply method returns a future of the T type. This future is completed with the value resulting from the asynchronous computation, b. Starting future computations Let's see how to start a future computation in an example. We first import the contents of the scala.concurrent package. We then import the global execution context from the Implicits object. This makes sure that the future computations execute on the global context—the default execution context you can use in most cases: import scala.concurrent._ import ExecutionContext.Implicits.global object FuturesCreate extends App { Future { log("the future is here") } log("the future is coming") Thread.sleep(1000) } The order in which the log method calls (in the future computation and the main thread) execute is nondeterministic. The Future singleton object followed by a block is syntactic sugar for calling the Future.apply method. In the following example, we can use the scala.io.Source object to read the contents of our build.sbt file in a future computation. The main thread calls the isCompleted method on the future value, buildFile, returned from the future computation. Chances are that the build file was not read so fast, so isCompleted returns false. After 250 milliseconds, the main thread calls isCompleted again, and this time, isCompleted returns true. Finally, the main thread calls the value method, which returns the contents of the build file: import scala.io.Source object FuturesDataType extends App { val buildFile: Future[String] = Future { val f = Source.fromFile("build.sbt") try f.getLines.mkString("n") finally f.close() } log(s"started reading the build file asynchronously") log(s"status: ${buildFile.isCompleted}") Thread.sleep(250) log(s"status: ${buildFile.isCompleted}") log(s"value: ${buildFile.value}") } In this example, we used polling to obtain the value of the future. The Future singleton object's polling methods are non-blocking, but they are also nondeterministic; the isCompleted method will repeatedly return false until the future is completed. Importantly, completion of the future is in a happens-before relationship with the polling calls. If the future completes before the invocation of the polling method, then its effects are visible to the thread after the polling completes. Shown graphically, polling looks as shown in the following figure: Polling diagram Polling is like calling your potential employer every five minutes to ask if you're hired. What you really want to do is hand in a job application and then apply for other jobs instead of busy-waiting for the employer's response. Once your employer decides to hire you, he will give you a call on the phone number you left him. We want futures to do the same; when they are completed, they should call a specific function that we left for them. Promises Promises are objects that can be assigned a value or an exception only once. This is why promises are sometimes also called single-assignment variables. A promise is represented with the Promise[T] type in Scala. To create a promise instance, we use the Promise.apply method on the Promise companion object: def apply[T](): Promise[T] This method returns a new promise instance. Like the Future.apply method, the Promise.apply method returns immediately; it is non-blocking. However, the Promise.apply method does not start an asynchronous computation, it just creates a fresh Promise object. When the Promise object is created, it does not contain a value or an exception. To assign a value or an exception to a promise, we use the success or failure method, respectively. Perhaps you have noticed that promises are very similar to futures. Both futures and promises are initially empty and can be completed with either a value or an exception. This is intentional—every promise object corresponds to exactly one future object. To obtain the future associated with a promise, we can call the future method on the promise. Calling this method multiple times always returns the same future object. A promise and a future represent two aspects of a single-assignment variable. The promise allows you to assign a value to the future object, whereas the future allows you to read that value. In the following code snippet, we create two promises, p and q, that can hold string values. We then install a foreach callback on the future associated with the p promise and wait for 1 second. The callback is not invoked until the p promise is completed by calling the success method. We then fail the q promise in the same way and install a failed.foreach callback: object PromisesCreate extends App { val p = Promise[String] val q = Promise[String] p.future foreach { case x => log(s"p succeeded with '$x'") } Thread.sleep(1000) p success "assigned" q failure new Exception("not kept") q.future.failed foreach { case t => log(s"q failed with $t") } Thread.sleep(1000) } Alternatively, we can use the complete method and specify a Try[T] object to complete the promise. Depending on whether the Try[T] object is a success or a failure, the promise is successfully completed or failed. Importantly, after a promise is either successfully completed or failed, it cannot be assigned an exception or a value again in any way. Trying to do so results in an exception. Note that this is true even when there are multiple threads simultaneously calling success or complete. Only one thread completes the promise, and the rest throw an exception. Assigning a value or an exception to an already completed promise is not allowed and throws an exception. We can also use the trySuccess, tryFailure, and tryComplete methods that correspond to success, failure, and complete states, respectively, but return a Boolean value to indicate whether the assignment was successful. Recall that using the Future.apply method and callback methods with referentially transparent functions results in deterministic concurrent programs. As long as we do not use the trySuccess, tryFailure, and tryComplete methods, and none of the success, failure, and complete methods ever throws an exception, we can use promises and retain determinism in our programs. We now have everything we need to implement our custom Future.apply method. We call it the myFuture method in the following example. The myFuture method takes a b by-name parameter that is the asynchronous computation. First, it creates a p promise. Then, it starts an asynchronous computation on the global execution context. This computation tries to evaluate b and complete the promise. However, if the b body throws a nonfatal exception, the asynchronous computation fails the promise with that exception. In the meanwhile, the myFuture method returns the future immediately after starting the asynchronous computation: import scala.util.control.NonFatal object PromisesCustomAsync extends App { def myFuture[T](b: =>T): Future[T] = { val p = Promise[T] global.execute(new Runnable { def run() = try { p.success(b) } catch { case NonFatal(e) => p.failure(e) } }) p.future } val f = myFuture { "naa" + "na" * 8 + " Katamari Damacy!" } f foreach { case text => log(text) } } This is a common pattern when producing futures. We create a promise, let some other computation complete that promise, and return the corresponding future. However, promises were not invented just for our custom future's myFuture computation method. In the following sections, we will study use cases in which promises are useful. Futures and blocking Futures and asynchronous computations mainly exist to avoid blocking, but in some cases, we cannot live without it. It is, therefore, valid to ask how blocking interacts with futures. There are two ways to block with futures. The first way is to wait until a future is completed. The second way is by blocking from within an asynchronous computation. We will study both the topics in this section. Awaiting futures In rare situations, we cannot use callbacks or future combinators to avoid blocking. For example, the main thread that starts multiple asynchronous computations has to wait for these computations to finish. If an execution context uses daemon threads, as is the case with the global execution context, the main thread needs to block to prevent the JVM process from terminating. In these exceptional circumstances, we use the ready and result methods on the Await object from the scala.concurrent package. The ready method blocks the caller thread until the specified future is completed. The result method also blocks the caller thread, but returns the value of the future if it was completed successfully or throws the exception in the future if the future is failed. Both the methods require specifying a timeout parameter; the longest duration that the caller should wait for the completion of the future before a TimeoutException method is thrown. To specify a timeout, we import the scala.concurrent.duration package. This allows us to write expressions such as 10.seconds: import scala.concurrent.duration._ object BlockingAwait extends App { val urlSpecSizeFuture = Future { val specUrl = "http://www.w3.org/Addressing/URL/url-spec.txt" Source.fromURL(specUrl).size } val urlSpecSize = Await.result(urlSpecSizeFuture, 10.seconds) log(s"url spec contains $urlSpecSize characters") } In this example, the main thread starts a computation that retrieves the URL specification and then awaits. By this time, the World Wide Web Consortium (W3C) is worried that a DOS attack is under way, so this is the last time we download the URL specification. Blocking in asynchronous computations Waiting for the completion of a future is not the only way to block. Some legacy APIs do not use callbacks to asynchronously return results. Instead, such APIs expose the blocking methods. After we call a blocking method, we lose control over the thread; it is up to the blocking method to unblock the thread and return the control back. Execution contexts are often implemented using thread pools. By starting future computations that block, it is possible to reduce parallelism and even cause deadlocks. This is illustrated in the following example, in which 16 separate future computations call the sleep method, and the main thread waits until they complete for an unbounded amount of time: val startTime = System.nanoTime val futures = for (_ <- 0 until 16) yield Future { Thread.sleep(1000) } for (f <- futures) Await.ready(f, Duration.Inf) val endTime = System.nanoTime log(s"Total time = ${(endTime - startTime) / 1000000} ms") log(s"Total CPUs = ${Runtime.getRuntime.availableProcessors}") Assume that you have eight cores in your processor. This program does not end in one second. Instead, a first batch of eight futures started by Future.apply will block all the worker threads for one second, and then another batch of eight futures will block for another second. As a result, none of our eight processor cores can do any useful work for one second. Avoid blocking in asynchronous computations, as it can cause thread starvation. If you absolutely must block, then the part of the code that blocks should be enclosed within the blocking call. This signals to the execution context that the worker thread is blocked and allows it to temporarily spawn additional worker threads if necessary:   val futures = for (_ <- 0 until 16) yield Future { blocking { Thread.sleep(1000) } } With the blocking call around the sleep call, the global execution context spawns additional threads when it detects that there is more work than the worker threads. All 16 future computations can execute concurrently and the program terminates after one second. The Await.ready and Await.result statements block the caller thread until the future is completed and are in most cases used outside asynchronous computations. They are blocking operations. The blocking statement is used inside asynchronous code to designate that the enclosed block of code contains a blocking call. It is not a blocking operation by itself. Alternative future frameworks Scala futures and promises API resulted from an attempt to consolidate several different APIs for asynchronous programming, among them, legacy Scala futures, Akka futures, Scalaz futures, and Twitter's Finagle futures. Legacy Scala futures and Akka futures have already converged to the futures and promises API that you've learned about so far in this article. Finagle's com.twitter.util.Future type is planned to eventually implement the same interface as scala.concurrent.Future package, while the Scalaz scalaz.concurrent.Future type implements a slightly different interface. In this section, we give a brief of Scalaz futures. To use Scalaz, we add the following dependency to the build.sbt file: libraryDependencies += "org.scalaz" %% "scalaz-concurrent" % "7.0.6" We now encode an asynchronous tombola program using Scalaz. The Future type in Scalaz does not have the foreach method. Instead, we use its runAsync method, which asynchronously runs the future computation to obtain its value and then calls the specified callback: import scalaz.concurrent._ object Scalaz extends App { val tombola = Future { scala.util.Random.shuffle((0 until 10000).toVector) } tombola.runAsync { numbers => log(s"And the winner is: ${numbers.head}") } tombola.runAsync { numbers => log(s"... ahem, winner is: ${numbers.head}") } } Unless you are terribly lucky and draw the same permutation twice, running this program reveals that the two runAsync calls print different numbers. Each runAsync call separately computes the permutation of the random numbers. This is not surprising, as Scalaz futures have the pull semantics, in which the value is computed each time some callback requests it, in contrast to Finagle and Scala futures' push semantics, in which the callback is stored, and applied if and when the asynchronously computed value becomes available. To achieve the same semantics, as we would have with Scala futures, we need to use the start combinator that runs the asynchronous computation once, and caches its result: val tombola = Future { scala.util.Random.shuffle((0 until 10000).toVector) } start With this change, the two runAsync calls use the same permutation of random numbers in the tombola variable and print the same values. We will not dive further into the internals of alternate frameworks. The fundamentals about futures and promises that you learned about in this article should be sufficient to easily familiarize yourself with other asynchronous programming libraries, should the need arise. Summary This article presented some powerful abstractions for asynchronous programming. We saw how to encode latency with the Future type. You learned that futures and promises are closely tied together and that promises allow interfacing with legacy callback-based systems. In cases, where blocking was unavoidable, you learned how to use the Await object and the blocking statement. Finally, you learned that the Scala Async library is a powerful alternative for expressing future computations more concisely. Resources for Article: Further resources on this subject: Introduction to Scala [article] Concurrency in Practice [article] Integrating Scala, Groovy, and Flex Development with Apache Maven [article]

In this article by Gastón C. Hillar, the author of the book Java 9 with JShell, we will learn about one of the most exciting features of object-oriented programming in Java 9: polymorphism. We will code many classes and then we will work with their instances in JShell to understand how objects can take many different forms. We will: Create concrete classes that inherit from abstract superclasses Work with instances of subclasses Understand polymorphism Control whether subclasses can or cannot override members Control whether classes can be subclassed Use methods that perform operations with instances of different subclasses (For more resources related to this topic, see here.) Creating concrete classes that inherit from abstract superclasses We will consider the existence of an abstract base class named VirtualAnimal and the following three abstract subclasses: VirtualMammal, VirtualDomesticMammal, and VirtualHorse. Next, we will code the following three concrete classes. Each class represents a different horse breed and is a subclass of the VirtualHorse abstract class. AmericanQuarterHorse: This class represents a virtual horse that belongs to the American Quarter Horse breed. ShireHorse: This class represents a virtual horse that belongs to the Shire Horse breed. Thoroughbred: This class represents a virtual horse that belongs to the Thoroughbred breed. The three concrete classes will implement the following three abstract methods they inherited from abstract superclasses: String getAsciiArt(): This abstract method is inherited from the VirtualAnimal abstract class. String getBaby(): This abstract method is inherited from the VirtualAnimal abstract class. String getBreed(): This abstract method is inherited from the VirtualHorse abstract class. The following UML diagram shows the members for the three concrete classes that we will code: AmericanQuarterHorse, ShireHorse, and Thoroughbred. We don’t use bold text format for the three methods that each of these concrete classes will declare because they aren’t overriding the methods, they are implementing the abstract methods that the classes inherited. First, we will create the AmericanQuarterHorse concrete class. The following lines show the code for this class in Java 9. Notice that there is no abstract keyword before class, and therefore, our class must make sure that it implements all the inherited abstract methods. public class AmericanQuarterHorse extends VirtualHorse { public AmericanQuarterHorse( int age, boolean isPregnant, String name, String favoriteToy) { super(age, isPregnant, name, favoriteToy); System.out.println("AmericanQuarterHorse created."); } public AmericanQuarterHorse( int age, String name, String favoriteToy) { this(age, false, name, favoriteToy); } public String getBaby() { return "AQH baby "; } public String getBreed() { return "American Quarter Horse"; } public String getAsciiArt() { return " >>\.n" + " /* )`.n" + " // _)`^)`. _.---. _n" + " (_,' \ `^-)'' `.\n" + " | | \n" + " \ / |n" + " / \ /.___.'\ (\ (_n" + " < ,'|| \ |`. \`-'n" + " \\ () )| )/n" + " |_>|> /_] //n" + " /_] /_]n"; } } Now, we will create the ShireHorse concrete class. The following lines show the code for this class in Java 9: public class ShireHorse extends VirtualHorse { public ShireHorse( int age, boolean isPregnant, String name, String favoriteToy) { super(age, isPregnant, name, favoriteToy); System.out.println("ShireHorse created."); } public ShireHorse( int age, String name, String favoriteToy) { this(age, false, name, favoriteToy); } public String getBaby() { return "ShireHorse baby "; } public String getBreed() { return "Shire Horse"; } public String getAsciiArt() { return " ;;n" + " .;;'*\n" + " __ .;;' ' \n" + " /' '\.~~.~' \ /'\.)n" + " ,;( ) / |n" + " ,;' \ /-.,,( )n" + " ) /| ) /|n" + " ||(_\ ||(_\n" + " (_\ (_\n"; } } Finally, we will create the Thoroughbred concrete class. The following lines show the code for this class in Java 9: public class Thoroughbred extends VirtualHorse { public Thoroughbred( int age, boolean isPregnant, String name, String favoriteToy) { super(age, isPregnant, name, favoriteToy); System.out.println("Thoroughbred created."); } public Thoroughbred( int age, String name, String favoriteToy) { this(age, false, name, favoriteToy); } public String getBaby() { return "Thoroughbred baby "; } public String getBreed() { return "Thoroughbred"; } public String getAsciiArt() { return " })\-=--.n" + " // *._.-'n" + " _.-=-...-' /n" + " {{| , |n" + " {{\ | \ /_n" + " }} \ ,'---'\___\n" + " / )/\\ \\ >\n" + " // >\ >\`-n" + " `- `- `-n"; } } We have more than one constructor defined for the three concrete classes. The first constructor that requires four arguments uses the super keyword to call the constructor from the base class or superclass, that is, the constructor defined in the VirtualHorse class. After the constructor defined in the superclass finishes its execution, the code prints a message indicating that an instance of each specific concrete class has been created. The constructor defined in each class prints a different message. The second constructor uses the this keyword to call the previously explained constructor with the received arguments and with false as the value for the isPregnant argument. Each class returns a different String in the implementation of the getBaby and getBreed methods. In addition, each class returns a different ASCII art representation for a virtual horse in the implementation of the getAsciiArt method. Understanding polymorphism We can use the same method, that is, a method with the same name and arguments, to cause different things to happen according to the class on which we invoke the method. In object-oriented programming, this feature is known as polymorphism. Polymorphism is the ability of an object to take on many forms, and we will see it in action by working with instances of the previously coded concrete classes. The following lines create a new instance of the AmericanQuarterHorse class named american and use one of its constructors that doesn’t require the isPregnant argument: AmericanQuarterHorse american = new AmericanQuarterHorse( 8, "American", "Equi-Spirit Ball"); american.printBreed(); The following lines show the messages that the different constructors displayed in JShell after we enter the previous code: VirtualAnimal created. VirtualMammal created. VirtualDomesticMammal created. VirtualHorse created. AmericanQuarterHorse created. The constructor defined in the AmericanQuarterHorse calls the constructor from its superclass, that is, the VirtualHorse class. Remember that each constructor calls its superclass constructor and prints a message indicating that an instance of the class is created. We don’t have five different instances; we just have one instance that calls the chained constructors of five different classes to perform all the necessary initialization to create an instance of AmericanQuarterHorse. If we execute the following lines in JShell, all of them will display true as a result, because american belongs to the VirtualAnimal, VirtualMammal, VirtualDomesticMammal, VirtualHorse, and AmericanQuarterHorse classes. System.out.println(american instanceof VirtualAnimal); System.out.println(american instanceof VirtualMammal); System.out.println(american instanceof VirtualDomesticMammal); System.out.println(american instanceof VirtualHorse); System.out.println(american instanceof AmericanQuarterHorse); The results of the previous lines mean that the instance of the AmericanQuarterHorse class, whose reference is saved in the american variable of type AmericanQuarterHorse, can take on the form of an instance of any of the following classes: VirtualAnimal VirtualMammal VirtualDomesticMammal VirtualHorse AmericanQuarterHorse The following screenshot shows the results of executing the previous lines in JShell: We coded the printBreed method within the VirtualHorse class, and we didn’t override this method in any of the subclasses. The following is the code for the printBreed method: public void printBreed() { System.out.println(getBreed()); } The code prints the String returned by the getBreed method, declared in the same class as an abstract method. The three concrete classes that inherit from VirtualHorse implemented the getBreed method and each of them returns a different String. When we called the american.printBreed method, JShell displayed American Quarter Horse. The following lines create an instance of the ShireHorse class named zelda. Note that in this case, we use the constructor that requires the isPregnant argument. As happened when we created an instance of the AmericanQuarterHorse class, JShell will display a message for each constructor that is executed as a result of the chained constructors we coded. ShireHorse zelda = new ShireHorse(9, true, "Zelda", "Tennis Ball"); The next lines call the printAverageNumberOfBabies and printAsciiArt instance methods for american, the instance of AmericanQuarterHorse, and zelda, which is the instance of ShireHorse. american.printAverageNumberOfBabies(); american.printAsciiArt(); zelda.printAverageNumberOfBabies(); zelda.printAsciiArt(); We coded the printAverageNumberOfBabies and printAsciiArt methods in the VirtualAnimal class, and we didn’t override them in any of its subclasses. Hence, when we call these methods for either american or zelda, Java will execute the code defined in the VirtualAnimal class. The printAverageNumberOfBabies method uses the int value returned by the getAverageNumberOfBabies and the String returned by the getBaby method to generate a String that represents the average number of babies for a virtual animal. The VirtualHorse class implemented the inherited getAverageNumberOfBabies abstract method with code that returns 1. The AmericanQuarterHorse and ShireHorse classes implemented the inherited getBaby abstract method with code that returns a String that represents a baby for the virtual horse breed: "AQH baby" and "ShireHorse baby". Thus, our call to the printAverageNumberOfBabies method will produce different results in each instance because they belong to a different class. The printAsciiArt method uses the String returned by the getAsciiArt method to print the ASCII art that represents a virtual horse. The AmericanQuarterHorse and ShireHorse classes implemented the inherited getAsciiArt abstract method with code that returns a String with the ASCII art that is appropriate for each virtual horse that the class represents. Thus, our call to the printAsciiArt method will produce different results in each instance because they belong to a different class. The following screenshot shows the results of executing the previous lines in JShell. Both instances run the same code for the two methods that were coded in the VirtualAnimal abstract class. However, each class provided a different implementation for the methods that end up being called to generated the result and cause the differences in the output. The following lines create an instance of the Thoroughbred class named willow, and then call its printAsciiArt method. As happened before, JShell will display a message for each constructor that is executed as a result of the chained constructors we coded. Thoroughbred willow = new Thoroughbred(5, "Willow", "Jolly Ball"); willow.printAsciiArt(); The following screenshot shows the results of executing the previous lines in JShell. The new instance is from a class that provides a different implementation of the getAsciiArt method, and therefore, we will see a different ASCII art than in the previous two calls to the same method for the other instances. The following lines call the neigh method for the instance named willow with a different number of arguments. This way, we take advantage of the neigh method that we overloaded four times with different arguments. Remember that we coded the four neigh methods in the VirtualHorse class and the Thoroughbred class inherits the overloaded methods from this superclass through its hierarchy tree. willow.neigh(); willow.neigh(2); willow.neigh(2, american); willow.neigh(3, zelda, true); american.nicker(); american.nicker(2); american.nicker(2, willow); american.nicker(3, willow, true); The following screenshot shows the results of calling the neigh and nicker methods with the different arguments in JShell: We called the four versions of the neigh method defined in the VirtualHorse class for the Thoroughbred instance named willow. The third and fourth lines that call the neigh method specify a value for the otherDomesticMammal argument of type VirtualDomesticMammal. The third line specifies american as the value for otherDomesticMammal and the fourth line specifies zelda as the value for the same argument. Both the AmericanQuarterHorse and ShireHorse concrete classes are subclasses of VirtualHorse, and VirtualHorse is a subclass or VirtualDomesticMammal. Hence, we can use american and zelda as arguments where a VirtualDomesticMammal instance is required. Then, we called the four versions of the nicker method defined in the VirtualHorse class for the AmericanQuarterHorse instance named american. The third and fourth lines that call the nicker method specify willow as the value for the otherDomesticMammal argument of type VirtualDomesticMammal. The Thoroughbred concrete class is also a subclass of VirtualHorse, and VirtualHorse is a subclass or VirtualDomesticMammal. Hence, we can use willow as an argument where a VirtualDomesticMammal instance is required. Controlling overridability of members in subclasses We will code the VirtualDomesticCat abstract class and its concrete subclass: MaineCoon. Then, we will code the VirtualBird abstract class, its VirtualDomesticBird abstract subclass and the Cockatiel concrete subclass. Finally, we will code the VirtualDomesticRabbit concrete class. While coding these classes, we will use Java 9 features that allow us to decide whether the subclasses can or cannot override specific members. All the virtual domestic cats must be able to talk, and therefore, we will override the talk method inherited from VirtualDomesticMammal to print the word that represents a cat meowing: "Meow". We also want to provide a method to print "Meow" a specific number of times. Hence, at this point, we realize that we can take advantage of the printSoundInWords method we had declared in the VirtualHorse class. We cannot access this instance method in the VirtualDomesticCat abstract class because it doesn’t inherit from VirtualHorse. Thus, we will move this method from the VirtualHorse class to its superclass: VirtualDomesticMammal. We will use the final keyword before the return type for the methods that we don’t want to be overridden in subclasses. When a method is marked as a final method, the subclasses cannot override the method and the Java 9 compiler shows an error if they try to do so. Not all the birds are able to fly in real-life. However, all our virtual birds are able to fly, and therefore, we will implement the inherited isAbleToFly abstract method as a final method that returns true. This way, we make sure that all the classes that inherit from the VirtualBird abstract class will always run this code for the isAbleToFly method and that they won’t be able to override it. The following UML diagram shows the members for the new abstract and concrete classes that we will code. In addition, the diagram shows the printSoundInWords method moved from the VirtualHorse abstract class to the VirtualDomesticMammal abstract class. First, we will create a new version of the VirtualDomesticMammal abstract class. We will add the printSoundInWords method that we have in the VirtualHorse abstract class and we will use the final keyword to indicate that we don’t want to allow subclasses to override this method. The following lines show the new code for the VirtualDomesticMammal class. public abstract class VirtualDomesticMammal extends VirtualMammal { public final String name; public String favoriteToy; public VirtualDomesticMammal( int age, boolean isPregnant, String name, String favoriteToy) { super(age, isPregnant); this.name = name; this.favoriteToy = favoriteToy; System.out.println("VirtualDomesticMammal created."); } public VirtualDomesticMammal( int age, String name, String favoriteToy) { this(age, false, name, favoriteToy); } protected final void printSoundInWords( String soundInWords, int times, VirtualDomesticMammal otherDomesticMammal, boolean isAngry) { String message = String.format("%s%s: %s%s", name, otherDomesticMammal == null ? "" : String.format(" to %s ", otherDomesticMammal.name), isAngry ? "Angry " : "", new String(new char[times]).replace(" ", soundInWords)); System.out.println(message); } public void talk() { System.out.println( String.format("%s: says something", name)); } } After we enter the previous lines, JShell will display the following messages: | update replaced class VirtualHorse which cannot be referenced until this error is corrected: | printSoundInWords(java.lang.String,int,VirtualDomesticMammal,boolean) in VirtualHorse cannot override printSoundInWords(java.lang.String,int,VirtualDomesticMammal,boolean) in VirtualDomesticMammal | overridden method is final | protected void printSoundInWords(String soundInWords, int times, | ^---------------------------------------------------------------... | update replaced class AmericanQuarterHorse which cannot be referenced until class VirtualHorse is declared | update replaced class ShireHorse which cannot be referenced until class VirtualHorse is declared | update replaced class Thoroughbred which cannot be referenced until class VirtualHorse is declared | update replaced variable american which cannot be referenced until class AmericanQuarterHorse is declared | update replaced variable zelda which cannot be referenced until class ShireHorse is declared | update replaced variable willow which cannot be referenced until class Thoroughbred is declared | update overwrote class VirtualDomesticMammal JShell indicates us that the VirtualHorse class and its subclasses cannot be referenced until we correct an error for this class. The class declares the printSoundInWords method and overrides the recently added method with the same name and arguments in the VirtualDomesticMammal. We used the final keyword in the new declaration to make sure that any subclass cannot override it, and therefore, the Java compiler generates the error message that JShell displays. Now, we will create a new version of the VirtualHorse abstract class. The following lines show the new version that removes the printSoundInWords method and uses the final keyword to make sure that many methods cannot be overridden by any of the subclasses. The declarations that use the final keyword to avoid the methods to be overridden are highlighted in the next lines. public abstract class VirtualHorse extends VirtualDomesticMammal { public VirtualHorse( int age, boolean isPregnant, String name, String favoriteToy) { super(age, isPregnant, name, favoriteToy); System.out.println("VirtualHorse created."); } public VirtualHorse( int age, String name, String favoriteToy) { this(age, false, name, favoriteToy); } public final boolean isAbleToFly() { return false; } public final boolean isRideable() { return true; } public final boolean isHervibore() { return true; } public final boolean isCarnivore() { return false; } public int getAverageNumberOfBabies() { return 1; } public abstract String getBreed(); public final void printBreed() { System.out.println(getBreed()); } public final void printNeigh( int times, VirtualDomesticMammal otherDomesticMammal, boolean isAngry) { printSoundInWords("Neigh ", times, otherDomesticMammal, isAngry); } public final void neigh() { printNeigh(1, null, false); } public final void neigh(int times) { printNeigh(times, null, false); } public final void neigh(int times, VirtualDomesticMammal otherDomesticMammal) { printNeigh(times, otherDomesticMammal, false); } public final void neigh(int times, VirtualDomesticMammal otherDomesticMammal, boolean isAngry) { printNeigh(times, otherDomesticMammal, isAngry); } public final void printNicker(int times, VirtualDomesticMammal otherDomesticMammal, boolean isAngry) { printSoundInWords("Nicker ", times, otherDomesticMammal, isAngry); } public final void nicker() { printNicker(1, null, false); } public final void nicker(int times) { printNicker(times, null, false); } public final void nicker(int times, VirtualDomesticMammal otherDomesticMammal) { printNicker(times, otherDomesticMammal, false); } public final void nicker(int times, VirtualDomesticMammal otherDomesticMammal, boolean isAngry) { printNicker(times, otherDomesticMammal, isAngry); } @Override public final void talk() { nicker(); } } After we enter the previous lines, JShell will display the following messages: | update replaced class AmericanQuarterHorse | update replaced class ShireHorse | update replaced class Thoroughbred | update replaced variable american, reset to null | update replaced variable zelda, reset to null | update replaced variable willow, reset to null | update overwrote class VirtualHorse We could replace the definition for the VirtualHorse class and the subclasses were also updated. It is important to know that the variables we declared in JShell that held references to instances of subclasses of VirtualHorse were set to null. Summary In this article, we created many abstract and concrete classes. We learned to control whether subclasses can or cannot override members, and whether classes can be subclassed. We worked with instances of many subclasses and we understood that objects can take many forms. We worked with many instances and their methods in JShell to understand how the classes and the methods that we coded are executed. We used methods that performed operations with instances of different classes that had a common superclass. Resources for Article: Further resources on this subject: Getting Started with Sorting Algorithms in Java [article]  Introduction to JavaScript [article]  Using Spring JMX within Java Applications [article]

OCS Inventory NG stands for Open Computer and Software Inventory Next Generation , and it is the name of an open source project that was started back in late 2005. The project matured into the first final release in the beginning of year 2007. It's an undertaking that is still actively maintained, fully documented, and has support forums. It has all of the requirements that an open source application should have in order to be competitive. There is a tricky part when it comes to open source solutions. Proposing them and getting them accepted by the management requires quite a bit of research. One side of the coin is that it is always favorable, everyone appreciates cutting down licensing costs. The problem with such a solution is that you cannot always take for granted their future support. In order to take an educated guess on whether an open source solution could be beneficial for the company, we need to look at the following criteria: how frequently is the project updated, check the download count, what is the feedback of the community, whether the application is thoroughly documented, and the existence of active community support. OCS-NG occupies a dominant position when it comes to open source projects on the area of inventorying computers and software. Brief overview on OCS Inventory NG's architecture The architecture of OCS-NG is based on the client-server model. The client program is called a network agent. These agents need to be deployed on the client computers that we want to include in our inventory. The management server is composed of four individual server roles: database server, communication server, deployment server, and the administration console server. More often than not, these can be run from the same machine. OCS Inventory NG is cross-platform and supports most Unices, BSD derivates (including Mac OS X), and all kinds of Windows-based operating systems. The server can be also be run on either platform. As it is an open source project, it's based on the popular LAMP or WAMP solution stack. This means that the main server-side prerequisites are Apache web server, MySQL database server, and PHP server. These are also the viable components of a fully functional web server. The network agents communicate with the management server under standardized HTTP protocols. The data that is exchanged is formatted under XML conventions. The screenshot below describes a general overview on the way clients communicate with the management server's sub-server components. Rough performance evaluation of OCS-NG The data that is collected in case of a fully - inventoried computer sums up to something around 5KB. That is a small amount and it will neither overload the server nor create network congestion. It is often said that around one million systems can be inventoried daily on a 3GHz bi-Xeon processor based server with 4 GB of RAM without any issues. Any modest old-generation server should suffice for the inventory of few thousand systems. When scalability is necessary such as over 10,000-20,000 inventoried systems, it is recommended to split those 4 server-role components on two individual servers. Should this be the case, the database server needs to be installed on the same machine with the communication server, and on another system with the administration server and the deployment server with a database replica. Any other combination is also possible. Although distributing the server components is possible, very rarely do we really need to do that. In this day and age, we can seamlessly virtualize up to four or more servers on any dual or quad-core new generation computer. OCS-NG's management server can be one of those VMs. If necessary, distributing server components in the future is possible. Meeting our inventory demands First and foremost, OCS Inventory NG network agents are able to collect all of the must-have attributes of a client computer and many more. Let's do a quick checkup on these: BIOS: System serial number, manufacturer, and model Bios manufacturer, version, and date Processors: Type, count (how many of them), manufacturer, speed, and cache Memory: Physical memory type, manufacturer, capacity, and slot number Total physical memory Total swap/paging memory Video: Video adapter: Chipset/model, manufacturer, memory size, speed, and screen resolution Display monitor: Manufacturer, description, refresh rate, type, serial number, and caption Storage/removable devices: Manufacturer, model, size, type, speed( all when applicable) Drive letter, filesystem type, partition/volume size, free space Network adapters/telephony: Manufacturer, model, type, speed, and description MAC and IP address, mask and IP gateway, DHCP server used Miscellaneous hardware Input devices: Keyboard, mouse, and pointing device Sound devices: Manufacturer name, type, and description System slots: Name, type, and designation System ports: Type, name, caption, and description Software Information: Operating system: Name, version, comments, and registration info Installed software: Name, publisher, version (from Add / Remove software or Programs and Features menu) Custom-specified registry queries (applicable to Windows OS) Not only computers but also networking components can be used for inventorying. OCS Inventory NG detects and collects network-specific information about these (such as MAC address and IP address, subnet mask, and so on.). Later on we can set labels and organize them appropriately. The place where OCS-NG comes as a surprise is its unique capability to make an inventory of hosts that are not on the network. The network agent can be run manually on these offline hosts and are then imported into the centralized management server. One of its features include intelligent auto-discovering functionalities and its ability to detect hosts that have not been inventoried. It is based on popular network diagnosing and auditing tools such as the nmap . The algorithm can decide whether it's an actual workstation computer or rather just a printer. If it's the former, the agent needs to be deployed. The network scanning is not done by the management server. It is delegated to network agents. This way the network is never overcrowded or congested. If the management server itself scans for populated networks spanning throughout different subnets, the process would be disastrous. This way the process is seamless and simply practical. Another interesting part is the election mechanism based on which the server is able to decide the most suited client to carry out the discovery. A rough sketch of this in action can be seen in the next figure. Set of functions and what it brings to the table At this moment, we're fully aware that the kind information that the network agents are getting into the database are relevant and more than enough for our inventorying needs. Nevertheless, we won't stop here. It's time to analyze and present its web interface. We will also shed a bit of light on the set of features it supports out of the box without any plugins or other mods yet. There will be a time for those too. Taking a glance at the OCS-NG web interface The web interface of OCS Inventory NG is slightly old-fashioned. One direct advantage of this is that the interface is really snappy. Queries are displayed quickly, and the UI won't lag. The other side of the coin is that intuitiveness is not the interface's strongest point. Getting used to it might take a while. At least it does not make you feel that the interface is overcrowded. However, the location and naming of buttons leaves plenty of room for improvement. Some people might prefer to see captions below the shortcuts as the meaning of the icons is not always obvious. After the first few minutes, we will easily get used to them. A picture is worth thousands of words, so let's exemplify our claims. The buttons that appear in the previous screenshot from left to right are the following: All computers Tag/Number of PC repartition Groups All softwares Search with various criteria In the same fashion, in this case the buttons in the previous screenshot stand for the following features: Deployment Security Dictionary Agent Configuration (this one is intuitive!) Registry (self-explanatory) Admin Info Duplicates Users Local Import Help When you click on the name of the specific icon, the drop-down menu appears right below on the cursor All in all, the web interface is not that bad after all. We must accept that the strongestpoint lies in its snappiness, and the wealth of information that is presented in a fraction of a second rather than its design or intuitiveness. We appreciate its overall simplicity and its quick response time. We are often struggling with new generation Java-based and AJAX-based overcrowded interfaces of network equipment that seem slow as hell. So, we'll choose OCS Inventory NG's UI over those anytime!

        Read more about this book       The WiX toolset ships with several User Interface wizards that are ready to use out of the box. We'll briefly discuss each of the available sets and then move on to learning how to create your own from scratch. In this article by Nick Ramirez, author of the book WiX: A Developer's Guide to Windows Installer XML, you'll learn about: Adding dialogs into the InstallUISequence Linking one dialog to another to form a complete wizard Getting basic text and window styling working Including necessary dialogs like those needed to display errors (For more resources on WiX, see here.) WiX standard dialog sets The wizards that come prebuilt with WiX won't fit every need, but they're a good place to get your feet wet. To add any one of them, you first have to add a project reference to WixUIExtension.dll, which can be found in the bin directory of your WiX program files. Adding this reference is sort of like adding a new source file. This one contains dialogs. To use one, you'll need to use a UIRef element to pull the dialog into the scope of your project. For example, this line, anywhere inside the Product element, will add the "Minimal" wizard to your installer: <UIRef Id="WixUI_Minimal" /> It's definitely minimal, containing just one screen. It gives you a license agreement, which you can change by adding a WixVariable element with an Id of WixUILicenseRtf and a Value attribute that points to a Rich Text Format (.rtf) file containing your new license agreement: <WixVariable Id="WixUILicenseRtf" Value="newLicense.rtf" /> You can also override the background image (red wheel on the left, white box on the right) by setting another WixVariable called WixUIDialogBmp to a new image. The dimensions used are 493x312. The other available wizards offer more and we'll cover them in the following sections. WixUI_Advanced The "Advanced" dialog set offers more: It has a screen that lets the user choose to install for just the current user or for all users, another where the end user can change the folder that files are installed to and a screen with a feature tree where features can be turned on or off. As in the following screenshot: You'll need to change your UIRef element to use WixUI_Advanced. This can be done by adding the following line: <UIRef Id="WixUI_Advanced" /> You'll also have to make sure that your install directory has an Id of APPLICATIONFOLDER, as in this example: <Directory Id="TARGETDIR" Name="SourceDir"> <Directory Id="ProgramFilesFolder"> <Directory Id="APPLICATIONFOLDER" Name="My Program" /> </Directory></Directory> Next, set two properties: ApplicationFolderName and WixAppFolder. The first sets the name of the install directory as it will be displayed in the UI. The second sets whether this install should default to being per user or per machine. It can be either WixPerMachineFolder or WixPerUserFolder. <Property Id="ApplicationFolderName" Value="My Program" /><Property Id="WixAppFolder" Value="WixPerMachineFolder" /> This dialog uses a bitmap that the Minimal installer doesn't: the white banner at the top. You can replace it with your own image by setting the WixUIBannerBmp variable. Its dimensions are 493x58. It would look something like this: <WixVariable Id="WixUIBannerBmp" Value="myBanner.bmp" /> WixUI_FeatureTree The WixUI_FeatureTree wizard shows a feature tree like the Advanced wizard, but it doesn't have a dialog that lets the user change the install path. To use it, you only need to set the UIRef to WixUI_FeatureTree, like so: <UIRef Id="WixUI_FeatureTree" /> This would produce a window that would allow you to choose features as show in the following screenshot: Notice that in the image, the Browse button is disabled. If any of your Feature elements have the ConfigurableDirectory attribute set to the Id of a Directory element, then this button will allow you to change where that feature is installed to. The Directory element's Id must be all uppercase. WixUI_InstallDir WixUI_InstallDir shows a dialog where the user can change the installation path. Change the UIRef to WixUI_InstallDir. Like so: <UIRef Id="WixUI_InstallDir" /> Here, the user can chose the installation path. This is seen in the following screenshot: You'll have to set a property called WIXUI_INSTALLDIR to the Id you gave your install directory. So, if your directory structure used INSTALLLDIR for the Id of the main install folder, use that as the value of the property. <Directory Id="TARGETDIR" Name="SourceDir"> <Directory Id="ProgramFilesFolder"> <Directory Id="INSTALLDIR" Name="My Program" /> </Directory></Directory> <Property Id="WIXUI_INSTALLDIR" Value="INSTALLDIR" /> WixUI_Mondo The WixUI_Mondo wizard gives the user the option of installing a "Typical", "Complete" or "Custom" install. Typical sets the INSTALLLEVEL property to 3 while Complete sets it to 1000. You can set the Level attribute of your Feature elements accordingly to include them in one group or the other. Selecting a Custom install will display a feature tree dialog where the user can choose exactly what they want. To use this wizard, change your UIRef element to WixUI_Mondo. <UIRef Id="WixUI_Mondo" /> This would result in a window like the following:

This article by Nathan Kozyra, author of Mastering Concurrency in Go, tackles one of the Internet's most famous and esteemed challenges and attempt to solve it with core Go packages. (For more resources related to this topic, see here.) We've built a few usable applications; things we can start with and leapfrog into real systems for everyday use. By doing so, we've been able to demonstrate the basic and intermediate-level patterns involved in Go's concurrent syntax and methodology. However, it's about time we take on a real-world problem—one that has vexed developers (and their managers and VPs) for a great deal of the early history of the Web. In addressing and, hopefully, solving this problem, we'll be able to develop a high-performance web server that can handle a very large volume of live, active traffic. For many years, the solution to this problem was solely to throw hardware or intrusive caching systems at the problem; so, alternately, solving it with programming methodology should excite any programmer. We'll be using every technique and language construct we've learned so far, but we'll do so in a more structured and deliberate way than we have up to now. Everything we've explored so far will come into play, including the following points: Creating a visual representation of our concurrent application Utilizing goroutines to handle requests in a way that will scale Building robust channels to manage communication between goroutines and the loop that will manage them Profiling and benchmarking tools (JMeter, ab) to examine the way our event loop actually works Timeouts and concurrency controls—when necessary—to ensure data and request consistency Attacking the C10K problem The genesis of the C10K problem is rooted in serial, blocking programming, which makes it ideal to demonstrate the strength of concurrent programming, especially in Go. When he asked this in 1999, for many server admins and engineers, serving 10,000 concurrent visitors was something that would be solved with hardware. The notion that a single server on common hardware could handle this type of CPU and network bandwidth without falling over seemed foreign to most. The crux of his proposed solutions relied on producing non-blocking code. Of course, in 1999, concurrency patterns and libraries were not widespread. C++ had some polling and queuing options available via some third-party libraries and the earliest predecessor to multithreaded syntaxes, later available through Boost and then C++11. Over the coming years, solutions to the problem began pouring in across various flavors of languages, programming design, and general approaches. Any performance and scalability problem will ultimately be bound to the underlying hardware, so as always, your mileage may vary. Squeezing 10,000 concurrent connections on a 486 processor with 500 MB of RAM will certainly be more challenging than doing so on a barebones Linux server stacked with memory and multiple cores. It's also worth noting that a simple echo server would obviously be able to assume more cores than a functional web server that returns larger amounts of data and accepts greater complexity in requests, sessions, and so on, as we'll be dealing with here. Failing of servers at 10,000 concurrent connections When the Web was born and the Internet commercialized, the level of interactivity was pretty minimal. If you're a graybeard, you may recall the transition from NNTP/IRC and the like and how extraordinarily rudimentary the Web was. To address the basic proposition of [page request] → [HTTP response], the requirements on a web server in the early 1990s were pretty lenient. Ignoring all of the error responses, header reading, and settings, and other essential (but unrelated to the in → out mechanism) functions, the essence of the early servers was shockingly simple, at least compared to the modern web servers. The first web server was developed by the father of the Web, Tim Berners-Lee. Developed at CERN (such as WWW/HTTP itself), CERN httpd handled many of the things you would expect in a web server today—hunting through the code, you'll find a lot of notation that will remind you that the very core of the HTTP protocol is largely unchanged. Unlike most technologies, HTTP has had an extraordinarily long shelf life. Written in C in 1990, it was unable to utilize a lot of concurrency strategies available in languages such as Erlang. Frankly, doing so was probably unnecessary—the majority of web traffic was a matter of basic file retrieval and protocol. The meat and potatoes of a web server were not dealing with traffic, but rather dealing with the rules surrounding the protocol itself. You can still access the original CERN httpd site and download the source code for yourself from http://www.w3.org/Daemon/. I highly recommend that you do so as both a history lesson and a way to look at the way the earliest web server addressed some of the earliest problems. However, the Web in 1990 and the Web when the C10K question was first posed were two very different environments. By 1999, most sites had some level of secondary or tertiary latency provided by third-party software, CGI, databases, and so on, all of which further complicated the matter. The notion of serving 10,000 flat files concurrently is a challenge in itself, but try doing so by running them on top of a Perl script that accesses a MySQL database without any caching layer; this challenge is immediately exacerbated. By the mid 1990s, the Apache web server had taken hold and largely controlled the market (by 2009, it had become the first server software to serve more than 100 million websites). Apache's approach was rooted heavily in the earliest days of the Internet. At its launch, connections were initially handled first in, first out. Soon, each connection was assigned a thread from the thread pool. There are two problems with the Apache server. They are as follows: Blocking connections can lead to a domino effect, wherein one or more slowly resolved connections could avalanche into inaccessibility Apache had hard limits on the number of threads/workers you could utilize, irrespective of hardware constraints It's easy to see the opportunity here, at least in retrospect. A concurrent server that utilizes actors (Erlang), agents (Clojure), or goroutines (Go) seems to fit the bill perfectly. Concurrency does not solve the C10k problem in itself, but it absolutely provides a methodology to facilitate it. The most notable and visible example of an approach to the C10K problem today is Nginx, which was developed using concurrency patterns, widely available in C by 2002 to address—and ultimately solve—the C10k problem. Nginx, today, represents either the #2 or #3 web server in the world, depending on the source. Using concurrency to attack C10K There are two primary approaches to handle a large volume of concurrent requests. The first involves allocating threads per connection. This is what Apache (and a few others) do. On the one hand, allocating a thread to a connection makes a lot of sense—it's isolated, controllable via the application's and kernel's context switching, and can scale with increased hardware. One problem for Linux servers—on which the majority of the Web lives—is that each allocated thread reserves 8 MB of memory for its stack by default. This can (and should) be redefined, but this imposes a largely unattainable amount of memory required for a single server. Even if you set the default stack size to 1 MB, we're dealing with a minimum of 10 GB of memory just to handle the overhead. This is an extreme example that's unlikely to be a real issue for a couple of reasons: first, because you can dictate the maximum amount of resources available to each thread, and second, because you can just as easily load balance across a few servers and instances rather than add 10 GB to 80 GB of RAM. Even in a threaded server environment, we're fundamentally bound to the issue that can lead to performance decreases (to the point of a crash). First, let's look at a server with connections bound to threads (as shown in the following diagram), and visualize how this can lead to logjams and, eventually, crashes: This is obviously what we want to avoid. Any I/O, network, or external process that can impose some slowdown can bring about that avalanche effect we talked about, such that our available threads are taken (or backlogged) and incoming requests begin to stack up. We can spawn more threads in this model, but as mentioned earlier, there are potential risks there too, and even this will fail to mitigate the underlying problem. Taking another approach In an attempt to create our web server that can handle 10,000 concurrent connections, we'll obviously leverage our goroutine/channel mechanism to put an event loop in front of our content delivery to keep new channels recycled or created constantly. For this example, we'll assume we're building a corporate website and infrastructure for a rapidly expanding company. To do this, we'll need to be able to serve both static and dynamic content. The reason we want to introduce dynamic content is not just for the purposes of demonstration—we want to challenge ourselves to show 10,000 true concurrent connections even when a secondary process gets in the way. As always, we'll attempt to map our concurrency strategy directly to goroutines and channels. In a lot of other languages and applications, this is directly analogous to an event loop, and we'll approach it as such. Within our loop, we'll manage the available goroutines, expire or reuse completed ones, and spawn new ones where necessary. In this example visualization, we show how an event loop (and corresponding goroutines) can allow us to scale our connections without employing too many hard resources such as CPU threads or RAM: The most important step for us here is to manage that event loop. We'll want to create an open, infinite loop to manage the creation and expiration of our goroutines and respective channels. As part of this, we will also want to do some internal logging of what's happening, both for benchmarking and debugging our application. Building our C10K web server Our web server will be responsible for taking requests, routing them, and serving either flat files or dynamic files with templates parsed against a few different data sources. As mentioned earlier, if we exclusively serve flat files and remove much of the processing and network latency, we'd have a much easier time with handling 10,000 concurrent connections. Our goal is to approach as much of a real-world scenario as we can—very little of the Web operates on a single server in a static fashion. Most websites and applications utilize databases, CDNs (Content Delivery Networks), dynamic and uncached template parsing, and so on. We need to replicate them whenever possible. For the sake of simplicity, we'll separate our content by type and filter them through URL routing, as follows: /static/[request]: This will serve request.html directly /template/[request]: This will serve request.tpl after its been parsed through Go /dynamic/[request][number]: This will also serve request.tpl and parse it against a database source's record By doing this, we should get a better mixture of possible HTTP request types that could impede the ability to serve large numbers of users simultaneously, especially in a blocking web server environment. You can find Go's exceptional library to generate safe data-driven templating at http://golang.org/pkg/html/template/. By safe, we're largely referring to the ability to accept data and move it directly into templates without worrying about the sort of injection issues that are behind a large amount of malware and cross-site scripting. For the database source, we'll use MySQL here, but feel free to experiment with other databases if you're more comfortable with them. Like the html/template package, we're not going to put a lot of time into outlining MySQL and/or its variants. Benchmarking against a blocking web server It's only fair to add some starting benchmarks against a blocking web server first so that we can measure the effect of concurrent versus nonconcurrent architecture. For our starting benchmarks, we'll eschew any framework, and we'll go with our old stalwart, Apache. For the sake of completeness here, we'll be using an Intel i5 3GHz machine with 8 GB of RAM. While we'll benchmark our final product on Ubuntu, Windows, and OS X here, we'll focus on Ubuntu for our example. Our localhost domain will have three plain HTML files in /static, each trimmed to 80 KB. As we're not using a framework, we don't need to worry about raw dynamic requests, but only about static and dynamic requests in addition to data source requests. For all examples, we'll use a MySQL database (named master) with a table called articles that will contain 10,000 duplicate entries. Our structure is as follows: CREATE TABLE articles ( article_id INT NOT NULL AUTO_INCREMENT, article_title VARCHAR(128) NOT NULL, article_text VARCHAR(128) NOT NULL, PRIMARY KEY (article_id) ) With ID indexes ranging sequentially from 0-10,000, we'll be able to generate random number requests, but for now, we just want to see what kind of basic response we can get out of Apache serving static pages with this machine. For this test, we'll use Apache's ab tool and then gnuplot to sequentially map the request time as the number of concurrent requests and pages; we'll do this for our final product as well, but we'll also go through a few other benchmarking tools for it to get some better details.   Apache's AB comes with the Apache web server itself. You can read more about it at http://httpd.apache.org/docs/2.2/programs/ab.html. You can download it for Linux, Windows, OS X, and more from http://httpd.apache.org/download.cgi. The gnuplot utility is available for the same operating systems at http://www.gnuplot.info/. So, let's see how we did it. Have a look at the following graph: Ouch! Not even close. There are things we can do to tune the connections available (and respective threads/workers) within Apache, but this is not really our goal. Mostly, we want to know what happens with an out-of-the-box Apache server. In these benchmarks, we start to drop or refuse connections at around 800 concurrent connections. More troubling is that as these requests start stacking up, we see some that exceed 20 seconds or more. When this happens in a blocking server, each request behind it is queued; requests behind that are similarly queued and the entire thing starts to fall apart. Even if we cannot hit 10,000 concurrent connections, there's a lot of room for improvement. While a single server of any capacity is no longer the way we expect a web server environment to be designed, being able to squeeze as much performance as possible out of that server, ostensibly with our concurrent, event-driven approach, should be our goal. Handling requests The Gorilla toolkit certainly makes this easier, but we should also know how to intercept the functionality to impose our own custom handler. Here is a simple web router wherein we handle and direct requests using a custom http.Server struct, as shown in the following code: var routes []string type customRouter struct { } func (customRouter) ServeHTTP(rw http.ResponseWriter, r *http.Request) { fmt.Println(r.URL.Path); } func main() { var cr customRouter; server := &http.Server { Addr: ":9000", Handler:cr, ReadTimeout: 10 * time.Second, WriteTimeout: 10 * time.Second, MaxHeaderBytes: 1 << 20, } server.ListenAndServe() } Here, instead of using a built-in URL routing muxer and dispatcher, we're creating a custom server and custom handler type to accept URLs and route requests. This allows us to be a little more robust with our URL handling. In this case, we created a basic, empty struct called customRouter and passed it to our custom server creation call. We can add more elements to our customRouter type, but we really don't need to for this simple example. All we need to do is to be able to access the URLs and pass them along to a handler function. We'll have three: one for static content, one for dynamic content, and one for dynamic content from a database. Before we go so far though, we should probably see what our absolute barebones, HTTP server written in Go, does when presented with the same traffic that we sent Apache's way. By old school, we mean that the server will simply accept a request and pass along a static, flat file. You could do this using a custom router as we did earlier, taking requests, opening files, and then serving them, but Go provides a much simpler mode to handle this basic task in the http.FileServer method. So, to get some benchmarks for the most basic of Go servers against Apache, we'll utilize a simple FileServer and test it against a test.html page (which contains the same 80 KB file that we had with Apache). As our goal with this test is to improve our performance in serving flat and dynamic pages, the actual specs for the test suite are somewhat immaterial. We'd expect that while the metrics will not match from environment to environment, we should see a similar trajectory. That said, it's only fair we supply the environment used for these tests; in this case, we used a MacBook Air with a 1.4 GHz i5 processor and 4 GB of memory. First, we'll do this with our absolute best performance out of the box with Apache, which had 850 concurrent connections and 900 total requests. The results are certainly encouraging as compared to Apache. Neither of our test systems were tweaked much (Apache as installed and basic FileServer in Go), but Go's FileServer handles 1,000 concurrent connections without so much as a blip, with the slowest clocking in at 411 ms. Apache has made a great number of strides pertaining to concurrency and performance options in the last five years, but to get there does require a bit of tuning and testing. The intent of this experiment is not intended to denigrate Apache, which is well tested and established. Instead, it's to compare the out-of-the-box performance of the world's number 1 web server against what we can do with Go. To really get a baseline of what we can achieve in Go, let's see if Go's FileServer can hit 10,000 connections on a single, modest machine out of the box: ab -n 10500 -c 10000 -g test.csv http://localhost:8080/a.html We will get the following output: Success! Go's FileServer by itself will easily handle 10,000 concurrent connections, serving flat, static content. Of course, this is not the goal of this particular project—we'll be implementing real-world obstacles such as template parsing and database access, but this alone should show you the kind of starting point that Go provides for anyone who needs a responsive server that can handle a large quantity of basic web traffic. Routing requests So, let's take a step back and look again at routing our traffic through a traditional web server to include not only our static content, but also the dynamic content. We'll want to create three functions that will route traffic from our customRouter:serveStatic():: read function and serve a flat file serveRendered():, parse a template to display serveDynamic():, connect to MySQL, apply data to a struct, and parse a template. To take our requests and reroute, we'll change the ServeHTTP method for our customRouter struct to handle three regular expressions. For the sake of brevity and clarity, we'll only be returning data on our three possible requests. Anything else will be ignored. In a real-world scenario, we can take this approach to aggressively and proactively reject connections for requests we think are invalid. This would include spiders and nefarious bots and processes, which offer no real value as nonusers.

Working with generics Visual Studio 2005 included .NET version 2.0 which included generics. Generics give developers the ability to design classes and methods that defer the specification of specific parts of a class or method's specification until declaration or instantiation. Generics offer features previously unavailable in .NET. One benefit to generics, that is potentially the most common, is for the implementation of collections that provide a consistent interface to collections of different data types without needing to write specific code for each data type. Constraints can be used to restrict the types that are supported by a generic method or class, or can guarantee specific interfaces. Limits of generics Constraints within generics in C# are currently limited to a parameter-less constructor, interfaces, or base classes, or whether or not the type is a struct or a class (value or reference type). This really means that code within a generic method or type can either be constructed or can make use of methods and properties. Due to these restrictions types within generic types or methods cannot have operators. Writing sequence and iterator members Visual Studio 2005 and C# 2.0 introduced the yield keyword. The yield keyword is used within an iterator member as a means to effectively implement an IEnumerable interface without needing to implement the entire IEnumerable interface. Iterator members are members that return a type of IEnumerable or IEnumerable<T>, and return individual elements in the enumerable via yield return, or deterministically terminates the enumerable via yield break. These members can be anything that can return a value, such as methods, properties, or operators. An iterator that returns without calling yield break has an implied yield break, just as a void method has an implied return. Iterators operate on a sequence but process and return each element as it is requested. This means that iterators implement what is known as deferred execution. Deferred execution is when some or all of the code, although reached in terms of where the instruction pointer is in relation to the code, hasn't entirely been executed yet. Iterators are methods that can be executed more than once and result in a different execution path for each execution. Let's look at an example: public static IEnumerable<DateTime> Iterator() { Thread.Sleep(1000); yield return DateTime.Now; Thread.Sleep(1000); yield return DateTime.Now; Thread.Sleep(1000); yield return DateTime.Now; }   The Iterator method returns IEnumerable which results in three DateTime values. The creation of those three DateTime values is actually invoked at different times. The Iterator method is actually compiled in such a way that a state machine is created under the covers to keep track of how many times the code is invoked and is implemented as a special IEnumerable<DateTime> object. The actual invocation of the code in the method is done through each call of the resulting IEnumerator. MoveNext method. The resulting IEnumerable is really implemented as a collection of delegates that are executed upon each invocation of the MoveNext method, where the state, in the simplest case, is really which of the delegates to invoke next. It's actually more complicated than that, especially when there are local variables and state that can change between invocations and is used across invocations. But the compiler takes care of all that. Effectively, iterators are broken up into individual bits of code between yield return statements that are executed independently, each using potentially local shared data. What are iterators good for other than a really cool interview question? Well, first of all, due to the deferred execution, we can technically create sequences that don't need to be stored in memory all at one time. This is often useful when we want to project one sequence into another. Couple that with a source sequence that is also implemented with deferred execution, we end up creating and processing IEnumerables (also known as collections) whose content is never all in memory at the same time. We can process large (or even infinite) collections without a huge strain on memory. For example, if we wanted to model the set of positive integer values (an infinite set) we could write an iterator method shown as follows: static IEnumerable<BigInteger> AllThePositiveIntegers() { var number = new BigInteger(0); while (true) yield return number++; }   We can then chain this iterator with another iterator, say something that gets all of the positive squares: static IEnumerable<BigInteger> AllThePostivieIntegerSquares( IEnumerable<BigInteger> sourceIntegers) { foreach(var value in sourceIntegers) yield return value*value; }   Which we could use as follows: foreach(var value in AllThePostivieIntegerSquares(AllThePositiveIntegers())) Console.WriteLine(value);   We've now effectively modeled two infi nite collections of integers in memory. Of course, our AllThePostiveIntegerSquares method could just as easily be used with fi nite sequences of values, for example: foreach (var value in AllThePostivieIntegerSquares( Enumerable.Range(0, int.MaxValue) .Select(v => new BigInteger(v)))) Console.WriteLine(value);   In this example we go through all of the positive Int32 values and square each one without ever holding a complete collection of the set of values in memory. As we see, this is a useful method for composing multiple steps that operate on, and result in, sequences of values. We could have easily done this without IEnumerable<T>, or created an IEnumerator class whose MoveNext method performed calculations instead of navigating an array. However, this would be tedious and is likely to be error-prone. In the case of not using IEnumerable<T>, we'd be unable to operate on the data as a collection with things such as foreach. Context: When modeling a sequence of values that is either known only at runtime, or each element can be reliably calculated at runtime. Practice: Consider using an iterator. Working with lambdas Visual Studio 2008 introduced C# 3.0 . In this version of C# lambda expressions were introduced. Lambda expressions are another form of anonymous functions. Lambdas were added to the language syntax primarily as an easier anonymous function syntax for LINQ queries. Although you can't really think of LINQ without lambda expressions, lambda expressions are a powerful aspect of the C# language in their own right. They are concise expressions that use implicitly-typed optional input parameters whose types are implied through the context of their use, rather than explicit de fi nition as with anonymous methods. Along with C# 3.0 in Visual Studio 2008, the .NET Framework 3.5 was introduced which included many new types to support LINQ expressions, such as Action<T> and Func<T>. These delegates are used primarily as definitions for different types of anonymous methods (including lambda expressions). The following is an example of passing a lambda expression to a method that takes a Func<T1, T2, TResult> delegate and the two arguments to pass along to the delegate: ExecuteFunc((f, s) => f + s, 1, 2);   The same statement with anonymous methods: ExecuteFunc(delegate(int f, int s) { return f + s; }, 1, 2);   It's clear that the lambda syntax has a tendency to be much more concise, replacing the delegate and braces with the "goes to" operator (=>). Prior to anonymous functions, member methods would need to be created to pass as delegates to methods. For example: ExecuteFunc(SomeMethod, 1, 2);   This, presumably, would use a method named SomeMethod that looked similar to: private static int SomeMethod(int first, int second) { return first + second; }   Lambda expressions are more powerful in the type inference abilities, as we've seen from our examples so far. We need to explicitly type the parameters within anonymous methods, which is only optional for parameters in lambda expressions. LINQ statements don't use lambda expressions exactly in their syntax. The lambda expressions are somewhat implicit. For example, if we wanted to create a new collection of integers from another collection of integers, with each value incremented by one, we could use the following LINQ statement: var x = from i in arr select i + 1;   The i + 1 expression isn't really a lambda expression, but it gets processed as if it were first converted to method syntax using a lambda expression: var x = arr.Select(i => i + 1);   The same with an anonymous method would be: var x = arr.Select(delegate(int i) { return i + 1; });   What we see in the LINQ statement is much closer to a lambda expression. Using lambda expressions for all anonymous functions means that you have more consistent looking code. Context: When using anonymous functions. Practice: Prefer lambda expressions over anonymous methods. Parameters to lambda expressions can be enclosed in parentheses. For example: var x = arr.Select((i) => i + 1);   The parentheses are only mandatory when there is more than one parameter: var total = arr.Aggregate(0, (l, r) => l + r);   Context: When writing lambdas with a single parameter. Practice: Prefer no parenthesis around the parameter declaration. Sometimes when using lambda expressions, the expression is being used as a delegate that takes an argument. The corresponding parameter in the lambda expression may not be used within the right-hand expression (or statements). In these cases, to reduce the clutter in the statement, it's common to use the underscore character (_) for the name of the parameter. For example: task.ContinueWith(_ => ProcessSecondHalfOfData());   The task.ContinueWith method takes an Action <Task> delegate. This means the previous lambda expression is actually given a task instance (the antecedent Task). In our example, we don't use that task and just perform some completely independent operation. In this case, we use (_) to not only signify that we know we don't use that parameter, but also to reduce the clutter and potential name collisions a little bit. Context: When writing lambda expression that take a single parameter but the parameter is not used. Practice: Use underscore (_) for the name of the parameter. There are two types of lambda expressions. So far, we've seen expression lambdas. Expression lambdas are a single expression on the right-hand side that evaluates to a value or void. There is another type of lambda expression called statement lambdas. These lambdas have one or more statements and are enclosed in braces. For example: task.ContinueWith(_ => { var value = 10; value += ProcessSecondHalfOfData(); ProcessSomeRandomValue(value); });   As we can see, statement lambdas can declare variables, as well as have multiple statements. Working with extension methods Along with lambda expressions and iterators, C# 3.0 brought us extension methods. These static methods (contained in a static class whose first argument is modified with the this modifier) were created for LINQ so IEnumerable types could be queried without needing to add copious amounts of methods to the IEnumerable interface. An extension method has the basic form of: public static class EnumerableExtensions { public static IEnumerable<int> IntegerSquares( this IEnumerable<int> source) { return source.Select(value => value * value); } }   As stated earlier, extension methods must be within a static class, be a static method, and the first parameter must be modified with the this modifier. Extension methods extend the available instance methods of a type. In our previous example, we've effectively added an instance member to IEnumerable<int> named IntegerSquares so we get a sequence of integer values that have been squared. For example, if we created an array of integer values, we will have added a Cubes method to that array that returns a sequence of the values cubed. For example: var values = new int[] {1, 2, 3}; foreach (var v in values.Cubes()) { Console.WriteLine(v); }   Having the ability to create new instance methods that operate on any public members of a specific type is a very powerful feature of the language. This, unfortunately, does not come without some caveats. Extension methods suffer inherently from a scoping problem. The only scoping that can occur with these methods is the namespaces that have been referenced for any given C# source file. For example, we could have two static classes that have two extension methods named Cubes. If those static classes are in the same namespace, we'd never be able to use those extensions methods as extension methods because the compiler would never be able to resolve which one to use. For example: public static class IntegerEnumerableExtensions { public static IEnumerable<int> Squares( this IEnumerable<int> source) { return source.Select(value => value * value); } public static IEnumerable<int> Cubes( this IEnumerable<int> source) { return source.Select(value => value * value * value); } } public static class EnumerableExtensions { public static IEnumerable<int> Cubes( this IEnumerable<int> source) { return source.Select(value => value * value * value); } }   If we tried to use Cubes as an extension method, we'd get a compile error, for example: var values = new int[] {1, 2, 3}; foreach (var v in values.Cubes()) { Console.WriteLine(v); }   This would result in error CS0121: The call is ambiguous between the following methods or properties. To resolve the problem, we'd need to move one (or both) of the classes to another namespace, for example: namespace Integers { public static class IntegerEnumerableExtensions { public static IEnumerable<int> Squares( this IEnumerable<int> source) { return source.Select(value => value*value); } public static IEnumerable<int> Cubes( this IEnumerable<int> source) { return source.Select(value => value*value*value); } } } namespace Numerical { public static class EnumerableExtensions { public static IEnumerable<int> Cubes( this IEnumerable<int> source) { return source.Select(value => value*value*value); } } }   Then, we can scope to a particular namespace to choose which Cubes to use: Context: When considering extension methods, due to potential scoping problems. Practice: Use extension methods sparingly. Context: When designing extension methods. Practice: Keep all extension methods that operate on a specific type in their own class. Context: When designing classes to contain methods to extend a specific type, TypeName. Practice: Consider naming the static class TypeNameExtensions. Context: When designing classes to contain methods to extend a specific type, in order to scope the extension methods. Practice: Consider placing the class in its own namespace. Generally, there isn't much need to use extension methods on types that you own. You can simply add an instance method to contain the logic that you want to have. Where extension methods really shine is for effectively creating instance methods on interfaces. Typically, when code is necessary for shared implementations of interfaces, an abstract base class is created so each implementation of the interface can derive from it to implement these shared methods. This is a bit cumbersome in that it uses the one-and-only inheritance slot in C#, so an interface implementation would not be able to derive or extend any other classes. Additionally, there's no guarantee that a given interface implementation will derive from the abstract base and runs the risk of not being able to be used in the way it was designed. Extension methods get around this problem by being entirely independent from the implementation of an interface while still being able to extend it. One of the most notable examples of this might be the System.Linq.Enumerable class introduced in .NET 3.5. The static Enumerable class almost entirely consists of extension methods that extend IEnumerable. It is easy to develop the same sort of thing for our own interfaces. For example, say we have an ICoordinate interface to model a three-dimensional position in relation to the Earth's surface: namespace ConsoleApplication { using Numerical; internal class Program { private static void Main(string[] args) { var values = new int[] {1, 2, 3}; foreach (var v in values.Cubes()) { Console.WriteLine(v); } } } }   We could create a static class to contain extension methods to provide shared functionality between any implementation of ICoordinate. For example: public interface ICoordinate { /// <summary>North/south degrees from equator.</summary> double Latitude { get; set; } /// <summary>East/west degrees from meridian.</summary> double Longitude { get; set; } /// <summary>Distance from sea level in meters.</summary> double Altitude { get; set; } }   Context: When designing interfaces that require shared code. Practice: Consider providing extension methods instead of abstract base implementations.

In this article by Arun Chinnachamy, the author of Redis Applied Design Patterns, we are going to see how to use Redis to build a basic autocomplete or autosuggest server. Also, we will see how to build a faceting engine using Redis. To build such a system, we will use sorted sets and operations involving ranges and intersections. To summarize, we will focus on the following topics in this article: (For more resources related to this topic, see here.) Autocompletion for words Multiword autosuggestion using a sorted set Faceted search using sets and operations such as union and intersection Autosuggest systems These days autosuggest is seen in virtually all e-commerce stores in addition to a host of others. Almost all websites are utilizing this functionality in one way or another from a basic website search to programming IDEs. The ease of use afforded by autosuggest has led every major website from Google and Amazon to Wikipedia to use this feature to make it easier for users to navigate to where they want to go. The primary metric for any autosuggest system is how fast we can respond with suggestions to a user's query. Usability research studies have found that the response time should be under a second to ensure that a user's attention and flow of thought are preserved. Redis is ideally suited for this task as it is one of the fastest data stores in the market right now. Let's see how to design such a structure and use Redis to build an autosuggest engine. We can tweak Redis to suit individual use case scenarios, ranging from the simple to the complex. For instance, if we want only to autocomplete a word, we can enable this functionality by using a sorted set. Let's see how to perform single word completion and then we will move on to more complex scenarios, such as phrase completion. Word completion in Redis In this section, we want to provide a simple word completion feature through Redis. We will use a sorted set for this exercise. The reason behind using a sorted set is that it always guarantees O(log(N)) operations. While it is commonly known that in a sorted set, elements are arranged based on the score, what is not widely acknowledged is that elements with the same scores are arranged lexicographically. This is going to form the basis for our word completion feature. Let's look at a scenario in which we have the words to autocomplete: jack, smith, scott, jacob, and jackeline. In order to complete a word, we need to use n-gram. Every word needs to be written as a contiguous sequence. n-gram is a contiguous sequence of n items from a given sequence of text or speech. To find out more, check http://en.wikipedia.org/wiki/N-gram. For example, n-gram of jack is as follows: j ja jac jack$ In order to signify the completed word, we can use a delimiter such as * or $. To add the word into a sorted set, we will be using ZADD in the following way: > zadd autocomplete 0 j > zadd autocomplete 0 ja > zadd autocomplete 0 jac > zadd autocomplete 0 jack$ Likewise, we need to add all the words we want to index for autocompletion. Once we are done, our sorted set will look as follows: > zrange autocomplete 0 -1 1) "j" 2) "ja" 3) "jac" 4) "jack$" 5) "jacke" 6) "jackel" 7) "jackeli" 8) "jackelin" 9) "jackeline$" 10) "jaco" 11) "jacob$" 12) "s" 13) "sc" 14) "sco" 15) "scot" 16) "scott$" 17) "sm" 18) "smi" 19) "smit" 20) "smith$" Now, we will use ZRANK and ZRANGE operations over the sorted set to achieve our desired functionality. To autocomplete for ja, we have to execute the following commands: > zrank autocomplete jac 2 zrange autocomplete 3 50 1) "jack$" 2) "jacke" 3) "jackel" 4) "jackeli" 5) "jackelin" 6) "jackeline$" 7) "jaco" 8) "jacob$" 9) "s" 10) "sc" 11) "sco" 12) "scot" 13) "scott$" 14) "sm" 15) "smi" 16) "smit" 17) "smith$" Another example on completing smi is as follows: zrank autocomplete smi 17 zrange autocomplete 18 50 1) "smit" 2) "smith$" Now, in our program, we have to do the following tasks: Iterate through the results set. Check if the word starts with the query and only use the words with $ as the last character. Though it looks like a lot of operations are performed, both ZRANGE and ZRANK are O(log(N)) operations. Therefore, there should be virtually no problem in handling a huge list of words. When it comes to memory usage, we will have n+1 elements for every word, where n is the number of characters in the word. For M words, we will have M(avg(n) + 1) records where avg(n) is the average characters in a word. The more the collision of characters in our universe, the less the memory usage. In order to conserve memory, we can use the EXPIRE command to expire unused long tail autocomplete terms. Multiword phrase completion In the previous section, we have seen how to use the autocomplete for a single word. However, in most real world scenarios, we will have to deal with multiword phrases. This is much more difficult to achieve as there are a few inherent challenges involved: Suggesting a phrase for all matching words. For instance, the same manufacturer has a lot of models available. We have to ensure that we list all models if a user decides to search for a manufacturer by name. Order the results based on overall popularity and relevance of the match instead of ordering lexicographically. The following screenshot shows the typical autosuggest box, which you find in popular e-commerce portals. This feature improves the user experience and also reduces the spell errors: For this case, we will use a sorted set along with hashes. We will use a sorted set to store the n-gram of the indexed data followed by getting the complete title from hashes. Instead of storing the n-grams into the same sorted set, we will store them in different sorted sets. Let's look at the following scenario in which we have model names of mobile phones along with their popularity: For this set, we will create multiple sorted sets. Let's take Apple iPhone 5S: ZADD a 9 apple_iphone_5s ZADD ap 9 apple_iphone_5s ZADD app 9 apple_iphone_5s ZADD apple 9 apple_iphone_5s ZADD i 9 apple_iphone_5s ZADD ip 9 apple_iphone_5s ZADD iph 9 apple_iphone_5s ZADD ipho 9 apple_iphone_5s ZADD iphon 9 apple_iphone_5s ZADD iphone 9 apple_iphone_5s ZADD 5 9 apple_iphone_5s ZADD 5s 9 apple_iphone_5s HSET titles apple_iphone_5s "Apple iPhone 5S" In the preceding scenario, we have added every n-gram value as a sorted set and created a hash that holds the original title. Likewise, we have to add all the titles into our index. Searching in the index Now that we have indexed the titles, we are ready to perform a search. Consider a situation where a user is querying with the term apple. We want to show the user the five best suggestions based on the popularity of the product. Here's how we can achieve this: > zrevrange apple 0 4 withscores 1) "apple_iphone_5s" 2) 9.0 3) "apple_iphone_5c" 4) 6.0 As the elements inside the sorted set are ordered by the element score, we get the matches ordered by the popularity which we inserted. To get the original title, type the following command: > hmget titles apple_iphone_5s 1) "Apple iPhone 5S" In the preceding scenario case, the query was a single word. Now imagine if the user has multiple words such as Samsung nex, and we have to suggest the autocomplete as Samsung Galaxy Nexus. To achieve this, we will use ZINTERSTORE as follows: > zinterstore samsung_nex 2 samsung nex aggregate max ZINTERSTORE destination numkeys key [key ...] [WEIGHTS weight [weight ...]] [AGGREGATE SUM|MIN|MAX] This computes the intersection of sorted sets given by the specified keys and stores the result in a destination. It is mandatory to provide the number of input keys before passing the input keys and other (optional) arguments. For more information about ZINTERSTORE, visit http://redis.io/commands/ZINTERSTORE. The previous command, which is zinterstore samsung_nex 2 samsung nex aggregate max, will compute the intersection of two sorted sets, samsung and nex, and stores it in another sorted set, samsung_nex. To see the result, type the following commands: > zrevrange samsung_nex 0 4 withscores 1) samsung_galaxy_nexus 2) 7 > hmget titles samsung_galaxy_nexus 1) Samsung Galaxy Nexus If you want to cache the result for multiword queries and remove it automatically, use an EXPIRE command and set expiry for temporary keys. Summary In this article, we have seen how to perform autosuggest and faceted searches using Redis. We have also understood how sorted sets and sets work. We have also seen how Redis can be used as a backend system for simple faceting and autosuggest system and make the system ultrafast. Further resources on this subject: Using Redis in a hostile environment (Advanced) [Article] Building Applications with Spring Data Redis [Article] Implementing persistence in Redis (Intermediate) [Article] Resources used for creating the article: Credit for the featured tiger image: Big Cat Facts - Tiger

The Sails Framework is a popular MVC framework that is designed for building practical, production-ready Node.js apps. Themes customize the look and feel of your app, but Sails does not come with a configuration or setting for handling themes by itself. This two-part post shows one of the ways you can set up theming for your Sails application, thus making use of some of Sails’ capabilities. You may have an application that needs to handle theming for different reasons, like custom branding, licensing, dynamic theme configuration, and so on. You can adjust the theming of your application, based on external factors, like patterns in the domain of the site you are browsing. Imagine you have an application that handles deliveries that you customize per client. So, your app renders the default theme when browsed as http://www.smartdelivery.com, but when logged in as a customer, let's say, "Burrito", it changes the domain name as http://burrito.smartdelivery.com. In this series we make use of Less as our language to define our CSS. Sails already handles Less right out of the box. The default Less file is located in /assets/styles/importer.less. We will also use Bootstrap as our base CSS Framework, importing its Less file into our importer.less file. The technique showed here consists of having a base CSS, and a theme CSS that varies according to the host name. Step 1 - Adding Bootstrap to Sails We use Bower to add Bootstrap to our project. First, install it by issuing the following command: npm install bower --save-dev Then, initialize the Bower configuration file. node_modules/bower/bin/bower init This command allows us to configure our bower.json file. Answer the questions asked by bower. ? name sails-themed-application ? description Sails Themed Application ? main file app.js ? keywords ? authors lobo ? license MIT ? homepage ? set currently installed components as dependencies? Yes ? add commonly ignored files to ignore list? Yes ? would you like to mark this package as private which prevents it from being accidentally published to the registry? No { name: 'sails-themed-application', description: 'Sails Themed Application', main: 'app.js', authors: [ 'lobo' ], license: 'MIT', homepage: '', ignore: [ '**/.*', 'node_modules', 'bower_components', 'assets/vendor', 'test', 'tests' ] } This generates a bower.json file in the root of your project. Now we need to tell bower to install everything in a specific directory. Create a file named .bowerrc and put this configuration into it: {"directory" : "assets/vendor"} Finally, install Bootstrap: node_modules/bower/bin/bower install bootstrap --save --production This action creates a folder in assets named vendor, with boostrap inside of it. Since Bootstrap uses JQuery, you also have a jquery folder: ├── api │ ├── controllers │ ├── models │ ├── policies │ ├── responses │ └── services ├── assets │ ├── images │ ├── js │ │ └── dependencies │ ├── styles │ ├── templates │ ├── themes │ └── vendor │ ├── bootstrap │ │ ├── dist │ │ │ ├── css │ │ │ ├── fonts │ │ │ └── js │ │ ├── fonts │ │ ├── grunt │ │ ├── js │ │ ├── less │ │ │ └── mixins │ │ └── nuget │ └── jquery │ ├── dist │ ├── external │ │ └── sizzle │ │ └── dist │ └── src │ ├── ajax │ │ └── var │ ├── attributes │ ├── core │ │ └── var │ ├── css │ │ └── var │ ├── data │ │ └── var │ ├── effects │ ├── event │ ├── exports │ ├── manipulation │ │ └── var │ ├── queue │ ├── traversing │ │ └── var │ └── var ├── config │ ├── env │ └── locales ├── tasks │ ├── config │ └── register └── views We need now to add Bootstrap into our importer. Edit /assets/styles/importer.less and add this instruction at the end of it: @import "../vendor/bootstrap/less/bootstrap.less"; Now you need to tell Sails where to import Bootstrap and JQuery JavaScript files from. Edit /tasks/pipeline.js and add the following code after it loads the sails.io.js file: // Load sails.io before everything else 'js/dependencies/sails.io.js', // <ADD THESE LINES> // JQuery JS 'vendor/jquery/dist/jquery.min.js', // Bootstrap JS 'vendor/bootstrap/dist/js/bootstrap.min.js', // </ADD THESE LINES> Now you have to edit your views layout and pages to use the Bootstrap style. In this series I created an application from scratch, so I have the default views and layouts. In your layout, insert the following line after your tag: <link rel="stylesheet" href="/themes/<%= typeof theme == 'undefined' ? 'default' : theme %>.css"> This loads a second CSS file, which defaults to /themes/default.css, into your views. As a sample, here are the /views/layout.ejs and /views/homepage.ejs I changed (the text under the headings is random text): /views/layout.ejs <!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1"> <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags --> <title><%= typeof title == 'undefined' ? 'Sails Themed Application' : title %></title> <!--STYLES--> <link rel="stylesheet" href="/styles/importer.css"> <!--STYLES END--> <!-- THIS IS WHERE THE THEME CSS IS LOADED --> <link rel="stylesheet" href="/themes/<%= typeof theme == 'undefined' ? 'default' : theme %>.css"> </head> <body> <%- body %> <!--TEMPLATES--> <!--TEMPLATES END--> <!--SCRIPTS--> <script src="/js/dependencies/sails.io.js"></script> <script src="/vendor/jquery/dist/jquery.min.js"></script> <script src="/vendor/bootstrap/dist/js/bootstrap.min.js"></script> <!--SCRIPTS END--> </body> </html> Notice the lines after the <!--STYLES END--> tag. /views/homepage.ejs <nav class="navbar navbar-inverse navbar-fixed-top"> <div class="container"> <div class="navbar-header"> <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar"> <span class="sr-only">Toggle navigation</span> <span class="icon-bar"></span> <span class="icon-bar"></span> <span class="icon-bar"></span> </button> <a class="navbar-brand" href="#">Project name</a> </div> <div id="navbar" class="navbar-collapse collapse"> <form class="navbar-form navbar-right"> <div class="form-group"> <input type="text" placeholder="Email" class="form-control"> </div> <div class="form-group"> <input type="password" placeholder="Password" class="form-control"> </div> <button type="submit" class="btn btn-success">Sign in</button> </form> </div><!--/.navbar-collapse --> </div> </nav> <!-- Main jumbotron for a primary marketing message or call to action --> <div class="jumbotron"> <div class="container"> <h1>Hello, world!</h1> <p>This is a template for a simple marketing or informational website. It includes a large callout called a jumbotron and three supporting pieces of content. Use it as a starting point to create something more unique.</p> <p><a class="btn btn-primary btn-lg" href="#" role="button">Learn more &raquo;</a></p> </div> </div> <div class="container"> <!-- Example row of columns --> <div class="row"> <div class="col-md-4"> <h2>Heading</h2> <p>Donec id elit non mi porta gravida at eget metus. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus. Etiam porta sem malesuada magna mollis euismod. Donec sed odio dui. </p> <p><a class="btn btn-default" href="#" role="button">View details &raquo;</a></p> </div> <div class="col-md-4"> <h2>Heading</h2> <p>Donec id elit non mi porta gravida at eget metus. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus. Etiam porta sem malesuada magna mollis euismod. Donec sed odio dui. </p> <p><a class="btn btn-default" href="#" role="button">View details &raquo;</a></p> </div> <div class="col-md-4"> <h2>Heading</h2> <p>Donec sed odio dui. Cras justo odio, dapibus ac facilisis in, egestas eget quam. Vestibulum id ligula porta felis euismod semper. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.</p> <p><a class="btn btn-default" href="#" role="button">View details &raquo;</a></p> </div> </div> <hr> <footer> <p>&copy; 2015 Company, Inc.</p> </footer> </div> <!-- /container --> You can now lift Sails and see your Bootstrapped Sails application. Now that we have our Bootstrapped Sails app set up, in Part 2 we will compile our theme’s CSS and the necessary Less files, and we will set bup the theme Sails hook to complete our application. About the author Luis Lobo Borobia is the CTO at FictionCity.NET, is a mentor and advisor, independent software engineer consultant, and conference speaker. He has a background as a software analyst and designer, creating, designing, and implementing Software products and solutions, frameworks, and platforms for several kinds of industries. In the last years he has focused on research and development for the Internet of Things, using the latest bleeding-edge software and hardware technologies available.

In this article by David R. Heffelfinger, the author of Java EE 7 Development with NetBeans 8, we will introduce Contexts and Dependency Injection (CDI) and other aspects of it. CDI can be used to simplify integrating the different layers of a Java EE application. For example, CDI allows us to use a session bean as a managed bean, so that we can take advantage of the EJB features, such as transactions, directly in our managed beans. In this article, we will cover the following topics: Introduction to CDI Qualifiers Stereotypes Interceptor binding types Custom scopes (For more resources related to this topic, see here.) Introduction to CDI JavaServer Faces (JSF) web applications employing CDI are very similar to JSF applications without CDI; the main difference is that instead of using JSF managed beans for our model and controllers, we use CDI named beans. What makes CDI applications easier to develop and maintain are the excellent dependency injection capabilities of the CDI API. Just as with other JSF applications, CDI applications use facelets as their view technology. The following example illustrates a typical markup for a JSF page using CDI: <?xml version='1.0' encoding='UTF-8' ?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html      >    <h:head>        <title>Create New Customer</title>    </h:head>    <h:body>        <h:form>            <h3>Create New Customer</h3>            <h:panelGrid columns="3">                <h:outputLabel for="firstName" value="First Name"/>                <h:inputText id="firstName" value="#{customer.firstName}"/>                <h:message for="firstName"/>                  <h:outputLabel for="middleName" value="Middle Name"/>                <h:inputText id="middleName"                  value="#{customer.middleName}"/>                <h:message for="middleName"/>                  <h:outputLabel for="lastName" value="Last Name"/>                <h:inputText id="lastName" value="#{customer.lastName}"/>                <h:message for="lastName"/>                  <h:outputLabel for="email" value="Email Address"/>                <h:inputText id="email" value="#{customer.email}"/>                <h:message for="email"/>                <h:panelGroup/>                <h:commandButton value="Submit"                  action="#{customerController.navigateToConfirmation}"/>            </h:panelGrid>        </h:form>    </h:body> </html> As we can see, the preceding markup doesn't look any different from the markup used for a JSF application that does not use CDI. The page renders as follows (shown after entering some data): In our page markup, we have JSF components that use Unified Expression Language expressions to bind themselves to CDI named bean properties and methods. Let's take a look at the customer bean first: package com.ensode.cdiintro.model;   import java.io.Serializable; import javax.enterprise.context.RequestScoped; import javax.inject.Named;   @Named @RequestScoped public class Customer implements Serializable {      private String firstName;    private String middleName;    private String lastName;    private String email;      public Customer() {    }      public String getFirstName() {        return firstName;    }      public void setFirstName(String firstName) {        this.firstName = firstName;    }      public String getMiddleName() {        return middleName;    }      public void setMiddleName(String middleName) {        this.middleName = middleName;    }      public String getLastName() {        return lastName;    }      public void setLastName(String lastName) {        this.lastName = lastName;    }      public String getEmail() {        return email;    }      public void setEmail(String email) {        this.email = email;    } } The @Named annotation marks this class as a CDI named bean. By default, the bean's name will be the class name with its first character switched to lowercase (in our example, the name of the bean is "customer", since the class name is Customer). We can override this behavior if we wish by passing the desired name to the value attribute of the @Named annotation, as follows: @Named(value="customerBean") A CDI named bean's methods and properties are accessible via facelets, just like regular JSF managed beans. Just like JSF managed beans, CDI named beans can have one of several scopes as listed in the following table. The preceding named bean has a scope of request, as denoted by the @RequestScoped annotation. Scope Annotation Description Request @RequestScoped Request scoped beans are shared through the duration of a single request. A single request could refer to an HTTP request, an invocation to a method in an EJB, a web service invocation, or sending a JMS message to a message-driven bean. Session @SessionScoped Session scoped beans are shared across all requests in an HTTP session. Each user of an application gets their own instance of a session scoped bean. Application @ApplicationScoped Application scoped beans live through the whole application lifetime. Beans in this scope are shared across user sessions. Conversation @ConversationScoped The conversation scope can span multiple requests, and is typically shorter than the session scope. Dependent @Dependent Dependent scoped beans are not shared. Any time a dependent scoped bean is injected, a new instance is created. As we can see, CDI has equivalent scopes to all JSF scopes. Additionally, CDI adds two additional scopes. The first CDI-specific scope is the conversation scope, which allows us to have a scope that spans across multiple requests, but is shorter than the session scope. The second CDI-specific scope is the dependent scope, which is a pseudo scope. A CDI bean in the dependent scope is a dependent object of another object; beans in this scope are instantiated when the object they belong to is instantiated and they are destroyed when the object they belong to is destroyed. Our application has two CDI named beans. We already discussed the customer bean. The other CDI named bean in our application is the controller bean: package com.ensode.cdiintro.controller;   import com.ensode.cdiintro.model.Customer; import javax.enterprise.context.RequestScoped; import javax.inject.Inject; import javax.inject.Named;   @Named @RequestScoped public class CustomerController {      @Inject    private Customer customer;      public Customer getCustomer() {        return customer;    }      public void setCustomer(Customer customer) {        this.customer = customer;    }      public String navigateToConfirmation() {        //In a real application we would        //Save customer data to the database here.          return "confirmation";    } } In the preceding class, an instance of the Customer class is injected at runtime; this is accomplished via the @Inject annotation. This annotation allows us to easily use dependency injection in CDI applications. Since the Customer class is annotated with the @RequestScoped annotation, a new instance of Customer will be injected for every request. The navigateToConfirmation() method in the preceding class is invoked when the user clicks on the Submit button on the page. The navigateToConfirmation() method works just like an equivalent method in a JSF managed bean would, that is, it returns a string and the application navigates to an appropriate page based on the value of that string. Like with JSF, by default, the target page's name with an .xhtml extension is the return value of this method. For example, if no exceptions are thrown in the navigateToConfirmation() method, the user is directed to a page named confirmation.xhtml: <?xml version='1.0' encoding='UTF-8' ?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html      >    <h:head>        <title>Success</title>    </h:head>    <h:body>        New Customer created successfully.        <h:panelGrid columns="2" border="1" cellspacing="0">            <h:outputLabel for="firstName" value="First Name"/>            <h:outputText id="firstName" value="#{customer.firstName}"/>              <h:outputLabel for="middleName" value="Middle Name"/>            <h:outputText id="middleName"              value="#{customer.middleName}"/>              <h:outputLabel for="lastName" value="Last Name"/>            <h:outputText id="lastName" value="#{customer.lastName}"/>              <h:outputLabel for="email" value="Email Address"/>            <h:outputText id="email" value="#{customer.email}"/>          </h:panelGrid>    </h:body> </html> Again, there is nothing special we need to do to access the named beans properties from the preceding markup. It works just as if the bean was a JSF managed bean. The preceding page renders as follows: As we can see, CDI applications work just like JSF applications. However, CDI applications have several advantages over JSF, for example (as we mentioned previously) CDI beans have additional scopes not found in JSF. Additionally, using CDI allows us to decouple our Java code from the JSF API. Also, as we mentioned previously, CDI allows us to use session beans as named beans. Qualifiers In some instances, the type of bean we wish to inject into our code may be an interface or a Java superclass, but we may be interested in injecting a subclass or a class implementing the interface. For cases like this, CDI provides qualifiers we can use to indicate the specific type we wish to inject into our code. A CDI qualifier is an annotation that must be decorated with the @Qualifier annotation. This annotation can then be used to decorate the specific subclass or interface. In this section, we will develop a Premium qualifier for our customer bean; premium customers could get perks that are not available to regular customers, for example, discounts. Creating a CDI qualifier with NetBeans is very easy; all we need to do is go to File | New File, select the Contexts and Dependency Injection category, and select the Qualifier Type file type. In the next step in the wizard, we need to enter a name and a package for our qualifier. After these two simple steps, NetBeans generates the code for our qualifier: package com.ensode.cdiintro.qualifier;   import static java.lang.annotation.ElementType.TYPE; import static java.lang.annotation.ElementType.FIELD; import static java.lang.annotation.ElementType.PARAMETER; import static java.lang.annotation.ElementType.METHOD; import static java.lang.annotation.RetentionPolicy.RUNTIME; import java.lang.annotation.Retention; import java.lang.annotation.Target; import javax.inject.Qualifier;   @Qualifier @Retention(RUNTIME) @Target({METHOD, FIELD, PARAMETER, TYPE}) public @interface Premium { } Qualifiers are standard Java annotations. Typically, they have retention of runtime and can target methods, fields, parameters, or types. The only difference between a qualifier and a standard annotation is that qualifiers are decorated with the @Qualifier annotation. Once we have our qualifier in place, we need to use it to decorate the specific subclass or interface implementation, as shown in the following code: package com.ensode.cdiintro.model;   import com.ensode.cdiintro.qualifier.Premium; import javax.enterprise.context.RequestScoped; import javax.inject.Named;   @Named @RequestScoped @Premium public class PremiumCustomer extends Customer {      private Integer discountCode;      public Integer getDiscountCode() {        return discountCode;    }      public void setDiscountCode(Integer discountCode) {        this.discountCode = discountCode;    } } Once we have decorated the specific instance we need to qualify, we can use our qualifiers in the client code to specify the exact type of dependency we need: package com.ensode.cdiintro.controller;   import com.ensode.cdiintro.model.Customer; import com.ensode.cdiintro.model.PremiumCustomer; import com.ensode.cdiintro.qualifier.Premium;   import java.util.logging.Level; import java.util.logging.Logger; import javax.enterprise.context.RequestScoped; import javax.inject.Inject; import javax.inject.Named;   @Named @RequestScoped public class PremiumCustomerController {      private static final Logger logger = Logger.getLogger(            PremiumCustomerController.class.getName());    @Inject    @Premium    private Customer customer;      public String saveCustomer() {          PremiumCustomer premiumCustomer =          (PremiumCustomer) customer;          logger.log(Level.INFO, "Saving the following information n"                + "{0} {1}, discount code = {2}",                new Object[]{premiumCustomer.getFirstName(),                    premiumCustomer.getLastName(),                    premiumCustomer.getDiscountCode()});          //If this was a real application, we would have code to save        //customer data to the database here.          return "premium_customer_confirmation";    } } Since we used our @Premium qualifier to decorate the customer field, an instance of the PremiumCustomer class is injected into that field. This is because this class is also decorated with the @Premium qualifier. As far as our JSF pages go, we simply access our named bean as usual using its name, as shown in the following code; <?xml version='1.0' encoding='UTF-8' ?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html      >    <h:head>        <title>Create New Premium Customer</title>    </h:head>    <h:body>        <h:form>            <h3>Create New Premium Customer</h3>            <h:panelGrid columns="3">                <h:outputLabel for="firstName" value="First Name"/>                 <h:inputText id="firstName"                    value="#{premiumCustomer.firstName}"/>                <h:message for="firstName"/>                  <h:outputLabel for="middleName" value="Middle Name"/>                <h:inputText id="middleName"                     value="#{premiumCustomer.middleName}"/>                <h:message for="middleName"/>                  <h:outputLabel for="lastName" value="Last Name"/>                <h:inputText id="lastName"                    value="#{premiumCustomer.lastName}"/>                <h:message for="lastName"/>                  <h:outputLabel for="email" value="Email Address"/>                <h:inputText id="email"                    value="#{premiumCustomer.email}"/>                <h:message for="email"/>                  <h:outputLabel for="discountCode" value="Discount Code"/>                <h:inputText id="discountCode"                    value="#{premiumCustomer.discountCode}"/>                <h:message for="discountCode"/>                   <h:panelGroup/>                <h:commandButton value="Submit"                      action="#{premiumCustomerController.saveCustomer}"/>            </h:panelGrid>        </h:form>    </h:body> </html> In this example, we are using the default name for our bean, which is the class name with the first letter switched to lowercase. Now, we are ready to test our application: After submitting the page, we can see the confirmation page. Stereotypes A CDI stereotype allows us to create new annotations that bundle up several CDI annotations. For example, if we need to create several CDI named beans with a scope of session, we would have to use two annotations in each of these beans, namely @Named and @SessionScoped. Instead of having to add two annotations to each of our beans, we could create a stereotype and annotate our beans with it. To create a CDI stereotype in NetBeans, we simply need to create a new file by selecting the Contexts and Dependency Injection category and the Stereotype file type. Then, we need to enter a name and package for our new stereotype. At this point, NetBeans generates the following code: package com.ensode.cdiintro.stereotype;   import static java.lang.annotation.ElementType.TYPE; import static java.lang.annotation.ElementType.FIELD; import static java.lang.annotation.ElementType.METHOD; import static java.lang.annotation.RetentionPolicy.RUNTIME; import java.lang.annotation.Retention; import java.lang.annotation.Target; import javax.enterprise.inject.Stereotype;   @Stereotype @Retention(RUNTIME) @Target({METHOD, FIELD, TYPE}) public @interface NamedSessionScoped { } Now, we simply need to add the CDI annotations that we want the classes annotated with our stereotype to use. In our case, we want them to be named beans and have a scope of session; therefore, we add the @Named and @SessionScoped annotations as shown in the following code: package com.ensode.cdiintro.stereotype;   import static java.lang.annotation.ElementType.TYPE; import static java.lang.annotation.ElementType.FIELD; import static java.lang.annotation.ElementType.METHOD; import static java.lang.annotation.RetentionPolicy.RUNTIME; import java.lang.annotation.Retention; import java.lang.annotation.Target; import javax.enterprise.context.SessionScoped; import javax.enterprise.inject.Stereotype; import javax.inject.Named;   @Named @SessionScoped @Stereotype @Retention(RUNTIME) @Target({METHOD, FIELD, TYPE}) public @interface NamedSessionScoped { } Now we can use our stereotype in our own code: package com.ensode.cdiintro.beans;   import com.ensode.cdiintro.stereotype.NamedSessionScoped; import java.io.Serializable;   @NamedSessionScoped public class StereotypeClient implements Serializable {      private String property1;    private String property2;      public String getProperty1() {        return property1;    }      public void setProperty1(String property1) {        this.property1 = property1;    }      public String getProperty2() {        return property2;    }      public void setProperty2(String property2) {        this.property2 = property2;    } } We annotated the StereotypeClient class with our NamedSessionScoped stereotype, which is equivalent to using the @Named and @SessionScoped annotations. Interceptor binding types One of the advantages of EJBs is that they allow us to easily perform aspect-oriented programming (AOP) via interceptors. CDI allows us to write interceptor binding types; this lets us bind interceptors to beans and the beans do not have to depend on the interceptor directly. Interceptor binding types are annotations that are themselves annotated with @InterceptorBinding. Creating an interceptor binding type in NetBeans involves creating a new file, selecting the Contexts and Dependency Injection category, and selecting the Interceptor Binding Type file type. Then, we need to enter a class name and select or enter a package for our new interceptor binding type. At this point, NetBeans generates the code for our interceptor binding type: package com.ensode.cdiintro.interceptorbinding;   import static java.lang.annotation.ElementType.TYPE; import static java.lang.annotation.ElementType.METHOD; import static java.lang.annotation.RetentionPolicy.RUNTIME; import java.lang.annotation.Inherited; import java.lang.annotation.Retention; import java.lang.annotation.Target; import javax.interceptor.InterceptorBinding;   @Inherited @InterceptorBinding @Retention(RUNTIME) @Target({METHOD, TYPE}) public @interface LoggingInterceptorBinding { } The generated code is fully functional; we don't need to add anything to it. In order to use our interceptor binding type, we need to write an interceptor and annotate it with our interceptor binding type, as shown in the following code: package com.ensode.cdiintro.interceptor;   import com.ensode.cdiintro.interceptorbinding.LoggingInterceptorBinding; import java.io.Serializable; import java.util.logging.Level; import java.util.logging.Logger; import javax.interceptor.AroundInvoke; import javax.interceptor.Interceptor; import javax.interceptor.InvocationContext;   @LoggingInterceptorBinding @Interceptor public class LoggingInterceptor implements Serializable{      private static final Logger logger = Logger.getLogger(            LoggingInterceptor.class.getName());      @AroundInvoke    public Object logMethodCall(InvocationContext invocationContext)            throws Exception {          logger.log(Level.INFO, new StringBuilder("entering ").append(                invocationContext.getMethod().getName()).append(                " method").toString());          Object retVal = invocationContext.proceed();          logger.log(Level.INFO, new StringBuilder("leaving ").append(                invocationContext.getMethod().getName()).append(                " method").toString());          return retVal;    } } As we can see, other than being annotated with our interceptor binding type, the preceding class is a standard interceptor similar to the ones we use with EJB session beans. In order for our interceptor binding type to work properly, we need to add a CDI configuration file (beans.xml) to our project. Then, we need to register our interceptor in beans.xml as follows: <?xml version="1.0" encoding="UTF-8"?> <beans               xsi_schemaLocation="http://>    <interceptors>          <class>        com.ensode.cdiintro.interceptor.LoggingInterceptor      </class>    </interceptors> </beans> To register our interceptor, we need to set bean-discovery-mode to all in the generated beans.xml and add the <interceptor> tag in beans.xml, with one or more nested <class> tags containing the fully qualified names of our interceptors. The final step before we can use our interceptor binding type is to annotate the class to be intercepted with our interceptor binding type: package com.ensode.cdiintro.controller;   import com.ensode.cdiintro.interceptorbinding.LoggingInterceptorBinding; import com.ensode.cdiintro.model.Customer; import com.ensode.cdiintro.model.PremiumCustomer; import com.ensode.cdiintro.qualifier.Premium; import java.util.logging.Level; import java.util.logging.Logger; import javax.enterprise.context.RequestScoped; import javax.inject.Inject; import javax.inject.Named;   @LoggingInterceptorBinding @Named @RequestScoped public class PremiumCustomerController {      private static final Logger logger = Logger.getLogger(            PremiumCustomerController.class.getName());    @Inject    @Premium    private Customer customer;      public String saveCustomer() {          PremiumCustomer premiumCustomer = (PremiumCustomer) customer;          logger.log(Level.INFO, "Saving the following information n"                + "{0} {1}, discount code = {2}",                new Object[]{premiumCustomer.getFirstName(),                    premiumCustomer.getLastName(),                    premiumCustomer.getDiscountCode()});          //If this was a real application, we would have code to save        //customer data to the database here.          return "premium_customer_confirmation";    } } Now, we are ready to use our interceptor. After executing the preceding code and examining the GlassFish log, we can see our interceptor binding type in action. The lines entering saveCustomer method and leaving saveCustomer method were added to the log by our interceptor, which was indirectly invoked by our interceptor binding type. Custom scopes In addition to providing several prebuilt scopes, CDI allows us to define our own custom scopes. This functionality is primarily meant for developers building frameworks on top of CDI, not for application developers. Nevertheless, NetBeans provides a wizard for us to create our own CDI custom scopes. To create a new CDI custom scope, we need to go to File | New File, select the Contexts and Dependency Injection category, and select the Scope Type file type. Then, we need to enter a package and a name for our custom scope. After clicking on Finish, our new custom scope is created, as shown in the following code: package com.ensode.cdiintro.scopes;   import static java.lang.annotation.ElementType.TYPE; import static java.lang.annotation.ElementType.FIELD; import static java.lang.annotation.ElementType.METHOD; import static java.lang.annotation.RetentionPolicy.RUNTIME; import java.lang.annotation.Inherited; import java.lang.annotation.Retention; import java.lang.annotation.Target; import javax.inject.Scope;   @Inherited @Scope // or @javax.enterprise.context.NormalScope @Retention(RUNTIME) @Target({METHOD, FIELD, TYPE}) public @interface CustomScope { } To actually use our scope in our CDI applications, we would need to create a custom context which, as mentioned previously, is primarily a concern for framework developers and not for Java EE application developers. Therefore, it is beyond the scope of this article. Interested readers can refer to JBoss Weld CDI for Java Platform, Ken Finnigan, Packt Publishing. (JBoss Weld is a popular CDI implementation and it is included with GlassFish.) Summary In this article, we covered NetBeans support for CDI, a new Java EE API introduced in Java EE 6. We provided an introduction to CDI and explained additional functionality that the CDI API provides over standard JSF. We also covered how to disambiguate CDI injected beans via CDI Qualifiers. Additionally, we covered how to group together CDI annotations via CDI stereotypes. We also, we saw how CDI can help us with AOP via interceptor binding types. Finally, we covered how NetBeans can help us create custom CDI scopes. Resources for Article: Further resources on this subject: Java EE 7 Performance Tuning and Optimization [article] Java EE 7 Developer Handbook [article] Java EE 7 with GlassFish 4 Application Server [article]

The examples are: Plotting data from a database Plotting data from a web page Plotting the data extracted by parsing an Apache log file Plotting the data read from a comma-separated values (CSV) file Plotting extrapolated data using curve fitting Third-party tools using Matplotlib (NetworkX and mpmath) Let's begin Plotting data from a database Databases often tend to collect much more information than we can simply extract and watch in a tabular format (let's call it the "Excel sheet" report style). Databases not only use efficient techniques to store and retrieve data, but they are also very good at aggregating it. One suggestion we can give is to let the database do the work. For example, if we need to sum up a column, let's make the database sum the data, and not sum it up in the code. In this way, the whole process is much more efficient because: There is a smaller memory footprint for the Python code, since only the aggregate value is returned, not the whole result set to generate it The database has to read all the rows in any case. However, if it's smart enough, then it can sum values up as they are read The database can efficiently perform such an operation on more than one column at a time The data source we're going to query is from an open source project: the Debian distribution. Debian has an interesting project called UDD , Ultimate Debian Database, which is a relational database where a lot of information (either historical or actual) about the distribution is collected and can be analyzed. On the project website http://udd.debian.org/, we can fi nd a full dump of the database (quite big, honestly) that can be downloaded and imported into a local PostgreSQL instance (refer to http://wiki.debian.org/UltimateDebianDatabase/CreateLocalReplica for import instructions Now that we have a local replica of UDD, we can start querying it: # module to access PostgreSQL databasesimport psycopg2# matplotlib pyplot moduleimport matplotlib.pyplot as plt Since UDD is stored in a PostgreSQL database, we need psycopg2 to access it. psycopg2 is a third-party module available at http://initd.org/projects/psycopg # connect to UDD databaseconn = psycopg2.connect(database="udd")# prepare a cursorcur = conn.cursor() We will now connect to the database server to access the udd database instance, and then open a cursor on the connection just created. # this is the query we'll be makingquery = """select to_char(date AT TIME ZONE 'UTC', 'HH24'), count(*) from upload_history where to_char(date, 'YYYY') = '2008' group by 1 order by 1""" We have prepared the select statement to be executed on UDD. What we wish to do here is extract the number of packages uploaded to the Debian archive (per hour) in the whole year of 2008. date AT TIME ZONE 'UTC': As date field is of the type timestamp with time zone, it also contains time zone information, while we want something independent from the local time. This is the way to get a date in UTC time zone. group by 1: This is what we have encouraged earlier, that is, let the database do the work. We let the query return the already aggregated data, instead of coding it into the program. # execute the querycur.execute(query)# retrieve the whole result setdata = cur.fetchall() We execute the query and fetch the whole result set from it. # close cursor and connectioncur.close()conn.close() Remember to always close the resources that we've acquired in order to avoid memory or resource leakage and reduce the load on the server (removing connections that aren't needed anymore). # unpack data in hours (first column) and# uploads (second column)hours, uploads = zip(*data) The query result is a list of tuples, (in this case, hour and number of uploads), but we need two separate lists—one for the hours and another with the corresponding number of uploads. zip() solves this with *data, we unpack the list, returning the sublists as separate arguments to zip(), which in return, aggregates the elements in the same position in the parameters into separated lists. Consider the following example: In [1]: zip(['a1', 'a2'], ['b1', 'b2'])Out[1]: [('a1', 'b1'), ('a2', 'b2')] To complete the code: # graph codeplt.plot(hours, uploads)# the the x limits to the 'hours' limitplt.xlim(0, 23)# set the X ticks every 2 hoursplt.xticks(range(0, 23, 2))# draw a gridplt.grid()# set title, X/Y labelsplt.title("Debian packages uploads per hour in 2008")plt.xlabel("Hour (in UTC)")plt.ylabel("No. of uploads") The previous code snippet is the standard plotting code, which results in the following screenshot: From this graph we can see that in 2008, the main part of Debian packages uploads came from European contributors. In fact, uploads were made mainly in the evening hours (European time), after the working days are over (as we can expect from a voluntary project). Plotting data from the Web Often, the information we need is not distributed in an easy-to-use format such as XML or a database export but for example only on web sites. More and more often we find interesting data on a web page, and in that case we have to parse it to extract that information: this is called web scraping . In this example, we will parse a Wikipedia article to extracts some data to plot. The article is at http://it.wikipedia.org/wiki/Demografia_d'Italia and contains lots of information about Italian demography (it's in Italian because the English version lacks a lot of data); in particular, we are interested in the population evolution over the years. Probably the best known Python module for web scraping is BeautifulSoup ( http://www.crummy.com/software/BeautifulSoup/). It's a really nice library that gets the job done quickly, but there are situations (in particular with JavaScript embedded in the web page, such as for Wikipedia) that prevent it from working. As an alternative, we find lxml quite productive (http://codespeak.net/lxml/). It's a library mainly used to work with XML (as the name suggests), but it can also be used with HTML (given their quite similar structures), and it is powerful and easy–to-use. Let's dig into the code now: # to get the web pagesimport urllib2# lxml submodule for html parsingfrom lxml.html import parse# regular expression moduleimport re# Matplotlib moduleimport matplotlib.pyplot as plt Along with the Matplotlib module, we need the following modules: urllib2: This is the module (from the standard library) that is used to access resources through URL (we will download the webpage with this). lxml: This is the parsing library. re: Regular expressions are needed to parse the returned data to extract the information we need. re is a module from the standard library, so we don't need to install a third-party module to use it. # general urllib2 configuser_agent = 'Mozilla/5.0 (compatible; MSIE 5.5; Windows NT)'headers = { 'User-Agent' : user_agent }url = "http://it.wikipedia.org/wiki/Demografia_d'Italia" Here, we prepare some configuration for urllib2, in particular, the user_agent header is used to access Wikipedia and the URL of the page. # prepare the request and open the urlreq = urllib2.Request(url, headers=headers)response = urllib2.urlopen(req) Then we make a request for the URL and get the HTML back. # we parse the webpage, getroot() return the document rootdoc = parse(response).getroot() We parse the HTML using the parse() function of lxml.html and then we get the root element. XML can be seen as a tree, with a root element (the node at the top of the tree from where every other node descends), and a hierarchical structure of elements. # find the data table, using css elementstable = doc.cssselect('table.wikitable')[0] We leverage the structure of HTML accessing the first element of type table of class wikitable because that's the table we're interested in. # prepare data structures, will contain actual datayears = []people = [] Preparing the lists that will contain the parsed data. # iterate over the rows of the table, except first and last onesfor row in table.cssselect('tr')[1:-1]: We can start parsing the table. Since there is a header and a footer in the table, we skip the first and the last line from the lines (selected by the tr tag) to loop over. # get the row cell (we will use only the first two)data = row.cssselect('td') We get the element with the td tag that stands for table data: those are the cells in an HTML table. # the first cell is the yeartmp_years = data[0].text_content()# cleanup for cases like 'YYYY[N]' (date + footnote link)tmp_years = re.sub('[.]', '', tmp_years) We take the first cell that contains the year, but we need to remove the additional characters (used by Wikipedia to link to footnotes). # the second cell is the population counttmp_people = data[1].text_content()# cleanup from '.', used as separatortmp_people = tmp_people.replace('.', '') We also take the second cell that contains the population for a given year. It's quite common in Italy to separate thousands in number with a '.' character: we have to remove them to have an appropriate value. # append current data to data lists, converting to integersyears.append(int(tmp_years))people.append(int(tmp_people)) We append the parsed values to the data lists, explicitly converting them to integer values. # plot dataplt.plot(years,people)# ticks every 10 yearsplt.xticks(range(min(years), max(years), 10))plt.grid()# add a note for 2001 Censusplt.annotate("2001 Census", xy=(2001, people[years.index(2001)]), xytext=(1986, 54.5*10**6), arrowprops=dict(arrowstyle='fancy')) Running the example results in the following screenshot that clearly shows why the annotation is needed: In 2001, we had a national census in Italy, and that's the reason for the drop in that year: the values released from the National Institute for Statistics (and reported in the Wikipedia article) are just an estimation of the population. However, with a census, we have a precise count of the people living in Italy. Plotting data by parsing an Apache log file Plotting data from a log file can be seen as the art of extracting information from it. Every service has a log format different from the others. There are some exceptions of similar or same format (for example, for services that come from the same development teams) but then they may be customized and we're back at the beginning. The main differences in log files are: Fields orders: Some have time information at the beginning, others in the middle of the line, and so on Fields types: We can find several different data types such as integers, strings, and so on Fields meanings: For example, log levels can have very different meanings From all the data contained in the log file, we need to extract the information we are interested in from the surrounding data that we don't need (and hence we skip). In our example, we're going to analyze the log file of one of the most common services: Apache. In particular, we will parse the access.log file to extract the total number of hits and amount of data transferred per day. Apache is highly configurable, and so is the log format. Our Apache configuration, contained in the httpd.conf file, has this log format: "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" This is in LogFormat specification where Log directive Description %h The host making the request %l Identity of the client (which is usually not available) %u User making the request (usually not available) %t The time the request was received %r The request %>s The status code %b The size (in bytes) of the response sent to the client (excluding the headers) %{Referer}i The page from where the requests originated (for example, the HTML page where a PNG image is requested) %{User-Agent}i The user agent used to make the request

This article by, Michał Jaworski and Tarek Ziadé, the authors of the book, Expert Python Programming - Second Edition, will mainly focus on interfaces. (For more resources related to this topic, see here.) An interface is a definition of an API. It describes a list of methods and attributes a class should have to implement with the desired behavior. This description does not implement any code but just defines an explicit contract for any class that wishes to implement the interface. Any class can then implement one or several interfaces in whichever way it wants. While Python prefers duck-typing over explicit interface definitions, it may be better to use them sometimes. For instance, explicit interface definition makes it easier for a framework to define functionalities over interfaces. The benefit is that classes are loosely coupled, which is considered as a good practice. For example, to perform a given process, a class A does not depend on a class B, but rather on an interface I. Class B implements I, but it could be any other class. The support for such a technique is built-in in many statically typed languages such as Java or Go. The interfaces allow the functions or methods to limit the range of acceptable parameter objects that implement a given interface, no matter what kind of class it comes from. This allows for more flexibility than restricting arguments to given types or their subclasses. It is like an explicit version of duck-typing behavior: Java uses interfaces to verify a type safety at compile time rather than use duck-typing to tie things together at run time. Python has a completely different typing philosophy to Java, so it does not have native support for interfaces. Anyway, if you would like to have more explicit control on application interfaces, there are generally two solutions to choose from: Use some third-party framework that adds a notion of interfaces Use some of the advanced language features to build your methodology for handling interfaces. Using zope.interface There are a few frameworks that allow you to build explicit interfaces in Python. The most notable one is a part of the Zope project. It is the zope.interface package. Although, nowadays, Zope is not as popular as it used to be, the zope.interface package is still one of the main components of the Twisted framework. The core class of the zope.interface package is the Interface class. It allows you to explicitly define a new interface by subclassing. Let's assume that we want to define the obligatory interface for every implementation of a rectangle: from zope.interface import Interface, Attribute class IRectangle(Interface): width = Attribute("The width of rectangle") height = Attribute("The height of rectangle") def area(): """ Return area of rectangle """ def perimeter(): """ Return perimeter of rectangle """ Some important things to remember when defining interfaces with zope.interface are as follows: The common naming convention for interfaces is to use I as the name suffix. The methods of the interface must not take the self parameter. As the interface does not provide concrete implementation, it should consist only of empty methods. You can use the pass statement, raise NotImplementedError, or provide a docstring (preferred). An interface can also specify the required attributes using the Attribute class. When you have such a contract defined, you can then define new concrete classes that provide implementation for our IRectangle interface. In order to do that, you need to use the implementer() class decorator and implement all of the defined methods and attributes: @implementer(IRectangle) class Square: """ Concrete implementation of square with rectangle interface """ def __init__(self, size): self.size = size @property def width(self): return self.size @property def height(self): return self.size def area(self): return self.size ** 2 def perimeter(self): return 4 * self.size @implementer(IRectangle) class Rectangle: """ Concrete implementation of rectangle """ def __init__(self, width, height): self.width = width self.height = height def area(self): return self.width * self.height def perimeter(self): return self.width * 2 + self.height * 2 It is common to say that the interface defines a contract that a concrete implementation needs to fulfill. The main benefit of this design pattern is being able to verify consistency between contract and implementation before the object is being used. With the ordinary duck-typing approach, you only find inconsistencies when there is a missing attribute or method at runtime. With zope.interface, you can introspect the actual implementation using two methods from the zope.interface.verify module to find inconsistencies early on: verifyClass(interface, class_object): This verifies the class object for existence of methods and correctness of their signatures without looking for attributes verifyObject(interface, instance): This verifies the methods, their signatures, and also attributes of the actual object instance Since we have defined our interface and two concrete implementations, let's verify their contracts in an interactive session: >>> from zope.interface.verify import verifyClass, verifyObject >>> verifyObject(IRectangle, Square(2)) True >>> verifyClass(IRectangle, Square) True >>> verifyObject(IRectangle, Rectangle(2, 2)) True >>> verifyClass(IRectangle, Rectangle) True Nothing impressive. The Rectangle and Square classes carefully follow the defined contract so there is nothing more to see than a successful verification. But what happens when we make a mistake? Let's see an example of two classes that fail to provide full IRectangle interface implementation: @implementer(IRectangle) class Point: def __init__(self, x, y): self.x = x self.y = y @implementer(IRectangle) class Circle: def __init__(self, radius): self.radius = radius def area(self): return math.pi * self.radius ** 2 def perimeter(self): return 2 * math.pi * self.radius The Point class does not provide any method or attribute of the IRectangle interface, so its verification will show inconsistencies already on the class level: >>> verifyClass(IRectangle, Point) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "zope/interface/verify.py", line 102, in verifyClass return _verify(iface, candidate, tentative, vtype='c') File "zope/interface/verify.py", line 62, in _verify raise BrokenImplementation(iface, name) zope.interface.exceptions.BrokenImplementation: An object has failed to implement interface <InterfaceClass __main__.IRectangle> The perimeter attribute was not provided. The Circle class is a bit more problematic. It has all the interface methods defined but breaks the contract on the instance attribute level. This is the reason why, in most cases, you need to use the verifyObject() function to completely verify the interface implementation: >>> verifyObject(IRectangle, Circle(2)) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "zope/interface/verify.py", line 105, in verifyObject return _verify(iface, candidate, tentative, vtype='o') File "zope/interface/verify.py", line 62, in _verify raise BrokenImplementation(iface, name) zope.interface.exceptions.BrokenImplementation: An object has failed to implement interface <InterfaceClass __main__.IRectangle> The width attribute was not provided. Using zope.inteface is an interesting way to decouple your application. It allows you to enforce proper object interfaces without the need for the overblown complexity of multiple inheritance, and it also allows to catch inconsistencies early. However, the biggest downside of this approach is the requirement that you explicitly define that the given class follows some interface in order to be verified. This is especially troublesome if you need to verify instances coming from external classes of built-in libraries. zope.interface provides some solutions for that problem, and you can of course handle such issues on your own by using the adapter pattern, or even monkey-patching. Anyway, the simplicity of such solutions is at least arguable. Using function annotations and abstract base classes Design patterns are meant to make problem solving easier and not to provide you with more layers of complexity. The zope.interface is a great concept and may greatly fit some projects, but it is not a silver bullet. By using it, you may soon find yourself spending more time on fixing issues with incompatible interfaces for third-party classes and providing never-ending layers of adapters instead of writing the actual implementation. If you feel that way, then this is a sign that something went wrong. Fortunately, Python supports for building lightweight alternative to the interfaces. It's not a full-fledged solution like zope.interface or its alternatives but it generally provides more flexible applications. You may need to write a bit more code, but in the end you will have something that is more extensible, better handles external types, and may be more future proof. Note that Python in its core does not have explicit notions of interfaces, and probably will never have, but has some of the features that allow you to build something that resembles the functionality of interfaces. The features are: Abstract base classes (ABCs) Function annotations Type annotations The core of our solution is abstract base classes, so we will feature them first. As you probably know, the direct type comparison is considered harmful and not pythonic. You should always avoid comparisons as follows: assert type(instance) == list Comparing types in functions or methods that way completely breaks the ability to pass a class subtype as an argument to the function. The slightly better approach is to use the isinstance() function that will take the inheritance into account: assert isinstance(instance, list) The additional advantage of isinstance() is that you can use a larger range of types to check the type compatibility. For instance, if your function expects to receive some sort of sequence as the argument, you can compare against the list of basic types: assert isinstance(instance, (list, tuple, range)) Such a way of type compatibility checking is OK in some situations but it is still not perfect. It will work with any subclass of list, tuple, or range, but will fail if the user passes something that behaves exactly the same as one of these sequence types but does not inherit from any of them. For instance, let's relax our requirements and say that you want to accept any kind of iterable as an argument. What would you do? The list of basic types that are iterable is actually pretty long. You need to cover list, tuple, range, str, bytes, dict, set, generators, and a lot more. The list of applicable built-in types is long, and even if you cover all of them it will still not allow you to check against the custom class that defines the __iter__() method, but will instead inherit directly from object. And this is the kind of situation where abstract base classes (ABC) are the proper solution. ABC is a class that does not need to provide a concrete implementation but instead defines a blueprint of a class that may be used to check against type compatibility. This concept is very similar to the concept of abstract classes and virtual methods known in the C++ language. Abstract base classes are used for two purposes: Checking for implementation completeness Checking for implicit interface compatibility So, let's assume we want to define an interface which ensures that a class has a push() method. We need to create a new abstract base class using a special ABCMeta metaclass and an abstractmethod() decorator from the standard abc module: from abc import ABCMeta, abstractmethod class Pushable(metaclass=ABCMeta): @abstractmethod def push(self, x): """ Push argument no matter what it means """ The abc module also provides an ABC base class that can be used instead of the metaclass syntax: from abc import ABCMeta, abstractmethod class Pushable(metaclass=ABCMeta): @abstractmethod def push(self, x): """ Push argument no matter what it means """ Once it is done, we can use that Pushable class as a base class for concrete implementation and it will guard us from the instantiation of objects that would have incomplete implementation. Let's define DummyPushable, which implements all interface methods and the IncompletePushable that breaks the expected contract: class DummyPushable(Pushable): def push(self, x): return class IncompletePushable(Pushable): pass If you want to obtain the DummyPushable instance, there is no problem because it implements the only required push() method: >>> DummyPushable() <__main__.DummyPushable object at 0x10142bef0> But if you try to instantiate IncompletePushable, you will get TypeError because of missing implementation of the interface() method: >>> IncompletePushable() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: Can't instantiate abstract class IncompletePushable with abstract methods push The preceding approach is a great way to ensure implementation completeness of base classes but is as explicit as the zope.interface alternative. The DummyPushable instances are of course also instances of Pushable because Dummy is a subclass of Pushable. But how about other classes with the same methods but not descendants of Pushable? Let's create one and see: >>> class SomethingWithPush: ... def push(self, x): ... pass ... >>> isinstance(SomethingWithPush(), Pushable) False Something is still missing. The SomethingWithPush class definitely has a compatible interface but is not considered as an instance of Pushable yet. So, what is missing? The answer is the __subclasshook__(subclass) method that allows you to inject your own logic into the procedure that determines whether the object is an instance of a given class. Unfortunately, you need to provide it by yourself, as abc creators did not want to constrain the developers in overriding the whole isinstance() mechanism. We got full power over it, but we are forced to write some boilerplate code. Although you can do whatever you want to, usually the only reasonable thing to do in the __subclasshook__() method is to follow the common pattern. The standard procedure is to check whether the set of defined methods are available somewhere in the MRO of the given class: from abc import ABCMeta, abstractmethod class Pushable(metaclass=ABCMeta): @abstractmethod def push(self, x): """ Push argument no matter what it means """ @classmethod def __subclasshook__(cls, C): if cls is Pushable: if any("push" in B.__dict__ for B in C.__mro__): return True return NotImplemented With the __subclasshook__() method defined that way, you can now confirm that the instances that implement the interface implicitly are also considered instances of the interface: >>> class SomethingWithPush: ... def push(self, x): ... pass ... >>> isinstance(SomethingWithPush(), Pushable) True Unfortunately, this approach to the verification of type compatibility and implementation completeness does not take into account the signatures of class methods. So, if the number of expected arguments is different in implementation, it will still be considered compatible. In most cases, this is not an issue, but if you need such fine-grained control over interfaces, the zope.interface package allows for that. As already said, the __subclasshook__() method does not constrain you in adding more complexity to the isinstance() function's logic to achieve a similar level of control. The two other features that complement abstract base classes are function annotations and type hints. Function annotation is the syntax element. It allows you to annotate functions and their arguments with arbitrary expressions. This is only a feature stub that does not provide any syntactic meaning. There is no utility in the standard library that uses this feature to enforce any behavior. Anyway, you can use it as a convenient and lightweight way to inform the developer of the expected argument interface. For instance, consider this IRectangle interface rewritten from zope.interface to abstract the base class: from abc import ( ABCMeta, abstractmethod, abstractproperty ) class IRectangle(metaclass=ABCMeta): @abstractproperty def width(self): return @abstractproperty def height(self): return @abstractmethod def area(self): """ Return rectangle area """ @abstractmethod def perimeter(self): """ Return rectangle perimeter """ @classmethod def __subclasshook__(cls, C): if cls is IRectangle: if all([ any("area" in B.__dict__ for B in C.__mro__), any("perimeter" in B.__dict__ for B in C.__mro__), any("width" in B.__dict__ for B in C.__mro__), any("height" in B.__dict__ for B in C.__mro__), ]): return True return NotImplemented If you have a function that works only on rectangles, let's say draw_rectangle(), you could annotate the interface of the expected argument as follows: def draw_rectangle(rectangle: IRectange): ... This adds nothing more than information for the developer about expected information. And even this is done through an informal contract because, as we know, bare annotations contain no syntactic meaning. However, they are accessible at runtime, so we can do something more. Here is an example implementation of a generic decorator that is able to verify interface from function annotation if it is provided using abstract base classes: def ensure_interface(function): signature = inspect.signature(function) parameters = signature.parameters @wraps(function) def wrapped(*args, **kwargs): bound = signature.bind(*args, **kwargs) for name, value in bound.arguments.items(): annotation = parameters[name].annotation if not isinstance(annotation, ABCMeta): continue if not isinstance(value, annotation): raise TypeError( "{} does not implement {} interface" "".format(value, annotation) ) function(*args, **kwargs) return wrapped Once it is done, we can create some concrete class that implicitly implements the IRectangle interface (without inheriting from IRectangle) and update the implementation of the draw_rectangle() function to see how the whole solution works: class ImplicitRectangle: def __init__(self, width, height): self._width = width self._height = height @property def width(self): return self._width @property def height(self): return self._height def area(self): return self.width * self.height def perimeter(self): return self.width * 2 + self.height * 2 @ensure_interface def draw_rectangle(rectangle: IRectangle): print( "{} x {} rectangle drawing" "".format(rectangle.width, rectangle.height) ) If we feed the draw_rectangle() function with an incompatible object, it will now raise TypeError with a meaningful explanation: >>> draw_rectangle('foo') Traceback (most recent call last): File "<input>", line 1, in <module> File "<input>", line 101, in wrapped TypeError: foo does not implement <class 'IRectangle'> interface But if we use ImplicitRectangle or anything else that resembles the IRectangle interface, the function executes as it should: >>> draw_rectangle(ImplicitRectangle(2, 10)) 2 x 10 rectangle drawing Our example implementation of ensure_interface() is based on the typechecked() decorator from the typeannotations project that tries to provide run-time checking capabilities (refer to https://github.com/ceronman/typeannotations). Its source code might give you some interesting ideas about how to process type annotations to ensure run-time interface checking. The last feature that can be used to complement this interface pattern landscape are type hints. Type hints are described in detail by PEP 484 and were added to the language quite recently. They are exposed in the new typing module and are available from Python 3.5. Type hints are built on top of function annotations and reuse this slightly forgotten syntax feature of Python 3. They are intended to guide type hinting and check for various yet-to-come Python type checkers. The typing module and PEP 484 document aim to provide a standard hierarchy of types and classes that should be used for describing type annotations. Still, type hints do not seem to be something revolutionary because this feature does not come with any type checker built-in into the standard library. If you want to use type checking or enforce strict interface compatibility in your code, you need to create your own tool because there is none worth recommendation yet. This is why we won't dig into details of PEP 484. Anyway, type hints and the documents describing them are worth mentioning because if some extraordinary solution emerges in the field of type checking in Python, it is highly probable that it will be based on PEP 484. Using collections.abc Abstract base classes are like small building blocks for creating a higher level of abstraction. They allow you to implement really usable interfaces but are very generic and designed to handle lot more than this single design pattern. You can unleash your creativity and do magical things but building something generic and really usable may require a lot of work. Work that may never pay off. This is why custom abstract base classes are not used so often. Despite that, the collections.abc module provides a lot of predefined ABCs that allow to verify interface compatibility of many basic Python types. With base classes provided in this module, you can check, for example, whether a given object is callable, mapping, or if it supports iteration. Using them with the isinstance() function is way better than comparing them against the base python types. You should definitely know how to use these base classes even if you don't want to define your own custom interfaces with ABCMeta. The most common abstract base classes from collections.abc that you will use from time to time are: Container: This interface means that the object supports the in operator and implements the __contains__() method Iterable: This interface means that the object supports the iteration and implements the __iter__() method Callable: This interface means that it can be called like a function and implements the __call__() method Hashable: This interface means that the object is hashable (can be included in sets and as key in dictionaries) and implements the __hash__ method Sized: This interface means that the object has size (can be a subject of the len() function) and implements the __len__() method A full list of the available abstract base classes from the collections.abc module is available in the official Python documentation (refer to https://docs.python.org/3/library/collections.abc.html). Summary Design patterns are reusable, somewhat language-specific solutions to common problems in software design. They are a part of the culture of all developers, no matter what language they use. We learned a small part of design patterns in this article. We covered what interfaces are and how they can be used in Python. Resources for Article: Further resources on this subject: Creating User Interfaces [article] Graphical User Interfaces for OpenSIPS 1.6 [article]

It’s not always easy to know what to learn next if you’re a programmer. Industry shifts can be subtle but they can sometimes be dramatic, making it incredibly important to stay on top of what’s happening both in your field and beyond. No one person can make that decision for you. All the thought leadership and mentorship in the world isn’t going to be able to tell you what’s right for you when it comes to your career. But this list of videos, released last month, might give you a helping hand as to where to go next when it comes to your learning… New data science and artificial intelligence video courses for March Apache Spark is carving out a big presence as the go-to software for big data. Two videos from February focus on Spark - Distributed Deep Learning with Apache Spark and Apache Spark in 7 Days. If you’re new to Spark and want a crash course on the tool, then clearly, our video aims to get you up and running quickly. However, Distributed Deep Learning with Apache Spark offers a deeper exploration that shows you how to develop end to end deep learning pipelines that can leverage the full potential of cutting edge deep learning techniques. While we’re on the subject of machine learning, other choice video courses for March include TensorFlow 2.0 New Features (we’ve been eagerly awaiting it and it finally looks like we can see what it will be like), Hands On Machine Learning with JavaScript (yes, you can now do machine learning in the browser), and a handful of interesting videos on artificial intelligence and finance: AI for Finance Machine Learning for Algorithmic Trading Bots with Python Hands on Python for Finance Elsewhere, a number of data visualization video courses prove that communicating and presenting data remains an urgent challenge for those in the data space. Tableau remains one of the definitive tools - you can learn the latest version with Tableau 2019.1 for Data Scientists and Data Visualization Recipes with Python and Matplotlib 3.   New app and web development video courses for March 2019 There are a wealth of video courses for web and app developers to choose from this month. True, Hands-on Machine Learning for JavaScript is well worth a look, but moving past the machine learning hype, there are a number of video courses that take a practical look at popular tools and new approaches to app and web development. Angular’s death has been greatly exaggerated - it remains a pillar of the JavaScript world. While the project’s versioning has arguably been lacking some clarity, if you want to get up to speed with where the framework is today, try Angular 7: A Practical Guide. It’s a video that does exactly what it says on the proverbial tin - it shows off Angular 7 and demonstrates how to start using it in web projects. We’ve also been seeing some uptake of Angular by ASP.NET developers, as it offers a nice complement to the Microsoft framework on the front end side. Our latest video on the combination, Hands-on Web Development with ASP.NET Core and Angular, is another practical look at an effective and increasingly popular approach to full-stack development. Other picks for March include Building Mobile Apps with Ionic 4, a video that brings you right up to date with the recent update that launched in January (interestingly, the project is now backed by web components, not Angular), and a couple of Redux videos - Mastering Redux and Redux Recipes. Redux is still relatively new. Essentially, it’s a JavaScript library that helps you manage application state - because it can be used with a range of different frameworks and libraries, including both Angular and React, it’s likely to go from strength to strength in 2019. Infrastructure, admin and security video courses for March 2019 Node.js is becoming an important library for infrastructure and DevOps engineers. As we move to a cloud native world, it’s a great tool for developing lightweight and modular services. That’s why we’re picking Learn Serverless App Development with Node.js and Azure Functions as one of our top videos for this month. Azure has been growing at a rapid rate over the last 12 months, and while it’s still some way behind AWS, Microsoft’s focus on developer experience is making Azure an increasingly popular platform with developers. For Node developers, this video is a great place to begin - it’s also useful for anyone who simply wants to find out what serverless development actually feels like. Read next: Serverless computing wars: AWS Lambda vs. Azure Functions A partner to this, for anyone beginning Node, is the new Node.js Design Patterns video. In particular, if Node.js is an important tool in your architecture, following design patterns is a robust method of ensuring reliability and resilience. Elsewhere, we have Modern DevOps in Practice, cutting through the consultancy-speak to give you useful and applicable guidance on how to use DevOps thinking in your workflows and processes, and DevOps with Azure, another video that again demonstrates just how impressive Azure is. For those not Azure-inclined, there’s AWS Certified Developer Associate - A Practical Guide, a video that takes you through everything you need to know to pass the AWS Developer Associate exam. There’s also a completely cloud-agnostic video course in the form of Creating a Continuous Deployment Pipeline for Cloud Platforms that’s essential for infrastructure and operations engineers getting to grips with cloud native development.     Learn a new programming language with these new video courses for March Finally, there are a number of new video courses that can help you get to grips with a new programming language. So, perfect if you’ve been putting off your new year’s resolution to learn a new language… Java 11 in 7 Days is a new video that brings you bang up to date with everything in the latest version of Java, while Hands-on Functional Programming with Java will help you rethink and reevaluate the way you use Java. Together, the two videos are a great way for Java developers to kick start their learning and update their skill set.  

 In this article by Greg Moss, author of Working with Odoo, he explains that Odoo is a very feature-filled business application framework with literally hundreds of applications and modules available. We have done our best to cover the most essential features of the Odoo applications that you are most likely to use in your business. Setting up an Odoo system is no easy task. Many companies get into trouble believing that they can just install the software and throw in some data. Inevitably, the scope of the project grows and what was supposed to be a simple system ends up being a confusing mess. Fortunately, Odoo's modular design will allow you to take a systematic approach to implementing Odoo for your business. (For more resources related to this topic, see here.) What is an ERP system? An Enterprise Resource Planning (ERP) system is essentially a suite of business applications that are integrated together to assist a company in collecting, managing, and reporting information throughout core business processes. These business applications, typically called modules, can often be independently installed and configured based on the specific needs of the business. As the needs of the business change and grow, additional modules can be incorporated into an existing ERP system to better handle the new business requirements. This modular design of most ERP systems gives companies great flexibility in how they implement the system. In the past, ERP systems were primarily utilized in manufacturing operations. Over the years, the scope of ERP systems have grown to encompass a wide range of business-related functions. Recently, ERP systems have started to include more sophisticated communication and social networking features. Common ERP modules The core applications of an ERP system typically include: Sales Orders Purchase Orders Accounting and Finance Manufacturing Resource Planning (MRP) Customer Relationship Management (CRM) Human Resources (HR) Let's take a brief look at each of these modules and how they address specific business needs. Selling products to your customer Sales Orders, commonly abbreviated as SO, are documents that a business generates when they sell products and services to a customer. In an ERP system, the Sales Order module will usually allow management of customers and products to optimize efficiency for data entry of the sales order. Many sales orders begin as customer quotes. Quotes allow a salesperson to collect order information that may change as the customer makes decisions on what they want in their final order. Once a customer has decided exactly what they wish to purchase, the quote is turned into a sales order and is confirmed for processing. Depending on the requirements of the business, there are a variety of methods to determine when a customer is invoiced or billed for the order. This preceding screenshot shows a sample sales order in Odoo. Purchasing products from suppliers Purchase Orders, often known as PO, are documents that a business generates when they purchase products from a vendor. The Purchase Order module in an ERP system will typically include management of vendors (also called suppliers) as well as management of the products that the vendor carries. Much like sales order quotes, a purchase order system will allow a purchasing department to create draft purchase orders before they are finalized into a specific purchasing request. Often, a business will configure the Sales Order and Purchase Order modules to work together to streamline business operations. When a valid sales order is entered, most ERP systems will allow you to configure the system so that a purchase order can be automatically generated if the required products are not in stock to fulfill the sales order. ERP systems will allow you to set minimum quantities on-hand or order limits that will automatically generate purchase orders when inventory falls below a predetermined level. When properly configured, a purchase order system can save a significant amount of time in purchasing operations and assist in preventing supply shortages. Managing your accounts and financing in Odoo Accounting and finance modules integrate with an ERP system to organize and report business transactions. In many ERP systems, the accounting and finance module is known as GL for General Ledger. All accounting and finance modules are built around a structure known as the chart of accounts. The chart of accounts organizes groups of transactions into categories such as assets, liabilities, income, and expenses. ERP systems provide a lot of flexibility in defining the structure of your chart of accounts to meet the specific requirements for your business. Accounting transactions are grouped by date into periods (typically by month) for reporting purposes. These reports are most often known as financial statements. Common financial statements include balance sheets, income statements, cash flow statements, and statements of owner's equity. Handling your manufacturing operations The Manufacturing Resource Planning (MRP) module manages all the various business operations that go into the manufacturing of products. The fundamental transaction of an MRP module is a manufacturing order, which is also known as a production order in some ERP systems. A manufacturing order describes the raw products or subcomponents, steps, and routings required to produce a finished product. The raw products or subcomponents required to produce the finished product are typically broken down into a detailed list called a bill of materials or BOM. A BOM describes the exact quantities required of each component and are often used to define the raw material costs that go into manufacturing the final products for a company. Often an MRP module will incorporate several submodules that are necessary to define all the required operations. Warehouse management is used to define locations and sublocations to store materials and products as they move through the various manufacturing operations. For example, you may receive raw materials in one warehouse location, assemble those raw materials into subcomponents and store them in another location, then ultimately manufacture the end products and store them in a final location before delivering them to the customer. Managing customer relations in Odoo In today's business environment, quality customer service is essential to being competitive in most markets. A Customer Relationship Management (CRM) module assists a business in better handling the interactions they may have with each customer. Most CRM systems also incorporate a presales component that will manage opportunities, leads, and various marketing campaigns. Typically, a CRM system is utilized the most by the sales and marketing departments within a company. For this reason, CRM systems are often considered to be sales force automation tools or SFA tools. Sales personnel can set up appointments, schedule call backs, and employ tools to manage their communication. More modern CRM systems have started to incorporate social networking features to assist sales personnel in utilizing these newly emerging technologies. Configuring human resource applications in Odoo Human Resource modules, commonly known as HR, manage the workforce- or employee-related information in a business. Some of the processes ordinarily covered by HR systems are payroll, time and attendance, benefits administration, recruitment, and knowledge management. Increased regulations and complexities in payroll and benefits have led to HR modules becoming a major component of most ERP systems. Modern HR modules typically include employee kiosk functions to allow employees to self-administer many tasks such as putting in a leave request or checking on their available vacation time. Finding additional modules for your business requirements In addition to core ERP modules, Odoo has many more official and community-developed modules available. At the time of this article's publication, the Odoo application repository had 1,348 modules listed for version 7! Many of these modules provide small enhancements to improve usability like adding payment type to a sales order. Other modules offer e-commerce integration or complete application solutions, such as managing a school or hospital. Here is a short list of the more common modules you may wish to include in an Odoo installation: Point of Sale Project Management Analytic Accounting Document Management System Outlook Plug-in Country-Specific Accounting Templates OpenOffice Report Designer You will be introduced to various Odoo modules that extend the functionality of the base Odoo system. You can find a complete list of Odoo modules at http://apps.Odoo.com/. This preceding screenshot shows the module selection page in Odoo. Getting quickly into Odoo Do you want to jump in right now and get a look at Odoo 7 without any complex installations? Well, you are lucky! You can access an online installation of Odoo, where you can get a peek at many of the core modules right from your web browser. The installation is shared publicly, so you will not want to use this for any sensitive information. It is ideal, however, to get a quick overview of the software and to get an idea for how the interface functions. You can access a trial version of Odoo at https://www.Odoo.com/start. Odoo – an open source ERP solution Odoo is a collection of business applications that are available under an open source license. For this reason, Odoo can be used without paying license fees and can be customized to suit the specific needs of a business. There are many advantages to open source software solutions. We will discuss some of these advantages shortly. Free your company from expensive software license fees One of the primary downsides of most ERP systems is they often involve expensive license fees. Increasingly, companies must pay these license fees on an annual basis just to receive bug fixes and product updates. Because ERP systems can require companies to devote great amounts of time and money for setup, data conversion, integration, and training, it can be very expensive, often prohibitively so, to change ERP systems. For this reason, many companies feel trapped as their current ERP vendors increase license fees. Choosing open source software solutions, frees a company from the real possibility that a vendor will increase license fees in the years ahead. Modify the software to meet your business needs With proprietary ERP solutions, you are often forced to accept the software solution the vendor provides chiefly "as is". While you may have customization options and can sometimes pay the company to make specific changes, you rarely have the freedom to make changes directly to the source code yourself. The advantages to having the source code available to enterprise companies can be very significant. In a highly competitive market, being able to develop solutions that improve business processes and give your company the flexibility to meet future demands can make all the difference. Collaborative development Open source software does not rely on a group of developers who work secretly to write proprietary code. Instead, developers from all around the world work together transparently to develop modules, prepare bug fixes, and increase software usability. In the case of Odoo, the entire source code is available on Launchpad.net. Here, developers submit their code changes through a structure called branches. Changes can be peer reviewed, and once the changes are approved, they are incorporated into the final source code product. Odoo – AGPL open source license The term open source covers a wide range of open source licenses that have their own specific rights and limitations. Odoo and all of its modules are released under the Affero General Public License (AGPL) version 3. One key feature of this license is that any custom-developed module running under Odoo must be released with the source code. This stipulation protects the Odoo community as a whole from developers who may have a desire to hide their code from everyone else. This may have changed or has been appended recently with an alternative license. You can find the full AGPL license at http://www.gnu.org/licenses/agpl-3.0.html. A real-world case study using Odoo The goal is to do more than just walk through the various screens and reports of Odoo. Instead, we want to give you a solid understanding of how you would implement Odoo to solve real-world business problems. For this reason, this article will present a real-life case study in which Odoo was actually utilized to improve specific business processes. Silkworm, Inc. – a mid-sized screen printing company Silkworm, Inc. is a highly respected mid-sized silkscreen printer in the Midwest that manufactures and sells a variety of custom apparel products. They have been kind enough to allow us to include some basic aspects of their business processes as a set of real-world examples implementing Odoo into a manufacturing operation. Using Odoo, we will set up the company records (or system) from scratch and begin by walking through their most basic sales order process, selling T-shirts. From there, we will move on to manufacturing operations, where custom art designs are developed and then screen printed onto raw materials for shipment to customers. We will come back to this real-world example so that you can see how Odoo can be used to solve real-world business solutions. Although Silkworm is actively implementing Odoo, Silkworm, Inc. does not directly endorse or recommend Odoo for any specific business solution. Every company must do their own research to determine whether Odoo is a good fit for their operation. Summary In this article, we have learned about the ERP system and common ERP modules. An introduction about Odoo and features of it. Resources for Article: Further resources on this subject: Getting Started with Odoo Development[article] Machine Learning in IPython with scikit-learn [article] Making Goods with Manufacturing Resource Planning [article]

(For more resources related to this topic, see here.) The latest version of the example code for this article can be found at http://github.com/CamelCookbook/camel-cookbook-examples. You can also download the example code files for all Packt books you have purchased from your account at https://www.packtpub.com. If you purchased this book elsewhere, you can visit https://www.packtpub.com/books/content/support and register to have the files e-mailed directly to you. In this article we will explore a number of ways in which Camel performs message content transformation: Let us first look at some important concepts regarding transformation of messages in Camel: Using the transform statement. This allows you to reference Camel Expression Language code within the route to do message transformations. Calling a templating component, such as Camel's XSLT or Velocity template style components. This will typically reference an external template resource that is used in transforming your message. Calling a Java method (for example, beanref), defined by you, within a Camel route to perform the transformation. This is a special case processor that can invoke any referenced Java object method. Camel's Type Converter capability that can automatically cast data from one type to another transparently within your Camel route. This capability is extensible, so you can add your own Type Converters. Camel's Data Format capability that allows us to use built-in, or add our own, higher order message format converters. A Camel Data Format goes beyond simple data type converters, which handle simple data type translations such as String to int, or File to String. Data Formats are used to translate between a low-level representation (XML) and a high-level one (Java objects). Other examples include encrypting/decrypting data, and compressing/decompressing data. For more, see http://camel.apache.org/data-format.html. A number of Camel architectural concepts are used throughout this article. Full details can be found at the Apache Camel website at http://camel.apache.org. The code for this article is contained within the camel-cookbook-transformation module of the examples. Transforming using a Simple Expression When you want to transform a message in a relatively straightforward way, you use Camel's transform statement along with one of the Expression Languages provided by the framework. For example, Camel's Simple Expression Language provides you with a quick, inline mechanism for straightforward transformations. This recipe will show you how to use Camel's Simple Expression Language to transform the message body. Getting ready The Java code for this recipe is located in the org.camelcookbook.transformation.simple package. Spring XML files are located under src/main/resources/META-INF/spring and are prefixed with simple. How to do it... In a Camel route, use a transform DSL statement containing the Expression Language code to do your transformation. In the XML DSL, this is written as follows: <route> <from uri="direct:start"/> <transform> <simple>Hello ${body}</simple> </transform> </route> In the Java DSL, the same route is expressed as: from("direct:start") .transform(simple("Hello ${body}")); In this example, the message transformation prefixes the incoming message with the phrase Hello using the Simple Expression Language. The processing step after the transform statement will see the transformed message content in the body of the exchange. How it works... Camel's Simple Expression Language is quite good at manipulating the String content through its access to all aspects of the message being processed, through its rich String and logical operators. The result of your Simple Expression becomes the new message body after the transform step. This includes predicates such as using Simple's logical operators, to evaluate a true or false condition; the results of that Boolean operation become the new message body containing a String: "true" or "false". The advantage of using a distinct transform step within a route, as opposed to embedding it within a processor, is that the logic is clearly visible to other programmers. Ensure that the expression embedded within your route is kept simple so as to not distract the next developer from the overall purpose of the integration. It is best to move more complex (or just lengthy) transformation logic into its own subroute, and invoke it using direct: or seda:. There's more... The transform statement will work with any Expression Language available in Camel, so if you need more powerful message processing capabilities you can leverage scripting languages such as Groovy or JavaScript (among many others) as well. The Transforming inline with XQuery recipe will show you how to use the XQuery Expression Language to do transformations on XML messages. See also Message Translator: http://camel.apache.org/message-translator.html Camel Expression capabilities: http://camel.apache.org/expression.html Camel Simple Expression Language: http://camel.apache.org/simple.html Languages supported by Camel: http://camel.apache.org/languages.html The Transforming inline with XQuery recipe   Transforming inline with XQuery Camel supports the use of Camel's XQuery Expression Language along with the transform statement as a quick and easy way to transform an XML message within a route. This recipe will show you how to use an XQuery Expression to do in-route XML transformation. Getting ready The Java code for this recipe is located in the org.camelcookbook.transformation.xquery package. Spring XML files are located under src/main/resources/META-INF/spring and prefixed with xquery. To use the XQuery Expression Language, you need to add a dependency element for the camel-saxon library, which provides the implementation for the XQuery Expression Language. Add the following to the dependencies section of your Maven POM: <dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-saxon</artifactId> <version>${camel-version}</version> </dependency>   How to do it... In the Camel route, specify a transform statement followed by the XQuery Expression Language code to do your transformation. In the XML DSL, this is written as: <route> <from uri="direct:start"/> <transform> <xquery> <books>{ for $x in /bookstore/book where $x/price>30 order by $x/title return $x/title }</books> </xquery> </transform> </route> When using the XML DSL, remember to XML encode the XQuery embedded XML elements. Therefore, < becomes &lt; and > becomes &gt;. In the Java DSL, the same route is expressed as: from("direct:start") .transform(xquery("<books>{ for $x in /bookstore/book " + "where $x/price>30 order by $x/title " + "return $x/title }</books>")); Feed the following input XML message through the transformation: <bookstore> <book category="COOKING"> <title lang="en">Everyday Italian</title> <author>Giada De Laurentiis</author> <year>2005</year> <price>30.00</price> </book> <book category="CHILDREN"> <title lang="en">Harry Potter</title> <author>J K. Rowling</author> <year>2005</year> <price>29.99</price> </book> <book category="PROGRAMMING"> <title lang="en">Apache Camel Developer's Cookbook</title> <author>Scott Cranton</author> <author>Jakub Korab</author> <year>2013</year> <price>49.99</price> </book> <book category="WEB"> <title lang="en">Learning XML</title> <author>Erik T. Ray</author> <year>2003</year> <price>39.95</price> </book> </bookstore> The resulting message will be: <books> <title lang="en">Apache Camel Developer's Cookbook</title> <title lang="en">Learning XML</title> </books> The processing step after the transform statement will see the transformed message content in the body of the exchange. How it works... Camel's XQuery Expression Language is a good way to inline XML transformation code within your route. The result of the XQuery Expression becomes the new message body after the transform step. All of the message's body, headers, and properties are made available to the XQuery Processor, so you can reference them directly within your XQuery statement. This provides you with a powerful mechanism for transforming XML messages. If you are more comfortable with XSLT, take a look at the Transforming with XSLT recipe. In-lining the transformation within your integration route can sometimes be an advantage as you can clearly see what is being changed. However, when the transformation expression becomes so complex that it starts to overwhelm the integration route, you may want to consider moving the transformation expression outside of the route. See the Transforming using a Simple Expression recipe for another inline transformation example, and see the Transforming with XSLT recipe for an example of externalizing your transformation. You can fetch the XQuery Expression from an external file using Camel's resource reference syntax. To reference an XQuery file on the classpath you can specify: <transform> <xquery>resource:classpath:/path/to/myxquery.xml</xquery> </transform> This is equivalent to using XQuery as an endpoint: <to uri="xquery:classpath:/path/to/myxquery.xml"/>   There's more... The XQuery Expression Language allows you to pass in headers associated with the message. These will show up as XQuery variables that can be referenced within your XQuery statements. Consider, from the previous example, to allow the value of the books that are filtered to be passed in with the message body, that is, parameterize the XQuery, you can modify the XQuery statement as follows: <transform> <xquery> declare variable $in.headers.myParamValue as xs:integer external; <books value='{$in.headers.myParamValue}'&gt;{ for $x in /bookstore/book where $x/price>$in.headers.myParamValue order by $x/title return $x/title }&lt;/books&gt; </xquery> </transform> Message headers will be associated with an XQuery variable called in.headers.<name of header>. To use this in your XQuery, you need to explicitly declare an external variable of the same name and XML Schema (xs:) type as the value of the message header. The transform statement will work with any Expression Language enabled within Camel, so if you need more powerful message processing capabilities you can leverage scripting languages such as Groovy or JavaScript (among many others) as well. The Transforming using a Simple Expression recipe will show you how to use the Simple Expression Language to do transformations on String messages. See also Message Translator: http://camel.apache.org/message-translator.html Camel Expression capabilities: http://camel.apache.org/expression.html Camel XQuery Expression Language: http://camel.apache.org/xquery.html XQuery language: http://www.w3.org/XML/Query/ Languages supported by Camel: http://camel.apache.org/languages.html The Transforming with XSLT recipe The Transforming using a Simple Expression recipe   Transforming with XSLT When you want to transform an XML message using XSLT, use Camel's XSLT Component. This is similar to the Transforming inline with XQuery recipe except that there is no XSLT Expression Language, so it can only be used as an endpoint. This recipe will show you how to transform a message using an external XSLT resource. Getting ready The Java code for this recipe is located in the org.camelcookbook.transformation.xslt package. Spring XML files are located under src/main/resources/META-INF/spring and prefixed with xslt. How to do it... In a Camel route, add the xslt processor step into the route at the point where you want the XSLT transformation to occur. The XSLT file must be referenced as an external resource, and depending on where the file is located, prefixed with either classpath:(default if not using a prefix), file:, or http:. In the XML DSL, this is written as: <route> <from uri="direct:start"/> <to uri="xslt:book.xslt"/> </route> In the Java DSL, the same route is expressed as: from("direct:start") .to("xslt:book.xslt"); The next processing step in the route will see the transformed message content in the body of the exchange. How it works... The following example shows how the preceding steps will process an XML file. Consider the following input XML message: <bookstore> <book category="COOKING"> <title lang="en">Everyday Italian</title> <author>Giada De Laurentiis</author> <year>2005</year> <price>30.00</price> </book> <book category="CHILDREN"> <title lang="en">Harry Potter</title> <author>J K. Rowling</author> <year>2005</year> <price>29.99</price> </book> <book category="PROGRAMMING"> <title lang="en">Apache Camel Developer's Cookbook</title> <author>Scott Cranton</author> <author>Jakub Korab</author> <year>2013</year> <price>49.99</price> </book> <book category="WEB"> <title lang="en">Learning XML</title> <author>Erik T. Ray</author> <year>2003</year> <price>39.95</price> </book> </bookstore> Process this with the following XSLT contained in books.xslt: <?xml version="1.0" encoding="UTF-8"?> <xsl:stylesheet version="1.0" > <xsl:output omit-xml-declaration="yes"/> <xsl:template match="/"> <books> <xsl:apply-templates select="/bookstore/book/title[../price>30]"> <xsl:sort select="."/> </xsl:apply-templates> </books> </xsl:template> <xsl:template match="node()|@*"> <xsl:copy> <xsl:apply-templates select="node()|@*"/> </xsl:copy> </xsl:template> </xsl:stylesheet> The result will appear as follows: <books> <title lang="en">Apache Camel Developer's Cookbook</title> <title lang="en">Learning XML</title> </books> The Camel XSLT Processor internally runs the message body through a registered Java XML transformer using the XSLT file referenced by the endpoint. This processor uses Camel's Type Converter capabilities to convert the input message body type to one of the supported XML source models in the following order of priority: StAXSource (off by default; this can be enabled by setting allowStAX=true on the endpoint URI) SAXSource StreamSource DOMSource Camel's Type Converter can convert from most input types (String, File, byte[], and so on) to one of the XML source types for most XML content loaded through other Camel endpoints with no extra work on your part. The output data type for the message is, by default, a String, and is configurable using the output parameter on the xslt endpoint URI. There's more... The XSLT Processor passes in headers, properties, and parameters associated with the message. These will show up as XSLT parameters that can be referenced within your XSLT statements. You can pass in the names of the books as parameters to the XSLT template; to do so, modify the previous XLST as follows: <xsl:param name="myParamValue"/> <xsl:template match="/"> <books> <xsl:attribute name="value"> <xsl:value-of select="$myParamValue"/> </xsl:attribute> <xsl:apply-templates select="/bookstore/book/title[../price>$myParamValue]"> <xsl:sort select="."/> </xsl:apply-templates> </books> </xsl:template> The Exchange instance will be associated with a parameter called exchange; the IN message with a parameter called in; and the message headers, properties, and parameters will be associated XSLT parameters with the same name. To use these in your XSLT, you need to explicitly declare a parameter of the same name in your XSLT file. In the previous example, it is possible to use either a message header or exchange property called myParamValue. See also Message Translator: http://camel.apache.org/message-translator.html Camel XSLT Component: http://camel.apache.org/xslt Camel Type Converter: http://camel.apache.org/type-converter.html XSL working group: http://www.w3.org/Style/XSL/ The Transforming inline with XQuery recipe   Transforming from Java to XML with JAXB Camel's JAXB Component is one of a number of components that can be used to convert your XML data back and forth from Java objects. It provides a Camel Data Format that allows you to use JAXB annotated Java classes, and then marshal (Java to XML) or unmarshal (XML to Java) your data. JAXB is a Java standard for translating between XML data and Java that is used by creating annotated Java classes that bind, or map, to your XML data schema. The framework takes care of the rest. This recipe will show you how to use the JAXB Camel Data Format to convert back and forth from Java to XML. Getting ready The Java code for this recipe is located in the org.camelcookbook.transformation.jaxb package. The Spring XML files are located under src/main/resources/META-INF/spring and prefixed with jaxb. To use Camel's JAXB Component, you need to add a dependency element for the camel-jaxb library, which provides the implementation for the JAXB Data Format. Add the following to the dependencies section of your Maven POM: <dependency> lt;groupId>org.apache.camel</groupId> <artifactId>camel-jaxb</artifactId> <version>${camel-version}</version> lt;/dependency>   How to do it... The main steps for converting between Java and XML are as follows: Given a JAXB annotated model, reference that model within a named Camel Data Format. Use that named Data Format within your Camel route using the marshal and unmarshal DSL statements. Create an annotated Java model using standard JAXB annotations. There are a number of external tools that can automate this creation from existing XML or XSD (XML Schema) files: @XmlAccessorType(XmlAccessType.FIELD) @XmlType(name = "", propOrder = { "title", "author", "year", "price" } ) @XmlRootElement(name = "book") public class Book { @XmlElement(required = true) protected Book.Title title; @XmlElement(required = true) protected List<String> author; protected int year; protected double price; // getters and setters } Instantiate a JAXB Data Format within your Camel route that refers to the Java package(s) containing your JAXB annotated classes. In the XML DSL, this is written as: <camelContext > <dataFormats> <jaxb id="myJaxb" contextPath="org.camelcookbook .transformation.myschema"/> </dataFormats> <!-- route definitions here --> </camelContext> In the Java DSL, the Data Format is defined as: public class JaxbRouteBuilder extends RouteBuilder { @Override public void configure() throws Exception { DataFormat myJaxb= new JaxbDataFormat( "org.camelcookbook.transformation.myschema"); // route definitions here } } Reference the Data Format within your route, choosing marshal(Java to XML) or unmarshal(XML to Java) as appropriate. In the XML DSL, this routing logic is written as: <route> <from uri="direct:unmarshal"/> <unmarshal ref="myJaxb"/> </route> In the Java DSL, this is expressed as: from("direct:unmarshal").unmarshal(myJaxb);   How it works... Using Camel JAXB to translate your XML data back and forth to Java makes it much easier for the Java processors defined later on in your route to do custom message processing. This is useful when the built-in XML translators (for example, XSLT or XQuery) are not enough, or you just want to call existing Java code. Camel JAXB eliminates the boilerplate code from your integration flows by providing a wrapper around the standard JAXB mechanisms for instantiating the Java binding for the XML data. There's more... Camel JAXB works just fine with existing JAXB tooling like the maven-jaxb2-plugin plugin, which can automatically create JAXB-annotated Java classes from an XML Schema (XSD). See also Camel JAXB: http://camel.apache.org/jaxb.html Available Data Formats: http://camel.apache.org/data-format.html JAXB Specification: http://jcp.org/en/jsr/detail?id=222   Transforming from Java to JSON Camel's JSON Component is used when you need to convert your JSON data back and forth from Java. It provides a Camel Data Format that, without any requirement for an annotated Java class, allows you to marshal (Java to JSON) or unmarshal (JSON to Java) your data. There is only one step to using Camel JSON to marshal and unmarshal XML data. Within your Camel route, insert the marshal(Java to JSON), or unmarshal(JSON to Java) statement, and configure it to use the JSON Data Format. This recipe will show you how to use the camel-xstream library to convert from Java to JSON, and back. Getting ready The Java code for this recipe is located in the org.camelcookbook.transformation.json package. The Spring XML files are located under src/main/resources/META-INF/spring and prefixed with json. To use Camel's JSON Component, you need to add a dependency element for the camel-xstream library, which provides an implementation for the JSON Data Format using the XStream library. Add the following to the dependencies section of your Maven POM: <dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-xstream</artifactId> <version>${camel-version}</version> </dependency>   How to do it... Reference the Data Format within your route, choosing the marshal (Java to JSON), or unmarshal (JSON to Java) statement, as appropriate: In the XML DSL, this is written as follows: <route> <from uri="direct:marshal"/> <marshal> <json/> </marshal> <to uri="mock:marshalResult"/> </route> In the Java DSL, this same route is expressed as: from("direct:marshal") .marshal().json() .to("mock:marshalResult");   How it works... Using Camel JSON simplifies translating your data between JSON and Java. This is convenient when you are dealing with REST endpoints and need Java processors in Camel to do custom message processing later on in the route. Camel JSON provides a wrapper around the JSON libraries for instantiating the Java binding for the JSON data, eliminating more boilerplate code from your integration flows. There's more... Camel JSON works with the XStream library by default, and can be configured to use other JSON libraries, such as Jackson or GSon. These other libraries provide additional features, more customization, and more flexibility that can be leveraged by Camel. To use them, include their respective Camel components, for example, camel-jackson, and specify the library within the json element: <dataFormats> <json id="myJson" library="Jackson"/> </dataFormats>   See also Camel JSON: http://camel.apache.org/json.html Available Data Formats: http://camel.apache.org/data-format.html   Transforming from XML to JSON Camel provides an XML JSON Component that converts your data back and forth between XML and JSON in a single step, without an intermediate Java object representation. It provides a Camel Data Format that allows you to marshal (XML to JSON), or unmarshal (JSON to XML) your data. This recipe will show you how to use the XML JSON Component to convert from XML to JSON, and back. Getting ready Java code for this recipe is located in the org.camelcookbook.transformation.xmljson package. Spring XML files are located under src/main/resources/META-INF/spring and prefixed with xmljson. To use Camel's XML JSON Component, you need to add a dependency element for the camel-xmljson library, which provides an implementation for the XML JSON Data Format. Add the following to the dependencies section of your Maven POM: <dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-xmljson</artifactId> <version>${camel-version}</version> </dependency>   How to do it... Reference the xmljson Data Format within your route, choosing the marshal(XML to JSON), or unmarshal(JSON to XML) statement, as appropriate: In the XML DSL, this is written as follows: <route> <from uri="direct:marshal"/> <marshal> <xmljson/> </marshal> <to uri="mock:marshalResult"/> </route> In the Java DSL, this same route is expressed as: from("direct:marshal") .marshal().xmljson() .to("mock:marshalResult");   How it works... Using the Camel XML JSON Component simplifies translating your data between XML and JSON, making it convenient to use when you are dealing with REST endpoints. The XML JSON Data Format wraps around the Json-lib library, which provides the core translation capabilities, eliminating more boilerplate code from your integration flows. There's more... You may need to configure XML JSON if you want to fine-tune the output of your transformation. For example, consider the following JSON: [{"@category":"PROGRAMMING","title":{"@lang":"en","#text": "Apache Camel Developer's Cookbook"},"author":[ "Scott Cranton","Jakub Korab"],"year":"2013","price":"49.99"}] This will be converted as follows, by default, which may not be exactly what you want (notice the <a> and <e> elements): <?xml version="1.0" encoding="UTF-8"?> <a> <e category="PROGRAMMING"> <author> <e>Scott Cranton</e> <e>Jakub Korab</e> </author> <price>49.99</price> <title lang="en">Apache Camel Developer's Cookbook</title> <year>2013</year> </e> </a> To configure XML JSON to use <bookstore> as the root element instead of <a>, use <book> for the individual elements instead of <e>, and expand the multiple author values to use a sequence of <author> elements, you would need to tune the configuration of the Data Format before referencing it in your route. In the XML DSL, the definition of the Data Format and the route that uses it is written as follows: <dataFormats> <xmljson id="myXmlJson" rootName="bookstore" elementName="book" expandableProperties="author author"/> </dataFormats> <route> <from uri="direct:unmarshalBookstore"/> <unmarshal ref="myXmlJson"/> <to uri="mock:unmarshalResult"/> </route> In the Java DSL, the same thing is expressed as: XmlJsonDataFormat xmlJsonFormat = new XmlJsonDataFormat(); xmlJsonFormat.setRootName("bookstore"); xmlJsonFormat.setElementName("book"); xmlJsonFormat.setExpandableProperties( Arrays.asList("author", "author")); from("direct:unmarshalBookstore") .unmarshal(xmlJsonFormat) .to("mock:unmarshalBookstoreResult"); This will result in the previous JSON being unmarshalled as follows: <?xml version="1.0" encoding="UTF-8"?> <bookstore> <book category="PROGRAMMING"> <author>Scott Cranton</author> <author>Jakub Korab</author><price>49.99</price> <title lang="en">Apache Camel Developer's Cookbook</title> <year>2013</year> </book> </bookstore>   See also Camel XML JSON: http://camel.apache.org/xmljson.html Available Data Formats: http://camel.apache.org/data-format.html Json-lib: http://json-lib.sourceforge.net   Summary We saw how Apache Camel is flexible in allowing us to transform and convert messages in various formats. This makes Apache Camel an ideal choice for integrating different systems together. Resources for Article: Further resources on this subject: Drools Integration Modules: Spring Framework and Apache Camel [Article] Installing Apache Karaf [Article] Working with AMQP [Article]

Datasource definition A datasource is what JasperReports uses to obtain data for generating a report. Data can be obtained from databases, XML files, arrays of objects, collections of objects, and XML files. In this article, we will focus on using databases as a datasource. Database for our reports We will use a MySQL database to obtain data for our reports. The database is a subset of public domain data that can be downloaded from http://dl.flightstats.us. The original download is 1.3 GB, so we deleted most of the tables and a lot of data to trim the download size considerably. MySQL dump of the modified database can be found as part of code download at http://www.packtpub.com/files/code/8082_Code.zip. The flightstats database contains the following tables: aircraft aircraft_models aircraft_types aircraft_engines aircraft_engine_types The database structure can be seen in the following diagram: The flightstats database uses the default MyISAM storage engine for the MySQL RDBMS, which does not support referential integrity (foreign keys). That is why we don't see any arrows in the diagram indicating dependencies between the tables. Let's create a report that will show the most powerful aircraft in the database. Let's say, those with horsepower of 1000 or above. The report will show the aircraft tail number and serial number, the aircraft model, and the aircraft's engine model. The following query will give us the required results: SELECT a.tail_num, a.aircraft_serial, am.model as aircraft_model, ae.model AS engine_model FROM aircraft a, aircraft_models am, aircraft_engines ae WHERE a.aircraft_engine_code in (select aircraft_engine_code from aircraft_engines where horsepower >= 1000) AND am.aircraft_model_code = a.aircraft_model_code AND ae.aircraft_engine_code = a.aircraft_engine_code The above query retrieves the following data from the database: Generating database reports There are two ways to generate database reports—either by embedding SQL queries into the JRXML report template or by passing data from the database to the compiled report through a datasource. We will discuss both of these techniques. We will first create the report by embedding the query into the JRXML template. Then, we will generate the same report by passing it through a datasource containing the database data. Embedding SQL queries into a report template JasperReports allows us to embed database queries into a report template. This can be achieved by using the <queryString> element of the JRXML file. The following example demonstrates this technique: <?xml version="1.0" encoding="UTF-8" ?> <jasperReport xsi_schemaLocation="http://jasperreports.sourceforge.net/jasperreports http://jasperreports.sourceforge.net/xsd/jasperreport.xsd" name="DbReport"> <queryString> <![CDATA[select a.tail_num, a.aircraft_serial, am.model as aircraft_model, ae.model as engine_model from aircraft a, aircraft_models am, aircraft_engines ae where a.aircraft_engine_code in ( select aircraft_engine_code from aircraft_engines where horsepower >= 1000) and am.aircraft_model_code = a.aircraft_model_code and ae.aircraft_engine_code = a.aircraft_engine_code]]> </queryString> <field name="tail_num" class="java.lang.String" /> <field name="aircraft_serial" class="java.lang.String" /> <field name="aircraft_model" class="java.lang.String" /> <field name="engine_model" class="java.lang.String" /> <pageHeader> <band height="30"> <staticText> <reportElement x="0" y="0" width="69" height="24" /> <textElement verticalAlignment="Bottom" /> <text> <![CDATA[Tail Number: ]]> </text> </staticText> <staticText> <reportElement x="140" y="0" width="79" height="24" /> <text> <![CDATA[Serial Number: ]]> </text> </staticText> <staticText> <reportElement x="280" y="0" width="69" height="24" /> <text> <![CDATA[Model: ]]> </text> </staticText> <staticText> <reportElement x="420" y="0" width="69" height="24" /> <text> <![CDATA[Engine: ]]> </text> </staticText> </band> </pageHeader> <detail> <band height="30"> <textField> <reportElement x="0" y="0" width="69" height="24" /> <textFieldExpression class="java.lang.String"> <![CDATA[$F{tail_num}]]> </textFieldExpression> </textField> <textField> <reportElement x="140" y="0" width="69" height="24" /> <textFieldExpression class="java.lang.String"> <![CDATA[$F{aircraft_serial}]]> </textFieldExpression> </textField> <textField> <reportElement x="280" y="0" width="69" height="24" /> <textFieldExpression class="java.lang.String"> <![CDATA[$F{aircraft_model}]]> </textFieldExpression> </textField> <textField> <reportElement x="420" y="0" width="69" height="24" /> <textFieldExpression class="java.lang.String"> <![CDATA[$F{engine_model}]]> </textFieldExpression> </textField> </band> </detail> </jasperReport> The <queryString> element is used to embed a database query into the report template. In the given code example, the <queryString> element contains the query wrapped in a CDATA block for execution. The <queryString> element has no attributes or subelements other than the CDATA block containing the query. Text wrapped inside an XML CDATA block is ignored by the XML parser. As seen in the given example, our query contains the > character, which would invalidate the XML block if it wasn't inside a CDATA block. A CDATA block is optional if the data inside it does not break the XML structure. However, for consistency and maintainability, we chose to use it wherever it is allowed in the example. The <field> element defines fields that are populated at runtime when the report is filled. Field names must match the column names or alias of the corresponding columns in the SQL query. The class attribute of the <field> element is optional; its default value is java.lang.String. Even though all of our fields are strings, we still added the class attribute for clarity. In the last example, the syntax to obtain the value of a report field is $F{field_name}, where field_name is the name of the field as defined. The next element that we'll discuss is the <textField> element. Text fields are used to display dynamic textual data in reports. In this case, we are using them to display the value of the fields. Like all the subelements of <band>, text fields must contain a <reportElement> subelement indicating the text field's height, width, and x, y coordinates within the band. The data that is displayed in text fields is defined by the <textFieldExpression> subelement of <textField>. The <textFieldExpresson> element has a single subelement, which is the report expression that will be displayed by the text field and wrapped in an XML CDATA block. In this example, each text field is displaying the value of a field. Therefore, the expression inside the <textFieldExpression> element uses the field syntax $F{field_name}, as explained before. Compiling a report containing a query is no different from compiling a report without a query. It can be done programmatically or by using the custom JasperReports jrc ANT task. Generating the report As we have mentioned previously, in JasperReports terminology, the action of generating a report from a binary report template is called filling the report. To fill a report containing an embedded database query, we must pass a database connection object to the report. The following example illustrates this process: package net.ensode.jasperbook; import java.sql.Connection; import java.sql.DriverManager; import java.sql.SQLException; import java.util.HashMap; import net.sf.jasperreports.engine.JRException; import net.sf.jasperreports.engine.JasperFillManager; public class DbReportFill { Connection connection; public void generateReport() { try { Class.forName("com.mysql.jdbc.Driver"); connection = DriverManager.getConnection("jdbc:mysql: //localhost:3306/flightstats?user=user&password=secret"); System.out.println("Filling report..."); JasperFillManager.fillReportToFile("reports/DbReport. jasper", new HashMap(), connection); System.out.println("Done!"); connection.close(); } catch (JRException e) { e.printStackTrace(); } catch (ClassNotFoundException e) { e.printStackTrace(); } catch (SQLException e) { e.printStackTrace(); } } public static void main(String[] args) { new DbReportFill().generateReport(); } } As seen in this example, a database connection is passed to the report in the form of a java.sql.Connection object as the last parameter of the static JasperFillManager.fillReportToFile() method. The first two parameters are as follows: a string (used to indicate the location of the binary report template or jasper file) and an instance of a class implementing the java.util.Map interface (used for passing additional parameters to the report). As we don't need to pass any additional parameters for this report, we used an empty HashMap. There are six overloaded versions of the JasperFillManager.fillReportToFile() method, three of which take a connection object as a parameter. For simplicity, our examples open and close database connections every time they are executed. It is usually a better idea to use a connection pool, as connection pools increase the performance considerably. Most Java EE application servers come with connection pooling functionality, and the commons-dbcp component of Apache Commons includes utility classes for adding connection pooling capabilities to the applications that do not make use of an application server. After executing the above example, a new report, or JRPRINT file is saved to disk. We can view it by using the JasperViewer utility included with JasperReports. In this example, we created the report and immediately saved it to disk. The JasperFillManager class also contains methods to send a report to an output stream or to store it in memory in the form of a JasperPrint object. Storing the compiled report in a JasperPrint object allows us to manipulate the report in our code further. We could, for example, export it to PDF or another format. The method used to store a report into a JasperPrint object is JasperFillManager.fillReport(). The method used for sending the report to an output stream is JasperFillManager.fillReportToStream(). These two methods accept the same parameters as JasperFillManager.fillReportToFile() and are trivial to use once we are familiar with this method. Refer to the JasperReports API for details. In the next example, we will fill our report and immediately export it to PDF by taking advantage of the net.sf.jasperreports.engine.JasperRunManager.runReportToPdfStream() method. package net.ensode.jasperbook; import java.io.IOException; import java.io.InputStream; import java.io.PrintWriter; import java.io.StringWriter; import java.sql.Connection; import java.sql.DriverManager; import java.util.HashMap; import javax.servlet.ServletException; import javax.servlet.ServletOutputStream; import javax.servlet.http.HttpServlet; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; import net.sf.jasperreports.engine.JasperRunManager; public class DbReportServlet extends HttpServlet { protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { Connection connection; response.setContentType("application/pdf"); ServletOutputStream servletOutputStream = response .getOutputStream(); InputStream reportStream = getServletConfig() .getServletContext().getResourceAsStream( "/reports/DbReport.jasper"); try { Class.forName("com.mysql.jdbc.Driver"); connection = DriverManager.getConnection("jdbc:mysql: //localhost:3306/flightstats?user=dbUser&password=secret"); JasperRunManager.runReportToPdfStream(reportStream, servletOutputStream, new HashMap(), connection); connection.close(); servletOutputStream.flush(); servletOutputStream.close(); } catch (Exception e) { // display stack trace in the browser StringWriter stringWriter = new StringWriter(); PrintWriter printWriter = new PrintWriter(stringWriter); e.printStackTrace(printWriter); response.setContentType("text/plain"); response.getOutputStream().print(stringWriter.toString()); } } } The only difference between static and dynamic reports is that for dynamic reports we pass a connection to the report for generating a database report. After deploying this servlet and pointing the browser to its URL, we should see a screen similar to the following screenshot: