How-To Tutorials - Programming

(For more resources related to this topic, see here.) In this article, we will learn how to work with Parse objects along with writing queries to set and get data from Parse. Every application has a different and specific Application ID associated with the Client Key, which remains same for all the applications of the same user. Parse is based on object-oriented principles. All the operations on Parse will be done in the form of objects. Parse saves your data in the form of objects you send, and helps you to fetch the data in the same format again. In this article, you will learn about objects and operations that can be performed on Parse objects. Parse objects All the data in Parse is saved in the form of PFObject. When you fetch any data from Parse by firing a query, the result will be in the form of PFObject. The detailed concept of PFObject is explained in the following section. PFObject Data stored on Parse is in the form of objects and it's developed around PFObject. PFObject can be defined as the key-value (dictionary format) pair of JSON data. The Parse data is schemaless, which means that you don't need to specify ahead of time what keys exist on each PFObject. Parse backend will take care of storing your data simply as a set of whatever key-value pair you want. Let's say you are tracking the visited count of the username with a user ID using your application. A single PFObject could contain the following code: visitedCount:1122, userName:"Jack Samuel", userId:1232333332 Parse accepts only string as Key. Values can be strings, numbers, Booleans, or even arrays, and dictionaries—anything that can be JSON encoded. The class name of PFObject is used to distinguish different sorts of data. Let's say you call the visitedCounts object of the user. Parse recommends you to write your class name NameYourClassLikeThis and nameYourKeysLikeThis just to provide readability to the code. As you have seen in the previous example, we have used visitedCounts to represent the visited count key. Operations on Parse objects You can perform save, update, and delete operations on Parse objects. Following is the detailed explanation of the operations that can be performed on Parse objects. Saving objects To save your User table on the Parse Cloud with additional fields, you need to follow the coding convention similar to the NSMutableDictionary method. After updating the data you have to call the saveInBackground method to save it on the Parse Cloud. Here is the example that explains how to save additional data on the Parse Cloud: PFObject *userObject = [PFObject currentUser];[userObject setObject:[NSNumber numberWithInt:1122]forKey:@"visitedCount"];[userObject setObject:@"Jack Samuel" forKey:@"userName"];[userObject setObject:@"1232333332" forKey:@"userId"];[userObject saveInBackground]; Just after executing the preceding piece of code, your data is saved on the Parse Cloud. You can check your data in Data Browser of your application on Parse. It should be something similar to the following line of code: objectId: "xWMyZ4YEGZ", visitedCount: 1122, userName: "JackSamuel", userId: "1232333332",createdAt:"2011-06-10T18:33:42Z", updatedAt:"2011-06-10T18:33:42Z" There are two things to note here: You don't have to configure or set up a new class called User before running your code. Parse will automatically create the class when it first encounters it. There are also a few fields you don't need to specify, those are provided as a convenience: objectId is a unique identifier for each saved object. createdAt and updatedAt represent the time that each object was created and last modified in the Parse Cloud. Each of these fields is filled in by Parse, so they don't exist on PFObject until a save operation has completed. You can provide additional logic after the success or failure of the callback operation using the saveInBackgroundWithBlock or saveInBackgroundWithTarget:selector: methods provided by Parse: [userObject saveInBackgroundWithBlock:^(BOOLsucceeded, NSError *error) {if (succeeded)NSLog(@"Success");elseNSLog(@"Error %@",error);}]; Fetching objects To fetch the saved data from the Parse Cloud is even easier than saving data. You can fetch the data from the Parse Cloud in the following way. You can fetch the complete object from its objectId using PFQuery. Methods to fetch data from the cloud are asynchronous. You can implement this either by using block-based or callback-based methods provided by Parse: PFQuery *query = [PFQuery queryWithClassName:@"GameScore"]; // 1[query getObjectInBackgroundWithId:@"xWMyZ4YEGZ" block:^(PFObject*gameScore, NSError *error) { //2// Do something with the returned PFObject in the gameScorevariable.int score = [[gameScore objectForKey:@"score"] intValue];NSString *playerName = [gameScore objectForKey:@"playerName"];//3BOOL cheatMode = [[gameScore objectForKey:@"cheatMode"]boolValue];NSLog(@"%@", gameScore);}];// The InBackground methods are asynchronous, so the code writtenafter this will be executed// immediately. The codes which are dependent on the query resultshould be moved// inside the completion block above. Lets analyze each line in here, as follows: Line 1: It creates a query object pointing to the class name given in the argument. Line 2: It calls an asynchronous method on the query object created in line 1 to download the complete object for objectId, provided as an argument. As we are using the block-based method, we can provide code inside the block, which will execute on success or failure. Line 3: It reads data from PFObject that we got in response to the query. Parse provides some common values of all Parse objects as properties: NSString *objectId = gameScore.objectId;NSDate *updatedAt = gameScore.updatedAt;NSDate *createdAt = gameScore.createdAt; To refresh the current Parse object, type: [myObject refresh]; This method can be called on any Parse object, which is useful when you want to refresh the data of the object. Let's say you want to re-authenticate a user, so you can call the refresh method on the user object to refresh it. Saving objects offline Parse provides you with the functions to save your data when the user is offline. So when the user is not connected to the Internet, the data will be saved locally in the objects, and as soon as the user is connected to the Internet, data will be saved automatically on the Parse Cloud. If your application is forcefully closed before establishing the connection, Parse will try again to save the object next time the application is opened. For such operations, Parse provides you with the saveEventually method, so that you will not lose any data even when the user is not connected to the Internet. Eventually all calls are executed in the order the request is made. The following code demonstrates the saveEventually call: // Create the object.PFObject *gameScore = [PFObject objectWithClassName:@"GameScore"];[gameScore setObject:[NSNumber numberWithInt:1337]forKey:@"score"];[gameScore setObject:@"Sean Plott" forKey:@"playerName"];[gameScore setObject:[NSNumber numberWithBool:NO]forKey:@"cheatMode"];[gameScore saveEventually]; Summary In this article, we explored Parse objects and the way to query the data available on Parse. We started by exploring Parse objects and the ways to save these objects on the cloud. Finally, we learned about the queries which will help us to fetch the saved data on Parse. Resources for Article: Further resources on this subject: New iPad Features in iOS 6 [Article] Creating a New iOS Social Project [Article] Installing Alfresco Software Development Kit (SDK) [Article]

(For more resources related to this topic, see here.) Event management An event in Qt is an object inherited from the abstract QEvent class which is a notification of something significant that has happened. Events become more useful in creating custom widgets on our own. An event can happen either within an application or as a result of an outside activity that the application needs to know about. When an event occurs, Qt creates an event object and notifies to the instance of an QObject class or one of its subclasses through their event() function. Events can be generated from both inside and outside the application. For instance, the QKeyEvent and QMouseEvent object represent some kind of keyboard and mouse interaction and they come from the window manager; the QTimerEvent objects are sent to QObject when one of its timers fires, and they usually come from the operating system; the QChildEvent objects are sent to QObject when a child is added or removed and they come from inside of your Qt application. The users of PySide usually get confused with events and signals. Events and signals are two parallel mechanisms used to accomplish the same thing. As a general difference, signals are useful when using a widget, whereas events are useful when implementing the widget. For example, when we are using a widget like QPushButton, we are more interested in its clicked() signal than in the low-level mouse press or key press events that caused the signal to be emitted. But if we are implementing the QPushButton class, we are more interested in the implementation of code for mouse and key events. Also, we usually handle events but get notified by signal emissions. Event loop All the events in Qt will go through an event loop. One main key concept to be noted here is that the events are not delivered as soon as they are generated; instead they're queued up in an event queue and processed later one-by-one. The event dispatcher will loop through this queue and dispatch these events to the target QObject and hence it is called an event loop. Qt's main event loop dispatcher, QCoreApplication.exec() will fetch the native window system events from the event queue and will process them, convert them into the QEvent objects, and send it to their respective target QObject. A simple event loop can be explained as described in the following pseudocode: while(application_is_active) { while(event_exists_in_event_queue) process_next_event(); wait_for_more_events(); } The Qt's main event loop starts with the QCoreApplication::exec() call and this gets blocked until QCoreApplication::exit() or QCoreApplication::quit() is called to terminate the loop. The wait_for_more_events() function blocks until some event is generated. This blocking is not a busy wait blocking and will not burn the CPU resources. Generally the event loop can be awaken by a window manager activity, socket activity, timers, or event posted by other threads. All these activities require a running event loop. It is more important not to block the event loop because when it is struck, widgets will not update themselves, timers won't fire, networking communications will slow down and stop. In short, your application will not respond to any external or internal events and hence it is advised to quickly react to events and return to the event loop as soon as possible. Event processing Qt offers five methods to do event processing. They are: By re-implementing a specific event handler like keyPressEvent(), paintEvent() By re-implementing the QObject::event() class Installing an event filter on a single QObject Installing an event filter on the QApplication object Subclassing QApplication and re-implementing notify() Generally, this can be broadly divided into re-implementing event handlers and installing event filters. We will see each of them in little detail. Reimplementing event handlers We can implement the task at hand or control a widget by reimplementing the virtual event handling functions. The following example will explain how to reimplement a few most commonly used events, a key press event, a mouse double-click event, and a window resize event. We will have a look at the code first and defer the explanation after the code: # Import necessary modules import sys from PySide.QtGui import * from PySide.QtCore import * # Our main widget class class MyWidget(QWidget): # Constructor function def __init__(self): QWidget.__init__(self) self.setWindowTitle("Reimplementing Events") self.setGeometry(300, 250, 300, 100) self.myLayout = QVBoxLayout() self.myLabel = QLabel("Press 'Esc' to close this App") self.infoLabel = QLabel() self.myLabel.setAlignment(Qt.AlignCenter) self.infoLabel.setAlignment(Qt.AlignCenter) self.myLayout.addWidget(self.myLabel) self.myLayout.addWidget(self.infoLabel) self.setLayout(self.myLayout) # Function reimplementing Key Press, Mouse Click and Resize Events def keyPressEvent(self, event): if event.key() == Qt.Key_Escape: self.close() def mouseDoubleClickEvent(self, event): self.close() def resizeEvent(self, event): self.infoLabel.setText("Window Resized to QSize(%d, %d)" % (event.size().width(), event.size().height())) if __name__ =='__main__': # Exception Handling try: myApp = QApplication(sys.argv) myWidget = MyWidget() myWidget.show() myApp.exec_() sys.exit(0) except NameError: print("Name Error:", sys.exc_info()[1]) except SystemExit: print("Closing Window...") except Exception: print(sys.exc_info()[1]) In the preceding code, the keyPressEvent() function reimplements the event generated as a result of pressing a key. We have implemented in such a way that the application closes when the Esc key is pressed. On running this code, we would get a output similar to the one shown in the following screenshot: The application will be closed if you press the Esc key. The same functionality is implemented on a mouse double-click event. The third event is a resize event. This event gets triggered when you try to resize the widget. The second line of text in the window will show the size of the window in (width, height) format. You could witness the same on resizing the window. Similar to keyPressEvent(), we could also implement keyReleaseEvent() that would be triggered on release of the key. Normally, we are not very interested in the key release events except for the keys where it is important. The specific keys where the release event holds importance are the modifier keys such as Ctrl, Shift, and Alt. These keys are called modifier keys and can be accessed using QKeyEvent::modifiers. For example, the key press of a Ctrl key can be checked using Qt.ControlModifier. The other modifiers are Qt.ShiftModifier and Qt.AltModifier. For instance, if we want to check the press event of combination of Ctrl + PageDown key, we could have the check as: if event.key() == Qt.Key_PageDown and event.modifiers() == Qt.ControlModifier: print("Ctrl+PgDn Key is pressed") Before any particular key press or mouse click event handler function, say, for example, keyPressEvent() is called, the widget's event() function is called first. The event() method may handle the event itself or may delegate the work to a specific event handler like resizeEvent() or keyPressEvent(). The implementation of the event() function is very helpful in some special cases like the Tab key press event. In most cases, the widget with the keyboard focuses the event() method will call setFocus() on the next widget in the tab order and will not pass the event to any of the specific handlers. So we might have to re-implement any specific functionality for the Tab key press event in the event() function. This behavior of propagating the key press events is the outcome of Qt's Parent-Child hierarchy. The event gets propagated to its parent or its grand-parent and so on if it is not handled at any particular level. If the top-level widget also doesn't handle the event it is safely ignored. The following code shows an example for reimplementing the event() function: class MyWidget(QWidget): # Constructor function def __init__(self): QWidget.__init__(self) self.setWindowTitle("Reimplementing Events") self.setGeometry(300, 250, 300, 100) self.myLayout = QVBoxLayout() self.myLabel1 = QLabel("Text 1") self.myLineEdit1 = QLineEdit() self.myLabel2 = QLabel("Text 2") self.myLineEdit2 = QLineEdit() self.myLabel3 = QLabel("Text 3") self.myLineEdit3 = QLineEdit() self.myLayout.addWidget(self.myLabel1) self.myLayout.addWidget(self.myLineEdit1) self.myLayout.addWidget(self.myLabel2) self.myLayout.addWidget(self.myLineEdit2) self.myLayout.addWidget(self.myLabel3) self.myLayout.addWidget(self.myLineEdit3) self.setLayout(self.myLayout) # Function reimplementing event() function def event(self, event): if event.type()== QEvent.KeyRelease and event.key()== Qt.Key_Tab: self.myLineEdit3.setFocus() return True return QWidget.event(self,event) In the preceding example, we try to mask the default behavior of the Tab key. If you haven't implemented the event() function, pressing the Tab key would have set focus to the next available input widget. You will not be able to detect the Tab key press in the keyPress() function as described in the previous examples, since the key press is never passed to them. Instead, we have to implement it in the event() function. If you execute the preceding code, you would see that every time you press the Tab key the focus will be set into the third QLineEdit widget of the application. Inside the event() function, it is more important to return the value from the function. If we have processed the required operation, True is returned to indicate that the event is handled successfully, else, we pass the event handling to the parent class's event() function. Installing event filters One of the interesting and notable features of Qt's event model is to allow a QObject instance to monitor the events of another QObject instance before the latter object is even notified of it. This feature is very useful in constructing custom widgets comprising of various widgets altogether. Consider that you have a requirement to implement a feature in an internal application for a customer such that pressing the Enter key must have to shift the focus to next input widget. One way to approach the problem is to reimplement the keyPressEvent() function for all the widgets present in the custom widget. Instead, this can be achieved by reimplementing the eventFilter() function for the custom widget. If we implement this, the events will first be passed on to the custom widget's eventFilter() function before being passed on to the target widget. An example is implemented as follows: def eventFilter(self, receiver, event): if(event.type() == QEvent.MouseButtonPress): QMessageBox.information(None,"Filtered Mouse Press Event!!",'Mouse Press Detected') return True return super(MyWidget,self).eventFilter(receiver, event) Remember to return the result of event handling, or pass it on to the parent's eventFilter() function. To invoke eventFilter(), it has to be registered as follows in the constructor function: self.installEventFilter(self) The event filters can also be implemented for the QApplication as a whole. This is left as an exercise for you to discover. Reimplementing the notify() function The final way of handling events is to reimplement the notify() function of the QApplication class. This is the only way to get all the events before any of the event filters discussed previously are notified. The event gets notified to this function first before it gets passed on to the event filters and specific event functions. The use of notify() and other event filters are generally discouraged unless it is absolutely necessary to implement them because handling them at top level might introduce unwanted results, and we might end up in handling the events that we don't want to. Instead, use the specific event functions to handle events. The following code excerpt shows an example of re-implementing the notify() function: class MyApplication(QApplication): def __init__(self, args): super(MyApplication, self).__init__(args) def notify(self, receiver, event): if (event.type() == QEvent.KeyPress): QMessageBox.information(None, "Received Key Release EVent", "You Pressed: "+ event.text()) return super(MyApplication, self).notify(receiver, event) Signals and slots The fundamental part of any GUI program is the communication between the objects. Signals and slots provide a mechanism to define this communication between the actions happened and the result proposed for the respective action. Prior to Qt's modern implementation of signal/slot mechanism, older toolkits achieve this kind of communication through callbacks. A callback is a pointer to a function, so if you want a processing function to notify about some event you pass a pointer to another function (the callback) to the processing function. The processing function then calls the callback whenever appropriate. This mechanism does not prove useful in the later advancements due to some flaws in the callback implementation. A signal is an observable event, or at least notification that the event has happened. A slot is a potential observer, more usually a function that is called. In order to establish communication between them, we connect a signal to a slot to establish the desired action. However, we have already seen the concept of connecting a signal to a slot in the earlier chapters while designing the text editor application. Those implementations handle and connect different signals to different objects. However, we may have different combinations as defined in the bullet points: One signal can be connected to many slots Many signals can be connected to the same slot A signal can be connected to other signals Connections can be removed PySide offers various predefined signals and slots such that we can connect a predefined signal to a predefined slot and do nothing else to achieve what we want. However, it is also possible to define our own signals and slots. Whenever a signal is emitted, Qt will simply throw it away. We can define the slot to catch and notice the signal that is being emitted. The first code excerpt that follows this text will be an example for connecting predefined signals to predefined slots and the latter will discuss the custom user defined signals and slots. The first example is a simple EMI calculator application that takes the Loan Amount, Rate of Interest, and Number of Years as its input, and calculates the EMI per month and displays it to the user. To start with, we set in a layout the components required for the EMI calculator application. The Amount will be a text input from the user. The rate of years will be taken from a spin box input or a dial input. A spin box is a GUI component which has its minimum and maximum value set, and the value can be modified using the up and down arrow buttons present at its side. The dial represents a clock like widget whose values can be changed by dragging the arrow. The Number of Years value is taken by a spin box input or a slider input: class MyWidget(QWidget): def __init__(self): QWidget.__init__(self) self.amtLabel = QLabel('Loan Amount') self.roiLabel = QLabel('Rate of Interest') self.yrsLabel = QLabel('No. of Years') self.emiLabel = QLabel('EMI per month') self.emiValue = QLCDNumber() self.emiValue.setSegmentStyle(QLCDNumber.Flat) self.emiValue.setFixedSize(QSize(130,30)) self.emiValue.setDigitCount(8) self.amtText = QLineEdit('10000') self.roiSpin = QSpinBox() self.roiSpin.setMinimum(1) self.roiSpin.setMaximum(15) self.yrsSpin = QSpinBox() self.yrsSpin.setMinimum(1) self.yrsSpin.setMaximum(20) self.roiDial = QDial() self.roiDial.setNotchesVisible(True) self.roiDial.setMaximum(15) self.roiDial.setMinimum(1) self.roiDial.setValue(1) self.yrsSlide = QSlider(Qt.Horizontal) self.yrsSlide.setMaximum(20) self.yrsSlide.setMinimum(1) self.calculateButton = QPushButton('Calculate EMI') self.myGridLayout = QGridLayout() self.myGridLayout.addWidget(self.amtLabel, 0, 0) self.myGridLayout.addWidget(self.roiLabel, 1, 0) self.myGridLayout.addWidget(self.yrsLabel, 2, 0) self.myGridLayout.addWidget(self.amtText, 0, 1) self.myGridLayout.addWidget(self.roiSpin, 1, 1) self.myGridLayout.addWidget(self.yrsSpin, 2, 1) self.myGridLayout.addWidget(self.roiDial, 1, 2) self.myGridLayout.addWidget(self.yrsSlide, 2, 2) self.myGridLayout.addWidget(self.calculateButton, 3, 1) self.setLayout(self.myGridLayout) self.setWindowTitle("A simple EMI calculator") Until now, we have set the components that are required for the application. Note that, the application layout uses a grid layout option. The next set of code is also defined in the contructor's __init__ function of the MyWidget class which will connect the different signals to slots. There are different ways by which you can use a connect function. The code explains the various options available: self.roiDial.valueChanged.connect(self.roiSpin.setValue) self.connect(self.roiSpin, SIGNAL("valueChanged(int)"), self.roiDial.setValue) In the first line of the previous code, we connect the valueChanged() signal of roiDial to call the slot of roiSpin, setValue(). So, if we change the value of roiDial, it emits a signal that connects to the roiSpin's setValue() function and will set the value accordingly. Here, we must note that changing either the spin or dial must change the other value because both represent a single entity. Hence, we induce a second line which calls roiDial's setValue() slot on changing the roiSpin's value. However, it is to be noted that the second form of connecting signals to slots is deprecated. It is given here just for reference and it is strongly discouraged to use this form. The following two lines of code execute the same for the number of years slider and spin: self.yrsSlide.valueChanged.connect(self.yrsSpin.setValue) self.connect(self.yrsSpin, SIGNAL("valueChanged(int)"), self.yrsSlide, SLOT("setValue(int)")) In order to calculate the EMI value, we connect the clicked signal of the push button to a function (slot) which calculates the EMI and displays it to the user: self.connect(self.calculateButton, SIGNAL("clicked()"), self.showEMI) The EMI calculation and display function is given for your reference: def showEMI(self): loanAmount = float(self.amtText.text()) rateInterest = float( float (self.roiSpin.value() / 12) / 100) noMonths = int(self.yrsSpin.value() * 12) emi = (loanAmount * rateInterest) * ( ( ( (1 + rateInterest) ** noMonths ) / ( ( (1 + rateInterest) ** noMonths ) - 1) )) self.emiValue.display(emi) self.myGridLayout.addWidget(self.emiLabel, 4, 0) self.myGridLayout.addWidget(self.emiValue, 4, 2) The sample output of the application is shown in the following screenshot: The EMI calculator application uses the predefined signals, say, for example, valueChanged(), clicked() and predefined slots, setValue(). However, the application also uses a user-defined slot showEMI() to calculate the EMI. As with slots, it is also possible to create a user-defined signal and emit it when required. The following program is an example for creating and emitting user-defined signals: import sys from PySide.QtCore import * # define a new slot that receives and prints a string def printText(text): print(text) class CustomSignal(QObject): # create a new signal mySignal = Signal(str) if __name__ == '__main__': try: myObject = CustomSignal() # connect signal and slot myObject.mySignal.connect(printText) # emit signal myObject.mySignal.emit("Hello, Universe!") except Exception: print(sys.exc_info()[1]) This is a very simple example of using custom signals. In the CustomSignal class, we create a signal named mySignal and we emit it in the main function. Also, we define that on emission of the signal mySignal, the printText() slot would be called. Many complex signal emissions can be built this way.

(For more resources related to this topic, see here.) In today's world, real-time information is continuously getting generated by applications (business, social, or any other type), and this information needs easy ways to be reliably and quickly routed to multiple types of receivers. Most of the time, applications that are producing information and applications that are consuming this information are well apart and inaccessible to each other. This, at times, leads to redevelopment of information of producers or consumers to provide an integration point between them. Therefore, a mechanism is required for seamless integration of information of producers and consumers to avoid any kind of rewriting of an application at either end. In the present era of big data, the first challenge is to collect the data and the second challenge is to analyze it. As it is a huge amount of data, the analysis typically includes the following and much more: User behavior data Application performance tracing Activity data in the form of logs Event messages Message publishing is a mechanism for connecting various applications with the help of messages that are routed between them, for example, by a message broker such as Kafka. Kafka is a solution to the real-time problems of any software solution, that is, to deal with real-time volumes of information and route it to multiple consumers quickly. Kafka provides seamless integration between information of producers and consumers without blocking the producers of the information, and without letting producers know who the final consumers are. Apache Kafka is an open source, distributed publish-subscribe messaging system, mainly designed with the following characteristics: Persistent messaging: To derive the real value from big data, any kind of information loss cannot be afforded. Apache Kafka is designed with O(1) disk structures that provide constant-time performance even with very large volumes of stored messages, which is in order of TB. High throughput: Keeping big data in mind, Kafka is designed to work on commodity hardware and to support millions of messages per second. Distributed: Apache Kafka explicitly supports messages partitioning over Kafka servers and distributing consumption over a cluster of consumer machines while maintaining per-partition ordering semantics. Multiple client support: Apache Kafka system supports easy integration of clients from different platforms such as Java, .NET, PHP, Ruby, and Python. Real time: Messages produced by the producer threads should be immediately visible to consumer threads; this feature is critical to event-based systems such as Complex Event Processing (CEP) systems. Kafka provides a real-time publish-subscribe solution, which overcomes the challenges of real-time data usage for consumption, for data volumes that may grow in order of magnitude, larger that the real data. Kafka also supports parallel data loading in the Hadoop systems. The following diagram shows a typical big data aggregation-and-analysis scenario supported by the Apache Kafka messaging system: At the production side, there are different kinds of producers, such as the following: Frontend web applications generating application logs Producer proxies generating web analytics logs Producer adapters generating transformation logs Producer services generating invocation trace logs At the consumption side, there are different kinds of consumers, such as the following: Offline consumers that are consuming messages and storing them in Hadoop or traditional data warehouse for offline analysis Near real-time consumers that are consuming messages and storing them in any NoSQL datastore such as HBase or Cassandra for near real-time analytics Real-time consumers that filter messages in the in-memory database and trigger alert events for related groups Need for Kafka A large amount of data is generated by companies having any form of web-based presence and activity. Data is one of the newer ingredients in these Internet-based systems. This data typically includes user-activity events corresponding to logins, page visits, clicks, social networking activities such as likes, sharing, and comments, and operational and system metrics. This data is typically handled by logging and traditional log aggregation solutions due to high throughput (millions of messages per second). These traditional solutions are the viable solutions for providing logging data to an offline analysis system such as Hadoop. However, the solutions are very limiting for building real-time processing systems. According to the new trends in Internet applications, activity data has become a part of production data and is used to run analytics at real time. These analytics can be: Search based on relevance Recommendations based on popularity, co-occurrence, or sentimental analysis Delivering advertisements to the masses Internet application security from spam or unauthorized data scraping Real-time usage of these multiple sets of data collected from production systems has become a challenge because of the volume of data collected and processed. Apache Kafka aims to unify offline and online processing by providing a mechanism for parallel load in Hadoop systems as well as the ability to partition real-time consumption over a cluster of machines. Kafka can be compared with Scribe or Flume as it is useful for processing activity stream data; but from the architecture perspective, it is closer to traditional messaging systems such as ActiveMQ or RabitMQ. Few Kafka usages Some of the companies that are using Apache Kafka in their respective use cases are as follows: LinkedIn (www.linkedin.com): Apache Kafka is used at LinkedIn for the streaming of activity data and operational metrics. This data powers various products such as LinkedIn news feed and LinkedIn Today in addition to offline analytics systems such as Hadoop. DataSift (www.datasift.com/): At DataSift, Kafka is used as a collector for monitoring events and as a tracker of users' consumption of data streams in real time. Twitter (www.twitter.com/): Twitter uses Kafka as a part of its Storm— a stream-processing infrastructure. Foursquare (www.foursquare.com/): Kafka powers online-to-online and online-to-offline messaging at Foursquare. It is used to integrate Foursquare monitoring and production systems with Foursquare, Hadoop-based offline infrastructures. Square (www.squareup.com/): Square uses Kafka as a bus to move all system events through Square's various datacenters. This includes metrics, logs, custom events, and so on. On the consumer side, it outputs into Splunk, Graphite, or Esper-like real-time alerting. The source of the above information is https: //cwiki. apache.org/confluence/display/KAFKA/Powered+By. Summary In this article, we have seen how companies are evolving the mechanism of collecting and processing application-generated data, and that of utilizing the real power of this data by running analytics over it. Resources for Article: Further resources on this subject: Apache Felix Gogo [Article] Hadoop and HDInsight in a Heartbeat [Article] Advanced Hadoop MapReduce Administration [Article]

Creating a torrent In order to share data, we first need to create and share a torrent. Go to File | Create New Torrent. We use this interface to create new torrents. Here we can select the source files we wish to share, the trackers to use, and configure other sharing behaviors. Click on the Add file / Add directory button. If you plan on sharing a file or sharing a directory, press the relevant button and find the data to share. If you wish to skip files in the directory you have chosen, add which files to skip in the Skip Files textbox. This textbox works by matching the filenames within a directory with a string, which can use wildcards (*) and pipes (|). Note If you want to match a specific file, enter the full filename (for example, filename.jpg). If you want to skip all .jpg files, enter *.jpg. If you want to skip all .jpg files and all .txt files, enter *.jpg|*.txt without spaces. Enter the trackers that this torrent will use. By default, uTorrent will enter trackers www.openbittorrent.com and www.publicbt.com, though you can add other trackers to this list if required. By adding more trackers to this list, the torrent will get more exposure to the potential participants. If you have the torrent contents accessible on a web server and want to list it as a backup location, enter the URL in the Web Seeds textbox. This option ensures the files inside the torrent are accessible even if no other participants are present to transfer data. Use the Comment textbox to enter any extra information that the users may require. Keep the Piece size as (auto detect) . This can be altered if desired, but it is better to leave it as it is, since the Piece size relates to how the files will be divided. If you manually choose 16 KB, the data will be divided into 16 KB chunks. Choosing the wrong piece size for the data size can make torrents inefficient, so it is better to let uTorrent decide. Enable the Start seeding checkbox. This option will make the torrent available to upload as soon as the torrent creation is complete. If you want this torrent to be private, enable the Private torrent checkbox. This will disable the DHT and PEX ( Peer Exchange ) protocols, making tracker the only method of interacting with this torrent.

(For more resources related to this topic, see here.) Drawing basics The screens of modern computers consist of a number of small squares, called pixels ( picture elements ). Each pixel can light in one color. You create pictures on the screen by changing the colors of the pixels. Graphics based on pixels is called raster graphics. Another kind of graphics is vector graphics, which is based on primitives such as lines and circles. Today, most computer screens are arrays of pixels and represent raster graphics. But images based on vector graphics (vector images) are still used in computer graphics. Vector images are drawn on raster screens using the rasterization procedure. The openFrameworks project can draw on the whole screen (when it is in fullscreen mode) or only in a window (when fullscreen mode is disabled). For simplicity, we will call the area where openFrameworks can draw, the screen . The current width and height of the screen in pixels may be obtained using the ofGetWidth() and ofGetHeight() functions. For pointing the pixels, openFrameworks uses the screen's coordinate system. This coordinate system has its origin on the top-left corner of the screen. The measurement unit is a pixel. So, each pixel on the screen with width w and height h pixels can be pointed by its coordinates (x, y), where x and y are integer values lying in the range 0 to w-1 and from 0 to h-1 respectively. In this article, we will deal with two-dimensional (2D) graphics, which is a number of methods and algorithms for drawing objects on the screen by specifying the two coordinates (x, y) in pixels. The other kind of graphics is three-dimensional (3D) graphics, which represents objects in 3D space using three coordinates (x, y, z) and performs rendering on the screen using some kind of projection of space (3D) to the screen (2D). The background color of the screen The drawing on the screen in openFrameworks should be performed in the testApp::draw() function. Before this function is called by openFrameworks, the entire screen is filled with a fixed color, which is set by the function ofSetBackground( r, g, b ). Here r, g, and b are integer values corresponding to red, green, and blue components of the background color in the range 0 to 255. Note that each of the ofSetBackground() function call fills the screen with the specified color immediately. You can make a gradient background using the ofBackgroundGradient() function. You can set the background color just once in the testApp::setup() function, but we often call ofSetBackground() in the beginning of the testApp::draw() function to not mix up the setup stage and the drawing stage. Pulsating background example You can think of ofSetBackground() as an opportunity to make the simplest drawings, as if the screen consists of one big pixel. Consider an example where the background color slowly changes from black to white and back using a sine wave. This is example 02-2D/01-PulsatingBackground. The project is based on the openFrameworks emptyExample example. Copy the folder with the example and rename it. Then fill the body of the testApp::draw() function with the following code: float time = ofGetElapsedTimef(); //Get time in seconds//Get periodic value in [-1,1],with wavelength equal to 1 second float value = sin( time * M_TWO_PI );//Map value from [-1,1] to [0,255] float v = ofMap( value, -1, 1, 0, 255 );ofBackground( v, v, v ); //Set background color This code gets the time lapsed from the start of the project using the ofGetElapsedTimef() function, and uses this value for computing value = sin( time * M_TWO_PI ). Here, M_TWO_PI is an openFrameworks constant equal to 2π; that is, approximately 6.283185. So, time * M_TWO_PI increases by 2π per second. The value 2π is equal to the period of the sine wave function, sin(). So, the argument of sin(...) will go through its wavelength in one second, hence value = sin(...) will run from -1 to 1 and back. Finally, we map the value to v, which changes in range from 0 to 255 using the ofMap() function, and set the background to a color with red, green, and blue components equal to v. Run the project; you will see how the screen color pulsates by smoothly changing its color from black to white and back. Replace the last line, which sets the background color to ofBackground( v, 0, 0 );, and the color will pulsate from black to red. Replace the argument of the sin(...) function to the formula time * M_TWO_PI * 2 and the speed of the pulsating increases by two times. We will return to background in the Drawing with an uncleared background section. Now we will consider how to draw geometric primitives. Geometric primitives In this article we will deal with 2D graphics. 2D graphics can be created in the following ways: Drawing geometric primitives such as lines, circles, and other curves and shapes like triangles and rectangles. This is the most natural way of creating graphics by programming. Generative art and creative coding projects are often based on this graphics method. We will consider this in the rest of the article. Drawing images lets you add more realism to the graphics. Setting the contents of the screen directly, pixel-by-pixel, is the most powerful way of generating graphics. But it is harder to use for simple things like drawing curves. So, such method is normally used together with both of the previous methods. A somewhat fast technique for drawing a screen pixel-by-pixel consists of filling an array with pixels colors, loading it in an image, and drawing the image on the screen. The fastest, but a little bit harder technique, is using fragment shaders. openFrameworks has the following functions for drawing primitives: ofLine( x1, y1, x2, y2 ): This function draws a line segment connecting points (x1, y1) and (x2, y2) ofRect( x, y, w, h ): This function draws a rectangle with the top-left corner (x, y), width w, and height h ofTriangle( x1, y1, x2, y2, x3, y3 ): This function draws a triangle with vertices (x1, y1), (x2, y2), and (x3, y3) ofCircle( x, y, r ): This function draws a circle with center (x, y) and radius r openFrameworks has no special function for changing the color of a separate pixel. To do so, you can draw the pixel (x, y) as a rectangle with width and height equal to 1 pixel; that is, ofRect( x, y, 1, 1 ). This is a very slow method, but we sometimes use it for educational and debugging purposes. All the coordinates in these functions are float type. Although the coordinates (x, y) of a particular pixel on the screen are integer values, openFrameworks uses float numbers for drawing geometric primitives. This is because a video card can draw objects with the float coordinates using modeling, as if the line goes between pixels. So the resultant picture of drawing with float coordinates is smoother than with integer coordinates. Using these functions, it is possible to create simple drawings. The simplest example of a flower Let's consider the example that draws a circle, line, and two triangles, which forms the simplest kind of flower. This is example 02-2D/02-FlowerSimplest. This example project is based on the openFrameworks emptyExample project. Fill the body of the testApp::draw() function with the following code: ofBackground( 255, 255, 255 ); //Set white background ofSetColor( 0, 0, 0 ); //Set black colorofCircle( 300, 100, 40 ); //Blossom ofLine( 300, 100, 300, 400 ); //Stem ofTriangle( 300, 270, 300, 300, 200, 220 ); //Left leaf ofTriangle( 300, 270, 300, 300, 400, 220 ); //Right leaf On running this code, you will see the following picture of the "flower": Controlling the drawing of primitives There are a number of functions for controlling the parameters for drawing primitives. ofSetColor( r, g, b ): This function sets the color of drawing primitives, where r, g, and b are integer values corresponding to red, green, and blue components of the color in the range 0 to 255. After calling ofSetColor(), all the primitives will be drawn using this color until another ofSetColor() calling. We will discuss colors in more detail in the Colors section. ofFill() and ofNoFill(): These functions enable and disable filling shapes like circles, rectangles, and triangles. After calling ofFill() or ofNoFill(), all the primitives will be drawn filled or unfilled until the next function is called. By default, the shapes are rendered filled with color. Add the line ofNoFill(); before ofCircle(...); in the previous example and you will see all the shapes unfilled, as follows: ofSetLineWidth( lineWidth ): This function sets the width of the rendered lines to the lineWidth value, which has type float. The default value is 1.0, and calling this function with larger values will result in thick lines. It only affects drawing unfilled shapes. The line thickness is changed up to some limit depending on the video card. Normally, this limit is not less than 8.0. Add the line ofSetLineWidth( 7 ); before the line drawing in the previous example, and you will see the flower with a thick vertical line, whereas all the filled shapes will remain unchanged. Note that we use the value 7; this is an odd number, so it gives symmetrical line thickening. Note that this method for obtaining thick lines is simple but not perfect, because adjacent lines are drawn quite crudely. For obtaining smooth thick lines, you should draw these as filled shapes. ofSetCircleResolution( res ): This function sets the circle resolution; that is, the number of line segments used for drawing circles to res. The default value is 20, but with such settings only small circles look good. For bigger circles, it is recommended to increase the circle resolution; for example, to 40 or 60. Add the line ofSetCircleResolution( 40 ); before ofCircle(...); in the previous example and you will see a smoother circle. Note that a large res value can decrease the performance of the project, so if you need to draw many small circles, consider using smaller res values. ofEnableSmoothing() and ofDisableSmoothing(): These functions enable and disable line smoothing. Such settings can be controlled by your video card. In our example, calling these functions will not have any effect. Performance considerations The functions discussed work well for drawings containing not more than a 1000 primitives. When you draw more primitives, the project's performance can decrease (it depends on your video card). The reason is that each command such as ofSetColor() or ofLine() is sent to drawing separately, which takes time. So, for drawing 10,000, 100,000, or even 1 million primitives, you should use advanced methods, which draw many primitives at once. In openFrameworks, you can use the ofMesh and ofVboMesh classes for this. Using ofPoint Maybe you noted a problem when considering the preceding flower example: drawing primitives by specifying the coordinates of all the vertices is a little cumbersome. There are too many numbers in the code, so it is hard to understand the relation between primitives. To solve this problem, we will learn about using the ofPoint class and then apply it for drawing primitives using control points. ofPoint is a class that represents the coordinates of a 2D point. It has two main fields: x and y, which are float type. Actually, ofPoint has the third field z, so ofPoint can be used for representing 3D points too. If you do not specify z, it sets to zero by default, so in this case you can think of ofPoint as a 2D point indeed. Operations with points To represent some point, just declare an object of the ofPoint class. ofPoint p; To initialize the point, set its coordinates. p.x = 100.0; p.y = 200.0; Or, alternatively, use the constructor. p = ofPoint( 100.0, 200.0 ); You can operate with points just as you do with numbers. If you have a point q, the following operations are valid: p + q or p - q provides points with coordinates (p.x + q.x, p.y + q.y) or (p.x - q.x, p.y - q.y) p * k or p / k, where k is the float value, provides the points (p.x * k, p.y * k) or (p.x / k, p.y / k) p += q or p -= q adds or subtracts q from p There are a number of useful functions for simplifying 2D vector mathematics, as follows: p.length(): This function returns the length of the vector p, which is equal to sqrt( p.x * p.x + p.y * p.y ). p.normalize(): This function normalizes the point so it has the unit length p = p / p.length(). Also, this function handles the case correctly when p.length() is equal to zero. See the full list of functions for ofPoint in the libs/openFrameworks/math/ofVec3f.h file. Actually, ofPoint is just another name for the ofVec3f class, representing 3D vectors and corresponding functions. All functions' drawing primitives have overloaded versions working with ofPoint: ofLine( p1, p2 ) draws a line segment connecting the points p1 and p2 ofRect( p, w, h ) draws a rectangle with top-left corner p, width w, and height h ofTriangle( p1, p2, p3 ) draws a triangle with the vertices p1, p2, and p3 ofCircle( p, r ) draws a circle with center p and radius r Using control points example We are ready to solve the problem stated in the beginning of the Using ofPoint section. To avoid using many numbers in drawing code, we can declare a number of points and use them as vertices for primitive drawing. In computer graphics, such points are called control points . Let's specify the following control points for the flower in our simplest flower example: Now we implement this in the code. This is example 02-2D/03-FlowerControlPoints. Add the following declaration of control points in the testApp class declaration in the testApp.h file: ofPoint stem0, stem1, stem2, stem3, leftLeaf, rightLeaf; Then set values for points in the testApp::update() function as follows: stem0 = ofPoint( 300, 100 ); stem1 = ofPoint( 300, 270 ); stem2 = ofPoint( 300, 300 ); stem3 = ofPoint( 300, 400 ); leftLeaf = ofPoint( 200, 220 ); rightLeaf = ofPoint( 400, 220 ); Finally, use these control points for drawing the flower in the testApp::draw() function: ofBackground( 255, 255, 255 ); //Set white background ofSetColor( 0, 0, 0 ); //Set black colorofCircle ( stem0, 40 ); //Blossom ofLine( stem0, stem3 ); //Stem ofTriangle( stem1, stem2, leftLeaf ); //Left leaf ofTriangle( stem1, stem2, rightLeaf ); //Right leaf You will observe that when drawing with control points the code is much easier to understand. Furthermore, there is one more advantage of using control points: we can easily change control points' positions and hence obtain animated drawings. See the full example code in 02-2D/03-FlowerControlPoints. In addition to the already explained code, it contains a code for shifting the leftLeaf and rightLeaf points depending on time. So, when you run the code, you will see the flower with moving leaves. Coordinate system transformations Sometimes we need to translate, rotate, and resize drawings. For example, arcade games are based on the characters moving across the screen. When we perform drawing using control points, the straightforward solution for translating, rotating, and resizing graphics is in applying desired transformations to control points using corresponding mathematical formulas. Such idea works, but sometimes leads to complicated formulas in the code (especially when we need to rotate graphics). The more elegant solution is in using coordinate system transformations. This is a method of temporarily changing the coordinate system during drawing, which lets you translate, rotate, and resize drawings without changing the drawing algorithm. The current coordinate system is represented in openFrameworks with a matrix. All coordinate system transformations are made by changing this matrix in some way. When openFrameworks draws something using the changed coordinate system, it performs exactly the same number of computations as with the original matrix. It means that you can apply as many coordinate system transformations as you want without any decrease in the performance of the drawing. Coordinate system transformations are managed in openFrameworks with the following functions: ofPushMatrix(): This function pushes the current coordinate system in a matrix stack. This stack is a special container that holds the coordinate system matrices. It gives you the ability to restore coordinate system transformations when you do not need them. ofPopMatrix(): This function pops the last added coordinate system from a matrix stack and uses it as the current coordinate system. You should take care to see that the number of ofPopMatrix() calls don't exceed the number of ofPushMatrix() calls. Though the coordinate system is restored before testApp::draw() is called, we recommend that the number of ofPushMatrix() and ofPopMatrix() callings in your project should be exactly the same. It will simplify the project's debugging and further development. ofTranslate( x, y ) or ofTranslate( p ): This function moves the current coordinate system at the vector (x, y) or, equivalently, at the vector p. If x and y are equal to zero, the coordinate system remains unchanged. ofScale( scaleX, scaleY ): This function scales the current coordinate system at scaleX in the x axis and at scaleY in the y axis. If both parameters are equal to 1.0, the coordinate system remains unchanged. The value -1.0 means inverting the coordinate axis in the opposite direction. ofRotate( angle ): This function rotates the current coordinate system around its origin at angle degrees clockwise. If the angle value is equal to 0, or k * 360 with k as an integer, the coordinate system remains unchanged. All transformations can be applied in any sequence; for example, translating, scaling, rotating, translating again, and so on. The typical usage of these functions is the following: Store the current transformation matrix using ofPushMatrix(). Change the coordinate system by calling any of these functions: ofTranslate(), ofScale(), or ofRotate(). Draw something. Restore the original transformation matrix using ofPopMatrix(). Step 3 can include steps 1 to 4 again. For example, for moving the origin of the coordinate system to the center of the screen, use the following code in testApp::draw(): ofPushMatrix(); ofTranslate( ofGetWidth() / 2, ofGetHeight() / 2 ); //Draw something ofPopMatrix(); If you replace the //Draw something comment to ofCircle( 0, 0, 100 );, you will see the circle in the center of the screen. This transformation significantly simplifies coding the drawings that should be located at the center of the screen. Now let's use coordinate system transformation for adding triangular petals to the flower. For further exploring coordinate system transformations.

(For more resources related to this topic, see here.) In the early days of Java, when projects were small and didn't have many external dependencies, developers tended to manage dependencies manually by copying the required JAR files in the lib folder and checking it in their SCM/VCS with their code. This is still followed by a lot of developers, even today. But due to the aforementioned issues, this is not an option for larger projects. In many enterprises, there are central servers, FTP, shared drives, and so on, which store the approved libraries for use and also internally released libraries. But managing and tracking them manually is never easy. They end up relying on scripts and build files. Maven came and standardized this process. Maven defines standards for the project format to define its dependencies, formats for repositories to store libraries, the automated process to fetch transitive dependencies, and much more. Most of the systems today either back onto Maven's dependency management system or on Ivy's, which can function in the same way, and also provides its own standards, which is heavily inspired by Maven. SBT uses Ivy in the backend for dependency management, but uses a custom DSL to specify the dependency. Quick introduction to Maven or Ivy dependency management Apache Maven is not a dependency management tool. It is a project management and a comprehension tool. Maven is configured using a Project Object Model (POM), which is represented in an XML file. A POM has all the details related to the project right from the basic ones, such as groupId, artifactId, version, and so on, to environment settings such as prerequisites, and repositories. Apache Ivy is a dependency management tool and a subproject of Apache Ant. Ivy integrates publicly available artifact repositories automatically. The project dependencies are declared using XML in a file called ivy.xml. This is commonly known as the Ivy file. Ivy is configured using a settings file. The settings file (ivysettings.xml) defines a set of dependency resolvers. Each resolver points to an Ivy file and/or artifacts. So, the configuration essentially indicates which resource should be used to resolve a module. How Ivy works The following diagram depicts the usual cycle of Ivy modules between different locations: The tags along the arrows are the Ivy commands that need to be run for that task, which are explained in detail in the following sections. Resolve Resolve is the phase where Ivy resolves the dependencies of a module by accessing the Ivy file defined for that module. For each dependency in the Ivy file, Ivy finds the module using the configuration. A module could be an Ivy file or artifact. Once a module is found, its Ivy file is downloaded to the Ivy cache. Then, Ivy checks for the dependencies of that module. If the module has dependencies on other modules, Ivy recursively traverses the graph of dependencies, handling conflicts simultaneously. After traversing the whole graph, Ivy downloads all the dependencies that are not already in the cache and have not been evicted by conflict management. Ivy uses a filesystem-based cache to avoid loading dependencies already available in the cache. In the end, an XML report of the dependencies of the module is generated in the cache. Retrieve Retrieve is the act of copying artifacts from the cache to another directory structure. The destination for the files to be copied is specified using a pattern. Before copying, Ivy checks if the files are not already copied to maximize performance. After dependencies have been copied, the build becomes independent of Ivy. Publish Ivy can then be used to publish the module to a repository. This can be done by manually running a task or from a continuous integration server. Dependency management in SBT In SBT, library dependencies can be managed in the following two ways: By specifying the libraries in the build definition By manually adding the JAR files of the library Manual addition of JAR files may seem simple in the beginning of a project. But as the project grows, it may depend on a lot of other projects, or the projects it depends on may have newer versions. These situations make handling dependencies manually a cumbersome task. Hence, most developers prefer to automate dependency management. Automatic dependency management SBT uses Apache Ivy to handle automatic dependency management. When dependencies are configured in this manner, SBT handles the retrieval and update of the dependencies. An update does not happen every time there is a change, since that slows down all the processes. To update the dependencies, you need to execute the update task. Other tasks depend on the output generated through the update. Whenever dependencies are modified, an update should be run for these changes to get reflected. There are three ways in which project dependencies can be specified. They are as follows: Declarations within the build definition Maven dependency files, that is, POM files Configuration and settings files used for Ivy Adding JAR files manually Declaring dependencies in the build definition The Setting key libraryDependencies is used to configure the dependencies of a project. The following are some of the possible syntaxes for libraryDependencies: libraryDependencies += groupID % artifactID % revision libraryDependencies += groupID %% artifactID % revision libraryDependencies += groupID % artifactID % revision % configuration libraryDependencies ++= Seq( groupID %% artifactID % revision, groupID %% otherID % otherRevision ) Let's explain some of these examples in more detail: groupID: This is the organization/group's ID by whom it was published artifactID: This is the project's name on which there is a dependency revision: This is the Ivy revision of the project on which there is a dependency configuration: This is the Ivy configuration for which we want to specify the dependency Notice that the first and second syntax are not the same. The second one has a %% symbol after groupID. This tells SBT to append the project's Scala version to artifactID. So, in a project with Scala Version 2.9.1, libraryDependencies ++= Seq("mysql" %% "mysql-connector-java" % "5.1.18") is equivalent to libraryDependencies ++= Seq("mysql" % "mysql-connector-java_2.9.1" % "5.1.18"). The %% symbol is very helpful for cross-building a project. Cross-building is the process of building a project for multiple Scala versions. SBT uses the crossScalaVersion key's value to configure dependencies for multiple versions of Scala. Cross-building is possible only for Scala Version 2.8.0 or higher. The %% symbol simply appends the current Scala version, so it should not be used when you know that there is no dependency for a given Scala version, although it is compatible with an older version. In such cases, you have to hardcode the version using the first syntax. Using the third syntax, we could add a dependency only for a specific configuration. This is very useful as some dependencies are not required by all configurations. For example, the dependency on a testing library is only for the test configuration. We could declare this as follows: libraryDependencies ++= Seq("org.specs2" % "specs2_2.9.1" % "1.12.3" % "test") We could also specify dependency for the provided scope (where the JDK or container provides the dependency at runtime).This scope is only available on compilation and test classpath, and is not transitive. Generally, servlet-api dependencies are declared in this scope: libraryDependencies += "javax.servlet" % "javax.servlet-api" % "3.0.1" % "provided" The revision does not have to be a single-fixed version, that is, it can be set with some constraints, and Ivy will select the one that matches best. For example, it could be latest integration or 12.0 or higher, or even a range of versions. A URL for the dependency JAR If the dependency is not published to a repository, you can also specify a direct URL to the JAR file: libraryDependencies += groupID %% artifactID % revision from directURL directURL is used only if the dependency cannot be found in the specified repositories and is not included in published metadata. For example: libraryDependencies += "slinky" % "slinky" % "2.1" from "http://slinky2.googlecode.com/svn/artifacts/2.1/slinky.jar" Extra attributes SBT also supports Ivy's extra attributes. To specify extra attributes, one could use the extra method. Consider that the project has a dependency on the following Ivy module: <ivy-module version ="2.0" > <info organization="packt" module = "introduction" e:media = "screen" status = "integration" e:codeWord = "PP1872"</ivy-module> A dependency on this can be declared by using the following: libraryDependencies += "packt" % "introduction" % "latest.integration" extra( "media"->"screen", "codeWord"-> "PP1872") The extra method can also be used to specify extra attributes for the current project, so that when it is published to the repository its Ivy file will also have extra attributes. An example for this is as follows: projectID << projectID {id => id extra( "codeWord"-> "PP1952")} Classifiers Classifiers ensure that the dependency being loaded is compatible with the platform for which the project is written. For example, to fetch the dependency relevant to JDK 1.5, use the following: libraryDependencies += "org.testng" % "testng" % "5.7" classifier "jdk15" We could also have multiple classifiers, as follows: libraryDependencies += "org.lwjgl.lwjgl" % "lwjgl-platform" % lwjglVersion classifier "natives-windows" classifier "natives-linux" classifier "natives-osx" Transitivity In logic and mathematics, a relationship between three elements is said to be transitive. If the relationship holds between the first and second elements and between the second and third elements, it implies that it also holds a relationship between the first and third elements. Relating this to the dependencies of a project, imagine that you have a project that depends on the project Foo for some of its functionality. Now, Foo depends on another project, Bar, for some of its functionality. If a change in the project Bar affects your project's functionality, then this implies that your project indirectly depends on project Bar. This means that your project has a transitive dependency on the project Bar. But if in this case a change in the project Bar does not affect your project's functionality, then your project does not depend on the project Bar. This means that your project does not have a dependency on the project Bar. SBT cannot know whether your project has a transitive dependency or not, so to avoid dependency issues, it loads the library dependencies transitively by default. In situations where this is not required for your project, you can disable it using intransitive() or notTransitive(). A common case where artifact dependencies are not required is in projects using the Felix OSGI framework (only its main JAR is required). The dependency can be declared as follows: libraryDependencies += "org.apache.felix" % "org.apache.felix.framework" % "1.8.0" intransitive() Or, it can be declared as follows: libraryDependencies += "org.apache.felix" % "org.apache.felix.framework" % "1.8.0" notTransitive() If we need to exclude certain transitive dependencies of a dependency, we could use the excludeAll or exclude method. libraryDependencies += "log4j" % "log4j" % "1.2.15" exclude("javax.jms", "jms")libraryDependencies += "log4j" % "log4j" % "1.2.15" excludeAll( ExclusionRule(organization = "com.sun.jdmk"), ExclusionRule(organization = "com.sun.jmx"), ExclusionRule(organization = "javax.jms") ) Although excludeAll provides more flexibility, it should not be used in projects that will be published in the Maven style as it cannot be represented in a pom.xml file. The exclude method is more useful in projects that require pom.xml when being published, since it requires both organizationID and name to exclude a module. Download documentation Generally, an IDE plugin is used to download the source and API documentation JAR files. However, one can configure SBT to download the documentation without using an IDE plugin. To download the dependency's sources, add withSources() to the dependency definition. For example: libraryDependencies += "org.apache.felix" % "org.apache.felix.framework" % "1.8.0" withSources() To download API JAR files, add withJavaDoc() to the dependency definition. For example: libraryDependencies += "org.apache.felix" % "org.apache.felix.framework" % "1.8.0" withSources() withJavadoc() The documentation downloaded like this is not transitive. You must use the update-classifiers task to do so. Dependencies using Maven files SBT can be configured to use a Maven POM file to handle the project dependencies by using the externalPom method. The following statements can be used in the build definition: externalPom(): This will set pom.xml in the project's base directory as the source for project dependencies externalPom(baseDirectory{base=>base/"myProjectPom"}): This will set the custom-named POM file myProjectPom.xml in the project's base directory as the source for project dependencies There are a few restrictions with using a POM file, as follows: It can be used only for configuring dependencies. The repositories mentioned in POM will not be considered. They need to be specified explicitly in the build definition or in an Ivy settings file. There is no support for relativePath in the parent element of POM and its existence will result in an error. Dependencies using Ivy files or Ivy XML Both Ivy settings and dependencies can be used to configure project dependencies in SBT through the build definition. They can either be loaded from a file or can be given inline in the build definition. The Ivy XML can be declared as follows: ivyXML := <dependencies> <dependency org="org.specs2" name="specs2" rev="1.12.3"></dependency></ dependencies> The commands to load from a file are as follows: externalIvySettings(): This will set ivysettings.xml in the project's base directory as the source for dependency settings. externalIvySettings(baseDirectory{base=>base/"myIvySettings"}): This will set the custom-named settings file myIvySettings.xml in the project's base directory as the source for dependency settings. externalIvySettingsURL(url("settingsURL")): This will set the settings file at settingsURL as the source for dependency settings. externalIvyFile(): This will set ivy.xml in the project's base directory as the source for dependency. externalIvyFile(baseDirectory(_/"myIvy"): This will set the custom-named settings file myIvy.xml in the project's base directory as the source for project dependencies. When using Ivy settings and configuration files, the configurations need to be mapped, because Ivy files specify their own configurations. So, classpathConfiguration must be set for the three main configurations. For example: classpathConfiguration in Compile := Compile classpathConfiguration in Test := Test classpathConfiguration in Runtime := Runtime Adding JAR files manually To handle dependencies manually in SBT, you need to create a lib folder in the project and add the JAR files to it. That is the default location where SBT looks for unmanaged dependencies. If you have the JAR files located in some other folder, you could specify that in the build definition. The key used to specify the source for manually added JAR files is unmanagedBase. For example, if the JAR files your project depends on are in project/extras/dependencies instead of project/lib, modify the value of unmanagedBase as follows: unmanagedBase <<= baseDirectory {base => base/"extras/dependencies"} Here, baseDirectory is the project's root directory. unmanagedJars is a task which lists the JAR files from the unmanagedBase directory. To see the list of JAR files in the interactive shell type, type the following: > show unmanaged-jars [info] ArrayBuffer() Or in the project folder, type: $ sbt show unmanaged-jars If you add a Spring JAR (org.springframework.aop-3.0.1.jar) to the dependencies folder, then the result of the previous command would be: > show unmanaged-jars [info] ArrayBuffer(Attributed(/home/introduction/extras/dependencies/ org.springframework.aop-3.0.1.jar)) It is also possible to specify the path(s) of JAR files for different configurations using unmanagedJars. In the build definition, the unmanagedJars task may need to be replaced when the jars are in multiple directories and other complex cases. unmanagedJars in Compile += file("/home/downloads/ org.springframework.aop-3.0.1.jar") Resolvers Resolvers are alternate resources provided for the projects on which there is a dependency. If the specified project's JAR is not found in the default repository, these are tried. The default repository used by SBT is Maven2 and the local Ivy repository. The simplest ways of adding a repository are as follows: resolvers += name at location. For example: resolvers += "releases" at "http://oss.sonatype.org/content/ repositories/releases" resolvers ++= Seq (name1 at location1, name2 at location2). For example: resolvers ++= Seq("snapshots" at "http://oss.sonatype.org/ content/repositories/snapshots", "releases" at "http://oss.sonatype.org/content/repositories/releases") resolvers := Seq (name1 at location1, name2 at location2). For example: resolvers := Seq("sgodbillon" at "https://bitbucket.org/ sgodbillon/repository/raw/master/snapshots/", "Typesafe backup repo" at " http://repo.typesafe.com/typesafe/repo/", "Maven repo1" at "http://repo1.maven.org/") ) You can also add their own local Maven repository as a resource using the following syntax: resolvers += "Local Maven Repository" at "file://"+Path.userHome.absolutePath+"/.m2/repository" An Ivy repository of the file types URL, SSH, or SFTP can also be added as resources using sbt.Resolver. Note that sbt.Resolver is a class with factories for interfaces to Ivy repositories that require a hostname, port, and patterns. Let's see how to use the Resolver class. For filesystem repositories, the following line defines an atomic filesystem repository in the test directory of the current working directory: resolvers += Resolver.file ("my-test-repo", file("test")) transactional() For URL repositories, the following line defines a URL repository at http://example.org/repo-releases/: resolvers += Resolver.url(" my-test-repo", url("http://example.org/repo-releases/")) The following line defines an Ivy repository at http://joscha.github.com/play-easymail/repo/releases/: resolvers += Resolver.url("my-test-repo", url("http://joscha.github.com/play-easymail/repo/releases/")) (Resolver.ivyStylePatterns) For SFTP repositories, the following line defines a repository that is served by SFTP from the host example.org: resolvers += Resolver.sftp(" my-sftp-repo", "example.org") The following line defines a repository that is served by SFTP from the host example.org at port 22: resolvers += Resolver.sftp("my-sftp-repo", "example.org", 22) The following line defines a repository that is served by SFTP from the host example.org with maven2/repo-releases/ as the base path: resolvers += Resolver.sftp("my-sftp-repo", "example.org", "maven2/repo-releases/") For SSH repositories, the following line defines an SSH repository with user-password authentication: resolvers += Resolver.ssh("my-ssh-repo", "example.org") as("user", "password") The following line defines an SSH repository with an access request for the given user. The user will be prompted to enter the password to complete the download. resolvers += Resolver.ssh("my-ssh-repo", "example.org") as("user") The following line defines an SSH repository using key authentication: resolvers += { val keyFile: File = ... Resolver.ssh("my-ssh-repo", "example.org") as("user", keyFile, "keyFilePassword") } The next line defines an SSH repository using key authentication where no keyFile password is required to be prompted for before download: resolvers += Resolver.ssh("my-ssh-repo", "example.org") as("user", keyFile) The following line defines an SSH repository with the permissions. It is a mode specification such as chmod: resolvers += Resolver.ssh("my-ssh-repo", "example.org") withPermissions("0644") SFTP authentication can be handled in the same way as shown for SSH in the previous examples. Ivy patterns can also be given to the factory methods. Each factory method uses a Patterns instance which defines the patterns to be used. The default pattern passed to the factory methods gives the Maven-style layout. To use a different layout, provide a Patterns object describing it. The following are some examples that specify custom repository layouts using patterns: resolvers += Resolver.url("my-test-repo", url)( Patterns("[organisation]/[module]/ [revision]/[artifact].[ext]") ) You can specify multiple patterns or patterns for the metadata and artifacts separately. For filesystem and URL repositories, you can specify absolute patterns by omitting the base URL, passing an empty patterns instance, and using Ivy instances and artifacts. resolvers += Resolver.url("my-test-repo") artifacts "http://example.org/[organisation]/[module]/ [revision]/[artifact].[ext]" When you do not need the default repositories, you must override externalResolvers. It is the combination of resolvers and default repositories. To use the local Ivy repository without the Maven repository, define externalResolvers as follows: externalResolvers <<= resolvers map { rs => Resolver.withDefaultResolvers(rs, mavenCentral = false) } Summary In this article, we have seen how dependency management tools such as Maven and Ivy work and how SBT handles project dependencies. This article also talked about the different options that SBT provides to handle your project dependencies and configuring resolvers for the module on which your project has a dependency. Resources for Article : Further resources on this subject: So, what is Play? [Article] Play! Framework 2 – Dealing with Content [Article] Integrating Scala, Groovy, and Flex Development with Apache Maven [Article]

If you were to create an API for Python, you should write it using Cython to create a more type-safe Python API. Or, you could take the C types from Cython to implement the same algorithms in your Python code, and they will be faster because you're specifying the types and you avoid a lot of the type conversion required. Consider you are implementing a fresh project in C. There are a few issues we always come across in starting fresh; for example, choosing the logging or configuration system we will use or implement. With Cython, we can reuse the Python logging system as well as the ConfigParser standard libraries from Python in our C code to get a head start. If this doesn't prove to be the correct solution, we can chop and change easily. We can even extend and get Python to handle all usage. Since the Python API is very powerful, we might as well make Python do as much as it can to get us off the ground. Another question is do we want Python be our "driver" (main entry function) or do we want to handle this from our C code? Cython cdef In the next two examples, I will demonstrate how we can reuse the Python logging and Python ConfigParser modules directly from C code. But there are a few formalities to get over first, namely the Python initialization API and the link load model for fully embedded Python applications for using the shared library method. It's very simple to embed Python within a C/C++ application; you will require the following boilerplate: #include <Python.h>int main (int argc, char ** argv){Py_SetProgramName (argv [0]);Py_Initialize ();/* Do all your stuff in side here...*/Py_Finalize ();return 0;} Make sure you always put the Python.h header at the very beginning of each C file, because Python contains a lot of headers defined for system headers to turn things on and off to make things behave correctly on your system. Later, I will introduce some important concepts about the GIL that you should know and the relevant Python API code you will need to use from time to time. But for now, these few calls will be enough for you to get off the ground. Linking models Linking models are extremely important when considering how we can extend or embed things in native applications. There are two main linking models for Cython: fully embedded Python and code, which looks like the following figure: This demonstrates a fully embedded Python application where the Python runtime is linked into the final binary. This means we already have the Python runtime, whereas before we had to run the Python interpreter to call into our Cython module. There is also a Python shared object module as shown in the following figure: We have now fully modularized Python. This would be a more Pythonic approach to Cython, and if your code base is mostly Python, this is the approach you should take if you simply want to have a native module to call into some native code, as this lends your code to be more dynamic and reusable. The public keyword Moving on from linking models, we should next look at the public keyword, which allows Cython to generate a C/C++ header file that we can include with the prototypes to call directly into Python code from C. The main caveat if you're going to call Python public declarations directly from C is if your link model is fully embedded and linked against libpython.so; you need to use the boilerplate code as shown in the previous section. And before calling anything with the function, you need to initialize the Python module example if you have a cythonfile.pyx file and compile it with public declarations such as the following: cdef public void cythonFunction ():print "inside cython function!!!" You will not only get a cythonfile.c file but also cythonfile.h; this declares a function called extern void initcythonfile (void). So, before calling anything to do with the Cython code, use the following: /* Boiler plate init Python */Py_SetProgramName (argv [0]);Py_Initialize ();/* Init our config module into Python memory */initpublicTest ();cythonFunction ();/* cleanup python before exit ... */Py_Finalize (); Calling initcythonfile can be considered as the following in Python: import cythonfile Just like the previous examples, this only affects you if you're generating a fully embedded Python binary. Logging into Python A good example of Cython's abilities in my opinion is reusing the Python logging module directly from C. So, for example, we want a few macros we can rely on, such as info (…) that can handle VA_ARGS and feels as if we are calling a simple printf method. I think that after this example, you should start to see how things might work when mixing C and Python now that the cdef and public keywords start to bring things to life: import loggingcdef public void initLogging (char * logfile):logging.basicConfig (filename = logfile,level = logging.DEBUG,format = '%(levelname)s %(asctime)s:%(message)s',datefmt = '%m/%d/%Y %I:%M:%S')cdef public void pyinfo (char * message):logging.info (message)cdef public void pydebug (char * message):logging.debug (message)cdef public void pyerror (char * message):logging.error (message) This could serve as a simple wrapper for calling directly into the Python logger, but we can make this even more awesome in our C code with C99 __VA_ARGS__ and an attribute that is similar to GCC printf. This will make it look and work just like any function that is similar to printf. We can define some headers to wrap our calls to this in C as follows: #ifndef __MAIN_H__#define __MAIN_H__#include <Python.h>#include <stdio.h>#include <stdarg.h>#define printflike __attribute__ ((format (printf, 3, 4)))extern void printflike cinfo (const char *, unsigned, const char *,...);extern void printflike cdebug (const char *, unsigned, const char *,...);extern void printflike cerror (const char *, unsigned, const char *,...);#define info(...) cinfo (__FILE__, __LINE__, __VA_ARGS__)#define error(...) cerror (__FILE__, __LINE__, __VA_ARGS__)#define debug(...) cdebug (__FILE__, __LINE__, __VA_ARGS__)#include "logger.h" // remember to import our cython public's#endif //__MAIN_H__ Now we have these macros calling cinfo and the rest, and we can see the file and line number where we call these logging functions: void cdebug (const char * file, unsigned line,const char * fmt, ...){char buffer [256];va_list args;va_start (args, fmt);vsprintf (buffer, fmt, args);va_end (args);char buf [512];snprintf (buf, sizeof (buf), "%s-%i -> %s",file, line, buffer);pydebug (buf);} On calling debug ("debug message"), we see the following output: Philips-MacBook:cpy-logging redbrain$ ./example log Philips-MacBook:cpy-logging redbrain$ cat log INFO 05/06/2013 12:28:24: main.c-62 -> info message DEBUG 05/06/2013 12:28:24: main.c-63 -> debug messageERROR 05/06/2013 12:28:24: main.c-64 -> error message Also, you should note that we import and do everything we would do in Python as we would in here, so don't be afraid to make lists or classes and use these to help out. Remember if you had a Cython module with public declarations calling into the logging module, this integrates your applications as if it were one. More importantly, you only need all of this boilerplate when you fully embed Python, not when you compile your module to a shared library. Python ConfigParser Another useful case is to make Python's ConfigParser accessible in some way from C; ideally, all we really want is to have a function to which we pass the path to a config file to receive a STATUS OK/FAIL message and a filled buffer of the configuration that we need: from ConfigParser import SafeConfigParser, NoSectionErrorcdef extern from "main.h":struct config:char * pathint numbercdef config myconfig Here, we've Cythoned our struct and declared an instance on the stack for easier management: cdef public config * parseConfig (char * cfg):# initialize the global stack variable for our config...myconfig.path = NULLmyconfig.number = 0# buffers for assigning python types into C typescdef char * path = NULLcdef number = 0parser = SafeConfigParser ()try:parser.readfp (open (cfg))pynumber = int (parser.get ("example", "number"))pypath = parser.get ("example", "path")except NoSectionError:print "No section named example"return NULLexcept IOError:print "no such file ", cfgreturn NULLfinally:myconfig.number = pynumbermyconfig.path = pypathreturn &myconfig This is a fairly trivial piece of Cython code that will return NULL on error as well as the pointer to the struct containing the configuration: Philips-MacBook:cpy-configparser redbrain$ ./example sample.cfgcfg->path = some/path/to/somethingcfg-number = 15 As you can see, we easily parsed a config file without using any C code. I always found figuring out how I was going to parse config files in C to be a nightmare. I usually ended up writing my own mini domain-specific language using Flex and Bison as a parser as well as my own middle-end, which is just too involved.

(For more resources related to this topic, see here.) The following screenshot (Images and colors) shows the final result of this article:   Images and colors The following is the corresponding drawing.kv code: 64. # File name: drawing.kv (Images and colors) 65. <DrawingSpace>: 66. canvas: 67. Ellipse: 68. pos: 10,10 69. size: 80,80 70. source: 'kivy.png' 71. Rectangle: 72. pos: 110,10 73. size: 80,80 74. source: 'kivy.png' 75. Color: 76. rgba: 0,0,1,.75 77. Line: 78. points: 10,10,390,10 79. width: 10 80. cap: 'square' 81. Color: 82. rgba: 0,1,0,1 83. Rectangle: 84. pos: 210,10 85. size: 80,80 86. source: 'kivy.png' 87. Rectangle: 88. pos: 310,10 89. size: 80,80 This code starts with an Ellipse (line 67) and a Rectangle (line 71). We use the source property, which inserts an image to decorate the polygon. The image kivy.png is 80 x 80 pixels with a white background (without any alpha/transparency channel). The result is shown in the first two columns of the previous screenshot (Images and colors). In line 75, we use the context instruction Color to change the color (with the rgba property: red, green, blue, and alpha) of the coordinate space context. This means that the next VertexInstructions will be drawn with the color changed by rgba. A ContextInstruction changes the current coordinate space context. In the previous screenshot, the blue bar at the bottom (line 77) has a transparent blue (line 76) instead of the default white (1,1,1,1) as seen in the previous examples. We set the ends shape of the line to a square with the cap property (line 80). We change the color again in line 81. After that, we draw two more rectangles, one with the kivy.png image and other without it. In the previous screenshot (Images and color) you can see that the white part of the image has become as green as the basic Rectangle on the left. Be very careful with this. The Color instruction acts as a light that is illuminating the kivy.png image. This is why you can still see the Kivy logo on the background instead of it being all covered by the color. There is another important detail to notice in the previous screenshot. There is a blue line that crosses the first two polygons in front and then crosses behind the last two. This illustrates the fact that the instructions are executed in order and this might bring some unwanted results. In this example we have full control of the order but for more complicated scenarios Kivy provides an alternative. We can specify three Canvas instances (canvas.before, canvas, and canvas.after) for each Widget. They are useful to organize the order of execution to guarantee that the background component remains in the background, or to bring some of the elements to the foreground. The following drawing.kv file shows an example of these three sets (lines 92, 98, and 104) of instructions: 90. # File name: drawing.kv (Before and After Canvas) 91. <DrawingSpace>: 92. canvas.before: 93. Color: 94. rgba: 1,0,0,1 95. Rectangle: 96. pos: 0,0 97. size: 100,100 98. canvas: 99. Color: 100. rgba: 0,1,0,1 101. Rectangle: 102. pos: 100,0 103. size: 100,100 104. canvas.after: 105. Color: 106. rgba: 0,0,1,1 107. Rectangle: 108. pos: 200,0 109. size: 100,100 110. Button: 111. text: 'A very very very long button' 112. pos_hint: {'center_x': .5, 'center_y': .5} 113. size_hint: .9,.1 In each set, a Rectangle of different color is drawn (lines 95, 101, and 107). The following diagram illustrates the execution order of the canvas. The number on the top-left margin of each code block indicates the order of execution: Execution order of the canvas Please note that we didn't define any canvas, canvas.before, or canvas.after for the Button, but Kivy does. The Button is a Widget and it displays graphics on the screen. For example, the gray background is just a Rectangle. That means that it has instructions in its internal Canvas instances. The following screenshot shows the result (executed with python drawing.py --size=300x100): Before and after canvas The graphics of the Button (the child) are covered up by the graphics of instructions in the canvas.after. But what is executed between canvas.before and canvas? It could be code of a base class when we are working with inheritance and we want to add instructions in the subclass that should be executed before the base class Canvas instances. A practical example of this will be covered when we apply them in the last section of this article in the comic creator project. The canvas.before will also be useful when we study how to dynamically add instruction to Canvas instances For now, it is sufficient to understand that there are three sets of instructions (Canvas instances) that provide some flexibility when we are displaying graphics on the screen. We will now explore some more context instructions related to three basic transformations. Summary In this article we learned how to add images and colors to shapes and how to position graphics at a front or back level. Resources for Article: Further resources on this subject: Easily Writing SQL Queries with Spring Python [Article] Python Testing: Installing the Robot Framework [Article] Advanced Output Formats in Python 2.6 Text Processing [Article]

(For more resources related to this topic, see here.) The spatial component of a real-world feature is the geometric representation of its shape in some coordinate space (either in 2D or 3D), and in vector space, this is referred to as its geometry. Oracle Spatial is designed to make spatial data management easier and more natural to users of location-enabled business applications and geographic information system (GIS) applications. Oracle allows the storage of spatial data in a table using the SDO_GEOMETRY data type that is just like any other data type in the database. Once the spatial data is stored in the Oracle database, it can be easily manipulated, retrieved, and related to all other data stored in the database. A spatial database should be designed just like any other database with a fully specified model. A fully specified model that is application independent should control the spatial data storage. A good data model supports and enhances application access without compromising the quality. In addition to these features, database features can be used to support applications that have limited functionality when it comes to table and column design. For example, some applications mandate a single spatial column per table or only a single homogeneous geometry type per spatial column. These limitations can be accommodated quite easily using database features such as views and triggers. In addition, there are a number of issues that arise when designing a data model that directly affects the data quality, performance, and access. The goal of this article is to give readers an understanding of how to model spatial data as SDO_GEOMETRY columns within tables, how to support spatial constraints for improved data quality, how to use synchronous and asynchronous triggers for implementing topological constraint checking, and to present methods for coping with multiple representations for faster web service access. All these issues, with solutions, are covered in this article: Defining a sample schema Using spatial metadata Using Oracle metadata views Using OGC metadata views Using different types of geometric representations Implementing tables with homogeneous and heterogeneous columns Implementing multiple representations for a single object Implementing multiple instances of a single column, for example, pre-thinned data for different scales and reprojection for faster web service access Restricting data access via views Using views to expose a single geometry type when multiple geometry types are present in the table Using views to expose tables with single geometry columns when multiple geometry columns are present in the table Implementing spatial constraints at the database level Restricting geometry types Spatial topological constraints Implementation of synchronous triggers Implementation of asynchronous triggers Defining a sample schema We will first define a sample schema that will be used for all the examples in this article. The schema is intended to model typical spatial assets maintained in a city-level GIS. Oracle Spatial provides all the functionality needed to model or describe the spatial properties of an asset (in modeling, it is often called an entity). This spatial description of an asset should not be treated differently from any other descriptive attribute. In addition, a data model should describe all assets/entities within it independently of any application. This should include, to the best of the ability of SQL, all business rules that define or control these assets/entities within the database, and these rules should be implemented using standard database practices. Defining the data model We use a schema with 12 tables to represent a spatial database for a city. This schema has tables to represent administrative areas managed at the city level, such as land parcels and neighborhoods, along with tables to manage natural features such as water boundaries. The LAND_PARCELS table has information about land at the lowest administrative level of the city. Buildings have to be fully contained in these land parcels. A table called BUILDING_FOOTPRINTS has information about all the buildings in the city. This table has the footprint of each building along with other information, such as name, height, and other attributes. Sets of neighborhoods are defined as a collection of land parcels to create more granular administrative areas. These neighborhoods are stored in the PLANNING_NEIGHBORHOODS table. There is a master table, BASE_ADDRESSES, to store information about all the valid street addresses in the city. Every record in the BUILDING_FOOTPRINTS table must have one parent record in this master address table. Note that the master address table does not list all the addresses of the apartments in a building. Rather, it stores one record for each street level address. So, each record in the BUILDING_FOOTPRINTS table has only one corresponding record in the master address table. There is also a master table, ROADS, that is used to store information about all the roads in the city. ROADS stores one record for each named road in the city so that all common information for the road can be stored together in one table. This is the only table in the schema without any geometry information. Each road in turn maps to a set of road segments that are stored in the ROAD_CLINES table. This table is used to store the geometric representation of center lines of road segments. This table also stores information about address ranges on these road segments. Road segments typically have different address ranges on the left side of the road and on the right side of the road. Each road segment also has a parent ROAD_ID associated with it from the ROADS table. A city usually manages sidewalks and other assets, such as street lamps, trashcans, and benches that are placed on these sidewalks. The SIDEWALKS table stores the information for all the sidewalks managed by the city. The CITY_FURNITURE table stores all the data corresponding to the assets, such as benches, streetlights, and trashcans. The ORTHO_PHOTOS table stores the collected information using aerial photography. The raster information stored in this table can be used to look for changes over time for the built-in features of the city. The water features of the city are stored in two different tables: the WATER_LINE table is used to store the water line features, such as creeks, rivers, and canals. The WATER_AREA table is used to store area features, such as lakes, rivers, and bays. The following figure shows the entity-relationship (E-R) diagram for this data model: The following figure shows the further E-R diagram for same data model: Creating tables in the schema Create a user called BOOK and assign it a password. Load the script <schema_load.sql> and it will create the tables required for running the examples described in this article. It will also create the Oracle Spatial metadata required for these tables. The following privileges are granted to the BOOK user: grant connect,resource to Book identified by <password>; grant connect, resource to book; grant create table to book; grant create view to book; grant create sequence to book; grant create synonym to book; grant create any directory to book; grant query rewrite to book; grant unlimited tablespace to book; Understanding spatial metadata Oracle Spatial requires certain metadata before the spatial data can be meaningfully used by applications. The database views that contain this metadata also act as a catalog for all the spatial data in the database. There are two basic views defined to store this metadata information: USER_SDO_GEOM_METADATA and ALL_SDO_GEOM_METADATA. The USER_ view is used to create a metadata entry for a single SDO_GEOMETRY column within a database table or view. An entry must be created for each SDO_GEOMETRY column within a table; entries for SDO_GEOMETRY columns in views are optional. If a table has more than one column of type SDO_GEOMETRY, then there is one metadata entry for each column of spatial data in that table. The ALL_ view shows all of the spatial layers that can be accessed by the current user. If a user has the Select grant on another user’s table with SDO_GEOMETRY columns, the first user can see the metadata entries for those tables in the ALL_ view. The views are set up so that owners of the spatial tables or views can create the metadata for them. And, the owner of a layer can grant read access to a layer to other users in the system. Granting a Select privilege on the table or view to other users will let them see the metadata for these tables and views. The ALL_ view displays all the spatial tables owned by the user along with other spatial tables for which the current user has read access. Spatial Reference System Each SDO_GEOMETRY object has a Spatial Reference System (SRS) associated with it, and all the SDO_GEOMETRY objects in a column should have the same SRS. In Oracle Spatial, a Spatial Reference ID (SRID) is used to associate an SRS with SDO_GEOMETRY objects. There are cases (for example, engineering drawings) where there is no SRS associated with an SDO_GEOMETRY object. In such cases, a NULL SRID is used to denote that the spatial data has no spatial reference information. An SRS can be geographic or non-geographic. A geographic SRS is used when the spatial data is used to represent features on the surface of the Earth. These types of SRS usually have a reference system that can relate the coordinates of the spatial data to locations on Earth. A unit of measurement is also associated with an SRS so that measurements can be done using a well-defined system. A non-geographic SRS is used when the spatial data is not directly related to locations on Earth. But these systems usually have a unit of measurement associated with them. Building floor plans is a good example of spatial data that is often not directly related to locations on Earth. A geographic system can be either geodetic or projected. Coordinates in a geodetic system are often described using longitude and latitude. In Oracle Spatial, the convention is to use longitude as the first coordinate and latitude as the second coordinate. A projected system is a Cartesian system that is defined as a planar projection based on the datum and projection parameters. Before an entry is created for a layer of data, the SRID associated with the data should be identified along with the tolerance to be used for the spatial layer. All the spatial data has an inherent accuracy associated with it. Hence, the tolerance value used for a spatial layer is very important and should be determined based on the accuracy of the data. Once these two values are identified, you are ready to create the metadata for the spatial layer. More on Spatial Reference Systems Oracle Spatial supports hundreds of SRSs, and it is very important to choose the right SRS for any given data set. The definition of an SRS can be easily obtained by looking at the well-known text (WKT) for that SRS. The WKTs for all the SRSs supplied as part of Oracle Spatial are available from the MDSYS.CS_SRS view. In addition to this view, there are several other metadata tables under MDSYS that contain more details on how these SRSs are defined. Oracle Spatial also supports the EPSG standard-based SRSs. SRS in Oracle Spatial is flexible and allows users to define new reference systems if they are not present in the supplied SRSs. Creating spatial metadata The tables used in our sample schema contain data that is geographically referenced. Spatial metadata can be created using Insert statements into the USER_SDO_GEOM_METADATA view. This view is defined as a public-writable view on top of MDSYS.SDO_GEOM_METADATA_TABLE that is used to store metadata for all the spatial columns in the database. Let us look at the metadata creation process for some of the spatial tables. Most of the tables used in the current schema have spatial data in the California State Plane Zone 3 Coordinate System. In Oracle Spatial, the corresponding SRID for this SRS is 2872. This coordinate system has foot as the unit of measurement and we will use 0.05 as our tolerance (that is, five-hundredths of a foot). The metadata is created using the Insert statement as shown in the following code: Insert Into USER_SDO_GEOM_METDATA Values (‘LAND_PARCELS’, ‘GEOM’, SDO_DIM_ARRAY(SDO_DIM_ARRAY(SDO_DIM_ELEMENT(‘X’, 5900000, 6100000, 0.05), SDO_DIM_ELEMENT(‘Y’,2000000, 2200000, 0.05)), 2872); The SDO_DIM_ELEMENT object is used to specify the lower and upper bounds for each dimension of the coordinate system along with the tolerance value. The metadata allows one entry for each dimension, even though it is very common to use the same tolerance value for the X and Y dimensions. When storing 3D data, it is very common to use a different tolerance value for the Z dimension. The BASE_ADDRESSES table has geometries stored in two columns: GEOMETRY and GEOD_GEOMETRY. The GEOMETRY column has data in the 2872 SRID, while the GEOD_GEOMETRY column has data in longitude and latitude. As this is a geodetic system, the tolerance for such systems is required to be in meters. So, a tolerance of 0.05 means a tolerance of 5cm. For geodetic data, it is recommended that the tolerance should not be less than 5cm for all of the topology and distance-based operations. Insert Into USER_SDO_GEOM_METDATA Values(‘BASE_ADDRESSES’, ‘GEOD_GEOM’, SDO_DIM_ARRAY((SDO_DIM_ELEMENT(‘Longitude’, -122.51436, -122.36638, .05), SDO_DIM_ELEMENT(‘Latitude’, 37.7081463, 37.8309382, .05)), 8307); As this is a geodetic system, the longitude range goes from -180 to 180 and the latitude range goes from -90 to 90. Even though it is normal practice to use these ranges for the metadata entry, many developers use the actual ranges spanned by the SDO_GEOMETRY object. Mapping tools and applications typically use this extent from the metadata to compute the initial extent of the data in each column of the spatial data. Any application looking for all the spatial columns in the database should select the data from the ALL_SDO_GEOM_METADATA view. This will return one row for each column of spatial data in the database that is visible to the current user. OGC-defined metadata views Open Geospatial Consortium (OGC) defines a different set of standardized metadata views. OGC standard metadata can be defined using a new set of tables or views in Oracle Spatial. For a simple solution for the OGC metadata schema, we will show a view-based implementation using the Oracle Spatial metadata table. All Oracle supplied packages, functions, and types of Oracle Spatial are in the MDSYS schema. It is generally not recommended to create any user objects under this schema as it might cause problems during database upgrades. Oracle also supplies another predefined schema called MDDATA that can be used for Oracle Spatial-related user objects that are general purpose in nature. We use this MDDATA schema to create the OGC metadata views. This user comes locked and it is recommended that you do not unlock this user. But, it does require a few privileges to make the following code work, so grant those privileges as required. Connect to the database as a user with SYSDBA privileges and execute all the following steps as the MDDATA user by changing the current schema to MDDATA. We need to grant an explicit Select privilege on SDO_GEOM_METADATA_TABLE to MDDATA. Alter session set current_schema=MDDATA; GRANT Select on MDSYS.SDO_GEOM_METADATA_TABLE to MDDATA; The OGC standard requires the geometry type as part of the metadata view. But, this is not part of the MDSYS owned metadata view and has to be computed based on the geometry table information stored in the MDSYS table. So, first define a function that can compute the geometry type based on the rows in the spatial tables. Note that this function just looks at the first non-NULL geometry and returns the type of that geometry. Users can modify this to make it look at the whole table to decide on the geometry type, but it can be a very expensive operation. Create Or Replace Function MDDATA.GET_GEOMETRY_TYPE (tsname varchar2, tname varchar2, cname varchar2) Return Number IS gtype number; Begin Begin execute immediate ‘ Select a.’|| SYS.DBMS_ASSERT.ENQUOTE_NAME(cname, false)|| ‘.sdo_gtype From ‘|| SYS.DBMS_ASSERT.ENQUOTE_NAME(tsname, false)||’.’|| SYS.DBMS_ASSERT.ENQUOTE_NAME(tname, false)|| ‘ a Where a.’|| SYS.DBMS_ASSERT.ENQUOTE_NAME(cname, false)|| ‘ is not null and rownum < 2’ Into gtype; Return gtype MOD 100; EXCEPTION When OTHERS Then Return 4; End; End; Notice all the uses of the ENQUOTE_NAME function from the SYS.DBMS_ASSERT package. This is used to avoid any possible SQL injection issues typically associated with functions that create SQL statements using the user supplied SQL. As we are creating a general purpose function that can be invoked by any user directly or indirectly, it is a good idea to protect the function from any possible SQL injection. Next, we define an OGC metadata view to see all the rows from the MDSYS owned metadata table. Create Or Replace View MDDATA.OGC_GEOMETRY_COLUMNS As Select GM.SDO_OWNER As F_TABLE_SCHEMA, GM.SDO_TABLE_NAME As F_TABLE_NAME, GM.SDO_COLUMN_NAME As F_GEOMETRY_COLUMN, Get_Geometry_Type(GM.sdo_owner, GM.sdo_table_name, GM.sdo_column_name) As GEOMETRY_TYPE, (Select count(*) From Table(GM.SDO_DIMINFO) ) As COORD_DIMENSION, GM.SDO_SRID As SRID From MDSYS.SDO_GEOM_METADATA_TABLE GM; And finally, we define a user view that will show all the geometry columns that are visible to the current user. Create Or Replace View GEOMETRY_COLUMNS As Select b.F_TABLE_SCHEMA , b.F_TABLE_NAME , b.F_GEOMETRY_COLUMN, b.COORD_DIMENSION, b.SRID, b.GEOMETRY_TYPE From MDDATA.OGC_GEOMETRY_COLUMNS b, ALL_OBJECTS a Where b.F_TABLE_NAME = a.OBJECT_NAME And b.F_TABLE_SCHEMA = a.OWNER And a.OBJECT_TYPE in (‘TABLE’, ‘SYNONYM’, ‘VIEW’); Grant Select On MDDATA.GEOMETRY_COLUMNS to public; Create PUBLIC SYNONYM GEOMETRY_COLUMNS FOR MDDATA.GEOMETRY_COLUMNS; Tolerance in Oracle Spatial Tolerance is used in Oracle Spatial to associate a level of precision with the data and to check the validity of geometries among other things. Tolerance should be derived based on the resolution and accuracy of the data. If the devices or methods used to collect the spatial data are correct up to a five-meter resolution, the tolerance for that layer should be set to 5 meters. The actual tolerance value, inserted into the metadata view depends on the real-world tolerance value and the unit of measurement used in the coordinate system is used for the column of spatial data. For example, let the tolerance for the spatial data be 5 centimeters and the unit of measurement of the coordinate system used for the spatial column is feet. Then, the five-centimeter value should first be converted to feet (1 centimeter is 0.032 feet)—this comes out to be 0.164 feet. So, you use a value of 0.164 for tolerance in the metadata. In practice, Oracle Spatial uses the following rules based on tolerance to determine if the geometry is valid or not. These are in addition to other topological consistency rules (as described by the OGC Simple Feature Specification) used to check the validity of geometries: If the distance between two consecutive vertices in the geometry is less than the tolerance value, the geometry is invalid. This rule applies to line-string and polygon type geometries. If the distance between a vertex and the nearest edge to that vertex in a polygon is less than the tolerance value, the geometry is invalid. This rule only applies to the polygon type geometries. Managing homogeneous and heterogeneous data If a spatial column in a table contains data of one single geometry type (for example, polygon or line-string, but not both), we can say that spatial data in that column is homogeneous. In other situations, a column may contain data from one or more geometry types (heterogeneous representation). For example, the spatial description of a rare and endangered flora object may normally be a single plant (a plant via a point), but in other situations, it may be an area (a patch via a polygon). Consider the CITY_FURNITURE table that is used for storing city assets like benches, trashcans, and streetlights. The geometry for the benches is represented using line-strings, while streetlights and trashcans are represented with points. It is perfectly correct, semantically, to store different types of observations within a single geometry column. However, while some mapping software systems can cope with multiple geometry types per SDO_GEOMETRY column, others, such as some traditional GIS packages, require homogeneity. We will describe how to achieve this next, when a column in the table has heterogeneous data. We define three views on top of the CITY_FURNITURE table corresponding to each of the types of data stored in the table. This table has three classes of objects: benches, trashcans, and streetlights. After the views are defined, we also need to create metadata entries for these views in USER_SDO_GEOM_METADATA so that any GIS tool can discover these views as if they are tables. Create the database views corresponding to each of the three types of data stored in the CITY_FURNITURE table. -- DDL for View CITY_FURN_BENCHES Create Or Replace FORCE VIEW CITY_FURN_BENCHES (FID, FEATTYPE, GEOM) As Select FID, FEATTYPE, GEOM From CITY_FURNITURE Where FEATTYPE=’BENCH’; -- DDL for View CITY_FURN_LIGHTS Create Or Replace FORCE VIEW CITY_FURN_LIGHTS (FID, FEATTYPE, GEOM) As Select FID, FEATTYPE, GEOM From CITY_FURNITURE Where FEATTYPE=’LIGHT’; -- DDL for View CITY_FURN_TRASHCANS Create Or Replace FORCE VIEW CITY_FURN_TRASHCANS (FID, FEATTYPE, GEOM) As Select FID, FEATTYPE, GEOM From CITY_FURNITURE Where FEATTYPE=’TRASHCAN’; The preceding examples show how to use other relational attributes to create the required views. Another way to do this is to constrain based on the SDO_GTYPE attribute of the SDO_GEOMETRY column. The following example shows how to do this for one of the preceding views, as the rest can be done with similar SQL: -- DDL for View CITY__FURN_BENCHES Create Or Replace FORCE VIEW CITY_FURN_BENCHES (FID, FEATTYPE, GEOM) AS Select FID, FEATTYPE, GEOM From CITY_FURNITURE A Where A.GEOM.SDO_GTYPE = 2002; Now create the metadata for each of these views so that any GIS can access this as if it is stored in a separate table. Note that these additional metadata entries are not required for the correct usage of Oracle Spatial. They are created only to facilitate the GIS tools that don’t support heterogeneous data in spatial columns. Insert Into USER_SDO_GEOM_METADATA Values ( ‘CITY_FURN_BENCHES’, ‘GEOM’, SDO_DIM_ARRAY(SDO_DIM_ELEMENT(‘X’, 5900000, 6100000, .05), SDO_DIM_ELEMENT(‘Y’, 2000000, 2200000, .05)), 2872); Insert Into USER_SDO_GEOM_METADATA Values( ‘CITY_FURN_LIGHTS’, ‘GEOM’, SDO_DIM_ARRAY(SDO_DIM_ELEMENT(‘X’, 5900000, 6100000, .05), SDO_DIM_ELEMENT(‘Y’, 2000000, 2200000, .05)), 2872); Insert Into USER_SDO_GEOM_METADATA Values( ‘CITY_FURN_TRASHCANS’, ‘GEOM’, SDO_DIM_ARRAY(SDO_DIM_ELEMENT(‘X’, 5900000, 6100000, .05), SDO_DIM_ELEMENT(‘Y’, 2000000, 2200000, .05)), 2872); How metadata is used Applications typically look at the ALL_SDO_GEOM_METADATA view to see the spatial tables available in the database for a given user. If you select the data from this view, now you will see 11 rows returned: 8 rows corresponding to tables and 3 rows corresponding to the views defined in the CITY_FURNITURE table. From an application point of view, it does not make any difference whether this data is stored in a view or a table. It will all look the same to the application. Sometimes it is useful to constrain the type of spatial data stored in the table to be homogeneous. For example, the ROAD_CLINES table should contain only linear geometries, as the roads are usually geometries of line type. This can be done by constraints that can be imposed by the spatial index defined in the ROAD_CLINES table. While creating the spatial index, provide the LAYER_GTYPE keyword and specify the type of data that will be stored in this table. -- DDL for Index ROAD_CLINES_SIDX Create Index ROAD_CLINES_SIDX ON ROAD_CLINES (GEOM) INDEXTYPE IS MDSYS.SPATIAL_INDEX PARAMETERS (‘LAYER_GTYPE=LINE’); Now, if you try to insert a row with a geometry that has a different SDO_GTYPE attribute than 2002, it will raise an error. Insert Into ROAD_CLINES Values ( 198999, 402, 0, 0, 2300, 2498, 190201, 23564000, 23555000, 94107, 10, ‘Y’, ‘DPW’, ‘Potrero Hill’, ‘190201’, ‘03RD ST’, ‘3’, ‘3RD’, SDO_GEOMETRY(2001, 2872, NULL, SDO_ELEM_INFO_ARRAY(1, 1, 1), SDO_ORDINATE_ARRAY( 6015763.86, 2104882.29))); * ERROR at line 1: ORA-29875: failed in the execution of the ODCIINDEXInsert routine ORA-13375: the layer is of type [2002] while geometry inserted has type [2001] ORA-06512: at “MDSYS.SDO_INDEX_METHOD_10I”, line 720 ORA-06512: at “MDSYS.SDO_INDEX_METHOD_10I”, line 225 The error message clearly indicates that the row that is currently being inserted has geometry with the wrong SDO_GTYPE attribute. This is the easiest way to strictly enforce the GTYPE constraints on the spatial data. However, this has the problem of rejecting the whole row when the geometry type does not match the LAYER_GTYPE keyword. And it is also not easy to log these cases as the error is thrown and the database moves on to process the next Insert statement. In some cases, the user might still want to insert the row into the table, but record cthe fact that there is invalid data in the row. Users can then come back and look at all the invalid entries and fix the issues. We will describe a few methods to do this logging and error processing later in this article. Using database check constraints Specifying the layer_gtype keyword is not the only way to constrain the type in a spatial layer. One can also use table level constraints to achieve the same result. With constraints, users get the additional benefit of specifying more complex constraints, such as allowing only points and lines in a layer. However, if only a single geometry type constraint is required, it is better to implement that constraint using the LAYER_GTYPE method as this is more efficient than the check constraint. These constraints can also be enforced with database triggers, and these trigger-based constraints are discussed in a later section. Alter Table CITY_FURNITURE ADD Constraint city_furniture_gtype_ck CHECK ( geom.sdo_gtype in (2002, 2001) ); Insert Into CITY_FURNITURE Values (432432, 'BENCH', SDO_GEOMETRY(2003,2872,NULL, SDO_ELEM_INFO_ARRAY(1, 1003, 3), SDO_ORDINATE_ARRAY(6010548.53, 2091896.34, 6010550.45,2091890.11))); This will fail with the following error: ERROR at line 1: ORA-02290: check Constraint (BOOK.CITY_FURNITURE_GTYPE_CK) violated Similarly, this approach can be useful for an OBJTYPE column check in the CITY_FURNITURE table. Alter Table CITY_FURNITURE ADD Constraint city_furniture_type_ck CHECK ( feattype in (‘BENCH’,’LIGHT’,’TRASHCAN’) ); Insert Into CITY_FURNITURE Values (432432, ‘LIGHTS’, SDO_GEOMETRY(2001,2872, SDO_POINT_TYPE(6010548.53, 2091896.34, NULL), NULL, NULL)); ERROR at line 1: ORA-02290: check Constraint (BOOK.CITY_FURNITURE_TYPE_CK) violated Now these two constraints are checking two independent columns, but what we really need is a more complex check to ensure each value of OBJTYPE has the corresponding SDO_GEOMETRY with the right type. That is, we want to make sure that TRASHCAN and LIGHT types have a point geometry and BENCH has a line geometry. Alter Table CITY_FURNITURE Drop Constraint city_furniture_gtype_ck; Alter Table CITY_FURNITURE Drop Constraint city_furniture_type_ck; Alter Table CITY_FURNITURE ADD Constraint city_furniture_objtype_geom_ck CHECK ( ( (“FEATTYPE”=’TRASHCAN’ Or “FEATTYPE”=’LIGHT’) AND “GEOM”.”SDO_GTYPE”=2001 ) Or (“FEATTYPE”=’BENCH’ AND “GEOM”.”SDO_GTYPE”=2002) /* Else Invalid combination */ ) ; Insert Into CITY_FURNITURE Values (432432, ‘BENCH’, SDO_GEOMETRY(2001,2872, SDO_POINT_TYPE(6010548.53, 2091896.34, NULL), NULL, NULL)); ERROR at line 1: ORA-02290: check Constraint (BOOK.CITY_FURNITURE_TYPE_CK) violated Multiple representations for the same objects In some situations, it is beneficial to have multiple representations for the same geometric feature. For example, an address usually has a point representation for its location. If a footprint of a building is associated with the address, then that footprint will be represented as a polygon. In some cases, a building might have many different point locations associated with it. One point may allow a GIS application to draw an icon for the building depending on its function (for example, a fire station). Another point may allow the building to be labeled with its street address, and finally another one may show an alternate location that is used for main delivery or emergency services entry at the back of the building. Similarly, a land parcel table can have an interior point of the parcel represented as point geometry in addition to the polygon representation. For a visualization application, it is sometimes useful to represent the land parcel as a point. When a map is displayed at a smaller scale (city level), the map will be cluttered if each land parcel is displayed as a polygon. In such cases, if land parcels are displayed as points with a suitable icon, the map will be less cluttered. When the map is displayed at a larger scale (street level), the same land parcel can be displayed as a polygon. Oracle Spatial allows such multiple representations by allowing multiple SDO_GEOMETRY columns in the same table. We first start with the BUILDING_FOOTPRINTS table and alter it to add an additional SDO_GEOMETRY column to allow the street address to be represented at the center of the building via a point feature. We can use a spatial function that can compute a point inside a polygon automatically to populate this column. Summary A spatial database application should be designed just like any other database application. It should follow the same standard data model practices like any other database application. In this article, we introduced a data model that can be used in a city-wide spatial data management system. This data model is used throughout the article to illustrate different aspects of spatial database application development. We discussed the use of triggers and queues to manage spatial data in the database. We also showed how to design database level constraints for spatial data management. We introduced the concepts of database triggers and queues and showed how they can be used for spatial data management in the database. Resources for Article: Further resources on this subject: Remote Job Agent in Oracle 11g Database with Oracle Scheduler [Article] Introduction to Oracle Service Bus & Oracle Service Registry [Article] Configuration, Release and Change Management with Oracle [Article]

(For more resources related to this topic, see here.) Using Capybara outside of Cucumber Capybara is by no means coupled to Cucumber, and can be used in any setting you wish, within Test::Unit, RSpec, or just from vanilla Ruby code. In fact, if you are using Cucumber but want to abstract some logic out of your step definitions and into Page Objects, then you will still need to consider how to use Capybara outside of Cucumber's world (https://github.com/cucumber/cucumber/wiki/A-Whole-New-World). Including the modules The first option you have for using Capybara outside of Cucumber is to include the DSL modules into your own modules or classes: require 'capybara/dsl' require 'rspec/expectations' Capybara.default_driver = :selenium module MyModule include Capybara::DSL include RSpec::Matchers def play_song visit 'http://localhost/html/html5.html' click_on 'Play' find('#log').should have_text 'playing' end end class Runner include MyModule def run play_song end end Runner.new.run It is important to require the Capybara DSL file, as this contains all the Capybara methods that need to be "mixed in". In this example, we have our own Ruby module and crucially within this, the relevant Capybara module Capybara::DSL is included. In addition, the RSpec::Matchers module has also been included, which allows us to utilize standard RSpec Matchers (obviously you do not have to use RSpec; you could choose a different way to assert behavior). If you follow this pattern in your own code, you can now mix in any of the standard Capybara methods into your own module or class methods. It is worth remembering that if you include modules in a base class, then all subclasses will inherit the ability to use those module methods. This would come in handy, for example, if you were using a Page Object pattern, where the only class that would have to include the DSL module would be the base page. Using the session directly The other option for mixing Capybara into your code is to use the session directly, that is to say, you instantiate a new instance of the session object and then call the DSL methods on it. The following example implements the same test as before, but this time by using a session instance and raising a simple exception if the expected content is not found: require 'capybara' session = Capybara::Session.new :selenium session.visit('http://localhost/html/html5.html') session.click_on 'Play' raise 'song not playing' unless session.find('#log') == 'playing' If you are using an object-oriented model for building your tests, you will need to pass the session instance around or find an appropriate strategy to deal with this, as you do not have the benefit of the modules mixing in the DSL methods globally. Capybara and popular test frameworks Capybara provides out of the box integration with a number of popular test frameworks; this is not a subject we will cover in depth, simply because they are covered very well in the Capybara README, which can be found at https://github.com/jniklas/capybara. Cucumber Throughout this article, examples have been set in the context of Cucumber, so you should be happy with how to implement Capybara's API in step definitions and do some simple setup in the env.rb file, such as setting the default driver. There are a couple of additional pieces of functionality that Capybara adds when using in conjunction with Cucumber, which are worth examining. The first is that Capybara hooks into Cucumber's Before do block as follows: Before do Capybara.reset_sessions! Capybara.use_default_driver end Apart from setting the default driver, this crucially makes a call to reset_sessions!, and this in turn will invoke some code in the underlying driver. In the case of Selenium, this deletes all cookies to ensure that you start each scenario with no pollution from the previous one. For Rack::Test, this will destroy the browser instance, so a new one gets created each time; for any other drivers, you will need to check the implementation of the reset! method to see what they do. Finally, Capybara also hooks into any Cucumber scenarios you have tagged with @javascript and automatically switches you to the driver you have set up to handle JavaScript. For example, you may use Rack::Test as your default driver, but have set the following in your env.rb file: Capybara.javascript_dirver = :selenium In this case, any scenarios tagged with @javascript will result in Capybara starting a browser using Selenium as the driver. RSpec Outside of Cucumber, RSpec is one of the most popular Ruby test frameworks, lending itself well to unit, integration, and acceptance tests, and used inside and outside of Rails. Capybara adds the following features to RSpec: Lets you mix in the DSL to your specs by adding require 'capybara/rspec' into your spec_helper.rb file Lets you use :js => true to invoke the JavaScript driver Adds a DSL for writing descriptive "feature style" acceptance tests using RSpec For more details on these features, check out the README, which is available at https://github.com/jniklas/capybara. Test::Unit If you are using Ruby's basic unit test library outside of Rails, then using Capybara simply means mixing in the DSL module via include Capybara::DSL, as you saw earlier. As noted in the README, it makes a lot of sense to reset the browser session in your teardown method, for example: def teardown Capybara.reset_sessions! Capybara.use_default_driver end If you are using Rails, then there will be other considerations, such as turning off transactional fixtures, as these will not work with Selenium; again, the Capybara README details this behavior fully. MiniTest::Spec MiniTest is a new unit test framework introduced in Ruby 1.9, which also has the ability to support BDD style tests. Capybara does not have built-in support for MiniTest, because MiniTest does not use RSpec but rather uses its own matchers. There is another gem named capybara_minitest_spec (https://github.com/ordinaryzelig/capybara_minitest_spec), which adds support for these matchers to Capybara. Advanced interactions and accessing the driver directly Although we have covered a great deal of Capybara's API, there are still a few interactions that we have not addressed, for example, hovering over an element or dragging elements around. Capybara does provide support for a lot of these more advanced interactions; for example, a recent addition (Capybara 2.1) to methods you can call on an element is hover: find('#box1').hover In Selenium, this results in a call to the mouse.move_to method and works for both elements using CSS's hover property or JavaScript's mouseenter / mouseleave methods. Other drivers may implement this differently, and obviously in some it may not be supported at all, either where JavaScript support is non-existent (Rack::Test) or rudimentary (Celerity). You can also emulate drag-and-drop using the following line of code: find('#mydiv').drag_to '#droplocation' Again, driver support is likely to be patchy, but of course this will work fine in Selenium WebDriver. Despite all the bells and whistles offered by Capybara, there may still be occasions where you need to access the API that exists in the underlying driver, but that has not been mapped in Capybara. In this instance, you have two options: Call Capybara's native method on any Caybara::Element, and then call the driver method Use page.driver.browser.manage to call the driver methods that are not called on elements Using the native method If you wish to call a method in the underlying driver, and that method is one that is called on a retrieved DOM element, then you can use the native method. A good example is retrieving a computed CSS style value. Elements in the DOM can obtain CSS properties in a couple of ways; firstly, there is the inline style: <p style="font-weight:bold;">Bold Paragraph Text</p> This information could be easily retrieved by accessing the style attribute via the methods. The other way in which an element obtains CSS properties is via style elements or stylesheets that are referenced via link tags in the page. When the browser loads the stylesheets and applies styles to the specified DOM elements, these are known as computed styles. Capybara has no direct API for retrieving the computed style of an element, which was most likely a deliberate design decision as only a few drivers would ever support this. However, Selenium WebDriver does have this capability, and it is possible that you would want to access this information. Consider the following code, where we apply a CSS hover property to a div element, so that when the user hovers over the element, it changes color. <html> <head> <title>Hover Examples</title> <style> .box { height: 200px; width: 200px; margin: 10px; background-color: blue; } .box:hover { background-color: green; } </style> </head> <body> <div id="main"> <div id="box1" class="box"> </div> </div> </body> </html> The Cucumber step definitions that follow use Capybara and Selenium WebDriver to assert that the color has changed: When(/^I hover over an element whose color changes on hover using CSS$/) do visit 'http://capybara.local/html/chapter5/hover.html' find('#box1').hover end Then(/^I see the color change$/) do find('#box1').native.style('background-color').should == 'rgba(0, 128, 0, 1)' end Here, the style method within Selenium WebDriver is accessed via Capybara's native method, and the value can then be validated. Accessing driver methods using browser.manage The other use case is when we wish to access functionality provided in the underlying driver that is neither mapped by Capybara nor related to a specific element on the page. A good example of this is accessing cookie information. There has been a long running debate on the Capybara forums and GitHub issue tracker, about whether Capybara should expose an API for cookie getters and setters, but the overriding feeling has always been that this should not be exposed (arguably, you should not set cookies in your tests, as this is changing the state of the application as a user never would). Nevertheless, it is something you may well need to do, and given that Selenium WebDriver and a number of other drivers support this functionality, you will need to access the driver methods directly. Consider this page that sets a cookie using JavaScript: <html> <head> <title>Cookie Examples</title> <script> document.cookie = 'mycookie=foobar'; </script> </head> <body></body> </html> The following steps simply visits the page and then outputs to the console all the cookies that are available on the current page: When(/^I visit a page that sets a Cookie$/) do visit 'http://localhost/html/cookie.html' end Then(/^I can access the cookie using Selenium$/) do puts page.driver.browser.manage.all_cookies end We can access any method in the underlying driver by using page.driver.browser.manage, and then the method we wish to call; in this instance, we call Selenium WebDriver's all_cookies method, and the output will be as follows: [{:name=>"mycookie", :value=>"foobar", :path=>"/html/chapter5", :domain=>"localhost", :expires=>nil, :secure=>false}] Selenium WebDriver exposes a full API for accessing and setting cookies. The documentation can be found at http://rubydoc.info/gems/selenium-webdriver/0.0.28/Selenium/WebDriver/Driver. Advanced driver configuration So far, we have only set the default driver or the JavaScript driver using a symbol: Capybara.default_driver = :selenium It is quite likely that you will need to fine-tune the configuration of your driver or register multiple configurations, which you can select from at run time. An example of this might be that you are running tests from your office, and the corporate network sits behind an HTTP Proxy (the bane of a tester's life). If you are using Selenium WebDriver with Firefox, you could register a custom driver configuration in Capybara as follows: Capybara.register_driver :selenium_proxy do |app| profile = Selenium::WebDriver::Firefox::Profile.new profile["network.proxy.type"] = 1 profile["network.proxy.no_proxies_on"] = "capybara.local" profile["network.proxy.http"] = "cache-mycompany.com" profile["network.proxy.ssl"] = 'securecache-mycompany.com' profile["network.proxy.http_port"] = 9999 profile["network.proxy.ssl_port"] = 9999 profile.native_events = true Capybara::Selenium::Driver.new(app, :browser => :firefox, :profile => profile) end Capybara.default_driver = :selenium_proxy This configuration uses the Selenium WebDriver API to construct a custom Firefox profile, set the proxy details programmatically, register the driver with the name :selenium-proxy, and then make it the default driver. Obviously the possibilities here are endless, and in browsers such as Firefox and Chrome, the amount of options you can customize runs into hundreds, so knowing how to set these is important. For example, you could use this technique to create profiles with JavaScript disabled or Cookies disabled to ensure your application behaves correctly in these cases. The driver ecosystem Capybara bundles two drivers, which as you know are Rack::Test and Selenium WebDriver. However, Capybara is architected in such a way to make it easy for developers to implement other drivers, and indeed, there is a healthy ecosystem of pluggable drivers, which offer interesting alternatives to the two built-in options. Capybara-WebKit Pretty much every developer I know who is passionate about test automation is desperate for one thing—a headless browser with great JavaScript support. A headless browser is one that runs without a UI. Headless browsers typically run tests faster than opening a real browser, and make it easy to run in Continuous Integration (CI) environments, where often, a windowing environment (either MS Windows or X11 on Linux) may not be available. You can refer to the following links to find out more about Capybara-WebKit: https://github.com/thoughtbot/capybara-webkit http://qt.digia.com/ Capybara-WebKit is a driver that wraps QtWebKit and is maintained by the guys at Thoughtbot. The Qt project provides a cross-platform framework for building native GUI applications, and as part of this, it provides a WebKit browser implementation that lends itself to headless implementations. There is a slight catch with this particular implantation of QtWebKit; you will need to install the Qt system libraries separately, and there is still a reliance on X11 being present on Linux distributions despite the browser being headless. Poltergeist Poltergeist is another driver, which, in the background, will use QtWebKit. The difference here is that it wraps PhantomJS, which brings a couple of potential advantages over Capybara-WebKit. You can refer to the following links to find out more about Poltergeist: https://github.com/jonleighton/poltergeist http://phantomjs.org/ You will still need to install PhantomJS as a system dependency, but the PhantomJS project has tackled some of the issues around making QtWebKit purely headless, so you will not need X11, and you will not need to install the entire Qt framework, because PhantomJS bundles this for you. Capybara-Mechanize Mechanize is a headless browser implemented purely in Ruby, and has long been a stalwart for the Ruby community; it uses Nokogiri at its core to provide DOM access, and then builds browser capabilities around this. You can refer to the following links to find out more about Capybara-Mechanize: https://github.com/jeroenvandijk/capybara-mechanize https://github.com/sparklemotion/mechanize Capybara-Mechanize is a driver that allows you to use Mechanize to run your tests. It is a very powerful option, but there are some important aspects to consider, such as: No JavaScript support: Mechanize does not contain a JavaScript engine; therefore, none of the JavaScript on any of your pages will be run No rendering engine: Mechanize does not contain a rendering engine; therefore, no computed styles will be evaluated Fast: Because it's not attempting to evaluate JavaScript and do graphical rendering, it will be super fast The benefits of using Mechanize may not be obvious at first sight, and a lot will depend on how your site is implemented. For sites whose functionality is wholly dependent on JavaScript, this option is clearly not feasible; however, if your site follows the principles of "progressive enhancement", where JavaScript is simply used to enrich the user's experience, Mechanize is a great option. You can implement all the core functional tests using Mechanize, which will be ensure they run with minimal latency and will be far less fragile than using Selenium, and then just implement a sprinkling of Selenium or WebKit tests to test the JavaScript dependent features. Capybara-Celerity The final driver that is worth considering is Capybara-Celerity, which wraps the Ruby library Celerity. This is especially worth a look if you are running your test code on JRuby (Ruby running on the JVM) as Celerity itself wraps the Java library HtmlUnit. You can refer to the following links to find out more about Capybara-Celerity: https://github.com/sobrinho/capybara-celerity https://github.com/jarib/celerity http://htmlunit.sourceforge.net/ Going back a few years, HtmlUnit would have been a serious consideration for anybody wanting a headless browser, indeed it was the default headless browser for the Selenium WebDriver project. HtmlUnit is a pure Java implementation of a browser and uses Rhino (https://developer.mozilla.org/en/docs/Rhino) to run JavaScript. Unfortunately, you may find that the JavaScript support still fails when attempting to evaluate heavy-weight JS libraries, as the project is not actively maintained, and keeping pace with the demands of modern browsers is unrealistic for a small project. If you are looking for a headless alternative to Mechanize, Celerity is worth a look, but don't rely on it for JavaScript support. Summary This article has taken you from confidently implementing Capybara tests against your application to being a Capybara ninja. By understanding how to use Capybara outside the comfort zone of Cucumber, you can now use it in almost any setting and even write your own custom framework. This is important even if you use Cucumber, because as your Cucumber tests grow, you will probably want to implement some Page Objects and mix Capybara::DSL into these. Another important aspect of writing effective tests is being able to configure the underlying driver and access it directly when required; there is nothing magical about this, and it means you can harness the full power of your chosen driver. Finally, we introduced some alternative drivers that should whet your appetite and prove why Capybara is such a powerful framework. You write your tests once and then run them in multitude of compatible drivers. Win! Resources for Article: Further resources on this subject: First Steps with Selenium RC [Article] Behavior-driven Development with Selenium WebDriver [Article] Setting up environment for Cucumber BDD Rails [Article]

(For more resources related to this topic, see here.) Sprites Let's briefly discuss sprites. In gaming, sprites are usually used for animation sequences; a sprite is a single image in which individual frames of a character animation are stored. We are going use sprites in our animations. If you already have knowledge of graphics design, it's good for you because it is an edge for you to define how you want your game to look like and how you want to define animation sequences in sprites. You can try out tools such as Sprite Maker for making your own sprites with ease; you can get a copy of Sprite Maker at http://www.spriteland.com/sprites/sprite-maker.zip. The following is an sample animation sprite by Marc Russell, which is available for free at http://opengameart.org/content/gfxlib-fuzed you can find other open source sprites at http://opengameart.org/content/platformersidescroller-tiles: The preceding sprite will play the animation of the character moving to the right. The character sequence is well organized using an invisible grid, as shown in the following screenshot: The grid is 32 x 32; the size of our grid is very important in setting up the quads for our game. A quad in LÖVE is a specific part of an image. Because our sprite is a single image file, quads will be used to specify each of the sequences we want to draw per unit time and will be the largest part part of our animation algorithm. Animation The animation algorithm will simply play the sprite like a tape of film; we'll be using a basic technique here as LÖVE doesn't have an official module for that. Some members of the LÖVE forum have come up with different libraries to ease the way we play animations. First of all let us load our file: function love.load() sprite = love.graphics.newImage "sprite.png" end Then we create quads for each part of the sprite by using love.graphics. newQuad(x, y, width, height, sw, sh), where x is the top-left position of the quad along the x axis, y is the top-left position of the quad along the y axis, width is the width of the quad, height is the height of the quad, sw is the sprite's width, and sh is the sprite's height: love.graphics.newQuad(0, 0, 32, 32, 256, 32) --- first quad love.graphics.newQuad(32, 0, 32, 32, 256, 32) --- second quad love.graphics.newQuad(64, 0, 32, 32, 256, 32) --- Third quad love.graphics.newQuad(96, 0, 32, 32, 256, 32) --- Fourth quad love.graphics.newQuad(128, 0, 32, 32, 256, 32) --- Fifth quad love.graphics.newQuad(160, 0, 32, 32, 256, 32) --- Sixth quad love.graphics.newQuad(192, 0, 32, 32, 256, 32) --- Seventh quad love.graphics.newQuad(224, 0, 32, 32, 256, 32) --- Eighth quad The preceding code can be rewritten in a more concise loop as shown in the following code snippet: for i=1,8 do love.graphics.newQuad((i-1)*32, 0, 32, 32, 256, 32) end As advised by LÖVE, we shouldn't state our quads in the draw() or update() functions, because it will cause the quad data to be repeatedly loaded into memory with every frame, which is a bad practice. So what we'll do is pretty simple; we'll load our quad parameters in a table, while love.graphics.newQuad will be referenced locally outside the functions. So the new code will look like the following for the animation in the right direction: local Quad = love.graphics.newQuad function love.load() sprite = love.graphics.newImage "sprite.png" quads = {} quads['right'] ={} quads['left'] = {} for j=1,8 do quads['right'][j] = Quad((j-1)*32, 0, 32, 32, 256, 32); quads['left'][j] = Quad((j-1)*32, 0, 32, 32, 256, 32); -- for the character to face the opposite direction, the quad need to be flipped by using the Quad:flip(x, y) method, where x and why are Boolean. quads.left[j]:flip(true, false) --flip horizontally x = true, y = false end end Now that our animation table is set, it is important that we set a Boolean value for the state of our character. At the start of the game our character is idle, so we set idle to true. Also, there are a number of quads the algorithm should read in order to play our animation. In our case, we have eight quads, so we need a maximum of eight iterations, as shown in the following code snippet: local Quad = love.graphics.newQuad function love.load() character= {} character.player = love.graphics.newImage("sprite.png") character.x = 50 character.y = 50 direction = "right" iteration = 1 max = 8 idle = true timer = 0.1 quads = {} quads['right'] ={} quads['left'] = {} for j=1,8 do quads['right'][j] = Quad((j-1)*32, 0, 32, 32, 256, 32); quads['left'][j] = Quad((j-1)*32, 0, 32, 32, 256, 32); -- for the character to face the opposite direction, the quad need to be flipped by using the Quad:flip(x, y) method, where x and why are Boolean. quads.left[j]:flip(true, false) --flip horizontally x = true, y = false end end Now let us update our motion; if a certain key is pressed, the animation should play; if the key is released, the animation should stop. Also, if the key is pressed, the character should change position. We'll be using the love.keypressed callback function here, as shown in the following code snippet: function love.update(dt) if idle == false then timer = timer + dt if timer > 0.2 then timer = 0.1 -- The animation will play as the iteration increases, so we just write iteration = iteration + 1, also we'll stop reset our iteration at the maximum of 8 with a timer update to keep the animation smooth. iteration = iteration + 1 if love.keyboard.isDown('right') then sprite.x = sprite.x + 5 end if love.keyboard.isDown('left') then sprite.x = sprite.x - 5 end if iteration > max then iteration = 1 end end end end function love.keypressed(key) if quads[key] then direction = key idle = false end end function love.keyreleased(key) if quads[key] and direction == key then idle = true iteration = 1 direction = "right" end end Finally, we can draw our character on the screen. Here we'll be using love.graphics.drawq(image, quad, x, y), where image is the image data, quad will load our quads table, x is the position in x axis and y is the position in the y axis: function love.draw() love.graphics.drawq(sprite.player, quads[direction][iteration], sprite.x, sprite.y) end So let's package our game and run it to see the magic in action by pressing the left or right navigation key: Summary That is all for this article. We have learned how to draw 2D objects on the screen and move the objects in four directions. We have delved into the usage of sprites for animations and how to play these animations with code. Resources for Article: Further resources on this subject: Panda3D Game Development: Scene Effects and Shaders [Article] Microsoft XNA 4.0 Game Development: Receiving Player Input [Article] Introduction to Game Development Using Unity 3D [Article]

(For more resources related to this topic, see here.) We have learned how to create snippets, how to work with forms, and Ajax, how to test your code, and how to create a REST API, and all this is awesome! However, we did not learn how to persist data to make it durable. In this article, we will see how to use Mapper, an object-relational mapping ( ORM ) system for relational databases, included with Lift. The idea is to create a map of database tables into a well-organized structure of objects; for example, if you have a table that holds users, you will have a class that represents the user table. Let's say that a user can have several phone numbers. Probably, you will have a table containing a column for the phone numbers and a column containing the ID of the user owning that particular phone number. This means that you will have a class to represent the table that holds phone numbers, and the user class will have an attribute to hold the list of its phone numbers; this is known as a one-to-many relationship . Mapper is a system that helps us build such mappings by providing useful features and abstractions that make working with databases a simple task. For example, Mapper provides several types of fields such as MappedString, MappedInt, and MappedDate which we can use to map the attributes of the class versus the columns in the table being mapped. It also provides useful methods such as findAll that is used to get a list of records or save, to persist the data. There is a lot more that Mapper can do, and we'll see what this is through the course of this article. Configuring a connection to database The first thing we need to learn while working with databases is how to connect the application that we will build with the database itself. In this recipe, we will show you how to configure Lift to connect with the database of our choice. For this recipe, we will use PostgreSQL; however, other databases can also be used: Getting ready Start a new blank project. Edit the build.sbt file to add the lift-mapper and PostgreSQL driver dependencies: "net.liftweb" %% "lift-mapper" % liftVersion % "compile", "org.postgresql" % "postgresql" % "9.2-1003-jdbc4" % "compile" Create a new database. Create a new user. How to do it... Now carry out the following steps to configure a connection with the database: Add the following lines into the default.props file: db.driver=org.postgresql.Driver db.url=jdbc:postgresql:liftbook db.user=<place here the user you've created> db.password=<place here the user password> Add the following import statement in the Boot.scala file: import net.liftweb.mapper._ Create a new method named configureDB() in the Boot.scala file with the following code: def configureDB() { for { driver <- Props.get("db.driver") url <- Props.get("db.url") } yield { val standardVendor = new StandardDBVendor(driver, url, Props.get("db.user"), Props.get("db.password")) LiftRules.unloadHooks.append(standardVendor.closeAllConnections_! _) DB.defineConnectionManager(DefaultConnectionIdentifier, standardVendor) } } Then, invoke configureDB() from inside the boot method. How it works... Lift offers the net.liftweb.mapper.StandardDBVendor class, which we can use to create connections to the database easily. This class takes four arguments: driver, URL, user, and password. These are described as follows: driver : The driver argument is the JDBC driver that we will use, that is, org.postgresql.Driver URL : The URL argument is the JDBC URL that the driver will use, that is, jdbc:postgresql:liftbook user and password : The user and password arguments are the values you set when you created the user in the database. After creating the default vendor, we need to bind it to a Java Naming and Directory Interface (JNDI) name that will be used by Lift to manage the connection with the database. To create this bind, we invoked the defineConnectionManager method from the DB object. This method adds the connection identifier and the database vendor into a HashMap, using the connection identifier as the key and the database vendor as the value. The DefaultConnectionIdentifier object provides a default JNDI name that we can use without having to worry about creating our own connection identifier. You can create your own connection identifier if you want. You just need to create an object that extends ConnectionIdentifier with a method called jndiName that should return a string. Finally, we told Lift to close all the connections while shutting down the application by appending a function to the unloadHooks variable. We did this to avoid locking connections while shutting the application down. There's more... It is possible to configure Lift to use a JNDI datasource instead of using the JDBC driver directly. In this way, we can allow the container to create a pool of connections and then tell Lift to use this pool. To use a JNDI datasource, we will need to perform the following steps: Create a file called jetty-env.xml in the WEB-INF folder under src/main/webapp/ with the following content: <!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure.dtd"> <Configure class="org.eclipse.jetty.webapp.WebAppContext"> <New id="dsliftbook" class="org.eclipse.jetty.plus.jndi.Resource"> <Arg>jdbc/dsliftbook</Arg> <Arg> <New class="org.postgresql.ds.PGSimpleDataSource"> <Set name="User">place here the user you've created</Set> <Set name="Password">place here the user password</Set> <Set name="DatabaseName">liftbook</Set> <Set name="ServerName">localhost</Set> <Set name="PortNumber">5432</Set> </New> </Arg> </New> </Configure> Add the following line into the build .sbt file: env in Compile := Some(file("./src/main/webapp/WEB-INF/jetty-env.xml") asFile) Remove all the jetty dependencies and add the following: "org.eclipse.jetty" % "jetty-webapp" % "8.0.4.v20111024" % "container", "org.eclipse.jetty" % "jetty-plus" % "8.0.4.v20111024" % "container", Add the following code into the web.xml file. <resource-ref> <res-ref-name>jdbc/dsliftbook</res-ref-name> <res-type>javax.sql.DataSource</res-type> <res-auth>Container</res-auth> </resource-ref> Remove the configureDB method. Replace the invocation of configureDB method with the following line of code: DefaultConnectionIdentifier.jndiName = "jdbc/dsliftbook" The creation of the file jetty-envy.xml and the change we made in the web.xml file were to create the datasource in jetty and make it available to Lift. Since the connections will be managed by jetty now, we don't need to append hooks so that Lift can close the connections or any other configuration when shutting down the application. All we need to do is tell Lift to get the connections from the JNDI datasource that we have configured into the jetty. We do this by setting the jndiName variable of the default connection identifier, DefaultConnectionIdentifier as follows: DefaultConnectionIdentifier.jndiName = "jdbc/dsliftbook" The change we've made to the build.sbt file was to make the jetty-env.xml file available to the embedded jetty. So, we can use it when we get the application started by using the container:start command. See also... You can learn more about how to configure a JNDI datasource in Jetty at the following address: http://wiki.eclipse.org/Jetty/Howto/Configure_JNDI_Datasource Mapping a table to a Scala class Now that we know how to connect Lift applications to the database, the next step is to learn how to create mappings between a database table and a Scala object using Mapper. Getting ready We will re-use the project we created in the previous recipe since it already has the connection configured. How to do it... Carry out the following steps to map a table into a Scala object using Mapper: Create a new file named Contact.scala inside the model package under src/main/scala/code/ with the following code: package code.model import net.liftweb.mapper.{MappedString, LongKeyedMetaMapper, LongKeyedMapper, IdPK} class Contact extends LongKeyedMapper[Contact] with IdPK { def getSingleton = Contact object name extends MappedString(this, 100) } object Contact extends Contact with LongKeyedMetaMapper[Contact] { override def dbTableName = "contacts" } Add the following import statement in the Boot.scala file: import code.model.Contact Add the following code into the boot method of the Boot class: Schemifier.schemify( true, Schemifier.infoF _, Contact ) Create a new snippet named Contacts with the following code: package code.snippet import code.model.Contact import scala.xml.Text import net.liftweb.util.BindHelpers._ class Contacts { def prepareContacts_!() { Contact.findAll().map(_.delete_!) val contactsNames = "John" :: "Joe" :: "Lisa" :: Nil contactsNames.foreach(Contact.create.name(_).save()) } def list = { prepareContacts_!() "li *" #> Contact.findAll().map { c => { c.name.get } } } } Edit the index.html file by replacing the content of the div element with main as the value of id using the following code : <div data-list="Contacts.list"> <ul> <li></li> </ul> </div> Start the application. Access your local host (http://localhost:8080), and you will see a page with three names, as shown in the following screenshot: How it works... To work with Mapper, we use a class and an object. The first is the class that is a representation of the table; in this class we define the attributes the object will have based on the columns of the table we are mapping. The second one is the class' companion object where we define some metadata and helper methods. The Contact class extends the LongKeyedMapper[Contact] trait and mixes the trait IdPK. This means that we are defining a class that has an attribute called id (the primary key), and this attribute has the type Long. We are also saying that the type of this Mapper is Contact. To define the attributes that our class will have, we need to create objects that extend "something". This "something" is the type of the column. So, when we say an object name extends MappedString(this, 100), we are telling Lift that our Contact class will have an attribute called name, which will be a string that can be 100 characters long. After defining the basics, we need to tell our Mapper where it can get the metadata about the database table. This is done by defining the getSingleton method. The Contact object is the object that Mapper uses to get the database table metadata. By default, Lift will use the name of this object as the table name. Since we don't want our table to be called contact but contacts, we've overridden the method dbTableName. What we have done here is created an object called Contact, which is a representation of a table in the database called contacts that has two columns: id and name. Here, id is the primary key and of the type Long, and name is of the type String, which will be mapped to a varchar datatype. This is all we need to map a database table to a Scala class, and now that we've got the mapping done, we can use it. To demonstrate how to use the mapping, we have created the snippet Contacts. This snippet has two methods. The list method does two things; it first invokes the prepareContacts_!() method, and then invokes the findAll method from the Contact companion object. The prepareContacts_!() method also does two things: it first deletes all the contacts from the database and then creates three contacts: John, Joe, and Lisa. To delete all the contacts from the database, it first fetches all of them using the findAll method, which executes a select * from contacts query and returns a list of Contact objects, one for each existing row in the table. Then, it iterates over the collection using the foreach method and for each contact, it invokes the delete_! method which as you can imagine will execute a delete from contacts where id = contactId query is valid. After deleting all the contacts from the database table, it iterates the contactsNames list, and for each element it invokes the create method of the Contact companion object, which then creates an instance of the Contact class. Once we have the instance of the Contact class, we can set the value of the name attribute by passing the value instance.name(value). You can chain commands while working with Mapper objects because they return themselves. For example, let's say our Contact class has firstName and lastName as attributes. Then, we could do something like this to create and save a new contact: Contact.create.firstName("John").lastName("Doe").save() Finally, we invoke the save() method of the instance, which will make Lift execute an insert query, thus saving the data into the database by creating a new record. Getting back to the list method, we fetch all the contacts again by invoking the findAll method, and then create a li tag for each contact we have fetched from the database. The content of the li tags is the contact name, which we get by calling the get method of the attribute we want the value of. So, when you say contact.name.get, you are telling Lift that you want to get the value of the name attribute from the contact object, which is an instance of the Contact class. There's more... Lift comes with a variety of built-in field types that we can use; MappedString is just one of them. The others include, MappedInt, MappedLong, and MappedBoolean. All these fields come with some built-in features such as the toForm method, which returns the HTML needed to generate a form field, and the validate method that validates the value of the field. By default, Lift will use the name of the object as the name of the table's column; for example, if you define your object as name—as we did—Lift will assume that the column name is name. Lift comes with a built-in list of database-reserved words such as limit, order, user, and so on. If the attribute you are mapping is a database-reserved word, Lift will append _c at the end of the column's name when using Schemifier. For example, if you create an attribute called user, Lift will create a database column called user_c. You can change this behavior by overriding the dbColumnName method, as shown in the following code: object name extends MappedString(this, 100) { override def dbColumnName = "some_new_name" } In this case, we are telling Lift that the name of the column is some_new_name. We have seen that you can fetch data from the database using the findAll method. However, this method will fetch every single row from the database. To avoid this, you can filter the result using the By object; for example, let's say you want to get only the contacts with the name Joe. To accomplish this, you would add a By object as the parameter of the findAll method as follows: Contact.findAll(By(Contact.name, "Joe")). There are also other filters such as ByList and NotBy. And if for some reason the features offered by Mapper to build select queries aren't enough, you can use methods such as findAllByPreparedStatement and findAllByInsecureSQL where you can use raw SQL to build the queries. The last thing left to talk about here is how this example would work if we didn't create any table in the database. Well, I hope you remember that we added the following lines of code to the Boot.scala file: Schemifier.schemify( true, Schemifier.infoF _, Contact ) As it turns out, the Schemifier object is a helper object that assures that the database has the correct schema based on a list of MetaMappers. This means that for each MetaMapper we pass to the Schemifier object, the object will compare the MetaMapper with the database schema and act accordingly so that both the MetaMapper and the database schema match. So, in our example, Schemifier created the table for us. If you change MetaMapper by adding attributes, Schemifier will create the proper columns in the table. See also... To learn more about query parameters, you can find more at: https://www.assembla.com/spaces/liftweb/wiki/Mapper#querying_the_database Creating one-to-many relationships In the previous recipe, we learned how to map a Scala class to a database table using Mapper. However, we have mapped a simple class with only one attribute, and of course, we will not face this while dealing with real-world applications. We will probably need to work with more complex data such as the one having one-to-many or many-to-many relationships. An example of this kind of relationship would be an application for a store where you'll have customers and orders, and we need to associate each customer with the orders he or she placed. This means that one customer can have many orders. In this recipe, we will learn how to create such relationships using Mapper. Getting ready We will modify the code from the last section by adding a one-to-many relationship into the Contact class. You can use the same project from before or duplicate it and create a new project. How to do it... Carry out the following steps: Create a new class named Phone into the model package under src/main/scala/code/ using the following code: package code.model import net.liftweb.mapper._ class Phone extends LongKeyedMapper[Phone] with IdPK { def getSingleton = Phone object number extends MappedString(this, 20) object contact extends MappedLongForeignKey(this, Contact) } object Phone extends Phone with LongKeyedMetaMapper[Phone] { override def dbTableName = "phones" } Change the Contact class declaration from: class Contact extends LongKeyedMapper[Contact] with IdPK To: class Contact extends LongKeyedMapper[Contact] with IdPK with OneToMany[Long, Contact] Add the attribute that will hold the phone numbers to the Contact class as follows: object phones extends MappedOneToMany(Phone, Phone.contact, OrderBy(Phone.id, Ascending)) with Owned[Phone] with Cascade[Phone] Insert the new class into the Schemifier list of parameters. It should look like the following code: Schemifier.schemify( true, Schemifier.infoF _, Contact, Phone ) In the Contacts snippet, replace the import statement from the following code: import code.model.Contact To: import code.model._ Modify the Contacts.prepareContacts_! method to associate a phone number to each contact. Your method should be similar to the one in the following lines of code: def prepareContacts_!() { Contact.findAll().map(_.delete_!) val contactsNames = "John" :: "Joe" :: "Lisa" :: Nil val phones = "5555-5555" :: "5555-4444" :: "5555-3333" :: "5555-2222" :: "5555-1111" :: Nil contactsNames.map(name => { val contact = Contact.create.name(name) val phone = Phone.create.number(phones((new Random()).nextInt(5))).saveMe() contact.phones.append(phone) contact.save() }) } Replace the list method's code with the following code: def list = { prepareContacts_!() ".contact *" #> Contact.findAll().map { contact => { ".name *" #> contact.name.get & ".phone *" #> contact.phones.map(_.number.get) } } } In the index.html file, replace the ul tag with the one in the following code snippet: <ul> <li class="contact"> <span class="name"></span> <ul> <li class="phone"></li> </ul> </li> </ul> Start the application. Access http://localhost:8080. Now you will see a web page with three names and their phone numbers, as shown in the following screenshot: How it works... In order to create a one-to-many relationship, we have mapped a second table called phone in the same way we've created the Contact class. The difference here is that we have added a new attribute called contact. This attribute extends MappedLongForeignKe y, which is the class that we need to use to tell Lift that this attribute is a foreign key from another table. In this case, we are telling Lift that contact is a foreign key from the contacts table in the phones table. The first parameter is the owner, which is the class that owns the foreign key, and the second parameter is MetaMapper, which maps the parent table. After mapping the phones table and telling Lift that it has a foreign key, we need to tell Lift what the "one" side of the one-to-many relationship will be. To do this, we need to mix the OneToMany trait in the Contact class. This trait will add features to manage the one-to-many relationship in the Contact class. Then, we need to add one attribute to hold the collection of children records; in other words, we need to add an attribute to hold the contact's phone numbers. Note that the phones attribute extends MappedOneToMany, and that the first two parameters of the MappedOneToMany constructor are Phone and Phone.contact. This means that we are telling Lift that this attribute will hold records from the phones table and that it should use the contact attribute from the Phone MetaMapper to do the join. The last parameter, which is optional, is the OrderBy object. We've added it to tell Lift that it should order, in an ascending order, the list of phone numbers by their id. We also added a few more traits into the phones attribute to show some nice features. One of the traits is Owned; this trait tells Lift to delete all orphan fields before saving the record. This means that if, for some reason, there are any records in the child table that has no owner in the parent table, Lift will delete them when you invoke the save method, helping you keep your databases consistent and clean. Another trait we've added is the Cascade trait. As its name implies, it tells Lift to delete all the child records while deleting the parent record; for example, if you invoke contact.delete_! for a given contact instance, Lift will delete all phone records associated with this contact instance. As you can see, Lift offers an easy and handy way to deal with one-to-many relationships. See also... You can read more about one-to-many relationships at the following URL: https://www.assembla.com/wiki/show/liftweb/Mapper#onetomany

(For more resources related to this topic, see here.) Dependencies can be injected in a consumer class using different patterns and injecting them into a constructor is just one of them. While there are some patterns that can be followed for injecting dependencies, there are also some patterns that are recommended to be avoided, as they usually lead to undesirable results. In this article, we will examine only those patterns and antipatterns that are somehow relevant to Ninject features. Constructor Injection Constructor Injection is the most common and recommended pattern for injecting dependencies in a class. Generally this pattern should always be used as the primary injection pattern unless we have to use other ones. In this pattern, a list of all class dependencies should be introduced in the constructor. The question is what if the class has more than one constructor. Although Ninject's strategy for selecting constructor is customizable, its default behavior is selecting the constructor with more parameters, provided all of them are resolvable by Ninject. So, although in the following code the second constructor introduces more parameters, Ninject will select the first one if it cannot resolve IService2 and it will even use the default constructor if IService1 is not registered either. But if both dependencies are registered and resolvable, Ninject will select the second constructor because it has more parameters: public class Consumer { private readonly IService1 dependency1; private readonly IService2 dependency2; public Consumer(IService1 dependency1) { this.dependency1 = dependency1; } public Consumer(IService1 dependency1, IService2 dependency2) { this.dependency1 = dependency1; this.dependency2 = dependency2; } } If the preceding class had another constructor with two resolvable parameters, Ninject would throw an ActivationException exception notifying that several constructors had the same priority. There are two approaches to override this default behavior and explicitly select a constructor. The first approach is to indicate the desired constructor in a binding as follows: Bind<Consumer>().ToConstructor(arg => new Consumer(arg.Inject<IService1>())); In the preceding example, we explicitly selected the first constructor. Using the Inject<T> method that the arg argument provides, we requested Ninject to resolve IService1 in order to be injected into the specified constructor. The second method is to indicate the desired constructor using the [Inject] attribute: [Inject] public Consumer(IService1 dependency1) { this.dependency1 = dependency1; } In the preceding example, we applied the Ninject's [Inject] attribute on the first constructor to explicitly specify that we need to initialize the class by injecting dependencies into this constructor; even though the second constructor has more parameters and the default strategy of Ninject would be to select the second one. Note that applying this attribute on more than one constructor will result in the ActivationException. Ninject is highly customizable and it is even possible to substitute the default [Inject] attribute with another one, so that we don't need to add reference to the Ninject library from our consumer classes just because of an attribute: kernel.Settings.Set("InjectAttribute",typeof(MyAttribute)); Initializer methods and properties Apart from constructor injection, Ninject supports the injection of dependencies using initializer methods and property setters. We can specify as many methods and properties as required using the [Inject] attribute to inject dependencies. Although the dependencies will be injected to them as soon as the class is constructed, it is not possible to predict in which order they will receive their dependencies. The following code shows how to specify a property for injection: [Inject]public IService Service{ get { return dependency; } set { dependency = value; }} Here is an example of injecting dependencies using an injector method: [Inject]public void Setup(IService dependency){ this.dependency = dependency;} Note that only public members and constructors will be injected and even the internals will be ignored unless Ninject is configured to inject nonpublic members. In Constructor Injection, the constructor is a single point where we can consume all of the dependencies as soon as the class is activated. But when we use initializer methods the dependencies will be injected via multiple points in an unpredictable order, so we cannot decide in which method all of the dependencies will be ready to consume. In order to solve this problem, Ninject offers the IInitializable interface. This interface has an Initialize method which will be called once all of the dependencies have been injected: public class Consumer:IInitializable{ private IService1 dependency1; private IService2 dependency2; [Inject] public IService Service1 { get { return dependency1; } set { dependency1 = value; } } [Inject] public IService Service2 { get { return dependency2; } set { dependency2 = value; } } public void Initialize() { // Consume all dependencies here }} Although Ninject supports injection using properties and methods, Constructor Injection should be the superior approach. First of all, Constructor Injection makes the class more reusable, because a list of all class dependencies are visible, while in the initializer property or method the user of the class should investigate all of the class members or go through the class documentations (if any), to discover its dependencies. Initialization of the class is easier while using Constructor Injection because all the dependencies get injected at the same time and we can easily consume them at the same place where the constructor initializes the class. As we have seen in the preceding examples the only case where the backing fields could be readonly was in the Constructor Injection scenario. As the readonly fields are initializable only in the constructor, we need to make them writable to be able to use initializer methods and properties. This can lead to potential mutation of backing fields. Service Locator Service Locator is a design pattern introduced by Martin Fowler regarding which there have been some controversies. Although it can be useful in particular circumstances, it is generally considered as an antipattern and preferably should be avoided. Ninject can easily be misused as a Service Locator if we are not familiar to this pattern. The following example demonstrates misusing the Ninject kernel as a Service Locator rather than a DI container: public class Consumer{ public void Consume() { var kernel = new StandardKernel(); var depenency1 = kernel.Get<IService1>(); var depenency2 = kernel.Get<IService2>(); ... }} There are two significant downsides with the preceding code. The first one is that although we are using a DI container, we are not at all implementing DI. The class is tied to the Ninject kernel while it is not really a dependency of this class. This class and all of its prospective consumers will always have to drag their unnecessary dependency on the kernel object and Ninject library. On the other hand, the real dependencies of class (IService1 and IService2) are invisible from the consumers, and this reduces its reusability. Even if we change the design of this class to the following one, the problems still exist: public class Consumer{ private readonly IKernel kernel; public Consumer(IKernel kernel) { this.kernel = kernel; } public void Consume() { var depenency1 = kernel.Get<IService1>(); var depenency2 = kernel.Get<IService2>(); ... }} The preceding class still depends on the Ninject library while it doesn't have to and its actual dependencies are still invisible to its consumers. It can easily be refactored using the Constructor Injection pattern: public Consumer(IService1 dependency1, IService2 dependency2){ this.dependency1 = dependency1; this.dependency2 = dependency2;} Summary In this article we studied the most common DI patterns and anti-patterns related to Ninject. Resources for Article: Further resources on this subject: Introduction to JBoss Clustering [Article] Configuring Clusters in GlassFish [Article] Designing Secure Java EE Applications in GlassFish [Article]

(For more resources related to this topic, see here.) An introduction to image filtering Morphological operations and edge detection are actually types of image filtering, even though we used them in a black box sense, without really looking under the hood. Hopefully, this approach will get you accustomed to the details of image filtering a little faster. First of all, let's give a general definition of image filtering; it can be explained as the process of modifying the values of the pixels using a function that is typically applied on a local neighborhood of the image. In many situations, applying the function on a neighborhood involves a special operation, called convolution, with an operand called kernel. In this sense, you have already applied such a process in the case of erosion or dilation and even in the case of edge detection. The former processes used the strel function to create a kernel, while the latter used a kernel based on your choice of the edge detection method. But let's not get ahead of ourselves. We will try to take things one step at a time, starting by explaining neighborhood processing. Processing neighborhoods of pixels In the previous paragraph, we mentioned that the filtering process typically takes place on a specific neighborhood of pixels. When this neighborhood process is applied for all pixels, it is called sliding neighborhood operation. In it, we slide a rectangular neighborhood window through all possible positions of the image and modify its central pixel using a function of the pixels in the neighborhood. Let's see how this is done, using a numeric example. We'll start with something simple, like a linear filtering process, that is, averaging. Let's suppose that we have a small image, sized 8x8 pixels and we want to modify its pixel values, so that they get assigned with the rounded average of the pixels' values in their 3x3 neighborhoods. This will be easier to explain by using a real numeric example. Let's explain what happens in the step shown in the following image, in which the central pixel of the highlighted 3x3 neighborhood (in the fourth row and sixth column) will be replaced by the average value of all the pixels in the neighborhood (rounded to the nearest integer): Let the image be called I, the result in pixel I(4,6) will be: Substituting the values of the pixels, we can calculate the average value: Hence, the value of the central pixel of the neighborhood will become 121 (the closest integer to 120.89). By repeating the process described previously for all the pixels of the image, we get a result commonly known as mean filtering or average filtering. The final result of the entire process is shown in the following figure: You may be wondering now; the choice of neighborhood, for the example, was very convenient, but what happens when we want to change the value of a pixel on the borders of the image such as let's say pixel I(1,4)? Why was it set to 77 as shown in the image? This is indeed a valid and natural question, and you are very intuitive if you already thought about it. The answer is that the way to tackle this problem when you want your resulting image to be the same size as your original image is to involve only the neighboring pixels that exist in your calculations. However, since in our example, the calculation that has to be performed is averaging the neighborhood pixels, the denominator will still be 9, hence, it will be like we pad the rest of the neighborhood with zeros. Let's demonstrate this example as well: As shown in the previous image, the central pixel value gets evaluated as follows: Of course, since there is no 0th line, the first three operands of the addition are non-existent, hence set to zero: Therefore, the result of the averaging process for the aforementioned neighborhood will be equal to 77 (as shown in the image). This approach is not the only one we have for the image borders. We could assign the maximum possible value (255 for our example) to the non-existent pixels, or assign them the mean value of the rest of the neighborhood, and so on. The choice we make affects the quality of the borders of the image, as we will see in real pictures later on. The basics of convolution The process described previously was performed in overlapping neighborhoods of the image, but no use of a kernel was mentioned. So, what is this all about? And how does the convolution fit in this framework? Well, the truth is that the process described previously is actually describing the essence of convolution, which is passing a kernel over all possible equally sized neighborhoods of the image and using it to modify the value of the central pixel. The only problem in our case is that we did not use a specific kernel in the process described. Or did we? Let's try to find out using MATLAB code to perform two-dimensional convolution. The 3x3 neighborhood we used for the described process can be replaced by a 3x3 kernel, as long as the final result remains the same. The kernel that accomplishes this effect is a 3x3 matrix with all pixels set to 1/9. Convolving this kernel with the original image produces the same result as the aforementioned example. To demonstrate the process, we can use the two-dimensional convolution MATLAB function conv2 as follows, to get the result: >> original = [132 101 101 107 115 121 110 92 120 124 122 120 129 123 121 129 134 146 144 134 134 132 134 138 143 147 136 121 121 115 107 107 145 147 138 129 119 113 113 122 162 155 152 149 142 129 118 122 127 122 115 113 117 102 95 94 67 74 78 80 89 89 107 109]; % Create original image >> kernel = ones(3,3)*(1/9); % Create kernel >> conv_result = conv2(original, kernel,'same'); % Perform convolution >> final_result = round(conv_result) % Rounding of result The final result obtained is as follows: final_result = 53 78 75 77 79 80 77 50 84 125 122 123 124 124 122 80 90 135 133 129 125 124 123 82 96 142 138 131 124 121 120 80 100 147 142 134 126 120 116 77 95 140 136 130 124 116 112 74 79 117 115 115 112 110 107 72 43 65 65 66 66 67 66 45 As expected, the result is the same as the one calculated using the analytical process described before. The convolution kernel has done its job. In our process, we used a 8x8 original image and a 3x3 kernel with the values of all pixels as 1/9 (this is what happens when you get a 3x3 matrix with all instances of 1 and multiply it by 1/9, as we did) and finally ordered the conv2 function to produce the result using the padding process described earlier for the borders, hence calculating a result with the same dimensions as the original. But how did it do it? What exactly is convolution? Now it is time to fully understand convolution. But first, you must get acquainted with its mathematical equations. Since learning math is not the purpose of this book, we will try to give you just the basics, so that you get an idea of what this operation is all about, as it is invaluable for image filtering. The ugly mathematical truth Let's start with the mathematical definition of convolution for discrete functions (since in digital image processing all functions are discrete). To form our problem in a signal processing sense, we can define it as passing an input image I, through a Linear Space Invariant (LSI) system, performing convolution with a kernel h (also called a filter), to produce an output image, g. Hence, we get the following block diagram: This process is described mathematically by the following equation: where * is the symbol for convolution and the large Σ denotes a sum. The reason we have two sums is because our process is two-dimensional. Without going into too much detail, we can summarize the process described previously using the following steps, which are also followed in the implementation of conv2: Rotate the convolution kernel by 180 degrees to abide by the process in the double sum of the equation. Determine the central pixel of the neighborhood. This is straightforward when the neighborhood has an odd number of rows and columns, but must be based on some rule if either of the dimensions is even. Apply the rotated kernel to each pixel of the input image. This is a multiplication of each pixel in the rotated kernel by the corresponding pixel on the image neighborhood processed. It can be thought of as the weighted sum of the neighborhood pixels. The result of conv2 can be either of the following choices: full: Larger than the original image, taking into account all the pixels that can be computed using the convolution kernel, even if their center falls out of the image. This is the default choice for the function. same: Same size as the original image, using zeros to calculate border pixel values. valid: Smaller than the original image, so that it uses only pixels that have full valid neighbors in the computations. This means that when you want to produce a convolution result with the same size as the original image, you will have to use same as an input, as we did in our previous example. By now, those of you that are not very much into math may be tempted to stop reading. So, let's stop the mathematical jargon and dive into the practical examples. We know what a convolution does and we have seen an example on the pixels of a very small image, using an averaging convolution kernel. So, what does this process really do to an image? Time for action – applying averaging filters in images We will start off with an easy-to-follow example, so that all the theory described previously is demonstrated. In this example, we will also introduce some new MATLAB functions, to facilitate your understanding. Let's start: First, we load our image, which is holiday_image2.bmp: >> img = imread('holiday_image2.bmp'); Then, we generate our convolution kernel, using function fspecial and then rotate it 180 degrees: >> kernel = fspecial('average',3); >> kernel = rot90(kernel,2) The output of the code will be as follows: kernel = 0.1111 0.1111 0.1111 0.1111 0.1111 0.1111 0.1111 0.1111 0.1111 Now, it is time to use the three different ways of convolving our image: >> con1 = conv2(img,kernel); % Default usage ('full') >> con2 = conv2(img,kernel,'same'); % convolution using 'same' >> con3 = conv2(img,kernel,'valid'); % convolution using 'valid' In the previous step, you probably got a warning saying: Warning: CONV2 on values of class UINT8 is obsolete. Use CONV2(DOUBLE(A),DOUBLE(B)) or CONV2(SINGLE(A),SINGLE(B)) instead. This actually means that UNIT8 type will not be supported by conv2 in the future. To be on the safe side, you might want to use the suggestion by MATLAB and convert your image to single prior to convolving it: >> img = single(img); >> kernel = fspecial('average',3); % Create 3x3 averaging kernel >> con1 = conv2(img,kernel); % Default usage ('full') >> con2 = conv2(img,kernel,'same'); % convolution using 'same' >> con3 = conv2(img,kernel,'valid'); % convolution using 'valid' Now, we can show our results in one figure, along with the original image. This time, we are going to use an empty matrix as the second argument in imshow, to avoid having to convert our results to UNIT8: >> figure;subplot(2,2,1),imshow(img,[]),title('Original') >> subplot(2,2,2),imshow(con1,[]),title('full') >> subplot(2,2,3),imshow(con2,[]),title('same') >> subplot(2,2,4),imshow(con3,[]),title('valid') It is obvious that the three results are identical, but there is a small detail. Their size is not. So let's see if we got what we expected. In the Workspace window, you can see the difference in sizes: Let's now discuss the physical, qualitative meaning of averaging an image. What does it exactly do? The answer is; it performs blurring of the image. To examine this effect, we can crop the tower from our original and averaged image and display the result. The tower can be cropped using the following coordinates: >> tower_original = img(51:210,321:440); >> tower_blurred = con2(51:210,321:440); figure >> subplot(1,2,1),imshow(tower_original),title('Original tower') >> subplot(1,2,2),imshow(tower_blurred),title('Blurred tower') The original image and the blurred image are as follows: What just happened? The process described in the previous example demonstrated the usage of convolution in its various implementations, using the averaging kernel produced using fspecial. This function is designed to generate kernels for popular filtering tasks, as we will further analyze in the following sections. In our case, we created a 3x3 kernel of values equal to 1/9 (which is almost equal to 0.1111, hence the result in step 2). Then, the three different choices of convolution were applied and the results were displayed along with the original image. Of course, a detail such as the size of the borders cannot be easily observed in full scale, so we observed the difference in the sizes of the results. Finally, we displayed a part of the original image next to the same part of the same convolution result, to prove that the result of the averaging process is a blurring of the image. Alternatives to convolution Convolution is not the only way to perform image filtering. There is also correlation, which gives us the same result. Filtering an image using correlation can be accomplished by using the MATLAB function called filter2, which performs, as its name implies, a two-dimensional filtering of two images. The first input in this case is a kernel (filter) and the second input is an image (or in a more general case a two-dimensional matrix). We will not go into detail here, just point out that one main difference between the two methods is that correlation does not need the kernel to be rotated. The border issue remains, having the same three approaches as in the case of convolution using conv2. A demonstration on the equivalence of the two functions is given if we type in the following commands: >> img = imread('holiday_image2.bmp'); >> img = img(51:210,321:440); >> kernel = fspecial('average',3); >> kernel180 = rot90(kernel,3); >> conv_result = conv2(img,kernel180,'same'); >> corr_result = filter2(kernel,img,'same'); >> subplot(1,3,1),imshow(img),title('Original') >> subplot(1,3,2),imshow(uint8(conv_result)),title('Blurred - conv2') >> subplot(1,3,3),imshow(uint8(corr_result)),title('Blurred - filter2') The result of the preceding code is displayed as follows: In our example, the two kernels used for conv2 and filter2 are identical, since the averaging filter used is square (3x3) and all its elements are equal. The generalized process shown will be useful when we have a more complex kernel. Using imfilter The two alternative solutions for performing image filtering presented so far have their origin in general two-dimensional signal processing theory. This means that they should be expanded for three-dimensional signals when we have to deal with colored image filtering. The process is pretty straightforward and involves repeating the process for all three separate colored channels. But why do that, when we have a function that takes care of checking the image before applying the filter and then selecting the correct method? This specialized function is called imfilter and it is designed for handling images, regardless if they are grayscale or color. This function can implement both filtering methods described in previous paragraphs and it can also define the result to be same or full. Its extra functionality comes in the selection of the way it handles boundary values, and the automatic processing of color images. Furthermore, this function performs the needed conversions, in case the image input is integer-valued. Combined with the fspecial function, this will probably be your most valuable tool in MATLAB when it comes to image filtering.

(For more resources related to this topic, see here.) Getting ready This recipe is going to log a large JSON message to the console, so large that it will actually overflow STS's default console buffer. To ensure that we don't miss anything, we're going to change STS's configuration to have an unlimited console buffer. Open the Preferences dialog by selecting Window | Preferences from the menu if you are running Windows, or by pressing ?+, if you are using a Mac. Open the Run/Debug item and select the Console item. Uncheck the Limit console output checkbox, and then click on the OK button. How to do it... Let's start by creating our project and updating its dependencies. Launch the New Spring Project wizard by navigating to File | New | Spring Project in the menu. In the dialog box, enter the project name WebServiceProject, expand the Integration folder, and select Spring Integration Project (Standalone) – Simple. Click on the Next button. Allow STS to download the project template if you are prompted to do so. Enter the groupId as com.example, artifactId as web-service-project, and top-level package as com.example.webserviceproject. Click on the Finish button to create the project. Open the Maven pom.xml file. Click on the Dependencies tab, and add the following dependencies: org.springframework.integration : spring-integrationhttp : ${spring.integration.version} org.springframework : spring-test : 3.1.3.RELEASE (Spring Integration 2.2.4.RELEASE depends on this version of Spring, and it's a good idea to be consistent) org.codehaus.jackson : jackson-core-asl : 1.9.12 org.codehaus.jackson : jackson-mapper-asl : 1.9.12 Save the file to update the project's dependencies. Now let's create an integration test. Expand the src/test/java source folder, right-click on the top-level package under it, and select New | JUnit Test Case from the pop-up menu. Name the test WebServiceTest, and click on the Finish button. We're creating a Spring context test, so add the following annotations to the top of the class: @ContextConfiguration(locations = "classpath:META-INF/spring/integration/spring-integration-context.xml")@RunWith(SpringJUnit4ClassRunner.class) Our test is going to send a Spring Integration message down one channel and then wait for a response on another channel. We need to inject the following two channels into our test: @Autowired private MessageChannel weatherRequest;@Autowired private SubscribableChannel weatherResponse; Create the following test case: @Testpublic void shouldRetrieveWeatherDataFromWebService() {weatherResponse.subscribe(new MessageHandler() {@Overridepublic void handleMessage(Message<?> message)throws MessagingException {Map payload = (Map) message.getPayload();Map observations = (Map) payload.get("observations");List data = (List) observations.get("data");Map latestData = (Map) data.get(0);assertNotNull(latestData);assertEquals("Sydney - Observatory Hill",latestData.get("name"));assertNotNull(latestData.get("air_temp"));assertNotNull(latestData.get("local_date_time_full"));}});weatherRequest.send(MessageBuilder.withPayload(new WeatherRequest("N", "94768")).build());} Add all required imports using Quick Fixes. The instantiation of the WeatherRequest object will cause a compilation error because the class does not yet exist. Use a Quick Fix to create the class, taking care to change its source folder to WebServiceProject/src/main/java. Create the class as follows: package com.example.webserviceproject;public class WeatherRequest {private final String stateCode;private final String locationId;public WeatherRequest(String stateCode, String locationId) {this.stateCode = stateCode;this.locationId = locationId;}public String getStateCode() {return stateCode;}public String getLocationId() {return locationId;}} Run the test. It will fail, complaining that no Spring beans could be injected into our test. We have our failing test. Now it's time to create our integration pipeline. Open spring-integration-context.xml (found in the src/main/ resources/META-INF/spring/integration folder). Click on the Namespaces tab. You will find that the int namespace is already checked. Check the int-http namespace as well. Click on the integration-graph tab and maximize the editor to give yourself more space. This template project has already created an integration pipeline for us, but we want to create our own, so select all of the components in the workspace area by dragging a box around them, and then press the Delete key to remove them. Expand the Channels category and drag a channel component into the workspace. Double-click on it to bring up its properties in the Properties view. Change its id to weatherRequest. Remember to close the Properties view after editing the properties of each integration component. Drag another channel component into the workspace, and use the Properties view to set its id to weatherServiceResponse. Drag a publish-subscribe-channel component into the workspace. Give it the id weatherResponse. Drag a logging-channel-adapter component into the workspace. Give it the id logger, set its level to INFO, and its logger-name to com.example. webserviceproject.webserviceresponse. Now expand the http category and drag an outbound-gateway component into the workspace. Set its id property to weatherService, its url property to http://www.bom.gov.au/fwo/ID{stateCode}60901/ ID{stateCode}60901.{locationId}.json, its http-method to GET, and expected-response-type to java.lang.String. Expand the Transformation category and drag a json-to-object-transformer component into the workspace. Set its type property to java.util.HashMap. Now we need to connect everything together. Select the connection tool from the palette and make the following connections: weatherRequest channel to weatherService gateway weatherService gateway to weatherServiceResponse channel weatherServiceResponse channel to json-to-object-transformer json-to-object-transformer to weatherResponse channel Select the mapping/recipient/wire-tap tool from the palette and create a connection from the weatherServiceResponse channel to the logging-channel-adapter channel. Our pipeline should now look like the following figure: Select the integration tab. In the Integration Overview panel, expand all of the beans and then right-click on the int-http:outbound-gateway bean. Select Insert <int-http:uri-variable> element from the pop-up menu. This will create a new sub element and show its properties on the right side of editor. Set its name property to stateCode and its expression to payload.stateCode. Create another uri-variable property, setting its name to locationId, and its expression to payload.locationId. And we're done! Save the file, then re-run the WebServiceTest. If all was configured correctly, the test will take a few seconds to run (it has to contact the web service), and then you should be rewarded with a green bar. Maximize the Console view, and you will find the JSON string returned from the web service. How it works... In this recipe, we used a few different Spring Integration components to make a request to a web service and to transform its response to a Map class so that our test could pick out pieces of data. We also used a wire tap component to log messages to the console. Channels We have seen a couple of different channel implementations in action in this recipe. The weatherRequest and weatherService channels are examples of the default Spring Integration channel, DirectChannel , which is a simple point-to-point channel (that is, there is one sender on one side of the channel and one receiver on the other). By contrast, the weatherRequest channel is a PublishSubscribeChannel , which allows for multiple receivers to register for message events (in our test, we only had the one).Spring Integration provides a number of other channel types designed for different requirements. See the Spring Integration Reference Manual ( http://static.springsource.org/spring-integration/reference/htmlsingle ) for further information. URI variables You would have noticed when setting the URL property of the HTTP outbound gateway that the value included two placeholders, namely {stateCode} and {locationId}. We configured a URL variable for each of these, giving them the values payload.stateCode and payload.locationId respectively. These are simply JavaBean property references. In our case, the payload was a WeatherRequest instance, and we had defined both the stateCode and locationId JavaBean properties on that class. The values of those properties are substituted into the placeholders in the configured URL string to determine the actual URL requested. Wire taps and logging When we connected the weatherServiceResponse channel with logging-channel-adapter, under the hood, STS configured a special type of channel interceptor called a wire tap. A wire tap takes messages passing through a channel, and sends a copy to another channel. This makes them excellent for logging and auditing purposes. When we configured our logging-channel-adapter , we told it to log at the INFO level using the logger name com.example.webserviceproject.webserviceresponse . We could have used any name, of course, but the log4j configuration provided in the project template (the log4j.xml file under src/main/resources ) already had an INFO level logging threshold configured for com.example.webserviceproject loggers, so it was convenient for us to choose the logger name that we did. Summary In this article, we retrieved weather data from the Australian Bureau of Meteorology and also learned about how STS and Spring Integration allow us to interact with web services with very little effort. Resources for Article : Further resources on this subject: Vaadin project with Spring and Handling login with Spring [Article] So, what is Spring for Android? [Article] Migration to Spring Security 3 [Article]