Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7019 Articles
article-image-using-content-type-effectively-plone-intranet
Packt
04 Aug 2010
4 min read
Save for later

Using Content Type Effectively with Plone Intranet

Packt
04 Aug 2010
4 min read
(For more resources on Plone, see here.) Designing our intranet information architecture No one uses a knowledge system (such as our intranet) if the information stored in it is hard to find or consume. We will have to specially emphasize on thinking about not only a good navigation schema, but also a successful one for our intranet. The definition of success is different for every interested group, organization, enterprise, or any kind of entity our intranet will serve. There are a lot of navigation schemas we may want to implement, but it is our task to find out what will be more suitable for our organization. To achieve this, we will have to use both hierarchy and metadata taxonomy wisely. Obviously, the use of folders and collections will help achieve this endeavor. The first-level folders or sections are very important and we will have to keep an eye on them when designing our intranet. Also, we should not forget the next levels of folders, because they have a key role in a success navigation schema. The use of metadata, and specifically categorization of content, will also play an important role in our intranet. The continuous content cataloging is crucial to achieve a good content search and the users should be made aware of it. An intranet where the search of content is inefficient and difficult is an unsuccessful intranet, and with time, the users will abandon it. At this point, we should analyze the navigation needs of our intranet. Think about how the people will use it, how will they contribute contents to it, and how will they find things stored in it. In this analysis, it is very important to think about security. Navigation and security are closely related because most probably we define security by containers. There are some standard schemas: by organization structure, by process, by product, and so on. By organization is the most usual case. Everybody has a very clear idea of the organizational schema of an enterprise or organization, and this factor makes it easier to implement this type of schema. In this kind of schema, the first-level sections are divided into departments, teams, or main groups of interest. If our intranet is small and dedicated to one or few points of interest, then these must take precedence over the first level section folders. Keep the following things in mind: Our intranet will be more usable if we can keep our intranet sections clean and clear Fight against those people who believe that his (or her) department is more important than others and want to assault our intranet sections Let them know that maintaining a good intranet structure will be more useful and will help contribute to its success Second levels are also very important. They should be perdurable in time, interesting to users of all sections, and they should divide information and contents clearly. Two subsections shouldn't contain elements of the same subject or kind. For example, these might be a typical second level: Documentation Meetings Events News Forums, tracker, or some application specific to the current section All of these are very commonly seen in an intranet. It is a good practice to create these second-level sections in advance, so that people can adapt to them. Teach people to categorize content. This will help intranet searches incredibly and will help create collections and manage contents more effectively. If needed, make a well-known set of categories publicly available for people to use. This would prevent the repetition of categories and the rational use of them. Notice that there can be several types of categories: Subject: Terms that describe the subject of the content Process: Terms that identify the content with the organizational process Flags: Flags such as Strongly Recommended Products: Terms from the products, standards, and technology names that describe the subject matter of the resource Labels: Terms used to ensure that the resource is listed under the appropriate label Keywords: Terms used to describe the resource Events: Terms used to identify events which are recurrent with the content There are other metadata also which influence the improvement of the navigation and search abilities of the intranet such as: Title Description URL, the ID of each content Don't forget to teach your users about content contribution best practices before deploying the intranet. We and our intranet users will appreciate it a lot. Once we have settled down on some practices which are best for information architecture, we should know how to use some interesting Plone features that will help us build navigation and sort the information on our intranet.
Read more
  • 0
  • 0
  • 2946

article-image-python-using-doctest-documentation
Packt
13 May 2011
6 min read
Save for later

Python: Using doctest for Documentation

Packt
13 May 2011
6 min read
  Python Testing Cookbook Over 70 simple but incredibly effective recipes for taking control of automated testing using powerful Python testing tools The reader can benefit from the previous article on Testing in Python using doctest.   Documenting the basics Python provides out-of-the-box capability to put comments in code known as docstrings. Docstrings can be read when looking at the source and also when inspecting the code interactively from a Python shell. In this recipe, we will demonstrate how these interactive docstrings can be used as runnable tests. What does this provide? It offers easy-to-read code samples for the users. Not only are the code samples readable, they are also runnable, meaning we can ensure the documentation stays up to date. How to do it... With the following steps, we will create an application combined with runnable docstring comments, and see how to execute these tests: Create a new file named recipe16.py to contain all the code we write for this recipe. Create a function that converts base-10 numbers to any other base using recursion. def convert_to_basen(value, base): import math def _convert(remaining_value, base, exp): def stringify(value): if value > 9: return chr(value + ord('a')-10) else: return str(value) if remaining_value >= 0 and exp >= 0: factor = int(math.pow(base, exp)) if factor <= remaining_value: multiple = remaining_value / factor return stringify(multiple) + _convert(remaining_value-multiple*factor, base, exp-1) else: return "0" + _convert(remaining_value, base, exp-1) else: return "" return "%s/%s" % (_convert(value, base, int(math.log(value, base))), base) Add a docstring just below the external function, as shown in the highlighted section of the following code. This docstring declaration includes several examples of using the function. def convert_to_basen(value, base): """Convert a base10 number to basen.edur >>> convert_to_basen(1, 2) '1/2' >>> convert_to_basen(2, 2) '10/2' >>> convert_to_basen(3, 2) '11/2' >>> convert_to_basen(4, 2) '100/2' >>> convert_to_basen(5, 2) '101/2' >>> convert_to_basen(6, 2) '110/2' >>> convert_to_basen(7, 2) '111/2' >>> convert_to_basen(1, 16) '1/16' >>> convert_to_basen(10, 16) 'a/16' >>> convert_to_basen(15, 16) 'f/16' >>> convert_to_basen(16, 16) '10/16' >>> convert_to_basen(31, 16) '1f/16' >>> convert_to_basen(32, 16) '20/16' """ import math Add a test runner block that invokes Python's doctest module. if __name__ == "__main__": import doctest doctest.testmod()   From an interactive Python shell, import the recipe and view its documentation. Run the code from the command line. In the next screenshot, notice how nothing is printed. This is what happens when all the tests pass. Run the code from the command line with -v to increase verbosity. In the following screenshot, we see a piece of the output, showing what was run and what was expected. This can be useful when debugging doctest. How it works... The doctest module looks for blocks of Python inside docstrings and runs it like real code. >>> is the same prompt we see when we use the interactive Python shell. The following line shows the expected output. doctest runs the statements it sees and then compares the actual with the expected output. There's more... doctest is very picky when matching expected output with actual results. An extraneous space or tab can cause things to break. Structures like dictionaries are tricky to test, because Python doesn't guarantee the order of items. On each test run, the items could be stored in a different order. Simply printing out a dictionary is bound to break it. It is strongly advised not to include object references in expected outputs. These values also vary every time the test is run.   Catching stack traces It's a common fallacy to write tests only for successful code paths. We also need to code against error conditions including the ones that generate stack traces. With this recipe, we will explore how stack traces are pattern-matched in doc testing that allows us to confirm expected errors. How to do it... With the following steps, we will see how to use doctest to verify error conditions: Create a new file called recipe17.py to write all our code for this recipe. Create a function that converts base 10 numbers to any other base using recursion. def convert_to_basen(value, base): import math def _convert(remaining_value, base, exp): def stringify(value): if value > 9: return chr(value + ord('a')-10) else: return str(value) if remaining_value >= 0 and exp >= 0: factor = int(math.pow(base, exp)) if factor <= remaining_value: multiple = remaining_value / factor return stringify(multiple) + _convert(remaining_value-multiple*factor, base, exp-1) else: return "0" + _convert(remaining_value, base, exp-1) else: return "" return "%s/%s" % (_convert(value, base, int(math.log(value, base))), base) Add a docstring just below the external function declaration that includes two examples that are expected to generate stack traces. def convert_to_basen(value, base): """Convert a base10 number to basen. >>> convert_to_basen(0, 2) Traceback (most recent call last): ... ValueError: math domain error >>> convert_to_basen(-1, 2) Traceback (most recent call last): ... ValueError: math domain error """ import math Add a test runner block that invokes Python's doctest module. if __name__ == "__main__": import doctest doctest.testmod() Run the code from the command line. In the following screenshot, notice how nothing is printed. This is what happens when all the tests pass. Run the code from the command line with -v to increase verbosity. In the next screenshot, we can see that 0 and -1 generate math domain errors. This is due to using math.log to find the starting exponent. How it works... The doctest module looks for blocks of Python inside docstrings and runs it like real code. >>>; is the same prompt we see when we use the interactive Python shell. The following line shows the expected output. doctest runs the statements it sees and then compares the actual output with the expected output. With regard to stack traces, there is a lot of detailed information provided in the stack trace. Pattern matching the entire trace is ineffective. By using the ellipsis, we are able to skip the intermediate parts of the stack trace and just match on the distinguishing part: ValueError: math domain error. This is valuable, because our users can see not only the way it handles good values, but will also observe what errors to expect when bad values are provided.  
Read more
  • 0
  • 0
  • 2946

article-image-linux-e-mail-providing-webmail-access-part-1
Packt
05 Feb 2010
10 min read
Save for later

Linux E-mail: Providing Webmail Access (Part 1)

Packt
05 Feb 2010
10 min read
The webmail solution A webmail solution is a program or a series of scripts that is run on a server, is accessible over the web, and provides access to e-mail functions similar to a conventional mail client. It is used by Yahoo! Mail, Microsoft Hotmail, Microsoft Outlook Web Access, and Gmail as the primary interface to their e-mail solutions. You may already be familiar with various forms of webmail. Though we will be examining the SquirrelMail webmail solution specifically, the benefits and drawbacks of SquirrelMail apply to most webmail systems in the market. From this point of view, we will approach the issue from a general perspective, and then in detail for the SquirrelMail package. The benefits This section will focus on the advantages offered by installing and maintaining a webmail solution. As with any list, it is not entirely comprehensive. Many benefits will be specific to a particular case; it is important to carefully examine and consider how the following qualities impact your individual situation. The main benefits we will explore in this section are as follows: Easy and quick access with little or no setup Easy remote access No need to maintain client software or configuration Provision of a user interface to configure mail server options Possible security benefits Easy and quick access Although well suited to certain situations, traditional mail access solutions can often be difficult to set up and maintain. Generally, this involves installing software on a client's local computer and configuring it. This can be difficult, especially in cases where users need to set up the software themselves. Configuration can often be even more problematic as some users may not be competent enough to follow even a very detailed set of instructions. These instructions also need to be provided and maintained for many different mail clients on many different platforms. However, a webmail solution does not have most of these problems. All of the user's settings can be configured on the server as the application itself resides on the server. This translates to almost zero set up time for the user. Once they have received their login credentials, they can visit the webmail site and instantly have access to all of their mail. The user is able to access the site instantly to send and receive e-mail. As the Internet is so common now, many users will be familiar with webmail sites such as Google Mail and Windows Live Hotmail, which offer free e-mail services. However, the user interface provided by an open source package may be more primitive and lack some visual features. Squirrelmail provides access to e-mail, including the ability to send and receive attachments, and offers a good user interface. It is also worth mentioning that a webmail solution can offer what certain traditional mail clients call groupware features. These features let groups communicate and coordinate in ways that complement e-mail communication. Examples of groupware components are private calendars, shared calendars, meeting scheduling, To-do lists, and other similar tools. These applications can be preconfigured so that a user can instantly begin using them without having to configure them on their own. Several SquirrelMail plugins which implement these features are available from the SquirrelMail website. Easy remote access Another problem with traditional mail access software is that it is not portable, as an e-mail client needs to be installed and configured on a computer. Once it has been downloaded, installed, and configured on a particular computer, it is accessible only on that computer. Without webmail, users on the road will not be able to access e-mail from friends' computers, mobile devices, or Internet booths at airports. However, in a webmail solution, e-mail can be accessed from any location with an Internet connection. Employees can access their work e-mail from any computer with an Internet connection and a suitable browser. As the administrator, you can choose to permit or deny users from accessing e-mail in insecure situations. By requiring the connection to be encrypted, you can ensure that when a user is in a remote location, their communication with the server is secure. No need to maintain clients Even if software mail clients have been installed and properly configured, they must be maintained. When a new version is released, all clients must be updated. This is not necessarily an easy task. Software that does not work as expected can result in a large number of support-desk calls. Updating the software on each client can be a very large administrative burden. In fact, many expensive software packages are designed for the specific purpose of updating software on individual machines automatically. Despite this, problems specific to each local machine often arise and must be solved individually. It may also be difficult to convey instructions or notifications to remote branch locations or remote workers. With a webmail solution, this is not necessary. In contrast to this, a webmail solution is centrally maintained and administered. The webmail application resides on the server. With webmail, only the web server and the webmail package need to be upgraded. Any exceptions or problems that arise can be dealt with before or during the upgrade. The software upgrade itself can be run through on a test system before it is deployed on a live system. Although changes in settings are rare with SquirrelMail, it is possible to update a user's settings to make them compatible with the changes introduced in an updated version. Additionally, while upgrading or changing a mail server platform, testing effort can be greatly reduced as only supported browser versions need to be tested. It is advisable to mandate particular browser versions for corporate computers. In contrast with e-mail clients, there is no need to test on all of the possible clients and software platforms. Configuring mail server interface via the user interface Many traditional desktop e-mail clients provide only e-mail functionality and nothing more. Often there is no support for other essential tasks (such as changing the access password) that are performed on behalf of a mail user. Certain configuration options that reside on the server may require additional software applications or external solutions to provide for these needs. Examples of mail server options that may need to be configured include each user's password and junk mail filtering settings. In the case of the SquirrelMail webmail application, many plugins have been developed that provide these features. For example, a user is able to change his/her password directly from the webmail interface. Also, there are plugins and systems that allow users to easily sign up without any direct human intervention. This may be useful if you are interested in providing a service where users can sign up without needing an administrative overhead. Possible security benefits This issue can be seen in two different ways—it is for this reason that the title is listed as "Possible" security benefits. Nonetheless, this is still an interesting point to examine. In the software client access model, e-mail is traditionally downloaded onto the local user's computer, being stored in one or more personal folders. From a security perspective, this may be a bad thing. Users of the system may not be as conscientious or knowledgeable about computer security as a trained computer administrator might be. It is often much easier to gain unauthorized access to an end user's computer than a properly configured and secured server. The implication is that someone who stole a company laptop might be able to access all the e-mail stored on that computer. There is one more disadvantage associated with the client access model. Even if an employee is terminated, he/she may still have access to all of the e-mail that  resides on his/her local office computer. It may take a certain amount of time before important information may be secured. A disgruntled worker might easily connect an external storage source to their local office computer and download any data they desire. It is also worth noting that in a webmail model, all e-mail is centrally stored. If an attacker were to gain access to the central e-mail server, he/she might access all the e-mail stored on that server. However, it is possible that an attacker will gain access to all the e-mail if the central mail server is compromised even if a webmail system is not used. The disadvantages This section focuses on the disadvantages resulting from providing and supporting a webmail solution. The warning given in the previous section applies: This list is not entirely comprehensive. Each situation is unique, and may bring its unique disadvantages. We will go over the following disadvantages of a webmail solution: Performance issues Compatibility with large e-mail volumes Compatibility with e-mail attachments Security issues Performance The traditional e-mail client is designed in the client-server model. One mail server accepts and delivers e-mail to and from other mail servers. However, a desktop mail client can offer many additional productivity-enhancing features such as message sorting, searching, contact list management, attachment handling, along with more recent ones such as spam filtering and message encryption. Each of these features may require a certain amount of processing power. The required level of processing power may be negligible when it comes to storing one user's e-mail on a desktop computer, but providing these features may be problematic when applied on a larger scale to a single server. When examining the performance issue, it is important to consider the number of potential users that will access the webmail application and size a server accordingly. A single server may be able to easily handle something like 300 users, but if the number of users increases significantly, server load may become an issue. For example, searching through several years' archived mail may take a few seconds on a client's computer. When one user performs this task using webmail, the load will be similar. However, if many clients request this operation at short intervals or concurrently, it may be difficult for the server to process all the requests in a timely manner. This may result in pages being served at a slower rate or, in extreme circumstances, the server failing to respond. Optimally, load testing in the appropriate conditions should be performed if there is any concern that a server may not be able to handle a particular load of users. Compatibility with large e-mail volumes The webmail solution is not well suited to large mail volumes. This disadvantage is related to the previous one, but is more related to the amount of data sent. Even with a relatively low number of users, a large volume of e-mails may be difficult to manage in a webmail application. There are mainly the following two reasons for this: Firstly, every e-mail viewed and every folder listed must be sent from the server each time. With a traditional e-mail client, the client software can manage e-mail messages, creating lists and views to suit the user. However, with a webmail solution, this is performed on the server. So, if there are many users, this overhead may use a significant proportion of the server's resources. Secondly, each interaction with the webmail application requires a Hypertext Transfer Protocol (HTTP) request and response. These messages will typically be larger than those between an e-mail server and a desktop e-mail client. There may also be less parallelism when using a webmail client, inother words, fewer things going on at the same time. A desktop e-mail client may be able to check for new e-mails in several folders at the same time, but a webmail client will typically perform these tasks one after the other, if they occur automatically at all.
Read more
  • 0
  • 0
  • 2946

article-image-launching-spark-cluster
Packt
31 Mar 2016
7 min read
Save for later

Launching a Spark Cluster

Packt
31 Mar 2016
7 min read
 In this article by Omar Khedher, author of OpenStack Sahara Essentials we will use Sahara to create and launch a Spark cluster. Sahara provides several plugins to provision Hadoop clusters on top of OpenStack. We will be using Spark plugins to provision Apache Spark clusters using Horizon. (For more resources related to this topic, see here.) General settings The following diagram illustrates our Spark cluster topology, which includes: One Spark master node: This runs the Spark Master and the HDFS NameNode Three Spark slave nodes: These run a Spark Slave and an HDFS DataNode each Preparing the Spark image The following link provides several Sahara images available for download for different plugins: http://sahara-files.mirantis.com/images/upstream/liberty. Note that the upstream Sahara image files are destined for the OpenStack Liberty release. From Horizon, click on Compute and select Images, click on Create Image and add the new image, as shown here: We will need to upload the downloaded image to Glance so that it can be registered in the Sahara image registry catalog. Make sure that the new image is active. Click on the Data Processing tab and select Image Registry. Click on Register Image to register the new uploaded Glance image to Sahara, as shown here: Click on Done and the new Spark image is ready to start launching the Spark cluster. Creating the Spark master group template Node group templates in Sahara facilitate the configuration of a set of instances that have same properties, such as RAM and CPU. We will start by creating the first node group template for the Spark master. From the Data Processing tab, select Node Group Templates and click on Create Template. Our first node group template will be based on Apache Spark with Version 1.3.1, as shown here: The next wizard will guide to specifying the name of the template, the instance flavor, the storage location, and which floating IP pool will be assigned to the cluster instance: The next tab in same wizard will guide you to selecting which kind of process the nodes in the cluster will run. In our case, the Spark master node group template will include Spark master and HDFS namenode processes, as shown here: The next tab in the wizard exposes more choices regarding the security groups that will be applied for the template cluster nodes: Auto security group: This will automatically create a set of security groups that will be directly applied to the instances of the node group template Default security group: Any existing security groups in the OpenStack environment configured as default will be applied the instances of the node group template The last tab in the wizard exposes more specific HDFS configuration that depend on the available resources of the cluster, such as disk space, CPU and memory: dfs.datanode.handler.count: How many server threads there are for the datanode dfs.datanode.du.reserved: How much of the available disk space will not be taken into account for HDFS use dfs.namenode.handler.count: How many server threads there are for the namenode dfs.datanode.failed.volumes.tolerated: How many volumes are allowed to fail before a datanode instance stops dfs.datanode.max.xcievers: What is the maximum number of threads to be used in order to transfer data to/from the DataNode instance. Name Node Heap Size: How much memory will be assigned to the heap size to handle workload per NameNode instance Data Node Heap Size: How much memory will be assigned to the heap size to handle workload per DataNode instance Creating the Spark slave group template Creating the Spark slave group template will be performed in the same way as the Spark master group template except the assignment of the node processes. The Spark slave nodes will be running Spark slave and HDFS datanode processes, as shown here: Security groups and HDFS parameters can be configured the same as the Spark master node group template. Creating the Spark cluster template Now that we have defined the basic templates for the Spark cluster, we will need to compile both entities into one cluster template. In the Sahara dashboard, select Cluster Templates and click on Create Template. Select Apache Spark as the Plugin name, with version 1.3.1, as follows: Give the cluster template a name and small description. It is also possible to mention which process in the Spark cluster will run in a different compute node for high-availability purposes. This is only valid when you have more than one compute node in the OpenStack environment. The next tab in the same wizard allows you to add the necessary number of Spark instances based on the node group templates created previously. In our case, we will use one master Spark instance and three slave Spark instances, as shown here: The next tab, General Parameters, provides more advanced cluster configuration, including the following: Timeout for disk preparing: The cluster will fail when the duration of formatting and mounting the disk per node exceeds the timeout value. Enable NTP service: This option will enable all the instances of the cluster to synchronized time. An NTP file can be found under /tmp when cluster nodes are active. URL of NTP server: If mentioned, the Spark cluster will use the URL of the NTP server for time synchronization. Heat Wait Condition timeout: Heat will throw an error message to Sahara and the cluster will fail when a node is not able to boot up after a certain amount of time. This will prevent Sahara spawning instances indefinitely. Enable XFS: Allows XFS disk formatting. Decommissioning Timeout: This will throw an error when scaling data nodes in the Spark cluster takes more than the time mentioned. Enable Swift: Allows using Swift object storage to pull and push data during job execution. The Spark Parameters tab allows you to specify the following: Master webui port: Which port will access the Spark master web user interface. Work webui port: Which port will access the Spark slave web user interface. Worker memory: How much memory will be reserved for Spark applications. By default, if all is selected, Spark will use all the available RAM is the instance minus 1 GB. Spark will not run properly when using a flavor having RAM less than 1 GB. Launching the Spark cluster Based on the cluster template, the last step will require you to only push the button Launch Cluster from the Clusters tab in the Sahara dashboard. You will need only to select the plugin name, Apache Spark, with version 1.3.1. Next, you will need to name the new cluster, select the right cluster template created previously, and the base image registered in Sahara. Additionally, if you intend to access the cluster instances via SSH, select an existing SSH keypair. It is also possible to select from which network segment you will be able to manage the cluster instances; in our case, an existing private network, Private_Net10, will be used for this purpose. Launch the cluster; this will take a while to finish spawning four instances forming the Spark cluster. The Spark cluster instances can be listed in the Compute Instances tab, as shown here: Summary In this article, we created a Spark cluster using Sahara in OpenStack by means of the Apache Spark plugin. The provisioned cluster includes one Spark master node and three Spark slave nodes. When the cluster status changes to theactive state, it is possible to start executing jobs. Resources for Article:  Further resources on this subject: Introducing OpenStack Trove [article] OpenStack Performance, Availability [article] Monitoring Physical Network Bandwidth Using OpenStack Ceilometer [article]
Read more
  • 0
  • 0
  • 2945

article-image-simple-item-selector-using-jquery
Packt
09 Nov 2009
4 min read
Save for later

Simple Item Selector Using jQuery

Packt
09 Nov 2009
4 min read
(For more resources on jQuery, see here.) Adding jQuery to your page You can download the latest version of jQuery from jQuery site (http://jquery.com/) and can be added as a reference to your web pages accordingly. You can reference a local copy of jQuery using <script> tag in the page. Either you can reference your local copy or you can directly reference remote copy from jQuery.com or Google Ajax API (http://ajax.googleapis.com/ajax/libs/jquery/1.3/jquery.min.js) Prerequisite Knowledge In order to understand the code, one should have the basic knowledge of HTML, CSS, JavaScript and basic knowledge of jQuery. Ingredients Used HTML CSS jQuery Photoshop (Used for Designing of Image Buttons and Backgrounds) Preview / Download If you would like to see the working example, please do click here http://www.developersnippets.com/snippets/jquery/item_selector/item_selector.html). And if you would like to download the snippet, click here (http://www.developersnippets.com/snippets/jquery/item_selector/item_selector.zip) Figure 1: Snapshot of "Simple Item Selector using jQuery" Figure 2: Overview of div containers and image buttons used Successfully tested The above application has been successfully tested on various browsers like IE 6.0, IE 7, IE 8, Mozilla Firefox (Latest Version), Google Chrome and Safari Browser (4.0.2) respectively. HTML Code Below is the HTML code with comments for you to understand it better. <!-- Container --><div id="container"> <!-- From Container --> <div class="from_container"> <select id="fromSelectBox" multiple="multiple"> <option value="1">Adobe</option> <option value="2">Oracle</option> <option value="3">Google</option> <option value="4">Microsoft</option> <option value="5">Google Talk</option> <option value="6">Google Wave</option> <option value="7">Microsoft Silver Light</option> <option value="8">Adobe Flex Professional</option> <option value="9">Oracle DataBase</option> <option value="10">Microsoft Bing</option> </select><br /> <input type="image" src="images/selectall.jpg" class="selectall" onclick="selectAll('fromSelectBox')" /><input type="image" src="images/deselectall.jpg" class="deselectall" onclick="clearAll('fromSelectBox')" /> </div> <!-- From Container [Close] --> <!-- Buttons Container --> <div class="buttons_container"> <input type="image" src="images/topmost.jpg" id="topmost" /><br /> <input type="image" src="images/moveup.jpg" id="moveup" /><br /> <input type="image" src="images/moveright.jpg" id="moveright" /><br /> <input type="image" src="images/moveleft.jpg" id="moveleft" /><br /> <input type="image" src="images/movedown.jpg" id="movedown" /><br /> <input type="image" src="images/bottommost.jpg" id="bottommost" /><br /> </div> <!-- Buttons Container [Close] --> <!-- To Container --> <div class="to_container"> <select id="toSelectBox" multiple="multiple"></select><br /> <input type="image" src="images/selectall.jpg" class="selectall" onclick="selectAll('toSelectBox')" /><input type="image" src="images/deselectall.jpg" class="deselectall" onclick="clearAll('toSelectBox')" /> </div> <!-- To Container [Close] --> <!-- To Container --> <div class="ascdes_container"> <input type="image" src="images/ascending.jpg" id="ascendingorder" style="margin:1px 0px 2px 0px;" onclick="ascOrderFunction()" /><br /> <input type="image" src="images/descending.jpg" id="descendingorder" onclick="desOrderFunction()" /> </div> <!-- To Container [Close] --> <div style="clear:both"></div></div><!-- Container [Close] -->
Read more
  • 0
  • 0
  • 2945

article-image-understanding-tdd
Packt
03 Sep 2015
31 min read
Save for later

Understanding TDD

Packt
03 Sep 2015
31 min read
 In this article by Viktor Farcic and Alex Garcia, the authors of the book Test-Driven Java Development, we will go through TDD in a simple procedure of writing tests before the actual implementation. It's an inversion of a traditional approach where testing is performed after the code is written. (For more resources related to this topic, see here.) Red-green-refactor Test-driven development is a process that relies on the repetition of a very short development cycle. It is based on the test-first concept of extreme programming (XP) that encourages simple design with a high level of confidence. The procedure that drives this cycle is called red-green-refactor. The procedure itself is simple and it consists of a few steps that are repeated over and over again: Write a test. Run all tests. Write the implementation code. Run all tests. Refactor. Run all tests. Since a test is written before the actual implementation, it is supposed to fail. If it doesn't, the test is wrong. It describes something that already exists or it was written incorrectly. Being in the green state while writing tests is a sign of a false positive. Tests like these should be removed or refactored. While writing tests, we are in the red state. When the implementation of a test is finished, all tests should pass and then we will be in the green state. If the last test failed, implementation is wrong and should be corrected. Either the test we just finished is incorrect or the implementation of that test did not meet the specification we had set. If any but the last test failed, we broke something and changes should be reverted. When this happens, the natural reaction is to spend as much time as needed to fix the code so that all tests are passing. However, this is wrong. If a fix is not done in a matter of minutes, the best thing to do is to revert the changes. After all, everything worked not long ago. Implementation that broke something is obviously wrong, so why not go back to where we started and think again about the correct way to implement the test? That way, we wasted minutes on a wrong implementation instead of wasting much more time to correct something that was not done right in the first place. Existing test coverage (excluding the implementation of the last test) should be sacred. We change the existing code through intentional refactoring, not as a way to fix recently written code. Do not make the implementation of the last test final, but provide just enough code for this test to pass. Write the code in any way you want, but do it fast. Once everything is green, we have confidence that there is a safety net in the form of tests. From this moment on, we can proceed to refactor the code. This means that we are making the code better and more optimum without introducing new features. While refactoring is in place, all tests should be passing all the time. If, while refactoring, one of the tests failed, refactor broke an existing functionality and, as before, changes should be reverted. Not only that at this stage we are not changing any features, but we are also not introducing any new tests. All we're doing is making the code better while continuously running all tests to make sure that nothing got broken. At the same time, we're proving code correctness and cutting down on future maintenance costs. Once refactoring is finished, the process is repeated. It's an endless loop of a very short cycle. Speed is the key Imagine a game of ping pong (or table tennis). The game is very fast; sometimes it is hard to even follow the ball when professionals play the game. TDD is very similar. TDD veterans tend not to spend more than a minute on either side of the table (test and implementation). Write a short test and run all tests (ping), write the implementation and run all tests (pong), write another test (ping), write implementation of that test (pong), refactor and confirm that all tests are passing (score), and then repeat—ping, pong, ping, pong, ping, pong, score, serve again. Do not try to make the perfect code. Instead, try to keep the ball rolling until you think that the time is right to score (refactor). Time between switching from tests to implementation (and vice versa) should be measured in minutes (if not seconds). It's not about testing T in TDD is often misunderstood. Test-driven development is the way we approach the design. It is the way to force us to think about the implementation and to what the code needs to do before writing it. It is the way to focus on requirements and implementation of just one thing at a time—organize your thoughts and better structure the code. This does not mean that tests resulting from TDD are useless—it is far from that. They are very useful and they allow us to develop with great speed without being afraid that something will be broken. This is especially true when refactoring takes place. Being able to reorganize the code while having the confidence that no functionality is broken is a huge boost to the quality. The main objective of test-driven development is testable code design with tests as a very useful side product. Testing Even though the main objective of test-driven development is the approach to code design, tests are still a very important aspect of TDD and we should have a clear understanding of two major groups of techniques as follows: Black-box testing White-box testing The black-box testing Black-box testing (also known as functional testing) treats software under test as a black-box without knowing its internals. Tests use software interfaces and try to ensure that they work as expected. As long as functionality of interfaces remains unchanged, tests should pass even if internals are changed. Tester is aware of what the program should do, but does not have the knowledge of how it does it. Black-box testing is most commonly used type of testing in traditional organizations that have testers as a separate department, especially when they are not proficient in coding and have difficulties understanding it. This technique provides an external perspective on the software under test. Some of the advantages of black-box testing are as follows: Efficient for large segments of code Code access, understanding the code, and ability to code are not required Separation between user's and developer's perspectives Some of the disadvantages of black-box testing are as follows: Limited coverage, since only a fraction of test scenarios is performed Inefficient testing due to tester's lack of knowledge about software internals Blind coverage, since tester has limited knowledge about the application If tests are driving the development, they are often done in the form of acceptance criteria that is later used as a definition of what should be developed. Automated black-box testing relies on some form of automation such as behavior-driven development (BDD). The white-box testing White-box testing (also known as clear-box testing, glass-box testing, transparent-box testing, and structural testing) looks inside the software that is being tested and uses that knowledge as part of the testing process. If, for example, an exception should be thrown under certain conditions, a test might want to reproduce those conditions. White-box testing requires internal knowledge of the system and programming skills. It provides an internal perspective on the software under test. Some of the advantages of white-box testing are as follows: Efficient in finding errors and problems Required knowledge of internals of the software under test is beneficial for thorough testing Allows finding hidden errors Programmers introspection Helps optimizing the code Due to the required internal knowledge of the software, maximum coverage is obtained Some of the disadvantages of white-box testing are as follows: It might not find unimplemented or missing features Requires high-level knowledge of internals of the software under test Requires code access Tests are often tightly coupled to the implementation details of the production code, causing unwanted test failures when the code is refactored. White-box testing is almost always automated and, in most cases, has the form of unit tests. When white-box testing is done before the implementation, it takes the form of TDD. The difference between quality checking and quality assurance The approach to testing can also be distinguished by looking at the objectives they are trying to accomplish. Those objectives are often split between quality checking (QC) and quality assurance (QA). While quality checking is focused on defects identification, quality assurance tries to prevent them. QC is product-oriented and intends to make sure that results are as expected. On the other hand, QA is more focused on processes that assure that quality is built-in. It tries to make sure that correct things are done in the correct way. While quality checking had a more important role in the past, with the emergence of TDD, acceptance test-driven development (ATDD), and later on behavior-driven development (BDD), focus has been shifting towards quality assurance. Better tests No matter whether one is using black-box, white-box, or both types of testing, the order in which they are written is very important. Requirements (specifications and user stories) are written before the code that implements them. They come first so they define the code, not the other way around. The same can be said for tests. If they are written after the code is done, in a certain way, that code (and the functionalities it implements) is defining tests. Tests that are defined by an already existing application are biased. They have a tendency to confirm what code does, and not to test whether client's expectations are met, or that the code is behaving as expected. With manual testing, that is less the case since it is often done by a siloed QC department (even though it's often called QA). They tend to work on tests' definition in isolation from developers. That in itself leads to bigger problems caused by inevitably poor communication and the police syndrome where testers are not trying to help the team to write applications with quality built-in, but to find faults at the end of the process. The sooner we find problems, the cheaper it is to fix them. Tests written in the TDD fashion (including its flavors such as ATDD and BDD) are an attempt to develop applications with quality built-in from the very start. It's an attempt to avoid having problems in the first place. Mocking In order for tests to run fast and provide constant feedback, code needs to be organized in such a way that the methods, functions, and classes can be easily replaced with mocks and stubs. A common word for this type of replacements of the actual code is test double. Speed of the execution can be severely affected with external dependencies; for example, our code might need to communicate with the database. By mocking external dependencies, we are able to increase that speed drastically. Whole unit tests suite execution should be measured in minutes, if not seconds. Designing the code in a way that it can be easily mocked and stubbed, forces us to better structure that code by applying separation of concerns. More important than speed is the benefit of removal of external factors. Setting up databases, web servers, external APIs, and other dependencies that our code might need, is both time consuming and unreliable. In many cases, those dependencies might not even be available. For example, we might need to create a code that communicates with a database and have someone else create a schema. Without mocks, we would need to wait until that schema is set. With or without mocks, the code should be written in a way that we can easily replace one dependency with another. Executable documentation Another very useful aspect of TDD (and well-structured tests in general) is documentation. In most cases, it is much easier to find out what the code does by looking at tests than the implementation itself. What is the purpose of some methods? Look at the tests associated with it. What is the desired functionality of some part of the application UI? Look at the tests associated with it. Documentation written in the form of tests is one of the pillars of TDD and deserves further explanation. The main problem with (traditional) software documentation is that it is not up to date most of the time. As soon as some part of the code changes, the documentation stops reflecting the actual situation. This statement applies to almost any type of documentation, with requirements and test cases being the most affected. The necessity to document code is often a sign that the code itself is not well written.Moreover, no matter how hard we try, documentation inevitably gets outdated. Developers shouldn't rely on system documentation because it is almost never up to date. Besides, no documentation can provide as detailed and up-to-date description of the code as the code itself. Using code as documentation, does not exclude other types of documents. The key is to avoid duplication. If details of the system can be obtained by reading the code, other types of documentation can provide quick guidelines and a high-level overview. Non-code documentation should answer questions such as what the general purpose of the system is and what technologies are used by the system. In many cases, a simple README is enough to provide the quick start that developers need. Sections such as project description, environment setup, installation, and build and packaging instructions are very helpful for newcomers. From there on, code is the bible. Implementation code provides all needed details while test code acts as the description of the intent behind the production code. Tests are executable documentation with TDD being the most common way to create and maintain it. Assuming that some form of Continuous Integration (CI) is in use, if some part of test-documentation is incorrect, it will fail and be fixed soon afterwards. CI solves the problem of incorrect test-documentation, but it does not ensure that all functionality is documented. For this reason (among many others), test-documentation should be created in the TDD fashion. If all functionality is defined as tests before the implementation code is written and execution of all tests is successful, then tests act as a complete and up-to-date information that can be used by developers. What should we do with the rest of the team? Testers, customers, managers, and other non coders might not be able to obtain the necessary information from the production and test code. As we saw earlier, two most common types of testing are black-box and white-box testing. This division is important since it also divides testers into those who do know how to write or at least read code (white-box testing) and those who don't (black-box testing). In some cases, testers can do both types. However, more often than not, they do not know how to code so the documentation that is usable for developers is not usable for them. If documentation needs to be decoupled from the code, unit tests are not a good match. That is one of the reasons why BDD came in to being. BDD can provide documentation necessary for non-coders, while still maintaining the advantages of TDD and automation. Customers need to be able to define new functionality of the system, as well as to be able to get information about all the important aspects of the current system. That documentation should not be too technical (code is not an option), but it still must be always up to date. BDD narratives and scenarios are one of the best ways to provide this type of documentation. Ability to act as acceptance criteria (written before the code), be executed frequently (preferably on every commit), and be written in natural language makes BDD stories not only always up to date, but usable by those who do not want to inspect the code. Documentation is an integral part of the software. As with any other part of the code, it needs to be tested often so that we're sure that it is accurate and up to date. The only cost-effective way to have accurate and up-to-date information is to have executable documentation that can be integrated into your continuous integration system. TDD as a methodology is a good way to move towards this direction. On a low level, unit tests are a best fit. On the other hand, BDD provides a good way to work on a functional level while maintaining understanding accomplished using natural language. No debugging We (authors of this article) almost never debug applications we're working on! This statement might sound pompous, but it's true. We almost never debug because there is rarely a reason to debug an application. When tests are written before the code and the code coverage is high, we can have high confidence that the application works as expected. This does not mean that applications written using TDD do not have bugs—they do. All applications do. However, when that happens, it is easy to isolate them by simply looking for the code that is not covered with tests. Tests themselves might not include some cases. In that situation, the action is to write additional tests. With high code coverage, finding the cause of some bug is much faster through tests than spending time debugging line by line until the culprit is found. With all this in mind, let's go through the TDD best practices. Best practices Coding best practices are a set of informal rules that the software development community has learned over time, which can help improve the quality of software. While each application needs a level of creativity and originality (after all, we're trying to build something new or better), coding practices help us avoid some of the problems others faced before us. If you're just starting with TDD, it is a good idea to apply some (if not all) of the best practices generated by others. For easier classification of test-driven development best practices, we divided them into four categories: Naming conventions Processes Development practices Tools As you'll see, not all of them are exclusive to TDD. Since a big part of test-driven development consists of writing tests, many of the best practices presented in the following sections apply to testing in general, while others are related to general coding best practices. No matter the origin, all of them are useful when practicing TDD. Take the advice with a certain dose of skepticism. Being a great programmer is not only about knowing how to code, but also about being able to decide which practice, framework or style best suits the project and the team. Being agile is not about following someone else's rules, but about knowing how to adapt to circumstances and choose the best tools and practices that suit the team and the project. Naming conventions Naming conventions help to organize tests better, so that it is easier for developers to find what they're looking for. Another benefit is that many tools expect that those conventions are followed. There are many naming conventions in use, and those presented here are just a drop in the ocean. The logic is that any naming convention is better than none. Most important is that everyone on the team knows what conventions are being used and are comfortable with them. Choosing more popular conventions has the advantage that newcomers to the team can get up to speed fast since they can leverage existing knowledge to find their way around. Separate the implementation from the test code Benefits: It avoids accidentally packaging tests together with production binaries; many build tools expect tests to be in a certain source directory. Common practice is to have at least two source directories. Implementation code should be located in src/main/java and test code in src/test/java. In bigger projects, the number of source directories can increase but the separation between implementation and tests should remain as is. Build tools such as Gradle and Maven expect source directories separation as well as naming conventions. You might have noticed that the build.gradle files that we used throughout this article did not have explicitly specified what to test nor what classes to use to create a .jar file. Gradle assumes that tests are in src/test/java and that the implementation code that should be packaged into a jar file is in src/main/java. Place test classes in the same package as implementation Benefits: Knowing that tests are in the same package as the code helps finding code faster. As stated in the previous practice, even though packages are the same, classes are in the separate source directories. All exercises throughout this article followed this convention. Name test classes in a similar fashion to the classes they test Benefits: Knowing that tests have a similar name to the classes they are testing helps in finding the classes faster. One commonly used practice is to name tests the same as the implementation classes, with the suffix Test. If, for example, the implementation class is TickTackToe, the test class should be TickTackToeTest. However, in all cases, with the exception of those we used throughout the refactoring exercises, we prefer the suffix Spec. It helps to make a clear distinction that test methods are primarily created as a way to specify what will be developed. Testing is a great subproduct of those specifications. Use descriptive names for test methods Benefits: It helps in understanding the objective of tests. Using method names that describe tests is beneficial when trying to figure out why some tests failed or when the coverage should be increased with more tests. It should be clear what conditions are set before the test, what actions are performed and what is the expected outcome. There are many different ways to name test methods and our preferred method is to name them using the Given/When/Then syntax used in the BDD scenarios. Given describes (pre)conditions, When describes actions, and Then describes the expected outcome. If some test does not have preconditions (usually set using @Before and @BeforeClass annotations), Given can be skipped. Let's take a look at one of the specifications we created for our TickTackToe application:   @Test public void whenPlayAndWholeHorizontalLineThenWinner() { ticTacToe.play(1, 1); // X ticTacToe.play(1, 2); // O ticTacToe.play(2, 1); // X ticTacToe.play(2, 2); // O String actual = ticTacToe.play(3, 1); // X assertEquals("X is the winner", actual); } Just by reading the name of the method, we can understand what it is about. When we play and the whole horizontal line is populated, then we have a winner. Do not rely only on comments to provide information about the test objective. Comments do not appear when tests are executed from your favorite IDE nor do they appear in reports generated by CI or build tools. Processes TDD processes are the core set of practices. Successful implementation of TDD depends on practices described in this section. Write a test before writing the implementation code Benefits: It ensures that testable code is written; ensures that every line of code gets tests written for it. By writing or modifying the test first, the developer is focused on requirements before starting to work on the implementation code. This is the main difference compared to writing tests after the implementation is done. The additional benefit is that with the tests written first, we are avoiding the danger that the tests work as quality checking instead of quality assurance. We're trying to ensure that quality is built in as opposed to checking later whether we met quality objectives. Only write new code when the test is failing Benefits: It confirms that the test does not work without the implementation. If tests are passing without the need to write or modify the implementation code, then either the functionality is already implemented or the test is defective. If new functionality is indeed missing, then the test always passes and is therefore useless. Tests should fail for the expected reason. Even though there are no guarantees that the test is verifying the right thing, with fail first and for the expected reason, confidence that verification is correct should be high. Rerun all tests every time the implementation code changes Benefits: It ensures that there is no unexpected side effect caused by code changes. Every time any part of the implementation code changes, all tests should be run. Ideally, tests are fast to execute and can be run by the developer locally. Once code is submitted to version control, all tests should be run again to ensure that there was no problem due to code merges. This is specially important when more than one developer is working on the code. Continuous integration tools such as Jenkins (http://jenkins-ci.org/), Hudson (http://hudson-ci.org/), Travis (https://travis-ci.org/), and Bamboo (https://www.atlassian.com/software/bamboo) should be used to pull the code from the repository, compile it, and run tests. All tests should pass before a new test is written Benefits: The focus is maintained on a small unit of work; implementation code is (almost) always in working condition. It is sometimes tempting to write multiple tests before the actual implementation. In other cases, developers ignore problems detected by existing tests and move towards new features. This should be avoided whenever possible. In most cases, breaking this rule will only introduce technical debt that will need to be paid with interest. One of the goals of TDD is that the implementation code is (almost) always working as expected. Some projects, due to pressures to reach the delivery date or maintain the budget, break this rule and dedicate time to new features, leaving the task of fixing the code associated with failed tests for later. These projects usually end up postponing the inevitable. Refactor only after all tests are passing Benefits: This type of refactoring is safe. If all implementation code that can be affected has tests and they are all passing, it is relatively safe to refactor. In most cases, there is no need for new tests. Small modifications to existing tests should be enough. The expected outcome of refactoring is to have all tests passing both before and after the code is modified. Development practices Practices listed in this section are focused on the best way to write tests. Write the simplest code to pass the test Benefits: It ensures cleaner and clearer design; avoids unnecessary features. The idea is that the simpler the implementation, the better and easier it is to maintain the product. The idea adheres to the keep it simple stupid (KISS) principle. This states that most systems work best if they are kept simple rather than made complex; therefore, simplicity should be a key goal in design, and unnecessary complexity should be avoided. Write assertions first, act later Benefits: This clarifies the purpose of the requirements and tests early. Once the assertion is written, the purpose of the test is clear and the developer can concentrate on the code that will accomplish that assertion and, later on, on the actual implementation. Minimize assertions in each test Benefits: This avoids assertion roulette; allows execution of more asserts. If multiple assertions are used within one test method, it might be hard to tell which of them caused a test failure. This is especially common when tests are executed as part of the continuous integration process. If the problem cannot be reproduced on a developer's machine (as may be the case if the problem is caused by environmental issues), fixing the problem may be difficult and time consuming. When one assert fails, execution of that test method stops. If there are other asserts in that method, they will not be run and information that can be used in debugging is lost. Last but not least, having multiple asserts creates confusion about the objective of the test. This practice does not mean that there should always be only one assert per test method. If there are other asserts that test the same logical condition or unit of functionality, they can be used within the same method. Let's go through few examples: @Test public final void whenOneNumberIsUsedThenReturnValueIsThatSameNumber() { Assert.assertEquals(3, StringCalculator.add("3")); } @Test public final void whenTwoNumbersAreUsedThenReturnValueIsTheirSum() { Assert.assertEquals(3+6, StringCalculator.add("3,6")); } The preceding code contains two specifications that clearly define what the objective of those tests is. By reading the method names and looking at the assert, there should be clarity on what is being tested. Consider the following for example: @Test public final void whenNegativeNumbersAreUsedThenRuntimeExceptionIsThrown() { RuntimeException exception = null; try { StringCalculator.add("3,-6,15,-18,46,33"); } catch (RuntimeException e) { exception = e; } Assert.assertNotNull("Exception was not thrown", exception); Assert.assertEquals("Negatives not allowed: [-6, -18]", exception.getMessage()); } This specification has more than one assert, but they are testing the same logical unit of functionality. The first assert is confirming that the exception exists, and the second that its message is correct. When multiple asserts are used in one test method, they should all contain messages that explain the failure. This way debugging the failed assert is easier. In the case of one assert per test method, messages are welcome, but not necessary since it should be clear from the method name what the objective of the test is. @Test public final void whenAddIsUsedThenItWorks() { Assert.assertEquals(0, StringCalculator.add("")); Assert.assertEquals(3, StringCalculator.add("3")); Assert.assertEquals(3+6, StringCalculator.add("3,6")); Assert.assertEquals(3+6+15+18+46+33, StringCalculator.add("3,6,15,18,46,33")); Assert.assertEquals(3+6+15, StringCalculator.add("3,6n15")); Assert.assertEquals(3+6+15, StringCalculator.add("//;n3;6;15")); Assert.assertEquals(3+1000+6, StringCalculator.add("3,1000,1001,6,1234")); } This test has many asserts. It is unclear what the functionality is, and if one of them fails, it is unknown whether the rest would work or not. It might be hard to understand the failure when this test is executed through some of the CI tools. Do not introduce dependencies between tests Benefits: The tests work in any order independently, whether all or only a subset is run Each test should be independent from the others. Developers should be able to execute any individual test, a set of tests, or all of them. Often, due to the test runner's design, there is no guarantee that tests will be executed in any particular order. If there are dependencies between tests, they might easily be broken with the introduction of new ones. Tests should run fast Benefits: These tests are used often. If it takes a lot of time to run tests, developers will stop using them or run only a small subset related to the changes they are making. The benefit of fast tests, besides fostering their usage, is quick feedback. The sooner the problem is detected, the easier it is to fix it. Knowledge about the code that produced the problem is still fresh. If the developer already started working on the next feature while waiting for the completion of the execution of the tests, he might decide to postpone fixing the problem until that new feature is developed. On the other hand, if he drops his current work to fix the bug, time is lost in context switching. Tests should be so quick that developers can run all of them after each change without getting bored or frustrated. Use test doubles Benefits: This reduces code dependency and test execution will be faster. Mocks are prerequisites for fast execution of tests and ability to concentrate on a single unit of functionality. By mocking dependencies external to the method that is being tested, the developer is able to focus on the task at hand without spending time in setting them up. In the case of bigger teams, those dependencies might not even be developed. Also, the execution of tests without mocks tends to be slow. Good candidates for mocks are databases, other products, services, and so on. Use set-up and tear-down methods Benefits: This allows set-up and tear-down code to be executed before and after the class or each method. In many cases, some code needs to be executed before the test class or before each method in a class. For this purpose, JUnit has @BeforeClass and @Before annotations that should be used as the setup phase. @BeforeClass executes the associated method before the class is loaded (before the first test method is run). @Before executes the associated method before each test is run. Both should be used when there are certain preconditions required by tests. The most common example is setting up test data in the (hopefully in-memory) database. At the opposite end are @After and @AfterClass annotations, which should be used as the tear-down phase. Their main purpose is to destroy data or a state created during the setup phase or by the tests themselves. As stated in one of the previous practices, each test should be independent from the others. Moreover, no test should be affected by the others. Tear-down phase helps to maintain the system as if no test was previously executed. Do not use base classes in tests Benefits: It provides test clarity. Developers often approach test code in the same way as implementation. One of the common mistakes is to create base classes that are extended by tests. This practice avoids code duplication at the expense of tests clarity. When possible, base classes used for testing should be avoided or limited. Having to navigate from the test class to its parent, parent of the parent, and so on in order to understand the logic behind tests introduces often unnecessary confusion. Clarity in tests should be more important than avoiding code duplication. Tools TDD, coding and testing in general, are heavily dependent on other tools and processes. Some of the most important ones are as follows. Each of them is too big a topic to be explored in this article, so they will be described only briefly. Code coverage and Continuous integration (CI) Benefits: It gives assurance that everything is tested Code coverage practice and tools are very valuable in determining that all code, branches, and complexity is tested. Some of the tools are JaCoCo (http://www.eclemma.org/jacoco/), Clover (https://www.atlassian.com/software/clover/overview), and Cobertura (http://cobertura.github.io/cobertura/). Continuous Integration (CI) tools are a must for all except the most trivial projects. Some of the most used tools are Jenkins (http://jenkins-ci.org/), Hudson (http://hudson-ci.org/), Travis (https://travis-ci.org/), and Bamboo (https://www.atlassian.com/software/bamboo). Use TDD together with BDD Benefits: Both developer unit tests and functional customer facing tests are covered. While TDD with unit tests is a great practice, in many cases, it does not provide all the testing that projects need. TDD is fast to develop, helps the design process, and gives confidence through fast feedback. On the other hand, BDD is more suitable for integration and functional testing, provides better process for requirement gathering through narratives, and is a better way of communicating with clients through scenarios. Both should be used, and together they provide a full process that involves all stakeholders and team members. TDD (based on unit tests) and BDD should be driving the development process. Our recommendation is to use TDD for high code coverage and fast feedback, and BDD as automated acceptance tests. While TDD is mostly oriented towards white-box, BDD often aims at black-box testing. Both TDD and BDD are trying to focus on quality assurance instead of quality checking. Summary You learned that it is a way to design the code through short and repeatable cycle called red-green-refactor. Failure is an expected state that should not only be embraced, but enforced throughout the TDD process. This cycle is so short that we move from one phase to another with great speed. While code design is the main objective, tests created throughout the TDD process are a valuable asset that should be utilized and severely impact on our view of traditional testing practices. We went through the most common of those practices such as white-box and black-box testing, tried to put them into the TDD perspective, and showed benefits that they can bring to each other. You discovered that mocks are a very important tool that is often a must when writing tests. Finally, we discussed how tests can and should be utilized as executable documentation and how TDD can make debugging much less necessary. Now that we are armed with theoretical knowledge, it is time to set up the development environment and get an overview and comparison of different testing frameworks and tools. Now we will walk you through all the TDD best practices in detail and refresh the knowledge and experience you gained throughout this article. Resources for Article: Further resources on this subject: RESTful Services JAX-RS 2.0[article] Java Refactoring in NetBeans[article] Developing a JavaFX Application for iOS [article]
Read more
  • 0
  • 0
  • 2942
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-authentication-and-authorization-modx
Packt
20 Oct 2009
1 min read
Save for later

Authentication and Authorization in MODx

Packt
20 Oct 2009
1 min read
It is vital to keep this distinction in mind to be able to understand the complexities explained in this article. You will also learn how MODx allows grouping of documents, users, and permissions. Create web users Let us start by creating a web user. Web users are users who can access restricted document groups in the web site frontend; they do not have Manager access. Web users can identify themselves at login by using login forms. They are allowed to log in from the user page, but they cannot log in using the Manager interface. To create a web user, perform the following steps: Click on the Web Users menu item in the Security menu. Click on New Web User. Fill in the fields with the following information: Field Name Value Username samira Password samira123 Email Address xyz@configurelater.com    
Read more
  • 0
  • 0
  • 2941

article-image-migrating-ms-sql-server-2008-enterprisedb
Packt
09 Oct 2009
2 min read
Save for later

Migrating from MS SQL Server 2008 to EnterpriseDB

Packt
09 Oct 2009
2 min read
With many database vendor products in the market and data intensive applications using them, it is often required to port the application to use the data or, migrate the data so that the application can use it. Migration of data is therefore one of the realities of the IT Industry. Some of the author's previous articles on migration can be found at this link. You may find more if you do a search on his blog site. Table to be migrated in SQL Server 2008 The following figure shows the Categories table in the Microsoft SQL Server 2008's Management Studio that will be migrated to the Postgres database. Creating a database in Postgres Studio Right click Databases node in the Advanced Server 8.3 and click on New Database... menu as shown. The New Database... window gets displayed as shown. Create an empty database PGNorthwind in Postgres Studio by entering information shown in the next figure. This creates a new database and related objects as shown in the next figure. This also creates the script in the Properties pane as shown. Review the properties. The database may be dropped using the Drop Database statement. Starting the Migration Studio Click on Start | All Programs | Postgres Advanced Server 8.3 to display the drop-down menu as shown. Click on the Migration Studio drop-down item. This opens the EnterpriseDB Migration Studio 8.3(Migration Studio for the rest of the tutorial) with a modal form with the title Edit Server Advanced Server 8.3(localhost:5432). This is the server we installed in the previous tutorial. Enter the User Name and Password and click OK. You will get a message displaying the result as shown in the next figure. Click OK to both the open windows and the EnterpriseDB Migration Studio shows up as shown here. Click on the File in the main menu on the Migration Studio and pick Add Server. This brings up the Add Server window with a default as shown.
Read more
  • 0
  • 0
  • 2939

article-image-linux4afrika-interview-founder
Packt
22 Oct 2009
5 min read
Save for later

Linux4afrika: An Interview with the Founder

Packt
22 Oct 2009
5 min read
  "Linux4afrika has the objective of bridging the digital divide between developed and disadvantaged countries, especially in Africa, by supporting  access to information technology. This is done through the collection of used computers in Germany, the terminal server project and Ubuntu software which is open source, and by providing support to the involved schools and institutions." In this interview with the founder Hans-Peter Merkel, Packt's Kushal Sharma explores the idea, support, and the future of this movement. Kushal Sharma: Who are the chief promoters of this movement? Hans-Peter Merkel: FreiOSS (established in 2004) with currently about 300 members started the Linux4afrika project in 2006. The input was provided by some African trainees doing their internship at St. Ursula High school in Freiburg where we currently have 2 Terminal servers running. The asked FreiOSS to run similar projects in Africa. KS: What initiated this vision to bridge the IT gap between the developed and the under developed nations? During 2002 to 2005 we conducted IT trainings on Open Source products during 3 InWEnt trainings “Information Technology in African business” (see http://www.it-ab.org) with 60 African trainees (20 each year). This made FreiOSS to move OSS out of the local area and include other countries, especially those countries we had participants from. KS: Can you briefly recount the history of this movement, from the time it started to its popularity today? HPM: As mentioned before, the Linux4afrika project has its roots with FreiOSS and St. Ursula High school. There itself the idea was born. I conduct Open Source trainings and security trainings in several African countries (see http://www.hpmerkel.com/events). During a training in Dar es Salaam I demonstrated the Terminal Server solution to participants in a security training. One of the participants informed a Minister of Tanzanian Parliament who immediately came to get more information on this idea. He asked whether Linux4afrika could collect about 100 Computers and ship them to Tanzania. Tanzania would cover the shipping costs. After retuning to Germany I informed FreiOSS regarding this, and the collection activity started. We found out more information about the container costs and found that a container would fit about 200 computers for the same price. Therefore we decided to change the number from 100 to 200. One Terminalserver (AMD 64 Dual Core with 2 GB Memory) can run about 20 Thin Clients. This would serve about 10 schools in Tanzania. The Ubuntu Community Germany heard about our project and invited us to Linuxtag in Berlin (2007). This was a breakthrough for us; many organizations donated hardware. 3SAT TV also added greatly to our popularity by sending a 5 minute broadcast about our project (see http://www.linux4afrika.de). In June we met Markus Feilner from German Linux Magazin who contacted us and also published serveral online reports. In September Linux4afrika was invited to the German Parliament to join a meeting about education strategies for the under developed countries. In October Linux4afrika will start collection for a second container which will be shipped end of the year. In early 2008 about 5 members of FreiOSS will fly to Dar es Salaam on their own costs to conduct a one week training where teachers will be trained. This will be an addon to the service from Agumba Computers Ltd. (see http://www.agumba.biz). Agumba offers support in Tanzania to keep the network running. During the InWEnt trainings from 2002-2005, three employees from Agumba were in that training. Currently, 2 other people from Agumba are here for a three month internship to become familiar with our solution and make the project sustainable. KS: Who are the major contributors? HPM: Currently FreiOSS in Germany and Agumba Computers in Tanzania are the major contributors. KS: Do you get any internal support in Tanzania and Mozambique? Do their Governments support open source? HPM: Yes, we do. In Tanzania, it's Augumba Computers and in Mozambique we have some support from CENFOSS. All trainings conducted by me on Security and Forensics had a 70 percent part on Open Source in Tanzania. Currently, the Governmental agencies are implementing those technologies mainly on servers. KS: Do you have any individuals working full-time for this project? If so, how do the full-time individuals support themselves financially? HPM: All supporters are helping us without any financial support. They all come after work to our meetings which take place about once a month. After some starting problems the group is now able to configure and test about 50 Thin clients per evening meetings. KS: Tell us something more about the training program: what topics do you cover, how many participants do you have so far, etc.? HPM: Tanzania shows a big interest in Security trainings. Agumba computers offers those trainings for about 4-6 weeks a year. Participants come from Tanzania Revenue Authority, Police, Presidents office, banks, water/electricity companies and others. Currently Tanzania Revenue Authority has sent 5 participants to conduct a 3 month Forensic training in Germany. In Tanzania about 120 participants joined the trainings so far. Sessions for next year will start in January 2007. KS: Packt supported the project by sending some copies of our OpenVPN book. How will these be used and what do you hope to gain from them? HPM: Markus Feilner (the author of the OpenVPN book) is currently in Tanzania. He will conduct a one and a half day training on OpenVPN in Dar es Salaam. The participants in Germany who received the books will receive practical training on IPCop and OpenVPN for Microsoft and Linux clients. This will help them establish secure Wireless in their country. KS: What does the future hold for Linux4afrika? Our current plans include the second container, the visit to Dar early 2008, and Linuxtag 2008. Further actions will be discussed therafter. We already have a few requests to expand the Terminalserver Solution to other under developed countries. Also, currently we have a request to support Martinique after the hurricane has destroyed huge parts of the island. KS:Thanks very much Hans-Peter for taking out time for us, and all the very best for your plans.
Read more
  • 0
  • 0
  • 2939

article-image-indexing-data-solr-14-enterprise-search-server-part2
Packt
05 Oct 2009
9 min read
Save for later

Indexing Data in Solr 1.4 Enterprise Search Server: Part2

Packt
05 Oct 2009
9 min read
(For more resources on Solr, see here.) Direct database and XML import The capability for Solr to get data directly from a database or HTTP GET accessible XML is distributed with Solr as a contrib module, and it is known as the DataImportHandler (DIH in short). The complete reference documentation for this capability is here at http://wiki.apache.org/solr/DataImportHandler, and it's rather thorough. In this article, we'll only walk through an example to see how it can be used with the MusicBrainz data set. In short, the DIH offers the following capabilities: Imports data from databases through JDBC (Java Database Connectivity) Imports XML data from a URL (HTTP GET) or a file Can combine data from different tables or sources in various ways Extraction/Transformation of the data Import of updated (delta) data from a database, assuming a last-updated date A diagnostic/development web page Extensible to support alternative data sources and transformation steps As the MusicBrainz data is in a database, the most direct method to get data into Solr is definitely through the DIH using JDBC. Getting started with DIH DIH is not a direct part of Solr. Hence it might not be included in your Solr setup. It amounts to a JAR file named something like apache-solr-dataimporthandler-1.4.jar, which is probably already embedded within the solr.war file. You can use jar -tf solr.war to see. Alternatively, it may be placed in <solr-home>/lib, which is alongside the conf directory we've been working with. For database connectivity, we need to ensure that the JDBC driver is on the Java classpath. Placing it in <solr-home>/lib is a convenient way to do this. The DIH needs to be registered with Solr in solrconfig.xml. Here is how it is done: <requestHandler name="/dataimport"class="org.apache.solr.handler.dataimport.DataImportHandler"><lst name="defaults"><str name="config">mb-dih-artists-jdbc.xml</str></lst></requestHandler> mb-dih-artists-jdbc.xml (mb being short for MusicBrainz) is a file in <solr-home>/conf, which is used to configure DIH. It is possible to specify some configuration aspects in this request handler configuration instead of the dedicated configuration file. However, I recommend that it all be in the DIHconfig file, as in our example here. Given below is an mb-dih-artists-jdbc.xml file with a rather long SQL query: <dataConfig> <dataSource name="jdbc" driver="org.postgresql.Driver" url="jdbc:postgresql://localhost/musicbrainz_db" user="musicbrainz" readOnly="true" autoCommit="false" /> <document> <entity name="artist" dataSource="jdbc" pk="id" query=" select a.id as id, a.name as a_name, a.sortname as a_name_sort, a.begindate as a_begin_date, a.enddate as a_end_date, a.type as a_type ,array_to_string( array(select aa.name from artistalias aa where aa.ref = a.id ) , '|') as a_alias ,array_to_string( array(select am.name from v_artist_members am where am.band = a.id order by am.id) , '|') as a_member_name ,array_to_string( array(select am.id from v_artist_members am where am.band = a.id order by am.id) , '|') as a_member_id, (select re.releasedate from release re inner join album r on re.album = r.id where r.artist = a.id order by releasedate desc limit 1) as a_release_date_latest from artist a " transformer="RegexTransformer,DateFormatTransformer, TemplateTransformer"> <field column = "id" template="Artist:${artist.id}" /> <field column = "type" template="Artist" /> <field column = "a_begin_date" dateTimeFormat="yyyy-MM-dd" /> <field column = "a_end_date" dateTimeFormat="yyyy-MM-dd" /> <field column = "a_alias" splitBy="|" /> <field column = "a_member_name" splitBy="|"/> <field column = "a_member_id" splitBy="|" /> </entity> </document></dataConfig> The DIH development console Before describing the configuration details, we're going to take a look at the DIH development console. It is accessed by going to this URL (modifications may be needed for your host, port, core, and so on):http://localhost:8983/solr/admin/dataimport.jsp The development console looks like the following screenshot: The screen is divided into two panes: on the left is the DIH control form, which includes an editable version of the DIH configuration file and on the right is the command output as raw XML. The screen works quite simply. The form essentially results in submitting a URL to the right pane. There's no real server-side logic to this interface beyond the standard DIH command invocations being executed on the right. The last section on DIH in this article goes into more detail on submitting a command to the DIH. DIH DataSources of type JdbcDataSource The DIH configuration file starts with the declaration of one or more data sources using the element <dataSource/>, which refers to either a database, a file, or an HTTP URL, depending on the type attribute. It defaults to a value of JdbcDataSource. Those familiar with JDBC should find the driver and url attributes with accompanying user and password straightforward—consult the documentation for your driver/database for further information. readOnly is a boolean that will set a variety of other JDBC options appropriately when set to true. And batchSize is an alias for the JDBC fetchSize and defaults to 500. There are numerous JDBC oriented attributes that can be set as well. I would not normally recommend learning about a feature by reading source code, but this is an exception. For further information, read org.apache.solr.handler.dataimport.JdbcDataSource.java Efficient JDBC configuration Many database drivers in the default configurations (including those for PostgreSQL and MySQL) fetch all of the query results into the memory instead of on-demand or using a batch/fetch size. This may work well for typical database usage like OLTP (Online Transaction Processing systems), but is completely unworkable for ETL (Extract Transform and Load) usage such as this. Configuring the driver to stream the data requires driver-specific configuration options. You may need to consult relevant documentation for the JDBC driver. For PostgreSQL, set autoCommit to false. For MySQL, set batchSize to -1(The DIH detects the -1 and replaces it with Java's Integer.MIN_VALUE, which triggers the special behavior in MySQL's JDBC driver). For Microsoft SQL Server, set responseBuffering to adaptive. Further information about specific databases is at :http://wiki.apache.org/solr/DataImportHandlerFaq.. DIH documents, entities After the declaration of <dataSource/> element(s) is the <document/> element. In turn, this element contains one or more <entity/> elements. In this sample configuration, we're only getting artists. However, if we wanted to have more than one type in the same index, then another could be added. The dataSource attribute references a correspondingly named element earlier. It is only necessary if there are multiple to choose from, but we've put it here explicitly anyway. The main piece of an entity used with a JDBC data source is the query attribute, which is the SQL query to be evaluated. You'll notice that this query involves some sub-queries, which are made into arrays and then transformed into strings joined by spaces. The particular functions used to do these sorts of things are generally database specific. This is done to shoe-horn multi-valued data into a single row in the results. It may create a more complicated query, but it does mean that the database does all of the heavy lifting so that all of the data Solr needs for an artist is in the row. An alternative with DIH is to declare other entities within the entity. If you aren't using a database or if you wish to mix in another data source (even if it's of a different type), then you will be forced to do that. See the Solr DIH Wiki page for examples: http://wiki.apache.org/solr/DataImportHandler. The DIH also supports a delta query, which is a query that selects time-stamped data with dates after the last queried date. This won't be covered here, but you can find more information at the previous URL. DIH fields and transformers Within the <entity/> are some <field/>elements that declare how the columns in the query map to Solr. The field element must have a column attribute that matches the corresponding named column in the SQL query. The name attribute is the Solr schema field name that the column is going into. If it is not specified (and it never is for our example), then it defaults to the column name. Use the SQL as a keyword as we've done to use the same names as the Solr schema instead of the database schema. This reduces the number of explicit mappings needed in <field/> elements and shortens existing ones. When a column in the result can be placed directly into Solr without further processing, there is no need to specify the field declaration, because it is implied. An attribute of the entity declaration that we didn't mention yet is transformer. This declares a comma-separated list of transformers that manipulate the transfer of data from the JDBC resultset into a Solr field. These transformers evaluate a field, if it has an attribute it uses to do its job. More than one might operate on a given field. Therefore, the order in which the transformers are declared in matters. Here are the attributes we've used: template: It is used by TemplateTransformer and declares text, which might include variable name substitutions using ${name} syntax. To access an existing field, use the entityname.columnname syntax. splitBy: It is used by RegexTransformer and splits a single string value into a multi-value by looking for the specified character. dateTimeFormat: It is used by DateFormatTransformer. This is a Java date/time format pattern http://java.sun.com/j2se/1.5.0/docs/api/java/text/SimpleDateFormat.html). If the type of the field in the schema is a date, then it is necessary to ensure Solr can interpret the format. Alternatively, ensure that the string matches the ISO-8601 format, which looks like this: 1976-10-23T23:59:59.000Z. As in all cases in Solr, when specifying dates you can use its so-called "DateMath" syntax such as appending /DAY to tell Solr to round the date to a day.
Read more
  • 0
  • 0
  • 2936
article-image-updating-software-koha
Packt
16 Nov 2010
4 min read
Save for later

Updating Software in Koha

Packt
16 Nov 2010
4 min read
  Koha 3 Library Management System Install, configure, and maintain your Koha installation with this easy-to-follow guide A self-sufficient guide to installing and configuring Koha Take control of your libraries with Koha library management system Get a clear understanding of software maintenance, customization, and other advanced topics such as LDAP and Internationalization Written in a style that applies to all Linux flavors and Koha versions Orientation to updating software Before we can update the Koha software, let us learn about Koha's software versions and how to choose the version to upgrade to. In this section we also learn about the components of a software update, and how to install each component of the update properly. Understanding Koha's software versions To choose which new version to upgrade to, let us first understand how the Koha software is organized. Branches At any given point Koha has at least two main software branches: Stable: This branch is older and is considered stable or bug free for the most part. Only bug fixes are allowed on this branch. Development: This branch is where new features are developed. This branch is ahead of the stable branch, meaning it has all the features of the stable branch and the new features in development. Heads Both branches—stable and development have heads. A heads is the tip of the branch, pointing to the latest change made in that branch. At the time of writing of this article, there are two heads available in Koha's Git repository. 3.0.x: This is the tip of the stable branch master: This is the tip of the development branch Tags Both branches have multiple tags. Tags point to specific points in a branch's change history. For instance we see these tags related to the stable branch: v3.00.06: This is the latest stable branch v3.00.05: An earlier version of the 3.0.x branch v3.00.04: An earlier version of the 3.0.x branch v3.00.03: An earlier version of the 3.0.x branch And these tags are available for the development branch: v3.02.00-beta: This is the 3.02 branch in the beta testing stage v3.03.00-alpha: This is the 3.02 branch when released for alpha testing Choosing a version to update to We can choose to move to the head of the stable branch or the head of the development branch or to any tag in one of these branches. Here are some pointers to help you decide: On production servers, we upgrade to the latest stable tag in the stable branch To take an early look at new features being developed, switch to the alpha or beta tag in the development branch, if available If you want to take a look at the very latest version of the software, switch to head of the development branch Understanding components of software updates When bugs are fixed or new features are added in Koha, different types of files and programs can change such as these: Perl, Java script, HTML, CSS, and other types of files in kohaclone folder Tables, columns, constraints, indexes, system preferences, and other types of changes in Koha's database Indexes and properties in Zebra configuration files Directives in Koha's Apache2 configuration files An overview of the installation process To ensure that software updates are installed properly, we need to follow these steps: Download software updates: We can download updates using Git. Git automatically detects our current version and downloads updates from Koha's online repository. Switch to a specific software version: Depending on our purposes, we will choose a version that we want to upgrade to. Install Perl module prerequisites: The new version of the software may depend on new Perl modules; we will need to install these. Install the new version of Koha: We will install the new Koha version using the make utility; this process is similar to that of a fresh Koha install. Configure Apache2: The new version of the software may have an updated Apache2 configuration file. We will need to configure this new file. Upgrade the database: We will use Koha's web installer to upgrade the database to the new version. Rebuild Zebra indexes: The new software version may contain updates to Zebra configuration files. To have these changes reflected in search results, we will need to do a full rebuild of Zebra's indexes. Restart Zebra server: To load new Zebra configurations we will have to restart zebrasrv.
Read more
  • 0
  • 0
  • 2936

article-image-business-intelligence-and-data-warehouse-solution-architecture-and-design
Packt
24 Jul 2012
4 min read
Save for later

Business Intelligence and Data Warehouse Solution - Architecture and Design

Packt
24 Jul 2012
4 min read
Choosing your database type There are three broad categories for setting up a database when you install Oracle database using the installation wizard: Transaction Processing: Online transactio processing is typically for applications with multiple users inserting and updating low volumes per transaction General Purpose: A batch application that loads large volumes of data during specific windows Data Warehouse: An application that can have multiple users (transactions) requesting large volumes of data for read-only Getting ready Identify the key stakeholders, business process owners, and system owners within the identified process flows. How to do it... To determine the initial configuration of the database, it is important to get an understanding of the number of users and the expected volumes you will be processing: Schedule meetings with the stakeholders. Ask key questions to understand the nature of the solution: How many anticipated users do you expect for the solution? What time of the month/day do you expect the highest number of users utilizing the information? Does the information need to be updated real time, and why? Schedule meetings with the business process owners. Ask key questions to understand the nature of the business process: Is the business process updated real time by users, or is information entered and then processed within a batch? How many business users are involved with the business process? Schedule meetings with the system owners. Ask key questions to understand the source system: What is the database upon which the system resides? How large is the current database? Where is the system located? How is the system connected to the network? Does the system do a lot of batch processing? Can we connect to the system directly? What is the preferred method to connect to the system to extract information? What is the process to request access to extract information? How it works... There are many other questions that can and should be asked. The aforementioned questions are focused on gathering information to determine how many databases should be created, and the general configuration for each database. Generally, a business intelligence solution is logically structured in the following way: Information comes from multiple sources, and is loaded into a staging area. From the staging area, the information is transformed and loaded into the data warehouse. In the data warehouse, this information can be summarized or enhanced (additional calculations), ready to be consumed from the presentation area. For the information sessions, if you have determined that you will be continuously extracting large volumes of data from multiple sources, which are geographically dispersed, then you will probably require a separate staging database to collect and consolidate the information. It is advantageous to select a separate database, so that as your information grows over a period of time, you can tune the stage environment to suit the requirements. When you perform the installation for Oracle, you can select a General Purpose template as the initial configuration for the stage database. For the data warehouse, if you have a large number of users who access information 24 hours per day, 7 days per week, then it is recommended that the data warehouse again be a separate database. This will allow you to prepare the information within the staging database and load the results into the data warehouse. You can minimize the impact of the data transformation and loading on the users, and only require a smaller window to load information. The Data Warehouse template should be used when creating this database. For the presentation area, if you are using a relational tool to publish information, then this should be located within the same database as the data warehouse. The main reason for this is the ease of administration and the ability to utilize the Oracle technology (summary management, query optimization, and so on), and minimize the amount of data movement. For most projects, if you have a handful of sources with data loads happening during a preset window and reasonable user counts, a single General Purpose database will be the correct place to start. Data Marts would also be classified under this definition. There's more... Oracle has published a white paper called Best Practices for a Data Warehouse on Oracle Database 11g, An Oracle White Paper, November 2010. This has a lot of very valuable information with regards to Balanced Configuration and Disk Layout for your database.
Read more
  • 0
  • 0
  • 2935

article-image-boost-your-search
Packt
19 Mar 2014
11 min read
Save for later

Boost Your search

Packt
19 Mar 2014
11 min read
(For more resources related to this topic, see here.) The dismax query parser Before we understand how to boost our search using the dismax query parser, we will learn what a dismax query parser is and the features that make it more demanding than the Lucene query parser. While using the Lucene query parser, a very vital problem was noticed. It restricts the query to be well formed, with certain syntax rules that have balanced quotes and parenthesis. The Lucene query parser is not sophisticated enough to understand that the end users might be laymen. Thus, these users might type anything for a query as they are unaware of such restrictions and are prone to end up with either an error or unexpected search results. To tackle such situations, the dismax query parser came into play. It has been named after Lucene's DisjunctionMaxQuery, which addresses the previously discussed issue along with incorporating a number of features that enhance search relevancy (that is, boosting or scoring). Now, let us do a comparative study of the features provided by the dismax query parser with those provided by the Lucene query parser. Here we go: Search is relevant to multiple fields that have different boost scores The query syntax is limited to the essentiality Auto-boosting of phrases out of the search query Convenient query boosting parameters, usually used with the function queries You can specify a cut-off count of words to match the query I believe you are aware of the q parameter, how the parser for user queries is set using the defType parameter, and the usage of qf, mm, and q.alt parameters. If not, I recommend that you refer to the Dismax query parser documentation at https://cwiki.apache.org/confluence/display/solr/The+DisMax+Query+Parser. Lucene DisjunctionMaxQuery Lucene DisjunctionMaxQuery provides the capability to search across multiple fields with different boosts. Let us consider the following example wherein the query string is mohan; we may configure dismax in such a way that it acts in a very similar way to DisjunctionMaxQuery. Our Boolean query looks as follows: fieldX:mohan^2.1 OR fieldY:mohan^1.4 OR fieldZ:mohan^0.3 Due to the difference in the scoring of the preceding query, we may infer that the query is not quite equivalent to what the dismax query actually does. As far as the dismax query is concerned, in such scenarios, (in case of Boolean queries) the final score is taken as the sum for each of the clauses, whereas DisjunctionMaxQuery considers the highest score as the final one. To understand this practically, let us calculate and compare the final scores in each of the following two behaviors: Fscore_dismax = 2.1 + 1.4 + 0.3 = 3.8 Fscore_disjunctionMaxQuery = 2.1 (the highest of the three) Based on the preceding calculation, we can infer that the score produced out of the dismax query parser is always greater than that of the DisjunctionMaxQuery query parser; hence, there is better search relevancy provided that we are searching for the same keyword in multiple fields. Now, we will look into another parameter, which is known as tie, that boosts the search relevance even further. The value of the tie parameter ranges from 0 to 1, 0 being the default value. Raising this value above 0 begins to favor the documents that match multiple search keywords over those that were boosted higher. Value of the tie parameter can go up to 1, which means that the score is very close to that of the Boolean query. Practically speaking, a smaller value such as 0.1 is the best as well as an effective choice we may have. Autophrase boosting Let us assume that a user searches for Surendra Mohan. Solr interprets this as two different search keywords, and depending on how the request handler has been configured, either both the terms or just one would be found in the document. There might be a case wherein one of the matching documents Surendra is the name of an organization and they have an employee named Mohan. It is quite obvious that Solr will find this document and it might probably be of interest to the user due to the fact that it contains both the terms the user typed. It is quite likely that the document field containing the keyword Surendra Mohan typed by the user represents a closer match to the document the user is actually looking for. However, in such scenarios, it is quite difficult to predict the relative score, though it contains the relevant documents the user was looking for. To tackle such situations and improve scoring, you might be tempted to quote the user's query automatically; however, this would omit the documents that don't have adjacent words. In such a scenario, dismax can add a phrased form of the user's query onto the entered query as an optional clause. It rewrites the query as follows: Surendra Mohan This query can be rewritten as follows: +(Surendra Mohan) "Surendra Mohan" The rewritten query depicts that the entered query is mandatory by using + and shows that we have added an optional phrase. So, a document that contains the phrase Surendra Mohan not only matches that clause in the rewritten query, but also matches each of the terms individually (that is, Surendra and Mohan). Thus, in totality, we have three clauses that Solr would love to play around with. Assume that there is another document where this phrase doesn't match, but it has both the terms available individually and scattered out in there. In this case, only two of the clauses would match. As par Lucene's scoring algorithm, the coordination factor for the first document (which matched the complete phrase) would be higher, assuming that all the other factors remain the same. Configuring autophrase boosting Let me inform you, autophrase boosting is not enabled by default. In order to avail this feature, you have to use the pf parameter (phrase fields), whose syntax is very much identical to that of the qf parameter. To play around with the pf value, it is recommended that you start with the same value as that of qf and then make the necessary adjustments. There are a few reasons why we should vary the pf value instead of qf. They are as follows: The pf value helps us to use varied boost factors so that the impact caused due to phrase boosting isn't overwhelming. In order to omit fields that are always a single termed, for example, identifier, due to the fact that in such a case there is no point in searching for phrases. To omit some of the fields having numerous text count in order to retain the search performance to a major extent. Substitute a field with the other having the same data, but are analyzed differently. You may use different text analysis techniques to achieve this, for example, Shingle or Common-grams. To learn more about text analysis techniques and their usage, I would recommend you to refer to http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters. Configuring the phrase slop Before we learn how to configure the phrase slop, let us understand what it actually is. Slop stands for term proximity, and is primarily used to factorize the distance between two or more terms to a relevant calculation. As discussed earlier in this section, if the two terms Surendra and Mohan are adjacent to each other in a document, that document will have a better score for the search keyword Surendra Mohan compared to the document that contains the terms Surendra and Mohan spread individually throughout the document. On the other hand, when used in conjunction with the OR operator, the relevancy of documents returned in the search results are likely to be improved. The following example shows the syntax of using slop, which is a phrase (in double quotes) followed by a tilde (~) and a number: "Surendra Mohan"~1 Dismax allows two parameters to be added so that slop can be automatically set; qs for any input phrase queries entered by the user and ps for phrase boosting. In case the slop is not specified, it means there is no slop and its value remains 0. The following is the sample configuration setting for slop : <str name="qs" >1</str> <str name="ps">0</str> Boosting a partial phrase You might come across a situation where you need to boost your search for consecutive word pairs or even triples out of a phrase query. To tackle such a situation, you need to use edismax, and this can be configured by setting pf2 and pf3 for word pairs and triples, respectively. The parameters pf2 and pf3 are de fi ned in a manner identical to that of the pf parameter. For instance, consider the following query: how who now cow This query becomes: +(how who now cow) "how who now cow" "how who" "who now" "now cow" "how who now" "who now cow" This feature is unaffected by the ps parameter due to the fact that it is only applicable to the entire phrase boost and has no impact on partial phrase boosting. Moreover, you may expect better relevancy for longer queries; however, the longer the query, the slower its execution. To handle this situation and make the longer queries execute faster, you need to explore and use text analysis techniques such as Shingle or Common-grams. Boost queries Apart from the other boosting techniques we discussed earlier, boost queries are another technique that impact the score of the document to a major extent. Implementing boost queries involves specifying multiple additional queries using the bq parameter or a set of parameters of the dismax query parser. Just like the autophrase boost, this parameter(s) gets added to the user's query in a very similar fashion. Let us not forget that boosting only impacts the scores of the documents that already matched the user's query in the q parameter. So, to achieve a higher score for a document, we need to make sure the document matches a bq query. To understand boost queries better and learn how to work with them, let us consider a realistic example of a music composition and a commerce product. We will primarily be concerned about the music type and the composer's fields with the field names wm_type and wm_composer, respectively. The wm_type field holds the Orchestral, Chamber, and Vocal values along with others and the wm_composer field holds the values Mohan, Webber, and so on. We don't wish to arrange the search results based on these parameters, due to the fact that we are targeting to implement the natural scoring algorithm so that the user's query can be considered relevant; on the other hand, we want the score to be impacted based on these parameters. For instance, let us assume that the music type chamber is the most relevant one, whereas vocal is the least relevant. Moreover, we assume that the composer Mohan is more relevant than Webber or others. Now, let us see how we can express this using the following boost query: <str name="bq">wm_type:Chamber^2 (*:* -wm_type:Vocal)^2 wm_ composer:Mohan^2</str> Based on the search results for any keyword entered by the user (for instance, Opera Simmy), we can infer that our boost query did its job successfully by breaking a tie score, wherein the music type and composer names are the same with varied attributes. In practical scenarios, to achieve a better and desired relevancy boost, boosting on each of the keywords (in our case, three keywords) can be tweaked by examining the debugQuery output minutely. In the preceding boost query, you must have noticed (*:* -wm_type:Vocal)^2, which actually boosts all the documents except the vocal music type. You might think of using wm_type:Vocal^0.5 instead, but let us understand that it would still add value to the score; hence, it wouldn't be able to serve our purpose. We have used *:* to instruct the parser that we would like to match all the documents. In case you don't want any document to match (that is, to achieve 0 results), simply use -*:* instead. Compared to function queries, boost queries are not much effective, primarily due to the fact that edismax supports multiplied boost, which is obviously demanding compared to addition. You might think of a painful situation wherein you want an equivalent boost for both the Chamber wm_type and Mohan wm_composer types. To tackle such situations, you need to execute the query with debugQuery enabled so as to analyze the scores of each of the terms (which is going to be different). Then, you need to use disproportionate boosts so that when multiplied by their score (resultant scores from debugQuery) ends up with the same value. Summary This article briefly described scoring and function queries. It also gave an idea about the Lucene's DisjunctionMaxQuery. Resources for Article: Further resources on this subject: Getting Started with Apache Solr [Article] Apache Solr: Analyzing your Text Data [Article] Apache Solr: Spellchecker, Statistics, and Grouping Mechanism [Article]
Read more
  • 0
  • 0
  • 2935
article-image-moodle-20-faqs
Packt
14 Oct 2010
8 min read
Save for later

Moodle 2.0 FAQs

Packt
14 Oct 2010
8 min read
Moodle 2.0 First Look Discover what's new in Moodle 2.0, how the new features work, and how it will impact you Get an insight into the new features of Moodle 2.0 Discover the benefits of brand new additions such as Comments and Conditional Activities Master the changes in administration with Moodle 2.0 The first and only book that covers all of the fantastic new features of Moodle 2.0         Read more about this book       (For more resources on Moodle, see here.)   Question: What are the basic requirements for Moodle 2.0 to function? Answer: It's important that either you (if you're doing this yourself) or your Moodle admin or webhost are aware of the requirements for Moodle 2.0. It needs: PHP must be 5.2.8 or later One of the following databases: MySQL 5.0.25 or later (InnoDB storage engine highly recommended) PostgreSQL 8.3 or later Oracle 10.2 or later MS SQL 2005 or later One of the following browsers: Firefox 3 or later Safari 3 or later Google Chrome 4 or later Opera 9 or later MS Internet Explorer 7 or later   Question: How can I upgrade to Moodle 2.0? Answer: If you already have an installation of Moodle, you will find instructions for upgrading in the docs on the main Moodle site here http://docs.moodle.org/en/Upgrading_to_Moodle_2.0. If you are upgrading from an earlier version of Moodle (such as 1.8) then you should upgrade to Moodle 1.9 first before going to 2.0. You must update incrementally; shortcuts – for example. updating from 1.7 directly to 2.0 -- are simply not possible. Read the docs carefully if you are planning on upgrading from very early versions such as 1.5 or 1.6.   Question: What are the potential problems with upgrading? Answer: There are a few challenges that one may come across while upgrading from Moodle 1.9 to 2.0 which are listed below: Themes: The way themes work has changed completely. While this allows for more flexible coding and templating, it does mean that if you had a customized theme it will not transfer over to Moodle 2 without some redesigning beforehand. Third party add-ons and custom code: The same applies to third party add-ons and custom code: it is highly unlikely they will work without significant alterations. Backup and Restore: Making courses from 1.9 or earlier restore into Moodle 2. 0 has proved very problematic and is still not entirely achievable. Although this is a priority for the Moodle developers, there is at the time of writing only a workaround involving restoring your course to a 1.9 site and then upgrading it to 2.0.   Question: How can teachers and students manage their learning? Answer: The two new features of Moodle 2.0 help teacher and students manage their learning: Conditional activities: A way to organize a course so that tasks are only available dependent on certain grades being obtained or criteria being met beforehand. Completion tracking: A way for students to have checkboxes next to their tasks that are either automatically marked as complete or which students themselves can manually mark if they feel they've finished the exercise – or alternatively a way for whole courses to be checked off as finished.   Question: What are the changes in the Themes structure for Moodle 2.0? Answer: The themes structure has been completely rewritten for Moodle 2.0. Themes that worked in 1.9 needed to be updated to work in 2.0. There is a wide variety of attractive new themes available. If you need to update your own theme or would like information on Moodle 2.0 theming, you will find the documentation at http://docs.moodle.org/en/Development:Themes_2.0 helpful. New to Moodle 2.0 are the following: Designer Mode: Turn this on so you're not served cached versions of themes, if you are designing themes or developing code. Allow theme changes in the URL: Enabling this will let users alter their theme via their Moodle URL using the syntax Allow blocks to use the dock: Enabling this will allow users to dock blocks if the theme supports it.   Question: Can we customize the MyMoodle page in Moodle 2.0? Answer: Yes, we can customize the default MyMoodle page. It's worth noting that on the MyMoodle page we can add blocks to the middle as well as the sides. With editing turned on, we're given the option to move a block to a central location.   Question: Can we Comment on the Moodle blog? Answer: Commenting on the Moodle blog is a bit of a workaround really; the Moodle blog doesn't really have a built-in commenting facility like, say WordPress. Rather, Moodle is making use of the new Comments feature which ordinarily appears as a block anywhere you want to add it.   Question: What are the improvements in the Blog option in Moodle 2.0 as compared to the previous version? Answer: There has always been a blogging option in a standard Moodle install. However, some users have found it unsatisfactory because of the following reasons: The blog is attached to the user profile so you can only have one blog There is no way to attach a blog or blog entry to a particular course There is no way for other people to comment on your blog For this reason, alternative blog systems (such as the contributed OU blog module) have become popular as they give users a wider range of options. The standard blog in Moodle 2.0 has changed, and now: A blog entry can optionally be associated with a course It is possible to comment on a blog entry Blog entries from outside of Moodle can be copied in It is now possible to search blog entries   Question: How to enable/disable the docking facility in Moodle 2.0? Answer: The docking facility can be managed in Moodle 2.0 as follows: The "docking" facility may be enabled or disabled for themes in Site Administration | Appearance | Themes | Theme settings. If we click the icon shown in the following screenshot, we also have the option of "docking" this over to the far left as a narrow tab.   Question: Has the HTML editor been replaced by some other editing tool? What is its advantage? Answer: In Moodle 2.0, the HTML editor has been replaced with a version known as Tiny MCE, a very popular Open Source editor you might have encountered in content management systems or blogging software such as WordPress. Along with Internet Explorer and Firefox, it will work with web browsers such as Safari, Chrome, and Opera, unlike Moodle's previous HTML editor. The following screenshot shows the new editor (on the bottom) with the original editor (on the top): There are many more options available to us when adding descriptions of our materials or summaries of our courses. However, one of the most powerful new features is the ability to add and embed media directly from within this new HTML editor.   Question: What have been the improvements related to Moodle Quiz? Answer: The following are the improvements to Moodle Quiz: The set up page has been simplified Creating questions has been simplified It's possible to flag questions for later referral Questions can be accessed with one click in the post-quiz review and correct/ incorrect questions are color-coded in an easy-to access navigation block   Question: What are Cohorts? Answer: Cohort is Moodle 2.0's take on the long wished for site-wide groups. When we click on the link we're taken to the following screen where we click on Add to enter details of the cohort we want to create:   Question: Has there been any modification in the Filters menu as compared to the previous versions On/Off options? Answer: The Manage Filters in Moodle 2.0 equates to the Filters menu in Moodle 1.9. The Manage Filters screen looks like the following screenshot (note—the screenshot only displays the first three filters): Previously, filters were either On or Off. Now we have three choices: Disabled: Nobody, in any course, can enable a filter. On: A filter is enabled by default and teachers can disable if they wish to. Off but available: A filter is off but teachers can enable it in their own courses.   Question: What are the changes in Site Administration? Answer: Perhaps the simplest way to explore this is to look at how this menu has altered since Moodle 1.9: Notifications/Registrations: A small but important change in Moodle 1.9, the Notifications screen contained a button you could click to register your site with http://moodle.org/. The page this took you to now has its own billing in Moodle 2.0, as the Registration link. Community hubs: The main Moodle community hub is known as MOOCH and you register with it here. You can also register your site with other community hubs. If you register with hubs, then teachers can add a Community block in their courses where users can search for a suitable course to enroll in or download. Summary In this article we took a look at the queries regarding what Moodle 2.0 has to offer with the exciting new modules and enhanced features, and the major overhauls in the file uploading and navigation system. Further resources on this subject: Moodle 1.9 Math [Book] Moodle Administration [Book] Moodle 1.9 for Teaching Special Education Children (5-10): Beginner's Guide [Book] Moodle 2.0: What's New in Add a Resource [Article] What's New in Moodle 2.0 [Article]
Read more
  • 0
  • 0
  • 2932

article-image-so-what-openelec
Packt
04 Sep 2013
3 min read
Save for later

So, what is OpenELEC?

Packt
04 Sep 2013
3 min read
(For more resources related to this topic, see here.) Open Embedded Linux Entertainment Center (OpenELEC) is an open source embedded operating system developed specifically for the purpose of running complete and easy-to-use media center solutions on a wide variety of hardware. Built around the great open source media player and organizer, XBMC, OpenELEC is optimized to deliver a smooth, intuitive, and efficient user experience. It is developed with the goal of making the task of setting up and maintaining a Home Theater PC (HTPC) easy and straightforward for all users, regardless of their technical skills. This makes OpenELEC the obvious choice for anyone looking to enhance their home media experience with a fully functional media center. The OpenELEC project is open source, so anyone can download and use the operating system completely free of charge. This is a great enticement to get started with setting up a media center of your own, capable of providing many hours of digital entertainment for yourself and your friends and family to enjoy. Media center features Once installed and configured, OpenELEC incorporates XBMC to provide a wide variety of advanced features in a streamlined and straightforward interface. The following are the media types that can be indexed and/or browsed: Movies TV shows Music Pictures The experience of browsing indexed media is enhanced by automatic inclusion of media information and relevant images available from online databases. This provides easy access to ratings, resumes, artwork, and trailers, because everything is incorporated directly in the interface. Hardware requirements Because OpenELEC is very versatile in hardware compatibility, you will most likely be able to turn "that old PC, which has just been lying around" into a fully functional media center, just by installing OpenELEC on it. This approach is great for a fun hobby project, giving you the opportunity to experiment with OpenELEC at no cost. Alternatively, you can get brand new PCs small enough to fit behind a TV, making them the obvious choice for HTPC use. With a very small footprint and compatibility with small atom- or fusion-based platforms, this is where OpenELEC shows its full potential as a lightweight embedded operating system. Summary So, what is OpenELEC? finds out what OpenELEC actually is, what you can do with it, and why it’s so great. Resources for Article : Further resources on this subject: Creating a file server (Samba) [Article] Webcam and Video Wizardry [Article] Our First Project – A Basic Thermometer [Article]
Read more
  • 0
  • 0
  • 2931
Modal Close icon
Modal Close icon