How-To Tutorials

(For more resources related to this topic, see here.) General tips Before diving into some advanced strategies for improving performance and scalability, let's briefly recap some of the general performance tips already spread across the book: When mapping your entity classes for Hibernate Search, use the optional elements of the @Field annotation to strip the unnecessary bloat from your Lucene indexes: If you are definitely not using index-time boosting , then there is no reason to store the information needed to make this possible. Set the norms element to Norms.NO . By default, the information needed for a projection-based query is not stored unless you set the store element to Store.YES or Store. COMPRESS. If you had projection-based queries that are no longer being used, then remove this element as part of the cleanup. Use conditional indexing and partial indexing to reduce the size of Lucene indexes. Rely on filters to narrow your results at the Lucene level, rather than using a WHERE clause at the database query level. Experiment with projection-based queries wherever possible , to reduce or eliminate the need for database calls. Be aware that with advanced database caching, the benefits might not always justify the added complexity. Test various index manager options , such as trying the near-real-time index manager or the async worker execution mode. Running applications in a cluster Making modern Java applications scale in a production environment usually involves running them in a cluster of server instances. Hibernate Search is perfectly at home in a clustered environment, and offers multiple approaches for configuring a solution. Simple clusters The most straightforward approach requires very little Hibernate Search configuration. Just set up a file server for hosting your Lucene indexes and make it available to every server instance in your cluster (for example, NFS, Samba, and so on): A simple cluster with multiple server nodes using a common Lucene index on a shared drive Each application instance in the cluster uses the default index manager, and the usual filesystem directory provider. In this arrangement, all of the server nodes are true peers. They each read from the same Lucene index, and no matter which node performs an update, that node is responsible for the write. To prevent corruption, Hibernate Search depends on simultaneous writes being blocked, by the locking strategy (that is, either "simple" or "native"). Recall that the "near-real-time" index manager is explicitly incompatible with a clustered environment. The advantage of this approach is two-fold. First and foremost is simplicity. The only steps involved are setting up a filesystem share, and pointing each application instance's directory provider to the same location. Secondly, this approach ensures that Lucene updates are instantly visible to all the nodes in the cluster. However, a serious downside is that this approach can only scale so far. Very small clusters may work fine, but larger numbers of nodes trying to simultaneously access the same shared files will eventually lead to lock contention. Also, the file server on which the Lucene indexes are hosted is a single point of failure. If the file share goes down, then your search functionality breaks catastrophically and instantly across the entire cluster. Master-slave clusters When your scalability needs outgrow the limitations of a simple cluster, Hibernate Search offers more advanced models to consider. The common element among them is the idea of a master node being responsible for all Lucene write operations. Clusters may also include any number of slave nodes. Slave nodes may still initiate Lucene updates, and the application code can't really tell the difference. However, under the covers, slave nodes delegate that work to be actually performed by the master node. Directory providers In a master-slave cluster, there is still an "overall master" Lucene index, which logically stands apart from all of the nodes. This may be filesystem-based, just as it is with a simple cluster. However, it may instead be based on JBoss Infinispan (http://www.jboss.org/infinispan), an open source in-memory NoSQL datastore sponsored by the same company that principally sponsors Hibernate development: In a filesystem-based approach, all nodes keep their own local copies of the Lucene indexes. The master node actually performs updates on the overall master indexes, and all of the nodes periodically read from that overall master to refresh their local copies. In an Infinispan-based approach, the nodes all read from the Infinispan index (although it is still recommended to delegate writes to a master node). Therefore, the nodes do not need to maintain their own local index copies. In reality, because Infinispan is a distributed datastore, portions of the index will reside on each node anyway. However, it is still best to visualize the overall index as a separate entity. Worker backends There are two available mechanisms by which slave nodes delegate write operations to the master node: A JMS message queue provider creates a queue, and slave nodes send messages to this queue with details about Lucene update requests. The master node monitors this queue, retrieves the messages, and actually performs the update operations. You may instead replace JMS with JGroups (http://www.jgroups.org), an open source multicast communication system for Java applications. This has the advantage of being faster and more immediate. Messages are received in real-time, synchronously rather than asynchronously. However, JMS messages are generally persisted to a disk while awaiting retrieval, and therefore can be recovered and processed later, in the event of an application crash. If you are using JGroups and the master node goes offline, then all the update requests sent by slave nodes during that outage period will be lost. To fully recover, you would likely need to reindex your Lucene indexes manually. A master-slave cluster using a directory provider based on filesystem or Infinispan, and worker based on JMS or JGroups. Note that when using Infinispan, nodes do not need their own separate index copies.   Summary In this article, we explored the options for running applications in multi-node server clusters, to spread out and handle user requests in a distributed fashion. We also learned how to use sharding to help make our Lucene indexes faster and more manageable. Resources for Article : Further resources on this subject: Integrating Spring Framework with Hibernate ORM Framework: Part 1 [Article] Developing Applications with JBoss and Hibernate: Part 1 [Article] Hibernate Types [Article]

(For more resources related to this topic, see here.) Parallelizing processing with pmap The easiest way to parallelize data is to take a loop we already have and handle each item in it in a thread. That is essentially what pmap does. If we replace a call to map with pmap, it takes each call to the function argument and executes it in a thread pool. pmap is not completely lazy, but it's not completely strict, either: it stays just ahead of the output consumed. So if the output is never used, it won't be fully realized. For this recipe, we'll calculate the Mandelbrot set. Each point in the output takes enough time that this is a good candidate to parallelize. We can just swap map for pmap and immediately see a speed-up. How to do it... The Mandelbrot set can be found by looking for points that don't settle on a value after passing through the formula that defines the set quickly. We need a function that takes a point and the maximum number of iterations to try and return the iteration that it escapes on. That just means that the value gets above 4. (defn get-escape-point [scaled-x scaled-y max-iterations] (loop [x 0, y 0, iteration 0] (let [x2 (* x x), y2 (* y y)] (if (and (< (+ x2 y2) 4) (< iteration max-iterations)) (recur (+ (- x2 y2) scaled-x) (+ (* 2 x y) scaled-y) (inc iteration)) iteration)))) The scaled points are the pixel points in the output, scaled to relative positions in the Mandelbrot set. Here are the functions that handle the scaling. Along with a particular x-y coordinate in the output, they're given the range of the set and the number of pixels each direction. (defn scale-to ([pixel maximum [lower upper]] (+ (* (/ pixel maximum) (Math/abs (- upper lower))) lower))) (defn scale-point ([pixel-x pixel-y max-x max-y set-range] [(scale-to pixel-x max-x (:x set-range)) (scale-to pixel-y max-y (:y set-range))])) The function output-points returns a sequence of x, y values for each of the pixels in the final output. (defn output-points ([max-x max-y] (let [range-y (range max-y)] (mapcat (fn [x] (map #(vector x %) range-y)) (range max-x))))) For each output pixel, we need to scale it to a location in the range of the Mandelbrot set and then get the escape point for that location. (defn mandelbrot-pixel ([max-x max-y max-iterations set-range] (partial mandelbrot-pixel max-x max-y max-iterations set-range)) ([max-x max-y max-iterations set-range [pixel-x pixel-y]] (let [[x y] (scale-point pixel-x pixel-y max-x max-y set-range)] (get-escape-point x y max-iterations)))) At this point, we can simply map mandelbrot-pixel over the results of outputpoints. We'll also pass in the function to use (map or pmap). (defn mandelbrot ([mapper max-iterations max-x max-y set-range] (doall (mapper (mandelbrot-pixel max-x max-y max-iterations set-range) (output-points max-x max-y))))) Finally, we have to define the range that the Mandelbrot set covers. (def mandelbrot-range {:x [-2.5, 1.0], :y [-1.0, 1.0]}) How do these two compare? A lot depends on the parameters we pass them. user=> (def m (time (mandelbrot map 500 1000 1000 mandelbrot-range))) "Elapsed time: 28981.112 msecs" #'user/m user=> (def m (time (mandelbrot pmap 500 1000 1000 mandelbrot-range))) "Elapsed time: 34205.122 msecs" #'user/m user=> (def m (time (mandelbrot map 1000 10001000 mandelbrot-range))) "Elapsed time: 85308.706 msecs" #'user/m user=> (def m (time (mandelbrot pmap 1000 10001000 mandelbrot-range))) "Elapsed time: 49067.584 msecs" #'user/m Refer to the following chart: If we only iterate at most 500 times for each point, it's slightly faster to use map and work sequentially. However, if we iterate 1,000 times each, pmap is faster. How it works... This shows that parallelization is a balancing act. If each separate work item is small, the overhead of creating the threads, coordinating them, and passing data back and forth takes more time than doing the work itself. However, when each thread has enough to do to make it worth it, we can get nice speed-ups just by using pmap. Behind the scenes, pmap takes each item and uses future to run it in a thread pool. It forces only a couple more items than you have processors, so it keeps your machine busy, without generating more work or data than you need. There's more... For an in-depth, excellent discussion of the nuts and bolts of pmap, along with pointers about things to watch out for, see David Liebke's talk, From Concurrency to Parallelism (http://blip.tv/clojure/david-liebke-from-concurrency-to-parallelism-4663526). See also The Partitioning Monte Carlo Simulations for better pmap performance recipe Parallelizing processing with Incanter One of its nice features is that it uses the Parallel Colt Java library (http://sourceforge.net/projects/parallelcolt/) to actually handle its processing, so when you use a lot of the matrix, statistical, or other functions, they're automatically executed on multiple threads. For this, we'll revisit the Virginia housing-unit census data and we'll fit it to a linear regression. Getting ready We'll need to add Incanter to our list of dependencies in our Leiningen project.clj file: :dependencies [[org.clojure/clojure "1.5.0"] [incanter "1.3.0"]] We'll also need to pull those libraries into our REPL or script: (use '(incanter core datasets io optimize charts stats)) We can use the following filename: (def data-file "data/all_160_in_51.P35.csv") How to do it... For this recipe, we'll extract the data to analyze and perform the linear regression. We'll then graph the data afterwards. First, we'll read in the data and pull the population and housing unit columns into their own matrix. (def data (to-matrix (sel (read-dataset data-file :header true) :cols [:POP100 :HU100]))) From this matrix, we can bind the population and the housing unit data to their own names. (def population (sel data :cols 0)) (def housing-units (sel data :cols 1)) Now that we have those, we can use Incanter to fit the data. (def lm (linear-model housing-units population)) Incanter makes it so easy, it's hard not to look at it. (def plot (scatter-plot population housing-units :legend true)) (add-lines plot population (:fitted lm)) (view plot) Here we can see that the graph of housing units to families makes a very straight line: How it works… Under the covers, Incanter takes the data matrix and partitions it into chunks. It then spreads those over the available CPUs to speed up processing. Of course, we don't have to worry about this. That's part of what makes Incanter so powerful. Partitioning Monte Carlo simulations for better pmap performance In the Parallelizing processing with pmap recipe, we found that while using pmap is easy enough, knowing when to use it is more complicated. Processing each task in the collection has to take enough time to make the costs of threading, coordinating processing, and communicating the data worth it. Otherwise, the program will spend more time concerned with how (parallelization) and not enough time with what (the task). The way to get around this is to make sure that pmap has enough to do at each step that it parallelizes. The easiest way to do that is to partition the input collection into chunks and run pmap on groups of the input. For this recipe, we'll use Monte Carlo methods to approximate pi . We'll compare a serial version against a naïve parallel version against a version that uses parallelization and partitions. Getting ready We'll use Criterium to handle benchmarking, so we'll need to include it as a dependency in our Leiningen project.clj file, shown as follows: :dependencies [[org.clojure/clojure "1.5.0"] [criterium "0.3.0"]] We'll use these dependencies and the java.lang.Math class in our script or REPL. (use 'criterium.core) (import [java.lang Math]) How to do it… To implement this, we'll define some core functions and then implement a Monte Carlo method for estimating pi that uses pmap. We need to define the functions necessary for the simulation. We'll have one that generates a random two-dimensional point that will fall somewhere in the unit square. (defn rand-point [] [(rand) (rand)]) Now, we need a function to return a point's distance from the origin. (defn center-dist [[x y]] (Math/sqrt (+ (* x x) (* y y)))) Next we'll define a function that takes a number of points to process, and creates that many random points. It will return the number of points that fall inside a circle. (defn count-in-circle [n] (->> (repeatedly n rand-point) (map center-dist) (filter #(<= % 1.0)) count)) That simplifies our definition of the base (serial) version. This calls count-incircle to get the proportion of random points in a unit square that fall inside a circle. It multiplies this by 4, which should approximate pi. (defn mc-pi [n] (* 4.0 (/ (count-in-circle n) n))) We'll use a different approach for the simple pmap version. The function that we'll parallelize will take a point and return 1 if it's in the circle, or 0 if not. Then we can add those up to find the number in the circle. (defn in-circle-flag [p] (if (<= (center-dist p) 1.0) 1 0)) (defn mc-pi-pmap [n] (let [in-circle (->> (repeatedly n rand-point) (pmap in-circle-flag) (reduce + 0))] (* 4.0 (/ in-circle n)))) For the version that chunks the input, we'll do something different again. Instead of creating the sequence of random points and partitioning that, we'll have a sequence that tells how large each partition should be and have pmap walk across that, calling count-in-circle. This means that creating the larger sequences are also parallelized. (defn mc-pi-part ([n] (mc-pi-part 512 n)) ([chunk-size n] (let [step (int (Math/floor (float (/ n chunk-size)))) remainder (mod n chunk-size) parts (lazy-seq (cons remainder (repeat step chunk-size))) in-circle (reduce + 0 (pmap count-in-circle parts))] (* 4.0 (/ in-circle n))))) Now, how do these work? We'll bind our parameters to names, and then we'll run one set of benchmarks before we look at a table of all of them. We'll discuss the results in the next section. user=> (def chunk-size 4096) #'user/chunk-size user=> (def input-size 1000000) #'user/input-size user=> (quick-bench (mc-pi input-size)) WARNING: Final GC required 4.001679309213317 % of runtime Evaluation count : 6 in 6 samples of 1 calls. Execution time mean :634.387833 ms Execution time std-deviation : 33.222001 ms Execution time lower quantile : 606.122000 ms ( 2.5%) Execution time upper quantile : 677.273125 ms (97.5%) nil Here's all the information in the form of a table: Function Input Size Chunk Size Mean Std Dev. GC Time mc-pi 1,000,000 NA 634.39ms 33.22 ms 4.0%   mc-pi-pmap 1,000,000 NA 1.92 sec 888.52 ms 2.60%   mc-pi-part 1,000,000 4,096 455.94 ms 4.19 ms 8.75%   Here's a chart with the same information: How it works… There are a couple of things we should talk about here. Primarily, we'll need to look at chunking the inputs for pmap, but we should also discuss Monte Carlo methods. Estimating with Monte Carlo simulations Monte Carlo simulations work by throwing random data at a problem that is fundamentally deterministic, but when it's practically infeasible to attempt a more straightforward solution. Calculating pi is one example of this. By randomly filling in points in a unit square, p/4 will be approximately the ratio of points that will fall within a circle centered on 0, 0. The more random points that we use, the better the approximation. I should note that this makes a good demonstration of Monte Carlo methods, but it's a terrible way to calculate pi. It tends to be both slower and less accurate than the other methods. Although not good for this task, Monte Carlo methods have been used for designing heat shields, simulating pollution, ray tracing, financial option pricing, evaluating business or financial products, and many, many more things. For a more in-depth discussion, Wikipedia has a good introduction to Monte Carlo methods at http://en.wikipedia.org/wiki/Monte_Carlo_method. Chunking data for pmap The table we saw earlier makes it clear that partitioning helped: the partitioned version took just 72 percent of the time that the serial version did, while the naïve parallel version took more than three times longer. Based on the standard deviations, the results were also more consistent. The speed up is because each thread is able to spend longer on each task. There is a performance penalty to spreading the work over multiple threads. Context switching (that is, switching between threads) costs time, and coordinating between threads does as well. But we expect to be able to make that time and more up by doing more things at once. However, if each task itself doesn't take long enough, then the benefit won't out-weigh the costs. Chunking the input—and effectively creating larger individual tasks for each thread— gets around this by giving each thread more to do, and thereby spending less time context switching and coordinating, relative to the overall time spent running.

(For more resources related to this topic, see here.) Getting ready To get the best result after setting up your lab, you should plan it properly at first. Your lab will be used to practise certain penetration testing skills. Therefore, in order to properly plan your lab environment, you should first consider which skills you want to practise. Although you could also have non-common or even unique reasons to build a lab, I can provide you with the average list of skills one might need to practice: Essential skills Discovery techniques Enumeration techniques Scanning techniques Network vulnerability exploitation Privilege escalation OWASP TOP 10 vulnerabilities discovery and exploitation Password and hash attacks Wireless attacks Additional skills Modifying and testing exploits Tunneling Fuzzing Vulnerability research Documenting the penetration testing process All these skills are applied in real-life penetration testing projects, depending on its depth and the penetration tester's qualifications. The following skills could be practised at three lab types or their combinations: Network security lab Web application lab Wi-Fi lab I should mention that the lab planning process for each of the three lab types listed consists of the same four phases: Determining the lab environment requirements: This phase helps you to actually understand what your lab should include. In this phase, all the necessary lab environment components should be listed and their importance for practising different penetration testing skills should be assessed. Determining the lab environment size: The number of various lab environment components should be defined in this phase. Determining the required resources: The point of this phase is to choose which hardware and software could be used for building the lab with the specified parameters and fit it with what you actually have or are able to get. Determining the lab environment architecture: This phase designs the network topology and network address space How to do it... Now, I want to describe step by step how to plan a common lab combined of all three lab types listed in the preceding section using the following four-phase approach: Determine the lab environment requirements: To fit our goal and practise particular skills, the lab should contain the following components: Skills to practice Necessary components Discovery techniques Several different hosts with various OSs Firewall IPS Enumeration techniques Scanning techniques Network vulnerability exploitation OWASP TOP 10 vulnerabilities discovery and exploitation Web server Web application Database server Web Application Firewall Password and hash attacks Workstations Servers Domain controller FTP server Wireless attacks Wireless router Radius server Laptop or any other host with Wi-Fi adapter Modifying and testing exploits Any host Vulnerable network service Debugger Privilege escalation Any host Tunnelling Several hosts Fuzzing Any host Vulnerable network service Debugger Vulnerability research Documenting the penetration testing process Specialized software Now, we can make our component list and define the importance of each component for our lab (importance ranges between less important, Additional, and most important, Essential): Components Importance Windows server Essential Linux server Important FreeBSD server Additional Domain controller Important Web server Essential FTP Server Important Web site Essential Web 2.0 application Important Web Application Firewall Additional Database server Essential Windows workstation Essential Linux workstation Additional Laptop or any other host with Wi-Fi adapter Essential Wireless router Essential Radius server Important Firewall Important IPS Additional Debugger Additional Determine the lab environment size: In this step, we should decide how many instances of each component we need in our lab. We will count only the essential and important components' numbers, so let's exclude all additional components. This means that we've now got the following numbers: Components Number Windows server 2 Linux server 1 Domain controller 1 Web server 1 FTP Server 1 Web site 1 Web 2.0 application 1 Database server 1 Windows workstation 2 Host with Wi-Fi adapter 2 Wireless router 1 Radius server 1 Firewall 2 Determine required resources: Now, we will discuss the required resources: Server and victim workstations will be virtual machines based on VMWare Workstation 8.0. To run the virtual machines without any trouble, you will need to have an appropriate hardware platform based on a CPU with two or more cores and at least 4 GB RAM. Windows servers OSs will work under Microsoft Windows 2008 Server and Microsoft Windows Server 2003. We will use Ubuntu 12.04 LTS as a Linux server OS. Workstations will work under Microsoft Windows XP SP3 and Microsoft Windows 7. ASUS WL-520gc will be used as the LAN and WLAN router. Any laptop as the attacker's host. Samsung Galaxy Tab as the Wi-Fi victim (or other device supporting Wi-Fi). We will use free software as a web server, an FTP-server, and a web application, so there is no need for any hardware or financial resources to get these requirements. Determine the lab environment architecture: Now, we need to design our lab network and draw a scheme: Address space parameters DHCP server: 192.168.1.1 Gateway: 192.168.1.1 Address pool: 192.168.1.2-15 Subnet mask: 255.255.255.0 How it works... In the first step, we discovered which types of lab components we need by determining what could be used to practise the following skills: All OSs and network services are suitable for practicing discovery, enumeration, and scanning techniques and also for network vulnerability exploitation. We also need at least two firewalls – windows built-in software and router built-in firewall functions. Firewalls are necessary for learning different scanning techniques and firewall rules detection knowledge. Additionally, you can use any IPS for practicing evasion techniques. A web server, a website, and a web application are necessary for learning how to disclose and exploit OWASP TOP 10 vulnerabilities. Though a Web Application Firewall (WAF) is not necessary, it helps to improve web penetration testing skills to higher level. An FTP service ideally fits to practice password brute-forcing. Microsoft domain services are necessary to understand and try Windows domain passwords and hash attacks including relaying. This is why we need at least one network service with remote password authentication and at least one Windows domain controller with two Windows workstations. A wireless access point is essential for performing various wireless attacks, but it is better to combine LAN router and Wi-Fi access point in one device. So, we will use Wi-Fi router with several LAN ports. A radius server is necessary for practicing attacks on WLAN with WPA-Enterprise security. A Laptop and a tablet PC with any Wi-Fi adapters will work as an attacker, and victim in wireless attacks. Tunnelling techniques could be practiced at any two hosts; it does not matter whether we use Windows or any other OS. Testing and modifying exploits as well as fuzzing and vulnerability research need a debugger installed on a vulnerable host. To properly document a penetration testing process, one can use just any test processor software, but there are several specialized software solutions, which make a thing much more comfortable and easier. In the second step, we determined which software and hardware we can use as instances of chosen component types and set their importance based on a common lab for a basic and intermediate professional level penetration tester. In the third step, we understood which solutions will be suitable for our tasks and what we can afford. I have tried to choose a cheaper option, which is why I am going to use virtualization software. The ASUS WL-520gc router combines the LAN router and Wi-Fi access point in the same device, so it is cheaper and more comfortable than using dedicated devices. A laptop and a tablet PC are also chosen for practising wireless attacks, but it is not the cheapest solution. In the fourth step, we designed our lab network based on determined resources. We have chosen to put all the hosts in the same subnet to set up the lab in an easier way. The subnet has its own DHCP server to dynamically assign network addresses to hosts. There's more... Let me give you an account of alternative ways to plan the lab environment details. Lab environment components variations It is not necessary to use a laptop as the attacker machine and a tablet PC as the victim – you just need two PCs with connected Wi-Fi adapters to perform various wireless attacks. As an alternative to virtual machines, a laptop, and a tablet PC or old unused computers (if you have them) could also be used to work as hardware hosts. There is only one condition – their hardware resources should be enough for planned OSs to work. An IPS could be either a software or hardware, but hardware systems are more expensive. For our needs, it is enough to use any freeware Internet security software including both the firewall and IPS functionality. It is not essential to choose the same OS as I have chosen in this chapter; you can use any other OSs that support the required functionality. The same is true about network services – it is not necessary to use an FTP service; you can use any other service that supports network password authentication such as telnet and SSH. You will have to additionally install any debugger on one of the victim's workstations in order to test the new or modified exploits and perform vulnerability research, if you need to. Finally, you can use any other hardware or virtual router that supports LAN routing and Wi-Fi access point functionality. A connected, dedicated LAN router and Wi-Fi access point are also suitable for the lab. Choosing virtualization solutions – pros and cons Here, I want to list some pros and cons of the different virtualization solutions in table format: Solution Pros Cons VMWare ESXi Enterprise solution Powerful solution Easily supports a lot of virtual machines on the same physical server as separate partitions No need to install the OS Very high cost Requires a powerful server Requires processor virtualization support VMWare workstation Comfortable to work with User friendly GUI Easy install Virtual *nix systems work fast Better works with virtual graphics Shareware It sometimes faces problems with USB Wi-Fi adapters on Windows 7 Demanding towards system resources Does not support 64-bit guest OS Virtual Windows systems work slowly VMWare player Freeware User-friendly GUI Easy to install Cannot create new virtual machines Poor functionality Micrisoft Virtual PC Freeware Great compatibility and stability with Microsoft systems Good USB support Easy to install Works only on Windows and only with Windows Does not support a lot of features that concurrent solutions do Oracle Virtual Box Freeware Virtual Windows systems work fast User-friendly GUI Easy to install Works on Mac OS and Solaris as well as on Windows and Linux Supports the "Teleportation" technology Paid USB support; Virtual *nix systems work slowly Here, I have listed only the leaders of the virtualization market in my opinion. Historically, I am mostly accustomed to VMWare Workstation, but of course, you can choose any other solutions that you may like. You can find more comparison info at http://virt.kernelnewbies.org/TechComparison. Summary This article explained how you can plan your lab environment. Resources for Article : Further resources on this subject: FAQs on BackTrack 4 [Article] CISSP: Vulnerability and Penetration Testing for Access Control [Article] BackTrack 4: Target Scoping [Article]

(For more resources related to this topic, see here.) The Raspberry Pi with attached file storage functions well as a file server. Such a file server could be used as a central location for sharing files and documents, for storing backups of other computers, and for storing large media files such as photo, music, and video files. This recipe installs and configures samba and samba-common-bin. The Samba software distribution package, samba, contains a server for the SMB (CIFS) protocol used by Windows computers for setting up 'shared drives' or 'shared folders'. The samba-common-bin package contains a small collection of utilities for managing access to shared files. The recipe includes setting a file-sharing password for the user pi and providing read/write access to the files in pi user's home directory. However, it does not set up a new file share or show how to share a USB disk. The next recipe shows how to do that. After completing this recipe, other computers can exchange files with the default user pi. Getting ready The following are the ingredients: A Raspberry Pi with a 5V power supply An installed and configured "official" Raspbian Linux SD card A network connection A client PC connected to the same network as the Raspberry Pi This recipe does not require the desktop GUI and could either be run from the text-based console or from within an LXTerminals. If the Raspberry Pi's Secure Shell server is running and it has a network connection, this recipe can be remotely completed using a Secure Shell client. How to do it... The following are the steps for creating a file server: Log in to the Raspberry Pi either directly or remotely. Execute the following command: apt-get install samba samba-common-bin Download and install samba with the other packages that it depends on. This command needs to be run as a privileged user (use sudo). The package management application aptitude could be used as an alternative to the apt-get command utility for downloading and installing samba and samba-common-bin. In the preceding screenshot, the apt-get command is used to install the samba and samba-common-bin software distribution packages. Execute the following command: nano /etc/samba/smb.conf Edit the samba configuration file. The smb.conf file is protected and needs to be accessed as a privileged user (use sudo). The preceding screenshot starts the nano editor to edit the /etc/samba/smb.conf file. Change the security = user line. Uncomment the line (remove the hash, #, from the beginning of the line). The preceding screenshot of the Samba configuration file shows how to change samba security to use the Raspberry Pi's user accounts. Change the read only = yes line to be read only = no, as shown in the following screenshot: The preceding screenshot shows how to change the Samba configuration file to permit adding new files to user shares (read only = no). Save and exit the nano editor. Execute the following command: /etc/init.d/samba reload Tell the Samba server to reload its configuration file. This command is privileged (use sudo). In the preceding screenshot, the Samba server's configuration file is reloaded with the /etc/init.d/samba command. Execute the following command: smbpasswd –a pi This command needs to be run as a privileged user (use sudo). Enter the password (twice) that will be used for SMB (CIFS) file sharing. The preceding screenshot shows how to add an SMB password for the pi user. The Raspberry Pi is now accessible as a Windows share! From a Windows computer, use Map network drive to mount the Raspberry Pi as a network disk, as follows: The preceding screenshot starts mapping a network drive to the Raspberry Pi on Windows 7. Enter the UNC address raspberrypipi as the network folder. Choose an appropriate drive letter. The example uses the Z: drive. Select Connect using different credentials and click on Finish, as shown in the following screenshot: The preceding screenshot finishes mapping a network drive to the Raspberry Pi. Log in using the newly configured SMB (CIFS) password (from step 7). In the screenshot, a dialog box is displayed for logging in to the Raspberry Pi with the SMB (CIFS) username and password. The Raspberry Pi is now accessible as a Windows share! Only the home directory of the pi user is accessible at this point. The next recipe configures a USB disk for using it as a shared drive. How it works... This recipe begins by installing two software distribution packages – samba and samba-common-bin. This recipe uses the apt-get install command; however, the aptitude package management application could also be used to install software packages. The Samba package contains an implementation of the Server Message Block (SMB) protocol (also known as the Common Internet File System, CIFS). The SMB protocol is used by Microsoft Windows computers for sharing files and printers. The samba-common-bin package contains the smbpasswd command. This command is used to set up user passwords exclusively for using them with the SMB protocol. After the packages are installed, the Samba configuration file /etc/samba/smb.conf is updated. The file is updated to turn on user security and to enable writing files to user home directories. The smbpasswd command is used to add (-a) the pi user to the list of users authorized to share files with the Raspberry Pi using the SMB protocol. The passwords for file sharing are managed separately from the passwords used to log in to the Raspberry Pi either directly or remotely. The smbpasswd command is used to set the password for Samba file sharing. After the password has been added for the pi user, the Raspberry Pi should be accessible from any machine on the local network that is configured for the SMB protocol. The last steps of the recipe configure access to the Raspberry Pi from a Windows 7 PC using a mapped network drive. The UNC name for the file share, raspberrypipi, could also be used to access the share directly from Windows Explorer. There's more... This is a very simple configuration for sharing files. It enables file sharing for users with a login to the Raspberry Pi. However, it only permits the files in the user home directories to be shared. The next recipe describes how to add a new a file share. In addition to the SMB protocol server, smbd, the Samba software distribution package also contains a NetBIOS name server, nmbd. The NetBIOS name server provides naming services to computers using the SMB protocol. The nmbd server broadcasts the configured name of the Raspberry Pi, raspberrypi, to other computers on the local network. In addition to file sharing, a Samba server could also be used as a Primary Domain Controller (PDC) — a central network server that is used to provide logins and security for all computers on a LAN. More information on using the Samba package as a PDC can be found on the links given next. See Also Samba (software) http://en.wikipedia.org/wiki/Samba_(software) A Wikipedia article on the Samba software suite. nmbd - NetBIOS over IP naming service http://manpages.debian.net/cgi-bin/man.cgi?query=nmbd The Debian man page for nmbd. samba – a Windows SMB/CIFS file server for UNIX http://manpages.debian.net/cgi-bin/man.cgi?query=samba The Debian man page for samba. smb.conf - the configuration file for the Samba suite http://manpages.debian.net/cgi-bin/man.cgi?query=smb.conf The Debian man page for smb.conf. smbd - server to provide SMB/CIFS services to clients http://manpages.debian.net/cgi-bin/man.cgi?query=smbd The Debian man page for smbd. smbpasswd - change a user's SMB password http://manpages.debian.net/cgi-bin/man.cgi?query=smbpasswd The Debian man page for smbpasswd. System initialization http://www.debian.org/doc/manuals/debian-reference/ch04.en.html The Debian Reference Manual article on system initialization. Samba.org http://www.samba.org The Samba software website. Resources for Article : Further resources on this subject: Using PVR with Raspbmc [Article] Our First Project – A Basic Thermometer [Article] Instant Minecraft Designs – Building a Tudor-style house [Article]

(For more resources related to this topic, see here.) Types of testing Testing can happen on many different levels. From the code level to integration and even testing individual functions of the user-facing implementation of an enterprise application, there are numerous tools and techniques to test your application. In particular, we will cover the following: Unit testing Functional testing Browser testing Black box versus white box testing Testing is often talked about within the context of black box versus white box testing. This is a useful metaphor in understanding testing at different levels. With black box testing, you look at your application as a black box knowing nothing of its internals—typically from the perspective of a user of the system. You simply execute functionality of the application and test whether the expected outcomes match the actual outcomes. White box differs from black box testing in that you know the internals of the application upfront and can thus pinpoint failures directly and test for specific conditions. In this case, you simply feed in data into specific parts of the system and test whether the expected output matches the actual output. Unit testing The first level of testing is at the code level. When you are te sting specific and individual units of code on whether they meet their stated goals, you are unit testing. Unit testing is often talked about in conjunction with test-driven development, the practice of writing unit tests first and then writing the minimal amount of code necessary to pass those tests. Having a suite of unit tests against your code and employing test-driven processes—when done right—can keep your code focused and help to ensure the stability of your enterprise application. Typically, unit tests are set up in a separate folder in your codebase. Each test case is composed of the following parts: Setup to build the test conditions under which the code or module is being tested An instantiation and invocation of the code or module being tested A verification of the results returned Setting up your unit test You usually start by setting up your test data. For example, if you are testing a piece of code that requires an authenticated account, you might consider creating a set of test users of your enterprise application. It is advisable that your test data be coupled with your test so that your tests are not dependent on your system being in a specific state. Invoking your target Once you have set up your test data and the conditions in which the code you are testing needs to run, you are ready to invoke it. This can be as simple as invoking a method. Mocking is a very important concept to understand when unit testing. Consider a set of unit tests for a business logic module that has a dependency on some external application programming interface (API). Now imagine if the API goes down. The tests would fail. While it is nice to get an indication that the API you are dependent upon is having issues, a failing unit test because of this is misleading because the goal of the unit test is to test the business logic rather than external resources on which you are dependent. This is where mock objects come into the picture. Mock objects are stubs that replicate the interface of a resource. They are set up to always return the same data the external resource would under normal conditions. This way you are isolating your test to just the unit of code you are testing. Mocking employs a pattern called dependency injection or inversion of control. Sure, the code you are testing may be dependent on an external resource. Yet how will you swap it in a mock resource? Code that is easy to unit test allows you to pass in or "inject" these dependencies when invoking it. Dependency injection is a design pattern where code that is dependent on an external resource has that dependency passed into it thereby decoupling your code from that dependency. The following code snippet is difficult to test since the dependency is encapsulated into the function being tested. We are at an impasse. var doSomething = function() {var api = getApi();//A bunch of codeapi.call();}var testOfDoSomething = function() {var mockApi = getMockApi();//What do I do now???} The following new code snippet uses dependency injection to circumvent the problem by instantiating the dependency and passing it into the function being tested: var doSomething = function(api) {//A bunch of codeapi.call();}var testOfDoSomething = function() {var mockApi = getMockApi();doSomething(mockApi);} In general, this is good practice not just for unit testing but for keeping your code clean and easy to manage. Instantiating a dependency once and injecting where it is needed makes it easier to change that dependency if the need occurs. There are many mocking frameworks available including JsMockito (http://jsmockito.org/ ) for JavaScript and Mockery (https://github.com/padraic/mockery)for PHP. Verifying the results Once you have invoked the code being tested, you need to capture the results and verify them. Verification comes in the form of assertions. Every unit testing framework comes with its own set of assertion methods, but the concept is the same: take a result and test it against an expectation. You can assert whether two things are equal. You can assert whether two things are not equal. You can assert whether a result is a valid number of a string. You can assert whether one value is greater than another. The general idea is you are testing actual data against your hypothesis. Assertions usually bubble up to the framework's reporting module and are manifested as a list of passed or failed tests. Frameworks and tools A bevy of tools have arisen in the past few years that aid in unit testing of JavaScript. What follows is a brief survey of notable frameworks and tools used to unit test JavaScript code. JsTestDriver JsTestDriver is a framework built at Google for unit testing. It has a server that runs on multiple browsers on a machine and will allow you to execute test cases in the Eclipse IDE. This screenshot shows the results of JsTestDriver. When run, it executes all tests configured to run and displays the results. More information about JsTestDriver can be found at http://code.google.com/p/js-test-driver/. QUnit QUnit is a JavaScript unit testing framework created by John Resig of jQuery fame. To use it, you need to create only a test harness web page and include the QUnit library as a script reference. There is even a hosted version of the library. Once included, you need to only invoke the test method, passing in a function and a set of assertions. It will then generate a nice report. Although QUnit has no dependencies and can test standard JavaScript code, it is oriented around jQuery. More information about QUnit can be found at http://qunitjs.com/. Sinon.JS Often coupled with QUnit, Sinon.JS introduces the concept of spying wherein it records function calls, the arguments passed in, the return value, and even the value of the this object. You can also create fake objects such as fake servers and fake timers to make sure your code tests in isolation and your tests run as quickly as possible. This is particularly useful when you need to make fake AJAX requests. More information about Sinon.JS can be found at http://sinonjs.org/. Jasmine Jasmine is a testing framework based on the concept of behavior-driven development. Much akin to test-driven development, it extends it by infusing domain-driven design principles and seeks to frame unit tests back to user-oriented behavior and business value. Jasmine as well as other behavior-driven design based frameworks build test cases—called specs—using as much English as possible so that when a report is generated, it reads more naturally than a conventional unit test report. More information about Jasmine can be found at http://pivotal.github.com/jasmine/. Functional testing Selenium has become the name in website functional testing. Its browser automation capabilities allow you to record test cases in your favorite web browser and run them across multiple browsers. When you have this, you can automate your browser tests, integrate them with your build and continuous integration server, and run them simultaneously to get quicker results when you need them. Selenium includes the Selenium IDE, a utility for recording and running Selenium scripts. Built as a Firefox add-on, it allows you to create Selenium test cases by loading and clicking on web pages in Firefox. You can easily record what you do in the browser and replay it. You can then add tests to determine whether actual behavior matches expected behavior. It is very useful for quickly creating simple test cases for a web application. Information on installing it can be found at http://seleniumhq.org/docs/02_selenium_ide.html. The following screenshot shows the Selenium IDE. Click on the red circle graphic on the right-hand side to set it to record, and then browse to http://google.com in the browser window and search for "html5". Click on the red circle graphic to stop recording. You can then add assertions to test whether certain properties of the page match expectations. In this case, we are asserting that the text of the first link in the search results is for the Wikipedia page for HTML5. When we run our test, we see that it passes (of course, if the search results for "html5" on Google change, then this particular test will fail). Selenium includes WebDriver, an API that allows you to drive a browser natively either locally or remotely. Coupled with its automation capabilities, WebDriver can run tests against browsers on multiple remote machines to achieve greater scale. For our MovieNow application, we will set up functional testing by using the following components: The Selenium standalone server The php-webdriver connector from Facebook PHPUnit The Selenium standalone server The Selenium standalone server routes requests to the HTML5 application. It needs to be started for the tests to run. It can be deployed anywhere, but by default it is accessed at http://localhost:4444/wd/hub. You can download the latest version of the standalone server at http://code.google.com/p/selenium/downloads/list or you can fire up the version included in the sample code under the test/lib folder. To start the server, execute the following line via the command line (you will need to have Java installed on your machine): java -jar lib/selenium-server-standalone-#.jar Here, # indicates the version number. You should see something akin to the following: At this point, it is listening for connections. You will see log messages here as you run your tests. Keep this window open. The php-webdriver connector from Facebook The php-webdriver connector serves as a library for WebDriver in PHP. It gives you the ability to make and inspect web requests using drivers for all the major web browsers as well as HtmlUnit. Thus it allows you to create test cases against any web browser. You can download it at https://github.com/facebook/php-webdriver.We have included the files in the webdriver folder. PHPUnit PHPUnit is a unit testing framework that provides the constructs necessary for running our tests. It has the plumbing necessary for building and validating test cases. Any unit testing framework will work with Selenium; we have chosen PHPUnit since it is lightweight and works well with PHP. You can download and install PHPUnit any number of ways (you can go to http://www.phpunit.de/manual/current/en/installation.html for more information on installing it). We have included the phpunit.phar file in the test/lib folder for your convenience. You can simply run it by executing the following via the command line: php lib/phpunit.phar <your test suite>.php To begin, we will add some PHP files to the test folder. The first file is webtest. php. Create this file and add the following code: <?phprequire_once "webdriver/__init__.php";class WebTest extends PHPUnit_Framework_TestCase {protected $_session;protected $_web_driver;public function __construct() {parent::__construct();$_web_driver = new WebDriver();$this->_session = $_web_driver->session('firefox');}public function __destruct() {$this->_session->close();unset($this->_session);}}?> The WebTest class integrated WebDriver into PHPUnit via the php-webdriver connector. This will serve as the base class for all of our test cases. As you can see, it starts with the following: require_once "webdriver/__init__.php"; This is a reference to __init__.php in the php-webdriver files. This brings in all the classes needed for WebDriver. In the constructor, WebTest initializes the driver and session objects used in all test cases. In the destructor, it cleans up its connections. Now that we have everything set up, we can create our first functional test. Add a file called generictest.php to the test folder. We will import WebTest and extend that class as follows: <?phprequire_once "webtest.php";class GenericTest extends WebTest {}?> Inside of the GenericTest class, add the following test case: public function testForData() {$this->_session->open('http://localhost/html5-book/Chapter%2010/');sleep(5); //Wait for AJAX data to load$result = $this->_session->element("id", "movies-near-me")->text();//May need to change settings to always allow sharing of location$this->assertGreaterThan(0, strlen($result));} We will open a connection to our application (feel free to change the URL to wherever you are running your HTML5 application), wait 5 seconds for the initial AJAX to load, and then test for whether the movies-near-me div is populated with data. To run this test, go to the command line and execute the following lines: chmod +x lib/phpunit.pharphp lib/phpunit.phar generictest.php You should see the following: This indicates that the test is passed. Congratulations! Now let us see it fail. Add the following test case: public function testForTitle() {$this->_session->open('http://localhost/html5-book/Chapter%2010/');$result = $this->_session->title();$this->assertEquals('Some Title', $result);} Rerun PHPUnit and you should see something akin to the following: As you can see, it was expecting 'Some Title' but actually found 'MovieNow'. Now that we have gotten you started, we will let you create your own tests. Refer to http://www.phpunit.de/manual/3.7/en/index.html for guidance on the different assertions you can make using PHPUnit. More information about Selenium can be found at http://seleniumhq.org/. Browser testing HTML5 enterprise applications must involve actually looking at the application on different web browsers. Thankfully, many web browsers are offered on multiple platforms. Google Chrome, Mozilla Firefox, and Opera all have versions that will install easily on Windows, Mac OSX, and flavors of Linux such as Ubuntu. Safari has versions for Windows and Mac OSX, and there are ways to install it on Linux with some tweaking. Nevertheless, Internet Explorer can only run on Windows. One way to work around this limitation is to install virtualization software. Virtualization allows you to run an entire operating system virtually within a host operating system. It allows you to run Windows applications on Mac OSX or Linux applications on Windows. There are a number of notable virtualization packages including VirtualBox, VMWare Fusion, Parallels, and Virtual PC. Although Virtual PC runs only on Windows, Microsoft does offer a set of prepackaged virtual hard drives that include specific versions of Internet Explorer for testing purposes. See the following URLs for details: http://www.microsoft. com/en-us/download/details.aspx?id=11575. Another common way to test for compatibility is to use web-based browser virtualization. There are a number of services such as BrowserStack (http://www.browserstack.com/), CrossBrowserTesting (http://crossbrowsertesting.com/), and Sauce Labs (https://saucelabs.com/) that offer a service whereby you can enter a URL and see it rendered in an assortment of web browsers and platforms (including mobile) virtually through the web. Many of them even work through a proxy to allow you to view, test, and debug web applications running on your local machine. Continuous integration With any testing solution, it is important to create and deploy your builds and run your tests in an automated fashion. Continuous integration solutions like Hudson, Jenkins, CruiseControl, and TeamCity allow you to accomplish this. They merge code from multiple developers, and run a number of automated functions from deploying modules to running tests. They can be invoked to run on a schedule basis or can be triggered by events such as a commitment of code to a code repository via a postcommit hook. Summary We covered several types of testing in this article including unit testing, functional testing, and browser testing. For each type of testing, there are many tools to help you make sure that your enterprise application runs in a stable way, most of which we covered bar a few. Because every minute change to your application code has the potential to destabilize it, we must assume that that every change does. To ensure that your enterprise applications remain stable and with minimal defect, having a testing strategy in place with a rich suite of tests—from unit to functional—combined with a continuous integration server running those tests is essential. One must, of course, weigh the investment in time for writing and executing tests against the time needed for writing production code, but the savings in long-term maintenance costs can make that investment worthwhile. Resources for Article : Further resources on this subject: Building HTML5 Pages from Scratch [Article] Blocking versus Non blocking scripts [Article] HTML5: Generic Containers [Article]

(For more resources related to this topic, see here.) >>> data = scipy.stats.randint.rvs(0.4,10,size=(10,2))>>> triangulation = scipy.spatial.Delaunay(data) Any Delaunay class has the basic search attributes such as points (to obtain the set of points in the triangulation), vertices (that offers the indices of vertices forming simplices in the triangulation), neighbors (for the indices of neighbor simplices for each simplex—with the convention that "-1" indicates no neighbor for simplices at the boundary). More advanced attributes, for example convex_hull, indicate the indices of the vertices that form the convex hull of the given points. If we desire to search for the simplices that share a given vertex, we may do so with the vertex_to_simplex method. If, instead, we desire to locate the simplices that contain any given point in the space, we do so with the find_simplex method. At this stage we would like to point out the intimate relationship between triangulations and Voronoi diagrams, and offer a simple coding exercise. Let us start by choosing first a random set of points, and obtaining the corresponding triangulation. >>> locations=scipy.stats.randint.rvs(0,511,size=(2,8))>>> triangulation=scipy.spatial.Delaunay(locations.T) We may use the matplotlib.pyplot routine triplot to obtain a graphical representation of this triangulation. We first need to obtain the set of computed simplices. Delaunay offers us this set, but by means of the indices of the vertices instead of their coordinates. We thus need to map these indices to actual points before feeding the set of simplices to the triplot routine: >>>assign_vertex = lambda index: triangulation.points[index]>>>triangle_set = map(assign_vertex, triangulation.vertices)>>>matplotlib.pyplot.triplot(locations[1], locations[0], ... triangles=triangle_set, color='r') We will now obtain the edge map of the Voronoi diagram in a similar fashion as we did before, and plot it below the triangulation (since the former needs to be with either a pcolormesh or imshow command). Note how the triangulation and the corresponding Voronoi diagrams are dual of each other; each edge in the triangulation (red) is perpendicular with an edge in the Voronoi diagram (white). How should we use this observation to code an actual Voronoi diagram for a cloud of points? The actual Voronoi diagram is the set of vertices and edges that composes it, rather than a binary image containing an approximation to the edges as we have computed. Let us finish this Article with two applications to scientific computing that use these techniques extensively, in combination with routines from other SciPy modules. Structural model of oxides In this example we will cover the extraction of the structural model of a molecule of a bronze-type Niobium oxide, from HAADF-STEM micrographs. The following diagram shows HAADF-STEM micrograph of a bronze-type Niobium oxide (taken from http://www.microscopy.ethz.ch/BFDF-STEM.htm, courtesy of ETH Zurich): For pedagogical purposes, we took the following approach to solving this problem: Segmentation of the atoms by thresholding and morphological operations. Connected component labeling to extract each single atom for posterior examination. Computation of the centers of mass of each label identified as an atom. This presents us with a lattice of points in the plane that shows a first insight in the structural model of the oxide. Computation of the Voronoi diagram of the previous lattice of points. The combination of information with the output of the previous step will lead us to a decent (approximation of the actual) structural model of our sample. Let us proceed in this direction. Once retrieved, our HAADF-STEM images will be stored as big matrices with float32 precision. For this project, it is enough to retrieve some tools from the scipy.ndimage module, and some procedures from the matplotlib library. The preamble then looks like the following code: import numpyimport scipyfrom scipy.ndimage import *from scipy.misc import imfilterimport matplotlib.pyplot as plt The image is loaded with the imread(filename) command. This stores the image as a numpy.array with dtype = float32. Notice that the maxima and minima are 1.0 and 0.0, respectively. Other interesting information about the image can be retrieved: img=imread('/Users/blanco/Desktop/NbW-STEM.png')print "Image dtype: %s"%(img.dtype)print "Image size: %6d"%(img.size)print "Image shape: %3dx%3d"%(img.shape[0],img.shape[1])print "Max value %1.2f at pixel %6d"%(img.max(),img.argmax())print "Min value %1.2f at pixel %6d"%(img.min(),img.argmin())print "Variance: %1.5fnStandard deviation:%1.5f"%(img.var(),img.std()) This outputs the following information: Image dtype: float32Image size: 87025Image shape: 295x295Max value 1.00 at pixel 75440Min value 0.00 at pixel 5703Variance: 0.02580Standard deviation: 0.16062 We perform thresholding by imposing an inequality in the array holding the data. The output is a Boolean array where True (white) indicates that the inequality is fulfilled, and False (black) otherwise. We may perform at this point several thresholding operations and visualize them to obtain the best threshold for segmentation purposes. The following images show several examples (different thresholdings applied to the oxide image): By visual inspection of several different thresholds, we choose 0.62 as one that gives us a good map showing what we need for segmentation. We need to get rid of "outliers", though; small particles that might fulfill the given threshold but are small enough not to be considered as an actual atom. Therefore, in the next step we perform a morphological operation of opening to get rid of those small particles. We decided that anything smaller than a square of size 2 x 2 is to be eliminated from the output of thresholding: BWatoms = (img> 0.62)BWatoms = binary_opening(BWatoms,structure=numpy.ones((2,2))) We are ready for segmentation, which will be performed with the label routine from the scipy.ndimage module. It collects one slice per segmented atom, and offers the number of slices computed. We need to indicate the connectivity type. For example, in the following toy example, do we want to consider that situation as two atoms or one atom? It depends; we would rather have it now as two different connected components, but for some other applications we might consider that they are one. The way we indicate the connectivity to the label routine is by means of a structuring element that defines feature connections. For example, if our criterion for connectivity between two pixels is that they are in adjacent edges, and then the structuring element looks like the image shown on the left-hand side from the images shown next. If our criterion for connectivity between two pixels is that they are also allowed to share a corner, then the structuring element looks like the image on the right-hand side. For each pixel we impose the chosen structuring element and count the intersections; if there are no intersections, then the two pixels are not connected. Otherwise, they belong to the same connected component. We need to make sure that atoms that are too close in a diagonal direction are counted as two, rather than one, so we chose the structuring element on the left. The script then reads as follows: structuring_element = [[0,1,0],[1,1,1],[0,1,0]]segmentation,segments = label(BWatoms,structuring_element) The segmentation object contains a list of slices, each of them with a Boolean matrix containing each of the found atoms of the oxide. We may obtain for each slice a great deal of useful information. For example, the coordinates of the centers of mass of each atom can be retrieved with the following commands: coords = center_of_mass(img, segmentation, range(1,segments+1))xcoords = numpy.array([x[1] for x in coords])ycoords = numpy.array([x[0] for x in coords]) Note that, because of the way matrices are stored in memory, there is a transposition of the x and y coordinates of the locations of the pixels. We need to take it into account. Notice the overlap of the computed lattice of points over the original image (the left-hand side image from the two images shown next). We may obtain it with the following commands: >>>plt.imshow(img); plt.gray(); plt.axis('off')>>>plt.plot(xcoords,ycoords,'b.') We have successfully found the centers of mass for most atoms, although there are still about a dozen regions where we are not too satisfied with the result. It is time to fine-tune by the simple method of changing the values of some variables; play with the threshold, with the structuring element, with different morphological operations, and so on. We can even add all the obtained information for a wide range of those variables, and filter out outliers. An example with optimized segmentation is shown, as follows (look at the right-hand side image): For the purposes of this exposition, we are happy to keep it simple and continue working with the set of coordinates that we have already computed. We will be now offering an approximation to the lattice of the oxide, computed as the edge map of the Voronoi diagram of the lattice. L1,L2 = distance_transform_edt(segmentation==0,return_distances=False,return_indices=True)Voronoi = segmentation[L1,L2]Voronoi_edges= imfilter(Voronoi,'find_edges')Voronoi_edges=(Voronoi_edges>0) Let us overlay the result of Voronoi_edges with the locations of the found atoms: >>>plt.imshow(Voronoi_edges); plt.axis('off'); plt.gray()>>>plt.plot(xcoords,ycoords,'r.',markersize=2.0) This gives the following output, which represents the structural model we were searching for:

(For more resources related to this topic, see here.) Getting ready We will start with the packet flow of NetScaler and where content switching comes into play. The following diagram is self-explanatory (it is not the entire packet flow to the receiver's endpoint; the focus here is only to CS and LB): The content switching vserver can be used for HTTP/HTTPS/TCP and UDP protocols, and it can direct it only to another vserver, not to the backend service directly. The content switching vserver doesn't need an LB vserver to be bound to it for its status to be UP. Even with nothing bound to the CS vserver, the status would show UP (this comes in handy when you want to blackhole unwanted traffic).Hence, it is always recommended to check whether the load balancing vservers that are bound to the content switching vserver are up and running. If you want to avoid the preceding condition, the following CLI command will help you achieve it (by default, the value is disabled): root@ NetScaler> add cs vserver <name> <serviceType> (<IPAddress>) [-stateupdate ( ENABLED | DISABLED )] Content switching can be done based on the following client attributes: Mobile user/PC Images/videos Dynamic/static content Client with/without cookies Geographical locations. Per VLAN Similarly, server-side differentiations can also be made based on the following attributes: Server speed and capacity Source/destination port Source/destination IP SSL/HTTP Citrix also has an additional feature (starting from NetScaler version 9.3) that dynamically selects the load balancing feature based on any criteria or condition provided in the CS action/policy: >add cs action <name> -targetLBVserver <string-expression> >add cs policy <policyName> -rule <RULEValue> -action <actionName> The policy is then bound to the CS vserver CS vservers can be configured to process URLs in a case-sensitive manner. By default, this option is ON: >set cs vserver CSVserver -caseSensitive ON The load balancing vserver bound to the CS vserver need not have any IP address configured unless it is used in a different access as well. How to do it... We shall focus on a few case studies that we commonly come across, and that can be solved with the help of content switching: Case 1: Customer ABC accesses an online shopping portal and gets redirected to a secure connection at the payment gateway. For this scenario, an HTTP LB vserver is used and is bound to the CS vserver, which is on HTTPS: The configuration in the preceding screenshot shows that a CS policy as well as a responder policy is bound to the CS vserver named testVserver. The CS policy works on directing the traffic to the target LB vserver (if there are no CS policies bound at all, it goes to the default LB vserver; this default LB vserver should be configured on the CS Vserver). The responder policy, if bound to the CS vserver works on HTTP requests before matching any CS policy. The configuration is verified by using show cs vserver <vserver name>. A packet capture taken on NetScaler will clearly show the redirect from HTTP to HTTPS as <HTTP 302>. If there is any traffic that doesn't match any specific CS policies that are bound, then it uses the default policy. If there is no default policy, the user will get an error – HTTP 1.1 Service Unavailable error message. Case 2: The customer Star Networks has a single web application that contains two domains, namely www.starnetworks.com and www.starnetworks.com.edu and has a content switching setup, which works fine when accessing www.starnetworks.com, but throws an error when accessing www.starnetworks.com.edu. This happens because the peceding domains are not the same; they are different and the certificate that is bound to the CS vserver would be of type www.starnetworks.com only. To resolve this issue, we can bind multiple certificates to the CS vserver with the Server Name Indication (SNI) option enabled. The SNI option can be enabled in the SSL Parameters tab (this would pop up only if the SSL protocol is chosen while creating the vserver). The CLI command to enable SNI is as follows: >bind sslvserver star_cs_vserver -certkeyname -SNICert > bind sslvserver star_cs_vserver -certkeyname -SNICert For each domain added, NetScaler will establish a secure channel between itself and the client. With this solution, you can avoid configuring multiple CS vservers. Case 3: A Customer has a large pool of IP subnets that needs categorizing, and it would be a next to impossible task to configure that number of content switching policies; how does he go about deploying this scenario? The solution is as follows: A database file should be created that includes the IP address range and the domain: >shell #cd /var/ NetScaler/locdb # vi test.db Run the following command to apply the changes made to the database file: > add locationfile aol.db Bind the CS policy with an expression stating, for example, as follows: "CLIENT.IP.SRC.MATCHES_LOCATION ("star.*.*.*.*.*")"" How it works... The working of NetScaler in all three preceding scenarios is that it analyzes the incoming traffic directed to the CS VIP and parses through the bound CS policies, if any. If a match is found, it goes to the target LB vserver. If there are any other policies that are bound (for example, a responder policy or a rewrite policy), then the responder policy gets executed even before the CS policy is executed (since responder policies are usually applied to the HTTP requests).However, rewrite policies can be bound either at the CS or LB level, depending on whether the request or response needs to be modified. To recap what we have seen in the case studies mentioned before, the first case helps us to do a simple redirect from HTTP to HTTPS using a responder policy bound at the CS level. The second case shows us how multiple certificates with the SNI option are used to solve domain differences that would otherwise cause issues. The final case study shows us the basic but handy setting to map IP address ranges to target load balancing vservers. An important thing to note – there are scenarios where the vserver and the services that are bound to them may be different ports altogether (for example, HTTP LB VIP would be listening on port 80, but the services would be on port 8080). In such cases, the redirectPortRewrite feature should be enabled. There's more... This section concentrates on tidbits and troubleshooting techniques: Tips and troubleshooting We can start with checking the output of show cs and show lb vservers, to see if the services bound to them are up and running: root@ns > show cs vserver cs_star_vserver 1) cs_star_vserver (IP_ADDRESS_HERE:80) - HTTP Type: CONTENT State: UP Client Idle Timeout: 180 sec Down state flush: ENABLED Port Rewrite : DISABLED Default: lb_vserver Content Precedence: RULE Vserver IP and Port insertion: OFF Case Sensitivity: OFF If there are responder and rewrite policies, then we can check whether the number of hits on that policy are incrementing or not. Packet captures (using Wireshark) on the server and NetScaler. In some cases, the client would show us the packet flow in depth. The Down state flush feature of the NetScaler is useful for admins planning their downtimes in advance. This feature is enabled, by default, on the vserver and service level. When the feature is enabled, the connections that are already open and established will be terminated and the users will have to retry their connections again. The requests that are already being processed alone would be honored. When the feature is disabled, the open and established connections are honored, and no new connections will be accepted at this time. If enabled at the vserver level, and if the state of the vserver is DOWN, then the vserver will flush the client and server connections that are linked. Otherwise, it would terminate only the client facing connections. At the server level, if the service is marked as DOWN, then only the server facing connections would be flushed. There is another option on the Advanced tab of the CS/LB vserver to direct the excess traffic to a backup vserver. In cases where the backup server also overflows, there is an option to use the redirect URL, which is also found in the Advanced tab of the CS/LB vserver. Summary This article has explained the implementation of content switching using Citrix Security. Resources for Article : Further resources on this subject: Managing Citrix Policies [Article] Getting Started with XenApp 6 [Article] Getting Started with the Citrix Access Gateway Product Family [Article]

(For more resources related to this topic, see here.) How to do it... Building the front page: i. Open iBooks Author and select the Editorial template. We will replace the placeholder image on the first page with an image of Tower Bridge from http://wikitravel.org/en/File:Tower_bridge_ London_Twilight_-_November_2006.jpg. ii. Add the Tower Bridge image to your iPhoto library and then the image will be available for use within iBooks Author. iii. Within iBooks Author, now you can drag this image to the placeholder image on the first page to make your page look like the following: iv. Replace the dummy text on the introduction page with a generic description of London from WikiTravel. You may need to remove/edit some text to accommodate the spacing and layout; we have used the text "Noisy, vibrant and truly multicultural, London is a megalopolis of people, ideas and frenetic energy. The capital and largest city of both the United Kingdom and of England, it is also the largest city in Western Europe and the European Union. Situated on the River Thames, London remains an international capital of culture, music, education, fashion, politics, finance and trade." v. Now for the book's cover page, let us use the Clock Tower photo from http://hub.packtpub.com/wp-content/uploads/2013/04/FileClock_Tower.jpg. Now your book cover should look like the following; note that this is how the book will be displayed on the iBooks Bookshelf as well: Building the History page: i. The next page in our London CityGuide book will be a page that describes the history of the city of London. Click on page 2 of your book in iBooks; based on our template chosen earlier, page 2 of our book should look like the following screenshot: ii. We will now use information from Wikipedia (http://en.wikipedia.org/wiki/London) to build our History page. We copy the historical facts about London from Wikipedia, format it, and copy it to our page 2. As we keep adding content to our page, iBooks Author automatically adds new pages, depending upon the text. iii. Don't forget to rename the title of page 2 from Untitled to History; you can do so by right-clicking on page 2 and selecting the Rename option. iv. After adding our history information from Wikipedia, our History page should now look like the following: Building the Transportation page: i. We will create another section in our book to accommodate the transportation information for London; we do so by copying the Section 1 (History) page and renaming it to Transportation: Previewing the books on your iPad: i. Now would be a good time to have a quick look at the book and how it works on the iPad. The new iPad mini was used during the writing of this book. Now, we will move on to discussing how to preview books on your iPad (without submitting books to iBookstore). ii. From within iBooks Author, click on the Preview button on the toolbar in the header. Make sure you have an iPad connected, as well as the iBooks app open. iii. Here is how our London CityGuide book looks on an actual iPad, starting with the index page: iv. Clicking on Introduction or swiping right-to-left (page-turn effect), shows us the Introduction page: v. Similarly the History page (in landscape mode) looks stunning with the Thames river backdrop! vi. While the Transportation page looks like the following: Building the Culture page: i. In the previous section, we added a new section to our book, using the copy-and-paste method. iBooks Author also provides another way to add pages, sections, or even chapters to our book via the Add Pages button in the header toolbar. We use the same method to add a new section to our book and name it Culture. ii. We can also select the different layouts that we want for our section, as shown in the preceding image. We choose the Section Text layout for our Culture section. We then copy over the cultural information about London from Wikipedia onto our iBooks Author (after some formatting, by removing citations from the Wikipedia text). Note that on all the pages where we have used images to make our London CityGuide elegant, these images are sourced from Wikipedia and due attribution needs to be provided to the authors. iii. Our Culture page now looks like the following: Having added a few background and general information about London, let's move forward to adding new chapters for attractions, restaurants, and shopping. Adding chapters for attractions and restaurants: i. We start by adding a new chapter using the Add Pages button from the header toolbar within iBooks Author and naming it as Attractions. The content for this chapter will be sourced from the WikiTravel page for London— http://wikitravel.org/en/London#See. ii. Similarly, we will add chapters for restaurants and add content from open source travel guide websites such as www.Wikitravel.org and www.Wikipedia.org. We will not go into the details of adding all the pages, since it is redundant, and by now you, the reader, should have a good idea of how things work with iBooks Author! You can also choose to write you own content, maybe for your own hometown; get going around your town and get some pictures, and the rest is what we've already discussed. iii. Here is how the Attractions page should look like; for the benefit of the readers, we are including the full source code and text for this book; the name of the file is London_CityGuide.iba and it is available for download via the book's page on www.packtpub.com. Summary This article has helped us create London CityGuide with the help of iBook Author. Resources for Article : Further resources on this subject: Development of iPhone Applications [Article] iPhone JavaScript: Installing Frameworks [Article] Jailbreaking the iPad - in Ubuntu [Article]

(For more resources related to this topic, see here.) Comparison Choosing a technology does not merely involve a technical comparison. Several other factors related to documentation, maintainability, stability and maturity, vendor support, developer community, license, price, and the future of the product or the organization behind it also play important roles. Having said that, I must also add that technical comparison should continue to play a pivotal role. We will start a deep technical comparison of the previously mentioned products and then look at the semi-technical and non-technical aspects for the same. Technical comparison From a technical perspective, we compare on the following parameters: Implementation language Engine types Speed Implementation language One of the more important factors that come into play is how can, if required, the product be extended; the programming language in which the product itself is written determines a large part of it. Some of the database may provide a different language for writing plugins but it may not always be true: Amazon SimpleDB: It is available in cloud and has a client SDK for Java, .NET, PHP, and Ruby. There are libraries for Android and iOS as well. BaseX: Written in Java. To extend, one must code in Java. Cassandra: Everything in Java. CouchDB: Written in Erlang. To extend use Erlang. Google Datastore: It is available in cloud and has SDK for Java, Python, and Go. HBase: It is Java all the way. MemcacheDB: Written in C. Uses the same language to extend. MongoDB: Written in C++. Client drivers are available in several languages including but not limited to JavaScript, Java, PHP, Python, and Ruby. Neo4j: Like several others, it is Java all the way Redis: Written in C. So you can extend using C. Great, so the first parameter itself may have helped you shortlist the products that you may be interested to use based on the developers available in your team or for hire. You may still be tempted to get smart people onboard and then build competency based on the choice that you make, based on subsequent dimensions. Note that for the databases written in high-level languages like Java, it may still be possible to write extensions in languages like C or C++ by using interfaces like JNI or otherwise. Amazon SimpleDB provides access via the HTTP protocol and has SDK in multiple languages. If you do not find an SDK for yourself, say for example, in JavaScript for use with NodeJS, just write one. However, life is not open with Google Datastore that allows access only via its cloud platform App Engine and has SDKs only in Java, Python, and the Go languages. Since the access is provided natively from the cloud servers, you cannot do much about it. In fact, the top requested feature of the Google App Engine is support for PHP ( See http://code.google.com/p/googleappengine/issues/list). Engine types Engine types define how you will structure the data and what data design expertise your team will need. NoSQL provides multiple options to choose from. Database Column oriented Document store Key value store Graph Amazon SimpleDB No No Yes No BaseX No Yes No No Cassandra Yes Yes No No CouchDB No Yes No No Google Datastore Yes No No No HBase Yes No No No MemcacheDB No No Yes No MongoDB No Yes No No Neo4j No No No Yes Redis No Yes Yes No You may notice two aspects of this table – a lot of No and multiple Yes against some databases. I expect the table to be populated with a lot more Yes over the next couple of years. Specifically, I expect the open source databases written in Java to be developed and enhanced actively providing multiple options to the developers. Speed One of the primary reasons for choosing a NoSQL solution is speed. Comparing and benchmarking the databases is a non-trivial task considering that each database has its own set of hardware and other configuration requirements. Having said that, you can definitely find a whole gambit of benchmark results comparing one NoSQL database against the other with details of how the tests were executed. Of all that is available, my personal choice is the Yahoo! Cloud Serving Benchmark (YCSB) tool. It is open source and available on Github at https://github.com/brianfrankcooper/YCSB. It is written in Java and clients are available for Cassandra, DynamoDB, HBase, HyperTable, MongoDB, Redis apart from several others that we have not discuss in this book. Before showing some results from the YCSB, I did a quick run on a couple of easy-to-set-up databases myself. I executed them without any optimizations to just get a feel of how easy it is for software to incorporate it without needing any expert help. I ran it on MongoDB on my personal box (server as well as the client on the same machine), DynamoDB connecting from a High-CPU Medium (c1.medium) box, and MySQL on the same High-CPU Medium box with both server and client on the same machine. Detailed configurations with the results are shown as follows: Server configuration: Parameter MongoDB DynamoDB MySQL Processor 5 EC2 Compute Units N/A 5 EC2 Compute Units RAM 1.7 GB with Apache HTTP server running (effective free: 200 MB, after database is up and running) N/A 1.7GB with Apache HTTP server running (effective free: 500MB, after database is up and running) Hard disk Non-SSD N/A Non-SSD Network configuration N/A US-East-1 N/A Operating system Ubuntu 10.04, 64 bit N/A Ubuntu 10.04, 64 bit Database version 1.2.2 N/A 5.1.41 Configuration Default Max write: 500, Max read: 500 Default Client configuration: Parameter MongoDB DynamoDB MySQL Processor 5 EC2 Compute Units 5 EC2 Compute Units 5 EC2 Compute Units RAM 1.7GB with Apache HTTP server running (effective free: 200MB, after database is up and running) 1.7GB with Apache HTTP server running (effective free: 500MB, after database is up and running) 1.7GB with Apache HTTP server running (effective free: 500MB after database is up and running) Hard disk Non-SSD Non-SSD Non-SSD Network configuration Same Machine as server US-East-1 Same Machine as server Operating system Ubuntu 10.04, 64 bit Ubuntu 10.04, 64 bit Ubuntu 10.04, 64 bit Record count 1,000,000 1,000 1,000,000 Max connections 1 5 1 Operation count (workload a) 1,000,000 1,000 1,000,000 Operation count (workload f) 1,000,000 100,000 1,000,000 Results: Workload Parameter MongoDB DynamoDB MySQL Workload-a (load) Total time 290 seconds 16 seconds 300 seconds   Speed (operations/second) 2363 to 4180 (approximately 3700) Bump at 1278 50 to 82 (operations/second) 3135 to 3517 (approximately 3300)   Insert latency 245 to 416 microseconds (approximately 260) Bump at 875 microseconds 12 to 19 milliseconds 275 to 300 microseconds (approximately 290) Workload-a (run) Total time 428 seconds 17 seconds 240 seconds   Speed 324 to 4653 42 to 78 3970 to 4212   Update latency 272 to 2946 microseconds 13 to 23.7 microseconds 219 to 225.5 microseconds   Read latency 112 to 5358 microseconds 12.4 to 22.48 microseconds 240.6 to 248.9 microseconds Workload-f (load) Total time 286 seconds Did not execute 295 seconds   Speed 3708 to 4200   3254 to 3529   Insert latency 228 to 265 microseconds   275 to 299 microseconds Workload-f (run) Total time 412 seconds Did not execute 1022 seconds   Speed 192 to 4146   224 to 2096   Update latency 219 to 336 microseconds   216 to 233 microseconds, with two bursts at 600 and 2303 microseconds   Read latency 119 to 5701 microseconds   1360 to 8246 microseconds   Read Modify Write (RMW) latency 346 to 9170 microseconds   1417 to 14648 microseconds Do not read too much into these numbers as they are a result of the default configuration, out-of-the-box setup without any optimizations. Some of the results from YCSB published by Brian F. Cooper (http://www.brianfrankcooper.net/pubs/ycsb-v4.pdf) are shown next. For update-heavy, 50-50 read-update: For read-heavy, under varying hardware: There are some more from Sergey Sverchkov at Altoros (http://altoros.com/nosql-research) who published their white paper recently. Summary In this article, we did a detailed comparative study of ten NoSQL databases on few parameters, both technical and non-technical. Resources for Article : Further resources on this subject: Getting Started with CouchDB and Futon [Article] Ruby with MongoDB for Web Development [Article] An Introduction to Rhomobile [Article]  

(For more resources related to this topic, see here.) Show/hide rows Click a link to trigger hiding or displaying of table rows. Getting ready Once again, start off with an HTML table. This one is not quite as simple a table as in previous recipes. You'll need to create a few <td> tags that span the entire table, as well as provide some specific classes to certain elements. How to do it... Again, give the table an id attribute. Each of the rows that represent a department, specifically the rows that span the entire table, should have a class attribute value of dept. <table border="1" id="employeeTable"> <thead> <tr> <th>Last Name</th> <th>First Name</th> <th>Phone</th> </tr> </thead> <tbody> <tr> <td colspan="3" class="dept"> </td> </tr> Each of the department names should be links where the <a> elements have a class of rowToggler. <a href="#" class="rowToggler">Accounting</a> Each table row that contains employee data should have a class attribute value that corresponds to its department. Note that class names cannot contain spaces. So in the case of the Information Technology department, the class names should be InformationTechnology without a space. The issue of the space will be addressed later. <tr class="Accounting"> <td>Frang</td> <td>Corey</td> <td>555-1111</td> </tr> The following script makes use of the class names to create a table whose rows can be easily hidden or shown by clicking a link: <script type="text/javascript"> $( document ).ready( function() { $( "a.rowToggler" ).click( function( e ) { e.preventDefault(); var dept = $( this ).text().replace( /s/g, "" ); $( "tr[class=" + dept + "]" ).toggle(); }) }); </script> With the jQuery implemented, departments are "collapsed", and will only reveal the employees when the link is clicked. How it works... The jQuery will "listen" for a click event on any <a> element that has a class of rowToggler. In this case, capture a reference to the event that triggered the action by passing e to the click handler function. $( "a.rowToggler" ).click( function( e ) In this case, e is simply a variable name. It can be any valid variable name, but e is a standard convention. The important thing is that jQuery has a reference to the event. Why? Because in this case, the event was that an <a> was clicked. The browser's default behavior is to follow a link. This default behavior needs to be prevented. As luck would have it, jQuery has a built-in function called preventDefault(). The first line of the function makes use of this by way of the following: e.preventDefault(); Now that you've safely prevented the browser from leaving or reloading the page, set a variable with a value that corresponds to the name of the department that was just clicked. var dept = $( this ).text().replace( /s/g, "" ); Most of the preceding line should look familiar. $( this ) is a reference to the element that was clicked, and text() is something you've already used. You're getting the text of the <a> tag that was clicked. This will be the name of the department. But there's one small issue. If the department name contains a space, such as "Information Technology", then this space needs to be removed. .replace( /s/g, "" ) replace() is a standard JavaScript function that uses a regular expression to replace spaces with an empty string. This turns "Information Technology" into "InformationTechnology", which is a valid class name. The final step is to either show or hide any table row with a class that matches the department name that was clicked. Ordinarily, the selector would look similar to the following: $( "tr.InformationTechnology" ) Because the class name is a variable value, an alternate syntax is necessary. jQuery provides a way to select an element using any attribute name and value. The selector above can also be represented as follows: $( "tr[class=InformationTechnology]" ) The entire selector is a literal string, as indicated by the fact that it's enclosed in quotes. But the department name is stored in a variable. So concatenate the literal string with the variable value: $( "tr[class=" + dept + "]" ) With the desired elements selected, either hide them if they're displayed, or display them if they're hidden. jQuery makes this very easy with its built-in toggle() method. Highlighting cells Use built-in jQuery traversal methods and selectors to parse the contents of each cell in a table and apply a particular style (for example, a yellow background or a red border) to all cells that meet a specified set of criteria. Getting ready Borrowing some data from Tiobe (http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html), create a table of the top five programming languages for 2012. To make it "pop" a bit more, each <td> in the Ratings column that's over 10 percent will be highlighted in yellow, and each <td> in the Delta column that's less than zero will be highlighted in red. Each <td> in the Ratings column should have a class of ratings, and each <td> in the Delta column should have a class of delta. Additionally, set up two CSS classes for the highlights as follows: .highlight { background-color: #FFFF00; } /* yellow */ .highlight-negative { background-color: #FF0000; } /* red */ Initially, the table should look as follows: How to do it... Once again, give the table an id attribute (but by now, you knew that), as shown in the following code snippet: <table border="1" id="tiobeTable"> <thead> <tr> <th>Position<br />Dec 2012</th> <th>Position<br />Dec 2011</th> <th>Programming Language</th> <th>Ratings<br />Dec 2012</th> <th>Delta<br />Dec 2011</th> </tr> </thead> Apply the appropriate class names to the last two columns in each table row within the <tbody>, as shown in the following code snippet: <tbody> <tr> <td>1</td> <td>2</td> <td>C</td> <td class="ratings">18.696%</td> <td class="delta">+1.64%</td> </tr> With the table in place and properly marked up with the appropriate class names, write the script to apply the highlights as follows: <script type="text/javascript"> $( document ).ready( function() { $( "#tiobeTable tbody tr td.ratings" ).each( function( index ) { if ( parseFloat( $( this ).text() ) > 10 ) { $( this ).addClass( "highlight" ); } }); $( "#tiobeTable tbody tr td.delta" ).each( function( index ) { if ( parseFloat( $( this ).text() ) < 0 ) { $( this ).addClass( "highlight-negative" ); } }); }); </script> Now, you will see a much more interesting table with multiple visual cues: How it works... Select the <td> elements within the tbody tag's table rows that have a class of ratings. For each iteration of the loop, test whether or not the value (text) of the <td> is greater than 10. Because the values in <td> contain non-numeric characters (in this case, % signs), we use JavaScript's parseFloat() to convert the text to actual numbers: parseFloat( $( this ).text() ) Much of that should be review. $( this ) is a reference to the element in question. text() retrieves the text from the element. parseFloat() ensures that the value is numeric so that it can be accurately compared to the value 10. If the condition is met, use addClass() to apply the highlight class to <td>. Do the same thing for the Delta column. The only difference is in checking to see if the text is less than zero. If it is, apply the class highlight-negative. The end result makes it much easier to identify specific data within the table. Summary In this article we covered two recipes Show/hide rows and Highlighting cells. Resources for Article : Further resources on this subject: Tips and Tricks for Working with jQuery and WordPress5 [Article] Using jQuery Script for Creating Dynamic Table of Contents [Article] Getting Started with jQuery [Article]

(For more resources related to this topic, see here.) Getting ready Any learner taking a quiz may want to know how well he/she has answered the questions posed. Often, working with Moodle, the instructor is at a distance from the learner. Providing feedback is a great way of enhancing communication between learner and instructor. Learner feedback can be provided at multiple levels using Moodle Quiz. You can create feedback at various levels in both the questions and the overall quiz. Here we will examine feedback at the question level. General feedback When we add General Feedback to a question, every student sees the feedback, regardless of their answer to the question. This is good opportunity to provide clarification for the learner who had guessed a correct answer, as well as for the learner whose response was incorrect. Individual response feedback We can create feedback tailored to each possible response in a multiple choice question. This feedback can be more focused in nature. Often, a carefully crafted distracter in a multiple choice can reveal misconceptions and the feedback can provide the correction required as soon as the learner completes the quiz. Feedback given when the question is fresh in the learner's mind, is very effective. How to do it... Let's create some learner feedback for some of the questions that we have created in the question bank: First of all, let's add general feedback to a question. Returning to our True-False question on Texture, we can see that general feedback is effective when there are only two choices. Remember that this type of feedback will appear for all learners, regardless of the answer they submitted. The intention of this feedback is to reflect the correct solution and also give more background information to enhance the teaching opportunity. Let's take a look at how to create a specific feedback for each possible response that a learner may submit. This is done by adding individual response feedback. Returning to our multiple choice question on application of the element line, a specific feedback response tailored to each possible choice will provide helpful clarification for the student. This type of feedback is entered after each possible choice. Here is an example of a feedback to reinforce a correct response and a feedback for an incorrect response: In this case, the feedback the learner receives is tailored to the response they have submitted. This provides much more specific feedback to the learner's choice of responses. For the embedded question (Cloze), feedback is easy to add in Moodle 2.0. In the following screenshot, we can see the question that we created with feedback added: And this is what the feedback looks like to the student: How it works... We have now improved questions in our exam bank by providing feedback for the learner. We have created both general feedback that all learners will see and specific feedback for each response the learner may choose. As we think about the learning experience for the learner, we can see that immediate feedback with our questions is an effective way to reinforce learning. This is another feature that makes Moodle Quiz such a powerful tool. There's more... As we think about the type of feedback we want for the learner, we can combine feedback for individual responses with general feedback. Also there are options for feedback for any correct response, for any partially correct response, or for any incorrect response. Feedback serves to engage the learners and personalize the experience. We created question categories, organized our questions into categories, and learned how to add learner feedback at various levels inside the questions. We are now ready to configure a quiz. Summary In the article we have seen how we can add feedback to the questions of the Moodle Quiz. Resources for Article : Further resources on this subject: Integrating Moodle 2.0 with Mahara and GoogleDocs for Business [Article] What's New in Moodle 2.0 [Article] Moodle 2.0 FAQs [Article]

(For more resources related to this topic, see here.) Tuning Hadoop configurations for cluster deployments Getting ready Shut down the Hadoop cluster if it is already running, by executing the bin/stop-dfs.sh and bin/stop-mapred.sh commands from HADOOP_HOME. How to do it... We can control Hadoop configurations through the following three configuration files: conf/core-site.xml: This contains the configurations common to whole Hadoop distribution conf/hdfs-site.xml: This contains configurations for HDFS conf/mapred-site.xml: This contains configurations for MapReduce Each configuration file has name-value pairs expressed in an XML format, and they define the workings of different aspects of Hadoop. The following code snippet shows an example of a property in the configuration file. Here, the <configuration> tag is the top-level XML container, and the <property> tags that define individual properties go as child elements of the <configuration> tag. <configuration><property><name>mapred.reduce.parallel.copies</name><value>20</value></property>...</configuration> The following instructions show how to change the directory to which we write Hadoop logs and configure the maximum number of map and reduce tasks: Create a directory to store the logfiles. For example, /root/hadoop_logs. Uncomment the line that includes HADOOP_LOG_DIR in HADOOP_HOME/conf/ hadoop-env.sh and point it to the new directory. Add the following lines to the HADOOP_HOME/conf/mapred-site.xml file: <property><name>mapred.tasktracker.map.tasks.maximum</name><value>2 </value></property><property><name>mapred.tasktracker.reduce.tasks.maximum</name><value>2 </value></property> Restart the Hadoop cluster by running the bin/stop-mapred.sh and bin/start-mapred.sh commands from the HADOOP_HOME directory. You can verify the number of processes created using OS process monitoring tools. If you are in Linux, run the watch ps –ef|grep hadoop command. If you are in Windows or MacOS use the Task Manager. How it works... HADOOP_LOG_DIR redefines the location to which Hadoop writes its logs. The mapred. tasktracker.map.tasks.maximum and mapred.tasktracker.reduce.tasks. maximum properties define the maximum number of map and reduce tasks that can run within a single TaskTracker at a given moment. These and other server-side parameters are defined in the HADOOP_HOME/conf/*-site. xml files. Hadoop reloads these configurations after a restart. There's more... There are many similar configuration properties defined in Hadoop. You can see some of them in the following tables. The configuration properties for conf/core-site.xml are listed in the following table: Name Default value Description fs.inmemory.size.mb 100 This is the amount of memory allocated to the in-memory filesystem that is used to merge map outputs at reducers in MBs. io.sort.factor 100 This is the maximum number of streams merged while sorting files. io.file.buffer.size 131072 This is the size of the read/write buffer used by sequence files. The configuration properties for conf/mapred-site.xml are listed in the following table: Name Default value Description mapred.reduce. parallel.copies 5 This is the maximum number of parallel copies the reduce step will execute to fetch output from many parallel jobs. mapred.map.child.java. opts -Xmx200M This is for passing Java options into the map JVM. mapred.reduce.child. java.opts -Xmx200M This is for passing Java options into the reduce JVM. io.sort.mb 200 The memory limit while sorting data in MBs. The configuration properties for conf/hdfs-site.xml are listed in the following table: Name Default value Description dfs.block.size 67108864 This is the HDFS block size. dfs.namenode.handler. count 40 This is the number of server threads to handle RPC calls in the NameNode. Running benchmarks to verify the Hadoop installation The Hadoop distribution comes with several benchmarks. We can use them to verify our Hadoop installation and measure Hadoop's performance. This recipe introduces these benchmarks and explains how to run them. Getting ready Start the Hadoop cluster. You can run these benchmarks either on a cluster setup or on a pseudo-distributed setup. How to do it... Let us run the sort benchmark. The sort benchmark consists of two jobs. First, we generate some random data using the randomwriter Hadoop job and then sort them using the sort sample. Change the directory to HADOOP_HOME. Run the randomwriter Hadoop job using the following command: >bin/hadoop jar hadoop-examples-1.0.0.jarrandomwriter-Dtest.randomwrite.bytes_per_map=100-Dtest.randomwriter.maps_per_host=10 /data/unsorted-data Here the two parameters, test.randomwrite.bytes_per_map and test. randomwriter.maps_per_host specify the size of data generated by a map and the number of maps respectively. Run the sort program: >bin/hadoop jar hadoop-examples-1.0.0.jar sort /data/unsorted-data/data/sorted-data Verify the final results by running the following command: >bin/hadoop jar hadoop-test-1.0.0.jar testmapredsort -sortInput /data/unsorted-data -sortOutput /data/sorted-data Finally, when everything is successful, the following message will be displayed: The job took 66 seconds.SUCCESS! Validated the MapReduce framework's 'sort' successfully. How it works... First, the randomwriter application runs a Hadoop job to generate random data that can be used by the second sort program. Then, we verify the results through testmapredsort job. If your computer has more capacity, you may run the initial randomwriter step with increased output sizes. There's more... Hadoop includes several other benchmarks. TestDFSIO: This tests the input output (I/O) performance of HDFS nnbench: This checks the NameNode hardware mrbench: This runs many small jobs TeraSort: This sorts a one terabyte of data More information about these benchmarks can be found at http://www.michaelnoll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoopcluster- with-terasort-testdfsio-nnbench-mrbench/. Reusing Java VMs to improve the performance In its default configuration, Hadoop starts a new JVM for each map or reduce task. However, running multiple tasks from the same JVM can sometimes significantly speed up the execution. This recipe explains how to control this behavior. How to do it... Run the WordCount sample by passing the following option as an argument: >bin/hadoop jar hadoop-examples-1.0.0.jar wordcount –Dmapred.job.reuse.jvm.num.tasks=-1 /data/input1 /data/output1 Monitor the number of processes created by Hadoop (through ps –ef|grephadoop command in Unix or task manager in Windows). Hadoop starts only a single JVM per task slot and then reuses it for an unlimited number of tasks in the job. However, passing arguments through the –D option only works if the job implements the org.apache.hadoop.util.Tools interface. Otherwise, you should set the option through the JobConf.setNumTasksToExecutePerJvm(-1) method. How it works... By setting the job configuration property through mapred.job.reuse.jvm.num.tasks, we can control the number of tasks for the JVM run by Hadoop. When the value is set to -1, Hadoop runs the tasks in the same JVM.

(For more resources related to this topic, see here.) Introducing line charts First let's start with a single series line chart. We will use one of the many data provided by The World Bank organization at www.worldbank.org. The following is the code snippet to create a simple line chart which shows the percentage of population ages, 65 and above, in Japan for the past three decades: var chart = new Highcharts.Chart({chart: {renderTo: 'container'},title: {text: 'Population ages 65 and over (% of total)',},credits: {position: {align: 'left',x: 20},text: 'Data from The World Bank'},yAxis: {title: {text: 'Percentage %'}},xAxis: {categories: ['1980', '1981','1982', ... ],labels: {step: 5}},series: [{name: 'Japan - 65 and over',data: [ 9, 9, 9, 10, 10, 10, 10 ... ]}]}); The following is the display of the simple chart: Instead of specifying the year number manually as strings in categories, we can use the pointStart option in the series config to initiate the x-axis value for the first point. So we have an empty xAxis config and series config, as follows: xAxis: {},series: [{pointStart: 1980,name: 'Japan - 65 and over',data: [ 9, 9, 9, 10, 10, 10, 10 ... ]}] Although this simplifies the example, the x-axis labels are automatically formatted by Highcharts utility method, numberFormat, which adds a comma after every three digits. The following is the outcome on the x axis: To resolve the x-axis label, we overwrite the label's formatter option by simply returning the value to bypass the numberFormat method being called. Also we need to set the allowDecimals option to false. The reason for that is when the chart is resized to elongate the x axis, decimal values are shown. The following is the final change to use pointStart for the year values: xAxis: {labels:{formatter: function() {// 'this' keyword is the label objectreturn this.value;}},allowDecimals: false},series: [{pointStart: 1980,name: 'Japan - 65 and over',data: [ 9, 9, 9, 10, 10, 10, 10 ... ]}] Extending to multiple series line charts We can include several more line series and set the Japan series by increasing the line width to be 6 pixels wide, as follows: series: [{lineWidth: 6,name: 'Japan',data: [ 9, 9, 9, 10, 10, 10, 10 ... ]}, {Name: 'Singapore',data: [ 5, 5, 5, 5, ... ]}, {...}] The line series for Japanese population becomes the focus in the chart, as shown in the following screenshot: Let's move on to a more complicated line graph. For the sake of demonstrating inverted line graphs, we use the chart.inverted option to flip the y and x axes to opposite orientations. Then we change the line colors of the axes to match the same series colors. We also disable data point markers for all the series and finally align the second series to the second entry in the y-axis array, as follows: chart: {renderTo: 'container',inverted: true,},yAxis: [{title: {text: 'Percentage %'},lineWidth: 2,lineColor: '#4572A7'}, {title: {text: 'Age'},opposite: true,lineWidth: 2,lineColor: '#AA4643'}],plotOptions: {series: {marker: {enabled: false}}},series: [{name: 'Japan - 65 and over',type: 'spline',data: [ 9, 9, 9, ... ]}, {name: 'Japan - Life Expectancy',yAxis: 1,data: [ 76, 76, 77, ... ]}] The following is the inverted graph with double y axes: The data representation of the chart may look slightly odd as the usual time labels are swapped to the y axis and the data trend is awkward to comprehend. The inverted option is normally used for showing data in a noncontinuous form and in bar format. If we interpret the data from the graph, 12 percent of the population is 65 and over, and the life expectancy is 79 in 1990. By setting plotOptions.series.marker.enabled to false it switches off all the data point markers. If we want to display a point marker for a particular series, we can either switch off the marker globally and then set the marker on an individual series, or the other way round. plotOptions: {series: {marker: {enabled: false}}},series: [{marker: {enabled: true},name: 'Japan - 65 and over',type: 'spline',data: [ 9, 9, 9, ... ]}, { The following graph demonstrates that only the 65 and over series has point markers: Sketching an area chart In this section, we are going to use our very first example and turn it into a more stylish graph (based on the design of wind energy poster by Kristin Clute), which is an area spline chart. An area spline chart is generated using the combined properties of area and spline charts. The main data line is plotted as a spline curve and the region underneath the line is filled in a similar color with a gradient and an opaque style. Firstly, we want to make the graph easier for viewers to look up the values for the current trend, so we move the y axis next to the latest year, that is, to the opposite side of the chart: yAxis: { ....opposite:true} The next thing is to remove the interval lines and have a thin axis line along the y axis: yAxis: { ....gridLineWidth: 0,lineWidth: 1,} Then we simplify the y-axis title with a percentage sign and align it to the top of the axis: yAxis: { ....title: {text: '(%)',rotation: 0,x: 10,y: 5,align: 'high'},} As for the x axis, we thicken the axis line with a red color and remove the interval ticks: xAxis: { ....lineColor: '#CC2929',lineWidth: 4,tickWidth: 0,offset: 2} For the chart title, we move the title to the right of the chart, increase the margin between the chart and the title, and then adopt a different font for the title: title: {text: 'Population ages 65 and over (% of total) -Japan ',margin: 40,align: 'right',style: {fontFamily: 'palatino'}} After that we are going to modify the whole series presentation, we first set the chart.type property from 'line' to 'areaspline'. Notice that setting the properties inside this series object will overwrite the same properties defined in plotOptions.areaspline and so on in plotOptions.series. Since so far there is only one series in the graph, there is no need to display the legend box. We can disable it with the showInLegend property. We then smarten the area part with gradient color and the spline with a darker color: series: [{showInLegend: false,lineColor: '#145252',fillColor: {linearGradient: {x1: 0, y1: 0,x2: 0, y2: 1},stops:[ [ 0.0, '#248F8F' ] ,[ 0.7, '#70DBDB' ],[ 1.0, '#EBFAFA' ] ]},data: [ ... ]}] After that, we introduce a couple of data labels along the line to indicate that the ranking of old age population has increased over time. We use the values in the series data array corresponding to the year 1995 and 2010, and then convert the numerical value entries into data point objects. Since we only want to show point markers for these two years, we turn off markers globally in plotOptions.series. marker.enabled and set the marker on, individually inside the point objects accompanied with style settings: plotOptions: {series: {marker: {enabled: false}}},series: [{ ...,data:[ 9, 9, 9, ...,{ marker: {radius: 2,lineColor: '#CC2929',lineWidth: 2,fillColor: '#CC2929',enabled: true},y: 14}, 15, 15, 16, ... ]}] We then set a bounding box around the data labels with round corners (borderRadius) in the same border color (borderColor) as the x axis. The data label positions are then finely adjusted with the x and y options. Finally, we change the default implementation of the data label formatter. Instead of returning the point value, we print the country ranking. series: [{ ...,data:[ 9, 9, 9, ...,{ marker: {...},dataLabels: {enabled: true,borderRadius: 3,borderColor: '#CC2929',borderWidth: 1,y: -23,formatter: function() {return "Rank: 15th";}},y: 14}, 15, 15, 16, ... ]}] The final touch is to apply a gray background to the chart and add extra space into spacingBottom. The extra space for spacingBottom is to avoid the credit label and x-axis label getting too close together, because we have disabled the legend box. chart: {renderTo: 'container',spacingBottom: 30,backgroundColor: '#EAEAEA'}, When all these configurations are put together, it produces the exact chart, as shown in the screenshot at the start of this section. Mixing line and area series In this section we are going to explore different plots including line and area series together, as follows: Projection chart, where a single trend line is joined with two series in different line styles Plotting an area spline chart with another step line series Exploring a stacked area spline chart, where two area spline series are stacked on top of each other Simulating a projection chart The projection chart has spline area with the section of real data and continues in a dashed line with projection data. To do that we separate the data into two series, one for real data and the other for projection data. The following is the series configuration code for the future data up to 2024. This data is based on the National Institute of Population and Social Security Research report (http://www.ipss.go.jp/pp-newest/e/ppfj02/ppfj02.pdf). series: [{name: 'project data',type: 'spline',showInLegend: false,lineColor: '#145252',dashStyle: 'Dash',data: [ [ 2010, 23 ], [ 2011, 22.8 ],... [ 2024, 28.5 ] ]}] The future series is configured as a spline in a dashed line style and the legend box is disabled, because we want to show both series as being from the same series. Then we set the future (second) series color the same as the first series. The final part is to construct the series data. As we specify the x-axis time data with the pointStart property, we need to align the projection data after 2010. There are two approaches that we can use to specify the time data in a continuous form, as follows: Insert null values into the second series data array for padding to align with the real data series Specify the second series data in tuples, which is an array with both time and projection data Next we are going to use the second approach because the series presentation is simpler. The following is the screenshot only for the future data series: The real data series is exactly the same as the graph in the screenshot at the start of the Sketching an area chart section, except without the point markers and data label decorations. The next step is to join both series together, as follows: series: [{name: 'real data',type: 'areaspline',....}, {name: 'project data',type: 'spline',....}] Since there is no overlap between both series data, they produce a smooth projection graph: Contrasting spline with step line In this section we are going to plot an area spline series with another line series but in a step presentation. The step line transverses vertically and horizontally only according to the changes in series data. It is generally used for presenting discrete data, that is, data without continuous/gradual movement. For the purpose of showing a step line, we will continue from the first area spline example. First of all, we need to enable the legend by removing the disabled showInLegend setting and also remove dataLabels in the series data. Next is to include a new series, Ages 0 to 14, in the chart with a default line type. Then we will change the line style slightly differently into steps. The following is the configuration for both series: series: [{name: 'Ages 65 and over',type: 'areaspline',lineColor: '#145252',pointStart: 1980,fillColor: {....},data: [ 9, 9, 9, 10, ...., 23 ]}, {name: 'Ages 0 to 14',// default type is line seriesstep: true,pointStart: 1980,data: [ 24, 23, 23, 23, 22, 22, 21,20, 20, 19, 18, 18, 17, 17, 16, 16, 16,15, 15, 15, 15, 14, 14, 14, 14, 14, 14,14, 14, 13, 13 ]}] The following screenshot shows the second series in the stepped line style:

(For more resources related to this topic, see here.) Tudor-style house In this recipe, we'll be building a Tudor-style house. We'll be employing some manual building methods, and we'll also introduce some WorldEdit CUI commands and VoxelSniper actions into our workflow. Getting ready Once you have installed the recommended mods, you will need to have Equip an Arrow tools equipped on your action bar. This is used by VoxelSniper to perform its functions. You will also need to equip a Wooden Axe as this item becomes the WorldEdit tool and will be used for making selections. Don't try and use them to break blocks especially if you have made a selection that you don't want to lose. Not only will they not break the block, they will also wreck your selection or worse. How to do it... Let's get started with building our Tudor-style house by performing the following steps: Find a nice area or clear one with roughly 40 x 40 squares of flat land. Mark out a selection of 37 x 13 blocks by left-clicking with the Wooden Axe to set the first point and then right-clicking for the second point. Hit your T key and type the //set 5:1 command. This will make all of the blocks in the selected area turn into Spruce Wood Planks. If you make a mistake, you can do //undo. The //undo command does not alter the selection itself, only the changes made to blocks. Now create a selection 20 x 13 that will complete the L shape of the mansion's bottom floor. Remember to left-click and right-click with the Wooden Axe tool. Now type //set 5:1. In the corner that will be at the end of the outside wall of the longest wing, place a stack of three Spruce Wood blocks on top of each other. Right beside this, place two stacked Wool blocks and one Spruce Wood block on top of them, as shown in the inset of the following screenshot: With the selection in place, we will now stack these six blocks horizontally along the 37 block wall. The stack command works in the direction you face. So face directly down the length of the floor and type //stack 17. If you make a mistake, do //undo. Go to the opposite end of the wall you just made and place a stack of three Spruce Wood blocks in the missing spot at the end. Then just like before, put two blocks of White Wool on the side of the corner Spruce Wood pole with one Spruce Wood block on top. Select these six blocks and facing along the short end wall, type //stack 5. Go to the end of this wall and complete it with the three Spruce Logs and two blocks of Wool with one Spruce block on top where the next wall will go. Select these six blocks. Remember! Wooden Axe, left-click, right-click. Facing down the inside wall, type //stack 11. Place another three Spruce Wood blocks upright in the corner and two Wool blocks with one Spruce block on top for the adjacent inner wall. Make a selection, face in the correct direction, and then type //stack 9. Repeat this same process of placing the six blocks, selecting them, facing in correct direction for the next wall, and typing //stack 5. Finally, type //stack 15 and your base should now be complete. On the corner section, we're going to make some bay windows. So let's create the reinforcing structure for those: Inset by two blocks from the corner place five of these reinforcement structures. They consist of one Spruce Wood upright and two upside down Nether Brick steps, each aligned to the Spruce Wood uprights behind them. Now we'll place the wall sections of the bay windows. You should be able to create these by referring to the right-hand section of the following screenshot: Now comes the use of VoxelSniper GUI. So let's add some windows using it. Hit your V key to bring up the VoxelSniper GUI. We're going to "snipe" some windows into place. The first section, Place, in the top left-hand side represents the block you wish to place. For this we will select Glass. The section directly below Place is the Replace panel. As the name suggests, this is the block you wish to replace. We wish to replace White Wool, so we'll select that. Scroll through and locate the Wool block. In the right-hand side, under the Ink panel scroll box, select the White Wool block. Make sure the No-Physics checkbox is not selected. In the right-hand panel, we will select the tool we wish to use. If it's not already selected, click on the Sized tab and choose Snipe. If you get lost, just follow the preceding screenshot. Choose your Arrow tool and right-click on the White Wool blocks you wish to change to Glass. VoxelSniper works from a distance hence the "Sniper" part of the name, so be careful when experimenting with this tool. If you make a mistake in VoxelSniper, use /u to undo. You can also do /u 5, or /u 7, or /u 22, and so on and so on if you wish to undo multiple actions. The upcoming screenshots should illustrate the sort of pattern we will implement along each of the walls. The VoxelSniper GUI tool retains the last settings used so you can just fill in all the Glass sections of the wall with Wool initially, and then replace them using VoxelSniper once you are done. For now, just do it for the two longest outer walls. The following screenshot shows the 37 and 33 block length walls: On the short wing end wall, we'll fill the whole area with White Wool. So let's type //set 35. On the short side, make a 21 x 4 selection like the one shown in the following screenshot (top-left section), and stand directly on the block as indicated by the player in the top-left section of the screenshot. Do //copy and then move to the pole on the opposite side. Once you are on the corner column like in the bottom-left section of the preceding screenshot, do //paste. To be sure that you are standing exactly on the right block, turn off flying (double-click Space bar), knock the block out below your feet, and make sure you fall down to the block below. Then jump up and replace the block. Do the same for the other wing. Select the wall section with the windows, repeat the process like you did for the previous wall, and then fill in the end wall with Wool blocks for now. Add a wooden floor that is level with the three Wool blocks below the Spruce Window frames. You can use the //set 5:1 command to fill in the large rectangular areas.

(For more resources related to this topic, see here.) The message queue A message queue, or technically a FIFO (First In First Out) queue is a fundamental and well-studied data structure. There are different queue implementations such as priority queues or double-ended queues that have different features, but the general idea is that the data is added in a queue and fetched when the data or the caller is ready. Imagine we are using a basic in-memory queue. In case of an issue, such as power outage or a hardware failure, the entire queue could be lost. Hence, another program that expects to receive a message will not receive any messages. However, adopting a message queue guarantees that messages will be delivered to the destination no matter what happens. Message queuing enables asynchronous communication between loosely-coupled components and also provides solid queuing consistency. In case of insufficient resources, which prevent you from immediately processing the data that is sent, you can queue them up in the message queue server that would store the data until the destination is ready to accept the messages. Message queuing has an important role in large-scaled distributed systems and enables asynchronous communication. Let's have a quick overview on the difference between synchronous and asynchronous systems. In ordinary synchronous systems, tasks are processed one at a time. A task is not processed until the task in-process is finished. This is the simplest way to get the job done. Synchronous system We could also implement this system with threads. In this case threads process each task in parallel. Threaded synchronous system In the threading model, threads are managed by the operating system itself on a single processor or multiple processors/cores. Asynchronous Input/Output (AIO) allows a program to continue its execution while processing input/output requests. AIO is mandatory in real-time applications. By using AIO, we could map several tasks to a single thread. Asynchronous system The traditional way of programming is to start a process and wait for it to complete. The downside of this approach is that it blocks the execution of the program while there is a task in progress. However, AIO has a different approach. In AIO, a task that does not depend on the process can still continue. You may wonder why you would use message queue instead of handling all processes with a single-threaded queue approach or multi-threaded queue approach. Let's consider a scenario where you have a web application similar to Google Images in which you let users type some URLs. Once they submit the form, your application fetches all the images from the given URLs. However: If you use a single-threaded queue, your application would not be able to process all the given URLs if there are too many users If you use a multi-threaded queue approach, your application would be vulnerable to a distributed denial of service attack (DDoS) You would lose all the given URLs in case of a hardware failure In this scenario, you know that you need to add the given URLs into a queue and process them. So, you would need a message queuing system. Introduction to ZeroMQ Until now we have covered what a message queue is, which brings us to the purpose of this article, that is, ZeroMQ. The community identifies ZeroMQ as "sockets on steroids". The formal definition of ZeroMQ is it is a messaging library that helps developers to design distributed and concurrent applications. The first thing we need to know about ZeroMQ is that it is not a traditional message queuing system, such as ActiveMQ, WebSphereMQ, or RabbitMQ. ZeroMQ is different. It gives us the tools to build our own message queuing system. It is a library. It runs on different architectures from ARM to Itanium, and has support for more than 20 programming languages. Simplicity ZeroMQ is simple. We can do some asynchronous I/O operations and ZeroMQ could queue the message in an I/O thread. ZeroMQ I/O threads are asynchronous when handling network traffic, so it can do the rest of the job for us. If you have worked on sockets before, you will know that it is quite painful to work on. However, ZeroMQ makes it easy to work on sockets. Performance ZeroMQ is fast. The website Second Life managed to get 13.4 microseconds end-to-end latencies and up to 4,100,000 messages per second. ZeroMQ can use multicast transport protocol, which is an efficient method to transmit data to multiple destinations. The brokerless design Unlike other traditional message queuing systems, ZeroMQ is brokerless. In traditional message queuing systems, there is a central message server (broker) in the middle of the network and every node is connected to this central node, and each node communicates with other nodes via the central broker. They do not directly communicate with each other. However, ZeroMQ is brokerless. In a brokerless design, applications can directly communicate with each other without any broker in the middle. ZeroMQ does not store messages on disk. Please do not even think about it. However, it is possible to use a local swap file to store messages if you set zmq.SWAP. Summary This article explained what a message queuing system is, discussed the importance of message queuing, and introduced ZeroMQ to the reader. Resources for Article : Further resources on this subject: RESTful Web Service Implementation with RESTEasy [Article] BizTalk Server: Standard Message Exchange Patterns and Types of Service [Article] AJAX Chat Implementation: Part 1 [Article]