How-To Tutorials

The path to mastering performance in Python has just started. Profiling only takes us half way there. Measuring how our program is using the resources at its disposal only tells us where the problem is, not how to fix it. In this article by Fernando Doglio, author of the book Mastering Python High Performance, we will cover the process of optimization, and to do that, we need to start with the basics. We'll keep it inside the language for now: no external tools, just Python and the right way to use it. We will cover the following topics in this article: Memoization / lookup tables Usage of default arguments (For more resources related to this topic, see here.) Memoization / lookup tables This is one of the most common techniques used to improve the performance of a piece of code (namely a function). We can save the results of expensive function calls associated to a specific set of input values and return the saved result (instead of redoing the whole computation) when the function is called with the remembered input. It might be confused with caching, since it is one case of it, although this term refers also, to other types of optimization (such as HTTP caching, buffering, and so on) This methodology is very powerful, because in practice, it'll turn what should have been a potentially very expensive call into a O(1) function call if the implementation is right. Normally, the parameters are used to create a unique key, which is then used on a dictionary to either save the result or obtain it if it's been already saved. There is, of course, a trade-off to this technique. If we're going to be remembering the returned values of a memoized function, then we'll be exchanging memory space for speed. This is a very acceptable trade-off, unless the saved data becomes more than what the system can handle. Classic use cases for this optimization are function calls that repeat the input parameters often. This will assure that most of the times, the memoized results are returned. If there are many function calls but with different parameters, we'll only store results and spend our memory without any real benefit, as seen in the following diagram: You can clearly see how the blue bar (Fixed params, memoized) is clearly the fastest use case, while the others are all similar due to their nature. Here is the code that generates values for the preceding chart. To generate some sort of time-consuming function, the code will call either the twoParams function or the twoParamsMemoized function several hundred times under different conditions, and it will log the execution time: import math import time import random class Memoized: def __init__(self, fn): self.fn = fn self.results = {} def __call__(self, *args): key = ''.join(map(str, args[0])) try: return self.results[key] except KeyError: self.results[key] = self.fn(*args) return self.results[key] @Memoized def twoParamsMemoized(values, period): totalSum = 0 for x in range(0, 100): for v in values: totalSum = math.pow((math.sqrt(v) * period), 4) + totalSum return totalSum def twoParams(values, period): totalSum = 0 for x in range(0, 100): for v in values: totalSum = math.pow((math.sqrt(v) * period), 4) + totalSum return totalSum def performTest(): valuesList = [] for i in range(0, 10): valuesList.append(random.sample(xrange(1, 101), 10)) start_time = time.clock() for x in range(0, 10): for values in valuesList: twoParamsMemoized(values, random.random()) end_time = time.clock() - start_time print "Fixed params, memoized: %s" % (end_time) start_time = time.clock() for x in range(0, 10): for values in valuesList: twoParams(values, random.random()) end_time = time.clock() - start_time print "Fixed params, without memoizing: %s" % (end_time) start_time = time.clock() for x in range(0, 10): for values in valuesList: twoParamsMemoized(random.sample(xrange(1,2000), 10), random.random()) end_time = time.clock() - start_time print "Random params, memoized: %s" % (end_time) start_time = time.clock() for x in range(0, 10): for values in valuesList: twoParams(random.sample(xrange(1,2000), 10), random.random()) end_time = time.clock() - start_time print "Random params, without memoizing: %s" % (end_time) performTest() The main insight to take from the preceding chart is that just like with every aspect of programming, there is no silver bullet algorithm that will work for all cases. Memoization is clearly a very basic way of optimizing code, but clearly, it won't optimize anything given the right circumstances. As for the code, there is not much to it. It is a very simple, non real-world example of the point I was trying to send across. The performTest function will take care of running a series of 10 tests for every use case and measure the total time each use case takes. Notice that we're not really using profilers at this point. We're just measuring time in a very basic and ad-hoc way, which works for us. The input for both functions is simply a set of numbers on which they will run some math functions, just for the sake of doing something. The other interesting bit about the arguments is that, since the first argument is a list of numbers, we can't just use the args parameter as a key inside the Memoized class' methods. This is why we have the following line: key = ''.join(map(str, args[0])) This line will concatenate all the numbers from the first parameter into a single value, which will act as the key. The second parameter is not used here, because it's always random, which would imply that the key would never be the same. Another variation of the preceding method is to pre-calculate all values from the function (assuming we have a limited number of inputs of course) during initialization and then refer to the lookup table during execution. This approach has several preconditions: The number of input values must be finite; otherwise it's impossible to precalculate everything The lookup table with all of its values must fit into memory Just like before, the input must be repeated, at least once, so the optimization both makes sense and is worth the extra effort There are different approaches when it comes to architecting the lookup table, all offering different types of optimizations. It all depends on the type of application and solution that you're trying to optimize. Here is a set of examples. Lookup on a list or linked list This solution works by iterating over an unsorted list and checking the key against each element, with the associated value as the result we're looking for. This is obviously a very slow method of implementation, with a big O notation of O(n) for both the average and worst case scenarios. Still, given the right circumstances, it could prove to be faster than calling the actual function every time. In this case, using a linked list would improve the performance of the algorithm over using a simple list. However, it would still depend heavily on the type of linked list it is (doubly linked list, simple linked list with direct access to the first and last elements, and so on). Simple lookup on a dictionary This method works using a one-dimensional dictionary lookup, indexed by a key consisting of the input parameters (enough of them create a unique key). In particular cases (like we covered earlier), this is probably one of the fastest lookups, even faster than binary search in some cases with a constant execution time (big O notation of O(1)). Note that this approach is efficient as long as the key-generation algorithm is capable of generating unique keys every time. Otherwise, the performance could degrade over time due to the many collisions on the dictionaries. Binary search This particular method is only possible if the list is sorted. This could potentially be an option depending on the values to sort. Yet, sorting them would require an extra effort that would hurt the performance of the entire effort. However, it presents very good results even in long lists (average big O notation of O(log n)). It works by determining in which half of the list the value is and repeating until either the value is found or the algorithm is able to determine that the value is not in the list. To put all of this into perspective, looking at the Memoized class mentioned earlier, it implements a simple lookup on a dictionary. However, this would be the place to implement either of the other algorithms. Use cases for lookup tables There are some classic example use cases for this type of optimization, but the most common one is probably the optimization of trigonometric functions. Based on the computing time, these functions are really slow. When used repeatedly, they can cause some serious damage to your program's performance. This is why it is normally recommended to precalculate the values of these functions. For functions that deal with an infinite domain universe of possible input values, this task becomes impossible. So, the developer is forced to sacrifice accuracy for performance by precalculating a discrete subdomain of the possible input values (that is, going from floating points down to integer numbers). This approach might not be ideal in some cases, since some systems require both performance and accuracy. So, the solution is to meet in the middle and use some form of interpolation to calculate the wanted value, based on the ones that have been precalculated. It will provide better accuracy. Even though it won't be as performant as using the lookup table directly, it should prove to be faster than doing the trigonometric calculation every time. Let's look at some examples of this, for instance, for the following trigonometric function: def complexTrigFunction(x): return math.sin(x) * math.cos(x)**2 We'll take a look at how simple precalculation won't be accurate enough and how some form of interpolation will result in a better level of accuracy. The following code will precalculate the values for the function on a range from -1000 to 1000 (only integer values). Then, it'll try to do the same calculation (only for a smaller range) for floating point numbers: import math import time from collections import defaultdict import itertools trig_lookup_table = defaultdict(lambda: 0) def drange(start, stop, step): assert(step != 0) sample_count = math.fabs((stop - start) / step) return itertools.islice(itertools.count(start, step), sample_count) def complexTrigFunction(x): return math.sin(x) * math.cos(x)**2 def lookUpTrig(x): return trig_lookup_table[int(x)] for x in range(-1000, 1000): trig_lookup_table[x] = complexTrigFunction(x) trig_results = [] lookup_results = [] init_time = time.clock() for x in drange(-100, 100, 0.1): trig_results.append(complexTrigFunction(x)) print "Trig results: %s" % (time.clock() - init_time) init_time = time.clock() for x in drange(-100, 100, 0.1): lookup_results.append(lookUpTrig(x)) print "Lookup results: %s" % (time.clock() - init_time) for idx in range(0, 200): print "%st%s" % (trig_results [idx], lookup_results[idx]) The results from the preceding code will help demonstrate how the simple lookup table approach is not accurate enough (see the following chart). However, it compensates for it on speed since the original function takes 0.001526 seconds to run while the lookup table only takes 0.000717 seconds. The preceding chart shows how the lack of interpolation hurts the accuracy. You can see how even though both plots are quite similar, the results from the lookup table execution aren't as accurate as the trig function used directly. So, now, let's take another look at the same problem. However, this time, we'll add some basic interpolation (we'll limit the rage of values from -PI to PI): import math import time from collections import defaultdict import itertools trig_lookup_table = defaultdict(lambda: 0) def drange(start, stop, step): assert(step != 0) sample_count = math.fabs((stop - start) / step) return itertools.islice(itertools.count(start, step), sample_count) def complexTrigFunction(x): return math.sin(x) * math.cos(x)**2 reverse_indexes = {} for x in range(-1000, 1000): trig_lookup_table[x] = complexTrigFunction(math.pi * x / 1000) complex_results = [] lookup_results = [] init_time = time.clock() for x in drange(-10, 10, 0.1): complex_results .append(complexTrigFunction(x)) print "Complex trig function: %s" % (time.clock() - init_time) init_time = time.clock() factor = 1000 / math.pi for x in drange(-10 * factor, 10 * factor, 0.1 * factor): lookup_results.append(trig_lookup_table[int(x)]) print "Lookup results: %s" % (time.clock() - init_time) for idx in range(0, len(lookup_results )): print "%st%s" % (complex_results [idx], lookup_results [idx]) As you might've noticed in the previous chart, the resulting plot is periodic (specially because we've limited the range from -PI to PI). So, we'll focus on a particular range of values that will generate one single segment of the plot. The output of the preceding script also shows that the interpolation solution is still faster than the original trigonometric function, although not as fast as it was earlier: Interpolation Solution Original function 0.000118 seconds 0.000343 seconds The following chart is a bit different from the previous one, especially because it shows (in green) the error percentage between the interpolated value and the original one: The biggest error we have is around 12 percent (which represents the peaks we see on the chart). However, it's for the smallest values, such as -0.000852248551417 versus -0.000798905501416. This is a case where the error percentage needs to be contextualized to see if it really matters. In our case, since the values related to that error are so small, we can ignore that error in practice. There are other use cases for lookup tables, such as in image processing. However, for the sake of this article, the preceding example should be enough to demonstrate their benefits and trade-off implied in their usage. Usage of default arguments Another optimization technique, one that is contrary to memoization, is not particularly generic. Instead, it is directly tied to how the Python interpreter works. Default arguments can be used to determine values once at function creation time instead of at run time. This can only be done for functions or objects that will not be changed during program execution. Let's look at an example of how this optimization can be applied. The following code shows two versions of the same function, which does some random trigonometric calculation: import math #original function def degree_sin(deg): return math.sin(deg * math.pi / 180.0) #optimized function, the factor variable is calculated during function creation time, #and so is the lookup of the math.sin method. def degree_sin(deg, factor=math.pi/180.0, sin=math.sin): return sin(deg * factor) This optimization can be problematic if not correctly documented. Since it uses attributes to precompute terms that should not change during the program's execution, it could lead to the creation of a confusing API. With a quick and simple test, we can double-check the performance gain from this optimization: import time import math def degree_sin(deg): return math.sin(deg * math.pi / 180.0) * math.cos(deg * math.pi / 180.0) def degree_sin_opt(deg, factor=math.pi/180.0, sin=math.sin, cos = math.cos): return sin(deg * factor) * cos(deg * factor) normal_times = [] optimized_times = [] for y in range(100): init = time.clock() for x in range(1000): degree_sin(x) normal_times.append(time.clock() - init) init = time.clock() for x in range(1000): degree_sin_opt(x) optimized_times.append(time.clock() - init) print "Normal function: %s" % (reduce(lambda x, y: x + y, normal_times, 0) / 100) print "Optimized function: %s" % (reduce(lambda x, y: x + y, optimized_times, 0 ) / 100) The preceding code measures the time it takes for the script to finish each of the versions of the function to run its code 1000 times. It saves those measurements, and finally, it does an average for each case. The result is displayed in the following chart: It clearly isn't an amazing optimization. However, it does shave off some microseconds from our execution time, so we'll keep it in mind. Just remember that this optimization could cause problems if you're working as part of an OS developer team. Summary In this article, we covered several optimization techniques. Some of them are meant to provide big boosts on speed, save memory. Some of them are just meant to provide minor speed improvements. Most of this article covered Python-specific techniques, but some of them can be translated into other languages as well. Resources for Article: Further resources on this subject: How to do Machine Learning with Python [article] The Essentials of Working with Python Collections [article] Symbolizers [article]

This virtual reality thing calls into question, what does it mean to "be somewhere"? Before cell phones, you would call someone and it would make no sense to say, "Hey, where are you?" You know where they are, you called their house, that's where they are. So then cell phones come around and you start to hear people say, "Hello. Oh, I'm at Starbucks," because the person on the other end wouldn't necessarily know where you are, because you became un-tethered from your house for voice communications. So when I saw a VR demo, I had this vision of coming home and my wife has got the kids settled down, she has a couple minutes to herself, and she's on the couch wearing goggles on her face. I come over and tap her on the shoulder, and I'm like, "Hey, where are you?" It's super weird. The person's sitting right in front of you, but you don't know where they are.                                                      -Jonathan Stark, mobile expert and podcaster In this article, by Jonathan Linowes, author of the book Unity Virtual Reality Projects, we will define virtual reality and illustrate how it can be applied not only to games but also many other areas of interest and productivity. Welcome to virtual reality! In this book, we will explore what it takes to create virtual reality experiences on our own. We will take a walk through a series of hands-on projects, step-by-step tutorials, and in-depth discussions using the Unity 5 3D game engine and other free or open source software. Though the virtual reality technology is rapidly advancing, we'll try to capture the basic principles and techniques that you can use to make your VR games and applications feel immersive and comfortable. This article discusses the following topics: What is virtual reality? Differences between virtual reality (VR) and augmented reality (AR) How VR applications may differ from VR games Types of VR experiences Technical skills that are necessary for the development of VR What is virtual reality to you? Today, we are witnesses to the burgeoning consumer virtual reality, an exciting technology that promises to transform in a fundamental way how we interact with information, our friends, and the world at large. What is virtual reality? In general, VR is the computer-generated simulation of a 3D environment, which seems very real to the person experiencing it, using special electronic equipment. The objective is to achieve a strong sense of being present in the virtual environment. Today's consumer tech VR involves wearing a head-mounted display (such as goggles) to view stereoscopic 3D scenes. You can look around by moving your head, and walk around by using hand controls or motion sensors. You are engaged in a fully immersive experience. It's as if you're really there in some other virtual world. The following image shows a guy experiencing an Oculus Rift Development Kit 2 (DK2): Virtual reality is not new. It's been here for decades, albeit hidden away in academic research labs and high-end industrial and military facilities. It was big, clunky, and expensive. Ivan Sutherland invented the first head-mounted display in 1966, which is shown in the following image. It was tethered to the ceiling! In the past, several failed attempts have been made to bring consumer-level virtual reality products to the market. In 2012, Palmer Luckey, the founder of Oculus VR LLC, gave a demonstration of a makeshift head-mounted VR display to John Carmack, the famed developer of Doom, Wolfenstein 3D, and Quake classic video games. Together, they ran a successful Kickstarter campaign and released a developer kit called Oculus Rift Development Kit 1 (DK1) to an enthusiastic community. This caught the attention of investors as well as Mark Zuckerberg, and in March 2014, Facebook bought the company for $2 billion. With no product, no customers, and an infinite promise, the money and attention that it attracted has helped fuel a new category of consumer products. Others have followed suit, including Google, Sony, Samsung, and Steam. New innovations and devices that enhance the VR experience continue to be introduced. Most of the basic research has already been done and the technology is now affordable thanks in large part to the mass adoption of devices that work on mobile technology. There is a huge community of developers with an experience in building 3D games and mobile apps. Creative content producers are joining in and the media is talking it up. At last, virtual reality is real! Say what? Virtual reality is real? Ha! If it's virtual, how can it be... Oh, never mind. Eventually, we will get past the focus on the emerging hardware devices and recognize that content is king. The current generation of 3D development software (commercial, free, and open source) that has spawned a plethora of indie, or independent, game developers can also be used to build non-game VR applications. Though VR finds most of its enthusiasts in the gaming community, the potential applications reach well beyond that. Any business that presently uses 3D modeling and computer graphics will be more effective if it uses the VR technology. The sense of immersive presence that is afforded by VR can enhance all common online experiences today, which includes engineering, social networking, shopping, marketing, entertainment, and business development. In the near future, viewing 3D websites with a VR headset may be as common as visiting ordinary flat websites today. Types of head-mounted displays Presently, there are two basic categories of head-mounted displays for virtual reality—desktop VR and mobile VR. Desktop VR With desktop VR (and console VR), your headset is a peripheral to a more powerful computer that processes the heavy graphics. The computer may be a Windows PC, Mac, Linux, or a game console. Most likely, the headset is connected to the computer with wires. The game runs on the remote machine and the head-mounted display (HMD) is a peripheral display device with a motion sensing input. The term desktop is an unfortunate misnomer since it's just as likely to be stationed in either a living room or a den. The Oculus Rift (https://www.oculus.com/) is an example of a device where the goggles have an integrated display and sensors. The games run on a separate PC. Other desktop headsets include HTC/Valve Vive and Sony's project Morpheus for PlayStation. The Oculus Rift is tethered to a desktop computer via video and USB cables, and generally, the more graphics processing unit (GPU) power, the better. However, for the purpose of this book, we won't have any heavy rendering in our projects, and you can get by even with a laptop (provided it has two USB ports and one HDMI port available). Mobile VR Mobile VR, exemplified by Google Cardboard (http://www.google.com/get/cardboard/), is a simple housing (device) for two lenses and a slot for your mobile phone. The phone's display is used to show the twin stereographic views. It has rotational head tracking, but it has no positional tracking. Cardboard also provides the user with the ability to click or tap its side to make selections in a game. The complexity of the imagery is limited because it uses your phone's processor for rendering the views on the phone display screen. Other mobile VR headsets include Samsung Gear VR and Zeiss VR One, among others. Google provides the open source specifications, and other manufacturers have developed ready-made models for purchase, with prices for the same as low as $15. If you want to find one, just Google it! There are versions of Cardboard-compatible headsets that are available for all sizes of phones—both Android and iOS. Although the quality of the VR experience with a Cardboard device is limited (some even say that it is inadequate) and it's probably a "starter" device that will just be quaint in a couple of years, Cardboard is fine for the small projects in this book, and we'll revisit its limitations from time to time. The difference between virtual reality and augmented reality It's probably worthwhile clarifying what virtual reality is not. A sister technology to VR is augmented reality (AR), which superimposes computer generated imagery (CGI) over views of the real world. Limited uses of AR can be found on smart phones, tablets, handheld gaming systems such as the Nintendo 3DS, and even in some science museum exhibits, which overlay the CGI on top of live video from a camera. The latest innovations in AR are the AR headsets, such as Microsoft HoloLens and Magic Leap, which show the computer graphics directly in your field of view; the graphics are not mixed into a video image. If the VR headsets are like closed goggles, the AR headsets are like translucent sunglasses that employ a technology called light fields to combine the real-world light rays with CGI. A challenge for AR is ensuring that the CGI is consistently aligned with and mapped onto the objects in the real-world space and eliminate latency while moving about so that they (the CGI and objects in real-world space) stay aligned. AR holds as much promise as VR for future applications, but it's different. Though AR intends to engage the user within their current surroundings, virtual reality is fully immersive. In AR, you may open your hand and see a log cabin resting in your palm, but in VR, you're transported directly inside the log cabin and you can walk around inside it. We can also expect to see hybrid devices that somehow either combine VR and AR, or let you switch between modes. Applications versus games The consumer-level virtual reality starts with gaming. Video gamers are already accustomed to being engaged in highly interactive hyper-realistic 3D environments. VR just ups the ante. Gamers are early adopters of high-end graphics technology. Mass production of gaming consoles and PC-based components in the tens of millions and competition between vendors leads to lower prices and higher performance. Game developers follow suit, often pushing the state-of-the-art, squeezing every ounce of performance out of hardware and software. Gamers are a very demanding bunch, and the market has consistently stepped up to keep them satisfied. It's no surprise that many, if not most, of the current wave of the VR hardware and software companies are first targeting the video gaming industry. A majority of the demos and downloads that are available on Oculus Share (https://share.oculus.com/) and Google Play for the Cardboard app (https://play.google.com/store/search?q=cardboard&c=apps) are games. Gamers are the most enthusiastic VR advocates and seriously appreciate its potential. Game developers know that the core of a game is the game mechanics, or the rules, which are largely independent of the skin, or the thematic topic of the game. Gameplay mechanics can include puzzles, chance, strategy, timing, or muscle memory (twitch). VR games can have the same mechanic elements but might need to be adjusted for the virtual environment. For example, a first-person character walking in a console video game is probably going about 1.5 times faster than their actual pace in real life. If this wasn't the case, the player would feel that the game is too slow and boring. Put the same character in a VR scene and they will feel that it is too fast; it could likely make the player feel nauseous. In VR, you will want your characters to walk a normal, earthly pace. Not all video games will map well to VR; it may not be fun to be in the middle of a war zone when you're actually there. That said, virtual reality is also being applied in areas other than gaming. Though games will remain important, non-gaming apps will eventually overshadow them. These applications may differ from games in a number of ways, with the most significant having much less emphasis on game mechanics and more emphasis on either the experience itself or application-specific goals. Of course, this doesn't preclude some game mechanics. For example, the application may be specifically designed to train the user at a specific skill. Sometimes, the gamification of a business or personal application makes it more fun and effective in driving the desired behavior through competition. In general, non-gaming VR applications are less about winning and more about the experience itself. Here are a few examples of the kinds of non-gaming applications that people are working on: Travel and tourism: Visit faraway places without leaving your home. Visit art museums in Paris, New York, and Tokyo in one afternoon. Take a walk on Mars. You can even enjoy Holi, the spring festival of colors, in India while sitting in your wintery cabin in Vermont. Mechanical engineering and industrial design: Computer-aided design software such as AutoCAD and SOLIDWORKS pioneered three-dimensional modeling, simulation, and visualization. With VR, engineers and designers can directly experience the hands-on end product before it's actually built and play with what-if scenarios at a very low cost. Consider iterating a new automobile design. How does it look? How does it perform? How does it appear sitting in the driver's seat? Architecture and civil engineering: Architects and engineers have always constructed scale models of their designs, if only to pitch the ideas to clients and investors, or more importantly, to validate the many assumptions about the design. Presently, modeling and rendering software is commonly used to build virtual models from architectural plans. With VR, the conversation with stakeholders can be so much more confident. Other personnel, such as the interior designers, HVAC, and electrical engineers, can be brought into the process sooner. Real estate: Real estate agents have been quick adopters of the Internet and visualization technology to attract buyers and close sales. Real estate search websites were some of the first successful uses of the Web. Online panoramic video walk-throughs of for-sale properties are commonplace today. With VR, I can be in New York and find a place to live in Los Angeles. This will become even easier with mobile 3D-sensing technologies such as Google Project Tango (https://www.google.com/atap/projecttango), which performs a 3D scan of a room using a smartphone and automatically builds a model of the space. Medicine: The potential of VR for health and medicine may literally be a matter of life and death. Every day, hospitals use MRI and other scanning devices to produce models of our bones and organs that are used for medical diagnosis and possibly pre-operative planning. Using VR to enhance visualization and measurement will provide a more intuitive analysis. Virtual reality is also being used for the simulation of surgery to train medical students. Mental health: Virtual reality experiences have been shown to be effective in a therapeutic context for the treatment of post traumatic stress disorder (PTSD) in what's called exposure therapy, where the patient, guided by a trained therapist, confronts their traumatic memories through the retelling of the experience. Similarly, VR is being used to treat arachnophobia (spiders) and the fear of flying. Education: The educational opportunities for VR are almost too obvious to mention. One of the first successful VR experiences is Titans of Space, which lets you explore the solar system first hand. Science, history, arts, and mathematics—VR will help students of all ages because, as they say, field trips are much more effective than textbooks. Training: Toyota has demonstrated a VR simulation of drivers' education to teach teenagers about the risks of distracted driving. In another project, vocational students got to experience the operating of cranes and other heavy construction equipment. Training for first responders, police, and the fire and rescue workers can be enhanced with VR by presenting highly risky situations and alternative virtual scenarios. The NFL is looking to VR for athletic training. Entertainment and journalism: Virtually attend rock concerts and sporting events. Watch music videos Erotica. Re-experience news events as if you were personally present. Enjoy 360-degree cinematic experiences. The art of storytelling will be transformed by virtual reality. Wow, that's quite a list! This is just the low-hanging fruit. The purpose of this book is not to dive too deeply into any of these applications. Rather, I hope that this survey helps stimulate your thinking and provides a perspective towards how virtual reality has the potential to be virtually anything for everyone. What this book covers This book takes a practical, project-based approach to teach the specifics of virtual reality development using the Unity 3D game development engine. You'll learn how to use Unity 5 to develop VR applications, which can be experienced with devices such as the Oculus Rift or Google Cardboard. However, we have a slight problem here—the technology is advancing very rapidly. Of course, this is a good problem to have. Actually, it's an awesome problem to have, unless you're a developer in the middle of a project or an author of a book on this technology! How does one write a book that does not have obsolete content the day it's published? Throughout the book, I have tried to distill some universal principles that should outlive any near-term advances in virtual reality technology, that includes the following: Categorization of different types of VR experiences with example projects Important technical ideas and skills, especially the ones relevant to the building of VR applications General explanations on how VR devices and software works Strategies to ensure user comfort and avoid VR motion sickness Instructions on using the Unity game engine to build VR experiences Once VR becomes mainstream, many of these lessons will perhaps be obvious rather than obsolete, just like the explanations from the 1980's on how to use a mouse would just be silly today. Who are you? If you are interested in virtual reality, want to learn how it works, or want to create VR experiences yourself, this book is for you. We will walk you through a series of hands-on projects, step-by-step tutorials, and in-depth discussions using the Unity 3D game engine. Whether you're a non-programmer who is unfamiliar with 3D computer graphics, or a person with experience in both but new to virtual reality, you will benefit from this book. It is not a cold start with Unity, but you do not need to be an expert either. Still, if you're new to Unity, you can pick up this book as long as you realize that you'll need to adapt to the pace of the book. Game developers may already be familiar with the concepts in the book, which are reapplied to the VR projects while learning many other ideas that are specific to VR. Engineers and 3D designers may understand many of the 3D concepts, but they may wish to learn to use the game engine for VR. Application developers may appreciate the potential non-gaming uses of VR and want to learn the tools that can make this happen. Whoever you are, we're going to turn you into a 3D Software VR Ninja. Well, OK, this may be a stretch goal for this little book, but we'll try to set you on the way. Types of VR experiences There is not just one kind of virtual reality experience. In fact, there are many. Consider the following types of virtual reality experiences: Diorama: In the simplest case, we build a 3D scene. You're observing from a third-person perspective. Your eye is the camera. Actually, each eye is a separate camera that gives you a stereographic view. You can look around. First-person experience: This time, you're immersed in the scene as a freely moving avatar. Using an input controller (keyboard, game controller, or some other technique), you can walk around and explore the virtual scene. Interactive virtual environment: This is like the first-person experience, but it has an additional feature—while you are in the scene, you can interact with the objects in it. Physics is at play. Objects may respond to you. You may be given specific goals to achieve and challenges with the game mechanics. You might even earn points and keep score. Riding on rails: In this kind of experience, you're seated and being transported through the environment (or, the environment changes around you). For example, you can ride a roller coaster via this virtual reality experience. However, it may not necessarily be an extreme thrill ride. It can be a simple real estate walk-through or even a slow, easy, and meditative experience. 360-degree media: Think panoramic images taken with GoPro® on steroids that are projected on the inside of a sphere. You're positioned at the center of the sphere and can look all around. Some purists don't consider this "real" virtual reality, because you're seeing a projection and not a model rendering. However, it can provide an effective sense of presence. Social VR: When multiple players enter the same VR space and can see and speak with each other's avatars, it becomes a remarkable social experience. In this book, we will implement a number of projects that demonstrate how to build each of these types of VR experience. For brevity, we'll need to keep it pure and simple, with suggestions for areas for further investigation. Technical skills that are important to VR You will learn about the following in this book: World scale: When building for a VR experience, attention to the 3D space and scale is important. One unit in Unity is usually equal to one meter in the virtual world. First-person controls: There are various techniques that can be used to control the movement of your avatar (first-person camera), such as the keyboard keys, game controllers, and head movements. User interfacecontrols: Unlike conventional video (and mobile) games, all user interface components are in world coordinates in VR, not screen coordinates. We'll explore ways to present notices, buttons, selectors, and other User interface (UI) controls to the users so that they can interact and make selections. Physics and gravity: Critical to the sense of presence and immersion in VR is the physics and gravity of the world. We'll use the Unity physics engine to our advantage. Animations: Moving objects within the scene is called "animation" (duh!) It can either be along predefined paths, or it may use AI (artificial intelligence) scripting that follows a logical algorithm in response to events in the environment. Multiuser services: Real-time networking and multiuser games are not easy to implement, but online services make it easy without you having to be a computer engineer. Build and run: Different HMDs use different developer kits (SDK) and assets to build applications that target a specific devise. We'll consider techniques that let you use a single interface for multiple devices. We will write scripts in the C# language and use features of Unity as and when they are needed to get things done. However, there are technical areas that we will not cover, such as realistic rendering, shaders, materials, and lighting. We will not go into modeling techniques, terrains, or humanoid animations. Effective use of advanced input devices and hand and body tracking is proving to be critical to VR, but we won't have a chance to get into it here either. We also won't discuss game mechanics, dynamics, and strategies. We will talk about rendering performance optimization, but not in depth. All of these are very important topics that may be necessary for you to learn (or for someone in your team), in addition to this book, to build complete, successful, and immersive VR applications. Summary In this article, we looked at virtual reality and realized that it can mean a lot of things to different people and can have different applications. There's no single definition, and it's a moving target. We are not alone, as everyone's still trying to figure it out. The fact is that virtual reality is a new medium that will take years, if not decades, to reach its potential. VR is not just for games; it can be a game changer for many different applications. We identified over a dozen. There are different kinds of VR experiences, which we'll explore in the projects in this book. VR headsets can be divided into those that require a separate processing unit (such as a desktop PC or a console) that runs with a powerful GPU and the ones that use your mobile phone for processing. In this book, we will use an Oculus Rift DK2 as an example of desktop VR and Google Cardboard as the example of mobile VR, although there are many alternative and new devices available. We're all pioneers living at an exciting time. Because you're reading this book, you're one, too. Whatever happens next is literally up to you. As the personal computing pioneer Alan Kay said, "The best way to predict the future is to invent it." So, let's get to it! Resources for Article: Further resources on this subject: Looking Back, Looking Forward [article] Unity Networking – The Pong Game [article] Getting Started with Mudbox 2013 [article]

In this article by Angelo Tadres, author of the book Extending Unity with Editor Scripting, we will explore a way to create a custom GUI for our properties using Property Drawers. If you've worked on a Unity project for a long time, you know that the bigger your scripts get, the more unwieldy they become. All your public variables take up space in the Inspector Window, and as they accumulate, they begin to convert into one giant and scary monster. Sometimes, organization is the clue. So, in this article, you will learn how to improve your inspectors using Property and Decorator Drawers. (For more resources related to this topic, see here.) A Property Drawer is an attribute that allows you to control how the GUI of a Serializable class or property is displayed in the inspector window. An Attribute is a C# way of defining declarative tags, which you can place on certain entities in your source code to specify additional information. The information that attributes contain is retrieved at runtime through reflection. This approach significantly reduces the amount of work you have to do for the GUI customization because you don't need to write an entire Custom Inspector; instead, you can just apply appropriate attributes to variables in your scripts to tell the editor how you want those properties to be drawn. Unity has several Property Drawers implemented by default. Let's take a look at one of them called Range: using UnityEngine; public class RangeDrawerDemo : MonoBehaviour { [Range (0, 100)] public int IntValue = 50; } If you attach this script to a game object and then select it, you will see the following: Using the Range attribute, we rendered a slider that moves between 0 and 100 instead of the common int field. This is valuable in terms of validating the input for this field. Avoid possible mistakes such as using a negative value to define the radius of a sphere collider, and so on. Let's take a look to the rest of the built-in Property Drawers. Built-in Property Drawers The Unity documentation has information about the built-in property drawers, but there is no such place where you can check all the available ones listed. In this section, we will resolve this. Range The Range attribute restricts a float or int variable in a script to a specific range. When this attribute is used, the float or int will be shown as a slider in the inspector instead of the default number field: public RangeAttribute(float min, float max); public RangeAttribute(float min, float max); [Range (0, 1)] public float FloatRange = 0.5f; [Range (0, 100)] public int IntRange = 50; You will get the following output: Multiline The Multiline attribute is used to show a string value in a multiline text area. You can decide the number of lines of text to make room for. The default is 3 and the text doesn't wrap. The following is an example of the Multiline attribute: public MultilineAttribute(); public MultilineAttribute(int lines); [Multiline (2)] public string StringMultiline = "This text is using a multiline property drawer"; You will get the following output: TextArea The TextArea attribute allows a string to be edited with a height-flexible and scrollable text area. You can specify the minimum and maximum values; a scrollbar will appear if the text is bigger than the area available. Its behavior is better compared to Multiline. The following is an example of the TextArea attribute: public TextAreaAttribute(); public TextAreaAttribute(int minLines, int maxLines); [TextArea (2, 4)] public string StringTextArea = "This text is using a textarea property drawer"; You will get the following output: ContextMenu The ContextMenu attribute adds the method to the context menu of the component. When the user selects the context menu, the method will be executed. The method has to be nonstatic. In the following example, we call the method DoSomething, printing a log in the console: public ContextMenu(string name); [ContextMenu ("Do Something")] public void DoSomething() { Debug.Log ("DoSomething called..."); } You will get the following output: ContextMenuItem The ContextMenuItem attribute is used to add a context menu to a field that calls a named method. In the following example, we call a method to reset the value of the IntReset variable to 0: public ContextMenuItemAttribute(string name, string function); [ContextMenuItem("Reset this value", "Reset")] public int IntReset = 100; public void Reset() { IntReset = 0; } You will get the following output: Built-in Decorator Drawers There is another kind of drawer called Decorator Drawer. These drawers are similar in composition to the Property Drawers, but the main difference is that Decorator Drawers are designed to draw decoration in the inspector and are unassociated with a specific field. This means, while you can only declare one Property Drawer per variable, you can stack multiple decorator drawers in the same field. Let's take a look in the following built-in Decorator Drawers. Header This is the attribute that adds a header to some fields in the inspector: public HeaderAttribute(string header); [Header("This is a group of variables")] public int VarA = 10; public int VarB = 20; You will get the following output: Space The space attribute adds some spacing in the inspector: public SpaceAttribute(float height); public int VarC = 10; [Space(40)] public int VarD = 20; You will get the following output: Tooltip This Tooltip attribute specifies a tooltip for a field: public TooltipAttribute(string tooltip); [Tooltip("This is a tooltip")] public int VarE = 30; You will get the following output: Creating you own Property Drawers If you have a serializable parameter or structure that repeats constantly in your video game and you would like to improve how this renders in the Inspector, you can try to write your own Property Drawer. We will create a property drawer for an integer meant to be a variable to save time in seconds. This Property Drawer will draw a normal int field but also a label with the number of seconds converted to the m:s or h:m:s time format. To implement a Property Drawer, you must create two scripts: The attribute: This declares the attribute and makes it usable in your MonoBehaviour scripts. This will be part of your video game scripts. The drawer: This is responsible for rendering the custom GUI and handling the input of the user. This is placed inside a folder called Editor. The Editor folder is one of the several special folders Unity has. All scripts inside this folder will be treated as editor scripts rather than runtime scripts. For the first script, create one file inside your Unity project called TimeAttribute.cs and then add the following code: using UnityEngine; public class TimeAttribute : PropertyAttribute { public readonly bool DisplayHours; public TimeAttribute (bool displayHours = false) { DisplayHours = displayHours; } } Here we defined the name of the attribute and its parameters. You must create your attribute class extending from the PropertyAttribute class. The name of the class contains the suffix "attribute"; however, when you want to use the attribute over a certain property, the suffix is not needed. In this case, we will use Time and not TimeAttribute to use the property drawer. The TimeAttribute has an optional parameter called DisplayHours. The idea is to display a label under the int field with the time in m:s format by default; if the DisplayHours parameter is true, this will be displayed in h:m:s format. Now is the moment to implement the drawer. To do this, let's create a new script called TimeDrawer.cs inside an Editor folder: using UnityEngine; using UnityEditor; [CustomPropertyDrawer (typeof(TimeAttribute))] public class TimeDrawer : PropertyDrawer { public override float GetPropertyHeight (SerializedProperty property, GUIContent label) { return EditorGUI.GetPropertyHeight (property) * 2; } public override void OnGUI (Rect position, SerializedProperty property, GUIContent label) { if (property.propertyType == SerializedPropertyType.Integer) { property.intValue = EditorGUI.IntField (new Rect (position.x, position.y, position.width, position.height / 2), label, Mathf.Abs(property.intValue)); EditorGUI.LabelField (new Rect (position.x, position.y + position.height / 2, position.width, position.height / 2), " ", TimeFormat (property.intValue)); } else { EditorGUI.LabelField (position, label.text, "Use Time with an int."); } } private string TimeFormat (int seconds) { TimeAttribute time = attribute as TimeAttribute; if (time.DisplayHours) { return string.Format ("{0}:{1}:{2} (h:m:s)", seconds / (60 * 60), ((seconds % (60 * 60)) / 60).ToString ().PadLeft(2,'0'), (seconds % 60).ToString ().PadLeft(2,'0')); } else { return string.Format ("{0}:{1} (m:s)", (seconds / 60).ToString (), (seconds % 60).ToString ().PadLeft(2,'0')); } } } Property Drawers don't support layouts to create a GUI; for this reason, the class you must use here is EditorGUI instead of EditorGUILayout. Using this class requires a little extra effort; you need to define the Rect function that will contain the GUI element each time you want to use it. The CustomPropertyDrawer attribute is part of the UnityEditor namespace, and this is what Unity uses to bind a drawer with a Property attribute. In this case, we passed the TimeAttribute. Your must extend the TimeAttribute from the PropertyDrawer class, and in this way, you will have access to the core methods to create Property Drawers: GetPropertyHeight: This method is responsible for handling the height of the drawer. You need to overwrite this method in order to use it. In our case, we force the size of the drawer to be double. OnGUI: This is where you place all the code related to rendering the GUI. You can create Decorator Drawers too. You just need to follow the same steps we performed to create a Property Drawer, but instead of extending your Drawer from PropertyDrawer, you need to extend from DecoratorDrawer. Also, you will have access to the variable attribute. This has a reference to the attribute class we created, and with this we can access to their variables. To test our code, create a new script called TimeDrawerDemo.cs and add the following code: using UnityEngine; using UnityEditor; [CustomPropertyDrawer (typeof(TimeAttribute))] public class TimeDrawer : PropertyDrawer { public override float GetPropertyHeight (SerializedProperty property, GUIContent label) { return EditorGUI.GetPropertyHeight (property) * 2; } public override void OnGUI (Rect position, SerializedProperty property, GUIContent label) { if (property.propertyType == SerializedPropertyType.Integer) { property.intValue = EditorGUI.IntField (new Rect (position.x, position.y, position.width, position.height / 2), label, Mathf.Abs(property.intValue)); EditorGUI.LabelField (new Rect (position.x, position.y + position.height / 2, position.width, position.height / 2), " ", TimeFormat (property.intValue)); } else { EditorGUI.LabelField (position, label.text, "Use Time with an int."); } } private string TimeFormat (int seconds) { TimeAttribute time = attribute as TimeAttribute; if (time.DisplayHours) { return string.Format ("{0}:{1}:{2} (h:m:s)", seconds / (60 * 60), ((seconds % (60 * 60)) / 60).ToString ().PadLeft(2,'0'), (seconds % 60).ToString ().PadLeft(2,'0')); } else { return string.Format ("{0}:{1} (m:s)", (seconds / 60).ToString (), (seconds % 60).ToString ().PadLeft(2,'0')); } } } After compiling, attach this script to a game object. You will see this on the inspector: The time property uses the time attribute. We can check the three possible scenarios here: The attribute uses a property that is not an int The attribute has the variable DisplayHours = false The attribute has the variable DisplayHours = true A little change makes it easier to set up this data in our game objects. Summary In this article, we created a custom Property Drawer to be used in properties mean to store time in seconds, and in the process, you learned how these are implemented. We also explored the available built-in Property Drawers and Decorator Drawers in Unity. Applying this knowledge to your projects will enable you to add validation to sensible data in your properties and make your scripts more developer friendly. This will also allow you to have a professional look. Resources for Article: Further resources on this subject: Adding a Graphical User Interface [article] Animation features in Unity 5 [article] Event-driven Programming [article]

In this article written by Diogo Resende, author of the book Node.js High Performance, we will discuss how high performance is hard, and how it depends on many factors. Best performance should be a constant goal for developers. To achieve it, a developer must know the programming language they use and, more importantly, how the language performs under heavy loads, these being disk, memory, network, and processor usage. (For more resources related to this topic, see here.) Developers will make the most out of a language if they know its weaknesses. In a perfect world, since every job is different, a developer should look for the best tool for the job. But this is not feasible and a developer wouldn't be able to know every best tool, so they have to look for the second best tool for every job. A developer will excel if they know few tools but master them. As a metaphor, a hammer is used to drive nails, and you can also use it to break objects apart or forge metals, but you shouldn't use it to drive screws. The same applies to languages and platforms. Some platforms are very good for a lot of jobs but perform really badly at other jobs. This performance can sometimes be mitigated, but at other times, can't be avoided and you should look for better tools. Node.js is not a language; it's actually a platform built on top of V8, Google's open source JavaScript engine. This engine implements ECMAScript, which itself is a simple and very flexible language. I say "simple" because it has no way of accessing the network, accessing the disk, or talking to other processes. It can't even stop execution since it has no kind of exit instruction. This language needs some kind of interface model on top of it to be useful. Node.js does this by exposing a (preferably) nonblocking I/O model using libuv. This nonblocking API allows you to access the filesystem, connect to network services and execute child processes. The API also has two other important elements: buffers and streams. Since JavaScript strings are Unicode friendly, buffers were introduced to help deal with binary data. Streams are used as simple event interfaces to pass data around. Buffers and streams are used all over the API when reading file contents or receiving network packets. A stream is a module, similar to the network module. When loaded, it provides access to some base classes that help create readable, writable, duplex, and transform streams. These can be used to perform all sorts of data manipulation in a simplified and unified format. The buffers module easily becomes your best friend when converting binary data formats to some other format, for example, JSON. Multiple read and write methods help you convert integers and floats, signed or not, big endian or little endian, from 8 bits to 8 bytes long. Most of the platform is designed to be simple, small, and stable. It's designed and ready to create some high-performance applications. Performance analysis Performance is the amount of work completed in a defined period of time and with a set of defined resources. It can be analyzed using one or more metrics that depend on the performance goal. The goal can be low latency, low memory footprint, reduced processor usage, or even reduced power consumption. The act of performance analysis is also called profiling. Profiling is very important for making optimized applications and is achieved by instrumenting either the source or the instance of the application. By instrumenting the source, developers can spot common performance weak spots. By instrumenting an application instance, they can test the application on different environments. This type of instrumentation can also be known by the name benchmarking. Node.js is known for being fast. Actually, it's not that fast; it's just as fast as your resources allow it. What Node.js is best at is not blocking your application because of an I/O task. The perception of performance can be misleading in Node.js applications. In some other languages, when an application task gets blocked—for example, by a disk operation—all other tasks can be affected. In the case of Node.js, this doesn't happen—usually. Some people look at the platform as being single threaded, which isn't true. Your code runs on a thread, but there are a few more threads responsible for I/O operations. Since these operations are extremely slow compared to the processor's performance, they run on a separate thread and signal the platform when they have information for your application. Applications blocking I/O operations perform poorly. Since Node.js doesn't block I/O unless you want it to, other operations can be performed while waiting for I/O. This greatly improves performance. V8 is an open source Google project and is the JavaScript engine behind Node.js. It's responsible for compiling and executing JavaScript, as well as managing your application's memory needs. It is designed with performance in mind. V8 follows several design principles to improve language performance. The engine has a profiler and one of the best and fast garbage collectors that exist, which is one of the keys to its performance. It also does not compile the language into byte code; it compiles it directly into machine code on the first execution. A good background in the development environment will greatly increase the chances of success in developing high-performance applications. It's very important to know how dereferencing works, or why your variables should avoid switching types. Here are other useful tips you would want to follow. You can use a style guide like JSCS and a linter like JSHint to enforce them to for yourself and your team. Here are some of them: Write small functions, as they're more easily optimized Use monomorphic parameters and variables Prefer arrays to manipulate data, as integer-indexed elements are faster Try to have small objects and avoid long prototype chains Avoid cloning objects because big objects will slow the operations Monitoring After an application is put into production mode, performance analysis becomes even more important, as users will be more demanding than you were. Users don't accept anything that takes more than a second, and monitoring the application's behavior over time and over some specific loads will be extremely important, as it will point to you where your platform is failing or will fail next. Yes, your application may fail, and the best you can do is be prepared. Create a backup plan, have fallback hardware, and create service probes. Essentially, anticipate all the scenarios you can think of, and remember that your application will still fail. Here are some of those scenarios and aspects that you should monitor: When in production, application usage is of extreme importance to understand where your application is heading in terms of data size or memory usage. It's important that you carefully define source code probes to monitor metrics—not only performance metrics, such as requests per second or concurrent requests, but also error rate and exception percentage per request served. Your application emits errors and sometimes throws exceptions; it's normal and you shouldn't ignore them. Don't forget the rest of the infrastructure. If your application must perform at high standards, your infrastructure should too. Your server power supply should be uninterruptible and stable, as instability will degrade your hardware faster than it should. Choose your disks wisely, as faster disks are more expensive and usually come in smaller storage sizes. Sometimes, however, this is actually not a bad decision when your application doesn't need that much storage and speed is considered more important. But don't just look at the gigabytes per dollar. Sometimes, it's more important to look at the gigabits per second per dollar. Also, your server temperature and server room should be monitored. High temperatures degrades performance and your hardware has an operation temperature limit. Security, both physical and virtual, is also very important. Everything counts for the standards of high performance, as an application that stops serving its users is not performing at all. Getting high performance Planning is essential in order to achieve the best results possible. High performance is built from the ground up and starts with how you plan and develop. It obviously depends on physical resources, as you can't perform well when you don't have sufficient memory to accomplish your task, but it also depends greatly on how you plan and develop an application. Mastering tools will give much better performance chances than just using them. Setting the bar high from the beginning of development will force the planning to be more prudent. Some bad planning of the database layer can really downgrade performance. Also, cautious planning will cause developers to think more about “use cases and program more consciously. High performance is when you have to think about a new set of resources (processor, memory, storage) because all that you have is exhausted, not just because one resource is. A high-performance application shouldn't need a second server when a little processor is used and the disk is full. In such a case, you just need bigger disks. Applications can't be designed as monolithic these days. An increasing user base enforces a distributed architecture, or at least one that can distribute load by having multiple instances. This is very important to accommodate in the beginning of the planning, as it will be harder to change an application that is already in production. Most common applications will start performing worse over time, not because of deficit of processing power but because of increasing data size on databases and disks. You'll notice that the importance of memory increases and fallback disks become critical to avoiding downtime. It's very important that an application be able to scale horizontally, whether to shard data across servers or across regions. A distributed architecture also increases performance. Geographically distributed servers can be more closed to clients and give a perception of performance. Also, databases distributed by more servers will handle more traffic as a whole and allow DevOps to accomplish zero downtime goals. This is also very useful for maintenance, as nodes can be brought down for support without affecting the application. Testing and benchmarking To know whether an application performs well or not under specific environments, we have to test it. This kind of test is called a benchmark. Benchmarking is important to do and it's specific to every application. Even for the same language and platform, different applications might perform differently, either because of the way in which some parts of an application were structured or the way in which a database was designed. Analyzing the performance will indicate bottleneck of your application, or if you may, the parts of the application that perform not good as others. These are the parts that need to be improved. Constantly trying to improve the worst performing parts will elevate the application's overall performance. There are plenty of tools out there, some more specific or focused on JavaScript applications, such as benchmarkjs (http://benchmarkjs.com/) and ben (https://github.com/substack/node-ben), and others more generic, such as ab (http://httpd.apache.org/docs/2.2/programs/ab.html) and httpload (https://github.com/perusio/httpload). There are several types of benchmark tests depending on the goal, they are as follows: Load testing is the simplest form of benchmarking. It is done to find out how the application performs under a specific load. You can test and find out how many connections an application accepts per second, or how many traffic bytes an application can handle. An application load can be checked by looking at the external performance, such as traffic, and also internal performance, such as the processor used or the memory consumed. Soak testing is used to see how an application performs during a more extended period of time. It is done when an application tends to degrade over time and analysis is needed to see how it reacts. This type of test is important in order to detect memory leaks, as some applications can perform well in some basic tests, but over time, the memory leaks and their performance can degrade. Spike testing is used when a load is increased very fast to see how the application reacts and performs. This test is very useful and important in applications that can have spike usages, and operators need to know how the application will react. Twitter is a good example of an application environment that can be affected by usage spikes (in world events such as sports or religious dates), and need to know how the infrastructure will handle them. All of these tests can become harder as your application grows. Since your user base gets bigger, your application scales and you lose the ability to be able to load test with the resources you have. It's good to be prepared for this moment, especially to be prepared to monitor performance and keep track of soaks and spikes as your application users start to be the ones responsible for continuously test load. Composition in applications Because of this continuous demand of performant applications, composition becomes very important. Composition is a practice where you split the application into several smaller and simpler parts, making them easier to understand, develop, and maintain. It also makes them easier to test and improve. Avoid creating big, monolithic code bases. They don't work well when you need to make a change, and they also don't work well if you need to test and analyze any part of the code to improve it and make it perform better. The Node.js platform helps you—and in some ways, forces you to—compose your code. Node.js Package Manager (NPM) is a great module publishing service. You can download other people's modules and publish your own as well. There are tens of thousands of modules published, which means that you don't have to reinvent the wheel in most cases. This is good since you can avoid wasting time on creating a module and use a module that is already in production and used by many people, which normally means that bugs will be tracked faster and improvements will be delivered even faster. The Node.js platform allows developers to easily separate code. You don't have to do this, as the platform doesn't force you to, but you should try and follow some good practices, such as the ones described in the following sections. Using NPM Don't rewrite code unless you need to. Take your time to try some available modules, and choose the one that is right for you. This reduces the probability of writing faulty code and helps published modules that have a bigger user base. Bugs will be spotted earlier, and more people in different environments will test fixes. Moreover, you will be using a more resilient module. One important and neglected task after starting to use some modules is to track changes and, whenever possible, keep using recent stable versions. If a dependency module has not been updated for a year, you can spot a problem later, but you will have a hard time figuring out what changed between two versions that are a year apart. Node.js modules tend to be improved over time and API changes are not rare. Always upgrade with caution and don't forget to test. Separating your code Again, you should always split your code into smaller parts. Node.js helps you do this in a very easy way. You should not have files bigger than 5 kB. If you have, you better think about splitting it. Also, as a good rule, each user-defined object should have its own separate file. Name your files accordingly: // MyObject.js module.exports = MyObject; function MyObject() { // … } MyObject.prototype.myMethod = function () { … }; Another good rule to check whether you have a file bigger than it should be; that is, it should be easy to read and understand in less than 5 minutes by someone new to the application. If not, it means that it's too complex and it will be harder to track and fix bugs later on. Remember that later on, when your application becomes huge, you will be like a new developer when opening a file to fix something. You can't remember all of the code of the application, and you need to absorb a file behavior fast. Garbage collection When writing applications, managing available memory is boring and difficult. When the application gets complex it's easy to start leaking memory. Many programming languages have automatic memory management, removing this management away from the developer by means of a Garbage Collector (GC). The GC is only a part of this memory management, but it's the most important one and is responsible for reclaiming memory that is no longer in use (garbage), by periodically looking at disposed referenced objects and freeing the memory associated with them. The most common technique used by GC is to monitor reference counting. This means that GC, for each object, holds the number (count) of other objects that reference it. When an object has no references to it, it can be collected, meaning, it can be disposed and its memory freed. CPU Profiling Profiling is boring but it's a good form of software analysis where you measure resource usage. This usage is measured over time and sometimes under specific workloads. Resources can mean anything the application is using, being it memory, disk, network or processor. More specifically, CPU profiling allows one to analyze how and how much your functions use the processor. You can also analyze the opposite, the non-usage of the processor, the idle time. When profiling the processor, we usually take samples of the call stack at a certain frequency and analyze how the stack changes (increases or decreases) over the sampling period. If we use profilers from the operating system we'll have more items in stack than you probably expect, as you'll get internal calls of node.js and V8. Summary Together, Node.js and NPM make a very good platform for developing high-performance applications. Since the language behind them is JavaScript and most applications these days are web applications, these combinations make it an even more appealing choice, as it's one less server-side language to learn (such as PHP or Ruby) and can ultimately allow a developer to share code on the client and server sides. Also, frontend and backend developers can share, read, and improve each other's code. Many developers pick this formula and bring with them many of their habits from the client side. Some of these habits are not applicable because on the server-side, asynchronous tasks must rule as there are many clients connected (as opposed to one) and performance becomes crucial. You now see how the garbage collector task is not that easy. It surely does a very good job managing memory automatically. You can help it a lot, especially if writing applications with performance in mind. Avoiding the GC old space growing is necessary to avoid long GC cycles, pausing your application and sometimes forcing your services to restart. Every time you create a new variable you allocate memory and you get closer to a new GC cycle. Even understanding how memory is managed, sometimes you need to inspect your memory usage behavior In environments seen nowadays, it's very important to be able to profile an application to identify bottlenecks, especially at the processor and memory level. Overall you should focus on your code quality before going into profiling. Resources for Article: Further resources on this subject: Node.js Fundamentals [Article] So, what is Node.js? [Article] Setting up Node [Article]

In this first of several articles Bill Pretty, author of Getting Started with Electronic Projects and Building a Home Security System with BeagleBone, shows us how to use a Zigbee point-to-point network to create our very own remote controlled RC car! What Network are we using? In the final chapter of Getting Started with Electronics Projects, I introduced you to the concept of ZigBee RF transmitters and receivers. We used the Type 1 version of XBee devices, because they were capable of “pin mirroring”. In simple terms, this means that whatever change occurs on the I/O pin of the transmitting device is reflected on the corresponding output pin of the receiving XBee device. This feature allows us to use the receiver XBees in a stand-alone configuration, like we did with the alarm system. The type 1 XBee network is configured as a “Star” network. That means that there is only one ‘master’ and in our case, only one end device. Figure 1 Star Network In the figure above, the controlling XBee is shown in red, while the end device(s) are shown in green. The RC cars that we are going to be using in this article have three ‘basic’ types of steering control. I will refer to them as ‘toy’ cars to differentiate them from higher end ‘model’ cars which tend to come out of the box with servo control and microprocessor control. We will be using toy cars that I got from the local thrift store for less than ten dollars, because the remote control was missing! While the remote would come in handy, it isn’t really necessary, because we will be replacing both ends of the system. The first kind of steering system is the simplest and the cheapest to manufacture. A command from the controller tells the front wheels to turn left or right and the wheels will continue to try and turn in the proper direction as long as they are receiving the command, even though the steering motor has hit the limit of the travel. This type of steering may have a series resistor to prevent the motor from burning out, or it may just rely on the limited current capacity of the battery. When the user centres the control, the motor is turned off and an extension spring returns the wheels to the centre position. This makes our control job really easy. We need four signals/commands; forward, reverse, left and right. The spring does the rest. The second type of steering circuit is a simple single pole, double throw switch; which is connected to the motor. The switch changes position when the motor reaches either the full left or full right position. There is no indication of ‘centre’, so this type of steering still relies on a spring for auto-centring. Figure 2 Switch Feedback The third and most complicated type of steering control tends to be seen in toys with rechargeable batteries, like NiCad or NiMH as well. These types of batteries provide more power than alkaline ‘flash light’ batteries. For that reason a slightly more complex steering system is required. This type of system uses an “analog feedback” servo system. What it is is a standard DC motor connected to a potentiometer; so that when the motor turns so does the pot. The wiper of the pot provides a DC voltage that is proportional to the position of the motor. Figure 3 Analog Feedback Servo The steering control circuit will receive three voltages that we are concerned about; full left, full right and centre. This type of servo requires a comparator circuit or a small microprocessor to detect the three positions. More on comparator circuits in future articles. The following circuit will work with the first two types of steering circuits. The circuit diagrams are shown below. Figure 4 In the circuit above, U2 is half of an SN754410 H-Bridge controller integrated circuit. In order for the motor to move the ‘M1’ and ‘M1/’ signals must be at a different level. A logic 0 on M1/ and a logic 1 on M1 will make the motor turn in one direction and a logic 1 on M1/ and a logic 0 on M1 will make it turn in the opposite direction. While either two logic 1’s or 0’s will cause the motor to stop moving. In this way we have either left/right/centre or forward/reverse/stop control. Figure 5 Figure 5 above, shows the connections to the XBee receiver module. As I said earlier, when an I/O pin on the controller goes low, the corresponding pin on the receiver “mirrors” that input. The outputs M1, M1/, M2 and M2/ are connected to the drive and steering motors. The SN754410 is capable of sourcing or sinking about 1 amp, so AUX-1 and AUX-2 can be used to drive LED’s or other devices. The controller circuit is shown below. The circuit is actually taken from a sample circuit I found on the XBee vendor site: http://examples.digi.com/sensors/joystick-with-xbee/ Figure 6 XBee Transmitter The colours represent the wires which come from the joystick. In this case it is a simple arcade style joystick, which uses four micro switches rather than potentiometers. So when the joystick is moved, one or more inputs to the XBee transmitter are grounded (Logic 0) and this information is sent to the receiver in the car. The joystick, available from Adafruit http://examples.digi.com/sensors/joystick-with-xbee/ is shown below. Figure 7 Adafruit Joystick The AUX-1 input is connected to a simple toggle switch, which in this case is used to control the vehicle’s headlights. This is a very simple controller which is designed to control the first two types of vehicles. A more complex design is required to control more complex vehicles. A single board computer such as an Arduino, RaspberryPi or Beaglebone would be required, so that we can read inputs from the joystick as well as digital and analog inputs sent to it from the vehicle. Summary So that’s it for this blog, I hope you enjoyed reading it. In future editions of this topic, I’ll show you how to configure the XBee modules using the Windows application. This method is a lot easier that the one presented in the example on the Digi site. About the Author Bill began his career in electronics in the early 80’s with a small telecom startup company that would eventually become a large multinational. He left there to pursue a career in commercial aviation in Canada’s north. From there he joined the Ontario Center for Microelectronics, a provincially funded research and development center. Bill left there for a career in the military as a civilian contractor at what was then called Defense Research Establishment Ottawa. That began a career which was to span the next 25 years, and continues today. Over the years Bill has acquired extensive knowledge in the field of technical security and started his own company in 2010. That company is called William Pretty Security Inc. and provides support in the form of research and development, to various law enforcement and private security agencies. Bill has published and presented a number of white papers on the subject of technical security. Bill was also a guest presenter for a number of years at the Western Canada Technical Conference, a law enforcement only conference held every year in western Canada. A selection of these papers is available for download from his web site. www.williamprettysecurity.com If you’re interested in building more of your own projects, then be sure to check out Bill’s titles available now in both print and eBook format! If you’re new to working with microcontrollers be sure to pick up Getting Started with Electronic Projects to start creating a whole host of great projects you can do in a single weekend with LM555, ZigBee, and BeagleBone components! If you’re looking for something more advanced to tinker with, then Bill’s other title - Building a Home Security System with BeagleBone – is perfect for hobbyists looking to make a bigger project!

In this article by Enrique Fernández, Luis Sánchez Crespo, Anil Mahtani, and Aaron Martinez, authors of the book Learning ROS for Robotics Programming - Second Edition, you will see the different levels of filesystems in ROS. (For more resources related to this topic, see here.) When you start to use or develop projects with ROS, you will see that although this concept can sound strange in the beginning, you will become familiar with it with time. Similar to an operating system, an ROS program is divided into folders, and these folders have files that describe their functionalities: Packages: Packages form the atomic level of ROS. A package has the minimum structure and content to create a program within ROS. It may have ROS runtime processes (nodes), configuration files, and so on. Package manifests: Package manifests provide information about a package, licenses, dependencies, compilation flags, and so on. A package manifest is managed with a file called package.xml. Metapackages: When you want to aggregate several packages in a group, you will use metapackages. In ROS Fuerte, this form for ordering packages was called Stacks. To maintain the simplicity of ROS, the stacks were removed, and now, metapackages make up this function. In ROS, there exist a lot of these metapackages; one of them is the navigation stack. Metapackage manifests: Metapackage manifests (package.xml) are similar to a normal package but with an export tag in XML. It also has certain restrictions in its structure. Message (msg) types: A message is the information that a process sends to other processes. ROS has a lot of standard types of messages. Message descriptions are stored in my_package/msg/MyMessageType.msg. Service (srv) types: Service descriptions, stored in my_package/srv/MyServiceType.srv, define the request and response data structures for services provided by each process in ROS. The workspace Basically, the workspace is a folder where we have packages, edit the source files or compile packages. It is useful when you want to compile various packages at the same time and is a good place to have all our developments localized. A typical workspace is shown in the following screenshot. Each folder is a different space with a different role: The Source space: In the Source space (the src folder), you put your packages, projects, clone packages, and so on. One of the most important files in this space is CMakeLists.txt. The src folder has this file because it is invoked by CMake when you configure the packages in the workspace. This file is created with the catkin_init_workspace command. The Build space: In the build folder, CMake and catkin keep the cache information, configuration, and other intermediate files for our packages and projects. The Development (devel) space: The devel folder is used to keep the compiled programs. This is used to test the programs without the installation step. Once the programs are tested, you can install or export the package to share with other developers. You have two options with regard to building packages with catkin. The first one is to use the standard CMake workflow. With this, you can compile one package at a time, as shown in the following commands: $ cmake packageToBuild/ $ make If you want to compile all your packages, you can use the catkin_make command line, as shown in the following commands: $ cd workspace $ catkin_make Both commands build the executables in the build space directory configured in ROS. Another interesting feature of ROS are its overlays. When you are working with a package of ROS, for example, Turtlesim, you can do it with the installed version, or you can download the source file and compile it to use your modified version. ROS permits you to use your version of this package instead of the installed version. This is very useful information if you are working on an upgrade of an installed package. Packages Usually, when we talk about packages, we refer to a typical structure of files and folders. This structure looks as follows: include/package_name/: This directory includes the headers of the libraries that you would need. msg/: If you develop nonstandard messages, put them here. scripts/: These are executable scripts that can be in Bash, Python, or any other scripting language. src/: This is where the source files of your programs are present. You can create a folder for nodes and nodelets or organize it as you want. srv/: This represents the service (srv) types. CMakeLists.txt: This is the CMake build file. package.xml: This is the package manifest. To create, modify, or work with packages, ROS gives us tools for assistance, some of which are as follows: rospack: This command is used to get information or find packages in the system. catkin_create_pkg: This command is used when you want to create a new package. catkin_make: This command is used to compile a workspace. rosdep: This command installs the system dependencies of a package. rqt_dep: This command is used to see the package dependencies as a graph. If you want to see the package dependencies as a graph, you will find a plugin called package graph in rqt. Select a package and see the dependencies. To move between packages and their folders and files, ROS gives us a very useful package called rosbash, which provides commands that are very similar to Linux commands. The following are a few examples: roscd: This command helps us change the directory. This is similar to the cd command in Linux. rosed: This command is used to edit a file. roscp: This command is used to copy a file from a package. rosd: This command lists the directories of a package. rosls: This command lists the files from a package. This is similar to the ls command in Linux. The package.xml file must be in a package, and it is used to specify information about the package. If you find this file inside a folder, probably this folder is a package or a metapackage. If you open the package.xml file, you will see information about the name of the package, dependencies, and so on. All of this is to make the installation and the distribution of these packages easy. Two typical tags that are used in the package.xml file are <build_depend> and <run _depend>. The <build_depend> tag shows what packages must be installed before installing the current package. This is because the new package might use a functionality of another package. Metapackages As we have shown earlier, metapackages are special packages with only one file inside; this file is package.xml. This package does not have other files, such as code, includes, and so on. Metapackages are used to refer to others packages that are normally grouped following a feature-like functionality, for example, navigation stack, ros_tutorials, and so on. You can convert your stacks and packages from ROS Fuerte to Hydro and catkin using certain rules for migration. These rules can be found at http://wiki.ros.org/catkin/migrating_from_rosbuild. In the following screenshot, you can see the content from the package.xml file in the ros_tutorials metapackage. You can see the <export> tag and the <run_depend> tag. These are necessary in the package manifest. If you want to locate the ros_tutorials metapackage, you can use the following command: $ rosstack find ros_tutorials The output will be a path, such as /opt/ros/hydro/share/ros_tutorials. To see the code inside, you can use the following command line: $ vim /opt/ros/hydro/share/ros_tutorials/package.xml Remember that Hydro uses metapackages, not stacks, but the rosstack find command line works to find metapackages. Messages ROS uses a simplified message description language to describe the data values that ROS nodes publish. With this description, ROS can generate the right source code for these types of messages in several programming languages. ROS has a lot of messages predefined, but if you develop a new message, it will be in the msg/ folder of your package. Inside that folder, certain files with the .msg extension define the messages. A message must have two principal parts: fields and constants. Fields define the type of data to be transmitted in the message, for example, int32, float32, and string, or new types that you have created earlier, such as type1 and type2. Constants define the name of the fields. An example of a msg file is as follows: int32 id float32 vel string name In ROS, you can find a lot of standard types to use in messages, as shown in the following table list: Primitive type Serialization C++ Python bool (1) unsigned 8-bit int uint8_t(2) bool int8 signed 8-bit int int8_t int uint8 unsigned 8-bit int uint8_t int(3) int16 signed 16-bit int int16_t int uint16 unsigned 16-bit int uint16_t int int32 signed 32-bit int int32_t int uint32 unsigned 32-bit int uint32_t int int64 signed 64-bit int int64_t long uint64 unsigned 64-bit int uint64_t long float32 32-bit IEEE float float float float64 64-bit IEEE float double float string ascii string (4) std::string string time secs/nsecs signed 32-bit ints ros::Time rospy.Time duration secs/nsecs signed 32-bit ints ros::Duration rospy.Duration A special type in ROS is the header type. This is used to add the time, frame, and so on. This permits you to have the messages numbered, to see who is sending the message, and to have more functions that are transparent for the user and that ROS is handling. The header type contains the following fields: uint32 seq time stamp string frame_id You can see the structure using the following command: $ rosmsg show std_msgs/Header Thanks to the header type, it is possible to record the timestamp and frame of what is happening with the robot. In ROS, there exist tools to work with messages. The rosmsg tool prints out the message definition information and can find the source files that use a message type. In the upcoming sections, we will see how to create messages with the right tools. Services ROS uses a simplified service description language to describe ROS service types. This builds directly upon the ROS msg format to enable request/response communication between nodes. Service descriptions are stored in .srv files in the srv/ subdirectory of a package. To call a service, you need to use the package name, along with the service name; for example, you will refer to the sample_package1/srv/sample1.srv file as sample_package1/sample1. There are tools that exist to perform functions with services. The rossrv tool prints out the service descriptions and packages that contain the .srv files, and finds source files that use a service type. If you want to create a service, ROS can help you with the service generator. These tools generate code from an initial specification of the service. You only need to add the gensrv() line to your CMakeLists.txt file. Summary In this article, we saw the different types of filesystems present in the ROS architecture. Resources for Article: Further resources on this subject: Building robots that can walk [article] Avoiding Obstacles Using Sensors [article] Managing Test Structure with Robot Framework [article]

In this article by Kevin Jackson, Cody Bunch, and Egle Sigler, the authors of the book OpenStack Cloud Computing Cookbook - Third Edition, we will cover the following topics: Using OpenStack Dashboard with LBaaS Using OpenStack Dashboard with OpenStack Orchestration (For more resources related to this topic, see here.) Using OpenStack Dashboard with LBaaS The OpenStack Dashboard has the ability to view, create, and edit Load Balancers, add Virtual IPs (VIPs), and add nodes behind a Load Balancer. Dashboard also provides a user interface for creating HA Proxy server Load Balance services for our instances. We do this first by creating load balancing pools and then adding running instances to those pools. In this section, we will create an HTTP Load Balance pool, create a VIP, and configure instances to be part of the pool. The result will be the ability to use the HTTP Load Balancer pool address to send traffic to two instances running Apache. Getting ready Load a web browser, point it to our OpenStack Dashboard address at http://192.168.100.200/, and log in as an admin user. How to do it... First we will create an HTTP Load Balance pool. Creating pools To create a Load Balancer pool for a logged in user, carry out the following steps: To manage Load Balancers within our OpenStack Dashboard, select the Load Balancers tab: This will show available Load Balancer pools. Since we currently do not have any created, click on the Add Pool button in the top-right corner to add a pool. After clicking on the Add Pool button, we are presented with a modal window. Fill out the details to add a new pool: Set name, description, and provider in the modal window. We name our pool web-pool and give an appropriate description. We choose to go with a default provider since we are creating an HA Proxy. Select a subnet for the pool by clicking on the drop-down menu. All of our instances are attached to the private network, so we select 10.200.0.0/24. We select the HTTP protocol, but HTTPS and TCP are also available. This selection will depend on what kind of applications you are running. Select your preferred routing algorithm; we choose the ROUND_ROBIN balancing method. Other options are LEAST_CONNECTIONS, and SOURCE_IP. We leave the Admin State set to UP. Click on the Add button to create a new pool. You should see the new pool created in the pool list: Adding pool members To add instances to the Load Balancer, follow these steps: After adding a pool, you should still be on the Load Balancer page. Click on the Members tab. You should see a list of active members, if you have any, or an empty list: On the Members tab, click on the Add Member button. This will present you with the following menu: We select the pool we just created, web-pool, and specify members or instances that will be part of this pool. Select weights for the members of the pool. In our case, both members of the pool will have equal weights, so we assign the weight as 1. The selected protocol port will be used to access all members of the pool and, since we are using HTTP, we set the port to 80. We set Admin State to UP. Click on the Add button to add members to the pool. Now the member list should contain two newly added nodes: Adding a VIP to the Load Balancer pool Creating a VIP on the external network will allow access to the Load Balance pool and the instances behind it. To create the VIP, carry out the following steps: From the Load Balancer page, select the Pools tab and click on the drop-down arrow next to the Edit Pool button. This will give you a drop-down menu with an option to add a VIP: Click on the Add VIP option. This will present you with the modal window for creating a new VIP: We enter a custom name and description for our VIP. For VIP Subnet, we pick external subnet 192.168.100.0/24, followed by an available IP in that subnet. We choose 192.168.100.12. Enter 80 for Protocol Port, followed by selecting HTTP for Protocol. Set -1 for Connection Limit if you do not wish to have a maximum number of connections for this VIP. Click on the Add button to create the VIP. This will create a new VIP and show the current pool list: Now, when we click on web-pool, we will see the following details: Click on the web-vip link in the details to view the VIP details: You can test this Load Balancer by entering the VIP's IP in a browser. If you selected ROUND_ROBIN for your routing algorithm, each time you refresh your browser it should hit a different node. Deleting the Load Balancer To delete the Load Balancer, we will first need to delete the attached VIP and then delete the pool. From the Load Balancer page, check the Pools tab and click on the drop-down arrow next to the Edit Pool button. This will give you a drop-down menu with an option to delete a VIP: Selecting the Delete VIP drop-down option will give you a warning and ask you to confirm the deletion of the VIP. Click on the Delete VIP button to confirm: After deleting the VIP, now we can delete the pool. From the Load Balancer page's Pools tab, click on the drop-down arrow next to the appropriate Edit Pool button: Select the Delete Pool option from the drop-down list to delete the pool. You will get asked to confirm the deletion. If you are ready to delete the Load Balance pool, click on the Delete Pool button: How it works... We created a Load Balance pool and added two instances with Apache to it. We also created a virtual IP to be used on the external network and assigned it to our pool. To do this, we executed the following steps: Create a pool from the Load Balancer page's Pools tab. Select the subnet to which all the nodes are attached when creating the pool. Add members to the pool. Create a VIP for the pool. Both the pool and VIP can be edited after being created. Additional members can also be added to the pool at a later time. Using OpenStack Dashboard with OpenStack Orchestration Heat is the OpenStack Orchestration engine that enables users to quickly spin up whole environments using templates. Heat templates, known as Heat Orchestration Templates (HOT), are Yet Another Markup Language (YAML) based files. The files describe the resources being used, the type and the size of the instances, the network an instance will be attached to, among other pieces of information required to run that environment. We showed you how to use the Heat command line client. In this section, we will show how to use an existing Heat template file in OpenStack Dashboard to spin up two web servers running Apache, connected to a third instance running HA Proxy. Getting ready Load a web browser, point it to our OpenStack Dashboard address at http://192.168.100.200/ and log in as an admin user. How to do it... First, we will launch stack within our OpenStack Dashboard. Launching stacks To launch a Heat stack for a logged in user, carry out the following steps: To view available Heat stacks within our OpenStack Dashboard, select the Stacks tab under the Orchestration menu: After clicking on the Stacks tab, you will see all running stacks in your environment. In our case, our list is empty: Click on the Launch Stack button to create a new stack. You will see the following window: There are several ways to specify what template source to use in a stack: File, Direct Input, or URL. Choose which option is the most convenient for you. For our example, you can either use the URL directly or upload a file. The template file can be downloaded from https://raw.githubusercontent.com/OpenStackCookbook/OpenStackCookbook/master/cookbook.yaml. We will upload files from our system. Just like we downloaded the cookbook.yaml file, we can also download the Environment Source file. In this case, we do not have to use the environment source, but it makes it convenient. The environment file stores the values we would have to enter into the browser manually, but instead loads the values for us on the Launch Stack screen, as shown in step 8. In our example, we are using the environment file that can be downloaded from https://raw.githubusercontent.com/OpenStackCookbook/OpenStackCookbook/master/cookbook-env.yaml. Update the public_net_id, private_net_id, and private_subnet_id fields to match your environment. After selecting the Template Source and Environment Source files, click on Next: Our sample environment file contains the following code: parameters: key_name: demokey image: trusty-image flavor: m1.tiny public_net_id: 5e5d24bd-9d1f-4ed1-84b5-0b7e2a9a233b private_net_id: 25153759-994f-4835-9b13-bf0ec77fb336 private_subnet_id: 4cf2c09c-b3d5-40ed-9127-ec40e5e38343 Clicking on Next will give you a Launch Stack window with all the inputs: Note that most of the inputs in our template are now populated. If you did not specify the environment source file in the previous step, you will need to enter the key_name, image, flavor, public_net_id, private_net_id, and private_subnet_id fields. These fields are specific to each template used. Your templates may have different fields. Enter the stack name and user password for your user. If you are logged in as admin or demo, the password is openstack. Click on the Launch button to start stack creation. If all inputs were correct, you should see your stack being created: After the stack creation finishes and if there were no errors during creation, you will see your stack's status updated to Complete: Viewing stack details After launching a stack, there is a lot of information associated with it, including inputs, outputs, and, in the case of errors, information about why stack creation failed. To view the details of the stack, click on the stack name from the Stacks list. The first available view is Topology: Explore the topology by clicking on the nodes. If the graph does not fully fit or you would like a different perspective, you can drag the graph around the window. The next tab under Stack Detail will provide all of the information that was used in creating the stack: Stack information available in the Overview tab is as follows:    Info    Status    Outputs    Stack parameters    Launch parameters The Resources tab will show all the HEAT resources that were created during stack launch: If there were any errors during stack launch, check this page to see which component's creation failed. The Events tab shows all the events that occurred when the stack was created. This page can also be very helpful in troubleshooting Heat templates: While your Heat stack is running, you can also see how many instances it created under the Compute tab's Instance option. The following is what our instances look like on the Instances page: Note that the test1 instance was not part of the stack creation. All the other VMs were created during the stack launch. Deleting stacks Stack deletion is simple; however, it will delete all resources that were created during stack launch. Follow these steps: To delete a stack, first view the available stacks on the Stacks page: Click on the appropriate Delete Stack button to delete a stack. You will be asked to confirm the deletion: After confirming deletion, all resources associated with the stack will be deleted. How it works... We have used the OpenStack Dashboard to launch, view, and delete Orchestration stacks. We first needed to download a sample HA Proxy Heat Orchestration Template from GitHub. Since we were using an environment file, we also had to modify the appropriate inputs. Your own templates may have different inputs. After launching our HA Proxy stack, we explored its topology, resources, and events. Resources created during stack launch will also be reflected in the rest of your environment. If you are launching new instances, all of them will also be available on the Instance page. Delete and modify resources created during the stack launch only through Orchestration section in the OpenStack dashboard or on the command line. Deleting stacks through the dashboard will delete all associated resources. Unlock the full potential of OpenStack with 'OpenStack Cloud Computing Third Edition' - updated and improved for the latest challenges and opportunities unleashed by cloud.

In this article by Eric Mayor, author of the book Learning Predictive Analytics with R, we will be discussing how to use PCA in R. Nowadays, the access to data is easier and cheaper than ever before. This leads to a proliferation of data in organizations' data warehouses and on the Internet. Analyzing this data is not trivial, as the quantity often makes the analysis difficult or unpractical. For instance, the data is often more abundant than available memory on the machines. The available computational power is also often not enough to analyze the data in a reasonable time frame. One solution is to have recourse to technologies that deal with high dimensionality in data (big data). These solutions typically use the memory and computing power of several machines for analysis (computer clusters). However, most organizations do not have such infrastructure. Therefore, a more practical solution is to reduce the dimensionality of the data while keeping the essential of the information intact. (For more resources related to this topic, see here.) Another reason to reduce dimensionality is that, in some cases, there are more attributes than observations. If some scientists were to store the genome of all inhabitants of Europe and the United States, the number of cases (approximately 1 billion) would be much less than the 3 billion base pairs in the human DNA. Most analyses do not work well or not at all when observations are less than attributes. Confronted with this problem, the data analysts might select groups of attributes, which go together (like height and weight, for instance) according to their domain knowledge, and reduce the dimensionality of the dataset. The data is often structured across several relatively independent dimensions, with each dimension measured using several attributes. This is where principal component analysis (PCA) is an essential tool, as it permits each observation to receive a score on each of the dimensions (determined by PCA itself) while allowing to discard the attributes from which the dimensions are computed. What is meant here is that for each of the obtained dimensions, values (scores) will be produced that combines several attributes. These can be used for further analysis. In the next section, we will use PCA to combine several attributes in the questionnaire data. We will see that the participants' self-report on items such as lively, excited, enthusiastic (and many more) can be combined in a single dimension we call positive arousal. In this sense, PCA performs both dimensionality reduction (discard the attributes) and feature extraction (compute the dimensions). The features can then be used for classification and regression problems. Another use of PCA is to check that the underlying structure of the data corresponds to a theoretical model. For instance, in a questionnaire, a group of questions might assess construct A, and another group construct B, and so on. PCA will find two different factors if indeed there is more similarity in the answers of participants within each group of questions compared to the overall questionnaire. Researchers in fields such as psychology use PCA mainly to test their theoretical model in this fashion. This article is a shortened version of the chapter with the same name in my book Learning predictive analytics with R. In what follows, we will examine how to use PCA in R. We will notably discover how to interpret the results and perform diagnostics. Learning PCA in R In this section, we will learn more about how to use PCA in order to obtain knowledge from data, and ultimately reduce the number of attributes (also called features). The first dataset we will use is the msg dataset from the psych package. The msq (for motivational state questionnaire) data set is composed of 92 attributes, of which 72 is the rating of adjectives by 3896 participants describing their mood. We will only use these 72 attributes for the current purpose, which is the exploration of the structure of the questionnaire. We will, therefore, start by installing and loading the package and the data, and assign the data we are interested in (the 75 attributes mentioned) to an object called motiv. install.packages("psych") library(psych) data(msq) motiv = msq[,1:72] Dealing with missing values Missing values are a common problem in real-life datasets like the one we use here. There are several ways to deal with them, but here we will only mention omitting the cases where missing values are encountered. Let's see how many missing values (we will call them NAs) there are for each attribute in this dataset: apply(is.na(motiv),2,sum) View of missing data in the dataset We can see that for many attributes this is unproblematic (there are only few missing values). However, in the case of several attributes, the number of NAs is quite high (anxious: 1849 NAs, cheerful: 1850, idle: 1848, inactive: 1846, tranquil: 1843). The most probable explanation is that these items have been dropped from some of the samples in which the data has been collected. Removing all cases with NAs from the dataset would result in an empty dataset in this particular case. Try the following in your console to verify this claim: na.omit(motiv) For this reason, we will simply deal with the problem by removing these attributes from the analysis. Remember that there are other ways, such as data imputation, to solve this issue while keeping the attributes. In order to do so, we first need to know which column number corresponds to the attributes we want to suppress. An easy way to do this is simply by printing the names of each columns vertically (using cbind() for something it was not exactly made for). We will omit cases with missing values on other attributes from the analysis later. head(cbind(names(motiv)),5) Here, we only print the result for the first five columns of the data frame: [,1] [1,] active [2,] afraid [3,] alert [4,] angry [5,] anxious We invite the reader to verify on his screen whether we suppress the correct columns from the analysis using the following vector to match the attributes: ToSuppress = c(5, 15, 37, 38, 66) Let's check whether it is correct: names(motiv[ToSuppress]) The following is the output: [1] "anxious" "cheerful" "idle" "inactive" "tranquil" Naming the components using the loadings We will run the analysis using the principal() function from the psych package. This will allow examining the loadings in a sorted fashion and making them independent from the others using a rotation. We will extract five components. The psych package has already been loaded, as the data we are analyzing comes from a dataset, which is included in psych. Yet, we need to install another package, the GPArotation package, which is required for the analysis we will perform next: install.packages("GPArotation") library(GPArotation) We can now run the analysis. This time we want to apply an orthogonal rotation varimax, in order to obtain independent factorial scores. Precisions are provided in the paper Principal component analysis by Abdi and Williams (2010). We also want that the analysis uses imputed data for missing values and estimates the scores for each observation on each retained principal component. We use the print.psych() function to print the sorted loadings, which will make the interpretation of the principal components easier: Pca2 = principal(motiv[,-ToSuppress],nfactors = 5, rotate = "varimax", missing = T, scores = T) print.psych(Pca2, sort =T) The annotated output displayed in the following screenshot has been slightly altered in order to allow it to fit on one page. Results of the PCA with the principal() function using varimax rotation The name of the items are displayed in the first column of the first matrix of results. The column item displays their order. The component loadings for the five retained components are displayed next (RC1 to RC5). We will comment on h2 and u2 later. The loadings of the attributes on the loadings can be used to name the principal components. As can be seen in the preceding screenshot, the first component could be called Positive arousal, the second component could be called Negative arousal, the third could be called Serenity, the fourth Exhaustion, and the last component could be called Fear. It is worth noting that the MSQ has theoretically four components obtained from two dimensions: energy and tension. Therefore, the fifth component we found is not accounted for by the model. The h2 value indicates the proportion of variance of the variable that is accounted for by the selected components. The u2 value indicates the part of variance not explained by the components. The sum of h2 and u2 is 1. Below the first result matrix, a second matrix indicates the proportion of variance explained by the components on the whole dataset. For instance, the first component explains 21 percent of the variance in the dataset Proportion Var and 38 percent of the variance explained by the five components. Overall, the five components explain 56 percent of the variance in the dataset Cumulative Var. As a remainder, the purpose of PCA is to replace all the original attributes by the scores on these components in further analysis. This is what we will discuss next. PCA scores At this point, you might wonder where data reduction comes into play. Well, the computation of the scores for each factor allows reducing the number of attributes for a dataset. This has been done in the previous section. Accessing the PCA scores We will now examine these scores more thoroughly. Let's start by checking whether the scores are indeed uncorrelated: round(cor(Pca2$scores),3) This is the output: RC1 RC2 RC3 RC4 RC5 RC1 1.000 -0.001 0.000 0.000 0.001 RC2 -0.001 1.000 0.000 0.000 0.000 RC3 0.000 0.000 1.000 -0.001 0.001 RC4 0.000 0.000 -0.001 1.000 0.000 RC5 0.001 0.000 0.001 0.000 1.000 As can be seen in the previous output, the correlations between the values are equal to or lower than 0.001 for every pair of components. The components are basically uncorrelated, as we requested. Let's start by checking that it indeed contains the same number of rows: nrow(Pca2$scores) == nrow(msq)  Here's the output: [1] TRUE As the principal() function didn't remove any cases but imputed the data instead, we can now append the factor scores to the original dataset (a copy of it): bound = cbind(msq,Pca2$score) PCA diagnostics Here, we will briefly discuss two diagnostics that can be performed on the dataset that will be subjected to analysis. These diagnostics should be performed analyzing the data with PCA in order to ascertain that PCA is an optimal analysis for the considered data. The first diagnostic is Bartlett's test of sphericity. This test examines the relationship between the variables together, instead of 2 x 2 as in a correlation. To be more specific, it tests whether the correlation matrix is different from an identity matrix (which has ones on the diagonals and zeroes elsewhere). The null hypothesis is that the variables are independent (no underlying structure). The paper When is a correlation matrix appropriate for factor analysis? Some decision rules by Dzuiban and Shirkey (1974) provides more information on the topic. This test can be performed by the cortest.normal() function in the psych package. In this case, the first argument is the correlation matrix of the data we want to subject to PCA, and the n1 argument is the number of cases. We will use the same data set as in the PCA (we first assign the name M to it): M = na.omit(motiv[-ToSuppress]) cortest.normal(cor(M), n1 = nrow(M)) The output is provided as follows: Tests of correlation matrices Call:cortest.normal(R1 = cor(M), n1 = nrow(M)) Chi Square value 829268 with df = 2211 with probability < 0 The last line of the output shows that the data subjected to the analysis is clearly different from an identity matrix: The probability to obtain these results if it were an identity matrix is close to zero (do not pay attention to the > 0, it is simply extremely close to zero). You might be curious about what is the output of an identity matrix. It turns out we have something close to it: the correlations of the PCA scores with the varimax rotation that we examined before. Let's subject the scores to the analysis: cortest.normal(cor(Pca2$scores), n1 = nrow(Pca2$scores)) Tests of correlation matrices Call:cortest.normal(R1 = cor(Pca2$scores), n1 = nrow(Pca2$scores)) Chi Square value 0.01 with df = 10 with probability < 1 In this case, the results show that the correlation matrix is not significantly different from an identity matrix. The other diagnostic is the Kaiser Meyer Olkin (KMO) index, which indicates the part of the data that can be explained by elements present in the dataset. The higher this score the more the proportion of the data is explainable, for instance, by PCA. The KMO (also called MSA for Measure of Sample Adequacy) ranges from 0 (nothing is explainable) to 1 (everything is explainable). It can be returned for each item separately, or for the overall dataset. We will examine this value. It is the first component of the object returned by the KMO()function from psych package. The second component is the list of the values for the individual items (not examined here). It simply takes a matrix, a data frame, or a correlation matrix as an argument. Let's run in on our data: KMO(motiv)[1] This returns a value of 0.9715381, meaning that most of our data is explainable by the analysis. Summary In this article, we have discussed how the PCA algorithm works, how to select the appropriate number of components and how to use PCA scores for further analysis. Resources for Article: Further resources on this subject: Big Data Analysis (R and Hadoop) [article] How to do Machine Learning with Python [article] Clustering and Other Unsupervised Learning Methods [article]

In this article by Mat Johns, author of the book Getting Started with Hazelcast - Second Edition, we will learn the following topics: Creating and using collection listeners Instance, lifecycle, and cluster membership listeners The partition migration listener Quorum functions and listeners (For more resources related to this topic, see here.) Listening to the goings-on One great feature of Hazelcast is its ability to notify us of the goings-on of our persisted data and the cluster as a whole, allowing us to register an interest in events. The listener concept is borrowed from Java. So, you should feel pretty familiar with it. To provide this, there are a number of listener interfaces that we can implement to receive, process, and handle different types of events—one of which we previously encountered. The following are the listener interfaces: Collection listeners: EntryListener is used for map-based (IMap and MultiMap) events ItemListener is used for flat collection-based (IList, ISet, and IQueue) events MessageListener is used to receive topic events, but as we've seen before, it is used as a part of the standard operation of topics QuorumListener is used for quorum state change events Cluster listeners: DistributedObjectListener is used for the collection, creation, and destruction of events MembershipListener is used for cluster membership events LifecycleListener is used for local node state events MigrationListener is used for partition migration state events The sound of our own data Being notified about data changes can be rather useful as we can make an application-level decision regarding whether the change is important or not and react accordingly. The first interface that we are going to look at is EntryListener. This class will notify us when changes are made to the entries stored in a map collection. If we take a look at the interface, we can see four entry event types and two map-wide events that we will be notified about. EntryListener has also being broken up into a number of individual super MapListener interfaces. So, should we be interested in only a subset of event types, we can implement the appropriate super interfaces as required. Let's take a look at the following code: void entryAdded(EntryEvent<K, V> event);void entryRemoved(EntryEvent<K, V> event);void entryUpdated(EntryEvent<K, V> event);void entryEvicted(EntryEvent<K, V> event);void mapCleared(MapEvent event);void mapEvicted(MapEvent event); Hopefully, the first three are pretty self-explanatory. However, the fourth is a little less clear and in fact, one of the most useful. The entryEvicted method is invoked when an entry is removed from a map non-programmatically (that is, Hazelcast has done it all by itself). This instance will occur in one of the following two scenarios: An entry's TTL has been reached and the entry has been expired The map size, according to the configured policy, has been reached, and the appropriate eviction policy has kicked in to clear out space in the map The first scenario allows us a capability that is very rarely found in data sources—to have our application be told when a time-bound record has expired and the ability to trigger some behavior based on it. For example, we can use it to automatically trigger a teardown operation if an entry is not correctly maintained by a user's interactions. This will allow us to generate an event based on the absence of activity, which is rather useful! Let's create an example MapEntryListener class to illustrate the various events firing off: public class MapEntryListenerimplements EntryListener<String, String> {@Overridepublic void entryAdded(EntryEvent<String, String> event) {System.err.println("Added: " + event);}@Overridepublic void entryRemoved(EntryEvent<String, String> event) {System.err.println("Removed: " + event);}@Overridepublic void entryUpdated(EntryEvent<String, String> event) {System.err.println("Updated: " + event);}@Overridepublic void entryEvicted(EntryEvent<String, String> event) {System.err.println("Evicted: " + event);}@Overridepublic void mapCleared(MapEvent event) {System.err.println("Map Cleared: " + event);}@Overridepublic void mapEvicted(MapEvent event) {System.err.println("Map Evicted: " + event);}} We shall see the various events firing off as expected, with a short 10-second wait for the Berlin entry to expire, which will trigger the eviction event, as follows: Added: EntryEvent {c:capitals} key=GB, oldValue=null, value=Winchester,event=ADDED, by Member [127.0.0.1]:5701 thisUpdated: EntryEvent {c:capitals} key=GB, oldValue=Winchester,value=London, event=UPDATED, by Member [127.0.0.1]:5701 thisAdded: EntryEvent {c:capitals} key=DE, oldValue=null, value=Berlin,event=ADDED, by Member [127.0.0.1]:5701 thisRemoved: EntryEvent {c:capitals} key=GB, oldValue=null, value=London,event=REMOVED, by Member [127.0.0.1]:5701 thisEvicted: EntryEvent {c:capitals} key=DE, oldValue=null, value=Berlin,event=EVICTED, by Member [127.0.0.1]:5701 this We can obviously implement the interface as extensively as possible to service our application, potentially creating no-op stubs should we wish not to handle a particular type of event. Continuously querying The previous example focuses on notifying us of all the entry events. However, what if we were only interested in some particular data? We can obviously filter out our listener to only handle entries that we are interested in. However, it is potentially expensive to have all the events flying around the cluster, especially if our interest lies only in a minority of the potential data. To address this, we can combine capabilities from the map-searching features that we looked at a while back. As and when we register the entry listener to a collection, we can optionally provide a search Predicate that can be used as a filter within Hazelcast itself. We can whittle events down to relevant data before they even reach our listener, as follows: IMap<String, String> capitals = hz.getMap("capitals");capitals.addEntryListener(new MapEntryListener(),new SqlPredicate("name = 'London'"), true); Listeners racing into action One issue with the previous example is that we retrospectively reconfigured the map to feature the listener after it was already in service. To avoid this race condition, we should wire up the listener before the node-entering service. We can do this by registering the listener within the map configuration, as follows: <hazelcast><map name="default"><entry-listeners><entry-listener include-value="true">com.packtpub.hazelcast.listeners.MapEntryListener</entry-listener></entry-listeners></map></hazelcast> However, in both the methods of configuration, we have provided a Boolean flag when registering the listener to the map. This include-value flag allows us to configure the listener when it is invoked, as if we are interested in just the key of the event entry or the entries value as well. The default behavior (true) is to include the value, but in case the use case does not require it, there is a performance benefit of not having to provide it to the listener. So, if the use case does not require this extra data, it will be beneficial to set this flag to false. Keyless collections Though the keyless collections (set, list, and queue) are very similar to map collections, they feature their own interface to define the available events, in this case, ItemListener. It is not as extensive as its map counterpart, featuring just the itemAdded and itemRemoved events, and can be used in the same way, though it only offers visibility of these two event types. Programmatic configuration ahead of time So far, most of the extra configurations that we applied have been done either by customizing the hazelcast.xml file, or retrospectively modifying a collection in the code. However, what if we want to programmatically configure Hazelcast without the race condition that we discovered earlier? Fortunately, there is a way. By creating an instance of the Config class, we can configure the appropriate behavior on it by using a hierarchy that is similar to the XML configuration, but in code. Before passing this configuration object over to the instance creation method, the previous example can be reconfigured to do so, as follows: public static void main(String[] args) {Config conf = new Config();conf.addListenerConfig(new EntryListenerConfig(new MapEntryListener(), false, true));HazelcastInstance hz = Hazelcast.newHazelcastInstance(conf); Events unfolding in the wider world Now that we can determine what is going on with our data within the cluster, we may wish to have a higher degree of visibility of the state of the cluster itself. We can use this either to trigger application-level responses to cluster instability, or provide mechanisms to enable graceful scaling. We are provided with a number of interfaces for different types of cluster activity. All of these listeners can be configured retrospectively, as we have seen in the previous examples. However, in production, it is better to configure them in advance for the same reasons regarding the race condition as the collection listeners. We can do this either by using the hazelcast.xml configuration, or by using the Config class, as follows: <hazelcast><listeners><listener>com.packtpub.hazelcast.MyClusterListener</listener></listeners></hazelcast> The first of these, DistributedObjectListener, simply notifies all the nodes in the cluster as to the new collection objects that are being created or destroyed. Again, let's create a new example listener, ClusterObjectListener, to receive events, as follows: public class ClusterObjectListenerimplements DistributedObjectListener {@Overridepublic void distributedObjectCreated(DistributedObjectEvent event) {System.err.println("Created: " + event);}@Overridepublic void distributedObjectDestroyed(DistributedObjectEvent event) {System.err.println("Destroyed: " + event);}} As these listeners are for cluster-wide events, the example usage of this listener is rather simple. It mainly creates an instance with the appropriate listener registered, as follows: public class ClusterListeningExample {public static void main(String[] args) {Config config = new Config();config.addListenerConfig(new ListenerConfig(new ClusterObjectListener()));HazelcastInstance hz = Hazelcast.newHazelcastInstance(config);}} When using the TestApp console, we can create and destroy some collections, as follows: hazelcast[default] > ns testnamespace: testhazelcast[test] > m.put foo barnullhazelcast[test] > m.destroyDestroyed! The preceding code will produce the following, logging on ALL the nodes that feature the listener: Created: DistributedObjectEvent{eventType=CREATED, serviceName='hz:impl:mapService', distributedObject=IMap{name='test'}} Destroyed: DistributedObjectEvent{eventType=DESTROYED, serviceName='hz:impl:mapService', distributedObject=IMap{name='test'}} The next type of cluster listener is MembershipListener, which notifies all the nodes as to the joining or leaving of a node from the cluster. Let's create another example class, this time ClusterMembershipListener, as follows: public class ClusterMembershipListenerimplements MembershipListener {@Overridepublic void memberAdded(MembershipEvent membershipEvent) {System.err.println("Added: " + membershipEvent);}@Overridepublic void memberRemoved(MembershipEvent membershipEvent) {System.err.println("Removed: " + membershipEvent);}@Overridepublic void memberAttributeChanged(MemberAttributeEventmemberAttributeEvent) {System.err.println("Changed: " + memberAttributeEvent);}} Let's add the following code to the previous example application: conf.addListenerConfig(new ListenerConfig(new   ClusterMembershipListener())); Lastly, we have LifecycleListener, which is local to an individual node and allows the application built on top of Hazelcast to understand its particular node state by being notified as it changes when starting, pausing, resuming, or even shutting down, as follows: public class NodeLifecycleListener implements LifecycleListener {@Overridepublic void stateChanged(LifecycleEvent event) {System.err.println(event);}} Moving data around the place The final listener is very useful as it lets an application know when Hazelcast is rebalancing the data within the cluster. This gives us an opportunity to prevent or even block the shutdown of a node, as we might be in a period of increased data resilience risk because we may be actively moving data around at the time. The interface used in this case is MigrationListener. It will notify the application when the partitions migrate from one node to another and when they complete: public class ClusterMigrationListener implements MigrationListener {@Overridepublic void migrationStarted(MigrationEvent migrationEvent) {System.err.println("Started: " + migrationEvent);}@Overridepublic void migrationCompleted(MigrationEvent migrationEvent) {System.err.println("Completed: " + migrationEvent);}@Overridepublic void migrationFailed(MigrationEvent migrationEvent) {System.err.println("Failed: " + migrationEvent);}} When you are registering this cluster listener in your example application and creating and destroying various nodes, you will see a deluge of events that show the ongoing migrations. The more astute among you may have previously spotted a repartitioning task logging when spinning up the multiple nodes: INFO: [127.0.0.1]:5701 [dev] [3.5] Re-partitioning cluster data... Migration queue size: 271 The previous code indicated that 271 tasks (one migration task for each partition being migrated) have been scheduled to rebalance the cluster. The new listener will now give us significantly more visibility on these events as they occur and hopefully, they will be completed successfully: Started: MigrationEvent{partitionId=98, oldOwner=Member [127.0.0.1]:5701, newOwner=Member [127.0.0.1]:5702 this} Completed: MigrationEvent{partitionId=98, oldOwner=Member [127.0.0.1]:5701, newOwner=Member [127.0.0.1]:5702 this} Started: MigrationEvent{partitionId=99, oldOwner=Member [127.0.0.1]:5701, newOwner=Member [127.0.0.1]:5702 this} Completed: MigrationEvent{partitionId=99, oldOwner=Member [127.0.0.1]:5701, newOwner=Member [127.0.0.1]:5702 this} However, this logging information is overwhelming and actually not all that useful to us. So, let's expand on the listener to try and provide the application with the ability to check whether the cluster is currently migrating data partitions or has recently done so. Let's create a new static class, MigrationStatus, to hold information about cluster migration and help us interrogate it as regards its current status: public abstract class MigrationStatus {private static final Map<Integer, Boolean> MIGRATION_STATE =new ConcurrentHashMap<Integer, Boolean>();private static final AtomicLong LAST_MIGRATION_TIME =new AtomicLong(System.currentTimeMillis());public static void migrationEvent(int partitionId, boolean state) {MIGRATION_STATE.put(partitionId, state);if (!state) {LAST_MIGRATION_TIME.set(System.currentTimeMillis());}}public static boolean isMigrating() {Collection<Boolean> migrationStates= MIGRATION_STATE.values();Long lastMigrationTime = LAST_MIGRATION_TIME.get();// did we recently (< 10 seconds ago) complete a migrationif (System.currentTimeMillis() < lastMigrationTime + 10000) {return true;}// are any partitions currently migratingfor (Boolean partition : migrationStates) {if (partition) return true;}// otherwise we're not migratingreturn false;}} Then, we will update the listener to pass through the appropriate calls in response to the events coming into it, as follows: @Overridepublic void migrationStarted(MigrationEvent migrationEvent) {MigrationStatus.migrationEvent(migrationEvent.getPartitionId(), true);}@Overridepublic void migrationCompleted(MigrationEvent migrationEvent) {MigrationStatus.migrationEvent(migrationEvent.getPartitionId(), false);}@Overridepublic void migrationFailed(MigrationEvent migrationEvent) {System.err.println("Failed: " + migrationEvent);MigrationStatus.migrationEvent(migrationEvent.getPartitionId(), false);} Finally, let's add a loop to the example application to print out the migration state over time, as follows: public static void main(String[] args) throws Exception {Config conf = new Config();conf.addListenerConfig(new ListenerConfig(new ClusterMembershipListener()));conf.addListenerConfig(new ListenerConfig(new MigrationStatusListener()));HazelcastInstance hz = Hazelcast.newHazelcastInstance(conf);while(true) {System.err.println("Is Migrating?: " + MigrationStatus.isMigrating());Thread.sleep(5000);}} When starting and stopping various nodes, we should see each node detect the presence of the rebalance occurring, but it passes by quite quickly. It is in these small, critical periods of time when data is being moved around that resilience is mostly at risk, albeit depending on the configured numbers of backup, the risk could potentially be quite small. Added: MembershipEvent {member=Member [127.0.0.1]:5703,type=added}Is Migrating?: trueIs Migrating?: trueIs Migrating?: false Extending quorum We previously saw how we can configure a simple cluster health check to ensure that a given number of nodes were present to support the application. However, should we need more detailed control over the quorum definition beyond a simple node count check, we can create our own quorum function that will allow us to programmatically define what it means to be healthy. This can be as simple or as complex as what the application requires. In the following example, we sourced an expected cluster size (probably from a suitable location than a hard-coded) and dynamically checked whether a majority of the nodes are present: public class QuorumExample {public static void main(String[] args) throws Exception {QuorumConfig quorumConf = new QuorumConfig();quorumConf.setName("atLeastTwoNodesWithMajority");quorumConf.setEnabled(true);quorumConf.setType(QuorumType.WRITE);final int expectedClusterSize = 5;quorumConf.setQuorumFunctionImplementation(new QuorumFunction() {@Overridepublic boolean apply(Collection<Member> members) {return members.size() >= 2&& members.size() > expectedClusterSize / 2;}});MapConfig mapConf = new MapConfig();mapConf.setName("default");mapConf.setQuorumName("atLeastTwoNodesWithMajority");Config conf = new Config();conf.addQuorumConfig(quorumConf);conf.addMapConfig(mapConf);HazelcastInstance hz = Hazelcast.newHazelcastInstance(conf);new ConsoleApp(hz).start(args);}} We can also create a listener for the quorum health check so that we can be notified when the state of the quorum changes, as follows: public class ClusterQuorumListener implements QuorumListener {@Overridepublic void onChange(QuorumEvent quorumEvent) {System.err.println("Changed: " + quorumEvent);}} Let's attach the new listener to the appropriate configuration, as follows: quorumConf.addListenerConfig(new QuorumListenerConfig(new   ClusterQuorumListener())); Summary Hazelcast allows us to be a first-hand witness to a lot of internal state information. By registering the listeners so that they can be notified as the events occur, we can further enhance an application not only in terms of its functionality, but also with respect to its resilience. By allowing the application to know when and what events are unfolding underneath it, we can add defensiveness to it, embracing the dynamic and destroyable nature of the ephemeral approaches towards applications and infrastructure. Resources for Article: Further resources on this subject: What is Hazelcast? [Article] Apache Solr and Big Data – integration with MongoDB [Article] Introduction to Apache ZooKeeper [Article]

In this article by Saurabh K. Gupta, author of the book Oracle Advanced PL/SQL Developer Professional Guide, Second Edition you will learn new features in Oracle 12c SQL and PL/SQL. (For more resources related to this topic, see here.) Oracle 12c SQL and PL/SQL new features SQL is the most widely used data access language while PL/SQL is an exigent language that can integrate seamlessly with SQL commands. The biggest benefit of running PL/SQL is that the code processing happens natively within the Oracle Database. In the past, there have been debates and discussions on server side programming while the client invokes the PL/SQL routines to perform a task. The server side programming approach has many benefits. It reduces the network round trips between the client and the database. It reduces the code size and eases the code portability because PL/SQL can run on all platforms, wherever Oracle Database is supported. Oracle Database 12c introduces many language features and enhancements which are focused on SQL to PL/SQL integration, code migration, and ANSI compliance. This section discusses the SQL and PL/SQL new features in Oracle database 12c. IDENTITY columns Oracle Database 12c Release 1 introduces identity columns in the tables in compliance with the American National Standard Institute (ANSI) SQL standard. A table column, marked as IDENTITY, automatically generate an incremental numeric value at the time of record creation. Before the release of Oracle 12c, developers had to create an additional sequence object in the schema and assign its value to the column. The new feature simplifies code writing and benefits the migration of a non-Oracle database to Oracle. The following script declares an identity column in the table T_ID_COL: /*Create a table for demonstration purpose*/ CREATE TABLE t_id_col (id NUMBER GENERATED AS IDENTITY, name VARCHAR2(20)) / The identity column metadata can be queried from the dictionary views USER_TAB_COLSand USER_TAB_IDENTITY_COLS. Note that Oracle implicitly creates a sequence to generate the number values for the column. However, Oracle allows the configuration of the sequence attributes of an identity column. The custom sequence configuration is listed under IDENTITY_OPTIONS in USER_TAB_IDENTITY_COLS view: /*Query identity column information in USER_TAB_COLS*/ SELECT column_name, data_default, user_generated, identity_column FROM user_tab_cols WHERE table_name='T_ID_COL' / COLUMN_NAME DATA_DEFAULT USE IDE -------------- ------------------------------ --- --- ID "SCOTT"."ISEQ$$_93001".nextval YES YES NAME YES NO Let us check the attributes of the preceding sequence that Oracle has implicitly created. Note that the query uses REGEXP_SUBSTR to print the sequence configuration in multiple rows: /*Check the sequence configuration from USER_TAB_IDENTITY_COLS view*/ SELECT table_name,column_name, generation_type, REGEXP_SUBSTR(identity_options,'[^,]+', 1, LEVEL) identity_options FROM user_tab_identity_cols WHERE table_name = 'T_ID_COL' CONNECT BY REGEXP_SUBSTR(identity_options,'[^,]+',1,level) IS NOT NULL / TABLE_NAME COLUMN_NAME GENERATION IDENTITY_OPTIONS ---------- ---------------------- --------------------------------- T_ID_COL ID ALWAYS START WITH: 1 T_ID_COL ID ALWAYS INCREMENT BY: 1 T_ID_COL ID ALWAYS MAX_VALUE: 9999999999999999999999999999 T_ID_COL ID ALWAYS MIN_VALUE: 1 T_ID_COL ID ALWAYS CYCLE_FLAG: N T_ID_COL ID ALWAYS CACHE_SIZE: 20 T_ID_COL ID ALWAYS ORDER_FLAG: N 7 rows selected While inserting data in the table T_ID_COL, do not include the identity column as its value is automatically generated: /*Insert test data in the table*/ BEGIN INSERT INTO t_id_col (name) VALUES ('Allen'); INSERT INTO t_id_col (name) VALUES ('Matthew'); INSERT INTO t_id_col (name) VALUES ('Peter'); COMMIT; END; / Let us check the data in the table. Note the identity column values: /*Query the table*/ SELECT id, name FROM t_id_col / ID NAME ----- -------------------- 1 Allen 2 Matthew 3 Peter The sequence created under the covers for identity columns is tightly coupled with the column. If a user tries to insert a user-defined input for the identity column, the operation throws an exception ORA-32795: INSERT INTO t_id_col VALUES (7,'Steyn'); insert into t_id_col values (7,'Steyn') * ERROR at line 1: ORA-32795: cannot insert into a generated always identity column Default column value to a sequence in Oracle 12c Oracle Database 12c allows developers to default a column directly to a sequence‑generated value. The DEFAULT clause of a table column can be assigned to SEQUENCE.CURRVAL or SEQUENCE.NEXTVAL. The feature will be useful while migrating non-Oracle data definitions to Oracle. The DEFAULT ON NULL clause Starting with Oracle Database 12c, a column can be assigned a default non-null value whenever the user tries to insert NULL into the column. The default value will be specified in the DEFAULT clause of the column with a new ON NULL extension. Note that the DEFAULT ON NULL cannot be used with an object type column. The following script creates a table t_def_cols. A column ID has been defaulted to a sequence while the column DOJ will always have a non-null value: /*Create a sequence*/ CREATE SEQUENCE seq START WITH 100 INCREMENT BY 10 / /*Create a table with a column defaulted to the sequence value*/ CREATE TABLE t_def_cols ( id number default seq.nextval primary key, name varchar2(30), doj date default on null '01-Jan-2000' ) / The following PL/SQL block inserts the test data: /*Insert the test data in the table*/ BEGIN INSERT INTO t_def_cols (name, doj) values ('KATE', '27-FEB-2001'); INSERT INTO t_def_cols (name, doj) values ('NANCY', '17-JUN-1998'); INSERT INTO t_def_cols (name, doj) values ('LANCE', '03-JAN-2004'); INSERT INTO t_def_cols (name) values ('MARY'); COMMIT; END; / Query the table and check the values for the ID and DOJ columns. ID gets the value from the sequence SEQ while DOJ for MARY has been defaulted to 01-JAN-2000. /*Query the table to verify sequence and default on null values*/ SELECT * FROM t_def_cols / ID NAME DOJ ---------- -------- --------- 100 KATE 27-FEB-01 110 NANCY 17-JUN-98 120 LANCE 03-JAN-04 130 MARY 01-JAN-00 Support for 32K VARCHAR2 Oracle Database 12c supports the VARCHAR2, NVARCHAR2, and RAW datatypes up to 32,767 bytes in size. The previous maximum limit for the VARCHAR2 (and NVARCHAR2) and RAW datatypes was 4,000 bytes and 2,000 bytes respectively. The support for extended string datatypes will benefit the non-Oracle to Oracle migrations. The feature can be controlled using the initialization parameter MAX_STRING_SIZE. It accepts two values: STANDARD (default)—The maximum size prior to the release of Oracle Database 12c will apply. EXTENDED—The new size limit for string datatypes apply. Note that, after the parameter is set to EXTENDED, the setting cannot be rolled back. The steps to increase the maximum string size in a database are: Restart the database in UPGRADE mode. In the case of a pluggable database, the PDB must be opened in MIGRATEmode. Use the ALTER SYSTEM command to set MAX_STRING_SIZE to EXTENDED. As SYSDBA, execute the $ORACLE_HOME/rdbms/admin/utl32k.sql script. The script is used to increase the maximum size limit of VARCHAR2, NVARCHAR2, and RAW wherever required. Restart the database in NORMAL mode. As SYSDBA, execute utlrp.sql to recompile the schema objects with invalid status. The points to be considered while working with the 32k support for string types are: COMPATIBLE must be 12.0.0.0 After the parameter is set to EXTENDED, the parameter cannot be rolled back to STANDARD In RAC environments, all the instances of the database comply with the setting of MAX_STRING_SIZE Row limiting using FETCH FIRST For Top-N queries, Oracle Database 12c introduces a new clause, FETCH FIRST, to simplify the code and comply with ANSI SQL standard guidelines. The clause is used to limit the number of rows returned by a query. The new clause can be used in conjunction with ORDER BY to retrieve Top-N results. The row limiting clause can be used with the FOR UPDATE clause in a SQL query. In the case of a materialized view, the defining query should not contain the FETCH clause. Another new clause, OFFSET, can be used to skip the records from the top or middle, before limiting the number of rows. For consistent results, the offset value must be a positive number, less than the total number of rows returned by the query. For all other offset values, the value is counted as zero. Keywords with the FETCH FIRST clause are: FIRST | NEXT—Specify FIRST to begin row limiting from the top. Use NEXT with OFFSET to skip certain rows. ROWS | PERCENT—Specify the size of the result set as a fixed number of rows or percentage of total number of rows returned by the query. ONLY | WITH TIES—Use ONLY to fix the size of the result set, irrespective of duplicate sort keys. If you want records with matching sort keys, specify WITH TIES. The following query demonstrates the use of the FETCH FIRST and OFFSET clauses in Top-N queries: /*Create the test table*/ CREATE TABLE t_fetch_first (empno VARCHAR2(30), deptno NUMBER, sal NUMBER, hiredate DATE) / The following PL/SQL block inserts sample data for testing: /*Insert the test data in T_FETCH_FIRST table*/ BEGIN INSERT INTO t_fetch_first VALUES (101, 10, 1500, '01-FEB-2011'); INSERT INTO t_fetch_first VALUES (102, 20, 1100, '15-JUN-2001'); INSERT INTO t_fetch_first VALUES (103, 20, 1300, '20-JUN-2000'); INSERT INTO t_fetch_first VALUES (104, 30, 1550, '30-DEC-2001'); INSERT INTO t_fetch_first VALUES (105, 10, 1200, '11-JUL-2012'); INSERT INTO t_fetch_first VALUES (106, 30, 1400, '16-AUG-2004'); INSERT INTO t_fetch_first VALUES (107, 20, 1350, '05-JAN-2007'); INSERT INTO t_fetch_first VALUES (108, 20, 1000, '18-JAN-2009'); COMMIT; END; / The SELECT query pulls in the top-5 rows when sorted by their salary: /*Query to list top-5 employees by salary*/ SELECT * FROM t_fetch_first ORDER BY sal DESC FETCH FIRST 5 ROWS ONLY / EMPNO DEPTNO SAL HIREDATE -------- ------ ------- --------- 104 30 1550 30-DEC-01 101 10 1500 01-FEB-11 106 30 1400 16-AUG-04 107 20 1350 05-JAN-07 103 20 1300 20-JUN-00 The SELECT query lists the top 25% of employees (2) when sorted by their hiredate: /*Query to list top-25% employees by hiredate*/ SELECT * FROM t_fetch_first ORDER BY hiredate FETCH FIRST 25 PERCENT ROW ONLY / EMPNO DEPTNO SAL HIREDATE -------- ------ ----- --------- 103 20 1300 20-JUN-00 102 20 1100 15-JUN-01 The SELECT query skips the first five employees and displays the next two—the 6th and 7th employee data: /*Query to list 2 employees after skipping first 5 employees*/ SELECT * FROM t_fetch_first ORDER BY SAL DESC OFFSET 5 ROWS FETCH NEXT 2 ROWS ONLY / Invisible columns Oracle Database 12c supports invisible columns, which implies that the visibility of a column. A column marked invisible does not appear in the following operations: SELECT * FROM queries on the table SQL* Plus DESCRIBE command Local records of %ROWTYPE Oracle Call Interface (OCI) description A column can be made invisible by specifying the INVISIBLE clause against the column. Columns of all types (except user-defined types), including virtual columns, can be marked invisible, provided the tables are not temporary tables, external tables, or clustered ones. An invisible column can be explicitly selected by the SELECT statement. Similarly, the INSERTstatement will not insert values in an invisible column unless explicitly specified. Furthermore, a table can be partitioned based on an invisible column. A column retains its nullity feature even after it is made invisible. An invisible column can be made visible, but the ordering of the column in the table may change. In the following script, the column NICKNAME is set as invisible in the table t_inv_col: /*Create a table to demonstrate invisible columns*/ CREATE TABLE t_inv_col (id NUMBER, name VARCHAR2(30), nickname VARCHAR2 (10) INVISIBLE, dob DATE ) / The information about the invisible columns can be found in user_tab_cols. Note that the invisible column is marked as hidden: /*Query the USER_TAB_COLS for metadata information*/ SELECT column_id, column_name, hidden_column FROM user_tab_cols WHERE table_name = 'T_INV_COL' ORDER BY column_id / COLUMN_ID COLUMN_NAME HID ---------- ------------ --- 1 ID NO 2 NAME NO 3 DOB NO NICKNAME YES Hidden columns are different from invisible columns. Invisible columns can be made visible and vice versa, but hidden columns cannot be made visible. If we try to make the NICKNAME visible and NAME invisible, observe the change in column ordering: /*Script to change visibility of NICKNAME column*/ ALTER TABLE t_inv_col MODIFY nickname VISIBLE / /*Script to change visibility of NAME column*/ ALTER TABLE t_inv_col MODIFY name INVISIBLE / /*Query the USER_TAB_COLS for metadata information*/ SELECT column_id, column_name, hidden_column FROM user_tab_cols WHERE table_name = 'T_INV_COL' ORDER BY column_id / COLUMN_ID COLUMN_NAME HID ---------- ------------ --- 1 ID NO 2 DOB NO 3 NICKNAME NO NAME YES Temporal databases Temporal databases were released as a new feature in ANSI SQL:2011. The term temporal data can be understood as a piece of information that can be associated with a period within which the information is valid. Before the feature was included in Oracle Database 12c, data whose validity is linked with a time period had to be handled either by the application or using multiple predicates in the queries. Oracle 12c partially inherits the feature from the ANSI SQL:2011 standard to support the entities whose business validity can be bracketed with a time dimension. If you're relating a temporal database with the Oracle Database 11g Total Recall feature, you're wrong. The total recall feature records the transaction time of the data in the database to secure the transaction validity and not the functional validity. For example, an investment scheme is active between January to December. The date recorded in the database at the time of data loading is the transaction timestamp. [box type="info" align="" class="" width=""]Starting from Oracle 12c, the Total Recall feature has been rebranded as Flashback Data Archive and has been made available for all versions of Oracle Database.[/box] The valid time temporal can be enabled for a table by adding a time dimension using the PERIOD FOR clause on the date or timestamp columns of the table. The following script creates a table t_tmp_db with the valid time temporal: /*Create table with valid time temporal*/ CREATE TABLE t_tmp_db( id NUMBER, name VARCHAR2(30), policy_no VARCHAR2(50), policy_term number, pol_st_date date, pol_end_date date, PERIOD FOR pol_valid_time (pol_st_date, pol_end_date)) / Create some sample data in the table: /*Insert test data in the table*/ BEGIN INSERT INTO t_tmp_db VALUES (100, 'Packt', 'PACKT_POL1', 1, '01-JAN-2015', '31-DEC-2015'); INSERT INTO t_tmp_db VALUES (110, 'Packt', 'PACKT_POL2', 2, '01-JAN-2015', '30-JUN-2015'); INSERT INTO t_tmp_db VALUES (120, 'Packt', 'PACKT_POL3', 3, '01-JUL-2015', '31-DEC-2015'); COMMIT; END; / Let us set the current time period window using DBMS_FLASHBACK_ARCHIVE. Grant the EXECUTE privilege on the package to the scott user. /*Connect to sysdba to grant execute privilege to scott*/ conn / as sysdba GRANT EXECUTE ON dbms_flashback_archive to scott / Grant succeeded. /*Connect to scott*/ conn scott/tiger /*Set the valid time period as CURRENT*/ EXEC DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME('CURRENT'); PL/SQL procedure successfully completed. Setting the valid time period as CURRENT means that all the tables with a valid time temporal will only list the rows that are valid with respect to today's date. You can set the valid time to a particular date too. /*Query the table*/ SELECT * from t_tmp_db / ID POLICY_NO POL_ST_DATE POL_END_DATE --------- ---------- ----------------------- ------------------- 100 PACKT_POL1 01-JAN-15 31-DEC-15 110 PACKT_POL2 01-JAN-15 30-JUN-15 [box type="info" align="" class="" width=""]Due to dependency on the current date, the result may vary when the reader runs the preceding queries.[/box] The query lists only those policies that are active as of March 2015. Since the third policy starts in July 2015, it is currently not active. In-Database Archiving Oracle Database 12c introduces In-Database Archiving to archive the low priority data in a table. The inactive data remains in the database but is not visible to the application. You can mark old data for archival, which is not actively required in the application except for regulatory purposes. Although the archived data is not visible to the application, it is available for querying and manipulation. In addition, the archived data can be compressed to improve backup performance. A table can be enabled by specifying the ROW ARCHIVAL clause at the table level, which adds a hidden column ORA_ARCHIVE_STATE to the table structure. The column value must be updated to mark a row for archival. For example: /*Create a table with row archiving*/ CREATE TABLE t_row_arch( x number, y number, z number) ROW ARCHIVAL / When we query the table structure in the USER_TAB_COLS view, we find an additional hidden column, which Oracle implicitly adds to the table: /*Query the columns information from user_tab_cols view*/ SELECT column_id,column_name,data_type, hidden_column FROM user_tab_cols WHERE table_name='T_ROW_ARCH' / COLUMN_ID COLUMN_NAME DATA_TYPE HID ---------- ------------------ ---------- --- ORA_ARCHIVE_STATE VARCHAR2 YES 1 X NUMBER NO 2 Y NUMBER NO 3 Z NUMBER NO Let us create test data in the table: /Insert test data in the table*/ BEGIN INSERT INTO t_row_arch VALUES (10,20,30); INSERT INTO t_row_arch VALUES (11,22,33); INSERT INTO t_row_arch VALUES (21,32,43); INSERT INTO t_row_arch VALUES (51,82,13); commit; END; / For testing purpose, let us archive the rows in the table where X > 50 by updating the ora_archive_state column: /*Update ORA_ARCHIVE_STATE column in the table*/ UPDATE t_row_arch SET ora_archive_state = 1 WHERE x > 50 / COMMIT / By default, the session displays only the active records from an archival-enabled table: /*Query the table*/ SELECT * FROM t_row_arch / X Y Z ------ -------- ---------- 10 20 30 11 22 33 21 32 43 If you wish to display all the records, change the session setting: /*Change the session parameter to display the archived records*/ ALTER SESSION SET ROW ARCHIVAL VISIBILITY = ALL / Session altered. /*Query the table*/ SELECT * FROM t_row_arch / X Y Z ---------- ---------- ---------- 10 20 30 11 22 33 21 32 43 51 82 13 Defining a PL/SQL subprogram in the SELECT query and PRAGMA UDF Oracle Database 12c includes two new features to enhance the performance of functions when called from SELECT statements. With Oracle 12c, a PL/SQL subprogram can be created inline with the SELECT query in the WITH clause declaration. The function created in the WITH clause subquery is not stored in the database schema and is available for use only in the current query. Since a procedure created in the WITH clause cannot be called from the SELECT query, it can be called in the function created in the declaration section. The feature can be very handy in read-only databases where the developers were not able to create PL/SQL wrappers. Oracle Database 12c adds the new PRAGMA UDF to create a standalone function with the same objective. Earlier, the SELECT queries could invoke a PL/SQL function, provided the function didn't change the database purity state. The query performance used to degrade because of the context switch from SQL to the PL/SQL engine (and vice versa) and the different memory representations of data type representation in the processing engines. In the following example, the function fun_with_plsql calculates the annual compensation of an employee's monthly salary: /*Create a function in WITH clause declaration*/ WITH FUNCTION fun_with_plsql (p_sal NUMBER) RETURN NUMBER IS BEGIN RETURN (p_sal * 12); END; SELECT ename, deptno, fun_with_plsql (sal) "annual_sal" FROM emp / ENAME DEPTNO annual_sal ---------- --------- ---------- SMITH 20 9600 ALLEN 30 19200 WARD 30 15000 JONES 20 35700 MARTIN 30 15000 BLAKE 30 34200 CLARK 10 29400 SCOTT 20 36000 KING 10 60000 TURNER 30 18000 ADAMS 20 13200 JAMES 30 11400 FORD 20 36000 MILLER 10 15600 14 rows selected. [box type="info" align="" class="" width=""]If the query containing the WITH clause declaration is not a top-level statement, then the top level statement must use the WITH_PLSQL hint. The hint will be used if INSERT, UPDATE, or DELETE statements are trying to use a SELECT with a WITHclause definition. Failure to include the hint results in an exception ORA-32034: unsupported use of WITH clause.[/box] A function can be created with the PRAGMA UDF to inform the compiler that the function is always called in a SELECT statement. Note that the standalone function created in the following code carries the same name as the one in the last example. The local WITH clause declaration takes precedence over the standalone function in the schema. /*Create a function with PRAGMA UDF*/ CREATE OR REPLACE FUNCTION fun_with_plsql (p_sal NUMBER) RETURN NUMBER is PRAGMA UDF; BEGIN RETURN (p_sal *12); END; / Since the objective of the feature is performance, let us go ahead with a case study to compare the performance when using a standalone function, a PRAGMA UDF function, and a WITHclause declared function. Test setup The exercise uses a test table with 1 million rows, loaded with random data. /*Create a table for performance test study*/ CREATE TABLE t_fun_plsql (id number, str varchar2(30)) / /*Generate and load random data in the table*/ INSERT /*+APPEND*/ INTO t_fun_plsql SELECT ROWNUM, DBMS_RANDOM.STRING('X', 20) FROM dual CONNECT BY LEVEL <= 1000000 / COMMIT / Case 1: Create a PL/SQL standalone function as it used to be until Oracle Database 12c. The function counts the numbers in the str column of the table. /*Create a standalone function without Oracle 12c enhancements*/ CREATE OR REPLACE FUNCTION f_count_num (p_str VARCHAR2) RETURN PLS_INTEGER IS BEGIN RETURN (REGEXP_COUNT(p_str,'d')); END; / The PL/SQL block measures the elapsed and CPU time when working with a pre-Oracle 12c standalone function. These numbers will serve as the baseline for our case study. /*Set server output on to display messages*/ SET SERVEROUTPUT ON /*Anonymous block to measure performance of a standalone function*/ DECLARE l_el_time PLS_INTEGER; l_cpu_time PLS_INTEGER; CURSOR C1 IS SELECT f_count_num (str) FROM t_fun_plsql; TYPE t_tab_rec IS TABLE OF PLS_INTEGER; l_tab t_tab_rec; BEGIN l_el_time := DBMS_UTILITY.GET_TIME (); l_cpu_time := DBMS_UTILITY.GET_CPU_TIME (); OPEN c1; FETCH c1 BULK COLLECT INTO l_tab; CLOSE c1; DBMS_OUTPUT.PUT_LINE ('Case 1: Performance of a standalone function'); DBMS_OUTPUT.PUT_LINE ('Total elapsed time:'||to_char(DBMS_UTILITY.GET_TIME () - l_el_time)); DBMS_OUTPUT.PUT_LINE ('Total CPU time:'||to_char(DBMS_UTILITY.GET_CPU_TIME () - l_cpu_time)); END; / Performance of a standalone function: Total elapsed time:1559 Total CPU time:1366 PL/SQL procedure successfully completed. Case 2: Create a PL/SQL function using PRAGMA UDF to count the numbers in the str column. /*Create the function with PRAGMA UDF*/ CREATE OR REPLACE FUNCTION f_count_num_pragma (p_str VARCHAR2) RETURN PLS_INTEGER IS PRAGMA UDF; BEGIN RETURN (REGEXP_COUNT(p_str,'d')); END; / Let us now check the performance of the PRAGMA UDF function using the following PL/SQL block. /*Set server output on to display messages*/ SET SERVEROUTPUT ON /*Anonymous block to measure performance of a PRAGMA UDF function*/ DECLARE l_el_time PLS_INTEGER; l_cpu_time PLS_INTEGER; CURSOR C1 IS SELECT f_count_num_pragma (str) FROM t_fun_plsql; TYPE t_tab_rec IS TABLE OF PLS_INTEGER; l_tab t_tab_rec; BEGIN l_el_time := DBMS_UTILITY.GET_TIME (); l_cpu_time := DBMS_UTILITY.GET_CPU_TIME (); OPEN c1; FETCH c1 BULK COLLECT INTO l_tab; CLOSE c1; DBMS_OUTPUT.PUT_LINE ('Case 2: Performance of a PRAGMA UDF function'); DBMS_OUTPUT.PUT_LINE ('Total elapsed time:'||to_char(DBMS_UTILITY.GET_TIME () - l_el_time)); DBMS_OUTPUT.PUT_LINE ('Total CPU time:'||to_char(DBMS_UTILITY.GET_CPU_TIME () - l_cpu_time)); END; / Performance of a PRAGMA UDF function: Total elapsed time:664 Total CPU time:582 PL/SQL procedure successfully completed. Case 3: The following PL/SQL block dynamically executes the function in the WITH clause subquery. Note that, unlike other SELECT statements, a SELECT query with a WITH clause declaration cannot be executed statically in the body of a PL/SQL block. /*Set server output on to display messages*/ SET SERVEROUTPUT ON /*Anonymous block to measure performance of inline function*/ DECLARE l_el_time PLS_INTEGER; l_cpu_time PLS_INTEGER; l_sql VARCHAR2(32767); c1 sys_refcursor; TYPE t_tab_rec IS TABLE OF PLS_INTEGER; l_tab t_tab_rec; BEGIN l_el_time := DBMS_UTILITY.get_time; l_cpu_time := DBMS_UTILITY.get_cpu_time; l_sql := 'WITH FUNCTION f_count_num_with (p_str VARCHAR2) RETURN NUMBER IS BEGIN RETURN (REGEXP_COUNT(p_str,'''||''||'d'||''')); END; SELECT f_count_num_with(str) FROM t_fun_plsql'; OPEN c1 FOR l_sql; FETCH c1 bulk collect INTO l_tab; CLOSE c1; DBMS_OUTPUT.PUT_LINE ('Case 3: Performance of an inline function'); DBMS_OUTPUT.PUT_LINE ('Total elapsed time:'||to_char(DBMS_UTILITY.GET_TIME () - l_el_time)); DBMS_OUTPUT.PUT_LINE ('Total CPU time:'||to_char(DBMS_UTILITY.GET_CPU_TIME () - l_cpu_time)); END; / Performance of an inline function: Total elapsed time:830 Total CPU time:718 PL/SQL procedure successfully completed. Comparative analysis Comparing the results from the preceding three cases, it's clear that the Oracle 12c flavor of PL/SQL functions out-performs the pre-12c standalone function by a high margin. From the following matrix, it is apparent that the usage of the PRAGMA UDF or WITH clause declaration enhances the code performance by (roughly) a factor of 2. Case Description Elapsed Time CPU time Performance gain factor by CPU time Standalone PL/SQL function in pre-Oracle 12c database 1559 1336 1x Standalone PL/SQL PRAGMA UDF function in Oracle 12c 664 582 2.3x Function created in WITH clause declaration in Oracle 12c 830 718 1.9x   [box type="info" align="" class="" width=""]Note that the numbers may slightly differ in the reader's testing environment but you should be able to draw the same conclusion by comparing them.[/box] The PL/SQL program unit whitelisting Prior to Oracle 12c, a standalone or packaged PL/SQL unit could be invoked by all other programs in the session's schema. Oracle Database 12c allows users to prevent unauthorized access to PL/SQL program units. You can now specify the list of whitelist program units that can invoke a particular program. The PL/SQL program header or the package specification can specify the list of program units in the ACCESSIBLE BY clause in the program header. All other program units, including cross-schema references (even SYS owned objects), trying to access a protected subprogram will receive an exception, PLS-00904: insufficient privileges to access object [object name]. The feature can be very useful in an extremely sensitive development environment. Suppose, a package PKG_FIN_PROC contains the sensitive implementation routines for financial institutions, the packaged subprograms are called by another PL/SQL package PKG_FIN_INTERNALS. The API layer exposes a fixed list of programs through a public API called PKG_CLIENT_ACCESS. In order to restrict access to the packaged routines in PKG_FIN_PROC, the users can build a safety net so as to allow access to only authorized programs. The following PL/SQL package PKG_FIN_PROC contains two subprograms—P_FIN_QTR and P_FIN_ANN. The ACCESSIBLE BY clause includes PKG_FIN_INTERNALS which means that all other program units, including anonymous PL/SQL blocks, are blocked from invoking PKG_FIN_PROC constructs. /*Package with the accessible by clause*/ CREATE OR REPLACE PACKAGE pkg_fin_proc ACCESSIBLE BY (PACKAGE pkg_fin_internals) IS PROCEDURE p_fin_qtr; PROCEDURE p_fin_ann; END; / [box type="info" align="" class="" width=""]The ACCESSIBLE BY clause can be specified for schema-level programs only.[/box] Let's see what happens when we invoke the packaged subprogram from an anonymous PL/SQL block. /*Invoke the packaged subprogram from the PL/SQL block*/ BEGIN pkg_fin_proc.p_fin_qtr; END; / pkg_fin_proc.p_fin_qtr; * ERROR at line 2: ORA-06550: line 2, column 4: PLS-00904: insufficient privilege to access object PKG_FIN_PROC ORA-06550: line 2, column 4: PL/SQL: Statement ignored Well, the compiler throws an exception as invoking the whitelisted package from an anonymous block is not allowed. The ACCESSIBLE BY clause can be included in the header information of PL/SQL procedures and functions, packages, and object types. Granting roles to PL/SQL program units Before Oracle Database 12c, a PL/SQL unit created with the definer's rights (default AUTHID) always executed with the definer's rights, whether or not the invoker has the required privileges. It may lead to an unfair situation where the invoking user may perform unwanted operations without needing the correct set of privileges. Similarly for an invoker's right unit, if the invoking user possesses a higher set of privileges than the definer, he might end up performing unauthorized operations. Oracle Database 12c secures the definer's rights by allowing the defining user to grant complementary roles to individual PL/SQL subprograms and packages. From the security standpoint, the granting of roles to schema level subprograms provides granular control as the privileges of the invoker are validated at the time of execution. In the following example, we will create two users: U1 and U2. The user U1 creates a PL/SQL procedure P_INC_PRICE that adds a surcharge to the price of a product by a certain amount. U1 grants the execute privilege to user U2. Test setup Let's create two users and give them the required privileges. /*Create a user with a password*/ CREATE USER u1 IDENTIFIED BY u1 / User created. /*Grant connect privileges to the user*/ GRANT CONNECT, RESOURCE TO u1 / Grant succeeded. /*Create a user with a password*/ CREATE USER u2 IDENTIFIED BY u2 / User created. /*Grant connect privileges to the user*/ GRANT CONNECT, RESOURCE TO u2 / Grant succeeded. The user U1 contains the PRODUCTS table. Let's create and populate the table. /*Connect to U1*/ CONN u1/u1 /*Create the table PRODUCTS*/ CREATE TABLE products ( prod_id INTEGER, prod_name VARCHAR2(30), prod_cat VARCHAR2(30), price INTEGER ) / /*Insert the test data in the table*/ BEGIN DELETE FROM products; INSERT INTO products VALUES (101, 'Milk', 'Dairy', 20); INSERT INTO products VALUES (102, 'Cheese', 'Dairy', 50); INSERT INTO products VALUES (103, 'Butter', 'Dairy', 75); INSERT INTO products VALUES (104, 'Cream', 'Dairy', 80); INSERT INTO products VALUES (105, 'Curd', 'Dairy', 25); COMMIT; END; / The procedure p_inc_price is designed to increase the price of a product by a given amount. Note that the procedure is created with the definer's rights. /*Create the procedure with the definer's rights*/ CREATE OR REPLACE PROCEDURE p_inc_price (p_prod_id NUMBER, p_amt NUMBER) IS BEGIN UPDATE products SET price = price + p_amt WHERE prod_id = p_prod_id; END; / The user U1 grants execute privilege on p_inc_price to U2. /*Grant execute on the procedure to the user U2*/ GRANT EXECUTE ON p_inc_price TO U2 / The user U2 logs in and executes the procedure P_INC_PRICE to increase the price of Milk by 5 units. /*Connect to U2*/ CONN u2/u2 /*Invoke the procedure P_INC_PRICE in a PL/SQL block*/ BEGIN U1.P_INC_PRICE (101,5); COMMIT; END; / PL/SQL procedure successfully completed. The last code listing exposes a gray area. The user U2, though not authorized to view PRODUCTS data, manipulates its data with the definer's rights. We need a solution to the problem. The first step is to change the procedure from definer's rights to invoker's rights. /*Connect to U1*/ CONN u1/u1 /*Modify the privilege authentication for the procedure to invoker's rights*/ CREATE OR REPLACE PROCEDURE p_inc_price (p_prod_id NUMBER, p_amt NUMBER) AUTHID CURRENT_USER IS BEGIN UPDATE products SET price = price + p_amt WHERE prod_id = p_prod_id; END; / Now, if we execute the procedure from U2, it throws an exception because it couldn't find the PRODUCTS table in its schema. /*Connect to U2*/ CONN u2/u2 /*Invoke the procedure P_INC_PRICE in a PL/SQL block*/ BEGIN U1.P_INC_PRICE (101,5); COMMIT; END; / BEGIN * ERROR at line 1: ORA-00942: table or view does not exist ORA-06512: at "U1.P_INC_PRICE", line 5 ORA-06512: at line 2 In a similar scenario in the past, the database administrators could have easily granted select or updated privileges to U2, which is not an optimal solution from the security standpoint. Oracle 12c allows the users to create program units with invoker's rights but grant the required roles to the program units and not the users. So, an invoker right unit executes with invoker's privileges, plus the PL/SQL program role. Let's check out the steps to create a role and assign it to the procedure. SYSDBA creates the role and assigns it to the user U1. Using the ADMIN or DELEGATE option with the grant enables the user to grant the role to other entities. /*Connect to SYSDBA*/ CONN / as sysdba /*Create a role*/ CREATE ROLE prod_role / /*Grant role to user U1 with delegate option*/ GRANT prod_role TO U1 WITH DELEGATE OPTION / Now, user U1 assigns the required set of privileges to the role. The role is then assigned to the required subprogram. Note that only roles, and not individual privileges, can be assigned to the schema level subprograms. /*Connect to U1*/ CONN u1/u1 /*Grant SELECT and UPDATE privileges on PRODUCTS to the role*/ GRANT SELECT, UPDATE ON PRODUCTS TO prod_role / /*Grant role to the procedure*/ GRANT prod_role TO PROCEDURE p_inc_price / User U2 tries to execute the procedure again. The procedure is successfully executed which means the value of "Milk" has been increased by 5 units. /*Connect to U2*/ CONN u2/u2 /*Invoke the procedure P_INC_PRICE in a PL/SQL block*/ BEGIN U1.P_INC_PRICE (101,5); COMMIT; END; / PL/SQL procedure successfully completed. User U1 verifies the result with a SELECT query. /*Connect to U1*/ CONN u1/u1 /*Query the table to verify the change*/ SELECT * FROM products / PROD_ID PROD_NAME PROD_CAT PRICE ---------- ---------- ---------- ---------- 101 Milk Dairy 25 102 Cheese Dairy 50 103 Butter Dairy 75 104 Cream Dairy 80 105 Curd Dairy 25 Miscellaneous PL/SQL enhancements Besides the preceding key features, there are a lot of new features in Oracle 12c. The list of features is as follows: An invoker rights function can be result-cached—until Oracle 11g, only the definers' programs were allowed to cache their results. Oracle 12c adds the invoking user's identity to the result cache to make it independent of the definer. The compilation parameter PLSQL_DEBUG has been deprecated. Two conditional compilation inquiry directives $$PLSQL_UNIT_OWNER and $$PLSQL_UNIT_TYPE have been implemented. Summary This chapter covers the top rated and new features of Oracle 12c SQL and PL/SQL as well as some miscellaneous PL/SQL enhancements. This chapter covers the top rated and new features of Oracle 12c SQL and PL/SQL as well as some miscellaneous PL/SQL enhancements. Further resources on this subject: What is Oracle Public Cloud? [article] Oracle APEX 4.2 reporting [article] Oracle E-Business Suite with Desktop Integration [article]

Sometimes your Ionic app needs to tell the user something really important. It might be so important that you want to be sure they see it before your app does anything else. For this you can usually use an Ionic popup. Popups are a nice, clean way to do something similar to the browser's alert, with none of its bad parts. But popups have their limitations. Their design isn't very flexible. It has a title region on the top, a message region in the middle and a button region at the bottom, all on a nice white canvas, centered in the middle of the display. If you want something different, such as no buttons, no white background, just a dark screen with a message and a picture, then a popup is not for you. What can you use instead? In this post you will learn how to use Ionic's loading overlay in combination with its gestures to build an alternative modal to the popup, which is more flexible in its design. The Ionic loading overlay usually just shows a line of text or an ionic or some combination of the two. It is meant to let the user know that the app is loading in a resource. But like many things in Ionic it has a flexible design which allows us to use it for a slightly different purpose. The template property of the overlay can hold pretty much any HTML we give it. For our example app we will build upon the Ionic sidemenu starter template. We don't actually need to add much code to the starter template (we will use beta 14 of Ionic). First, we need to add a few references to the playlist controller. We need the Angular $timeout and the Ionic $ionicLoading and $ionicGesture, so we add their names to the parameters. We also add a variable, showingOverlay, which tells us if the overlay is currently being displayed and set it to false. if (!showingOverlay) { showingOverlay = true; console.log("First Pass Through - show the overlay"); $ionicLoading.show({ template: ' <h2>A message from cat central</h2> <img src="img/cat.jpg" alt="" width="256" height="256" /> <div>Stop playing angry birds and feed me.</div> ', content: 'Loading', animation: 'fade-in', showBackdrop: true, maxWidth: 300, showDelay: 0 }); } To display the loading overlay we call its show method. In the options that we pass to it, we set the template to the HTML we wish to display. In our case we include an image tag that points to a picture in our 'img' directory. If you'd like, you can use the templateUrl property instead of the template property. Don't forget to set the maxWidth property to something reasonable. It must be big enough to hold your content but not too big. You can also set other properties to your liking. In the example I've set the showBackdrop to true and the showDelay to zero. Now that we have our overlay displaying to the user how do we get rid of it? Good question. $ionicLoading blocks users from interacting with the page until we want them to. In order to interact with the overlay, we will have to hook it with a gesture event. Ionic comes with a complete set of gesture events for mobile devices. Most of the time you don't need to use them directly since Ionic widgets do that work for you, but in this example we will need to dig a bit deeper. When the $ionicLoading overlay is displayed it has no id but it does have a class of 'loading'. Once we find it, we can listen for the user to tap it, then dismiss the overlay and unbind the event. $timeout(function () { if (showingOverlay) { // we don't need anything fancy here, we use the DOM's query selector to find the loading div var element = angular.element(document.querySelector('.loading')), // and then we hook the tapGesture = $ionicGesture.on('tap', function (evt) { console.log("TAP RECEIVED!!!"); $ionicLoading.hide(); $ionicGesture.off(tapGesture, 'tap'); showingOverlay = false; }, element); } }, 50); Why have we wrapped the code in an Angular $timeout? We wrap the code in a $timeout because if we look for the overlay on the page immediately, it won't be there. The DOM won't have a chance to render it until our code has given control back to the browser. So we place the code in a timeout which lets the browser render the overlay. Once the timeout has elapsed, we can find overlay element on the page. Also note, we don't use jQuery, and Angular doesn't have a way to find an element by class or id. We instead use the document.querySelector method, which is in all of the devices which Ionic supports. Once the user has tapped the overlay, we need to do some clean up. First, we hide the overlay. Then we turn off the tap event. And finally we set the flag saying whether or not we are displaying the overlay back to false. To be honest this code really belongs in a directive and this is something I will probably do later. But it works and it neatly solves an issue I was having in my current project. If you want to check out the code for yourself it is on my GitHub repo at: https://github.com/Rockncoder/loading. About the author Troy Miles, aka the Rockncoder, began writing games in assembly language for early computers like the Apple II, Vic20, C64, and the IBM PC over 35 years ago. Currently he fills his days writing web apps for a Southern California based automotive valuation and information company. Nights and weekends he can usually be found writing cool apps for mobile and web or teaching other developers how to do so. He likes to post interesting code nuggets on his blog: http://therockncoder.com and videos on his YouTube channel: https://www.youtube.com/user/rockncoder. He can be reached at rockncoder@gmail.com

In this article by John Horton, the author of Android Game Programming by Example, we will look at developing the home screen UI for our game. (For more resources related to this topic, see here.) Creating the project Fire up Android Studio and create a new project by following these steps. On the welcome page of Android Studio, click on Start a new Android Studio project. In the Create New Project window shown next, we need to enter some basic information about our app. These bits of information will be used by Android Studio to determine the package name. In the following image, you can see the Edit link where you can customize the package name if required. If you will be copy/pasting the supplied code into your project, then use C1 Tappy Defender for the Application name field and gamecodeschool.com in the Company Domain field as shown in the following screenshot: Click on the Next button when you're ready. When asked to select the form factors your app will run on, we can accept the default settings (Phone and Tablet). So click on Next again. On the Add an activity to mobile dialog, just click on Blank Activity followed by the Next button. On the Choose options for your new file dialog, again we can accept the default settings because MainActivity seems like a good name for our main Activity. So click on the Finish button. What we did Android Studio has built the project and created a number of files, most of which you will see and edit during the course of building this game. As mentioned earlier, even if you are just copying and pasting the code, you need to go through this step because Android Studio is doing things behind the scenes to make your project work. Building the home screen UI The first and simplest part of your Tappy Defender game is the home screen. All you need is a neat picture with a scene about the game, a high score, and a button to start the game. The finished home screen will look a bit like this: When you built the project, Android Studio opens up two files ready for you to edit. You can see them as tabs in the following Android Studio UI designer. The files (and tabs) are MainActivity.java and activity_main.xml: The MainActivity.java file is the entry point to your game, and you will see this in more detail soon. The activity_main.xml file is the UI layout that your home screen will use. Now, you can go ahead and edit the activity_main.xml file, so it actually looks like your home screen should. First of all, your game will be played with the Android device in landscape mode. If you change your UI preview window to landscape, you will see your progress more accurately. Look for the button shown in the next image. It is just preceding the UI preview: Click on the button shown in the preceding screenshot, and your UI preview will switch to landscape like this: Make sure activity_main.xml is open by clicking on its tab. Now, you will set in a background image. You can use your own. Add your chosen image to the drawable folder of the project in Android Studio. In the Properties window of the UI designer, find and click on the background property as shown in the next image: Also, in the previous image the button labelled ... is outlined. It is just to the right of the background property. Click on that ... button and browse to and select the background image file that you will be using. Next, you need a TextView widget that you will use to display the high score. Note that there is already a TextView widget on the layout. It says Hello World. You will modify this and use it for your high score. Left click on and drag the TextView to where you want it. You can copy me if you intend using the supplied background or put it where it looks best with your background. Next, in the Properties window, find and click on the id property. Enter textHighScore. You can also edit the text property to say High Score: 99999 or similar so that the TextView looks the part. However, this isn't necessary because your Java code will take care of this later. Now, you will drag a button from the widget palette as shown in the following screenshot: Drag it to where it looks good on your background. You can copy me if using the supplied background or put it where it looks best with your background. What we did You now have a cool background with neatly arranged widgets (a TextView and a Button) for your home screen. You can add functionality via Java code to the Button widget next. Revisit the TextView for the player's high score. The important point is that both the widgets have been assigned a unique ID that you can use to reference and manipulate in your Java code. Coding the functionality Now, you have a simple layout for your game home screen. Now, you need to add the functionality that will allow the player to click on the Play button to start the game. Click on the tab for the MainActivity.java file. The code that was automatically generated for us is not exactly what we need. Therefore, we will start again as it is simpler and quicker than tinkering with what is already there. Delete the entire contents of the MainActivity.java file except the package name and enter the following code in it. Of course, your package name may be different. package com.gamecodeschool.c1tappydefender;import android.app.Activity;import android.os.Bundle;public class MainActivity extends Activity{    // This is the entry point to our game    @Override    protected void onCreate(Bundle savedInstanceState) {        super.onCreate(savedInstanceState);                //Here we set our UI layout as the view        setContentView(R.layout.activity_main);    }} The mentioned code is the current contents of your main MainActivity class and the entry point of your game, the onCreate method. The line of code that begins with setContentView... is the line that loads our UI layout from activity_main.xml to the players screen. We can run the game now and see our home screen. Now, let's handle the Play button on our home screen. Add the two highlighted lines of the following code into the onCreate method just after the call to setContentView(). The first new line creates a new Button object and gets a reference to Button in our UI layout. The second line is the code to listen for clicks on the button. //Here we set our UI layout as the viewsetContentView(R.layout.activity_main);// Get a reference to the button in our layoutfinal Button buttonPlay =   (Button)findViewById(R.id.buttonPlay);// Listen for clicksbuttonPlay.setOnClickListener(this); Note that you have a few errors in your code. You can resolve these errors by holding down the Alt keyboard key and then pressing Enter. This will add an import directive for the Button class. You still have one error. You need to implement an interface so that your code listens to the button clicks. Modify the MainActivity class declaration as highlighted: public class MainActivity extends Activity         implements View.OnClickListener{ When you implement the onClickListener interface, you must also implement the onClick method. This is where you will handle what happens when a button is clicked. You can automatically generate the onClick method by right-clicking somewhere after the onCreate method, but within the MainActivity class, and navigating to Generate | Implement methods | onClick(v:View):void. You also need to have Android Studio add another import directive for Android.view.View. Use the Alt | Enter keyboard combination again. You can now scroll to near the bottom of the MainActivity class and see that Android Studio has implemented an empty onClick method for you. You should have no errors in your code at this point. Here is the onClick method: @Overridepublic void onClick(View v) {  //Our code goes here} As you only have one Button object and one listener, you can safely assume that any clicks on your home screen are the player pressing your Play button. Android uses the Intent class to switch between activities. As you need to go to a new activity when the Play button is clicked, you will create a new Intent object and pass in the name of your future Activity class, GameActivity to its constructor. You can then use the Intent object to switch activities. Add the following code to the body of the onClick method: // must be the Play button.// Create a new Intent objectIntent i = new Intent(this, GameActivity.class);// Start our GameActivity class via the IntentstartActivity(i);// Now shut this activity downfinish(); Once again, you have errors in your code because you need to generate a new import directive, this time for the Intent class so use the Alt | Enter keyboard combination again. You still have one error in your code. This is because your GameActivity class does not exist yet. You will now solve this problem. Creating GameActivity You have seen that when the player clicks on the Play button, main activity will close and game activity will begin. Therefore, you need to create a new activity called GameActivity that will be where your game actually executes. From the main menu, navigate to File | New | Activity | Blank Activity. In the Choose options for your new file dialog, change the Activity name field to GameActivity. You can accept all the other default settings from this dialog, so click on Finish. As you did with your MainActivity class, you will code this class from scratch. Therefore, delete the entire code content from GameActivity.java. What we did Android Studio has generated two more files for you and done some work behind the scenes that you will investigate soon. The new files are GameActivity.java and activity_game.xml. They are both automatically opened for you in two new tabs, in the same place as the other tabs above the UI designer. You will never need activity_game.xml because you will build a dynamically generated game view, not a static UI. Feel free to close that now or just ignore it. You will come back to the GameActivity.java file, when you start to code your game for real. Configuring the AndroidManifest.xml file We briefly mentioned that when we create a new project or a new activity, Android Studio does more than just creating two files for us. This is why we create new projects/activities the way we do. One of the things going on behind the scenes is the creation and modification of the AndroidManifest.xml file in the manifests directory. This file is required for your app to work. Also, it needs to be edited to make your app work the way you want it to. Android Studio has automatically configured the basics for you, but you will now do two more things to this file. By editing the AndroidManifest.xml file, you will force both of your activities to run with a full screen, and you will also lock them to a landscape layout. Let's make these changes here: Open the manifests folder now, and double click on the AndroidManifest.xml file to open it in the code editor. In the AndroidManifest.xml file, find the following line of code:android:name=".MainActivity" Immediately following it, type or copy and paste these two lines to make MainActivity run full screen and lock it in the landscape orientation:android:theme="@android:style/Theme.NoTitleBar.Fullscreen"android:screenOrientation="landscape" In the AndroidManifest.xml file, find the following line of code:android:name=".GameActivity" Immediately following it, type or copy and paste these two lines to make GameActivity run full screen and lock it in the landscape orientation: android:theme="@android:style/Theme.NoTitleBar.Fullscreen"android:screenOrientation="landscape" What you did You have now configured both activities from your game to be full screen. This gives a much more pleasing appearance for your player. In addition, you have disabled the player's ability to affect your game by rotating their Android device. Continue building on what you've learnt so far with Android Game Programming by Example! Learn to implement the game rules, game mechanics, and game objects such as guns, life, money; and of course, the enemy. You even get to control a spaceship!

In this article by Salil Kapur and Nisarg Thakkar, authors of the book Mastering OpenCV Android Application Programming, we will look at the broader aspects of object tracking in Videos. Object tracking is one of the most important applications of computer vision. It can be used for many applications, some of which are as follows: Human–computer interaction: We might want to track the position of a person's finger and use its motion to control the cursor on our machines Surveillance: Street cameras can capture pedestrians' motions that can be tracked to detect suspicious activities Video stabilization and compression Statistics in sports: By tracking a player's movement in a game of football, we can provide statistics such as distance travelled, heat maps, and so on In this article, you will learn the following topics: Optical flow Image Pyramids (For more resources related to this topic, see here.) Optical flow Optical flow is an algorithm that detects the pattern of the motion of objects, or edges, between consecutive frames in a video. This motion may be caused by the motion of the object or the motion of the camera. Optical flow is a vector that depicts the motion of a point from the first frame to the second. The optical flow algorithm works under two basic assumptions: The pixel intensities are almost constant between consecutive frames The neighboring pixels have the same motion as the anchor pixel We can represent the intensity of a pixel in any frame by f(x,y,t). Here, the parameter t represents the frame in a video. Let's assume that, in the next dt time, the pixel moves by (dx,dy). Since we have assumed that the intensity doesn't change in consecutive frames, we can say: f(x,y,t) = f(x + dx,y + dy,t + dt) Now we take the Taylor series expansion of the RHS in the preceding equation: Cancelling the common term, we get: Where . Dividing both sides of the equation by dt we get: This equation is called the optical flow equation. Rearranging the equation we get: We can see that this represents the equation of a line in the (u,v) plane. However, with only one equation available and two unknowns, this problem is under constraint at the moment. The Horn and Schunck method By taking into account our assumptions, we get: We can say that the first term will be small due to our assumption that the brightness is constant between consecutive frames. So, the square of this term will be even smaller. The second term corresponds to the assumption that the neighboring pixels have similar motion to the anchor pixel. We need to minimize the preceding equation. For this, we differentiate the preceding equation with respect to u and v. We get the following equations: Here, and  are the Laplacians of u and v respectively. The Lucas and Kanade method We start off with the optical flow equation that we derived earlier and noticed that it is under constrained as it has one equation and two variables: To overcome this problem, we make use of the assumption that pixels in a 3x3 neighborhood have the same optical flow: We can rewrite these equations in the form of matrices, as shown here: This can be rewritten in the form: Where: As we can see, A is a 9x2 matrix, U is a 2x1 matrix, and b is a 9x1 matrix. Ideally, to solve for U, we just need to multiply by A-1on both sides of the equation. However, this is not possible, as we can only take the inverse of square matrices. Thus, we try to transform A into a square matrix by first multiplying the equation by AT on both sides of the equation: Now is a square matrix of dimension 2x2. Hence, we can take its inverse: On solving this equation, we get: This method of multiplying the transpose and then taking an inverse is called pseudo-inverse. This equation can also be obtained by finding the minimum of the following equation: According to the optical flow equation and our assumptions, this value should be equal to zero. Since the neighborhood pixels do not have exactly the same values as the anchor pixel, this value is very small. This method is called Least Square Error. To solve for the minimum, we differentiate this equation with respect to u and v, and equate it to zero. We get the following equations: Now we have two equations and two variables, so this system of equations can be solved. We rewrite the preceding equations as follows: So, by arranging these equations in the form of a matrix, we get the same equation as obtained earlier: Since, the matrix A is now a 2x2 matrix, it is possible to take an inverse. On taking the inverse, the equation obtained is as follows: This can be simplified as: Solving for u and v, we get: Now we have the values for all the , , and . Thus, we can find the values of u and v for each pixel. When we implement this algorithm, it is observed that the optical flow is not very smooth near the edges of the objects. This is due to the brightness constraint not being satisfied. To overcome this situation, we use image pyramids. Checking out the optical flow on Android To see the optical flow in action on Android, we will create a grid of points over a video feed from the camera, and then the lines will be drawn for each point that will depict the motion of the point on the video, which is superimposed by the point on the grid. Before we begin, we will set up our project to use OpenCV and obtain the feed from the camera. We will process the frames to calculate the optical flow. First, create a new project in Android Studio. We will set the activity name to MainActivity.java and the XML resource file as activity_main.xml. Second, we will give the app the permissions to access the camera. In the AndroidManifest.xml file, add the following lines to the manifest tag: <uses-permission android_name="android.permission.CAMERA" /> Make sure that your activity tag for MainActivity contains the following line as an attribute: android:screenOrientation="landscape" Our activity_main.xml file will contain a simple JavaCameraView. This is a custom OpenCV defined layout that enables us to access the camera frames and processes them as normal Mat objects. The XML code has been shown here: <LinearLayout       android_layout_width="match_parent"    android_layout_height="match_parent"    android_orientation="horizontal">      <org.opencv.android.JavaCameraView        android_layout_width="fill_parent"        android_layout_height="fill_parent"        android_id="@+id/main_activity_surface_view" />   </LinearLayout> Now, let's work on some Java code. First, we'll define some global variables that we will use later in the code: private static final String   TAG = "com.packtpub.masteringopencvandroid.chapter5.MainActivity";      private static final int       VIEW_MODE_KLT_TRACKER = 0;    private static final int       VIEW_MODE_OPTICAL_FLOW = 1;      private int                   mViewMode;    private Mat                   mRgba;    private Mat                   mIntermediateMat;    private Mat                   mGray;    private Mat                   mPrevGray;      MatOfPoint2f prevFeatures, nextFeatures;    MatOfPoint features;      MatOfByte status;    MatOfFloat err;      private MenuItem               mItemPreviewOpticalFlow, mItemPreviewKLT;      private CameraBridgeViewBase   mOpenCvCameraView; We will need to create a callback function for OpenCV, like we did earlier. In addition to the code we used earlier, we will also enable CameraView to capture frames for processing: private BaseLoaderCallback mLoaderCallback = new BaseLoaderCallback(this) {        @Override        public void onManagerConnected(int status) {            switch (status) {                case LoaderCallbackInterface.SUCCESS:                {                    Log.i(TAG, "OpenCV loaded successfully");                      mOpenCvCameraView.enableView();                } break;                default:                {                    super.onManagerConnected(status);                } break;            }        }    }; We will now check whether the OpenCV manager is installed on the phone, which contains the required libraries. In the onResume function, add the following line of code: OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_2_4_10,   this, mLoaderCallback); In the onCreate() function, add the following line before calling setContentView to prevent the screen from turning off, while using the app: getWindow().addFlags(WindowManager.LayoutParams. FLAG_KEEP_SCREEN_ON); We will now initialize our JavaCameraView object. Add the following lines after setContentView has been called: mOpenCvCameraView = (CameraBridgeViewBase)   findViewById(R.id.main_activity_surface_view); mOpenCvCameraView.setCvCameraViewListener(this); Notice that we called setCvCameraViewListener with the this parameter. For this, we need to make our activity implement the CvCameraViewListener2 interface. So, your class definition for the MainActivity class should look like the following code: public class MainActivity extends Activity   implements CvCameraViewListener2 We will add a menu to this activity to toggle between different examples in this article. Add the following lines to the onCreateOptionsMenu function: mItemPreviewKLT = menu.add("KLT Tracker"); mItemPreviewOpticalFlow = menu.add("Optical Flow"); We will now add some actions to the menu items. In the onOptionsItemSelected function, add the following lines: if (item == mItemPreviewOpticalFlow) {            mViewMode = VIEW_MODE_OPTICAL_FLOW;            resetVars();        } else if (item == mItemPreviewKLT){            mViewMode = VIEW_MODE_KLT_TRACKER;            resetVars();        }          return true; We used a resetVars function to reset all the Mat objects. It has been defined as follows: private void resetVars(){        mPrevGray = new Mat(mGray.rows(), mGray.cols(), CvType.CV_8UC1);        features = new MatOfPoint();        prevFeatures = new MatOfPoint2f();        nextFeatures = new MatOfPoint2f();        status = new MatOfByte();        err = new MatOfFloat();    } We will also add the code to make sure that the camera is released for use by other applications, whenever our application is suspended or killed. So, add the following snippet of code to the onPause and onDestroy functions: if (mOpenCvCameraView != null)            mOpenCvCameraView.disableView(); After the OpenCV camera has been started, the onCameraViewStarted function is called, which is where we will add all our object initializations: public void onCameraViewStarted(int width, int height) {        mRgba = new Mat(height, width, CvType.CV_8UC4);        mIntermediateMat = new Mat(height, width, CvType.CV_8UC4);        mGray = new Mat(height, width, CvType.CV_8UC1);        resetVars();    } Similarly, the onCameraViewStopped function is called when we stop capturing frames. Here we will release all the objects we created when the view was started: public void onCameraViewStopped() {        mRgba.release();        mGray.release();        mIntermediateMat.release();    } Now we will add the implementation to process each frame of the feed that we captured from the camera. OpenCV calls the onCameraFrame method for each frame, with the frame as a parameter. We will use this to process each frame. We will use the viewMode variable to distinguish between the optical flow and the KLT tracker, and have different case constructs for the two: public Mat onCameraFrame(CvCameraViewFrame inputFrame) {        final int viewMode = mViewMode;        switch (viewMode) {            case VIEW_MODE_OPTICAL_FLOW: We will use the gray()function to obtain the Mat object that contains the captured frame in a grayscale format. OpenCV also provides a similar function called rgba() to obtain a colored frame. Then we will check whether this is the first run. If this is the first run, we will create and fill up a features array that stores the position of all the points in a grid, where we will compute the optical flow:                mGray = inputFrame.gray();                if(features.toArray().length==0){                   int rowStep = 50, colStep = 100;                    int nRows = mGray.rows()/rowStep, nCols = mGray.cols()/colStep;                      Point points[] = new Point[nRows*nCols];                    for(int i=0; i<nRows; i++){                        for(int j=0; j<nCols; j++){                            points[i*nCols+j]=new Point(j*colStep, i*rowStep);                        }                    }                      features.fromArray(points);                      prevFeatures.fromList(features.toList());                    mPrevGray = mGray.clone();                    break;                } The mPrevGray object refers to the previous frame in a grayscale format. We copied the points to a prevFeatures object that we will use to calculate the optical flow and store the corresponding points in the next frame in nextFeatures. All of the computation is carried out in the calcOpticalFlowPyrLK OpenCV defined function. This function takes in the grayscale version of the previous frame, the current grayscale frame, an object that contains the feature points whose optical flow needs to be calculated, and an object that will store the position of the corresponding points in the current frame:                nextFeatures.fromArray(prevFeatures.toArray());                Video.calcOpticalFlowPyrLK(mPrevGray, mGray,                    prevFeatures, nextFeatures, status, err); Now, we have the position of the grid of points and their position in the next frame as well. So, we will now draw a line that depicts the motion of each point on the grid:                List<Point> prevList=features.toList(), nextList=nextFeatures.toList();                Scalar color = new Scalar(255);                  for(int i = 0; i<prevList.size(); i++){                    Core.line(mGray, prevList.get(i), nextList.get(i), color);                } Before the loop ends, we have to copy the current frame to mPrevGray so that we can calculate the optical flow in the subsequent frames:                mPrevGray = mGray.clone();                break; default: mViewMode = VIEW_MODE_OPTICAL_FLOW; After we end the switch case construct, we will return a Mat object. This is the image that will be displayed as an output to the user of the application. Here, since all our operations and processing were performed on the grayscale image, we will return this image: return mGray; So, this is all about optical flow. The result can be seen in the following image: Optical flow at various points in the camera feed Image pyramids Pyramids are multiple copies of the same images that differ in their sizes. They are represented as layers, as shown in the following figure. Each level in the pyramid is obtained by reducing the rows and columns by half. Thus, effectively, we make the image's size one quarter of its original size: Relative sizes of pyramids Pyramids intrinsically define reduce and expand as their two operations. Reduce refers to a reduction in the image's size, whereas expand refers to an increase in its size. We will use a convention that lower levels in a pyramid mean downsized images and higher levels mean upsized images. Gaussian pyramids In the reduce operation, the equation that we use to successively find levels in pyramids, while using a 5x5 sliding window, has been written as follows. Notice that the size of the image reduces to a quarter of its original size: The elements of the weight kernel, w, should add up to 1. We use a 5x5 Gaussian kernel for this task. This operation is similar to convolution with the exception that the resulting image doesn't have the same size as the original image. The following image shows you the reduce operation: The reduce operation The expand operation is the reverse process of reduce. We try to generate images of a higher size from images that belong to lower layers. Thus, the resulting image is blurred and is of a lower resolution. The equation we use to perform expansion is as follows: The weight kernel in this case, w, is the same as the one used to perform the reduce operation. The following image shows you the expand operation: The expand operation The weights are calculated using the Gaussian function to perform Gaussian blur. Summary In this article, we have seen how to detect a local and global motion in a video, and how we can track objects. We have also learned about Gaussian pyramids, and how they can be used to improve the performance of some computer vision tasks. Resources for Article: Further resources on this subject: New functionality in OpenCV 3.0 [article] Seeing a Heartbeat with a Motion Amplifying Camera [article] Camera Calibration [article]

In this two part post, you'll learn the basics of how to benchmark and optmize Go code. You'll do this by trying out a few different implementations for calculating a specific element in Pascal's Triangle (a.k.a. the binomial coeffecient). In Part 1 we setup Go, covered the structure of our example, created an interface, and conducted some testing. Here in Part 2 we will cover benchmarking our example for performance, and look at a recursive and bultin implementation. All of the code for these two posts is available on github. Benchmarking for Performance Since we've verified that our implementation works in Part 1, let's benchmark it to see how fast it is. Benchmarking one implementation by itself is not incredibly useful, but soon we will benchmark other implementations too and compare the results. In Go, benchmarks are simply functions that start with the word "Benchmark" and accept *testing.B as an argument. Like testing functions, benchmark functions also need to exist in a file that ends in "_test.go". As we did for our tests, we'll write a utility function that we can use to benchmark any implementation. Add the following to test/benchmark_test.go package test import ( "github.com/albrow/go-benchmark-example/common" "github.com/albrow/go-benchmark-example/implementations" "testing" ) func BenchmarkNaive(b *testing.B) { benchmarkPascaler(b, implementations.Naive) } func benchmarkPascaler(b *testing.B, p common.Pascaler) { for i := 0; i < b.N; i++ { p.Pascal(32, 16) } } We'll use the parameters n=32 and m=16 for our benchmarks. You could actually choose anything you want, but if n gets too big, you'll overflow any int values (which are stored in 64 bits on 64-bit operating systems). If that happens, any implementation that relies on storing numbers as ints will return incorrect results (and may even panic, depending on what math you do). When I tested the limit, overflow first started happening around n=67, m=33. You can run the benchmark by adding the bench flag to the test command: go test ./test -bench .. If it works, your output should look like this: ? github.com/albrow/go-benchmark-example/common [no test files] ? github.com/albrow/go-benchmark-example/implementations [no test files] PASS BenchmarkNaive 100000 15246 ns/op ok github.com/albrow/go-benchmark-example/test 3.970s The important bit is the line that starts with BenchmarkNaive. The first number tells us that this particular benchmark was run 100,000 times. Go will automatically run the benchmark as many times as it needs to until it gets a consistent result. Typically, faster benchmarks will be run more times. The second number tells us that it took an average of 15,246 ns per operation. The exact number can vary widely from machine to machine, which is why it doesn't mean much on its own. Let's write another implementation so we have something to compare it to. Recursive Implementation You may have noticed that our naive implementation was doing more work than it needed to. We don't actually need to construct the entire triangle to figure out a specific element. We actually only need to calculate the elements directly above the one we want (along with the elements directly above those, and so on). What if instead of iterating through from top to bottom, we started at the bottom and iterated upwards, calculating only the elements we needed? It should perform better because we're doing less calculations, right? Let's write an implementation that does exactly that and benchmark it to find out for sure. The idea of moving from bottom to top leads pretty naturally to a recursive implementation. For the base case, we know that the top element is 1. We also know that any element on the edges, where m == 0 or m == n is 1. For all other cases, the element is equal to the sum of the elements directly above it. We'll write the implementation in implementations/recursive.go, and it looks like this: package implementations type recursiveType struct{} var Recursive = &recursiveType{} func (p *recursiveType) Pascal(n, m int) int { if n == 0 || m == 0 || n == m { return 1 } return p.Pascal(n-1, m-1) + p.Pascal(n-1, m) } func (p *recursiveType) String() string { return "Recursive Implementation" } Personally, I think this implementation is much cleaner and easier to understand. Let's test it for correctness by adding a few lines to test/pascal_test.go: func TestRecursive(t *testing.T) { testPascaler(t, implementations.Recursive) } Then run the tests again with go test ./.... If it works you should see the same output as before. Now, let's benchmark this implementation to see how fast it is compared to our first one. Add the following lines to test/benchmark_test.go: func BenchmarkRecursive(b *testing.B) { benchmarkPascaler(b, implementations.Recursive) } Then run the benchmarks with go test ./test -bench .. You might be surprised to find that with the parameters n=32, m=16, the recursive implementation is much, much slower! In fact, if you increase n much more the benchmark will timeout because the recursive implementation takes too long. These were the results on my machine: BenchmarkNaive 100000 15246 ns/op BenchmarkRecursive 1 4005199128 ns/op This shows the recursive implementation took just over 4 seconds, whereas the "naive" implementation took a tiny fraction of a millisecond! The reason this happens has to do with the way Go allocates stack frames for functions. Currently, the Go compiler does not optimize much for recursion (though it might in the future), which means our code results in a lot of extra memory allocations. In languages/compilers that heavily optimize recursive function calls, it's possible that the recursive implementation would be faster, but not so in Go. A naive runtime analysis that didn't take stack frame allocation into account would incorrectly predict that the recursive implementation would be faster. This example illustrates why benchmarking is important whenever you care about performance! Built-in Implementation There are a lot of techniques you could use to optimize the function further. You could try writing an implementation that moves from bottom to top without using recursion. You could take advantage of the symmetry of Pascal's Triangle and cut number of calculations in half. You could even take advantage of memoization or use a lookup table to prevent repeated calculations. But for the sake of time, in this two part series we're just going to try one more implementation. If you are familiar with Combinatorics or Statistics, you might recognize that what we've been trying to calculate with our Pascal function is the equivalent of the binomial coefficient. There are formulas for calcuating the binomial coefficient directly, such as without building a complicated data structure. The most common formula involves factorials and is as follows: binomial(n, k) = n! / (k! * (n-k)!) One unexpected problem with this formula is that the intermediate values in the numerator and denominator can get too large and overflow 64-bit integers. This can happen even when the final result can be represented perfectly with 64 bits. Luckily, there is a package in the Go standard library called "big" for dealing with numbers that are too big to fit in a 64-bit int. That package also has a function for calculating the binomial coeffecient, which is exactly what we're looking for. Here's how we can use the builtin function to create our own implementation in implementations/builtin.go: package implementations import ( "math/big" ) type builtinType struct{} var Builtin = builtinType{} func (p builtinType) Pascal(n, m int) int { z := big.NewInt(0) z.Binomial(int64(n), int64(m)) return int(z.Int64()) } func (p builtinType) String() string { return "Builtin Implementation" } Test it for correctness by adding the following lines to test/pascal_test.go and running the tests again: func TestBuiltin(t *testing.T) { testPascaler(t, implementations.Builtin) } Then, we can compare the performance by adding the following lines to test/benchmark_test.go and running the benchmarks: func BenchmarkBuiltin(b *testing.B) { benchmarkPascaler(b, implementations.Builtin) } On my machine, the builtin implementation was only slightly faster for parameters n=32 and m=16: func BenchmarkBuiltin(b *testing.B) { benchmarkPascaler(b, implementations.Builtin) } Let's see what happens if we increase n and m. You will probably need to skip the recursive implementation by adding the following line inside of the BenchmarkRecursive function: b.Skip("Skipping slower recursive implementation") Here are the results on my machine for n=64, m=32: BenchmarkNaive 50000 40234 ns/op BenchmarkBuiltin 50000 25233 ns/op As you can see, when we increase the difficulty, the builtin implementation really starts to out-perform the others. We can go even higher, but keep in mind that doing so will overflow 64-bit ints. Since the builtin implementation uses the big package, we could work around this restriction. However, the benchmark results for the naive implementation wouldn't mean much since it can't return the correct values. To demonstrate how the builtin implementation performs with higher values of n, here's the results of n=10,000 and m=5,000 on my machine (skipping the other implementations): BenchmarkBuiltin 300 4978329 ns/op That's only 4 ms to compute a really high coeffecient. It appears that for high values of n and m, the builtin implementation is by far the best, not to mention the fact that it is the only one out of the three that returns correct results for higher values of n and m. Conclusion In this two part series, we tested and benchmarked three different implementations of the Pascal function, we learned how to write benchmarking functions, and we found that sometimes the fastest implementation is not necessarily the one you would expect. Benchmarking is critical whenever you are writing performance-sensitive code. Looking for a challenge? See if you can come up with an implementation to beat the builtin one. Some ideas are using parallelization, memoization, GPU acceleration, or binding to some optimized c libraries with cgo. Don't forget to test each implementation for correctness. Feel free to send a pull request here if you are able to beat it. Good luck! About the Author Alex Browne is a programmer and entrepreneur with about 4 years of product development experience and 3 years experience working on small startups. He has worked with a wide variety of technologies and has single-handedly built many production-grade applications. He is currently working with two co-founders on an early stage startup exploring ways of applying machine learning and computer vision to the manufacturing industry.His favorite programming language is Go.

In the past years, many types of attacks on the WEP protocol have been undertaken. Being successful with such an attack is an important milestone for anyone who wants to undertake penetration tests of wireless networks. In this article by Marco Alamanni, the author of Kali Linux Wireless Penetration Testing Essentials, we will take a look at the basics and the most common types of WEP protocols. What is the WEP protocol? The WEP protocol was introduced with the original 802.11 standard as a means to provide authentication and encryption to wireless LAN implementations. It is based on the Rivest Cipher 4 (RC4) stream cypher with a Pre-shared Secret Key (PSK) of 40 or 104 bits, depending on the implementation. A 24-bit pseudorandom Initialization Vector (IV) is concatenated with the pre-shared key to generate the per-packet keystream used by RC4 for the actual encryption and decryption process. Thus, the resulting keystream could be 64 or 128 bits long. In the encryption phase, the keystream is encrypted with the XOR cypher with the plaintext data to obtain the encrypted data. While in the decryption phase, the encrypted data is XOR-encrypted with the keystream to obtain the plaintext data. The encryption process is shown in the following diagram: Attacks against WEP and why do they occur? WEP is an insecure protocol and has been deprecated by the Wi-Fi Alliance. It suffers from various vulnerabilities related to the generation of the keystreams, to the use of IVs (initialization vectors), and to the length of the keys. The IV is used to add randomness to the keystream, trying to avoid the reuse of the same keystream to encrypt different packets. This purpose has not been accomplished in the design of WEP because the IV is only 24 bits long (with 2^24 =16,777,216 possible values) and it is transmitted in clear text within each frame. Thus, after a certain period of time (depending on the network traffic), the same IV and consequently the same keystream will be reused, allowing the attacker to collect the relative cypher texts and perform statistical attacks to recover plain texts and the key. FMS attacks on WEP The first well-known attack against WEP was the Fluhrer, Mantin, and Shamir (FMS) attack back in 2001. The FMS attack relies on the way WEP generates the keystreams and on the fact that it also uses weak IV to generate weak keystreams, making it possible for an attacker to collect a sufficient number of packets encrypted with these keys, to analyze them, and recover the key. The number of IVs to be collected to complete the FMS attack is about 250,000 for 40-bit keys and 1,500,000 for 104-bit keys. The FMS attack has been enhanced by Korek, improving its performance. Andreas Klein found more correlations between the RC4 keystream and the key than the ones discovered by Fluhrer, Mantin, and Shamir, which can be used to crack the WEP key. PTW attacks on WEP In 2007, Pyshkin, Tews, and Weinmann (PTW) extended Andreas Klein's research and improved the FMS attack, significantly reducing the number of IVs needed to successfully recover the WEP key. Indeed, the PTW attack does not rely on weak IVs such as the FMS attack does and is very fast and effective. It is able to recover a 104-bit WEP key with a success probability of 50% using less than 40,000 frames and with a probability of 95% with 85,000 frames. The PTW attack is the default method used by Aircrack-ng to crack WEP keys. ARP Request replay attacks on WEP Both FMS and PTW attacks need to collect quite a large number of frames to succeed and can be conducted passively, sniffing the wireless traffic on the same channel of the target AP and capturing frames. The problem is that, in normal conditions, we will have to spend quite a long time to passively collect all the necessary packets for the attacks, especially with the FMS attack. To accelerate the process, the idea is to reinject frames in the network to generate traffic in response so that we can collect the necessary IVs more quickly. A type of frame that is suitable for this purpose is the ARP request because the AP broadcasts it, each time with a new IV. As we are not associated with the AP, if we send frames to it directly, they are discarded and a de-authentication frame is sent. Instead, we can capture ARP requests from associated clients and retransmit them to the AP. This technique is called the ARP Request Replay attack and is also adopted by Aircrack-ng for the implementation of the PTW attack. Find out more to become a master penetration tester by reading Kali Linux Wireless Penetration Testing Essentials