How-To Tutorials

In this article by Kevin Jackson, Cody Bunch, and Egle Sigler, the authors of the book OpenStack Cloud Computing Cookbook - Third Edition, we will cover the following topics: Using OpenStack Dashboard with LBaaS Using OpenStack Dashboard with OpenStack Orchestration (For more resources related to this topic, see here.) Using OpenStack Dashboard with LBaaS The OpenStack Dashboard has the ability to view, create, and edit Load Balancers, add Virtual IPs (VIPs), and add nodes behind a Load Balancer. Dashboard also provides a user interface for creating HA Proxy server Load Balance services for our instances. We do this first by creating load balancing pools and then adding running instances to those pools. In this section, we will create an HTTP Load Balance pool, create a VIP, and configure instances to be part of the pool. The result will be the ability to use the HTTP Load Balancer pool address to send traffic to two instances running Apache. Getting ready Load a web browser, point it to our OpenStack Dashboard address at http://192.168.100.200/, and log in as an admin user. How to do it... First we will create an HTTP Load Balance pool. Creating pools To create a Load Balancer pool for a logged in user, carry out the following steps: To manage Load Balancers within our OpenStack Dashboard, select the Load Balancers tab: This will show available Load Balancer pools. Since we currently do not have any created, click on the Add Pool button in the top-right corner to add a pool. After clicking on the Add Pool button, we are presented with a modal window. Fill out the details to add a new pool: Set name, description, and provider in the modal window. We name our pool web-pool and give an appropriate description. We choose to go with a default provider since we are creating an HA Proxy. Select a subnet for the pool by clicking on the drop-down menu. All of our instances are attached to the private network, so we select 10.200.0.0/24. We select the HTTP protocol, but HTTPS and TCP are also available. This selection will depend on what kind of applications you are running. Select your preferred routing algorithm; we choose the ROUND_ROBIN balancing method. Other options are LEAST_CONNECTIONS, and SOURCE_IP. We leave the Admin State set to UP. Click on the Add button to create a new pool. You should see the new pool created in the pool list: Adding pool members To add instances to the Load Balancer, follow these steps: After adding a pool, you should still be on the Load Balancer page. Click on the Members tab. You should see a list of active members, if you have any, or an empty list: On the Members tab, click on the Add Member button. This will present you with the following menu: We select the pool we just created, web-pool, and specify members or instances that will be part of this pool. Select weights for the members of the pool. In our case, both members of the pool will have equal weights, so we assign the weight as 1. The selected protocol port will be used to access all members of the pool and, since we are using HTTP, we set the port to 80. We set Admin State to UP. Click on the Add button to add members to the pool. Now the member list should contain two newly added nodes: Adding a VIP to the Load Balancer pool Creating a VIP on the external network will allow access to the Load Balance pool and the instances behind it. To create the VIP, carry out the following steps: From the Load Balancer page, select the Pools tab and click on the drop-down arrow next to the Edit Pool button. This will give you a drop-down menu with an option to add a VIP: Click on the Add VIP option. This will present you with the modal window for creating a new VIP: We enter a custom name and description for our VIP. For VIP Subnet, we pick external subnet 192.168.100.0/24, followed by an available IP in that subnet. We choose 192.168.100.12. Enter 80 for Protocol Port, followed by selecting HTTP for Protocol. Set -1 for Connection Limit if you do not wish to have a maximum number of connections for this VIP. Click on the Add button to create the VIP. This will create a new VIP and show the current pool list: Now, when we click on web-pool, we will see the following details: Click on the web-vip link in the details to view the VIP details: You can test this Load Balancer by entering the VIP's IP in a browser. If you selected ROUND_ROBIN for your routing algorithm, each time you refresh your browser it should hit a different node. Deleting the Load Balancer To delete the Load Balancer, we will first need to delete the attached VIP and then delete the pool. From the Load Balancer page, check the Pools tab and click on the drop-down arrow next to the Edit Pool button. This will give you a drop-down menu with an option to delete a VIP: Selecting the Delete VIP drop-down option will give you a warning and ask you to confirm the deletion of the VIP. Click on the Delete VIP button to confirm: After deleting the VIP, now we can delete the pool. From the Load Balancer page's Pools tab, click on the drop-down arrow next to the appropriate Edit Pool button: Select the Delete Pool option from the drop-down list to delete the pool. You will get asked to confirm the deletion. If you are ready to delete the Load Balance pool, click on the Delete Pool button: How it works... We created a Load Balance pool and added two instances with Apache to it. We also created a virtual IP to be used on the external network and assigned it to our pool. To do this, we executed the following steps: Create a pool from the Load Balancer page's Pools tab. Select the subnet to which all the nodes are attached when creating the pool. Add members to the pool. Create a VIP for the pool. Both the pool and VIP can be edited after being created. Additional members can also be added to the pool at a later time. Using OpenStack Dashboard with OpenStack Orchestration Heat is the OpenStack Orchestration engine that enables users to quickly spin up whole environments using templates. Heat templates, known as Heat Orchestration Templates (HOT), are Yet Another Markup Language (YAML) based files. The files describe the resources being used, the type and the size of the instances, the network an instance will be attached to, among other pieces of information required to run that environment. We showed you how to use the Heat command line client. In this section, we will show how to use an existing Heat template file in OpenStack Dashboard to spin up two web servers running Apache, connected to a third instance running HA Proxy. Getting ready Load a web browser, point it to our OpenStack Dashboard address at http://192.168.100.200/ and log in as an admin user. How to do it... First, we will launch stack within our OpenStack Dashboard. Launching stacks To launch a Heat stack for a logged in user, carry out the following steps: To view available Heat stacks within our OpenStack Dashboard, select the Stacks tab under the Orchestration menu: After clicking on the Stacks tab, you will see all running stacks in your environment. In our case, our list is empty: Click on the Launch Stack button to create a new stack. You will see the following window: There are several ways to specify what template source to use in a stack: File, Direct Input, or URL. Choose which option is the most convenient for you. For our example, you can either use the URL directly or upload a file. The template file can be downloaded from https://raw.githubusercontent.com/OpenStackCookbook/OpenStackCookbook/master/cookbook.yaml. We will upload files from our system. Just like we downloaded the cookbook.yaml file, we can also download the Environment Source file. In this case, we do not have to use the environment source, but it makes it convenient. The environment file stores the values we would have to enter into the browser manually, but instead loads the values for us on the Launch Stack screen, as shown in step 8. In our example, we are using the environment file that can be downloaded from https://raw.githubusercontent.com/OpenStackCookbook/OpenStackCookbook/master/cookbook-env.yaml. Update the public_net_id, private_net_id, and private_subnet_id fields to match your environment. After selecting the Template Source and Environment Source files, click on Next: Our sample environment file contains the following code: parameters: key_name: demokey image: trusty-image flavor: m1.tiny public_net_id: 5e5d24bd-9d1f-4ed1-84b5-0b7e2a9a233b private_net_id: 25153759-994f-4835-9b13-bf0ec77fb336 private_subnet_id: 4cf2c09c-b3d5-40ed-9127-ec40e5e38343 Clicking on Next will give you a Launch Stack window with all the inputs: Note that most of the inputs in our template are now populated. If you did not specify the environment source file in the previous step, you will need to enter the key_name, image, flavor, public_net_id, private_net_id, and private_subnet_id fields. These fields are specific to each template used. Your templates may have different fields. Enter the stack name and user password for your user. If you are logged in as admin or demo, the password is openstack. Click on the Launch button to start stack creation. If all inputs were correct, you should see your stack being created: After the stack creation finishes and if there were no errors during creation, you will see your stack's status updated to Complete: Viewing stack details After launching a stack, there is a lot of information associated with it, including inputs, outputs, and, in the case of errors, information about why stack creation failed. To view the details of the stack, click on the stack name from the Stacks list. The first available view is Topology: Explore the topology by clicking on the nodes. If the graph does not fully fit or you would like a different perspective, you can drag the graph around the window. The next tab under Stack Detail will provide all of the information that was used in creating the stack: Stack information available in the Overview tab is as follows:    Info    Status    Outputs    Stack parameters    Launch parameters The Resources tab will show all the HEAT resources that were created during stack launch: If there were any errors during stack launch, check this page to see which component's creation failed. The Events tab shows all the events that occurred when the stack was created. This page can also be very helpful in troubleshooting Heat templates: While your Heat stack is running, you can also see how many instances it created under the Compute tab's Instance option. The following is what our instances look like on the Instances page: Note that the test1 instance was not part of the stack creation. All the other VMs were created during the stack launch. Deleting stacks Stack deletion is simple; however, it will delete all resources that were created during stack launch. Follow these steps: To delete a stack, first view the available stacks on the Stacks page: Click on the appropriate Delete Stack button to delete a stack. You will be asked to confirm the deletion: After confirming deletion, all resources associated with the stack will be deleted. How it works... We have used the OpenStack Dashboard to launch, view, and delete Orchestration stacks. We first needed to download a sample HA Proxy Heat Orchestration Template from GitHub. Since we were using an environment file, we also had to modify the appropriate inputs. Your own templates may have different inputs. After launching our HA Proxy stack, we explored its topology, resources, and events. Resources created during stack launch will also be reflected in the rest of your environment. If you are launching new instances, all of them will also be available on the Instance page. Delete and modify resources created during the stack launch only through Orchestration section in the OpenStack dashboard or on the command line. Deleting stacks through the dashboard will delete all associated resources. Unlock the full potential of OpenStack with 'OpenStack Cloud Computing Third Edition' - updated and improved for the latest challenges and opportunities unleashed by cloud.

In this first of several articles Bill Pretty, author of Getting Started with Electronic Projects and Building a Home Security System with BeagleBone, shows us how to use a Zigbee point-to-point network to create our very own remote controlled RC car! What Network are we using? In the final chapter of Getting Started with Electronics Projects, I introduced you to the concept of ZigBee RF transmitters and receivers. We used the Type 1 version of XBee devices, because they were capable of “pin mirroring”. In simple terms, this means that whatever change occurs on the I/O pin of the transmitting device is reflected on the corresponding output pin of the receiving XBee device. This feature allows us to use the receiver XBees in a stand-alone configuration, like we did with the alarm system. The type 1 XBee network is configured as a “Star” network. That means that there is only one ‘master’ and in our case, only one end device. Figure 1 Star Network In the figure above, the controlling XBee is shown in red, while the end device(s) are shown in green. The RC cars that we are going to be using in this article have three ‘basic’ types of steering control. I will refer to them as ‘toy’ cars to differentiate them from higher end ‘model’ cars which tend to come out of the box with servo control and microprocessor control. We will be using toy cars that I got from the local thrift store for less than ten dollars, because the remote control was missing! While the remote would come in handy, it isn’t really necessary, because we will be replacing both ends of the system. The first kind of steering system is the simplest and the cheapest to manufacture. A command from the controller tells the front wheels to turn left or right and the wheels will continue to try and turn in the proper direction as long as they are receiving the command, even though the steering motor has hit the limit of the travel. This type of steering may have a series resistor to prevent the motor from burning out, or it may just rely on the limited current capacity of the battery. When the user centres the control, the motor is turned off and an extension spring returns the wheels to the centre position. This makes our control job really easy. We need four signals/commands; forward, reverse, left and right. The spring does the rest. The second type of steering circuit is a simple single pole, double throw switch; which is connected to the motor. The switch changes position when the motor reaches either the full left or full right position. There is no indication of ‘centre’, so this type of steering still relies on a spring for auto-centring. Figure 2 Switch Feedback The third and most complicated type of steering control tends to be seen in toys with rechargeable batteries, like NiCad or NiMH as well. These types of batteries provide more power than alkaline ‘flash light’ batteries. For that reason a slightly more complex steering system is required. This type of system uses an “analog feedback” servo system. What it is is a standard DC motor connected to a potentiometer; so that when the motor turns so does the pot. The wiper of the pot provides a DC voltage that is proportional to the position of the motor. Figure 3 Analog Feedback Servo The steering control circuit will receive three voltages that we are concerned about; full left, full right and centre. This type of servo requires a comparator circuit or a small microprocessor to detect the three positions. More on comparator circuits in future articles. The following circuit will work with the first two types of steering circuits. The circuit diagrams are shown below. Figure 4 In the circuit above, U2 is half of an SN754410 H-Bridge controller integrated circuit. In order for the motor to move the ‘M1’ and ‘M1/’ signals must be at a different level. A logic 0 on M1/ and a logic 1 on M1 will make the motor turn in one direction and a logic 1 on M1/ and a logic 0 on M1 will make it turn in the opposite direction. While either two logic 1’s or 0’s will cause the motor to stop moving. In this way we have either left/right/centre or forward/reverse/stop control. Figure 5 Figure 5 above, shows the connections to the XBee receiver module. As I said earlier, when an I/O pin on the controller goes low, the corresponding pin on the receiver “mirrors” that input. The outputs M1, M1/, M2 and M2/ are connected to the drive and steering motors. The SN754410 is capable of sourcing or sinking about 1 amp, so AUX-1 and AUX-2 can be used to drive LED’s or other devices. The controller circuit is shown below. The circuit is actually taken from a sample circuit I found on the XBee vendor site: http://examples.digi.com/sensors/joystick-with-xbee/ Figure 6 XBee Transmitter The colours represent the wires which come from the joystick. In this case it is a simple arcade style joystick, which uses four micro switches rather than potentiometers. So when the joystick is moved, one or more inputs to the XBee transmitter are grounded (Logic 0) and this information is sent to the receiver in the car. The joystick, available from Adafruit http://examples.digi.com/sensors/joystick-with-xbee/ is shown below. Figure 7 Adafruit Joystick The AUX-1 input is connected to a simple toggle switch, which in this case is used to control the vehicle’s headlights. This is a very simple controller which is designed to control the first two types of vehicles. A more complex design is required to control more complex vehicles. A single board computer such as an Arduino, RaspberryPi or Beaglebone would be required, so that we can read inputs from the joystick as well as digital and analog inputs sent to it from the vehicle. Summary So that’s it for this blog, I hope you enjoyed reading it. In future editions of this topic, I’ll show you how to configure the XBee modules using the Windows application. This method is a lot easier that the one presented in the example on the Digi site. About the Author Bill began his career in electronics in the early 80’s with a small telecom startup company that would eventually become a large multinational. He left there to pursue a career in commercial aviation in Canada’s north. From there he joined the Ontario Center for Microelectronics, a provincially funded research and development center. Bill left there for a career in the military as a civilian contractor at what was then called Defense Research Establishment Ottawa. That began a career which was to span the next 25 years, and continues today. Over the years Bill has acquired extensive knowledge in the field of technical security and started his own company in 2010. That company is called William Pretty Security Inc. and provides support in the form of research and development, to various law enforcement and private security agencies. Bill has published and presented a number of white papers on the subject of technical security. Bill was also a guest presenter for a number of years at the Western Canada Technical Conference, a law enforcement only conference held every year in western Canada. A selection of these papers is available for download from his web site. www.williamprettysecurity.com If you’re interested in building more of your own projects, then be sure to check out Bill’s titles available now in both print and eBook format! If you’re new to working with microcontrollers be sure to pick up Getting Started with Electronic Projects to start creating a whole host of great projects you can do in a single weekend with LM555, ZigBee, and BeagleBone components! If you’re looking for something more advanced to tinker with, then Bill’s other title - Building a Home Security System with BeagleBone – is perfect for hobbyists looking to make a bigger project!

In this article by Enrique Fernández, Luis Sánchez Crespo, Anil Mahtani, and Aaron Martinez, authors of the book Learning ROS for Robotics Programming - Second Edition, you will see the different levels of filesystems in ROS. (For more resources related to this topic, see here.) When you start to use or develop projects with ROS, you will see that although this concept can sound strange in the beginning, you will become familiar with it with time. Similar to an operating system, an ROS program is divided into folders, and these folders have files that describe their functionalities: Packages: Packages form the atomic level of ROS. A package has the minimum structure and content to create a program within ROS. It may have ROS runtime processes (nodes), configuration files, and so on. Package manifests: Package manifests provide information about a package, licenses, dependencies, compilation flags, and so on. A package manifest is managed with a file called package.xml. Metapackages: When you want to aggregate several packages in a group, you will use metapackages. In ROS Fuerte, this form for ordering packages was called Stacks. To maintain the simplicity of ROS, the stacks were removed, and now, metapackages make up this function. In ROS, there exist a lot of these metapackages; one of them is the navigation stack. Metapackage manifests: Metapackage manifests (package.xml) are similar to a normal package but with an export tag in XML. It also has certain restrictions in its structure. Message (msg) types: A message is the information that a process sends to other processes. ROS has a lot of standard types of messages. Message descriptions are stored in my_package/msg/MyMessageType.msg. Service (srv) types: Service descriptions, stored in my_package/srv/MyServiceType.srv, define the request and response data structures for services provided by each process in ROS. The workspace Basically, the workspace is a folder where we have packages, edit the source files or compile packages. It is useful when you want to compile various packages at the same time and is a good place to have all our developments localized. A typical workspace is shown in the following screenshot. Each folder is a different space with a different role: The Source space: In the Source space (the src folder), you put your packages, projects, clone packages, and so on. One of the most important files in this space is CMakeLists.txt. The src folder has this file because it is invoked by CMake when you configure the packages in the workspace. This file is created with the catkin_init_workspace command. The Build space: In the build folder, CMake and catkin keep the cache information, configuration, and other intermediate files for our packages and projects. The Development (devel) space: The devel folder is used to keep the compiled programs. This is used to test the programs without the installation step. Once the programs are tested, you can install or export the package to share with other developers. You have two options with regard to building packages with catkin. The first one is to use the standard CMake workflow. With this, you can compile one package at a time, as shown in the following commands: $ cmake packageToBuild/ $ make If you want to compile all your packages, you can use the catkin_make command line, as shown in the following commands: $ cd workspace $ catkin_make Both commands build the executables in the build space directory configured in ROS. Another interesting feature of ROS are its overlays. When you are working with a package of ROS, for example, Turtlesim, you can do it with the installed version, or you can download the source file and compile it to use your modified version. ROS permits you to use your version of this package instead of the installed version. This is very useful information if you are working on an upgrade of an installed package. Packages Usually, when we talk about packages, we refer to a typical structure of files and folders. This structure looks as follows: include/package_name/: This directory includes the headers of the libraries that you would need. msg/: If you develop nonstandard messages, put them here. scripts/: These are executable scripts that can be in Bash, Python, or any other scripting language. src/: This is where the source files of your programs are present. You can create a folder for nodes and nodelets or organize it as you want. srv/: This represents the service (srv) types. CMakeLists.txt: This is the CMake build file. package.xml: This is the package manifest. To create, modify, or work with packages, ROS gives us tools for assistance, some of which are as follows: rospack: This command is used to get information or find packages in the system. catkin_create_pkg: This command is used when you want to create a new package. catkin_make: This command is used to compile a workspace. rosdep: This command installs the system dependencies of a package. rqt_dep: This command is used to see the package dependencies as a graph. If you want to see the package dependencies as a graph, you will find a plugin called package graph in rqt. Select a package and see the dependencies. To move between packages and their folders and files, ROS gives us a very useful package called rosbash, which provides commands that are very similar to Linux commands. The following are a few examples: roscd: This command helps us change the directory. This is similar to the cd command in Linux. rosed: This command is used to edit a file. roscp: This command is used to copy a file from a package. rosd: This command lists the directories of a package. rosls: This command lists the files from a package. This is similar to the ls command in Linux. The package.xml file must be in a package, and it is used to specify information about the package. If you find this file inside a folder, probably this folder is a package or a metapackage. If you open the package.xml file, you will see information about the name of the package, dependencies, and so on. All of this is to make the installation and the distribution of these packages easy. Two typical tags that are used in the package.xml file are <build_depend> and <run _depend>. The <build_depend> tag shows what packages must be installed before installing the current package. This is because the new package might use a functionality of another package. Metapackages As we have shown earlier, metapackages are special packages with only one file inside; this file is package.xml. This package does not have other files, such as code, includes, and so on. Metapackages are used to refer to others packages that are normally grouped following a feature-like functionality, for example, navigation stack, ros_tutorials, and so on. You can convert your stacks and packages from ROS Fuerte to Hydro and catkin using certain rules for migration. These rules can be found at http://wiki.ros.org/catkin/migrating_from_rosbuild. In the following screenshot, you can see the content from the package.xml file in the ros_tutorials metapackage. You can see the <export> tag and the <run_depend> tag. These are necessary in the package manifest. If you want to locate the ros_tutorials metapackage, you can use the following command: $ rosstack find ros_tutorials The output will be a path, such as /opt/ros/hydro/share/ros_tutorials. To see the code inside, you can use the following command line: $ vim /opt/ros/hydro/share/ros_tutorials/package.xml Remember that Hydro uses metapackages, not stacks, but the rosstack find command line works to find metapackages. Messages ROS uses a simplified message description language to describe the data values that ROS nodes publish. With this description, ROS can generate the right source code for these types of messages in several programming languages. ROS has a lot of messages predefined, but if you develop a new message, it will be in the msg/ folder of your package. Inside that folder, certain files with the .msg extension define the messages. A message must have two principal parts: fields and constants. Fields define the type of data to be transmitted in the message, for example, int32, float32, and string, or new types that you have created earlier, such as type1 and type2. Constants define the name of the fields. An example of a msg file is as follows: int32 id float32 vel string name In ROS, you can find a lot of standard types to use in messages, as shown in the following table list: Primitive type Serialization C++ Python bool (1) unsigned 8-bit int uint8_t(2) bool int8 signed 8-bit int int8_t int uint8 unsigned 8-bit int uint8_t int(3) int16 signed 16-bit int int16_t int uint16 unsigned 16-bit int uint16_t int int32 signed 32-bit int int32_t int uint32 unsigned 32-bit int uint32_t int int64 signed 64-bit int int64_t long uint64 unsigned 64-bit int uint64_t long float32 32-bit IEEE float float float float64 64-bit IEEE float double float string ascii string (4) std::string string time secs/nsecs signed 32-bit ints ros::Time rospy.Time duration secs/nsecs signed 32-bit ints ros::Duration rospy.Duration A special type in ROS is the header type. This is used to add the time, frame, and so on. This permits you to have the messages numbered, to see who is sending the message, and to have more functions that are transparent for the user and that ROS is handling. The header type contains the following fields: uint32 seq time stamp string frame_id You can see the structure using the following command: $ rosmsg show std_msgs/Header Thanks to the header type, it is possible to record the timestamp and frame of what is happening with the robot. In ROS, there exist tools to work with messages. The rosmsg tool prints out the message definition information and can find the source files that use a message type. In the upcoming sections, we will see how to create messages with the right tools. Services ROS uses a simplified service description language to describe ROS service types. This builds directly upon the ROS msg format to enable request/response communication between nodes. Service descriptions are stored in .srv files in the srv/ subdirectory of a package. To call a service, you need to use the package name, along with the service name; for example, you will refer to the sample_package1/srv/sample1.srv file as sample_package1/sample1. There are tools that exist to perform functions with services. The rossrv tool prints out the service descriptions and packages that contain the .srv files, and finds source files that use a service type. If you want to create a service, ROS can help you with the service generator. These tools generate code from an initial specification of the service. You only need to add the gensrv() line to your CMakeLists.txt file. Summary In this article, we saw the different types of filesystems present in the ROS architecture. Resources for Article: Further resources on this subject: Building robots that can walk [article] Avoiding Obstacles Using Sensors [article] Managing Test Structure with Robot Framework [article]

In this article by Mat Johns, author of the book Getting Started with Hazelcast - Second Edition, we will learn the following topics: Creating and using collection listeners Instance, lifecycle, and cluster membership listeners The partition migration listener Quorum functions and listeners (For more resources related to this topic, see here.) Listening to the goings-on One great feature of Hazelcast is its ability to notify us of the goings-on of our persisted data and the cluster as a whole, allowing us to register an interest in events. The listener concept is borrowed from Java. So, you should feel pretty familiar with it. To provide this, there are a number of listener interfaces that we can implement to receive, process, and handle different types of events—one of which we previously encountered. The following are the listener interfaces: Collection listeners: EntryListener is used for map-based (IMap and MultiMap) events ItemListener is used for flat collection-based (IList, ISet, and IQueue) events MessageListener is used to receive topic events, but as we've seen before, it is used as a part of the standard operation of topics QuorumListener is used for quorum state change events Cluster listeners: DistributedObjectListener is used for the collection, creation, and destruction of events MembershipListener is used for cluster membership events LifecycleListener is used for local node state events MigrationListener is used for partition migration state events The sound of our own data Being notified about data changes can be rather useful as we can make an application-level decision regarding whether the change is important or not and react accordingly. The first interface that we are going to look at is EntryListener. This class will notify us when changes are made to the entries stored in a map collection. If we take a look at the interface, we can see four entry event types and two map-wide events that we will be notified about. EntryListener has also being broken up into a number of individual super MapListener interfaces. So, should we be interested in only a subset of event types, we can implement the appropriate super interfaces as required. Let's take a look at the following code: void entryAdded(EntryEvent<K, V> event);void entryRemoved(EntryEvent<K, V> event);void entryUpdated(EntryEvent<K, V> event);void entryEvicted(EntryEvent<K, V> event);void mapCleared(MapEvent event);void mapEvicted(MapEvent event); Hopefully, the first three are pretty self-explanatory. However, the fourth is a little less clear and in fact, one of the most useful. The entryEvicted method is invoked when an entry is removed from a map non-programmatically (that is, Hazelcast has done it all by itself). This instance will occur in one of the following two scenarios: An entry's TTL has been reached and the entry has been expired The map size, according to the configured policy, has been reached, and the appropriate eviction policy has kicked in to clear out space in the map The first scenario allows us a capability that is very rarely found in data sources—to have our application be told when a time-bound record has expired and the ability to trigger some behavior based on it. For example, we can use it to automatically trigger a teardown operation if an entry is not correctly maintained by a user's interactions. This will allow us to generate an event based on the absence of activity, which is rather useful! Let's create an example MapEntryListener class to illustrate the various events firing off: public class MapEntryListenerimplements EntryListener<String, String> {@Overridepublic void entryAdded(EntryEvent<String, String> event) {System.err.println("Added: " + event);}@Overridepublic void entryRemoved(EntryEvent<String, String> event) {System.err.println("Removed: " + event);}@Overridepublic void entryUpdated(EntryEvent<String, String> event) {System.err.println("Updated: " + event);}@Overridepublic void entryEvicted(EntryEvent<String, String> event) {System.err.println("Evicted: " + event);}@Overridepublic void mapCleared(MapEvent event) {System.err.println("Map Cleared: " + event);}@Overridepublic void mapEvicted(MapEvent event) {System.err.println("Map Evicted: " + event);}} We shall see the various events firing off as expected, with a short 10-second wait for the Berlin entry to expire, which will trigger the eviction event, as follows: Added: EntryEvent {c:capitals} key=GB, oldValue=null, value=Winchester,event=ADDED, by Member [127.0.0.1]:5701 thisUpdated: EntryEvent {c:capitals} key=GB, oldValue=Winchester,value=London, event=UPDATED, by Member [127.0.0.1]:5701 thisAdded: EntryEvent {c:capitals} key=DE, oldValue=null, value=Berlin,event=ADDED, by Member [127.0.0.1]:5701 thisRemoved: EntryEvent {c:capitals} key=GB, oldValue=null, value=London,event=REMOVED, by Member [127.0.0.1]:5701 thisEvicted: EntryEvent {c:capitals} key=DE, oldValue=null, value=Berlin,event=EVICTED, by Member [127.0.0.1]:5701 this We can obviously implement the interface as extensively as possible to service our application, potentially creating no-op stubs should we wish not to handle a particular type of event. Continuously querying The previous example focuses on notifying us of all the entry events. However, what if we were only interested in some particular data? We can obviously filter out our listener to only handle entries that we are interested in. However, it is potentially expensive to have all the events flying around the cluster, especially if our interest lies only in a minority of the potential data. To address this, we can combine capabilities from the map-searching features that we looked at a while back. As and when we register the entry listener to a collection, we can optionally provide a search Predicate that can be used as a filter within Hazelcast itself. We can whittle events down to relevant data before they even reach our listener, as follows: IMap<String, String> capitals = hz.getMap("capitals");capitals.addEntryListener(new MapEntryListener(),new SqlPredicate("name = 'London'"), true); Listeners racing into action One issue with the previous example is that we retrospectively reconfigured the map to feature the listener after it was already in service. To avoid this race condition, we should wire up the listener before the node-entering service. We can do this by registering the listener within the map configuration, as follows: <hazelcast><map name="default"><entry-listeners><entry-listener include-value="true">com.packtpub.hazelcast.listeners.MapEntryListener</entry-listener></entry-listeners></map></hazelcast> However, in both the methods of configuration, we have provided a Boolean flag when registering the listener to the map. This include-value flag allows us to configure the listener when it is invoked, as if we are interested in just the key of the event entry or the entries value as well. The default behavior (true) is to include the value, but in case the use case does not require it, there is a performance benefit of not having to provide it to the listener. So, if the use case does not require this extra data, it will be beneficial to set this flag to false. Keyless collections Though the keyless collections (set, list, and queue) are very similar to map collections, they feature their own interface to define the available events, in this case, ItemListener. It is not as extensive as its map counterpart, featuring just the itemAdded and itemRemoved events, and can be used in the same way, though it only offers visibility of these two event types. Programmatic configuration ahead of time So far, most of the extra configurations that we applied have been done either by customizing the hazelcast.xml file, or retrospectively modifying a collection in the code. However, what if we want to programmatically configure Hazelcast without the race condition that we discovered earlier? Fortunately, there is a way. By creating an instance of the Config class, we can configure the appropriate behavior on it by using a hierarchy that is similar to the XML configuration, but in code. Before passing this configuration object over to the instance creation method, the previous example can be reconfigured to do so, as follows: public static void main(String[] args) {Config conf = new Config();conf.addListenerConfig(new EntryListenerConfig(new MapEntryListener(), false, true));HazelcastInstance hz = Hazelcast.newHazelcastInstance(conf); Events unfolding in the wider world Now that we can determine what is going on with our data within the cluster, we may wish to have a higher degree of visibility of the state of the cluster itself. We can use this either to trigger application-level responses to cluster instability, or provide mechanisms to enable graceful scaling. We are provided with a number of interfaces for different types of cluster activity. All of these listeners can be configured retrospectively, as we have seen in the previous examples. However, in production, it is better to configure them in advance for the same reasons regarding the race condition as the collection listeners. We can do this either by using the hazelcast.xml configuration, or by using the Config class, as follows: <hazelcast><listeners><listener>com.packtpub.hazelcast.MyClusterListener</listener></listeners></hazelcast> The first of these, DistributedObjectListener, simply notifies all the nodes in the cluster as to the new collection objects that are being created or destroyed. Again, let's create a new example listener, ClusterObjectListener, to receive events, as follows: public class ClusterObjectListenerimplements DistributedObjectListener {@Overridepublic void distributedObjectCreated(DistributedObjectEvent event) {System.err.println("Created: " + event);}@Overridepublic void distributedObjectDestroyed(DistributedObjectEvent event) {System.err.println("Destroyed: " + event);}} As these listeners are for cluster-wide events, the example usage of this listener is rather simple. It mainly creates an instance with the appropriate listener registered, as follows: public class ClusterListeningExample {public static void main(String[] args) {Config config = new Config();config.addListenerConfig(new ListenerConfig(new ClusterObjectListener()));HazelcastInstance hz = Hazelcast.newHazelcastInstance(config);}} When using the TestApp console, we can create and destroy some collections, as follows: hazelcast[default] > ns testnamespace: testhazelcast[test] > m.put foo barnullhazelcast[test] > m.destroyDestroyed! The preceding code will produce the following, logging on ALL the nodes that feature the listener: Created: DistributedObjectEvent{eventType=CREATED, serviceName='hz:impl:mapService', distributedObject=IMap{name='test'}} Destroyed: DistributedObjectEvent{eventType=DESTROYED, serviceName='hz:impl:mapService', distributedObject=IMap{name='test'}} The next type of cluster listener is MembershipListener, which notifies all the nodes as to the joining or leaving of a node from the cluster. Let's create another example class, this time ClusterMembershipListener, as follows: public class ClusterMembershipListenerimplements MembershipListener {@Overridepublic void memberAdded(MembershipEvent membershipEvent) {System.err.println("Added: " + membershipEvent);}@Overridepublic void memberRemoved(MembershipEvent membershipEvent) {System.err.println("Removed: " + membershipEvent);}@Overridepublic void memberAttributeChanged(MemberAttributeEventmemberAttributeEvent) {System.err.println("Changed: " + memberAttributeEvent);}} Let's add the following code to the previous example application: conf.addListenerConfig(new ListenerConfig(new   ClusterMembershipListener())); Lastly, we have LifecycleListener, which is local to an individual node and allows the application built on top of Hazelcast to understand its particular node state by being notified as it changes when starting, pausing, resuming, or even shutting down, as follows: public class NodeLifecycleListener implements LifecycleListener {@Overridepublic void stateChanged(LifecycleEvent event) {System.err.println(event);}} Moving data around the place The final listener is very useful as it lets an application know when Hazelcast is rebalancing the data within the cluster. This gives us an opportunity to prevent or even block the shutdown of a node, as we might be in a period of increased data resilience risk because we may be actively moving data around at the time. The interface used in this case is MigrationListener. It will notify the application when the partitions migrate from one node to another and when they complete: public class ClusterMigrationListener implements MigrationListener {@Overridepublic void migrationStarted(MigrationEvent migrationEvent) {System.err.println("Started: " + migrationEvent);}@Overridepublic void migrationCompleted(MigrationEvent migrationEvent) {System.err.println("Completed: " + migrationEvent);}@Overridepublic void migrationFailed(MigrationEvent migrationEvent) {System.err.println("Failed: " + migrationEvent);}} When you are registering this cluster listener in your example application and creating and destroying various nodes, you will see a deluge of events that show the ongoing migrations. The more astute among you may have previously spotted a repartitioning task logging when spinning up the multiple nodes: INFO: [127.0.0.1]:5701 [dev] [3.5] Re-partitioning cluster data... Migration queue size: 271 The previous code indicated that 271 tasks (one migration task for each partition being migrated) have been scheduled to rebalance the cluster. The new listener will now give us significantly more visibility on these events as they occur and hopefully, they will be completed successfully: Started: MigrationEvent{partitionId=98, oldOwner=Member [127.0.0.1]:5701, newOwner=Member [127.0.0.1]:5702 this} Completed: MigrationEvent{partitionId=98, oldOwner=Member [127.0.0.1]:5701, newOwner=Member [127.0.0.1]:5702 this} Started: MigrationEvent{partitionId=99, oldOwner=Member [127.0.0.1]:5701, newOwner=Member [127.0.0.1]:5702 this} Completed: MigrationEvent{partitionId=99, oldOwner=Member [127.0.0.1]:5701, newOwner=Member [127.0.0.1]:5702 this} However, this logging information is overwhelming and actually not all that useful to us. So, let's expand on the listener to try and provide the application with the ability to check whether the cluster is currently migrating data partitions or has recently done so. Let's create a new static class, MigrationStatus, to hold information about cluster migration and help us interrogate it as regards its current status: public abstract class MigrationStatus {private static final Map<Integer, Boolean> MIGRATION_STATE =new ConcurrentHashMap<Integer, Boolean>();private static final AtomicLong LAST_MIGRATION_TIME =new AtomicLong(System.currentTimeMillis());public static void migrationEvent(int partitionId, boolean state) {MIGRATION_STATE.put(partitionId, state);if (!state) {LAST_MIGRATION_TIME.set(System.currentTimeMillis());}}public static boolean isMigrating() {Collection<Boolean> migrationStates= MIGRATION_STATE.values();Long lastMigrationTime = LAST_MIGRATION_TIME.get();// did we recently (< 10 seconds ago) complete a migrationif (System.currentTimeMillis() < lastMigrationTime + 10000) {return true;}// are any partitions currently migratingfor (Boolean partition : migrationStates) {if (partition) return true;}// otherwise we're not migratingreturn false;}} Then, we will update the listener to pass through the appropriate calls in response to the events coming into it, as follows: @Overridepublic void migrationStarted(MigrationEvent migrationEvent) {MigrationStatus.migrationEvent(migrationEvent.getPartitionId(), true);}@Overridepublic void migrationCompleted(MigrationEvent migrationEvent) {MigrationStatus.migrationEvent(migrationEvent.getPartitionId(), false);}@Overridepublic void migrationFailed(MigrationEvent migrationEvent) {System.err.println("Failed: " + migrationEvent);MigrationStatus.migrationEvent(migrationEvent.getPartitionId(), false);} Finally, let's add a loop to the example application to print out the migration state over time, as follows: public static void main(String[] args) throws Exception {Config conf = new Config();conf.addListenerConfig(new ListenerConfig(new ClusterMembershipListener()));conf.addListenerConfig(new ListenerConfig(new MigrationStatusListener()));HazelcastInstance hz = Hazelcast.newHazelcastInstance(conf);while(true) {System.err.println("Is Migrating?: " + MigrationStatus.isMigrating());Thread.sleep(5000);}} When starting and stopping various nodes, we should see each node detect the presence of the rebalance occurring, but it passes by quite quickly. It is in these small, critical periods of time when data is being moved around that resilience is mostly at risk, albeit depending on the configured numbers of backup, the risk could potentially be quite small. Added: MembershipEvent {member=Member [127.0.0.1]:5703,type=added}Is Migrating?: trueIs Migrating?: trueIs Migrating?: false Extending quorum We previously saw how we can configure a simple cluster health check to ensure that a given number of nodes were present to support the application. However, should we need more detailed control over the quorum definition beyond a simple node count check, we can create our own quorum function that will allow us to programmatically define what it means to be healthy. This can be as simple or as complex as what the application requires. In the following example, we sourced an expected cluster size (probably from a suitable location than a hard-coded) and dynamically checked whether a majority of the nodes are present: public class QuorumExample {public static void main(String[] args) throws Exception {QuorumConfig quorumConf = new QuorumConfig();quorumConf.setName("atLeastTwoNodesWithMajority");quorumConf.setEnabled(true);quorumConf.setType(QuorumType.WRITE);final int expectedClusterSize = 5;quorumConf.setQuorumFunctionImplementation(new QuorumFunction() {@Overridepublic boolean apply(Collection<Member> members) {return members.size() >= 2&& members.size() > expectedClusterSize / 2;}});MapConfig mapConf = new MapConfig();mapConf.setName("default");mapConf.setQuorumName("atLeastTwoNodesWithMajority");Config conf = new Config();conf.addQuorumConfig(quorumConf);conf.addMapConfig(mapConf);HazelcastInstance hz = Hazelcast.newHazelcastInstance(conf);new ConsoleApp(hz).start(args);}} We can also create a listener for the quorum health check so that we can be notified when the state of the quorum changes, as follows: public class ClusterQuorumListener implements QuorumListener {@Overridepublic void onChange(QuorumEvent quorumEvent) {System.err.println("Changed: " + quorumEvent);}} Let's attach the new listener to the appropriate configuration, as follows: quorumConf.addListenerConfig(new QuorumListenerConfig(new   ClusterQuorumListener())); Summary Hazelcast allows us to be a first-hand witness to a lot of internal state information. By registering the listeners so that they can be notified as the events occur, we can further enhance an application not only in terms of its functionality, but also with respect to its resilience. By allowing the application to know when and what events are unfolding underneath it, we can add defensiveness to it, embracing the dynamic and destroyable nature of the ephemeral approaches towards applications and infrastructure. Resources for Article: Further resources on this subject: What is Hazelcast? [Article] Apache Solr and Big Data – integration with MongoDB [Article] Introduction to Apache ZooKeeper [Article]

In this article by Saurabh K. Gupta, author of the book Oracle Advanced PL/SQL Developer Professional Guide, Second Edition you will learn new features in Oracle 12c SQL and PL/SQL. (For more resources related to this topic, see here.) Oracle 12c SQL and PL/SQL new features SQL is the most widely used data access language while PL/SQL is an exigent language that can integrate seamlessly with SQL commands. The biggest benefit of running PL/SQL is that the code processing happens natively within the Oracle Database. In the past, there have been debates and discussions on server side programming while the client invokes the PL/SQL routines to perform a task. The server side programming approach has many benefits. It reduces the network round trips between the client and the database. It reduces the code size and eases the code portability because PL/SQL can run on all platforms, wherever Oracle Database is supported. Oracle Database 12c introduces many language features and enhancements which are focused on SQL to PL/SQL integration, code migration, and ANSI compliance. This section discusses the SQL and PL/SQL new features in Oracle database 12c. IDENTITY columns Oracle Database 12c Release 1 introduces identity columns in the tables in compliance with the American National Standard Institute (ANSI) SQL standard. A table column, marked as IDENTITY, automatically generate an incremental numeric value at the time of record creation. Before the release of Oracle 12c, developers had to create an additional sequence object in the schema and assign its value to the column. The new feature simplifies code writing and benefits the migration of a non-Oracle database to Oracle. The following script declares an identity column in the table T_ID_COL: /*Create a table for demonstration purpose*/ CREATE TABLE t_id_col (id NUMBER GENERATED AS IDENTITY, name VARCHAR2(20)) / The identity column metadata can be queried from the dictionary views USER_TAB_COLSand USER_TAB_IDENTITY_COLS. Note that Oracle implicitly creates a sequence to generate the number values for the column. However, Oracle allows the configuration of the sequence attributes of an identity column. The custom sequence configuration is listed under IDENTITY_OPTIONS in USER_TAB_IDENTITY_COLS view: /*Query identity column information in USER_TAB_COLS*/ SELECT column_name, data_default, user_generated, identity_column FROM user_tab_cols WHERE table_name='T_ID_COL' / COLUMN_NAME DATA_DEFAULT USE IDE -------------- ------------------------------ --- --- ID "SCOTT"."ISEQ$$_93001".nextval YES YES NAME YES NO Let us check the attributes of the preceding sequence that Oracle has implicitly created. Note that the query uses REGEXP_SUBSTR to print the sequence configuration in multiple rows: /*Check the sequence configuration from USER_TAB_IDENTITY_COLS view*/ SELECT table_name,column_name, generation_type, REGEXP_SUBSTR(identity_options,'[^,]+', 1, LEVEL) identity_options FROM user_tab_identity_cols WHERE table_name = 'T_ID_COL' CONNECT BY REGEXP_SUBSTR(identity_options,'[^,]+',1,level) IS NOT NULL / TABLE_NAME COLUMN_NAME GENERATION IDENTITY_OPTIONS ---------- ---------------------- --------------------------------- T_ID_COL ID ALWAYS START WITH: 1 T_ID_COL ID ALWAYS INCREMENT BY: 1 T_ID_COL ID ALWAYS MAX_VALUE: 9999999999999999999999999999 T_ID_COL ID ALWAYS MIN_VALUE: 1 T_ID_COL ID ALWAYS CYCLE_FLAG: N T_ID_COL ID ALWAYS CACHE_SIZE: 20 T_ID_COL ID ALWAYS ORDER_FLAG: N 7 rows selected While inserting data in the table T_ID_COL, do not include the identity column as its value is automatically generated: /*Insert test data in the table*/ BEGIN INSERT INTO t_id_col (name) VALUES ('Allen'); INSERT INTO t_id_col (name) VALUES ('Matthew'); INSERT INTO t_id_col (name) VALUES ('Peter'); COMMIT; END; / Let us check the data in the table. Note the identity column values: /*Query the table*/ SELECT id, name FROM t_id_col / ID NAME ----- -------------------- 1 Allen 2 Matthew 3 Peter The sequence created under the covers for identity columns is tightly coupled with the column. If a user tries to insert a user-defined input for the identity column, the operation throws an exception ORA-32795: INSERT INTO t_id_col VALUES (7,'Steyn'); insert into t_id_col values (7,'Steyn') * ERROR at line 1: ORA-32795: cannot insert into a generated always identity column Default column value to a sequence in Oracle 12c Oracle Database 12c allows developers to default a column directly to a sequence‑generated value. The DEFAULT clause of a table column can be assigned to SEQUENCE.CURRVAL or SEQUENCE.NEXTVAL. The feature will be useful while migrating non-Oracle data definitions to Oracle. The DEFAULT ON NULL clause Starting with Oracle Database 12c, a column can be assigned a default non-null value whenever the user tries to insert NULL into the column. The default value will be specified in the DEFAULT clause of the column with a new ON NULL extension. Note that the DEFAULT ON NULL cannot be used with an object type column. The following script creates a table t_def_cols. A column ID has been defaulted to a sequence while the column DOJ will always have a non-null value: /*Create a sequence*/ CREATE SEQUENCE seq START WITH 100 INCREMENT BY 10 / /*Create a table with a column defaulted to the sequence value*/ CREATE TABLE t_def_cols ( id number default seq.nextval primary key, name varchar2(30), doj date default on null '01-Jan-2000' ) / The following PL/SQL block inserts the test data: /*Insert the test data in the table*/ BEGIN INSERT INTO t_def_cols (name, doj) values ('KATE', '27-FEB-2001'); INSERT INTO t_def_cols (name, doj) values ('NANCY', '17-JUN-1998'); INSERT INTO t_def_cols (name, doj) values ('LANCE', '03-JAN-2004'); INSERT INTO t_def_cols (name) values ('MARY'); COMMIT; END; / Query the table and check the values for the ID and DOJ columns. ID gets the value from the sequence SEQ while DOJ for MARY has been defaulted to 01-JAN-2000. /*Query the table to verify sequence and default on null values*/ SELECT * FROM t_def_cols / ID NAME DOJ ---------- -------- --------- 100 KATE 27-FEB-01 110 NANCY 17-JUN-98 120 LANCE 03-JAN-04 130 MARY 01-JAN-00 Support for 32K VARCHAR2 Oracle Database 12c supports the VARCHAR2, NVARCHAR2, and RAW datatypes up to 32,767 bytes in size. The previous maximum limit for the VARCHAR2 (and NVARCHAR2) and RAW datatypes was 4,000 bytes and 2,000 bytes respectively. The support for extended string datatypes will benefit the non-Oracle to Oracle migrations. The feature can be controlled using the initialization parameter MAX_STRING_SIZE. It accepts two values: STANDARD (default)—The maximum size prior to the release of Oracle Database 12c will apply. EXTENDED—The new size limit for string datatypes apply. Note that, after the parameter is set to EXTENDED, the setting cannot be rolled back. The steps to increase the maximum string size in a database are: Restart the database in UPGRADE mode. In the case of a pluggable database, the PDB must be opened in MIGRATEmode. Use the ALTER SYSTEM command to set MAX_STRING_SIZE to EXTENDED. As SYSDBA, execute the $ORACLE_HOME/rdbms/admin/utl32k.sql script. The script is used to increase the maximum size limit of VARCHAR2, NVARCHAR2, and RAW wherever required. Restart the database in NORMAL mode. As SYSDBA, execute utlrp.sql to recompile the schema objects with invalid status. The points to be considered while working with the 32k support for string types are: COMPATIBLE must be 12.0.0.0 After the parameter is set to EXTENDED, the parameter cannot be rolled back to STANDARD In RAC environments, all the instances of the database comply with the setting of MAX_STRING_SIZE Row limiting using FETCH FIRST For Top-N queries, Oracle Database 12c introduces a new clause, FETCH FIRST, to simplify the code and comply with ANSI SQL standard guidelines. The clause is used to limit the number of rows returned by a query. The new clause can be used in conjunction with ORDER BY to retrieve Top-N results. The row limiting clause can be used with the FOR UPDATE clause in a SQL query. In the case of a materialized view, the defining query should not contain the FETCH clause. Another new clause, OFFSET, can be used to skip the records from the top or middle, before limiting the number of rows. For consistent results, the offset value must be a positive number, less than the total number of rows returned by the query. For all other offset values, the value is counted as zero. Keywords with the FETCH FIRST clause are: FIRST | NEXT—Specify FIRST to begin row limiting from the top. Use NEXT with OFFSET to skip certain rows. ROWS | PERCENT—Specify the size of the result set as a fixed number of rows or percentage of total number of rows returned by the query. ONLY | WITH TIES—Use ONLY to fix the size of the result set, irrespective of duplicate sort keys. If you want records with matching sort keys, specify WITH TIES. The following query demonstrates the use of the FETCH FIRST and OFFSET clauses in Top-N queries: /*Create the test table*/ CREATE TABLE t_fetch_first (empno VARCHAR2(30), deptno NUMBER, sal NUMBER, hiredate DATE) / The following PL/SQL block inserts sample data for testing: /*Insert the test data in T_FETCH_FIRST table*/ BEGIN INSERT INTO t_fetch_first VALUES (101, 10, 1500, '01-FEB-2011'); INSERT INTO t_fetch_first VALUES (102, 20, 1100, '15-JUN-2001'); INSERT INTO t_fetch_first VALUES (103, 20, 1300, '20-JUN-2000'); INSERT INTO t_fetch_first VALUES (104, 30, 1550, '30-DEC-2001'); INSERT INTO t_fetch_first VALUES (105, 10, 1200, '11-JUL-2012'); INSERT INTO t_fetch_first VALUES (106, 30, 1400, '16-AUG-2004'); INSERT INTO t_fetch_first VALUES (107, 20, 1350, '05-JAN-2007'); INSERT INTO t_fetch_first VALUES (108, 20, 1000, '18-JAN-2009'); COMMIT; END; / The SELECT query pulls in the top-5 rows when sorted by their salary: /*Query to list top-5 employees by salary*/ SELECT * FROM t_fetch_first ORDER BY sal DESC FETCH FIRST 5 ROWS ONLY / EMPNO DEPTNO SAL HIREDATE -------- ------ ------- --------- 104 30 1550 30-DEC-01 101 10 1500 01-FEB-11 106 30 1400 16-AUG-04 107 20 1350 05-JAN-07 103 20 1300 20-JUN-00 The SELECT query lists the top 25% of employees (2) when sorted by their hiredate: /*Query to list top-25% employees by hiredate*/ SELECT * FROM t_fetch_first ORDER BY hiredate FETCH FIRST 25 PERCENT ROW ONLY / EMPNO DEPTNO SAL HIREDATE -------- ------ ----- --------- 103 20 1300 20-JUN-00 102 20 1100 15-JUN-01 The SELECT query skips the first five employees and displays the next two—the 6th and 7th employee data: /*Query to list 2 employees after skipping first 5 employees*/ SELECT * FROM t_fetch_first ORDER BY SAL DESC OFFSET 5 ROWS FETCH NEXT 2 ROWS ONLY / Invisible columns Oracle Database 12c supports invisible columns, which implies that the visibility of a column. A column marked invisible does not appear in the following operations: SELECT * FROM queries on the table SQL* Plus DESCRIBE command Local records of %ROWTYPE Oracle Call Interface (OCI) description A column can be made invisible by specifying the INVISIBLE clause against the column. Columns of all types (except user-defined types), including virtual columns, can be marked invisible, provided the tables are not temporary tables, external tables, or clustered ones. An invisible column can be explicitly selected by the SELECT statement. Similarly, the INSERTstatement will not insert values in an invisible column unless explicitly specified. Furthermore, a table can be partitioned based on an invisible column. A column retains its nullity feature even after it is made invisible. An invisible column can be made visible, but the ordering of the column in the table may change. In the following script, the column NICKNAME is set as invisible in the table t_inv_col: /*Create a table to demonstrate invisible columns*/ CREATE TABLE t_inv_col (id NUMBER, name VARCHAR2(30), nickname VARCHAR2 (10) INVISIBLE, dob DATE ) / The information about the invisible columns can be found in user_tab_cols. Note that the invisible column is marked as hidden: /*Query the USER_TAB_COLS for metadata information*/ SELECT column_id, column_name, hidden_column FROM user_tab_cols WHERE table_name = 'T_INV_COL' ORDER BY column_id / COLUMN_ID COLUMN_NAME HID ---------- ------------ --- 1 ID NO 2 NAME NO 3 DOB NO NICKNAME YES Hidden columns are different from invisible columns. Invisible columns can be made visible and vice versa, but hidden columns cannot be made visible. If we try to make the NICKNAME visible and NAME invisible, observe the change in column ordering: /*Script to change visibility of NICKNAME column*/ ALTER TABLE t_inv_col MODIFY nickname VISIBLE / /*Script to change visibility of NAME column*/ ALTER TABLE t_inv_col MODIFY name INVISIBLE / /*Query the USER_TAB_COLS for metadata information*/ SELECT column_id, column_name, hidden_column FROM user_tab_cols WHERE table_name = 'T_INV_COL' ORDER BY column_id / COLUMN_ID COLUMN_NAME HID ---------- ------------ --- 1 ID NO 2 DOB NO 3 NICKNAME NO NAME YES Temporal databases Temporal databases were released as a new feature in ANSI SQL:2011. The term temporal data can be understood as a piece of information that can be associated with a period within which the information is valid. Before the feature was included in Oracle Database 12c, data whose validity is linked with a time period had to be handled either by the application or using multiple predicates in the queries. Oracle 12c partially inherits the feature from the ANSI SQL:2011 standard to support the entities whose business validity can be bracketed with a time dimension. If you're relating a temporal database with the Oracle Database 11g Total Recall feature, you're wrong. The total recall feature records the transaction time of the data in the database to secure the transaction validity and not the functional validity. For example, an investment scheme is active between January to December. The date recorded in the database at the time of data loading is the transaction timestamp. [box type="info" align="" class="" width=""]Starting from Oracle 12c, the Total Recall feature has been rebranded as Flashback Data Archive and has been made available for all versions of Oracle Database.[/box] The valid time temporal can be enabled for a table by adding a time dimension using the PERIOD FOR clause on the date or timestamp columns of the table. The following script creates a table t_tmp_db with the valid time temporal: /*Create table with valid time temporal*/ CREATE TABLE t_tmp_db( id NUMBER, name VARCHAR2(30), policy_no VARCHAR2(50), policy_term number, pol_st_date date, pol_end_date date, PERIOD FOR pol_valid_time (pol_st_date, pol_end_date)) / Create some sample data in the table: /*Insert test data in the table*/ BEGIN INSERT INTO t_tmp_db VALUES (100, 'Packt', 'PACKT_POL1', 1, '01-JAN-2015', '31-DEC-2015'); INSERT INTO t_tmp_db VALUES (110, 'Packt', 'PACKT_POL2', 2, '01-JAN-2015', '30-JUN-2015'); INSERT INTO t_tmp_db VALUES (120, 'Packt', 'PACKT_POL3', 3, '01-JUL-2015', '31-DEC-2015'); COMMIT; END; / Let us set the current time period window using DBMS_FLASHBACK_ARCHIVE. Grant the EXECUTE privilege on the package to the scott user. /*Connect to sysdba to grant execute privilege to scott*/ conn / as sysdba GRANT EXECUTE ON dbms_flashback_archive to scott / Grant succeeded. /*Connect to scott*/ conn scott/tiger /*Set the valid time period as CURRENT*/ EXEC DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME('CURRENT'); PL/SQL procedure successfully completed. Setting the valid time period as CURRENT means that all the tables with a valid time temporal will only list the rows that are valid with respect to today's date. You can set the valid time to a particular date too. /*Query the table*/ SELECT * from t_tmp_db / ID POLICY_NO POL_ST_DATE POL_END_DATE --------- ---------- ----------------------- ------------------- 100 PACKT_POL1 01-JAN-15 31-DEC-15 110 PACKT_POL2 01-JAN-15 30-JUN-15 [box type="info" align="" class="" width=""]Due to dependency on the current date, the result may vary when the reader runs the preceding queries.[/box] The query lists only those policies that are active as of March 2015. Since the third policy starts in July 2015, it is currently not active. In-Database Archiving Oracle Database 12c introduces In-Database Archiving to archive the low priority data in a table. The inactive data remains in the database but is not visible to the application. You can mark old data for archival, which is not actively required in the application except for regulatory purposes. Although the archived data is not visible to the application, it is available for querying and manipulation. In addition, the archived data can be compressed to improve backup performance. A table can be enabled by specifying the ROW ARCHIVAL clause at the table level, which adds a hidden column ORA_ARCHIVE_STATE to the table structure. The column value must be updated to mark a row for archival. For example: /*Create a table with row archiving*/ CREATE TABLE t_row_arch( x number, y number, z number) ROW ARCHIVAL / When we query the table structure in the USER_TAB_COLS view, we find an additional hidden column, which Oracle implicitly adds to the table: /*Query the columns information from user_tab_cols view*/ SELECT column_id,column_name,data_type, hidden_column FROM user_tab_cols WHERE table_name='T_ROW_ARCH' / COLUMN_ID COLUMN_NAME DATA_TYPE HID ---------- ------------------ ---------- --- ORA_ARCHIVE_STATE VARCHAR2 YES 1 X NUMBER NO 2 Y NUMBER NO 3 Z NUMBER NO Let us create test data in the table: /Insert test data in the table*/ BEGIN INSERT INTO t_row_arch VALUES (10,20,30); INSERT INTO t_row_arch VALUES (11,22,33); INSERT INTO t_row_arch VALUES (21,32,43); INSERT INTO t_row_arch VALUES (51,82,13); commit; END; / For testing purpose, let us archive the rows in the table where X > 50 by updating the ora_archive_state column: /*Update ORA_ARCHIVE_STATE column in the table*/ UPDATE t_row_arch SET ora_archive_state = 1 WHERE x > 50 / COMMIT / By default, the session displays only the active records from an archival-enabled table: /*Query the table*/ SELECT * FROM t_row_arch / X Y Z ------ -------- ---------- 10 20 30 11 22 33 21 32 43 If you wish to display all the records, change the session setting: /*Change the session parameter to display the archived records*/ ALTER SESSION SET ROW ARCHIVAL VISIBILITY = ALL / Session altered. /*Query the table*/ SELECT * FROM t_row_arch / X Y Z ---------- ---------- ---------- 10 20 30 11 22 33 21 32 43 51 82 13 Defining a PL/SQL subprogram in the SELECT query and PRAGMA UDF Oracle Database 12c includes two new features to enhance the performance of functions when called from SELECT statements. With Oracle 12c, a PL/SQL subprogram can be created inline with the SELECT query in the WITH clause declaration. The function created in the WITH clause subquery is not stored in the database schema and is available for use only in the current query. Since a procedure created in the WITH clause cannot be called from the SELECT query, it can be called in the function created in the declaration section. The feature can be very handy in read-only databases where the developers were not able to create PL/SQL wrappers. Oracle Database 12c adds the new PRAGMA UDF to create a standalone function with the same objective. Earlier, the SELECT queries could invoke a PL/SQL function, provided the function didn't change the database purity state. The query performance used to degrade because of the context switch from SQL to the PL/SQL engine (and vice versa) and the different memory representations of data type representation in the processing engines. In the following example, the function fun_with_plsql calculates the annual compensation of an employee's monthly salary: /*Create a function in WITH clause declaration*/ WITH FUNCTION fun_with_plsql (p_sal NUMBER) RETURN NUMBER IS BEGIN RETURN (p_sal * 12); END; SELECT ename, deptno, fun_with_plsql (sal) "annual_sal" FROM emp / ENAME DEPTNO annual_sal ---------- --------- ---------- SMITH 20 9600 ALLEN 30 19200 WARD 30 15000 JONES 20 35700 MARTIN 30 15000 BLAKE 30 34200 CLARK 10 29400 SCOTT 20 36000 KING 10 60000 TURNER 30 18000 ADAMS 20 13200 JAMES 30 11400 FORD 20 36000 MILLER 10 15600 14 rows selected. [box type="info" align="" class="" width=""]If the query containing the WITH clause declaration is not a top-level statement, then the top level statement must use the WITH_PLSQL hint. The hint will be used if INSERT, UPDATE, or DELETE statements are trying to use a SELECT with a WITHclause definition. Failure to include the hint results in an exception ORA-32034: unsupported use of WITH clause.[/box] A function can be created with the PRAGMA UDF to inform the compiler that the function is always called in a SELECT statement. Note that the standalone function created in the following code carries the same name as the one in the last example. The local WITH clause declaration takes precedence over the standalone function in the schema. /*Create a function with PRAGMA UDF*/ CREATE OR REPLACE FUNCTION fun_with_plsql (p_sal NUMBER) RETURN NUMBER is PRAGMA UDF; BEGIN RETURN (p_sal *12); END; / Since the objective of the feature is performance, let us go ahead with a case study to compare the performance when using a standalone function, a PRAGMA UDF function, and a WITHclause declared function. Test setup The exercise uses a test table with 1 million rows, loaded with random data. /*Create a table for performance test study*/ CREATE TABLE t_fun_plsql (id number, str varchar2(30)) / /*Generate and load random data in the table*/ INSERT /*+APPEND*/ INTO t_fun_plsql SELECT ROWNUM, DBMS_RANDOM.STRING('X', 20) FROM dual CONNECT BY LEVEL <= 1000000 / COMMIT / Case 1: Create a PL/SQL standalone function as it used to be until Oracle Database 12c. The function counts the numbers in the str column of the table. /*Create a standalone function without Oracle 12c enhancements*/ CREATE OR REPLACE FUNCTION f_count_num (p_str VARCHAR2) RETURN PLS_INTEGER IS BEGIN RETURN (REGEXP_COUNT(p_str,'d')); END; / The PL/SQL block measures the elapsed and CPU time when working with a pre-Oracle 12c standalone function. These numbers will serve as the baseline for our case study. /*Set server output on to display messages*/ SET SERVEROUTPUT ON /*Anonymous block to measure performance of a standalone function*/ DECLARE l_el_time PLS_INTEGER; l_cpu_time PLS_INTEGER; CURSOR C1 IS SELECT f_count_num (str) FROM t_fun_plsql; TYPE t_tab_rec IS TABLE OF PLS_INTEGER; l_tab t_tab_rec; BEGIN l_el_time := DBMS_UTILITY.GET_TIME (); l_cpu_time := DBMS_UTILITY.GET_CPU_TIME (); OPEN c1; FETCH c1 BULK COLLECT INTO l_tab; CLOSE c1; DBMS_OUTPUT.PUT_LINE ('Case 1: Performance of a standalone function'); DBMS_OUTPUT.PUT_LINE ('Total elapsed time:'||to_char(DBMS_UTILITY.GET_TIME () - l_el_time)); DBMS_OUTPUT.PUT_LINE ('Total CPU time:'||to_char(DBMS_UTILITY.GET_CPU_TIME () - l_cpu_time)); END; / Performance of a standalone function: Total elapsed time:1559 Total CPU time:1366 PL/SQL procedure successfully completed. Case 2: Create a PL/SQL function using PRAGMA UDF to count the numbers in the str column. /*Create the function with PRAGMA UDF*/ CREATE OR REPLACE FUNCTION f_count_num_pragma (p_str VARCHAR2) RETURN PLS_INTEGER IS PRAGMA UDF; BEGIN RETURN (REGEXP_COUNT(p_str,'d')); END; / Let us now check the performance of the PRAGMA UDF function using the following PL/SQL block. /*Set server output on to display messages*/ SET SERVEROUTPUT ON /*Anonymous block to measure performance of a PRAGMA UDF function*/ DECLARE l_el_time PLS_INTEGER; l_cpu_time PLS_INTEGER; CURSOR C1 IS SELECT f_count_num_pragma (str) FROM t_fun_plsql; TYPE t_tab_rec IS TABLE OF PLS_INTEGER; l_tab t_tab_rec; BEGIN l_el_time := DBMS_UTILITY.GET_TIME (); l_cpu_time := DBMS_UTILITY.GET_CPU_TIME (); OPEN c1; FETCH c1 BULK COLLECT INTO l_tab; CLOSE c1; DBMS_OUTPUT.PUT_LINE ('Case 2: Performance of a PRAGMA UDF function'); DBMS_OUTPUT.PUT_LINE ('Total elapsed time:'||to_char(DBMS_UTILITY.GET_TIME () - l_el_time)); DBMS_OUTPUT.PUT_LINE ('Total CPU time:'||to_char(DBMS_UTILITY.GET_CPU_TIME () - l_cpu_time)); END; / Performance of a PRAGMA UDF function: Total elapsed time:664 Total CPU time:582 PL/SQL procedure successfully completed. Case 3: The following PL/SQL block dynamically executes the function in the WITH clause subquery. Note that, unlike other SELECT statements, a SELECT query with a WITH clause declaration cannot be executed statically in the body of a PL/SQL block. /*Set server output on to display messages*/ SET SERVEROUTPUT ON /*Anonymous block to measure performance of inline function*/ DECLARE l_el_time PLS_INTEGER; l_cpu_time PLS_INTEGER; l_sql VARCHAR2(32767); c1 sys_refcursor; TYPE t_tab_rec IS TABLE OF PLS_INTEGER; l_tab t_tab_rec; BEGIN l_el_time := DBMS_UTILITY.get_time; l_cpu_time := DBMS_UTILITY.get_cpu_time; l_sql := 'WITH FUNCTION f_count_num_with (p_str VARCHAR2) RETURN NUMBER IS BEGIN RETURN (REGEXP_COUNT(p_str,'''||''||'d'||''')); END; SELECT f_count_num_with(str) FROM t_fun_plsql'; OPEN c1 FOR l_sql; FETCH c1 bulk collect INTO l_tab; CLOSE c1; DBMS_OUTPUT.PUT_LINE ('Case 3: Performance of an inline function'); DBMS_OUTPUT.PUT_LINE ('Total elapsed time:'||to_char(DBMS_UTILITY.GET_TIME () - l_el_time)); DBMS_OUTPUT.PUT_LINE ('Total CPU time:'||to_char(DBMS_UTILITY.GET_CPU_TIME () - l_cpu_time)); END; / Performance of an inline function: Total elapsed time:830 Total CPU time:718 PL/SQL procedure successfully completed. Comparative analysis Comparing the results from the preceding three cases, it's clear that the Oracle 12c flavor of PL/SQL functions out-performs the pre-12c standalone function by a high margin. From the following matrix, it is apparent that the usage of the PRAGMA UDF or WITH clause declaration enhances the code performance by (roughly) a factor of 2. Case Description Elapsed Time CPU time Performance gain factor by CPU time Standalone PL/SQL function in pre-Oracle 12c database 1559 1336 1x Standalone PL/SQL PRAGMA UDF function in Oracle 12c 664 582 2.3x Function created in WITH clause declaration in Oracle 12c 830 718 1.9x   [box type="info" align="" class="" width=""]Note that the numbers may slightly differ in the reader's testing environment but you should be able to draw the same conclusion by comparing them.[/box] The PL/SQL program unit whitelisting Prior to Oracle 12c, a standalone or packaged PL/SQL unit could be invoked by all other programs in the session's schema. Oracle Database 12c allows users to prevent unauthorized access to PL/SQL program units. You can now specify the list of whitelist program units that can invoke a particular program. The PL/SQL program header or the package specification can specify the list of program units in the ACCESSIBLE BY clause in the program header. All other program units, including cross-schema references (even SYS owned objects), trying to access a protected subprogram will receive an exception, PLS-00904: insufficient privileges to access object [object name]. The feature can be very useful in an extremely sensitive development environment. Suppose, a package PKG_FIN_PROC contains the sensitive implementation routines for financial institutions, the packaged subprograms are called by another PL/SQL package PKG_FIN_INTERNALS. The API layer exposes a fixed list of programs through a public API called PKG_CLIENT_ACCESS. In order to restrict access to the packaged routines in PKG_FIN_PROC, the users can build a safety net so as to allow access to only authorized programs. The following PL/SQL package PKG_FIN_PROC contains two subprograms—P_FIN_QTR and P_FIN_ANN. The ACCESSIBLE BY clause includes PKG_FIN_INTERNALS which means that all other program units, including anonymous PL/SQL blocks, are blocked from invoking PKG_FIN_PROC constructs. /*Package with the accessible by clause*/ CREATE OR REPLACE PACKAGE pkg_fin_proc ACCESSIBLE BY (PACKAGE pkg_fin_internals) IS PROCEDURE p_fin_qtr; PROCEDURE p_fin_ann; END; / [box type="info" align="" class="" width=""]The ACCESSIBLE BY clause can be specified for schema-level programs only.[/box] Let's see what happens when we invoke the packaged subprogram from an anonymous PL/SQL block. /*Invoke the packaged subprogram from the PL/SQL block*/ BEGIN pkg_fin_proc.p_fin_qtr; END; / pkg_fin_proc.p_fin_qtr; * ERROR at line 2: ORA-06550: line 2, column 4: PLS-00904: insufficient privilege to access object PKG_FIN_PROC ORA-06550: line 2, column 4: PL/SQL: Statement ignored Well, the compiler throws an exception as invoking the whitelisted package from an anonymous block is not allowed. The ACCESSIBLE BY clause can be included in the header information of PL/SQL procedures and functions, packages, and object types. Granting roles to PL/SQL program units Before Oracle Database 12c, a PL/SQL unit created with the definer's rights (default AUTHID) always executed with the definer's rights, whether or not the invoker has the required privileges. It may lead to an unfair situation where the invoking user may perform unwanted operations without needing the correct set of privileges. Similarly for an invoker's right unit, if the invoking user possesses a higher set of privileges than the definer, he might end up performing unauthorized operations. Oracle Database 12c secures the definer's rights by allowing the defining user to grant complementary roles to individual PL/SQL subprograms and packages. From the security standpoint, the granting of roles to schema level subprograms provides granular control as the privileges of the invoker are validated at the time of execution. In the following example, we will create two users: U1 and U2. The user U1 creates a PL/SQL procedure P_INC_PRICE that adds a surcharge to the price of a product by a certain amount. U1 grants the execute privilege to user U2. Test setup Let's create two users and give them the required privileges. /*Create a user with a password*/ CREATE USER u1 IDENTIFIED BY u1 / User created. /*Grant connect privileges to the user*/ GRANT CONNECT, RESOURCE TO u1 / Grant succeeded. /*Create a user with a password*/ CREATE USER u2 IDENTIFIED BY u2 / User created. /*Grant connect privileges to the user*/ GRANT CONNECT, RESOURCE TO u2 / Grant succeeded. The user U1 contains the PRODUCTS table. Let's create and populate the table. /*Connect to U1*/ CONN u1/u1 /*Create the table PRODUCTS*/ CREATE TABLE products ( prod_id INTEGER, prod_name VARCHAR2(30), prod_cat VARCHAR2(30), price INTEGER ) / /*Insert the test data in the table*/ BEGIN DELETE FROM products; INSERT INTO products VALUES (101, 'Milk', 'Dairy', 20); INSERT INTO products VALUES (102, 'Cheese', 'Dairy', 50); INSERT INTO products VALUES (103, 'Butter', 'Dairy', 75); INSERT INTO products VALUES (104, 'Cream', 'Dairy', 80); INSERT INTO products VALUES (105, 'Curd', 'Dairy', 25); COMMIT; END; / The procedure p_inc_price is designed to increase the price of a product by a given amount. Note that the procedure is created with the definer's rights. /*Create the procedure with the definer's rights*/ CREATE OR REPLACE PROCEDURE p_inc_price (p_prod_id NUMBER, p_amt NUMBER) IS BEGIN UPDATE products SET price = price + p_amt WHERE prod_id = p_prod_id; END; / The user U1 grants execute privilege on p_inc_price to U2. /*Grant execute on the procedure to the user U2*/ GRANT EXECUTE ON p_inc_price TO U2 / The user U2 logs in and executes the procedure P_INC_PRICE to increase the price of Milk by 5 units. /*Connect to U2*/ CONN u2/u2 /*Invoke the procedure P_INC_PRICE in a PL/SQL block*/ BEGIN U1.P_INC_PRICE (101,5); COMMIT; END; / PL/SQL procedure successfully completed. The last code listing exposes a gray area. The user U2, though not authorized to view PRODUCTS data, manipulates its data with the definer's rights. We need a solution to the problem. The first step is to change the procedure from definer's rights to invoker's rights. /*Connect to U1*/ CONN u1/u1 /*Modify the privilege authentication for the procedure to invoker's rights*/ CREATE OR REPLACE PROCEDURE p_inc_price (p_prod_id NUMBER, p_amt NUMBER) AUTHID CURRENT_USER IS BEGIN UPDATE products SET price = price + p_amt WHERE prod_id = p_prod_id; END; / Now, if we execute the procedure from U2, it throws an exception because it couldn't find the PRODUCTS table in its schema. /*Connect to U2*/ CONN u2/u2 /*Invoke the procedure P_INC_PRICE in a PL/SQL block*/ BEGIN U1.P_INC_PRICE (101,5); COMMIT; END; / BEGIN * ERROR at line 1: ORA-00942: table or view does not exist ORA-06512: at "U1.P_INC_PRICE", line 5 ORA-06512: at line 2 In a similar scenario in the past, the database administrators could have easily granted select or updated privileges to U2, which is not an optimal solution from the security standpoint. Oracle 12c allows the users to create program units with invoker's rights but grant the required roles to the program units and not the users. So, an invoker right unit executes with invoker's privileges, plus the PL/SQL program role. Let's check out the steps to create a role and assign it to the procedure. SYSDBA creates the role and assigns it to the user U1. Using the ADMIN or DELEGATE option with the grant enables the user to grant the role to other entities. /*Connect to SYSDBA*/ CONN / as sysdba /*Create a role*/ CREATE ROLE prod_role / /*Grant role to user U1 with delegate option*/ GRANT prod_role TO U1 WITH DELEGATE OPTION / Now, user U1 assigns the required set of privileges to the role. The role is then assigned to the required subprogram. Note that only roles, and not individual privileges, can be assigned to the schema level subprograms. /*Connect to U1*/ CONN u1/u1 /*Grant SELECT and UPDATE privileges on PRODUCTS to the role*/ GRANT SELECT, UPDATE ON PRODUCTS TO prod_role / /*Grant role to the procedure*/ GRANT prod_role TO PROCEDURE p_inc_price / User U2 tries to execute the procedure again. The procedure is successfully executed which means the value of "Milk" has been increased by 5 units. /*Connect to U2*/ CONN u2/u2 /*Invoke the procedure P_INC_PRICE in a PL/SQL block*/ BEGIN U1.P_INC_PRICE (101,5); COMMIT; END; / PL/SQL procedure successfully completed. User U1 verifies the result with a SELECT query. /*Connect to U1*/ CONN u1/u1 /*Query the table to verify the change*/ SELECT * FROM products / PROD_ID PROD_NAME PROD_CAT PRICE ---------- ---------- ---------- ---------- 101 Milk Dairy 25 102 Cheese Dairy 50 103 Butter Dairy 75 104 Cream Dairy 80 105 Curd Dairy 25 Miscellaneous PL/SQL enhancements Besides the preceding key features, there are a lot of new features in Oracle 12c. The list of features is as follows: An invoker rights function can be result-cached—until Oracle 11g, only the definers' programs were allowed to cache their results. Oracle 12c adds the invoking user's identity to the result cache to make it independent of the definer. The compilation parameter PLSQL_DEBUG has been deprecated. Two conditional compilation inquiry directives $$PLSQL_UNIT_OWNER and $$PLSQL_UNIT_TYPE have been implemented. Summary This chapter covers the top rated and new features of Oracle 12c SQL and PL/SQL as well as some miscellaneous PL/SQL enhancements. This chapter covers the top rated and new features of Oracle 12c SQL and PL/SQL as well as some miscellaneous PL/SQL enhancements. Further resources on this subject: What is Oracle Public Cloud? [article] Oracle APEX 4.2 reporting [article] Oracle E-Business Suite with Desktop Integration [article]

Sometimes your Ionic app needs to tell the user something really important. It might be so important that you want to be sure they see it before your app does anything else. For this you can usually use an Ionic popup. Popups are a nice, clean way to do something similar to the browser's alert, with none of its bad parts. But popups have their limitations. Their design isn't very flexible. It has a title region on the top, a message region in the middle and a button region at the bottom, all on a nice white canvas, centered in the middle of the display. If you want something different, such as no buttons, no white background, just a dark screen with a message and a picture, then a popup is not for you. What can you use instead? In this post you will learn how to use Ionic's loading overlay in combination with its gestures to build an alternative modal to the popup, which is more flexible in its design. The Ionic loading overlay usually just shows a line of text or an ionic or some combination of the two. It is meant to let the user know that the app is loading in a resource. But like many things in Ionic it has a flexible design which allows us to use it for a slightly different purpose. The template property of the overlay can hold pretty much any HTML we give it. For our example app we will build upon the Ionic sidemenu starter template. We don't actually need to add much code to the starter template (we will use beta 14 of Ionic). First, we need to add a few references to the playlist controller. We need the Angular $timeout and the Ionic $ionicLoading and $ionicGesture, so we add their names to the parameters. We also add a variable, showingOverlay, which tells us if the overlay is currently being displayed and set it to false. if (!showingOverlay) { showingOverlay = true; console.log("First Pass Through - show the overlay"); $ionicLoading.show({ template: ' <h2>A message from cat central</h2> <img src="img/cat.jpg" alt="" width="256" height="256" /> <div>Stop playing angry birds and feed me.</div> ', content: 'Loading', animation: 'fade-in', showBackdrop: true, maxWidth: 300, showDelay: 0 }); } To display the loading overlay we call its show method. In the options that we pass to it, we set the template to the HTML we wish to display. In our case we include an image tag that points to a picture in our 'img' directory. If you'd like, you can use the templateUrl property instead of the template property. Don't forget to set the maxWidth property to something reasonable. It must be big enough to hold your content but not too big. You can also set other properties to your liking. In the example I've set the showBackdrop to true and the showDelay to zero. Now that we have our overlay displaying to the user how do we get rid of it? Good question. $ionicLoading blocks users from interacting with the page until we want them to. In order to interact with the overlay, we will have to hook it with a gesture event. Ionic comes with a complete set of gesture events for mobile devices. Most of the time you don't need to use them directly since Ionic widgets do that work for you, but in this example we will need to dig a bit deeper. When the $ionicLoading overlay is displayed it has no id but it does have a class of 'loading'. Once we find it, we can listen for the user to tap it, then dismiss the overlay and unbind the event. $timeout(function () { if (showingOverlay) { // we don't need anything fancy here, we use the DOM's query selector to find the loading div var element = angular.element(document.querySelector('.loading')), // and then we hook the tapGesture = $ionicGesture.on('tap', function (evt) { console.log("TAP RECEIVED!!!"); $ionicLoading.hide(); $ionicGesture.off(tapGesture, 'tap'); showingOverlay = false; }, element); } }, 50); Why have we wrapped the code in an Angular $timeout? We wrap the code in a $timeout because if we look for the overlay on the page immediately, it won't be there. The DOM won't have a chance to render it until our code has given control back to the browser. So we place the code in a timeout which lets the browser render the overlay. Once the timeout has elapsed, we can find overlay element on the page. Also note, we don't use jQuery, and Angular doesn't have a way to find an element by class or id. We instead use the document.querySelector method, which is in all of the devices which Ionic supports. Once the user has tapped the overlay, we need to do some clean up. First, we hide the overlay. Then we turn off the tap event. And finally we set the flag saying whether or not we are displaying the overlay back to false. To be honest this code really belongs in a directive and this is something I will probably do later. But it works and it neatly solves an issue I was having in my current project. If you want to check out the code for yourself it is on my GitHub repo at: https://github.com/Rockncoder/loading. About the author Troy Miles, aka the Rockncoder, began writing games in assembly language for early computers like the Apple II, Vic20, C64, and the IBM PC over 35 years ago. Currently he fills his days writing web apps for a Southern California based automotive valuation and information company. Nights and weekends he can usually be found writing cool apps for mobile and web or teaching other developers how to do so. He likes to post interesting code nuggets on his blog: http://therockncoder.com and videos on his YouTube channel: https://www.youtube.com/user/rockncoder. He can be reached at rockncoder@gmail.com

In this article by Salil Kapur and Nisarg Thakkar, authors of the book Mastering OpenCV Android Application Programming, we will look at the broader aspects of object tracking in Videos. Object tracking is one of the most important applications of computer vision. It can be used for many applications, some of which are as follows: Human–computer interaction: We might want to track the position of a person's finger and use its motion to control the cursor on our machines Surveillance: Street cameras can capture pedestrians' motions that can be tracked to detect suspicious activities Video stabilization and compression Statistics in sports: By tracking a player's movement in a game of football, we can provide statistics such as distance travelled, heat maps, and so on In this article, you will learn the following topics: Optical flow Image Pyramids (For more resources related to this topic, see here.) Optical flow Optical flow is an algorithm that detects the pattern of the motion of objects, or edges, between consecutive frames in a video. This motion may be caused by the motion of the object or the motion of the camera. Optical flow is a vector that depicts the motion of a point from the first frame to the second. The optical flow algorithm works under two basic assumptions: The pixel intensities are almost constant between consecutive frames The neighboring pixels have the same motion as the anchor pixel We can represent the intensity of a pixel in any frame by f(x,y,t). Here, the parameter t represents the frame in a video. Let's assume that, in the next dt time, the pixel moves by (dx,dy). Since we have assumed that the intensity doesn't change in consecutive frames, we can say: f(x,y,t) = f(x + dx,y + dy,t + dt) Now we take the Taylor series expansion of the RHS in the preceding equation: Cancelling the common term, we get: Where . Dividing both sides of the equation by dt we get: This equation is called the optical flow equation. Rearranging the equation we get: We can see that this represents the equation of a line in the (u,v) plane. However, with only one equation available and two unknowns, this problem is under constraint at the moment. The Horn and Schunck method By taking into account our assumptions, we get: We can say that the first term will be small due to our assumption that the brightness is constant between consecutive frames. So, the square of this term will be even smaller. The second term corresponds to the assumption that the neighboring pixels have similar motion to the anchor pixel. We need to minimize the preceding equation. For this, we differentiate the preceding equation with respect to u and v. We get the following equations: Here, and  are the Laplacians of u and v respectively. The Lucas and Kanade method We start off with the optical flow equation that we derived earlier and noticed that it is under constrained as it has one equation and two variables: To overcome this problem, we make use of the assumption that pixels in a 3x3 neighborhood have the same optical flow: We can rewrite these equations in the form of matrices, as shown here: This can be rewritten in the form: Where: As we can see, A is a 9x2 matrix, U is a 2x1 matrix, and b is a 9x1 matrix. Ideally, to solve for U, we just need to multiply by A-1on both sides of the equation. However, this is not possible, as we can only take the inverse of square matrices. Thus, we try to transform A into a square matrix by first multiplying the equation by AT on both sides of the equation: Now is a square matrix of dimension 2x2. Hence, we can take its inverse: On solving this equation, we get: This method of multiplying the transpose and then taking an inverse is called pseudo-inverse. This equation can also be obtained by finding the minimum of the following equation: According to the optical flow equation and our assumptions, this value should be equal to zero. Since the neighborhood pixels do not have exactly the same values as the anchor pixel, this value is very small. This method is called Least Square Error. To solve for the minimum, we differentiate this equation with respect to u and v, and equate it to zero. We get the following equations: Now we have two equations and two variables, so this system of equations can be solved. We rewrite the preceding equations as follows: So, by arranging these equations in the form of a matrix, we get the same equation as obtained earlier: Since, the matrix A is now a 2x2 matrix, it is possible to take an inverse. On taking the inverse, the equation obtained is as follows: This can be simplified as: Solving for u and v, we get: Now we have the values for all the , , and . Thus, we can find the values of u and v for each pixel. When we implement this algorithm, it is observed that the optical flow is not very smooth near the edges of the objects. This is due to the brightness constraint not being satisfied. To overcome this situation, we use image pyramids. Checking out the optical flow on Android To see the optical flow in action on Android, we will create a grid of points over a video feed from the camera, and then the lines will be drawn for each point that will depict the motion of the point on the video, which is superimposed by the point on the grid. Before we begin, we will set up our project to use OpenCV and obtain the feed from the camera. We will process the frames to calculate the optical flow. First, create a new project in Android Studio. We will set the activity name to MainActivity.java and the XML resource file as activity_main.xml. Second, we will give the app the permissions to access the camera. In the AndroidManifest.xml file, add the following lines to the manifest tag: <uses-permission android_name="android.permission.CAMERA" /> Make sure that your activity tag for MainActivity contains the following line as an attribute: android:screenOrientation="landscape" Our activity_main.xml file will contain a simple JavaCameraView. This is a custom OpenCV defined layout that enables us to access the camera frames and processes them as normal Mat objects. The XML code has been shown here: <LinearLayout       android_layout_width="match_parent"    android_layout_height="match_parent"    android_orientation="horizontal">      <org.opencv.android.JavaCameraView        android_layout_width="fill_parent"        android_layout_height="fill_parent"        android_id="@+id/main_activity_surface_view" />   </LinearLayout> Now, let's work on some Java code. First, we'll define some global variables that we will use later in the code: private static final String   TAG = "com.packtpub.masteringopencvandroid.chapter5.MainActivity";      private static final int       VIEW_MODE_KLT_TRACKER = 0;    private static final int       VIEW_MODE_OPTICAL_FLOW = 1;      private int                   mViewMode;    private Mat                   mRgba;    private Mat                   mIntermediateMat;    private Mat                   mGray;    private Mat                   mPrevGray;      MatOfPoint2f prevFeatures, nextFeatures;    MatOfPoint features;      MatOfByte status;    MatOfFloat err;      private MenuItem               mItemPreviewOpticalFlow, mItemPreviewKLT;      private CameraBridgeViewBase   mOpenCvCameraView; We will need to create a callback function for OpenCV, like we did earlier. In addition to the code we used earlier, we will also enable CameraView to capture frames for processing: private BaseLoaderCallback mLoaderCallback = new BaseLoaderCallback(this) {        @Override        public void onManagerConnected(int status) {            switch (status) {                case LoaderCallbackInterface.SUCCESS:                {                    Log.i(TAG, "OpenCV loaded successfully");                      mOpenCvCameraView.enableView();                } break;                default:                {                    super.onManagerConnected(status);                } break;            }        }    }; We will now check whether the OpenCV manager is installed on the phone, which contains the required libraries. In the onResume function, add the following line of code: OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_2_4_10,   this, mLoaderCallback); In the onCreate() function, add the following line before calling setContentView to prevent the screen from turning off, while using the app: getWindow().addFlags(WindowManager.LayoutParams. FLAG_KEEP_SCREEN_ON); We will now initialize our JavaCameraView object. Add the following lines after setContentView has been called: mOpenCvCameraView = (CameraBridgeViewBase)   findViewById(R.id.main_activity_surface_view); mOpenCvCameraView.setCvCameraViewListener(this); Notice that we called setCvCameraViewListener with the this parameter. For this, we need to make our activity implement the CvCameraViewListener2 interface. So, your class definition for the MainActivity class should look like the following code: public class MainActivity extends Activity   implements CvCameraViewListener2 We will add a menu to this activity to toggle between different examples in this article. Add the following lines to the onCreateOptionsMenu function: mItemPreviewKLT = menu.add("KLT Tracker"); mItemPreviewOpticalFlow = menu.add("Optical Flow"); We will now add some actions to the menu items. In the onOptionsItemSelected function, add the following lines: if (item == mItemPreviewOpticalFlow) {            mViewMode = VIEW_MODE_OPTICAL_FLOW;            resetVars();        } else if (item == mItemPreviewKLT){            mViewMode = VIEW_MODE_KLT_TRACKER;            resetVars();        }          return true; We used a resetVars function to reset all the Mat objects. It has been defined as follows: private void resetVars(){        mPrevGray = new Mat(mGray.rows(), mGray.cols(), CvType.CV_8UC1);        features = new MatOfPoint();        prevFeatures = new MatOfPoint2f();        nextFeatures = new MatOfPoint2f();        status = new MatOfByte();        err = new MatOfFloat();    } We will also add the code to make sure that the camera is released for use by other applications, whenever our application is suspended or killed. So, add the following snippet of code to the onPause and onDestroy functions: if (mOpenCvCameraView != null)            mOpenCvCameraView.disableView(); After the OpenCV camera has been started, the onCameraViewStarted function is called, which is where we will add all our object initializations: public void onCameraViewStarted(int width, int height) {        mRgba = new Mat(height, width, CvType.CV_8UC4);        mIntermediateMat = new Mat(height, width, CvType.CV_8UC4);        mGray = new Mat(height, width, CvType.CV_8UC1);        resetVars();    } Similarly, the onCameraViewStopped function is called when we stop capturing frames. Here we will release all the objects we created when the view was started: public void onCameraViewStopped() {        mRgba.release();        mGray.release();        mIntermediateMat.release();    } Now we will add the implementation to process each frame of the feed that we captured from the camera. OpenCV calls the onCameraFrame method for each frame, with the frame as a parameter. We will use this to process each frame. We will use the viewMode variable to distinguish between the optical flow and the KLT tracker, and have different case constructs for the two: public Mat onCameraFrame(CvCameraViewFrame inputFrame) {        final int viewMode = mViewMode;        switch (viewMode) {            case VIEW_MODE_OPTICAL_FLOW: We will use the gray()function to obtain the Mat object that contains the captured frame in a grayscale format. OpenCV also provides a similar function called rgba() to obtain a colored frame. Then we will check whether this is the first run. If this is the first run, we will create and fill up a features array that stores the position of all the points in a grid, where we will compute the optical flow:                mGray = inputFrame.gray();                if(features.toArray().length==0){                   int rowStep = 50, colStep = 100;                    int nRows = mGray.rows()/rowStep, nCols = mGray.cols()/colStep;                      Point points[] = new Point[nRows*nCols];                    for(int i=0; i<nRows; i++){                        for(int j=0; j<nCols; j++){                            points[i*nCols+j]=new Point(j*colStep, i*rowStep);                        }                    }                      features.fromArray(points);                      prevFeatures.fromList(features.toList());                    mPrevGray = mGray.clone();                    break;                } The mPrevGray object refers to the previous frame in a grayscale format. We copied the points to a prevFeatures object that we will use to calculate the optical flow and store the corresponding points in the next frame in nextFeatures. All of the computation is carried out in the calcOpticalFlowPyrLK OpenCV defined function. This function takes in the grayscale version of the previous frame, the current grayscale frame, an object that contains the feature points whose optical flow needs to be calculated, and an object that will store the position of the corresponding points in the current frame:                nextFeatures.fromArray(prevFeatures.toArray());                Video.calcOpticalFlowPyrLK(mPrevGray, mGray,                    prevFeatures, nextFeatures, status, err); Now, we have the position of the grid of points and their position in the next frame as well. So, we will now draw a line that depicts the motion of each point on the grid:                List<Point> prevList=features.toList(), nextList=nextFeatures.toList();                Scalar color = new Scalar(255);                  for(int i = 0; i<prevList.size(); i++){                    Core.line(mGray, prevList.get(i), nextList.get(i), color);                } Before the loop ends, we have to copy the current frame to mPrevGray so that we can calculate the optical flow in the subsequent frames:                mPrevGray = mGray.clone();                break; default: mViewMode = VIEW_MODE_OPTICAL_FLOW; After we end the switch case construct, we will return a Mat object. This is the image that will be displayed as an output to the user of the application. Here, since all our operations and processing were performed on the grayscale image, we will return this image: return mGray; So, this is all about optical flow. The result can be seen in the following image: Optical flow at various points in the camera feed Image pyramids Pyramids are multiple copies of the same images that differ in their sizes. They are represented as layers, as shown in the following figure. Each level in the pyramid is obtained by reducing the rows and columns by half. Thus, effectively, we make the image's size one quarter of its original size: Relative sizes of pyramids Pyramids intrinsically define reduce and expand as their two operations. Reduce refers to a reduction in the image's size, whereas expand refers to an increase in its size. We will use a convention that lower levels in a pyramid mean downsized images and higher levels mean upsized images. Gaussian pyramids In the reduce operation, the equation that we use to successively find levels in pyramids, while using a 5x5 sliding window, has been written as follows. Notice that the size of the image reduces to a quarter of its original size: The elements of the weight kernel, w, should add up to 1. We use a 5x5 Gaussian kernel for this task. This operation is similar to convolution with the exception that the resulting image doesn't have the same size as the original image. The following image shows you the reduce operation: The reduce operation The expand operation is the reverse process of reduce. We try to generate images of a higher size from images that belong to lower layers. Thus, the resulting image is blurred and is of a lower resolution. The equation we use to perform expansion is as follows: The weight kernel in this case, w, is the same as the one used to perform the reduce operation. The following image shows you the expand operation: The expand operation The weights are calculated using the Gaussian function to perform Gaussian blur. Summary In this article, we have seen how to detect a local and global motion in a video, and how we can track objects. We have also learned about Gaussian pyramids, and how they can be used to improve the performance of some computer vision tasks. Resources for Article: Further resources on this subject: New functionality in OpenCV 3.0 [article] Seeing a Heartbeat with a Motion Amplifying Camera [article] Camera Calibration [article]

In this article by John Horton, the author of Android Game Programming by Example, we will look at developing the home screen UI for our game. (For more resources related to this topic, see here.) Creating the project Fire up Android Studio and create a new project by following these steps. On the welcome page of Android Studio, click on Start a new Android Studio project. In the Create New Project window shown next, we need to enter some basic information about our app. These bits of information will be used by Android Studio to determine the package name. In the following image, you can see the Edit link where you can customize the package name if required. If you will be copy/pasting the supplied code into your project, then use C1 Tappy Defender for the Application name field and gamecodeschool.com in the Company Domain field as shown in the following screenshot: Click on the Next button when you're ready. When asked to select the form factors your app will run on, we can accept the default settings (Phone and Tablet). So click on Next again. On the Add an activity to mobile dialog, just click on Blank Activity followed by the Next button. On the Choose options for your new file dialog, again we can accept the default settings because MainActivity seems like a good name for our main Activity. So click on the Finish button. What we did Android Studio has built the project and created a number of files, most of which you will see and edit during the course of building this game. As mentioned earlier, even if you are just copying and pasting the code, you need to go through this step because Android Studio is doing things behind the scenes to make your project work. Building the home screen UI The first and simplest part of your Tappy Defender game is the home screen. All you need is a neat picture with a scene about the game, a high score, and a button to start the game. The finished home screen will look a bit like this: When you built the project, Android Studio opens up two files ready for you to edit. You can see them as tabs in the following Android Studio UI designer. The files (and tabs) are MainActivity.java and activity_main.xml: The MainActivity.java file is the entry point to your game, and you will see this in more detail soon. The activity_main.xml file is the UI layout that your home screen will use. Now, you can go ahead and edit the activity_main.xml file, so it actually looks like your home screen should. First of all, your game will be played with the Android device in landscape mode. If you change your UI preview window to landscape, you will see your progress more accurately. Look for the button shown in the next image. It is just preceding the UI preview: Click on the button shown in the preceding screenshot, and your UI preview will switch to landscape like this: Make sure activity_main.xml is open by clicking on its tab. Now, you will set in a background image. You can use your own. Add your chosen image to the drawable folder of the project in Android Studio. In the Properties window of the UI designer, find and click on the background property as shown in the next image: Also, in the previous image the button labelled ... is outlined. It is just to the right of the background property. Click on that ... button and browse to and select the background image file that you will be using. Next, you need a TextView widget that you will use to display the high score. Note that there is already a TextView widget on the layout. It says Hello World. You will modify this and use it for your high score. Left click on and drag the TextView to where you want it. You can copy me if you intend using the supplied background or put it where it looks best with your background. Next, in the Properties window, find and click on the id property. Enter textHighScore. You can also edit the text property to say High Score: 99999 or similar so that the TextView looks the part. However, this isn't necessary because your Java code will take care of this later. Now, you will drag a button from the widget palette as shown in the following screenshot: Drag it to where it looks good on your background. You can copy me if using the supplied background or put it where it looks best with your background. What we did You now have a cool background with neatly arranged widgets (a TextView and a Button) for your home screen. You can add functionality via Java code to the Button widget next. Revisit the TextView for the player's high score. The important point is that both the widgets have been assigned a unique ID that you can use to reference and manipulate in your Java code. Coding the functionality Now, you have a simple layout for your game home screen. Now, you need to add the functionality that will allow the player to click on the Play button to start the game. Click on the tab for the MainActivity.java file. The code that was automatically generated for us is not exactly what we need. Therefore, we will start again as it is simpler and quicker than tinkering with what is already there. Delete the entire contents of the MainActivity.java file except the package name and enter the following code in it. Of course, your package name may be different. package com.gamecodeschool.c1tappydefender;import android.app.Activity;import android.os.Bundle;public class MainActivity extends Activity{    // This is the entry point to our game    @Override    protected void onCreate(Bundle savedInstanceState) {        super.onCreate(savedInstanceState);                //Here we set our UI layout as the view        setContentView(R.layout.activity_main);    }} The mentioned code is the current contents of your main MainActivity class and the entry point of your game, the onCreate method. The line of code that begins with setContentView... is the line that loads our UI layout from activity_main.xml to the players screen. We can run the game now and see our home screen. Now, let's handle the Play button on our home screen. Add the two highlighted lines of the following code into the onCreate method just after the call to setContentView(). The first new line creates a new Button object and gets a reference to Button in our UI layout. The second line is the code to listen for clicks on the button. //Here we set our UI layout as the viewsetContentView(R.layout.activity_main);// Get a reference to the button in our layoutfinal Button buttonPlay =   (Button)findViewById(R.id.buttonPlay);// Listen for clicksbuttonPlay.setOnClickListener(this); Note that you have a few errors in your code. You can resolve these errors by holding down the Alt keyboard key and then pressing Enter. This will add an import directive for the Button class. You still have one error. You need to implement an interface so that your code listens to the button clicks. Modify the MainActivity class declaration as highlighted: public class MainActivity extends Activity         implements View.OnClickListener{ When you implement the onClickListener interface, you must also implement the onClick method. This is where you will handle what happens when a button is clicked. You can automatically generate the onClick method by right-clicking somewhere after the onCreate method, but within the MainActivity class, and navigating to Generate | Implement methods | onClick(v:View):void. You also need to have Android Studio add another import directive for Android.view.View. Use the Alt | Enter keyboard combination again. You can now scroll to near the bottom of the MainActivity class and see that Android Studio has implemented an empty onClick method for you. You should have no errors in your code at this point. Here is the onClick method: @Overridepublic void onClick(View v) {  //Our code goes here} As you only have one Button object and one listener, you can safely assume that any clicks on your home screen are the player pressing your Play button. Android uses the Intent class to switch between activities. As you need to go to a new activity when the Play button is clicked, you will create a new Intent object and pass in the name of your future Activity class, GameActivity to its constructor. You can then use the Intent object to switch activities. Add the following code to the body of the onClick method: // must be the Play button.// Create a new Intent objectIntent i = new Intent(this, GameActivity.class);// Start our GameActivity class via the IntentstartActivity(i);// Now shut this activity downfinish(); Once again, you have errors in your code because you need to generate a new import directive, this time for the Intent class so use the Alt | Enter keyboard combination again. You still have one error in your code. This is because your GameActivity class does not exist yet. You will now solve this problem. Creating GameActivity You have seen that when the player clicks on the Play button, main activity will close and game activity will begin. Therefore, you need to create a new activity called GameActivity that will be where your game actually executes. From the main menu, navigate to File | New | Activity | Blank Activity. In the Choose options for your new file dialog, change the Activity name field to GameActivity. You can accept all the other default settings from this dialog, so click on Finish. As you did with your MainActivity class, you will code this class from scratch. Therefore, delete the entire code content from GameActivity.java. What we did Android Studio has generated two more files for you and done some work behind the scenes that you will investigate soon. The new files are GameActivity.java and activity_game.xml. They are both automatically opened for you in two new tabs, in the same place as the other tabs above the UI designer. You will never need activity_game.xml because you will build a dynamically generated game view, not a static UI. Feel free to close that now or just ignore it. You will come back to the GameActivity.java file, when you start to code your game for real. Configuring the AndroidManifest.xml file We briefly mentioned that when we create a new project or a new activity, Android Studio does more than just creating two files for us. This is why we create new projects/activities the way we do. One of the things going on behind the scenes is the creation and modification of the AndroidManifest.xml file in the manifests directory. This file is required for your app to work. Also, it needs to be edited to make your app work the way you want it to. Android Studio has automatically configured the basics for you, but you will now do two more things to this file. By editing the AndroidManifest.xml file, you will force both of your activities to run with a full screen, and you will also lock them to a landscape layout. Let's make these changes here: Open the manifests folder now, and double click on the AndroidManifest.xml file to open it in the code editor. In the AndroidManifest.xml file, find the following line of code:android:name=".MainActivity" Immediately following it, type or copy and paste these two lines to make MainActivity run full screen and lock it in the landscape orientation:android:theme="@android:style/Theme.NoTitleBar.Fullscreen"android:screenOrientation="landscape" In the AndroidManifest.xml file, find the following line of code:android:name=".GameActivity" Immediately following it, type or copy and paste these two lines to make GameActivity run full screen and lock it in the landscape orientation: android:theme="@android:style/Theme.NoTitleBar.Fullscreen"android:screenOrientation="landscape" What you did You have now configured both activities from your game to be full screen. This gives a much more pleasing appearance for your player. In addition, you have disabled the player's ability to affect your game by rotating their Android device. Continue building on what you've learnt so far with Android Game Programming by Example! Learn to implement the game rules, game mechanics, and game objects such as guns, life, money; and of course, the enemy. You even get to control a spaceship!

In the past years, many types of attacks on the WEP protocol have been undertaken. Being successful with such an attack is an important milestone for anyone who wants to undertake penetration tests of wireless networks. In this article by Marco Alamanni, the author of Kali Linux Wireless Penetration Testing Essentials, we will take a look at the basics and the most common types of WEP protocols. What is the WEP protocol? The WEP protocol was introduced with the original 802.11 standard as a means to provide authentication and encryption to wireless LAN implementations. It is based on the Rivest Cipher 4 (RC4) stream cypher with a Pre-shared Secret Key (PSK) of 40 or 104 bits, depending on the implementation. A 24-bit pseudorandom Initialization Vector (IV) is concatenated with the pre-shared key to generate the per-packet keystream used by RC4 for the actual encryption and decryption process. Thus, the resulting keystream could be 64 or 128 bits long. In the encryption phase, the keystream is encrypted with the XOR cypher with the plaintext data to obtain the encrypted data. While in the decryption phase, the encrypted data is XOR-encrypted with the keystream to obtain the plaintext data. The encryption process is shown in the following diagram: Attacks against WEP and why do they occur? WEP is an insecure protocol and has been deprecated by the Wi-Fi Alliance. It suffers from various vulnerabilities related to the generation of the keystreams, to the use of IVs (initialization vectors), and to the length of the keys. The IV is used to add randomness to the keystream, trying to avoid the reuse of the same keystream to encrypt different packets. This purpose has not been accomplished in the design of WEP because the IV is only 24 bits long (with 2^24 =16,777,216 possible values) and it is transmitted in clear text within each frame. Thus, after a certain period of time (depending on the network traffic), the same IV and consequently the same keystream will be reused, allowing the attacker to collect the relative cypher texts and perform statistical attacks to recover plain texts and the key. FMS attacks on WEP The first well-known attack against WEP was the Fluhrer, Mantin, and Shamir (FMS) attack back in 2001. The FMS attack relies on the way WEP generates the keystreams and on the fact that it also uses weak IV to generate weak keystreams, making it possible for an attacker to collect a sufficient number of packets encrypted with these keys, to analyze them, and recover the key. The number of IVs to be collected to complete the FMS attack is about 250,000 for 40-bit keys and 1,500,000 for 104-bit keys. The FMS attack has been enhanced by Korek, improving its performance. Andreas Klein found more correlations between the RC4 keystream and the key than the ones discovered by Fluhrer, Mantin, and Shamir, which can be used to crack the WEP key. PTW attacks on WEP In 2007, Pyshkin, Tews, and Weinmann (PTW) extended Andreas Klein's research and improved the FMS attack, significantly reducing the number of IVs needed to successfully recover the WEP key. Indeed, the PTW attack does not rely on weak IVs such as the FMS attack does and is very fast and effective. It is able to recover a 104-bit WEP key with a success probability of 50% using less than 40,000 frames and with a probability of 95% with 85,000 frames. The PTW attack is the default method used by Aircrack-ng to crack WEP keys. ARP Request replay attacks on WEP Both FMS and PTW attacks need to collect quite a large number of frames to succeed and can be conducted passively, sniffing the wireless traffic on the same channel of the target AP and capturing frames. The problem is that, in normal conditions, we will have to spend quite a long time to passively collect all the necessary packets for the attacks, especially with the FMS attack. To accelerate the process, the idea is to reinject frames in the network to generate traffic in response so that we can collect the necessary IVs more quickly. A type of frame that is suitable for this purpose is the ARP request because the AP broadcasts it, each time with a new IV. As we are not associated with the AP, if we send frames to it directly, they are discarded and a de-authentication frame is sent. Instead, we can capture ARP requests from associated clients and retransmit them to the AP. This technique is called the ARP Request Replay attack and is also adopted by Aircrack-ng for the implementation of the PTW attack. Find out more to become a master penetration tester by reading Kali Linux Wireless Penetration Testing Essentials

In this article written by James Preston, author of the book Microsoft Application Virtualization Cookbook, we will cover the following topics: Enabling the App-V shared content store mode Publishing applications through Microsoft RemoteApp Pre-caching applications in the local store Publishing applications through Citrix StoreFront (For more resources related to this topic, see here.) App-V 5 is the perfect companion for your virtual session or desktop delivery environment, allowing you to abstract applications from the user and desktop, as shown in the following image, and, in turn, reducing infrastructure requirements through features such as shared content store mode.   In this article, we will cover how to deploy App-V5 in these environments. Enabling the App-V shared content store mode In this recipe, we will cover enabling the App-V shared content store mode, which prevents the caching of App-V files on a client so that the application is launched from the server hosting the application directly. This feature is ideal for environments where there is ample network bandwidth between remote desktop session hosts (or client virtual machines in a VDI deployment) and where administrators are looking to reduce the overall need for storage by the hosts. While some files are still cached on the local machine (for example, for shortcuts or Shell extensions), the following screenshot shows the amount of storage saved on an Office 2013 deployment, where the shared content store mode is turned on (the screenshot on the right):   With the shared content store mode enabled, you can check the amount of storage space used by a package by checking the size of the individual package's folders at the following path on a client where the package is deployed (where Package ID is the GUID assigned to that package): C:ProgramDataApp-V<Package ID>. Getting ready To complete these steps, you will need to deploy a Remote Desktop Services environment (on the server RDS). The server RDS must also have the App-V client and any prerequisites deployed on it. How to do it… The following list shows you the high-level tasks involved in this recipe and the tasks required to complete the recipe (all of the actions in this recipe will take place on the server DC): Link the App-V5 Settings Group Policy Object to the Remote Desktop Server's OU. Create a Group Policy object for the server RDS. Enable the shared content store mode within that policy. The implementation of the preceding tasks is as follows: On the server DC, load the Group Policy Management console. Expand the tree structure to display the Remote Desktop Server's Organizational Unit and click on Link an Existing GPO…. From the window that appears, select the App-V 5 Settings policy and click on OK. Next, right-click on the OU and select Create a GPO in this domain, and Link it here.... Set the name of the policy as App-V 5 Shared Content Store and click on OK. Let's take a look at the following screenshot: Right-click on the policy you have just created and click on Edit…. In the window that appears, right-click on App-V 5 Shared Content Store and click on Properties. Then, tick the Disable User Configuration settings box and click on OK. Next, navigate to Computer Configuration | Policies | Administrative Templates | System | App-V | Streaming and double-click on Shared Content (SCS) mode. Set the policy to Enabled and click on OK. There's more… To verify that the setting is applied on the server RDS, open a PowerShell session and run the following command: Get-AppvClientConfiguration If the SharedContentStoreMode value is 1 and the SetByGroupPolicy value is True, then the policy is correctly applied. Publishing applications through Microsoft RemoteApp In this recipe, we will publish the Audacity package to the RDS server to be accessed by users through the Remote Desktop Web Access. Getting ready To complete these steps, you will need to deploy a Remote Desktop Services environment (on the server RDS). How to do it… The following list shows you the high-level tasks involved in this recipe and the tasks required to complete the recipe (all of the actions in this recipe will take place on the server RDS): Create a Security Group for your remote desktop session hosts. Publish the Audacity package to that Security Group through the App-V Management console. Publish the Audacity package through Server Manager. The implementation of the preceding tasks is as follows: On the server DC, launch the Active Directory Users and Computers console and navigate to demo.org | Domain Groups, and create a new Security Group called RDS Session Hosts. Add the server RDS to the group that you just created. On your Windows 8 client PC, log in to the App-V Management console as Sam Adams, select the Audacity package and click on the Edit option next to the AD ACCESS option. Under FIND VALID ACTIVE DIRECTORY GROUP AND GRANT ACCESS, enter demo.orgRDS Session Hosts and click on Check. In the drop-down menu that appears, select RDS Session Hosts and click on Grant Access. On the server RDS, wait for the App-V Publishing Refresh to occur (or force the process manually) for the Audacity shortcut to appear on the desktop. Launch Server Manager, and from the left-hand side bar, select Remote Desktop. From the left-hand side, select QuickSessionCollection (the collection created by default). Under REMOTEAPP PROGRAMS, navigate to Tasks | Publish RemoteApp Programs. In the window that appears, tick the box next to Audacity and click on Next, as shown in the following screenshot: Note that the path to the Audacity application is pointing at the App-V Installation root in %SYSTEMDRIVE%ProgramDataMicrosoftAppV. Review the confirmation window and click on Publish. On your Windows 8 client, open Internet Explorer and browse to https://rds.demo.org/RDWeb, accepting any invalid SSL certificate prompts and allowing the Remote Desktop plugin to run. Log in as Sam Adams and launch the Audacity application.   There's more… It is possible to limit applications within a remote desktop collection to users in a specific Security Group. To do this, right-click on the application as it appears under REMOTEAPP PROGRAMS and click on Edit Properties:   In the window that appears, click on User Assignment and set the radio button to Only specified users and groups. You will now be able to access the Add… button, which brings up an Active Directory search dialog, from where you can add the Audacity Users security group to limit the application to only the users in that group.   Precaching applications in the local store As an alternative to using the Shared Content Store mode, applications can be forced to be cached within the local store on your RDS session hosts. This would be advantageous in scenarios where the bandwidth from a central high speed storage device is more expensive than providing dedicated storage to the RDS session hosts. Getting ready To complete these tasks, you will need to deploy a Remote Desktop Services environment (on the server RDS). How to do it… The following list shows you the high-level tasks involved in this recipe and the tasks required to complete the recipe (all of the actions in this recipe will take place on the server DC): Create a group policy object for the server RDS. Enable background application caching within that policy. The implementation of the preceding tasks is as follows: On the server DC, load the Group Policy Management console. Expand the tree structure to display Remote Desktop Servers Organizational Unit, right-click on the OU, and select Create a GPO in this domain, and Link it here.... Set the name of the policy to App-V 5 Cache Applications and click on OK. Right-click on the policy you have just created and click on Edit…. In the window that appears, right-click on App-V5 Cache Applications and click on Properties, tick the Disable User Configuration settings box and click on OK. Next, navigate to Computer Configuration | Policies | Administrative Templates | System | App-V | Specify what to load in background (aka AutoLoad). Set the policy to Enabled, with Autoload Options set to All, and click on OK. There's more… Individual applications can be targeted for caching using the Mount-AppvClientPackage PowerShell command. For example, to mount the package named Audacity 2.0.6 (which has been already published to the Remote Desktop session host), the administrator would run the following command: Mount-AppvClientPackage –Name "Audacity 2.0.6" This would generate the following result:   Note that the PercentLoaded value is shown as 100 to indicate that the package is completely loaded within the local store. Publishing applications through Citrix StoreFront Apart from being a great addition to the Microsoft Virtual environment, App-V is also supported by Citrix XenDesktop. In this recipe, we will look at publishing the Audacity package through Citrix StoreFront. Getting ready In addition to this, the server XenDesktop and XD-HOST will be used in this recipe. XenDesktop is configured with an installation of XenDesktop 7.6 with a Machine Catalogue containing the server XD-HOST (configured as a Server OS Machine) and a delivery group that has been set up to service both applications and desktops. The server XD-HOST should have the App-V RDS client installed. Finally, the App-V applications that you wish to deploy through Citrix StoreFront must also be published to the server XD-HOST through the App-V Management console; in this case, Audacity. How to do it… The following list shows you the high-level steps involved in this recipe and the tasks required to complete the recipe (all of the actions in this recipe will take place on the server XenDesktop): Set up App-V Publishing in Citrix Studio. Publish applications through the Delivery Group. The implementation of the preceding tasks is as follows: On the server XenDesktop, launch Citrix Studio. Navigate to Citrix Studio | Configuration, right-click on App-V Publishing, and click on Add App-V Publishing. In the window that appears, enter the details of your App-V Management and Publishing servers, click on Test connection… to confirm that the details are correct and then click on Save. Navigate to Delivery Groups, right-click on the delivery group you have created, and click on Add Applications. On the introduction page of the wizard that appears, click on Next. On the applications page of the wizard, select Audacity from the list provided (which will be discovered automatically from your server XD-HOST) and click on Next. Note that you can also select to publish multiple applications at the same time. Review the summary screen and click on Finish. There's more… Similar to publishing through the Microsoft remote desktop web app, it is possible to limit access to your applications to specific users or security groups. To limit access, right-click on your application in the Applications tab of the Delivery Groups page and click on Properties.   In the window that appears, select the Limit Visibility tab and select Limit visibility for this application to the users listed below. Click on the Add users… button to choose users and security groups from Active Directory to be included in the group. Summary In this article, we learned about enabling App-V shared content store mode which prevents the caching of App-V sored files on the client system. We also had a look into publishing applications through Microsoft RemoteApp which publishes the Audacity package to the RDS server which enables the user to access from the Remote Desktop Web Access. Then we learned about precaching application in the local store which forces the application to be cached in to the RDS session hosts, which has certain advantages. Finally, we learned about publishing the applications through Citrix StoreFront where we published the Audacity server through Citrix StoreFront. Resources for Article: Further resources on this subject: Virtualization [article] Customization in Microsoft Dynamics CRM [article] Installing Postgre SQL [article]

In this article by Robert Craig Layton, author of Learning Data Mining with Python, we will look at predicting the winner of games of the National Basketball Association (NBA) using a different type of classification algorithm—decision trees. Collecting the data The data we will be using is the match history data for the NBA, for the 2013-2014 season. The Basketball-Reference.com website contains a significant number of resources and statistics collected from the NBA and other leagues. Perform the following steps to download the dataset: Navigate to http://www.basketball-reference.com/leagues/NBA_2014_games.html in your web browser. Click on the Export button next to the Regular Season heading. Download the file to your data folder (and make a note of the path). This will download a CSV file containing the results of 1,230 games in the regular season of the NBA. We will load the file with the pandas library, which is an incredibly useful library for manipulating data. Python also contains a built-in library called csv that supports reading and writing CSV files. We will use pandas instead as it provides more powerful functions to work with datasets. For this article, you will need to install pandas. The easiest way to do that is to use pip3, which you may previously have used to install scikit-learn: $pip3 install pandas Using pandas to load the dataset We can load the dataset using the read_csv function in pandas as follows: import pandas as pddataset = pd.read_csv(data_filename) The result of this is a data frame, a data structure used by pandas. The pandas.read_csv function has parameters to fix some of the problems in the data, such as missing headings, which we can specify when loading the file: dataset = pd.read_csv(data_filename, parse_dates=["Date"],skiprows=[0,])dataset.columns = ["Date", "Score Type", "Visitor Team","VisitorPts", "Home Team", "HomePts", "OT?", "Notes"] We can now view a sample of the data frame: dataset.ix[:5] Extracting new features We extract our classes, 1 for a home win, and 0 for a visitor win. We can specify this using the following code to extract those wins into a NumPy array: dataset["HomeWin"] = dataset["VisitorPts"] < dataset["HomePts"] y_true = dataset["HomeWin"].values The first two new features we want to create are to indicate whether each of the two teams won their previous game. This would roughly approximate which team is currently playing well. We will compute this feature by iterating through the rows in order, and recording which team won. When we get to a new row, we look up whether the team won the last time: from collections import defaultdictwon_last = defaultdict(int) We can then iterate over all the rows and update the current row with the team's last result (win or loss): for index, row in dataset.iterrows():home_team = row["Home Team"]visitor_team = row["Visitor Team"]row["HomeLastWin"] = won_last[home_team]row["VisitorLastWin"] = won_last[visitor_team]dataset.ix[index] = row We then set our dictionary with each team's result (from this row) for the next time we see these teams: won_last[home_team] = row["HomeWin"]won_last[visitor_team] = not row["HomeWin"] Decision trees Decision trees are a class of classification algorithm such as a flow chart that consist of a sequence of nodes, where the values for a sample are used to make a decision on the next node to go to. We can use the DecisionTreeClassifier class to create a decision tree: from sklearn.tree import DecisionTreeClassifierclf = DecisionTreeClassifier(random_state=14) We now need to extract the dataset from our pandas data frame in order to use it with our scikit-learn classifier. We do this by specifying the columns we wish to use and using the values parameter of a view of the data frame: X_previouswins = dataset[["HomeLastWin", "VisitorLastWin"]].values Decision trees are estimators, and therefore, they have fit and predict methods. We can also use the cross_val_score method as before to get the average score: scores = cross_val_score(clf, X_previouswins, y_true,scoring='accuracy')print("Accuracy: {0:.1f}%".format(np.mean(scores) * 100)) This scores up to 56.1%; we are better off choosing randomly! Predicting sports outcomes We have a method for testing how accurate our models are using the cross_val_score method that allows us to try new features. For the first feature, we will create a feature that tells us whether the home team is generally better than the visitors by seeing whether they ranked higher in the previous season. To obtain the data, perform the following steps: Head to http://www.basketball-reference.com/leagues/NBA_2013_standings.html Scroll down to Expanded Standings. This gives us a single list for the entire league. Click on the Export link to the right of this heading. Save the download in your data folder. In your IPython Notebook, enter the following into a new cell. You'll need to ensure that the file was saved into the location pointed to by the data_folder variable: standings_filename = os.path.join(data_folder,"leagues_NBA_2013_standings_expanded-standings.csv")standings = pd.read_csv(standings_filename, skiprows=[0,1]) We then iterate over the rows and compare the team's standings: dataset["HomeTeamRanksHigher"] = 0for index, row in dataset.iterrows():home_team = row["Home Team"]visitor_team = row["Visitor Team"] Between 2013 and 2014, a team was renamed as follows: if home_team == "New Orleans Pelicans":home_team = "New Orleans Hornets"elif visitor_team == "New Orleans Pelicans":visitor_team = "New Orleans Hornets" Now, we can get the rankings for each team. We then compare them and update the feature in the row: home_rank = standings[standings["Team"] ==home_team]["Rk"].values[0]visitor_rank = standings[standings["Team"] ==visitor_team]["Rk"].values[0]row["HomeTeamRanksHigher"] = int(home_rank > visitor_rank)dataset.ix[index] = row Next, we use the cross_val_score function to test the result. First, we extract the dataset as before: X_homehigher = dataset[["HomeLastWin", "VisitorLastWin", "HomeTeamRanksHigher"]].values Then, we create a new DecisionTreeClassifier class and run the evaluation: clf = DecisionTreeClassifier(random_state=14)scores = cross_val_score(clf, X_homehigher, y_true,scoring='accuracy')print("Accuracy: {0:.1f}%".format(np.mean(scores) * 100)) This now scores up to 60.3%—even better than our previous result. Unleash the full power of Python machine learning with our 'Learning Data Mining with Python' book.

In this article by Artemij Fedosejev, the author of React.js Essentials, we will take a look at test suites, specs, and expectations. To write a test for JavaScript functions, you need a testing framework. Fortunately, Facebook built their own unit test framework for JavaScript called Jest. It is built on top of Jasmine - another well-known JavaScript test framework. If you’re familiar with Jasmine you’ll find Jest's approach to testing very similar. However I'll make no assumptions about your prior experience with testing frameworks and discuss the basics first. The fundamental idea of unit testing is that you test only one piece of functionality in your application that usually is implemented by one function. And you test it in isolation - meaning that all other parts of your application which that function depends on are not used by your tests. Instead, they are imitated by your tests. To imitate a JavaScript object is to create a fake one that simulates the behavior of the real object. In unit testing the fake object is called mock and the process of creating it is called mocking. Jest automatically mocks dependencies when you're running your tests. Better yet, it automatically finds tests to execute in your repository. Let's take a look at the example. Create a directory called ./snapterest/source/js/utils/ and create a new file called TweetUtils.js within it, with the following contents: function getListOfTweetIds(tweets) {  return Object.keys(tweets);}module.exports.getListOfTweetIds = getListOfTweetIds; TweetUtils.js file is a module with the getListOfTweetIds() utility function for our application to use. Given an object with tweets, getListOfTweetIds() returns an array of tweet IDs. Using the CommonJS module pattern we export this function: module.exports.getListOfTweetIds = getListOfTweetIds; Jest Unit Testing Now let's write our first unit test with Jest. We'll be testing our getListOfTweetIds() function. Create a new directory: ./snapterest/source/js/utils/__tests__/. Jest will run any tests in any __tests__ directories that it finds within your project structure. So it's important to name your directories with tests: __tests__. Create a TweetUtils-test.js file inside of __tests__:jest.dontMock('../TweetUtils');describe('Tweet utilities module', function () {  it('returns an array of tweet ids', function () {    var TweetUtils = require('../TweetUtils');    var tweetsMock = {      tweet1: {},      tweet2: {},      tweet3: {}    };    var expectedListOfTweetIds = ['tweet1', 'tweet2', 'tweet3'];    var actualListOfTweetIds = TweetUtils.getListOfTweetIds(tweetsMock);    expect(actualListOfTweetIds).toBe(expectedListOfTweetIds);  });}); First we tell Jest not to mock our TweetUtils module: jest.dontMock('../TweetUtils'); We do this because Jest will automatically mock modules returned by the require() function. In our test we're requiring the TweetUtils module: var TweetUtils = require('../TweetUtils'); Without the jest.dontMock('../TweetUtils') call, Jest would return an imitation of our TweetUtils module, instead of the real one. But in this case we actually need the real TweetUtils module, because that's what we're testing. Creating test suites Next we call a global Jest function describe(). In our TweetUtils-test.js file we're not just creating a single test, instead we're creating a suite of tests. A suite is a collection of tests that collectively test a bigger unit of functionality. For example a suite can have multiple tests which tests all individual parts of a larger module. In our example, we have a TweetUtils module with a number of utility functions. In that situation we would create a suite for the TweetUtils module and then create tests for each individual utility function, like getListOfTweetIds(). describe defines a suite and takes two parameters: Suite name - the description of what is being tested: 'Tweet utilities module'. Suit implementation: the function that implements this suite. In our example, the suite is: describe('Tweet utilities module', function () {  // Suite implementation goes here...}); Defining specs How do you create an individual test? In Jest, individual tests are called specs. They are defined by calling another global Jest function it(). Just like describe(), it() takes two parameters: Spec name: the title that describes what is being tested by this spec: 'returns an array of tweet ids'. Spec implementation: the function that implements this spec. In our example, the spec is: it('returns an array of tweet ids', function () {  // Spec implementation goes here...}); Let's take a closer look at the implementation of our spec: var TweetUtils = require('../TweetUtils');var tweetsMock = {  tweet1: {},  tweet2: {},  tweet3: {}};var expectedListOfTweetIds = ['tweet1', 'tweet2', 'tweet3'];var actualListOfTweetIds = TweetUtils.getListOfTweetIds(tweetsMock);expect(actualListOfTweetIds).toEqual(expectedListOfTweetIds); This spec tests whether getListOfTweetIds() method of our TweetUtils module returns an array of tweet IDs when given an object with tweets. First we import the TweetUtils module: var TweetUtils = require('../TweetUtils'); Then we create a mock object that simulates the real tweets object: var tweetsMock = {  tweet1: {},  tweet2: {},  tweet3: {}}; The only requirement for this mock object is to have tweet IDs as object keys. The values are not important hence we choose empty objects. Key names are not important either, so we can name them tweet1, tweet2 and tweet3. This mock object doesn't fully simulate the real tweet object. Its sole purpose is to simulate the fact that its keys are tweet IDs. The next step is to create an expected list of tweet IDs: var expectedListOfTweetIds = ['tweet1', 'tweet2', 'tweet3']; We know what tweet IDs to expect because we've mocked a tweets object with the same IDs. The next step is to extract the actual tweet IDs from our mocked tweets object. For that we use getListOfTweetIds()that takes the tweets object and returns an array of tweet IDs: var actualListOfTweetIds = TweetUtils.getListOfTweetIds(tweetsMock); We pass tweetsMock to that method and store the results in actualListOfTweetIds. The reason this variable is named actualListOfTweetIds is because this list of tweet IDs is produced by the actual getListOfTweetIds() function that we're testing. Setting Expectations The final step will introduce us to a new important concept: expect(actualListOfTweetIds).toEqual(expectedListOfTweetIds); Let's think about the process of testing. We need to take an actual value produced by the method that we're testing - getListOfTweetIds(), and match it to the expected value that we know in advance. The result of that match will determine if our test has passed or failed. The reason why we can guess what getListOfTweetIds() will return in advance is because we've prepared the input for it - that's our mock object: var tweetsMock = {  tweet1: {},  tweet2: {},  tweet3: {}}; So we can expect the following output from calling TweetUtils.getListOfTweetIds(tweetsMock): ['tweet1', 'tweet2', 'tweet3'] But because something can go wrong inside of getListOfTweetIds() we cannot guarantee this result - we can only expect it. That's why we need to create an expectation. In Jest, an Expectation is built using expect()which takes an actual value, for example: actualListOfTweetIds. expect(actualListOfTweetIds) Then we chain it with a Matcher function that compares the actual value with the expected value and tells Jest whether the expectation was met. expect(actualListOfTweetIds).toEqual(expectedListOfTweetIds); In our example we use the toEqual() matcher function to compare two arrays. Click here for a list of all built-in matcher functions in Jest. And that's how you create a spec. A spec contains one or more expectations. Each expectation tests the state of your code. A spec can be either a passing spec or a failing spec. A spec is a passing spec only when all expectations are met, otherwise it's a failing spec. Well done, you've written your first testing suite with a single spec that has one expectation. Continue reading React.js Essentials to continue your journey into testing.

In this two part post, you'll learn the basics of how to benchmark and optmize Go code. You'll do this by trying out a few different implementations for calculating a specific element in Pascal's Triangle (a.k.a. the binomial coeffecient). In Part 1 we setup Go, covered the structure of our example, created an interface, and conducted some testing. Here in Part 2 we will cover benchmarking our example for performance, and look at a recursive and bultin implementation. All of the code for these two posts is available on github. Benchmarking for Performance Since we've verified that our implementation works in Part 1, let's benchmark it to see how fast it is. Benchmarking one implementation by itself is not incredibly useful, but soon we will benchmark other implementations too and compare the results. In Go, benchmarks are simply functions that start with the word "Benchmark" and accept *testing.B as an argument. Like testing functions, benchmark functions also need to exist in a file that ends in "_test.go". As we did for our tests, we'll write a utility function that we can use to benchmark any implementation. Add the following to test/benchmark_test.go package test import ( "github.com/albrow/go-benchmark-example/common" "github.com/albrow/go-benchmark-example/implementations" "testing" ) func BenchmarkNaive(b *testing.B) { benchmarkPascaler(b, implementations.Naive) } func benchmarkPascaler(b *testing.B, p common.Pascaler) { for i := 0; i < b.N; i++ { p.Pascal(32, 16) } } We'll use the parameters n=32 and m=16 for our benchmarks. You could actually choose anything you want, but if n gets too big, you'll overflow any int values (which are stored in 64 bits on 64-bit operating systems). If that happens, any implementation that relies on storing numbers as ints will return incorrect results (and may even panic, depending on what math you do). When I tested the limit, overflow first started happening around n=67, m=33. You can run the benchmark by adding the bench flag to the test command: go test ./test -bench .. If it works, your output should look like this: ? github.com/albrow/go-benchmark-example/common [no test files] ? github.com/albrow/go-benchmark-example/implementations [no test files] PASS BenchmarkNaive 100000 15246 ns/op ok github.com/albrow/go-benchmark-example/test 3.970s The important bit is the line that starts with BenchmarkNaive. The first number tells us that this particular benchmark was run 100,000 times. Go will automatically run the benchmark as many times as it needs to until it gets a consistent result. Typically, faster benchmarks will be run more times. The second number tells us that it took an average of 15,246 ns per operation. The exact number can vary widely from machine to machine, which is why it doesn't mean much on its own. Let's write another implementation so we have something to compare it to. Recursive Implementation You may have noticed that our naive implementation was doing more work than it needed to. We don't actually need to construct the entire triangle to figure out a specific element. We actually only need to calculate the elements directly above the one we want (along with the elements directly above those, and so on). What if instead of iterating through from top to bottom, we started at the bottom and iterated upwards, calculating only the elements we needed? It should perform better because we're doing less calculations, right? Let's write an implementation that does exactly that and benchmark it to find out for sure. The idea of moving from bottom to top leads pretty naturally to a recursive implementation. For the base case, we know that the top element is 1. We also know that any element on the edges, where m == 0 or m == n is 1. For all other cases, the element is equal to the sum of the elements directly above it. We'll write the implementation in implementations/recursive.go, and it looks like this: package implementations type recursiveType struct{} var Recursive = &recursiveType{} func (p *recursiveType) Pascal(n, m int) int { if n == 0 || m == 0 || n == m { return 1 } return p.Pascal(n-1, m-1) + p.Pascal(n-1, m) } func (p *recursiveType) String() string { return "Recursive Implementation" } Personally, I think this implementation is much cleaner and easier to understand. Let's test it for correctness by adding a few lines to test/pascal_test.go: func TestRecursive(t *testing.T) { testPascaler(t, implementations.Recursive) } Then run the tests again with go test ./.... If it works you should see the same output as before. Now, let's benchmark this implementation to see how fast it is compared to our first one. Add the following lines to test/benchmark_test.go: func BenchmarkRecursive(b *testing.B) { benchmarkPascaler(b, implementations.Recursive) } Then run the benchmarks with go test ./test -bench .. You might be surprised to find that with the parameters n=32, m=16, the recursive implementation is much, much slower! In fact, if you increase n much more the benchmark will timeout because the recursive implementation takes too long. These were the results on my machine: BenchmarkNaive 100000 15246 ns/op BenchmarkRecursive 1 4005199128 ns/op This shows the recursive implementation took just over 4 seconds, whereas the "naive" implementation took a tiny fraction of a millisecond! The reason this happens has to do with the way Go allocates stack frames for functions. Currently, the Go compiler does not optimize much for recursion (though it might in the future), which means our code results in a lot of extra memory allocations. In languages/compilers that heavily optimize recursive function calls, it's possible that the recursive implementation would be faster, but not so in Go. A naive runtime analysis that didn't take stack frame allocation into account would incorrectly predict that the recursive implementation would be faster. This example illustrates why benchmarking is important whenever you care about performance! Built-in Implementation There are a lot of techniques you could use to optimize the function further. You could try writing an implementation that moves from bottom to top without using recursion. You could take advantage of the symmetry of Pascal's Triangle and cut number of calculations in half. You could even take advantage of memoization or use a lookup table to prevent repeated calculations. But for the sake of time, in this two part series we're just going to try one more implementation. If you are familiar with Combinatorics or Statistics, you might recognize that what we've been trying to calculate with our Pascal function is the equivalent of the binomial coefficient. There are formulas for calcuating the binomial coefficient directly, such as without building a complicated data structure. The most common formula involves factorials and is as follows: binomial(n, k) = n! / (k! * (n-k)!) One unexpected problem with this formula is that the intermediate values in the numerator and denominator can get too large and overflow 64-bit integers. This can happen even when the final result can be represented perfectly with 64 bits. Luckily, there is a package in the Go standard library called "big" for dealing with numbers that are too big to fit in a 64-bit int. That package also has a function for calculating the binomial coeffecient, which is exactly what we're looking for. Here's how we can use the builtin function to create our own implementation in implementations/builtin.go: package implementations import ( "math/big" ) type builtinType struct{} var Builtin = builtinType{} func (p builtinType) Pascal(n, m int) int { z := big.NewInt(0) z.Binomial(int64(n), int64(m)) return int(z.Int64()) } func (p builtinType) String() string { return "Builtin Implementation" } Test it for correctness by adding the following lines to test/pascal_test.go and running the tests again: func TestBuiltin(t *testing.T) { testPascaler(t, implementations.Builtin) } Then, we can compare the performance by adding the following lines to test/benchmark_test.go and running the benchmarks: func BenchmarkBuiltin(b *testing.B) { benchmarkPascaler(b, implementations.Builtin) } On my machine, the builtin implementation was only slightly faster for parameters n=32 and m=16: func BenchmarkBuiltin(b *testing.B) { benchmarkPascaler(b, implementations.Builtin) } Let's see what happens if we increase n and m. You will probably need to skip the recursive implementation by adding the following line inside of the BenchmarkRecursive function: b.Skip("Skipping slower recursive implementation") Here are the results on my machine for n=64, m=32: BenchmarkNaive 50000 40234 ns/op BenchmarkBuiltin 50000 25233 ns/op As you can see, when we increase the difficulty, the builtin implementation really starts to out-perform the others. We can go even higher, but keep in mind that doing so will overflow 64-bit ints. Since the builtin implementation uses the big package, we could work around this restriction. However, the benchmark results for the naive implementation wouldn't mean much since it can't return the correct values. To demonstrate how the builtin implementation performs with higher values of n, here's the results of n=10,000 and m=5,000 on my machine (skipping the other implementations): BenchmarkBuiltin 300 4978329 ns/op That's only 4 ms to compute a really high coeffecient. It appears that for high values of n and m, the builtin implementation is by far the best, not to mention the fact that it is the only one out of the three that returns correct results for higher values of n and m. Conclusion In this two part series, we tested and benchmarked three different implementations of the Pascal function, we learned how to write benchmarking functions, and we found that sometimes the fastest implementation is not necessarily the one you would expect. Benchmarking is critical whenever you are writing performance-sensitive code. Looking for a challenge? See if you can come up with an implementation to beat the builtin one. Some ideas are using parallelization, memoization, GPU acceleration, or binding to some optimized c libraries with cgo. Don't forget to test each implementation for correctness. Feel free to send a pull request here if you are able to beat it. Good luck! About the Author Alex Browne is a programmer and entrepreneur with about 4 years of product development experience and 3 years experience working on small startups. He has worked with a wide variety of technologies and has single-handedly built many production-grade applications. He is currently working with two co-founders on an early stage startup exploring ways of applying machine learning and computer vision to the manufacturing industry.His favorite programming language is Go.

Product Hunt addicts like me might have noticed how often a "developer" tab was available on landing pages. More and more modern products offer a special entry point tailored for coders who want deeper interaction, beyond standard end-user experience. Twitter, Myo, Estimote are great examples of technologies an engineer could leverage for its own tool/product. And Application Programming Interfaces (API) make it possible. Companies design them as a communication contract between the developer and their product. We can discern Representational State Transfer APIs (RESTful) from programmatic ones. The latter usually offer deeper technical integration, while the former tries to abstract most of the product's complexity behind intuitive remote resources (more on that later). The resulting simplicity owes a lot to the HTTP protocol and turns out to be trickier than one thinks. Both RESTful servers and clients often underestimates the value of HTTP historical rules or the challenges behind network failures. I will dump in this article my last experience in building an HTTP+JSON API client. We are going to build a small framework in python to interact with well-designed third party services. One should get out of it a consistent starting point for new projects, like remotely controlling its car ! Stack and Context Before diving in, let's state an important assumption : APIs our client will call are well designed. They enforce RFC standards, conventions and consistent resources. Sometimes, however, real world throws at us ugly interfaces. Always read the documentation (if any) and deal with it. The choice of Python should be seen as a minor implementation consideration. Nevertheless, it will bring us the powerful requests package and a nice repl to manually explore remote services. Its popularity also suggests we are likely to be able to integrate our future package in a future project. To keep things practical, requests will hit Consul HTTP endpoints, providing us with a handy interface for our infrastructure. Consul, as a whole, it is a tool for discovering and configuring services in your infrastructure. Just download the appropriate binary, move it in your $PATH and start a new server : consul agent -server -bootstrap-expect 1 -data-dir /tmp/consul -node consul-server We also need python 3.4 or 2.7, pip installed and, then, to download the single dependency we mentioned earlier with pip install requests==2.7.0. Now let's have a conversation with an API ! Sending requests APIs exposes resources for manipulation through HTTP verbs. Say we need to retrieve nodes in our cluster, Consul documentation requires us to perform a GET /v1/catalog/nodes. import requests def http_get(resource, payload=None): """ Perform an HTTP GET request against the given endpoint. """ # Avoid dangerous default function argument `{}` payload = payload or {} # versioning an API guarantees compatibility endpoint = '{}/{}/{}'.format('localhost:8500', 'v1', resource) return requests.get( endpoint, # attach parameters to the url, like `&foo=bar` params=payload, # tell the API we expect to parse JSON responses headers={'Accept': 'application/vnd.consul+json; version=1'}) Providing consul is running on the same host, we get the following result. In [4]: res = http_get('catalog/nodes') In [5]: res.json() Out[5]: [{'Address': '172.17.0.1', 'Node': 'consul-server'}] Awesome : a few lines of code gave us a really convenient access to Consul information. Let's leverage OOP to abstract further the nodes resource. Mapping resources The idea is to consider a Catalog class whose attributes are Consul API resources. A little bit of Python magic offers an elegant way to achieve that. class Catalog(object): # url specific path _path = 'catalog' def__getattr__(self, name): """ Extend built-in method to add support for attributes related to endpoints. Example: agent.members runs GET /v1/agent/members """ # Default behavior if name in self.__dict__: returnself.__dict__[name] # Dynamic attribute based on the property name else: return http_get('/'.join([self._path, name])) It might seem a little cryptic if you are not familiar with built-in Python's object methods but the usage is crystal clear : In [47]: catalog_ = Catalog() In [48]: catalog_.nodes.json() Out[48]: [{'Address': '172.17.0.1', 'Node': 'consul-server'}] The really nice benefit with this approach is that we become very productive in supporting new resources. Just rename the previous class ClientFactory and profit. class Status(ClientFactory): _path = 'status' In [58]: status_ = Status() In [59]: status_.peers.json() Out[59]: ['172.17.0.1:8300'] But... what if the resource we call does not exist ? And, although we provide a header with Accept: application/json, what if we actually don't get back a JSON object or reach our rate limit ? Reading responses Let's challenge our current implementation against those questions. In [61]: status_.not_there Out[61]: <Response [404]> In [68]: # ok, that's a consistent response In [69]: # 404 HTTP code means the resource wasn't found on server-side In [69]: status_.not_there.json() --------------------------------------------------------------------------- StopIteration Traceback (most recent call last) ... ValueError: Expecting value: line 1 column 1 (char 0) Well that's not safe at all. We're going to wrap our HTTP calls with a decorator in charge of inspecting the API response. def safe_request(fct): """ Return Go-like data (i.e. actual response and possible error) instead of raising errors. """ def inner(*args, **kwargs): data, error = {}; one try: res = fct(*args, **kwargs) except requests.exceptions.ConnectionErroras error: returnNone, {'message': str(error), 'id': -1} if res.status_code == 200 and res.headers['content-type'] == 'application/json': # expected behavior data = res.json() elif res.status_code == 206 and res.headers['content-type'] == 'application/json': # partial response, return as-is data = res.json() else: # something went wrong error = {'id': res.status_code, 'message': res.reason} return res, error return inner # update our old code @safe_request def http_get(resource): # ... This implementation stills require us to check for errors instead of disposing of the data right away. But we are dealing with network and unexpected failures will happen. Being aware of them without crashing or wrapping every resources with try/catch is a working compromise. In [71]: res, err = status_.not_there In [72]: print(err) {'id': 404, 'message': 'Not Found'} Conclusion We just covered an opinionated python abstraction for programmatically expose remote resources. Subclassing the objects above allows one to quickly interact with new services, through command line tools or interactive prompt. Yet, we only worked with the GET method. Most of the APIs allow resources deletion (DELETE), update (PUT) or creation (POST) to name a few HTTP verbs. Other future work could involve : authentication smarter HTTP code handler when dealing with forbidden, rate limiting, internal server error responses Given the incredible services that emerged lately (IBM Watson, Docker, ...), building API clients is a more and more productive option to develop innovative projects. About the Author Xavier Bruhiere is a Lead Developer at AppTurbo in Paris, where he develops innovative prototypes to support company growth. He is addicted to learning, hacking on intriguing hot techs (both soft and hard), and practicing high intensity sports.

In this article, Sunila Gollapudi, author of Practical Machine Learning, introduces the key aspects of machine learning semantics and various toolkit options in Python. Machine learning has been around for many years now and all of us, at some point in time, have been consumers of machine learning technology. One of the most common examples is facial recognition software, which can identify if a digital photograph includes a particular person. Today, Facebook users can see automatic suggestions to tag their friends in their uploaded photos. Some cameras and software such as iPhoto also have this capability. What is learning? Let's spend some time understanding what the "learning" in machine learning means. We are referring to learning from some kind of observation or data to automatically carry out further actions. An intelligent system cannot be built without using learning to get there. The following are some questions that you’ll need to answer to define your learning problem: What do you want to learn? What is the required data and where does it come from? Is the complete data available in one shot? What is the goal of learning or why should there be learning at all? Before we plunge into understanding the internals of each learning type, let's quickly understand a simple predictive analytics process for building and validating models that solve a problem with maximum accuracy: Identify whether the raw dataset is validated or cleansed and is broken into training, testing, and evaluation datasets. Pick a model that best suits and has an error function that will be minimized over the training set. Make sure this model works on the testing set. Iterate this process with other machine learning algorithms and/or attributes until there is a reasonable performance on the test set. This result can now be used to apply for new inputs and predict the output. The following diagram depicts how learning can be applied to predict behavior: Key aspects of machine learning semantics The following concept map shows the key aspects of machine learning semantics: Python Python is one of the most highly adopted programming or scripting languages in the field of machine learning and data science. Python is known for its ease of learning, implementation, and maintenance. Python is highly portable and can run on Unix, Windows, and Mac platforms. With the availability of libraries such as Pydoop and SciPy, its relevance in the world of big data analytics has tremendously increased. Some of the key reasons for the popularity of Python in solving machine learning problems are as follows: Python is well suited for data analysis It is a versatile scripting language that can be used to write some basic, quick and dirty scripts to test some basic functions or can be used in real-time applications leveraging its full-featured toolkits Python comes with mature machine learning packages and can be used in a plug-and-play manner Toolkit options in Python Before we go deeper into what toolkit options we have in Python, let's first understand what toolkit option trade-offs should be considered before choosing one: What are my performance priorities? Do I need offline or real-time processing implementations? How transparent are the toolkits? Can I customize the library myself? What is the community status? How fast are bugs fixed and how is the community support and expert communication availability? There are three options in Python: Python external bindings. These are interfaces to popular packages in the market such as Matlab, R, Octave, and so on. This option will work well if you already have existing implementations in these frameworks. Python-based toolkits. There are a number of toolkits written in Python which come with a bunch of algorithms. Write your own logic/toolkit. Python has two core toolkits that are more like building blocks. Almost all the following specialized toolkits use these core ones: NumPy: Fast and efficient arrays built in Python SciPy: A bunch of algorithms for standard operations built on NumPy There are also C/C++ based implementations such as LIBLINEAR, LIBSVM, OpenCV, and others. Some of the most popular Python toolkits are as follows: nltk: The natural language toolkit. This focuses on natural language processing (NLP). mlpy: The machine learning algorithms toolkit that comes with support for some key machine learning algorithms such as classifications, regression, and clustering, among others. PyML: This toolkit focuses on support vector machine (SVM). PyBrain: This toolkit focuses on neural network and related functions. mdp-toolkit: The focus of this toolkit is on data processing and it supports scheduling and parallelizing the processing. scikit-learn: This is one of the most popular toolkits and has been highly adopted by data scientists in the recent past. It has support for supervised and unsupervised learning and some special support for feature selection and visualizations. There is a large team that is actively building this toolkit and is known for its excellent documentation. PyDoop: Python integration with the Hadoop platform. PyDoop and SciPy are heavily deployed in big data analytics. Find out how to apply python machine learning to your working environment with our 'Practical Machine Learning' book