How-To Tutorials - Programming

 In this article written by John Torjo and Wisnu Anggoro, authors of the book Boost.Asio C++ Network Programming - Second Edition, the authors state that "Many programmers have used libraries since this simplifies the programming process. Because they do not need to write the function from scratch anymore, using a library can save much code development time". In this article, we are going to get acquainted with Boost C++ libraries. Let us prepare our own compiler and text editor to prove the power of Boost libraries. As we do so, we will discuss the following topics: Introducing the C++ standard template library Introducing Boost C++ libraries Setting up Boost C++ libraries in MinGW compiler Building Boost C++ libraries Compiling code that contains Boost C++ libraries (For more resources related to this topic, see here.) Introducing the C++ standard template library The C++ Standard Template Library (STL) is a generic template-based library that offers generic containers among other things. Instead of dealing with dynamic arrays, linked lists, binary trees, or hash tables, programmers can easily use an algorithm that is provided by STL. The STL is structured by containers, iterators, and algorithms, and their roles are as follows: Containers: Their main role is to manage the collection of objects of certain kinds, such as arrays of integers or linked lists of strings. Iterators: Their main role is to step through the element of the collections. The working of an iterator is similar to that of a pointer. We can increment the iterator by using the ++ operator and access the value by using the * operator. Algorithms: Their main role is to process the element of collections. An algorithm uses an iterator to step through all elements. After it iterates the elements, it processes each element, for example, modifying the element. It can also search and sort the element after it finishes iterating all the elements. Let us examine the three elements that structure STL by creating the following code: /* stl.cpp */ #include <vector> #include <iostream> #include <algorithm> int main(void) { int temp; std::vector<int> collection; std::cout << "Please input the collection of integer numbers, input 0 to STOP!n"; while(std::cin >> temp != 0) { if(temp == 0) break; collection.push_back(temp); } std::sort(collection.begin(), collection.end()); std::cout << "nThe sort collection of your integer numbers:n"; for(int i: collection) { std::cout << i << std::endl; } } Name the preceding code stl.cpp, and run the following command to compile it: g++ -Wall -ansi -std=c++11 stl.cpp -o stl Before we dissect this code, let us run it to see what happens. This program will ask users to enter as many as integer, and then it will sort the numbers. To stop the input and ask the program to start sorting, the user has to input 0. This means that 0 will not be included in the sorting process. Since we do not prevent users from entering non-integer numbers such as 3.14, or even the string, the program will soon stop waiting for the next number after the user enters a non-integer number. The code yields the following output: We have entered six integer: 43, 7, 568, 91, 2240, and 56. The last entry is 0 to stop the input process. Then the program starts to sort the numbers and we get the numbers sorted in sequential order: 7, 43, 56, 91, 568, and 2240. Now, let us examine our code to identify the containers, iterators, and algorithms that are contained in the STL. std::vector<int> collection; The preceding code snippet has containers from STL. There are several containers, and we use a vector in the code. A vector manages its elements in a dynamic array, and they can be accessed randomly and directly with the corresponding index. In our code, the container is prepared to hold integer numbers so we have to define the type of the value inside the angle brackets <int>. These angle brackets are also called generics in STL. collection.push_back(temp); std::sort(collection.begin(), collection.end()); The begin() and end() functions in the preceding code are algorithms in STL. They play the role of processing the data in the containers that are used to get the first and the last elements in the container. Before that, we can see the push_back() function, which is used to append an element to the container. for(int i: collection) { std::cout << i << std::endl; } The preceding for block will iterate each element of the integer which is called as collection. Each time the element is iterated, we can process the element separately. In the preceding example, we showed the number to the user. That is how the iterators in STL play their role. #include <vector> #include <algorithm> We include vector definition to define all vector functions and algorithm definition to invoke the sort() function. Introducing the Boost C++ libraries The Boost C++ libraries is a set of libraries to complement the C++ standard libraries. The set contains more than a hundred libraries that we can use to increase our productivity in C++ programming. It is also used when our requirements go beyond what is available in the STL. It provides source code under Boost Licence, which means that it allows us to use, modify, and distribute the libraries for free, even for commercial use. The development of Boost is handled by the Boost community, which consists of C++ developers from around the world. The mission of the community is to develop high-quality libraries as a complement to STL. Only proven libraries will be added to the Boost libraries. For detailed information about Boost libraries go to www.boost.org. And if you want to contribute developing libraries to Boost, you can join the developer mailing list at lists.boost.org/mailman/listinfo.cgi/boost. The entire source code of the libraries is available on the official GitHub page at github.com/boostorg. Advantages of Boost libraries As we know, using Boost libraries will increase programmer productivity. Moreover, by using Boost libraries, we will get advantages such as these: It is open source, so we can inspect the source code and modify it if needed. Its license allows us to develop both open source and close source projects. It also allows us to commercialize our software freely. It is well documented and we can find it libraries all explained along with sample code from the official site. It supports almost any modern operating system, such as Windows and Linux. It also supports many popular compilers. It is a complement to STL instead of a replacement. It means using Boost libraries will ease those programming processes which are not handled by STL yet. In fact, many parts of Boost are included in the standard C++ library. Preparing Boost libraries for MinGW compiler Before we go through to program our C++ application by using Boost libraries, the libraries need to be configured in order to be recognized by MinGW compiler. Here we are going to prepare our programming environment so that our compiler is able use Boost libraries. Downloading Boost libraries The best source from which to download Boost is the official download page. We can go there by pointing our internet browser to www.boost.org/users/download. Find the Download link in Current Release section. At the time of writing, the current version of Boost libraries is 1.58.0, but when you read this article, the version may have changed. If so, you can still choose the current release because the higher version must be compatible with the lower. However, you have to adjust as we're goning to talk about the setting later. Otherwise, choosing the same version will make it easy for you to follow all the instructions in this article. There are four file formats to be choose from for download; they are .zip, .tar.gz, .tar.bz2, and .7z. There is no difference among the four files but their file size. The largest file size is of the ZIP format and the lowest is that of the 7Z format. Because of the file size, Boost recommends that we download the 7Z format. See the following image for comparison: We can see, from the preceding image, the size of ZIP version is 123.1 MB while the size of the 7Z version is 65.2 MB. It means that the size of the ZIP version is almost twice that of the 7Z version. Therefore they suggest that you choose the 7Z format to reduce download and decompression time. Let us choose boost_1_58_0.7z to be downloaded and save it to our local storage. Deploying Boost libraries After we have got boost_1_58_0.7z in our local storage, decompress it using the 7ZIP application and save the decompression files to C:boost_1_58_0. The 7ZIP application can be grabbed from www.7-zip.org/download.html. The directory then should contain file structures as follows: Instead of browsing to the Boost download page and searching for the Boost version manually, we can go directly to sourceforge.net/projects/boost/files/boost/1.58.0. It will be useful when the 1.58.0 version is not the current release anymore. Using Boost libraries Most libraries in Boost are header-only; this means that all declarations and definitions of functions, including namespaces and macros, are visible to the compiler and there is no need to compile them separately. We can now try to use Boost with the program to convert the string into int value as follows: /* lexical.cpp */ #include <boost/lexical_cast.hpp> #include <string> #include <iostream> int main(void) { try { std::string str; std::cout << "Please input first number: "; std::cin >> str; int n1 = boost::lexical_cast<int>(str); std::cout << "Please input second number: "; std::cin >> str; int n2 = boost::lexical_cast<int>(str); std::cout << "The sum of the two numbers is "; std::cout << n1 + n2 << "n"; return 0; } catch (const boost::bad_lexical_cast &e) { std::cerr << e.what() << "n"; return 1; } } Open the Notepad++ application, type the preceding code, and save it as lexical.cpp in C:CPP. Now open the command prompt, point the active directory to C:CPP, and then type the following command: g++ -Wall -ansi lexical.cpp –Ic:boost_1_58_0 -o lexical We have a new option here, which is –I (the "include" option). This option is used along with the full path of the directory to inform the compiler that we have another header directory that we want to include to our code. Since we store our Boost libraries in c: boost_1_58_0, we can use –Ic:boost_1_58_0 as an additional parameter. In lexical.cpp, we apply boost::lexical_cast to convert string type data into int type data. The program will ask the user to input two numbers and will then automatically find the sum of both numbers. If a user inputs an inappropriate number, it will inform that an error has occurred. The Boost.LexicalCast library is provided by Boost for casting data type purpose (converting numeric types such as int, double, or floats into string types, and vice versa). Now let us dissect lexical.cpp to for a more detailed understanding of what it does: #include <boost/lexical_cast.hpp> #include <string> #include <iostream> We include boost/lexical_cast.hpp because the boost::lexical_cast function is declared lexical_cast.hpp header file whilst string header is included to apply std::string function and iostream header is included to apply std::cin, std::cout and std::cerr function. Other functions, such as std::cin and std::cout, and we saw what their functions are so we can skip those lines. #include <boost/lexical_cast.hpp> #include <string> #include <iostream> We used the preceding two separate lines to convert the user-provided input string into the int data type. Then, after converting the data type, we summed up both of the int values. We can also see the try-catch block in the preceding code. It is used to catch the error if user inputs an inappropriate number, except 0 to 9. catch (const boost::bad_lexical_cast &e) { std::cerr << e.what() << "n"; return 1; } The preceding code snippet will catch errors and inform the user what the error message exactly is by using boost::bad_lexical_cast. We call the e.what() function to obtain the string of the error message. Now let us run the application by typing lexical at the command prompt. We will get output like the following: I put 10 for first input and 20 for the second input. The result is 30 because it just sums up both input. But what will happen if I put in a non-numerical value, for instance Packt. Here is the output to try that condition: Once the application found the error, it will ignore the next statement and go directly to the catch block. By using the e.what() function, the application can get the error message and show it to the user. In our example, we obtain bad lexical cast: source type value could not be interpreted as target as the error message because we try to assign the string data to int type variable. Building Boost libraries As we discussed previously, most libraries in Boost are header-only, but not all of them. There are some libraries that have to be built separately. They are: Boost.Chrono: This is used to show the variety of clocks, such as current time, the range between two times, or calculating the time passed in the process. Boost.Context: This is used to create higher-level abstractions, such as coroutines and cooperative threads. Boost.Filesystem: This is used to deal with files and directories, such as obtaining the file path or checking whether a file or directory exists. Boost.GraphParallel: This is an extension to the Boost Graph Library (BGL) for parallel and distributed computing. Boost.IOStreams: This is used to write and read data using stream. For instance, it loads the content of a file to memory or writes compressed data in GZIP format. Boost.Locale: This is used to localize the application, in other words, translate the application interface to user's language. Boost.MPI: This is used to develop a program that executes tasks concurrently. MPI itself stands for Message Passing Interface. Boost.ProgramOptions: This is used to parse command-line options. Instead of using the argv variable in the main parameter, it uses double minus (--) to separate each command-line option. Boost.Python: This is used to parse Python language in C++ code. Boost.Regex: This is used to apply regular expression in our code. But if our development supports C++11, we do not depend on the Boost.Regex library anymore since it is available in the regex header file. Boost.Serialization: This is used to convert objects into a series of bytes that can be saved and then restored again into the same object. Boost.Signals: This is used to create signals. The signal will trigger an event to run a function on it. Boost.System: This is used to define errors. It contains four classes: system::error_code, system::error_category, system::error_condition, and system::system_error. All of these classes are inside the boost namespace. It is also supported in the C++11 environment, but because many Boost libraries use Boost.System, it is necessary to keep including Boost.System. Boost.Thread: This is used to apply threading programming. It provides classes to synchronize access on multiple-thread data. It is also supported in C++11 environments, but it offers extensions, such as we can interrupt thread in Boost.Thread. Boost.Timer: This is used to measure the code performance by using clocks. It measures time passed based on usual clock and CPU time, which states how much time has been spent to execute the code. Boost.Wave: This provides a reusable C preprocessor that we can use in our C++ code. There are also a few libraries that have optional, separately compiled binaries. They are as follows: Boost.DateTime: It is used to process time data; for instance, calendar dates and time. It has a binary component that is only needed if we use to_string, from_string, or serialization features. It is also needed if we target our application in Visual C++ 6.x or Borland. Boost.Graph: It is used to create two-dimensional graphics. It has a binary component that is only needed if we intend to parse GraphViz files. Boost.Math: It is used to deal with mathematical formulas. It has binary components for cmath functions. Boost.Random: It is used to generate random numbers. It has a binary component which is only needed if we want to use random_device. Boost.Test: It is used to write and organize test programs and their runtime execution. It can be used in header-only or separately compiled mode, but separate compilation is recommended for serious use. Boost.Exception: It is used to add data to an exception after it has been thrown. It provides non-intrusive implementation of exception_ptr for 32-bit _MSC_VER==1310 and _MSC_VER==1400, which requires a separately compiled binary. This is enabled by #define BOOST_ENABLE_NON_INTRUSIVE_EXCEPTION_PTR. Let us try to recreate the random number generator. But now we will use the Boost.Random library instead of std::rand() from the C++ standard function. Let us take a look at the following code: /* rangen_boost.cpp */ #include <boost/random/mersenne_twister.hpp> #include <boost/random/uniform_int_distribution.hpp> #include <iostream> int main(void) { int guessNumber; std::cout << "Select number among 0 to 10: "; std::cin >> guessNumber; if(guessNumber < 0 || guessNumber > 10) { return 1; } boost::random::mt19937 rng; boost::random::uniform_int_distribution<> ten(0,10); int randomNumber = ten(rng); if(guessNumber == randomNumber) { std::cout << "Congratulation, " << guessNumber << " is your lucky number.n"; } else { std::cout << "Sorry, I'm thinking about number " << randomNumber << "n"; } return 0; } We can compile the preceding source code by using the following command: g++ -Wall -ansi -Ic:/boost_1_58_0 rangen_boost.cpp -o rangen_boost Now, let us run the program. Unfortunately, for the three times that I ran the program, I always obtained the same random number as follows: As we can see from this example, we always get number 8. This is because we apply Mersenne Twister, a Pseudorandom Number Generator (PRNG), which uses the default seed as a source of randomness so it will generate the same number every time the program is run. And of course it is not the program that we expect. Now, we will rework the program once again, just in two lines. First, find the following line: #include <boost/random/mersenne_twister.hpp> Change it as follows: #include <boost/random/random_device.hpp> Next, find the following line: boost::random::mt19937 rng; Change it as follows: boost::random::random_device rng; Then, save the file as rangen2_boost.cpp and compile the rangen2_boost.cpp file by using the command like we compiled rangen_boost.cpp. The command will look like this: g++ -Wall -ansi -Ic:/boost_1_58_0 rangen2_boost.cpp -o rangen2_boost Sadly, there will be something wrong and the compiler will show the following error message: cc8KWVvX.o:rangen2_boost.cpp:(.text$_ZN5boost6random6detail20generate _uniform_intINS0_13random_deviceEjEET0_RT_S4_S4_N4mpl_5bool_ILb1EEE[_ ZN5boost6random6detail20generate_uniform_intINS0_13random_deviceEjEET 0_RT_S4_S4_N4mpl_5bool_ILb1EEE]+0x24f): more undefined references to boost::random::random_device::operator()()' follow collect2.exe: error: ld returned 1 exit status This is because, as we have discussed earlier, the Boost.Random library needs to be compiled separately if we want to use the random_device attribute. Boost libraries have a system to compile or build Boost itself, called Boost.Build library. There are two steps we have to achieve to install the Boost.Build library. First, run Bootstrap by pointing the active directory at the command prompt to C:boost_1_58_0 and typing the following command: bootstrap.bat mingw We use our MinGW compiler, as our toolset in compiling the Boost library. Wait a second and then we will get the following output if the process is a success: Building Boost.Build engine Bootstrapping is done. To build, run: .b2 To adjust configuration, edit 'project-config.jam'. Further information: - Command line help: .b2 --help - Getting started guide: http://boost.org/more/getting_started/windows.html - Boost.Build documentation: http://www.boost.org/build/doc/html/index.html In this step, we will find four new files in the Boost library's root directory. They are: b2.exe: This is an executable file to build Boost libraries. bjam.exe: This is exactly the same as b2.exe but it is a legacy version. bootstrap.log: This contains logs from the bootstrap process project-config.jam: This contains a setting that will be used in the building process when we run b2.exe. We also find that this step creates a new directory in C:boost_1_58_0toolsbuildsrcenginebin.ntx86 , which contains a bunch of .obj files associated with Boost libraries that needed to be compiled. After that, run the second step by typing the following command at the command prompt: b2 install toolset=gcc Grab yourself a cup of coffee after running that command because it will take about twenty to fifty minutes to finish the process, depending on your system specifications. The last output we will get will be like this: ...updated 12562 targets... This means that the process is complete and we have now built the Boost libraries. If we check in our explorer, the Boost.Build library adds C:boost_1_58_0stagelib, which contains a collection of static and dynamic libraries that we can use directly in our program. bootstrap.bat and b2.exe use msvc (Microsoft Visual C++ compiler) as the default toolset, and many Windows developers already have msvc installed on their machines. Since we have installed GCC compiler, we set the mingw and gcc toolset options in Boost's build. If you also have mvsc installed and want to use it in Boost's build, the toolset options can be omitted. Now, let us try to compile the rangen2_boost.cpp file again, but now with the following command: c:CPP>g++ -Wall -ansi -Ic:/boost_1_58_0 rangen2_boost.cpp - Lc:boost_1_58_0stagelib -lboost_random-mgw49-mt-1_58 - lboost_system-mgw49-mt-1_58 -o rangen2_boost We have two new options here, they are –L and –l. The -L option is used to define the path that contains the library file if it is not in the active directory. The –l option is used to define the name of library file but omitting the first lib word in front of the file name. In this case, the original library file name is libboost_random-mgw49-mt-1_58.a, and we omit the lib phrase and the file extension for option -l. The new file called rangen2_boost.exe will be created in C:CPP. But before we can run the program, we have to ensure that the directory which the program installed has contained the two dependencies library file. These are libboost_random-mgw49-mt-1_58.dll and libboost_system-mgw49-mt-1_58.dll, and we can get them from the library directory c:boost_1_58_0_1stagelib. Just to make it easy for us to run that program, run the following copy command to copy the two library files to C:CPP: copy c:boost_1_58_0_1stageliblibboost_random-mgw49-mt-1_58.dll c:cpp copy c:boost_1_58_0_1stageliblibboost_system-mgw49-mt-1_58.dll c:cpp And now the program should run smoothly. In order to create a network application, we are going to use the Boost.Asio library. We do not find Boost.Asio—the library we are going to use to create a network application—in the non-header-only library. It seems that we do not need to build the boost library since Boost.Asio is header-only library. This is true, but since Boost.Asio depends on Boost.System and Boost.System needs to be built before being used, it is important to build Boost first before we can use it to create our network application. For option –I and –L, the compiler does not care if we use backslash () or slash (/) to separate each directory name in the path because the compiler can handle both Windows and Unix path styles. Summary We saw that Boost C++ libraries were developed to complement the standard C++ library We have also been able to set up our MinGW compiler in order to compile the code which contains Boost libraries and build the binaries of libraries which have to be compiled separately. Please remember that though we can use the Boost.Asio library as a header-only library, it is better to build all Boost libraries by using the Boost.Build library. It will be easy for us to use all libraries without worrying about compiling failure. Resources for Article:   Further resources on this subject: Actors and Pawns[article] What is Quantitative Finance?[article] Program structure, execution flow, and runtime objects [article]

In this article by David Salter, the author of Mastering NetBeans, we'll cover the following topics: Running applications Debugging applications Profiling applications Testing applications On a day-to-day basis, developers spend much of their time writing and running applications. While writing applications, they typically debug, test, and profile them to ensure that they provide the best possible application to customers. Running, debugging, profiling, and testing are all integral parts of the development life cycle, and NetBeans provides excellent tooling to help us in all these areas. (For more resources related to this topic, see here.) Running applications Executing applications from within NetBeans is as simple as either pressing the F6 button on the keyboard or selecting the Run menu item or Project Context menu item. Choosing either of these options will launch your application without specifying any additional Java command-line parameters using the default platform JDK that NetBeans is currently using. Sometimes we want to change the options that are used for launching applications. NetBeans allows these options to be easily specified by a project's properties. Right-clicking on a project in the Projects window and selecting the Properties menu option opens the Project Properties dialog. Selecting the Run category allows the configuration options to be defined for launching an application. From this dialog, we can define and select multiple run configurations for the project via the Configuration dropdown. Selecting the New… button to the right of the Configuration dropdown allows us to enter a name for a new configuration. Once a new configuration is created, it is automatically selected as the active configuration. The Delete button can be used for removing any unwanted configurations. The preceding screenshot shows the Project Properties dialog for a standard Java project. Different project types (for example, web or mobile projects) have different options in the Project Properties window. As can be seen from the preceding Project Properties dialog, several pieces of information can be defined for a standard Java project, which together make up the launch configuration for a project: Runtime Platform: This option allows us to define which Java platform we will use when launching the application. From here, we can select from all the Java platforms that are configured within NetBeans. Selecting the Manage Platforms… button opens the Java Platform Manager dialog, allowing full configuration of the different Java platforms available (both Java Standard Edition and Remote Java Standard Edition). Selecting this button has the same effect as selecting the Tools and then Java Platforms menu options. Main Class: This option defines the main class that is used to launch the application. If the project has more than one main class, selecting the Browse… button will cause the Browse Main Classes dialog to be displayed, listing all the main classes defined in the project. Arguments: Different command-line arguments can be passed to the main class as defined in this option. Working Directory: This option allows the working directory for the application to be specified. VM Options: If different VM options (such as heap size) require setting, they can be specified by this option. Selecting the Customize button displays a dialog listing the different standard VM options available which can be selected (ticked) as required. Custom VM properties can also be defined in the dialog. For more information on the different VM properties for Java, check out http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html. From here, the VM properties for Java 7 (and earlier versions) and Java 8 for Windows, Solaris, Linux, and Mac OS X can be referenced. Run with Java Web Start: Selecting this option allows the application to be executed using Java Web Start technologies. This option is only available if Web Start is enabled in the Application | Web Start category. When running a web application, the project properties are different from those of a standalone Java application. In fact, the project properties for a Maven web application are different from those of a standard NetBeans web application. The following screenshot shows the properties for a Maven-based web application; as discussed previously, Maven is the standard project management tool for Java applications, and the recommended tool for creating and managing web applications: Debugging applications In the previous section, we saw how NetBeans provides the easy-to-use features to allow developers to launch their applications, but then it also provides more powerful additional features. The same is true for debugging applications. For simple debugging, NetBeans provides the standard facilities you would expect, such as stepping into or over methods, setting line breakpoints, and monitoring the values of variables. When debugging applications, NetBeans provides several different windows, enabling different types of information to be displayed and manipulated by the developer: Breakpoints Variables Call stack Loaded classes Sessions Threads Sources Debugging Analyze stack All of these windows are accessible from the Window and then Debugging main menu within NetBeans. Breakpoints NetBeans provides a simple approach to set breakpoints and a more comprehensive approach that provides many more useful features. Breakpoints can be easily added into Java source code by clicking on the gutter on the left-hand side of a line of Java source code. When a breakpoint is set, a small pink square is shown in the gutter and the entire line of source code is also highlighted in the same color. Clicking on the breakpoint square in the gutter toggles the breakpoint on and off. Once a breakpoint has been created, instead of removing it altogether, it can be disabled by right-clicking on the bookmark in the gutter and selecting the Breakpoint and then Enabled menu options. This has the effect of keeping the breakpoint within your codebase, but execution of the application does not stop when the breakpoint is hit. Creating a simple breakpoint like this can be a very powerful way of debugging applications. It allows you to stop the execution of an application when a line of code is hit. If we want to add a bit more control onto a simple breakpoint, we can edit the breakpoint's properties by right-clicking on the breakpoint in the gutter and selecting the Breakpoint and then Properties menu options. This causes the Breakpoint Properties dialog to be displayed: In this dialog, we can see the line number and the file that the breakpoint belongs to. The line number can be edited to move the breakpoint if it has been created on the wrong line. However, what's more interesting is the conditions that we can apply to the breakpoint. The Condition entry allows us to define a condition that has to be met for the breakpoint to stop the code execution. For example, we can stop the code when the variable i is equal to 20 by adding a condition, i==20. When we add conditions to a breakpoint, the breakpoint becomes known as a conditional breakpoint, and the icon in the gutter changes to a square with the lower-right quadrant removed. We can also cause the execution of the application to halt at a breakpoint when the breakpoint has been hit a certain number of times. The Break when hit count is condition can be set to Equal to, Greater than, or Multiple of to halt the execution of the application when the breakpoint has been hit the requisite number of times. Finally, we can specify what actions occur when a breakpoint is hit. The Suspend dropdown allows us to define what threads are suspended when a breakpoint is hit. NetBeans can suspend All threads, Breakpoint thread, or no threads at all. The text that is displayed in the Output window can be defined via the Print Text edit box and different breakpoint groups can be enabled or disabled via the Enable Group and Disable Group drop-down boxes. But what exactly is a breakpoint group? Simply put, a breakpoint group is a collection of breakpoints that can all be set or unset at the same time. It is a way of categorizing breakpoints into similar collections, for example, all the breakpoints in a particular file, or all the breakpoints relating to exceptions or unit tests. Breakpoint groups are created in the Breakpoints window. This is accessible by selecting the Debugging and then Breakpoints menu options from within the main NetBeans Window menu. To create a new breakpoint group, simply right-click on an existing breakpoint in the Breakpoints window and select the Move Into Group… and then New… menu options. The Set the Name of Breakpoints Group dialog is displayed in which the name of the new breakpoint group can be entered. After creating a breakpoint group and assigning one or more breakpoints into it, the entire group of breakpoints can be enabled or disabled, or even deleted by right-clicking on the group in the Breakpoints window and selecting the appropriate option. Any newly created breakpoint groups will also be available in the Breakpoint Properties window. So far, we've seen how to create breakpoints that stop on a single line of code, and also how to create conditional breakpoints so that we can cause an application to stop when certain conditions occur for a breakpoint. These are excellent techniques to help debug applications. NetBeans, however, also provides the ability to create more advanced breakpoints so that we can get even more control of when the execution of applications is halted by breakpoints. So, how do we create these breakpoints? These different types of breakpoints are all created from in the Breakpoints window by right-clicking and selecting the New Breakpoint… menu option. In the New Breakpoint dialog, we can create different types of breakpoints by selecting the appropriate entry from the Breakpoint Type drop-down list. The preceding screenshot shows an example of creating a Class breakpoint. The following types of breakpoints can be created: Class: This creates a breakpoint that halts execution when a class is loaded, unloaded, or either event occurs. Exception: This stops execution when the specified exception is caught, uncaught, or either event occurs. Field: This creates a breakpoint that halts execution when a field on a class is accessed, modified, or either event occurs. Line: This stops execution when the specified line of code is executed. It acts the same way as creating a breakpoint by clicking on the gutter of the Java source code editor window. Method: This creates a breakpoint that halts execution when a method is entered, exited, or when either event occurs. Optionally, the breakpoint can be created for all methods inside a specified class rather than a single method. Thread: This creates a breakpoint that stops execution when a thread is started, finished, or either event occurs. AWT/Swing Component: This creates a breakpoint that stops execution when a GUI component is accessed. For each of these different types of breakpoints, conditions and actions can be specified in the same way as on simple line-based breakpoints. The Variables debug window The Variables debug window lists all the variables that are currently within  the scope of execution of the application. This is therefore thread-specific, so if multiple threads are running at one time, the Variables window will only display variables in scope for the currently selected thread. In the Variables window, we can see the variables currently in scope for the selected thread, their type, and value. To display variables for a different thread to that currently selected, we must select an alternative thread via the Debugging window. Using the triangle button to the left of each variable, we can expand variables and drill down into the properties within them. When a variable is a simple primitive (for example, integers or strings), we can modify it or any property within it by altering the value in the Value column in the Variables window. The variable's value will then be changed within the running application to the newly entered value. By default, the Variables window shows three columns (Name, Type, and Value). We can modify which columns are visible by pressing the selection icon () at the top-right of the window. Selecting this displays the Change Visible Columns dialog, from which we can select from the Name, String value, Type, and Value columns: The Watches window The Watches window allows us to see the contents of variables and expressions during a debugging session, as can be seen in the following screenshot: In this screenshot, we can see that the variable i is being displayed along with the expressions 10+10 and i+20. New expressions can be watched by clicking on the <Enter new watch> option or by right-clicking on the Java source code editor and selecting the New Watch… menu option. Evaluating expressions In addition to watching variables in a debugging session, NetBeans also provides the facility to evaluate expressions. Expressions can contain any Java code that is valid for the running scope of the application. So, for example, local variables, class variables, or new instances of classes can be evaluated. To evaluate variables, open the Evaluate Expression window by selecting the Debug and then Evaluate Expression menu options. Enter an expression to be evaluated in this window and press the Evaluate Code Fragment button at the bottom-right corner of the window. As a shortcut, pressing the Ctrl + Enter keys will also evaluate the code fragment. Once an expression has been evaluated, it is shown in the Evaluation Result window. The Evaluation Result window shows a history of each expression that has previously been evaluated. Expressions can be added to the list of watched variables by right-clicking on the expression and selecting the Create Fixed Watch expression. The Call Stack window The Call Stack window displays the call stack for the currently executing thread: The call stack is displayed from top to bottom with the currently executing frame at the top of the list. Double-clicking on any entry in the call stack opens up the corresponding source code in the Java editor within NetBeans. Right-clicking on an entry in the call stack displays a pop-up menu with the choice to: Make Current: This makes the selected thread the current thread Pop To Here: This pops the execution of the call stack to the selected location Go To Source: This displays the selected code within the Java source editor Copy Stack: This copies the stack trace to the clipboard for use elsewhere When debugging, it can be useful to change the stack frame of the currently executing thread by selecting the Pop To Here option from within the stack trace window. Imagine the following code: // Get some magic int magic = getSomeMagicNumber(); // Perform calculation performCalculation(magic); During a debugging session, if after stepping over the getSomeMagicNumber() method, we decided that the method has not worked as expected, our course of action would probably be to debug into the getSomeMagicNumber() method. But, we've just stepped over the method, so what can we do? Well, we can stop the debugging session and start again or repeat the operation that called this section of code and hope there are no changes to the application state that affect the method we want to debug. A better solution, however, would be to select the line of code that calls the getSomeMagicNumber() method and pop the stack frame using the Pop To Here option. This would have the effect of rewinding the code execution so that we can then step into the method and see what is happening inside it. As well as using the Pop To Here functionality, NetBeans also offers several menu options for manipulating the stack frame, namely: Make Callee Current: This makes the callee of the current method the currently executing stack frame Make Caller Current: This makes the caller of the current method the currently executing stack frame Pop Topmost Call: This pops one stack frame, making the calling method the currently executing stack frame When moving around the call stack using these techniques, any operations performed by the currently executing method are not undone. So, for example, strange results may be seen if global or class-based variables are altered within a method and then an entry is popped from the call stack. Popping entries in the call stack is safest when no state changes are made within a method. The call stack displayed in the Debugging window for each thread behaves in the same way as in the Call Stack window itself. The Loaded Classes window The Loaded Classes window displays a list of all the classes that are currently loaded, showing how many instances there are of each class as a number and as a percentage of the total number of classes loaded. Depending upon the number of external libraries (including the standard Java runtime libraries) being used, you may find it difficult to locate instances of your own classes in this window. Fortunately, the filter at the bottom of the window allows the list of classes to be filtered, based upon an entered string. So, for example, entering the filter String will show all the classes with String in the fully qualified class name that are currently loaded, including java.lang.String and java.lang.StringBuffer. Since the filter works on the fully qualified name of a class, entering a package name will show all the classes listed in that package and subpackages. So, for example, entering a filter value as com.davidsalter.multithread would show only the classes listed in that package and subpackages. The Sessions window Within NetBeans, it is possible to perform multiple debugging sessions where either one project is being debugged multiple times, or more commonly, multiple projects are being debugged at the same time, where one is acting as a client application and the other is acting as a server application. The Sessions window displays a list of the currently running debug sessions, allowing the developer control over which one is the current session. Right-clicking on any of the sessions listed in the window provides the following options: Make Current: This makes the selected session the currently active debugging session Scope: This debugs the current thread or all the threads in the selected session Language: This options shows the language of the application being debugged—Java Finish: This finishes the selected debugging session Finish All: This finishes all the debugging sessions The Sessions window shows the name of the debug session (for example the main class being executed), its state (whether the application is Stopped or Running) and language being debugged. Clicking the selection icon () at the top-right of the window allows the user to choose which columns are displayed in the window. The default choice is to display all columns except for the Host Name column, which displays the name of the computer the session is running on. The Threads window The Threads window displays a hierarchical list of threads in use by the application currently being debugged. The current thread is displayed in bold. Double-clicking on any of the threads in the hierarchy makes the thread current. Similar to the Debugging window, threads can be made current, suspended, or interrupted by right-clicking on the thread and selecting the appropriate option. The default display for the Threads window is to show the thread's name and its state (Running, Waiting, or Sleeping). Clicking the selection icon () at the top-right of the window allows the user to choose which columns are displayed in the window. The Sources window The Sources window simply lists all of the source roots that NetBeans considers for the selected project. These are the only locations that NetBeans will search when looking for source code while debugging an application. If you find that you are debugging an application, and you cannot step into code, the most likely scenario is that the source root for the code you wish to debug is not included in the Sources window. To add a new source root, right-click in the Sources window and select the Add Source Root option. The Debugging window The Debugging window allows us to see which threads are running while debugging our application. This window is, therefore, particularly useful when debugging multithreaded applications. In this window, we can see the different threads that are running within our application. For each thread, we can see the name of the thread and the call stack leading to the breakpoint. The current thread is highlighted with a green band along the left-hand side edge of the window. Other threads created within our application are denoted with a yellow band along the left-hand side edge of the window. System threads are denoted with a gray band. We can make any of the threads the current thread by right-clicking on it and selecting the Make Current menu option. When we do this, the Variables and Call Stack windows are updated to show new information for the selected thread. The current thread can also be selected by clicking on the Debug and then Set Current Thread… menu options. Upon selecting this, a list of running threads is shown from which the current thread can be selected. Right-clicking on a thread and selecting the Resume option will cause the selected thread to continue execution until it hits another breakpoint. For each thread that is running, we can also Suspend, Interrupt, and Resume the thread by right-clicking on the thread and choosing the appropriate action. In each thread listing, the current methods call stack is displayed for each thread. This can be manipulated in the same way as from the Call Stack window. When debugging multithreaded applications, new breakpoints can be hit within different threads at any time. NetBeans helps us with multithreaded debugging by not automatically switching the user interface to a different thread when a breakpoint is hit on the non-current thread. When a breakpoint is hit on any thread other than the current thread, an indication is displayed at the bottom of the Debugging window, stating New Breakpoint Hit (an example of this can be seen in the previous window). Clicking on the icon to the right of the message shows all the breakpoints that have been hit together with the thread name in which they occur. Selecting the alternate thread will cause the relevant breakpoint to be opened within NetBeans and highlighted in the appropriate Java source code file. NetBeans provides several filters on the Debugging window so that we can show more/less information as appropriate. From left to right, these images allow us to: Show less (suspended and current threads only) Show thread groups Show suspend/resume table Show system threads Show monitors Show qualified names Sort by suspended/resumed state Sort by name Sort by default Debugging multithreaded applications can be a lot easier if you give your threads names. The thread's name is displayed in the Debugging window, and it's a lot easier to understand what a thread with a proper name is doing as opposed to a thread called Thread-1. Deadlock detection When debugging multithreaded applications, one of the problems that we can see is that a deadlock occurs between executing threads. A deadlock occurs when two or more threads become blocked forever because they are both waiting for a shared resource to become available. In Java, this typically occurs when the synchronized keyword is used. NetBeans allows us to easily check for deadlocks using the Check for Deadlock tool available on the Debug menu. When a deadlock is detected using this tool, the state of the deadlocked threads is set to On Monitor in the Threads window. Additionally, the threads are marked as deadlocked in the Debugging window. Each deadlocked thread is displayed with a red band on the left-hand side border and the Deadlock detected warning message is displayed. Analyze Stack Window When running an application within NetBeans, if an exception is thrown and not caught, the stack trace will be displayed in the Output window, allowing the developer to see exactly where errors have occurred. From the following screenshot, we can easily see that a NullPointerException was thrown from within the FaultyImplementation class in the doUntestedOperation() method at line 16. Looking before this in the stack trace (that is the entry underneath), we can see that the doUntestedOperation() method was called from within the main() method of the Main class at line 21: In the preceding example, the FaultyImplementation class is defined as follows: public class FaultyImplementation { public void doUntestedOperation() { throw new NullPointerException(); } } Java is providing an invaluable feature to developers, allowing us to easily see where exceptions are thrown and what the sequence of events was that led to the exception being thrown. NetBeans, however, enhances the display of the stack traces by making the class and line numbers clickable hyperlinks which, when clicked on, will navigate to the appropriate line in the code. This allows us to easily delve into a stack trace and view the code at all the levels of the stack trace. In the previous screenshot, we can click on the hyperlinks FaultyImplementation.java:16 and Main.java:21 to take us to the appropriate line in the appropriate Java file. This is an excellent time-saving feature when developing applications, but what do we do when someone e-mails us a stack trace to look at an error in a production system? How do we manage stack traces that are stored in log files? Fortunately, NetBeans provides an easy way to allow a stack trace to be turned into clickable hyperlinks so that we can browse through the stack trace without running the application. To load and manage stack traces into NetBeans, the first step is to copy the stack trace onto the system clipboard. Once the stack trace has been copied onto the clipboard, Analyze Stack Window can be opened within NetBeans by selecting the Window and then Debugging and then Analyze Stack menu options (the default installation for NetBeans has no keyboard shortcut assigned to this operation). Analyze Stack Window will default to showing the stack trace that is currently in the system clipboard. If no stack trace is in the clipboard, or any other data is in the system's clipboard, Analyze Stack Window will be displayed with no contents. To populate the window, copy a stack trace into the system's clipboard and select the Insert StackTrace From Clipboard button. Once a stack trace has been displayed in Analyze Stack Window, clicking on the hyperlinks in it will navigate to the appropriate location in the Java source files just as it does from the Output window when an exception is thrown from a running application. You can only navigate to source code from a stack trace if the project containing the relevant source code is open in the selected project group. Variable formatters When debugging an application, the NetBeans debugger can display the values of simple primitives in the Variables window. As we saw previously, we can also display the toString() representation of a variable if we select the appropriate columns to display in the window. Sometimes when debugging, however, the toString() representation is not the best way to display formatted information in the Variables window. In this window, we are showing the value of a complex number class that we have used in high school math. The ComplexNumber class being debugged in this example is defined as: public class ComplexNumber { private double realPart; private double imaginaryPart; public ComplexNumber(double realPart, double imaginaryPart) { this.realPart = realPart; this.imaginaryPart = imaginaryPart; } @Override public String toString() { return "ComplexNumber{" + "realPart=" + realPart + ", imaginaryPart=" + imaginaryPart + '}'; } // Getters and Setters omitted for brevity… } Looking at this class, we can see that it essentially holds two members—realPart and imaginaryPart. The toString() method outputs a string, detailing the name of the object and its parameters which would be very useful when writing ComplexNumbers to log files, for example: ComplexNumber{realPart=1.0, imaginaryPart=2.0} When debugging, however, this is a fairly complicated string to look at and comprehend—particularly, when there is a lot of debugging information being displayed. NetBeans, however, allows us to define custom formatters for classes that detail how an object will be displayed in the Variables window when being debugged. To define a custom formatter, select the Java option from the NetBeans Options dialog and then select the Java Debugger tab. From this tab, select the Variable Formatters category. On this screen, all the variable formatters that are defined within NetBeans are shown. To create a new variable formatter, select the Add… button to display the Add Variable Formatter dialog. In the Add Variable Formatter dialog, we need to enter Formatter Name and a list of Class types that NetBeans will apply the formatting to when displaying values in the debugger. To apply the formatter to multiple classes, enter the different classes, separated by commas. The value that is to be formatted is entered in the Value formatted as a result of code snippet field. This field takes the scope of the object being debugged. So, for example, to output the ComplexNumber class, we can enter the custom formatter as: "("+realPart+", "+imaginaryPart+"i)" We can see that the formatter is built up from concatenating static strings and the values of the members realPart and imaginaryPart. We can see the results of debugging variables using custom formatters in the following screenshot: Debugging remote applications The NetBeans debugger provides rapid access for debugging local applications that are executing within the same JVM as NetBeans. What happens though when we want to debug a remote application? A remote application isn't necessarily hosted on a separate server to your development machine, but is defined as any application running outside of the local JVM (that is the one that is running NetBeans). To debug a remote application, the NetBeans debugger can be "attached" to the remote application. Then, to all intents, the application can be debugged in exactly the same way as a local application is debugged, as described in the previous sections of this article. To attach to a remote application, select the Debug and then Attach Debugger… menu options. On the Attach dialog, the connector (SocketAttach, ProcessAttach, or SocketListen) must be specified to connect to the remote application. The appropriate connection details must then be entered to attach the debugger. For example, the process ID must be entered for the ProcessAttach connector and the host and port must be specified for the SocketAttach connector. Profiling applications Learning how to debug applications is an essential technique in software development. Another essential technique that is often overlooked is profiling applications. Profiling applications involves measuring various metrics such as the amount of heap memory used or the number of loaded classes or running threads. By profiling applications, we can gain an understanding of what our applications are actually doing and as such we can optimize them and make them function better. NetBeans provides first class profiling tools that are easy to use and provide results that are easy to interpret. The NetBeans profiler allows us to profile three specific areas: Application monitoring Performance monitoring Memory monitoring Each of these monitoring tools is accessible from the Profile menu within NetBeans. To commence profiling, select the Profile and then Profile Project menu options. After instructing NetBeans to profile a project, the profiler starts providing the choice of the type of profiling to perform. Testing applications Writing tests for applications is probably one of the most important aspects of modern software development. NetBeans provides the facility to write and run both JUnit and TestNG tests and test suites. In this section, we'll provide details on how NetBeans allows us to write and run these types of tests, but we'll assume that you have some knowledge of either JUnit or TestNG. TestNG support is provided by default with NetBeans, however, due to license concerns, JUnit may not have been installed when you installed NetBeans. If JUnit support is not installed, it can easily be added through the NetBeans Plugins system. In a project, NetBeans creates two separate source roots: one for application sources and the other for test sources. This allows us to keep tests separate from application source code so that when we ship applications, we do not need to ship tests with them. This separation of application source code and test source code enables us to write better tests and have less coupling between tests and applications. The best situation is for the test source root to have a dependency on application classes and the application classes to have no dependency on the tests that we have written. To write a test, we must first have a project. Any type of Java project can have tests added into it. To add tests into a project, we can use the New File wizard. In the Unit Tests category, there are templates for: JUnit Tests Tests for Existing Class (this is for JUnit tests) Test Suite (this is for JUnit tests) TestNG Test Case TestNG Test Suite When creating classes for these types of tests, NetBeans provides the option to automatically generate code; this is usually a good starting point for writing classes. When executing tests, NetBeans iterates through the test packages in a project looking for the classes that are suffixed with the word Test. It is therefore essential to properly name tests to ensure they are executed correctly. Once tests have been created, NetBeans provides several methods for running the tests. The first method is to run all the tests that we have defined for an application. Selecting the Run and then Test Project menu options runs all of the tests defined for a project. The type of the project doesn't matter (Java SE or Java EE), nor whether a project uses Maven or the NetBeans project build system (Ant projects are even supported if they have a valid test activity), all tests for the project will be run when selecting this option. After running the tests, the Test Results window will be displayed, highlighting successful tests in green and failed tests in red. In the Test Results window, we have several options to help categorize and manage the tests: Rerun all of the tests Rerun the failed tests Show only the passed tests Show only the failed tests Show errors Show aborted tests Show skipped tests Locate previous failure Locate next failure Always open test result window Always open test results in a new tab The second option within NetBeans for running tests it to run all the tests in a package or class. To perform these operations, simply right-click on a package in the Projects window and select Test Package or right-click on a Java class in the Projects window and select Test File. The final option for running tests it to execute a single test in a class. To perform this operation, right-click on a test in the Java source code editor and select the Run Focussed Test Method menu option. After creating tests, how do we keep them up to date when we add new methods to application code? We can keep tests suites up to date by manually editing them and adding new methods corresponding to new application code or we can use the Create/Update Tests menu. Selecting the Tools and then Create/Update Tests menu options displays the Create Tests dialog that allows us to edit the existing test classes and add new methods into them, based upon the existing application classes. Summary In this article, we looked at the typical tasks that a developer does on a day-to-day basis when writing applications. We saw how NetBeans can help us to run and debug applications and how to profile applications and write tests for them. Finally, we took a brief look at TDD, and saw how the Red-Green-Refactor cycle can be used to help us develop more stable applications. Resources for Article: Further resources on this subject: Contexts and Dependency Injection in NetBeans [article] Creating a JSF composite component [article] Getting to know NetBeans [article]

 In this article by Greg Moss, author of Working with Odoo, he explains that Odoo is a very feature-filled business application framework with literally hundreds of applications and modules available. We have done our best to cover the most essential features of the Odoo applications that you are most likely to use in your business. Setting up an Odoo system is no easy task. Many companies get into trouble believing that they can just install the software and throw in some data. Inevitably, the scope of the project grows and what was supposed to be a simple system ends up being a confusing mess. Fortunately, Odoo's modular design will allow you to take a systematic approach to implementing Odoo for your business. (For more resources related to this topic, see here.) What is an ERP system? An Enterprise Resource Planning (ERP) system is essentially a suite of business applications that are integrated together to assist a company in collecting, managing, and reporting information throughout core business processes. These business applications, typically called modules, can often be independently installed and configured based on the specific needs of the business. As the needs of the business change and grow, additional modules can be incorporated into an existing ERP system to better handle the new business requirements. This modular design of most ERP systems gives companies great flexibility in how they implement the system. In the past, ERP systems were primarily utilized in manufacturing operations. Over the years, the scope of ERP systems have grown to encompass a wide range of business-related functions. Recently, ERP systems have started to include more sophisticated communication and social networking features. Common ERP modules The core applications of an ERP system typically include: Sales Orders Purchase Orders Accounting and Finance Manufacturing Resource Planning (MRP) Customer Relationship Management (CRM) Human Resources (HR) Let's take a brief look at each of these modules and how they address specific business needs. Selling products to your customer Sales Orders, commonly abbreviated as SO, are documents that a business generates when they sell products and services to a customer. In an ERP system, the Sales Order module will usually allow management of customers and products to optimize efficiency for data entry of the sales order. Many sales orders begin as customer quotes. Quotes allow a salesperson to collect order information that may change as the customer makes decisions on what they want in their final order. Once a customer has decided exactly what they wish to purchase, the quote is turned into a sales order and is confirmed for processing. Depending on the requirements of the business, there are a variety of methods to determine when a customer is invoiced or billed for the order. This preceding screenshot shows a sample sales order in Odoo. Purchasing products from suppliers Purchase Orders, often known as PO, are documents that a business generates when they purchase products from a vendor. The Purchase Order module in an ERP system will typically include management of vendors (also called suppliers) as well as management of the products that the vendor carries. Much like sales order quotes, a purchase order system will allow a purchasing department to create draft purchase orders before they are finalized into a specific purchasing request. Often, a business will configure the Sales Order and Purchase Order modules to work together to streamline business operations. When a valid sales order is entered, most ERP systems will allow you to configure the system so that a purchase order can be automatically generated if the required products are not in stock to fulfill the sales order. ERP systems will allow you to set minimum quantities on-hand or order limits that will automatically generate purchase orders when inventory falls below a predetermined level. When properly configured, a purchase order system can save a significant amount of time in purchasing operations and assist in preventing supply shortages. Managing your accounts and financing in Odoo Accounting and finance modules integrate with an ERP system to organize and report business transactions. In many ERP systems, the accounting and finance module is known as GL for General Ledger. All accounting and finance modules are built around a structure known as the chart of accounts. The chart of accounts organizes groups of transactions into categories such as assets, liabilities, income, and expenses. ERP systems provide a lot of flexibility in defining the structure of your chart of accounts to meet the specific requirements for your business. Accounting transactions are grouped by date into periods (typically by month) for reporting purposes. These reports are most often known as financial statements. Common financial statements include balance sheets, income statements, cash flow statements, and statements of owner's equity. Handling your manufacturing operations The Manufacturing Resource Planning (MRP) module manages all the various business operations that go into the manufacturing of products. The fundamental transaction of an MRP module is a manufacturing order, which is also known as a production order in some ERP systems. A manufacturing order describes the raw products or subcomponents, steps, and routings required to produce a finished product. The raw products or subcomponents required to produce the finished product are typically broken down into a detailed list called a bill of materials or BOM. A BOM describes the exact quantities required of each component and are often used to define the raw material costs that go into manufacturing the final products for a company. Often an MRP module will incorporate several submodules that are necessary to define all the required operations. Warehouse management is used to define locations and sublocations to store materials and products as they move through the various manufacturing operations. For example, you may receive raw materials in one warehouse location, assemble those raw materials into subcomponents and store them in another location, then ultimately manufacture the end products and store them in a final location before delivering them to the customer. Managing customer relations in Odoo In today's business environment, quality customer service is essential to being competitive in most markets. A Customer Relationship Management (CRM) module assists a business in better handling the interactions they may have with each customer. Most CRM systems also incorporate a presales component that will manage opportunities, leads, and various marketing campaigns. Typically, a CRM system is utilized the most by the sales and marketing departments within a company. For this reason, CRM systems are often considered to be sales force automation tools or SFA tools. Sales personnel can set up appointments, schedule call backs, and employ tools to manage their communication. More modern CRM systems have started to incorporate social networking features to assist sales personnel in utilizing these newly emerging technologies. Configuring human resource applications in Odoo Human Resource modules, commonly known as HR, manage the workforce- or employee-related information in a business. Some of the processes ordinarily covered by HR systems are payroll, time and attendance, benefits administration, recruitment, and knowledge management. Increased regulations and complexities in payroll and benefits have led to HR modules becoming a major component of most ERP systems. Modern HR modules typically include employee kiosk functions to allow employees to self-administer many tasks such as putting in a leave request or checking on their available vacation time. Finding additional modules for your business requirements In addition to core ERP modules, Odoo has many more official and community-developed modules available. At the time of this article's publication, the Odoo application repository had 1,348 modules listed for version 7! Many of these modules provide small enhancements to improve usability like adding payment type to a sales order. Other modules offer e-commerce integration or complete application solutions, such as managing a school or hospital. Here is a short list of the more common modules you may wish to include in an Odoo installation: Point of Sale Project Management Analytic Accounting Document Management System Outlook Plug-in Country-Specific Accounting Templates OpenOffice Report Designer You will be introduced to various Odoo modules that extend the functionality of the base Odoo system. You can find a complete list of Odoo modules at http://apps.Odoo.com/. This preceding screenshot shows the module selection page in Odoo. Getting quickly into Odoo Do you want to jump in right now and get a look at Odoo 7 without any complex installations? Well, you are lucky! You can access an online installation of Odoo, where you can get a peek at many of the core modules right from your web browser. The installation is shared publicly, so you will not want to use this for any sensitive information. It is ideal, however, to get a quick overview of the software and to get an idea for how the interface functions. You can access a trial version of Odoo at https://www.Odoo.com/start. Odoo – an open source ERP solution Odoo is a collection of business applications that are available under an open source license. For this reason, Odoo can be used without paying license fees and can be customized to suit the specific needs of a business. There are many advantages to open source software solutions. We will discuss some of these advantages shortly. Free your company from expensive software license fees One of the primary downsides of most ERP systems is they often involve expensive license fees. Increasingly, companies must pay these license fees on an annual basis just to receive bug fixes and product updates. Because ERP systems can require companies to devote great amounts of time and money for setup, data conversion, integration, and training, it can be very expensive, often prohibitively so, to change ERP systems. For this reason, many companies feel trapped as their current ERP vendors increase license fees. Choosing open source software solutions, frees a company from the real possibility that a vendor will increase license fees in the years ahead. Modify the software to meet your business needs With proprietary ERP solutions, you are often forced to accept the software solution the vendor provides chiefly "as is". While you may have customization options and can sometimes pay the company to make specific changes, you rarely have the freedom to make changes directly to the source code yourself. The advantages to having the source code available to enterprise companies can be very significant. In a highly competitive market, being able to develop solutions that improve business processes and give your company the flexibility to meet future demands can make all the difference. Collaborative development Open source software does not rely on a group of developers who work secretly to write proprietary code. Instead, developers from all around the world work together transparently to develop modules, prepare bug fixes, and increase software usability. In the case of Odoo, the entire source code is available on Launchpad.net. Here, developers submit their code changes through a structure called branches. Changes can be peer reviewed, and once the changes are approved, they are incorporated into the final source code product. Odoo – AGPL open source license The term open source covers a wide range of open source licenses that have their own specific rights and limitations. Odoo and all of its modules are released under the Affero General Public License (AGPL) version 3. One key feature of this license is that any custom-developed module running under Odoo must be released with the source code. This stipulation protects the Odoo community as a whole from developers who may have a desire to hide their code from everyone else. This may have changed or has been appended recently with an alternative license. You can find the full AGPL license at http://www.gnu.org/licenses/agpl-3.0.html. A real-world case study using Odoo The goal is to do more than just walk through the various screens and reports of Odoo. Instead, we want to give you a solid understanding of how you would implement Odoo to solve real-world business problems. For this reason, this article will present a real-life case study in which Odoo was actually utilized to improve specific business processes. Silkworm, Inc. – a mid-sized screen printing company Silkworm, Inc. is a highly respected mid-sized silkscreen printer in the Midwest that manufactures and sells a variety of custom apparel products. They have been kind enough to allow us to include some basic aspects of their business processes as a set of real-world examples implementing Odoo into a manufacturing operation. Using Odoo, we will set up the company records (or system) from scratch and begin by walking through their most basic sales order process, selling T-shirts. From there, we will move on to manufacturing operations, where custom art designs are developed and then screen printed onto raw materials for shipment to customers. We will come back to this real-world example so that you can see how Odoo can be used to solve real-world business solutions. Although Silkworm is actively implementing Odoo, Silkworm, Inc. does not directly endorse or recommend Odoo for any specific business solution. Every company must do their own research to determine whether Odoo is a good fit for their operation. Summary In this article, we have learned about the ERP system and common ERP modules. An introduction about Odoo and features of it. Resources for Article: Further resources on this subject: Getting Started with Odoo Development[article] Machine Learning in IPython with scikit-learn [article] Making Goods with Manufacturing Resource Planning [article]

In this article by Navin Agarwal, author of the book Liferay Portal 6.2 Enterprise Intranets, we will learn that Liferay is an enterprise application solution. It provides a lot of functionalities, which helps an organization to grow and is a one-solution package as a portal and content management solution. In this article, we will look at the following topics: The complete features you want your organization's intranet solution to have Reasons why Liferay is an excellent choice to build your intranet Where and how Liferay is used besides intranet portals Easy integration with other open source tools and applications Getting into more technical information about what Liferay is and how it works So, let's start looking at exactly what kind of site we're going to build. (For more resources related to this topic, see here.) Liferay Portal makes life easy We're going to build a complete corporate intranet solution using Liferay. Let's discuss some of the features your intranet portal will have. Hosted discussions Are you still using e-mail for group discussions? Then, it's time you found a better way! Running group discussions over e-mail clogs up the team's inbox—this means you have to choose your distribution list in advance, and that makes it hard for team members to opt in and out of the discussion. Using Liferay, we will build a range of discussion boards for discussion within and between teams. The discussions are archived in one place, which means that it's always possible to go back and refer to them later. On one level, it's just more convenient to move e-mail discussions to a discussion forum designed for the purpose. But once the forum is in place, you will find that a more productive group discussion takes place here than it ever did over e-mail. Collaborative documents using wikis Your company probably has guideline documents that should be updated regularly but swiftly lose their relevance as practices and procedures change. Even worse, each of your staff will know useful, productive tricks and techniques—but there's probably no easy way to record that knowledge in a way that is easy for others to find and use. We will see how to host wikis within Liferay. A wiki enables anybody to create and edit web pages and link all of those web pages together without requiring any HTML or programming skills. You can put your guideline documents into a wiki, and as practices change, your frontline staff can quickly and effortlessly update the guideline documentation. Wikis can also act as a shared notebook, enabling team members to collaborate and share ideas and findings and work together on documents. Team and individual blogs Your company probably needs frequent, chronological publications of personal thoughts and web links in the intranet. Your company probably has teams and individuals working on specific projects in order to share files and blogs about a project process and more. By using the Liferay Blog features, you can use HTML text editors to create or update files and blogs and to provide RSS feeds. Liferay provides an easy way for teams and individuals to share files with the help of blogs. Blogs provide a straightforward blogging solution with features such as RSS, user and guest comments, browsable categories, tags and labels, and a rating system. Liferay's RSS with the subscription feature provides the ability to frequently read RSS feeds from within the portal framework. At the same time, What You See Is What You Get (WYSIWYG) editors provide the ability to edit web content, including the blogs' content. Less technical people can use the WYSIWYG editor instead of sifting through complex code. Shared calendars Many companies require calendar information and share the calendar among users from different departments. We will see how to share a calendar within Liferay. The shared calendar can satisfy the basic business requirements incorporated into a featured business intranet, such as scheduling meetings, sending meeting invitations, checking for attendees' availability, and so on. Therefore, you can provide an environment for users to manage events and share calendars. Document management – CMS When there is a need for document sharing and document management, Liferay's Documents and Media library helps you with lots of features. The Documents and Media portlet allows you to add folders and subfolders for documents and media files, and also allows users to publish documents. It serves as a repository for all types of files and makes Content management systems (CMSes) available for intranets. The Documents and Media library portlet is equipped with customizable folders and acts as a web-based solution to share documents and media files among all your team members—just as a shared drive would. All the intranet users will be able to access the files from anywhere, and the content is accessible only by those authorized by administrators. All the files are secured by the permission layer by the administrator. Web content management – WCM Your company may have a lot of images and documents, and you may need to manage all these images and documents as well. Therefore, you require the ability to manage a lot of web content and then publish web content in intranets. We will see how to manage web content and how to publish web content within Liferay. Liferay Journal (Web Content) not only provides high availability to publish, manage, and maintain web content and documents, but it also separates content from the layout. Liferay WCM allows us to create, edit, and publish web content (articles). It also allows quick changes in the preview of the web content by changing the layout. It has built-in functionality, such as workflow, search, article versioning, scheduling, and metadata. Personalization and internalization All users can get a personal space that can be either made public (published as a website with a unique, friendly URL) or kept private. You can also customize how the space looks, what tools and applications are included, what goes into Documents and Media, and who can view and access all of this content. In addition, Liferay supports multiple languages, where you can select your own language. Multilingual organizations get out-of-the-box support for up to 45 languages. Users can toggle among different language settings with just one click and produce/publish multilingual documents and web content. Users can make use of the internalization feature to define the specific site in a localized language. Workflow, staging, scheduling, and publishing You can use a workflow to manage definitions, instances, and predetermined sequences of connected steps. Workflow can be used for web content management, assets, and so on. Liferay's built-in workflow engine is called Kaleo. It allows users to set up the review and publishing process on the web content article of any document that needs to end up on the live site. Liferay 6.2 integrates with the powerful features of the workflow and data capabilities of dynamic data lists in Kaleo Forms; it's only available in Liferay Enterprise Edition. Staging environments are integrated with Liferay's workflow engine. To have a review process for staged pages, you need to make sure you have a workflow engine configured and you have a staging setup in the workflow. As a content creator, you can update what you've created and publish it in a staging workflow. Other users can then review and modify it. Moreover, content editors can make a decision on whether to publish web content from staging to live, that is, you can easily create and manage everything from a simple article of text and images to fully functional websites in staging and then publish them live. Before going live, you can schedule web content as well. For instance, you can publish web content immediately or schedule it for publishing on a specific date. Social networks and Social Office Liferay Portal supports social networks—you can easily manage your Google Plus, Facebook, MySpace, Twitter, and other social network accounts in Liferay. In addition, you can manage your instant messenger accounts, such as AIM, ICQ, Jabber, MSN, Skype, YM, and so on smoothly from inside Liferay. Liferay Social Office gives us a social collaboration on top of the portal—a fully virtual workspace that streamlines communication and builds up group cohesion. It provides holistic enhancement to the way you and your colleagues work together. All components in Social Office are tied together seamlessly, getting everyone on the same page by sharing the same look and feel. More importantly, the dynamic activity tracking gives us a bird's-eye view of who has been doing what and when within each individual site. Using Liferay Social Office, you can enhance your existing personal workflow with social tools, keep your team up to date, and turn collective knowledge into collective action. Note that Liferay 6.2 supports the Liferay Social Office 3.0 current version. Liferay Sync and Marketplace Liferay Sync is Liferay's newest product, designed to make file sharing as easy as a simple drag and drop! Liferay Sync is an add-on product for Liferay 6.1 CE, EE, and later versions, which makes it a more raw boost product and enables the end user to publish and access documents and files from multiple environments and devices, including Windows and MacOS systems, and iOS-based mobile platforms. Liferay Sync is one of the best features, and it is fully integrated into the Liferay platform. Liferay 6.1 introduced the new concept of the marketplace, which leverages the developers to develop any components or functionality and release and share it with other users. It's a user-friendly and one-stop place to share apps. Liferay Marketplace provides the portal product with add-on features with a new hub to share, browse, and download Liferay-compatible applications. In Liferay 6.2, Marketplace comes under App Manager, where all the app-related controls can be possible. More features The intranet also arranges staff members into teams and sites, provides a way of real-time IM and chatting, and gives each user an appropriate level of access. This means that they can get all the information they need and edit and add content as necessary but won't be able to mess with sensitive information that they have no reason to see. In particular, the portal provides an integrating framework so that you can integrate external applications easily. For example, you can integrate external applications with the portal, such as Alfresco, OpenX, LDAP, SSO CAS, Orbeon Forms, Konakart, PayPal, Solr, and so on. In a word, the portal offers compelling benefits to today's enterprises—reduced operational costs, improved customer satisfaction, and streamlined business processes. Everything in one place All of these features are useful on their own. However, it gets better when you consider that all of these features will be combined into one easy-to-use searchable portal. A user of the intranet, for example, can search for a topic—let's say financial report—and find the following in one go: Any group discussions about financial reports Blog entries within the intranet concerning financial reports Documents and files—perhaps the financial reports themselves Wiki entries with guidelines on preparing financial reports Calendar entries for meetings to discuss the financial report Of course, users can also restrict their search to just one area if they already know exactly what they are looking for. Liferay provides other features, such as tagging, in order to make it even easier to organize information across the whole intranet. We will do all of this and more. Introducing Palm Tree Publications We are going to build an intranet for a fictional company as an example, focusing on how to install, configure, and integrate it with other applications and also implement portals and plugins (portlets, themes, layout templates, hooks, and webs) within Liferay. By applying the instructions to your own business, you will be able to build an intranet to meet your own company's needs. "Palm Tree Publications" needs an intranet of its own, which we will call bookpub.com. The enterprise's global headquarters are in the United States. It has several departments—editorial, website, engineering, marketing, executive, and human resources. Each department has staff in the U.S., Germany, and India or in all three places. The intranet site provides a site called "Book Street and Book Workshop" consisting of users who have an interest in reading books. The enterprise needs to integrate collaboration tools, such as wikis, discussion forums, blogs, instant messaging, mail, RSS, shared calendars, tagging, and so on. Palm Tree Publications has more advanced needs too: a workflow to edit, approve, and publish books. Furthermore, the enterprise has a lot of content, such as books stored and managed alfresco currently. In order to build the intranet site, the following functionality should be considered: Installing the portal, experiencing the portal and portlets, and customizing the portal and personal web pages Bringing the features of enabling document sharing, calendar sharing, and other collaboration within a business to the users of the portal Discussion forums—employees should be able to discuss book ideas and proposals Wikis—keeping track of information about editorial guidance and other resources that require frequent editing Dissemination of information via blogs—small teams working on specific projects share files and blogs about a project process Sharing a calendar among employees Web content management creation by the content author and getting approved by the publisher Document repository—using effective content management systems (CMSes), a natural fit for a portal for secure access, permissions, and distinct roles (such as writers, editors, designers, administrators, and so on) Collaborative chat and instant messaging, social network, Social Office, and knowledge management tools Managing a site named Book Street and Book Workshop that consists of users who have the same interest in reading books as staging, scheduling, and publishing web content related to books Federated search for discussion forum entries, blog posts, wiki articles, users in the directory, and content in both the Document and Media libraries; search by tags Integrating back-of-the-house software applications, such as Alfresco, Orbeon Forms, the Drools rule server, Jasper Server, and BI/Reporting Pentaho; strong authentication and authorization with LDAP; and single authentication to access various company sites besides the intranet site The enterprise can have the following groups of people: Admin: This group installs systems, manages membership, users, user groups, organizations, roles and permissions, security on resources, workflow, servers and instances, and integrates with third-party systems Executives: Executive management handles approvals Marketing: This group handles websites, company brochures, marketing campaigns, projects, and digital assets Sales: This group makes presentations, contracts, documents, and reports Website editors: This group manages pages of the intranet—writes articles, reviews articles, designs the layout of articles, and publishes articles Book editors: This group writes, reviews, and publishes books and approves and rejects the publishing of books Human resources: This group manages corporate policy documents Finance: This group manages accounts documents, scanned invoices and checks accounts Corporate communications: This group manages external public relations, internal news releases, and syndication Engineering: This group sets up the development environment and collaborates on engineering projects and presentation templates Introducing Liferay Portal's architecture and framework Liferay Portal's architecture supports high availability for mission-critical applications using clustering and the fully distributed cache and replication support across multiple servers. The following diagram has been taken from the Liferay forum written by Jorge Ferrer. This diagram depicts the various architectural layers and functionalities of portlets: Figure 1.1: The Liferay architecture The preceding image was taken from https://www.liferay.com/web/jorge.ferrer/blog/-/blogs/liferay-s-architecture-the-beginning-of-a-blog-series site blog. The Liferay Portal architecture is designed in such a way that it provides tons of features at one place: Frontend layer: This layer is the end user's interface Service layer: This contains the great majority of the business logic for the portal platform and all of the portlets included out of the box Persistence layer: Liferay relies on Hibernate to do most of its database access Web services API layer: This handles web services, such as JSON and SOAP In Liferay, the service layer, persistence layer, and web services API layer are built automatically by that wonderful tool called Service Builder. Service Builder is the tool that glues together all of Liferay's layers and that hides the complexities of using Spring or Hibernate under the hood. Service-oriented architecture Liferay Portal uses service-oriented architecture (SOA) design principles throughout and provides the tools and framework to extend SOA to other enterprise applications. Under the Liferay enterprise architecture, not only can the users access the portal from traditional and wireless devices, but developers can also access it from the exposed APIs via REST, SOAP, RMI, XML-RPC, XML, JSON, Hessian, and Burlap. Liferay Portal is designed to deploy portlets that adhere to the portlet API compliant with both JSR-168 and JSR-286. A set of useful portlets are bundled with the portal, including Documents and Media, Calendar, Message Boards, Blogs, Wikis, and so on. They can be used as examples to add custom portlets. In a word, the key features of Liferay include using SOA design principles throughout, such as reliable security, integrating the portal with SSO and LDAP, multitier and limitless clustering, high availability, caching pages, dynamic virtual hosting, and so on. Understanding Enterprise Service Bus Enterprise Service Bus (ESB) is a central connection manager that allows applications and services to be added quickly to an enterprise infrastructure. When an application needs to be replaced, it can easily be disconnected from the bus at a single point. Liferay Portal uses Mule or ServiceMix as ESB. Through ESB, the portal can integrate with SharePoint, BPM (such as the jBPM workflow engine and Intalio | BPMS engine), BI Xforms reporting, JCR repository, and so on. It supports JSR 170 for content management systems with the integration of JCR repositories, such as Jackrabbit. It also uses Hibernate and JDBC to connect to any database. Furthermore, it supports an event system with synchronous and asynchronous messaging and a lightweight message bus. Liferay Portal uses the Spring framework for its business and data services layers. It also uses the Spring framework for its transaction management. Based on service interfaces, portal-impl is implemented and exposed only for internal usage—for example, they are used for the extension environment. portal-kernel and portal-service are provided for external usage (or for internal usage)—for example, they are used for the Plugins SDK environment. Custom portlets, both JSR-168 and JSR-286, and web services can be built based on portal-kernel and portal-service. In addition, the Web 2.0 Mail portlet and the Web 2.0 Chat portlet are supported as well. More interestingly, scheduled staging and remote staging and publishing serve as a foundation through the tunnel web for web content management and publishing. Liferay Portal supports web services to make it easy for different applications in an enterprise to communicate with each other. Java, .NET, and proprietary applications can work together easily because web services use XML standards. It also supports REST-style JSON web services for lightweight, maintainable code and supports AJAX-based user interfaces. Liferay Portal uses industry-standard, government-grade encryption technologies, including advanced algorithms, such as DES, MD5, and RSA. Liferay was benchmarked as one of the most secure portal platforms using LogicLibrary's Logiscan suite. Liferay offers customizable single sign-on (SSO) that integrates into Yale CAS, JAAS, LDAP, NTLM, CA Siteminder, Novell Identity Manager, OpenSSO, and more. Open ID, OpenAuth, Yale CAS, Siteminder, and OpenAM integration are offered by it out of the box. In short, Liferay Portal uses ESB in general with an abstraction layer on top of an enterprise messaging system. It allows integration architects to exploit the value of messaging systems, such as reporting, e-commerce, and advertisements. Understanding the advantages of using Liferay to build an intranet Of course, there are lots of ways to build a company intranet. What makes Liferay such a good choice to create an intranet portal? It has got the features we need All of the features we outlined for our intranet come built into Liferay: discussions, wikis, calendars, blogs, and so on are part of what Liferay is designed to do. It is also designed to tie all of these features together into one searchable portal, so we won't be dealing with lots of separate components when we build and use our intranet. Every part will work together with others. Easy to set up and use Liferay has an intuitive interface that uses icons, clear labels, and drag and drop to make it easy to configure and use the intranet. Setting up the intranet will require a bit more work than using it, of course. However, you will be pleasantly surprised by how simple it is—no programming is required to get your intranet up and running. Free and open source How much does Liferay cost? Nothing! It's a free, open source tool. Here, being free means that you can go to Liferay's website and download it without paying anything. You can then go ahead and install it and use it. Liferay comes with an enterprise edition too, for which users need to pay. In addition, Liferay provides full support and access to additional enterprise edition plugins/applications. Liferay makes its money by providing additional services, including training. However, the standard use of Liferay is completely free. Now you probably won't have to pay another penny to get your intranet working. Being open source means that the program code that makes Liferay work is available to anybody to look at and change. Even if you're not a programmer, this is still good for you: If you need Liferay to do something new, then you can hire a programmer to modify Liferay to do it. There are lots of developers studying the source code, looking for ways to make it better. Lots of improvements get incorporated into Liferay's main code. Developers are always working to create plugins—programs that work together with Liferay to add new features. Probably, for now, the big deal here is that it doesn't cost any money. However, as you use Liferay more, you will come to understand the other benefits of open source software for you. Grows with you Liferay is designed in a way that means it can work with thousands and thousands of users at once. No matter how big your business is or how much it grows, Liferay will still work and handle all of the information you throw at it. It also has features especially suited to large, international businesses. Are you opening offices in non-English speaking countries? No problem! Liferay has internationalization features tailored to many of the world's popular languages. Works with other tools Liferay is designed to work with other software tools—the ones that you're already using and the ones that you might use in the future—for instance: You can hook up Liferay to your LDAP directory server and SSO so that user details and login credentials are added to Liferay automatically Liferay can work with Alfresco—a popular and powerful Enterprise CMS (used to provide extremely advanced document management capabilities, which are far beyond what Liferay does on its own) Based on "standards" This is a more technical benefit; however, it is a very useful one if you ever want to use Liferay in a more specialized way. Liferay is based on standard technologies that are popular with developers and other IT experts and that confer the following benefits on users: Built using Java: Java is a popular programming language that can run on just about any computer. There are millions of Java programmers in the world, so it won't be too hard to find developers who can customize Liferay. Based on tried and tested components: With any tool, there's a danger of bugs. Liferay uses lots of well-known, widely tested components to minimize the likelihood of bugs creeping in. If you are interested, here are some of the well-known components and technologies Liferay uses—Apache ServiceMix, Mule, ehcache, Hibernate, ICEfaces, Java J2EE/JEE, jBPM, Activiti, JGroups, Alloy UI, Lucene, PHP, Ruby, Seam, Spring and AOP, Struts and Tiles, Tapestry, Velocity, and FreeMarker. Uses standard ways to communicate with other software: There are various standards established to share data between pieces of software. Liferay uses these so that you can easily get information from Liferay into other systems. The standards implemented by Liferay include AJAX, iCalendar and Microformat, JSR-168, JSR-127, JSR-170, JSR-286 (Portlet 2.0), JSR-314 (JSF 2.0), OpenSearch, the Open platform with support for web services, including JSON, Hessian, Burlap, REST, RMI, and WSRP, WebDAV, and CalDAV. Makes publication and collaboration tools Web Content Accessibility Guidelines 2.0 (WCAG 2.0) compliant: The new W3C recommendation is to make web content accessible to a wide range of people with disabilities, including blindness and low vision, deafness and hearing loss, learning disabilities, cognitive limitations, limited movement, speech disabilities, photosensitivity, and combinations of these. For example, the portal integrates CKEditor-standards support, such as W3C (WAI-AA and WCAG), 508 (Section 508). Alloy UI: The Liferay UI supports HTML 5, CSS 3, and Yahoo! User Interface Library 3 (YUI 3). Supports Apache Ant 1.8 and Maven 2: Liferay Portal can be built through Apache Ant by default, where you can build services; clean, compile, and build JavaScript CMD; build language native to ASCII, deploy, fast deploy; and so on. Moreover, Liferay supports Maven 2 SDK, providing Community Edition (CE) releases through public maven repositories as well as Enterprise Edition (EE) customers to install maven artifacts in their local maven repository. Bootstrap: Liferay 6.2 provides support for Twitter Bootstrap out of the box. With its fully responsive UI, the benefit of bootstrap is that it will support any device to render the content. Even content authors can use bootstrap markup and styles to make the content nicer. Many of these standards are things that you will never need to know much about, so don't worry if you've never heard of them. Liferay is better for using them, but mostly, you won't even know they are there. Other advantages of Liferay Liferay isn't just for intranets! Users and developers are building all kinds of different websites and systems based on Liferay. Corporate extranets An intranet is great for collaboration and information sharing within a company. An extranet extends this facility to suppliers and customers, who usually log in over the Internet. In many ways, this is similar to an intranet—however, there are a few technical differences. The main difference is that you create user accounts for people who are not part of your company. Collaborative websites Collaborative websites not only provide a secure and administrated framework, but they also empower users with collaborative tools, such as blogs, instant e-mail, message boards, instant messaging, shared calendars, and so on. Moreover, they encourage users to use other tools, such as tag administration, fine-grained permissions, delegable administrator privileges, enterprise taxonomy, and ad hoc user groups. By means of these tools, as an administrator, you can ultimately control what people can and cannot do in Liferay. In many ways, this is similar to an intranet too; however, there are a few technical differences. The main difference is that you use collaborative tools simply, such as blogs, instant e-mail, message boards, instant messaging, shared calendars, and so on. Content management and web publishing You can also use Liferay to run your public company website with content management and web publishing. Content management and web publishing are useful features in websites. It is a fact that the volume of digital content for any organization is increasing on a daily basis. Therefore, an effective CMS is a vital part of any organization. Meanwhile, document management is also useful and more effective when repositories have to be assigned to different departments and groups within the organization. Content management and document management are effective in Liferay. Moreover, when managing and publishing content, we may have to answer many questions, such as "who should be able to update and delete a document from the system?". Fortunately, Liferay's security and permissions model can satisfy the need for secure access and permissions and distinct roles (for example, writer, editor, designer, and administrator). Furthermore, Liferay integrates with the workflow engine. Thus, users can follow a flow to edit, approve, and publish content in the website. Content management and web publishing are similar to an intranet; however, there are a few technical differences. The main difference is that you can manage content and publish web content smoothly. Infrastructure portals Infrastructure portals integrate all possible functions, as we stated previously. This covers collaboration and information sharing within a company in the form of collaborative tools, content management, and web publishing. In infrastructure portals, users can create a unified interface to work with content, regardless of source via content interaction APIs. Furthermore, using the same API and the same interface as that of the built-in CMS, users can also manage content and publish web content from third-party systems, such as Alfresco, Vignette, Magnolia, FatWire, Microsoft SharePoint, and so on. Infrastructure portals are similar to an intranet; there are a few technical differences though. The main difference is that you can use collaborative tools, manage content, publish web content, and integrate other systems in one place. Why do you need a portal? The main reason is that a portal can serve as a framework to aggregate content and applications. A portal normally provides a secure and manageable framework where users can easily make new and existing enterprise applications available. In order to build an infrastructure portal smoothly, Liferay Portal provides an SOA-based framework to integrate third-party systems. Out-of-the-box portlets and features Liferay provides out-of-the-box (OOTB) portlets that have key features and can be used in the enterprise intranet very efficiently. These portlets are very scalable and powerful and provide the developer with the tools to customize it very easily. Let's see some of the most frequently used portlets in Liferay Portal. Content management Content management is a common feature in any web-based portal or website: The Web Content portlet has the features of full web publishing, office integration, and the asset library, which contains documents, images, and videos. This portlet also has the structure and templates that help with the designing of the web content's look and feel. Structure can be designed with the help of a visual editor with drag and drop. It has the integrated help feature with tooltips to name the attributes of the fields. The Asset Publisher portlet provides you with the feature to select any type of content/asset, such as wiki pages, web content, calendar events, message board messages, documents, media documents, and many more. It also allows us to use filter on them by types, categories, tags, and sources. The display settings provide configurable settings, which helps the content to be displayed to the end users perfectly. The Document and Media portlet is one of the most usable portlets to store any type of document. It allows you to store and manage your documents. It allows you to manage Liferay documents from your own machine's filesystem with the help of WebDAV integration. It has lots of new, built-in features, such as the inline document preview, image preview, and video player. Document metadata is displayed in document details, which makes it easier for you to review the metadata of the document. Also, Document and Media has features named checkin and checkout that helps editing the document in a group very easily. The Document and Media portlet has the multi-repository integration feature, which allows you to configure or mount any other repository very easily, such as SharePoint, Documentum, and Alfresco, utilizing the CMIS standard. Collaboration Collaboration features are generally ways in which users communicate with each other, such as the ones shown in the following list: The Dynamic data list portlet provides you with the facility of not writing a single line of code to create the form or data list. Say, for example, your corporate intranet needs the job posting done on a daily basis by the HR administrator. The administrator needs to develop the custom portlet to fulfill that requirement. Now, the dynamic data list portlet will allow the administrator to create a form for job posting. It's very easy to create and display new data types. The Blog portlet is one of the best features of Liferay. Blog portlets have two other related portlets, namely Recent Bloggers and Blogs Aggregator. The blog portlet provides the best possible ways for chronological publications of personal thoughts and web links in the intranet. Blog portlets can be placed for users of different sites/departments under the respective site//department page. The Calendar portlet provides the feature to create the event and schedule the event. It has many features that help the users in viewing the meeting schedule. The Message Board portlet is a full-featured forum solution with threaded views, categories, RSS capability, avatars, file attachments, previews, dynamic lists of recent posts, and forum statistics. Message Board portlets work with the fine-grained permissions and role-based access control model to give detailed levels of control to administrators and users. The Wiki portlet, like the Message Boards portlet, provides a straightforward wiki solution for both intranet and extranet portals that provides knowledge management among the users. It has all of the features you would expect in a state-of-the-art wiki. Again, it has the features of a file attachment preview, publishing the content, and versioning, and works with a fine-grained permission and role-based access control model. This again takes all the features of the Liferay platform. The Social Activity portlet allows you to tweak the measurements used to calculate user involvement within a site. The contribution and participation values determine the reward value of an action. It uses the blog entry, wiki, and message board points to calculate the user involvement in the site. The Marketplace portlet is placed inside the control panel. It's a hub for the applications provided by Liferay and other partners. You can find that many applications are free, and for certain applications, you need to pay an amount. It's more like an app store. This feature was introduced in Liferay Version 6.1. In the Liferay 6.2 control panel, under the Apps | Store link section, you will see apps that are stored in the Marketplace portlet. Liferay 6.2 comes with a new control panel that is very easy to manage for the portal's Admin users. Liferay Sync is not a portlet; it's a new feature of Liferay that allows you to synchronize documents of Liferay Document and Media with your local system. Liferay provide the Liferay Sync application, which has to be installed in your local system or mobile device. News RSS portlets provide RSS feeds. RSS portlets are used for the publishers by letting them syndicate content automatically. They benefit readers who want to subscribe to timely updates from their favorite websites or to aggregate feeds from many sites into one place. A Liferay RSS portlet is fully customizable, and it allows you to set the URL from which site you would like to get feeds. Social Activities portlets display portal-wide user activity, such as posting on message boards, creating wikis, and adding documents to Documents and Media. There are more portlets for social categories, such as User Statistics portlets, Group Statistics portlets, and Requests portlets. All these portlets are used for the social media. Tools The Search portlet provides faceted search features. When a search is performed, facet information will appear based on the results of the search. The number of each asset type and the most frequently occurring tags and categories as well as their frequency will all appear in the left-hand side column of the portlet. It searches through Bookmarks, Blogs Entries, Web Content Articles, Document Library Files, Users, Message Board, and Wiki. Finding more information on Liferay In this article, we looked at what Liferay can do for your corporate intranet and briefly saw why it's a good choice. If you want more background information on Liferay, the best place to start is the Liferay corporate website (http://www.liferay.com) itself. You can find the latest news and events, various training programs offered worldwide, presentations, demonstrations, and hosted trails. More interestingly, Liferay eats its own dog food; corporate websites within forums (called message boards), blogs, and wikis are built by Liferay using its own products. It is a real demo of Liferay Portal's software. Liferay is 100 percent open source and all downloads are available from the Liferay Portal website at http://www.liferay.com/web/guest/downloads/portal and the SourceForge website at http://sourceforge.net/projects/lportal/files. The source code repository is available at https://github.com/liferay. The Liferay website's wiki (http://www.liferay.com/web/guest/community/wiki) contains documentation, including a tutorial, user guide, developer guide, administrator guide, roadmap, and so on. The Liferay website's discussion forums can be accessed at http://www.liferay.com/web/guest/community/forums and the blogs at http://www.liferay.com/community/blogs/highlighted. The official plugins and the community plugins are available at http://www.liferay.com/marketplace and are the best place to share your thoughts, get tips and tricks about Liferay implementation, and use and contribute community plugins. If you would like to file a bug or know more about the fixes in a specific release, then you must visit the bug-tracking system at http://issues.liferay.com/. Summary In this article, we looked at what Liferay can offer your intranet and what we should consider while designing the company's enterprise site. We saw that our final intranet will provide shared documents, discussions, collaborative wikis, and more in a single, searchable portal. Well, Liferay is a great choice for an intranet because it provides so many features and is easy to use, free and open source, extensible, and well-integrated with other tools and standards. We also saw the other kinds of sites Liferay is good for, such as extranets, collaborative websites, content management, web publishing, and infrastructure portals. For the best example of an intranet and extranet, you can visit www.liferay.com. It will provide you with more background information. Resources for Article: Further resources on this subject: Working with a Liferay User / User Group / Organization[article] Liferay, its Installation and setup[article] Building your First Liferay Site [article]

In this article, by Mark Collin, the author of the book, Mastering Selenium WebDriver, we will look at how we can directly execute JavaScript snippets in Selenium. We will explore the sort of things that you can do and how they can help you work around some of the limitations that you will come across while writing your scripts. We will also have a look at some examples of things that you should avoid doing. (For more resources related to this topic, see here.) Introducing the JavaScript executor Selenium has a mature API that caters to the majority of automation tasks that you may want to throw at it. That being said, you will occasionally come across problems that the API doesn't really seem to support. This was very much on the development team's mind when Selenium was written. So, they provided a way for you to easily inject and execute arbitrary blocks of JavaScript. Let's have a look at a basic example of using a JavaScript executor in Selenium: JavascriptExecutor js = (JavascriptExecutor) driver; js.executeScript("console.log('I logged something to the Javascript console');"); Note that the first thing we do is cast a WebDriver object into a JavascriptExecutor object. The JavascriptExecutor interface is implemented through the RemoteWebDriver class. So, it's not a part of the core set of API functions. Since we normally pass around a WebDriver object, the executeScript functions will not be available unless we perform this cast. If you are directly using an instance of RemoteWebDriver or something that extends it (most driver implementations now do this), you will have direct access to the .executeScript() function. Here's an example: FirefoxDriver driver = new FirefoxDriver(new FirefoxProfile()); driver.executeScript("console.log('I logged something to the Javascript console');"); The second line (in both the preceding examples) is just telling Selenium to execute an arbitrary piece of JavaScript. In this case, we are just going to print something to the JavaScript console in the browser. We can also get the .executeScript() function to return things to us. For example, if we tweak the script of JavaScript in the first example, we can get Selenium to tell us whether it managed to write to the JavaScript console or not, as follows: JavascriptExecutor js = (JavascriptExecutor) driver; Object response = js.executeScript("return console.log('I logged something to the Javascript console');"); In the preceding example, we will get a result of true coming back from the JavaScript executor. Why does our JavaScript start with return? Well, the JavaScript executed by Selenium is executed as a body of an anonymous function. This means that if we did not add a return statement to the start of our JavaScript snippet, we would actually be running this JavaScript function using Selenium: var anonymous = function () { console.log('I logged something to the Javascript console'); }; This function does log to the console, but it does not return anything. So, we can't access the result of the JavaScript snippet. If we prefix it with a return, it will execute this anonymous function: var anonymous = function () { return console.log('I logged something to the Javascript console'); }; This does return something for us to work with. In this case, it will be the result of our attempt to write some text to the console. If we succeeded in writing some text to the console, we will get back a true value. If we failed, we will get back a false value. Note that in our example, we saved the response as an object—not a string or a Boolean. This is because the JavaScript executor can return lots of different types of objects. What we get as a response can be one of the following: If the result is null or there is no return value, a null will be returned If the result is an HTML element, a WebElement will be returned If the result is a decimal, a double will be returned If the result is a nondecimal number, a long will be returned If the result is a Boolean, a Boolean will be returned If the result is an array, a List object with each object that it contains, along with all of these rules, will be returned (nested lists are supported) For all other cases, a string will be returned It is an impressive list, and it makes you realize just how powerful this method is. There is more as well. You can also pass arguments into the .executeScript() function. The arguments that you pass in can be any one of the following: Number Boolean String WebElement List They are then put into a magic variable called arguments, which can be accessed by the JavaScript. Let's extend our example a little bit to pass in some arguments, as follows: String animal = "Lion"; int seen = 5; JavascriptExecutor js = (JavascriptExecutor) driver; js.executeScript("console.log('I have seen a ' + arguments[0] + ' ' + arguments[1] + ' times(s)');", animal, seen); This time, you will see that we managed to print the following text into the console: I have seen a Lion 5 times(s) As you can see, there is a huge amount of flexibility with the JavaScript executor. You can write some complex bits of JavaScript code and pass in lots of different types of arguments from your Java code. Think of all the things that you could do! Let's not get carried away We now know the basics of how one can execute JavaScript snippets in Selenium. This is where some people can start to get a bit carried away. If you go through the mailing list of the users of Selenium, you will see many instances of people asking why they can't click on an element. Most of the time, this is due to the element that they are trying to interact with not being visible, which is blocking a click action. The real solution to this problem is to perform an action (the same one that they would perform if they were manually using the website) to make the element visible so that they can interact with it. However, there is a shortcut offered by many, which is a very bad practice. You can use a JavaScript executor to trigger a click event on this element. Doing this will probably make your test pass. So why is it a bad solution? The Selenium development team has spent quite a lot of time writing code that works out if a user can interact with an element. It's pretty reliable. So, if Selenium says that you cannot currently interact with an element, it's highly unlikely that it's wrong. When figuring out whether you can interact with an element, lots of things are taken into account, including the z-index of an element. For example, you may have a transparent element that is covering the element that you want to click on and blocking the click action so that you can't reach it. Visually, it will be visible to you, but Selenium will correctly see it as not visible. If you now invoke a JavaScript executor to trigger a click event on this element, your test will pass, but users will not be able to interact with it when they try to manually use your website. However, what if Selenium got it wrong and I can interact with the element that I want to click manually? Well, that's great, but there are two things that you need to think about. First of all, does it work in all browsers? If Selenium thinks that it is something that you cannot interact with, it's probably for a good reason. Is the markup, or the CSS, overly complicated? Can it be simplified? Secondly, if you invoke a JavaScript executor, you will never know whether the element that you want to interact with really does get blocked at some point in the future. Your test may as well keep passing when your application is broken. Tests that can't fail when something goes wrong are worse than no test at all! If you think of Selenium as a toolbox, a JavaScript executor is a very powerful tool that is present in it. However, it really should be seen as a last resort when all other avenues have failed you. Too many people use it as a solution to any slightly sticky problem that they come across. If you are writing JavaScript code that attempts to mirror existing Selenium functions but are removing the restrictions, you are probably doing it wrong! Your code is unlikely to be better. The Selenium development team have been doing this for a long time with a lot of input from a lot of people, many of them being experts in their field. If you are thinking of writing methods to find elements on a page, don't! Use the .findElement() method provided by Selenium. Occasionally, you may find a bug in Selenium that prevents you from interacting with an element in the way you would expect to. Many people first respond by reaching for the JavascriptExecutor to code around the problem in Selenium. Hang on for just one moment though. Have you upgraded to the latest version of Selenium to see if that fixes your problem? Alternatively, did you just upgrade to the latest version of Selenium when you didn't need to? Using a slightly older version of Selenium that works correctly is perfectly acceptable. Don't feel forced to upgrade for no reason, especially if it means that you have to write your own hacks around problems that didn't exist before. The correct thing to do is to use a stable version of Selenium that works for you. You can always raise bugs for functionality that doesn't work, or even code a fix and submit a pull request. Don't give yourself the additional work of writing a workaround that's probably not the ideal solution, unless you need to. So what should we do with it? Let's have a look at some examples of the things that we can do with the JavaScript executor that aren't really possible using the base Selenium API. First of all, we will start off by getting the element text. Wait a minute, element text? But, that’s easy! You can use the existing Selenium API with the following code: WebElement myElement = driver.findElement(By.id("foo")); String elementText = myElement.getText(); So why would we want to use a JavaScript executor to find the text of an element? Getting text is easy using the Selenium API, but only under certain conditions. The element that you are collecting the text from needs to be displayed. If Selenium thinks that the element from which you are collecting the text is not displayed, it will return an empty string. If you want to collect some text from a hidden element, you are out of luck. You will need to implement a way to do it with a JavaScript executor. Why would you want to do this? Well, maybe you have a responsive website that shows different elements based on different resolutions. You may want to check whether these two different elements are displaying the same text to the user. To do this, you will need to get the text of the visible and invisible elements so that you can compare them. Let's create a method to collect some hidden text for us: private String getHiddenText(WebElement element) { JavascriptExecutor js = (JavascriptExecutor) ((RemoteWebElement) element).getWrappedDriver(); return (String) js.executeScript("return arguments[0].text", element); } There is some cleverness in this method. First of all, we took the element that we wanted to interact with and then extracted the driver object associated with it. We did this by casting the WebElement into a RemoteWebElement, which allowed us to use the getWrappedDriver() method. This removes the need to pass a driver object around the place all the time (this is something that happens a lot in some code bases). We then took the driver object and cast it into a JavascriptExecutor so that we would have the ability to invoke the executeScript() method. Next, we executed the JavaScript snippet and passed in the original element as an argument. Finally, we took the response of the executeScript() call and cast it into a string that we can return as a result of the method. Generally, getting text is a code smell. Your tests should not rely on specific text being displayed on a website because content always changes. Maintaining tests that check the content of a site is a lot of work, and it makes your functional tests brittle. The best thing to do is test the mechanism that injects the content into the website. If you use a CMS that injects text into a specific template key, you can test whether each element has the correct template key associated with it. I want to see a more complex example! So you want to see something more complicated. The Advanced User Interactions API cannot deal with HTML5 drag and drop. So, what happens if we come across an HTML5 drag-and-drop implementation that we want to automate? Well, we can use the JavascriptExecutor. Let's have a look at the markup for the HTML5 drag-and-drop page: <!DOCTYPE html> <html lang="en"> <head> <meta charset=utf-8> <title>Drag and drop</title> <style type="text/css"> li { list-style: none; } li a { text-decoration: none; color: #000; margin: 10px; width: 150px; border-width: 2px; border-color: black; border-style: groove; background: #eee; padding: 10px; display: block; } *[draggable=true] { cursor: move; } ul { margin-left: 200px; min-height: 300px; } #obliterate { background-color: green; height: 250px; width: 166px; float: left; border: 5px solid #000; position: relative; margin-top: 0; } #obliterate.over { background-color: red; } </style> </head> <body> <header> <h1>Drag and drop</h1> </header> <article> <p>Drag items over to the green square to remove them</p> <div id="obliterate"></div> <ul> <li><a id="one" href="#" draggable="true">one</a></li> <li><a id="two" href="#" draggable="true">two</a></li> <li><a id="three" href="#" draggable="true">three</a></li> <li><a id="four" href="#" draggable="true">four</a></li> <li><a id="five" href="#" draggable="true">five</a></li> </ul> </article> </body> <script> var draggableElements = document.querySelectorAll('li > a'), obliterator = document.getElementById('obliterate'); for (var i = 0; i < draggableElements.length; i++) { element = draggableElements[i]; element.addEventListener('dragstart', function (event) { event.dataTransfer.effectAllowed = 'copy'; event.dataTransfer.setData('being-dragged', this.id); }); } obliterator.addEventListener('dragover', function (event) { if (event.preventDefault) event.preventDefault(); obliterator.className = 'over'; event.dataTransfer.dropEffect = 'copy'; return false; }); obliterator.addEventListener('dragleave', function () { obliterator.className = ''; return false; }); obliterator.addEventListener('drop', function (event) { var elementToDelete = document.getElementById( event.dataTransfer.getData('being-dragged')); elementToDelete.parentNode.removeChild(elementToDelete); obliterator.className = ''; return false; }); </script> </html> Note that you need a browser that supports HTML5/CSS3 for this page to work. The latest versions of Google Chrome, Opera Blink, Safari, and Firefox will work. You may have issues with Internet Explorer (depending on the version that you are using). For an up-to-date list of HTML5/CSS3 support, have a look at http://caniuse.com. If you try to use the Advanced User Interactions API to automate this page, you will find that it just doesn't work. It looks like it's time to reach for JavascriptExecutor. First of all, we need to write some JavaScript that can simulate the events that we need to trigger to perform the drag-and-drop action. To do this, we are going to create three JavaScript functions. The first function is going to create a JavaScript event: function createEvent(typeOfEvent) { var event = document.createEvent("CustomEvent"); event.initCustomEvent(typeOfEvent, true, true, null); event.dataTransfer = { data: {}, setData: function (key, value) { this.data[key] = value; }, getData: function (key) { return this.data[key]; } }; return event; } We then need to write a function that will fire events that we have created. This also allows you to pass in the dataTransfer value set on an element. We need this to keep track of the element that we are dragging: function dispatchEvent(element, event, transferData) { if (transferData !== undefined) { event.dataTransfer = transferData; } if (element.dispatchEvent) { element.dispatchEvent(event); } else if (element.fireEvent) { element.fireEvent("on" + event.type, event); } } Finally, we need something that will use these two functions to simulate the drag-and-drop action: function simulateHTML5DragAndDrop(element, target) { var dragStartEvent = createEvent('dragstart'); dispatchEvent(element, dragStartEvent); var dropEvent = createEvent('drop'); dispatchEvent(target, dropEvent, dragStartEvent.dataTransfer); var dragEndEvent = createEvent('dragend'); dispatchEvent(element, dragEndEvent, dropEvent.dataTransfer); } Note that the simulateHTML5DragAndDrop function needs us to pass in two elements—the element that we want to drag, and the element that we want to drag it to. It's always a good idea to try out your JavaScript in a browser first. You can copy the preceding functions into the JavaScript console in a modern browser and then try using them to make sure that they work as expected. If things go wrong in your Selenium test, you then know that it is most likely an error invoking it via the JavascriptExecutor rather than a bad piece of JavaScript. We now need to take these scripts and put them into a JavascriptExecutor along with something that will call the simulateHTML5DragAndDrop function: private void simulateDragAndDrop(WebElement elementToDrag, WebElement target) throws Exception { WebDriver driver = getDriver(); JavascriptExecutor js = (JavascriptExecutor) driver; js.executeScript("function createEvent(typeOfEvent) {n" + "var event = document.createEvent("CustomEvent");n" + "event.initCustomEvent(typeOfEvent, true, true, null);n" + " event.dataTransfer = {n" + " data: {},n" + " setData: function (key, value) {n" + " this.data[key] = value;n" + " },n" + " getData: function (key) {n" + " return this.data[key];n" + " }n" + " };n" + " return event;n" + "}n" + "n" + "function dispatchEvent(element, event, transferData) {n" + " if (transferData !== undefined) {n" + " event.dataTransfer = transferData;n" + " }n" + " if (element.dispatchEvent) {n" + " element.dispatchEvent(event);n" + " } else if (element.fireEvent) {n" + " element.fireEvent("on" + event.type, event);n" + " }n" + "}n" + "n" + "function simulateHTML5DragAndDrop(element, target) {n" + " var dragStartEvent = createEvent('dragstart');n" + " dispatchEvent(element, dragStartEvent);n" + " var dropEvent = createEvent('drop');n" + " dispatchEvent(target, dropEvent, dragStartEvent.dataTransfer);n" + " var dragEndEvent = createEvent('dragend'); n" + " dispatchEvent(element, dragEndEvent, dropEvent.dataTransfer);n" + "}n" + "n" + "var elementToDrag = arguments[0];n" + "var target = arguments[1];n" + "simulateHTML5DragAndDrop(elementToDrag, target);", elementToDrag, target); } This method is really just a wrapper around the JavaScript code. We take a driver object and cast it into a JavascriptExecutor. We then pass the JavaScript code into the executor as a string. We have made a couple of additions to the JavaScript functions that we previously wrote. Firstly, we set a couple of variables (mainly for code clarity; they can quite easily be inlined) that take the WebElements that we have passed in as arguments. Finally, we invoke the simulateHTML5DragAndDrop function using these elements. The final piece of the puzzle is to write a test that utilizes the simulateDragAndDrop method, as follows: @Test public void dragAndDropHTML5() throws Exception { WebDriver driver = getDriver(); driver.get("http://ch6.masteringselenium.com/ dragAndDrop.html"); final By destroyableBoxes = By.cssSelector("ul > li > a"); WebElement obliterator = driver.findElement(By.id("obliterate")); WebElement firstBox = driver.findElement(By.id("one")); WebElement secondBox = driver.findElement(By.id("two")); assertThat(driver.findElements(destroyableBoxes).size(), is(equalTo(5))); simulateDragAndDrop(firstBox, obliterator); assertThat(driver.findElements(destroyableBoxes). size(), is(equalTo(4))); simulateDragAndDrop(secondBox, obliterator); assertThat(driver.findElements(destroyableBoxes). size(), is(equalTo(3))); } This test finds a couple of boxes and destroys them one by one using the simulated drag and drop. As you can see, the JavascriptExcutor is extremely powerful. Can I use JavaScript libraries? The logical progression is, of course, to write your own JavaScript libraries that you can import instead of sending everything over as a string. Alternatively, maybe you would just like to import an existing library. Let's write some code that allows you to import a JavaScript library of your choice. It's not a particularly complex JavaScript. All that we are going to do is create a new <script> element in a page and then load our library into it, as follows: public void injectScript(String scriptURL) throws Exception { WebDriver driver = getDriver(); JavascriptExecutor js = (JavascriptExecutor) driver; js.executeScript("function injectScript(url) {n" + " var script = document.createElement ('script');n" + " script.src = url;n" + " var head = document.getElementsByTagName( 'head')[0];n" + " head.appendChild(script);n" + "}n" + "n" + "var scriptURL = arguments[0];n" + "injectScript(scriptURL);" , scriptURL); } We have again set arguments[0] to a variable before injecting it for clarity, but you can inline this part if you want to. All that remains now is to inject this into a page and check whether it works. Let's write a test! We are going to use this function to inject jQuery into the Google website. The first thing that we need to do is write a method that can tell us whether jQuery has been loaded or not, as follows: public Boolean isjQueryLoaded() throws Exception { WebDriver driver = getDriver(); JavascriptExecutor js = (JavascriptExecutor) driver; return (Boolean) js.executeScript("return typeof jQuery != 'undefined';"); } Now, we need to put all of this together in a test, as follows: @Test public void injectjQueryIntoGoogle() throws Exception { WebDriver driver = DriverFactory.getDriver(); driver.get("http://www.google.com"); assertThat(isjQueryLoaded(), is(equalTo(false))); injectScript("https://code.jquery.com/jquery-latest.min.js"); assertThat(isjQueryLoaded(), is(equalTo(true))); } It's a very simple test. We loaded the Google website. Then, we checked whether jQuery existed. Once we were sure that it didn't exist, we injected jQuery into the page. Finally, we again checked whether jQuery existed. We have used jQuery in our example, but you don't have to use jQuery. You can inject any script that you desire. Should I inject JavaScript libraries? It's very easy to inject JavaScript into a page, but stop and think before you do it. Adding lots of different JavaScript libraries may affect the existing functionality of the site. You may have functions in your JavaScript that overwrite existing functions that are already on the page and break the core functionality. If you are testing a site, it may make all of your tests invalid. Failures may arise because there is a clash between the scripts that you inject and the existing scripts used on the site. The flip side is also true—injecting a script may make the functionality that is broken, work. If you are going to inject scripts into an existing site, be sure that you know what the consequences are. If you are going to regularly inject a script, it may be a good idea to add some assertions to ensure that the functions that you are injecting do not already exist before you inject the script. This way, your tests will fail if the developers add a JavaScript function with the same name at some point in the future without your knowledge. What about asynchronous scripts? Everything that we have looked at so far has been a synchronous piece of JavaScript. However, what if we wanted to perform some asynchronous JavaScript calls as a part of our test? Well, we can do this. The JavascriptExecutor also has a method called executeAsyncScript(). This will allow you to run some JavaScript that does not respond instantly. Let's have a look at some examples. First of all, we are going to write a very simple bit of JavaScript that will wait for 25 seconds before triggering a callback, as follows: @Test private void javascriptExample() throws Exception { WebDriver driver = DriverFactory.getDriver(); driver.manage().timeouts().setScriptTimeout(60, TimeUnit.SECONDS); JavascriptExecutor js = (JavascriptExecutor) driver; js.executeAsyncScript("var callback = arguments[ arguments.length - 1]; window.setTimeout(callback, 25000);"); driver.get("http://www.google.com"); } Note that we defined a JavaScript variable named callback, which uses a script argument that we have not set. For asynchronous scripts, Selenium needs to have a callback defined, which is used to detect when the JavaScript that you are executing has finished. This callback object is automatically added to the end of your arguments array. This is what we have defined as the callback variable. If we now run the script, it will load our browser and then sit there for 25 seconds as it waits for the JavaScript snippet to complete and call the callback. It will then load the Google website and finish. We have also set a script timeout on the driver object that will wait for up to 60 seconds for our piece of JavaScript to execute. Let's see what happens if our script takes longer to execute than the script timeout: @Test private void javascriptExample() throws Exception { WebDriver driver = DriverFactory.getDriver(); driver.manage().timeouts().setScriptTimeout(5, TimeUnit.SECONDS); JavascriptExecutor js = (JavascriptExecutor) driver; js.executeAsyncScript("var callback = arguments[ arguments.length - 1]; window.setTimeout(callback, 25000);"); driver.get("http://www.google.com"); } This time, when we run our test, it waits for 5 seconds and then throws a TimoutException. It is important to set a script timeout on the driver object when running asynchronous scripts, to give them enough time to execute. What do you think will happen if we execute this as a normal script? @Test private void javascriptExample() throws Exception { WebDriver driver = DriverFactory.getDriver(); driver.manage().timeouts().setScriptTimeout( 5, TimeUnit.SECONDS); JavascriptExecutor js = (JavascriptExecutor) driver; js.executeScript("var callback = arguments[arguments. length - 1]; window.setTimeout(callback, 25000);"); driver.get("http://www.google.com"); } You may have been expecting an error, but that's not what you got. The script got executed as normal because Selenium was not waiting for a callback; it didn't wait for it to complete. Since Selenium did not wait for the script to complete, it didn't hit the script timeout. Hence, no error was thrown. Wait a minute. What about the callback definition? There was no argument that was used to set the callback variable. Why didn't it blow up? Well, JavaScript isn't as strict as Java. What it has done is try and work out what arguments[arguments.length - 1] would resolve and realized that it is not defined. Since it is not defined, it has set the callback variable to null. Our test then completed before setTimeout() had a chance to complete its call. So, you won't see any console errors. As you can see, it's very easy to make a small error that stops things from working when working with asynchronous JavaScript. It's also very hard to find these errors because there can be very little user feedback. Always take extra care when using the JavascriptExecutor to execute asynchronous bits of JavaScript. Summary In this article, we: Learned how to use a JavaScript executor to execute JavaScript snippets in the browser through Selenium Learned about passing arguments into a JavaScript executor and the sort of arguments that are supported Learned what the possible return types are for a JavaScript executor Gained a good understanding of when we shouldn't use a JavaScript executor Worked through a series of examples that showed ways in which we can use a JavaScript executor to enhance our tests Resources for Article: Further resources on this subject: JavaScript tech page Cross-browser Tests using Selenium WebDriver Selenium Testing Tools Learning Selenium Testing Tools with Python

 In this article by Hussein Nasser, author of the book ArcGIS By Example we will discuss the following topics: Geodatabase editing Preparing the data and project Creating excavation features Viewing and editing excavation information (For more resources related to this topic, see here.) Geodatabase editing YharanamCo is a construction contractor experienced in executing efficient and economical excavations for utility and telecom companies. When YharanamCo's board of directors heard of ArcGIS technology, they wanted to use their expertise with the power of ArcGIS to come up with a solution that helps them cut costs even more. Soil type is not the only factor in the excavation, there are many factors including the green factor where you need to preserve the trees and green area while excavating for visual appeal. Using ArcGIS, YharanamCo can determine the soil type and green factor and calculate the cost of an excavation. The excavation planning manager is the application you will be writing on top of ArcGIS. This application will help YharanamCo to create multiple designs and scenarios for a given excavation. This way they can compare the cost for each one and consider how many trees they could save by going through another excavation route. YharanamCo has provided us with the geodatabase of a soil and trees data for one of their new projects for our development. So far we learned how to view and query the geodatabase and we were able to achieve that by opening what we called a workspace. However, changing the underlying data requires establishing an editing session. All edits that are performed during an edit sessions are queued, and the moment the session is saved, these edits are committed to the geodatabase. Geodatabase editing supports atomic transactions, which are referred to as operations in the geodatabase. Atomic transaction is a list of database operations that either all occur together or none. This is to ensure consistency and integrity. After this short introduction to geodatabase editing, we will prepare our data and project. Preparing the data and project Before we dive into the coding part, we need to do some preparation for our new project and our data. Preparing the Yharnam geodatabase and map The YharnamCo team has provided us with the geodatabase and map document, so we will simply copy the necessary files to your drive. Follow these steps to start your preparation of the data and map: Copy the entire yharnam folder in the supporting to C:\ArcGISByExample\. Run yharnam.mxd under C:\ArcGISByExample\yharnam\Data\yharnam.mxd. This should point to the geodatabase, which is located under C:\ArcGISByExample\yharnam\Data\yharnam.gdb, as illustrated in the following screenshot: Note there are three types of trees: Type1, Type2, and Type3. Also note there are two types of soil: rocky and sand. Close ArcMap and choose not to save any changes. Preparing the Yharnam project We will now start our project. First we need to create our Yharnam Visual Studio extending ArcObjects project. To do so, follow these steps: From the Start menu, run Visual Studio Express 2013 as administrator. Go to the File menu and then click on New Project. Expand the Templates node | Visual Basic | ArcGIS, and then click on Extending ArcObjects. You will see the list of projects displayed on the right. Select the Class Library (ArcMap) project. In the Name field, type Yharnam, and in the location, browse to C:\ArcGISByExample\yharnam\Code. If the Code folder is not there, create it. Click on OK. In the ArcGIS Project Wizard, you will be asked to select the references libraries you will need in your project. I always recommend selecting all referencing, and then at the end of your project, remove the unused ones. So, go ahead and right-click on Desktop ArcMap and click on Select All, as shown in the following screenshot: Click on Finish to create the project. This will take a while to add all references to your project. Once your project is created, you will see that one class called Class1 is added, which we won't need, so right-click on it and choose Delete. Then, click on OK to confirm. Go to File and click on Save All. Exit the Visual Studio application. You have finished preparing your Visual Studio with extending ArcObjects support. Move to the next section to write some code. Adding the new excavation tool The new excavation tool will be used to draw a polygon on the map, which represents the geometry of the excavation. Then, this will create a corresponding excavation feature using this geometry: If necessary, open Visual Studio Express in administrator mode; we need to do this since our project is actually writing to the registry this time, so it needs administrator permissions. To do that, right-click on Visual Studio and click on Run as administrator. Go to File, then click on Open Project, browse to the Yharnam project from C:\ArcGISByExample\yharnam\Code, and click on Open. Click on the Yharnam project from Solution Explorer to activate it. From the Project menu, click on Add Class. Expand the ArcGIS node and then click on the Extending ArcObjects node. Select Base Tool and name it tlNewExcavation.vb. Click on Add to open ArcGIS New Item Wizard Options. From ArcGIS New Item Wizard Options, select Desktop ArcMap tool since we will be programming against ArcMap. Click on OK. Take note of the Yharnam.tlNewExcavation Progid as we will be using this in the next section to add the tool to the toolbar. If necessary, double-click on tlNewExcavation.vb to edit it. In the New method, update the properties of the command as follows. This will update the name and caption and other properties of the command. There is a piece of code that loads the command icon. Leave that unchanged: Public Sub New() MyBase.New() ' TODO: Define values for the public properties ' TODO: Define values for the public properties MyBase.m_category = "Yharnam" 'localizable text MyBase.m_caption = "New Excavation" 'localizable text MyBase.m_message = "New Excavation" 'localizable text MyBase.m_toolTip = "New Excavation" 'localizable text MyBase.m_name = "Yharnam_NewExcavation" Try 'TODO: change resource name if necessary Dim bitmapResourceName As String = Me.GetType().Name + ".bmp" MyBase.m_bitmap = New Bitmap(Me.GetType(), bitmapResourceName) MyBase.m_cursor = New System.Windows.Forms.Cursor(Me.GetType(), Me.GetType().Name + ".cur") Catch ex As Exception System.Diagnostics.Trace.WriteLine(ex.Message, "Invalid Bitmap") End Try End Sub Adding the excavation editor tool The excavation editor is a tool that will let us click an excavation feature on the map and display the excavation information such as depth, area, and so on. It will also allow us to edit some of the information. We will now add a tool to our project. To do that, follow these steps: If necessary, open Visual Studio Express in administrator mode; we need to do this since our project is actually writing to the registry this time, so it needs administrator permissions. To do that, right-click on Visual Studio and click on Run as administrator. Go to File, then click on Open Project, browse to the Yharnam project from C:\ArcGISByExample\yharnam\Code, and click on Open. Click on the Yharnam project from Solution Explorer to activate it. From the Project menu, click on Add Class. Expand the ArcGIS node and then click on the Extending ArcObjects node. Select Base Tool and name it tlExcavationEditor.vb. Click on Add to open ArcGIS New Item Wizard Options. From ArcGIS New Item Wizard Options, select Desktop ArcMap Tool since we will be programming against ArcMap. Click on OK. Take note of the Yharnam.tlExcavationEditor Progid as we will be using this in the next section to add the tool to the toolbar. If necessary, double-click on tlExcavationEditor.vb to edit it. In the New method, update the properties of the command as follows. This will update the name and caption and other properties of the command: MyBase.m_category = "Yharnam" 'localizable text MyBase.m_caption = "Excavation Editor" 'localizable text MyBase.m_message = "Excavation Editor" 'localizable text MyBase.m_toolTip = "Excavation Editor" 'localizable text MyBase.m_name = "Yharnam_ExcavationEditor" In order to display the excavation information, we will need a form. To add the Yharnam Excavation Editor form, point to Project and then click on Add Windows Form. Name the form frmExcavationEditor.vb and click on Add. Use the form designer to add and set the controls shown in the following table: Control Name Properties Label lblDesignID Text: Design ID Label lblExcavationOID Text: Excavation ObjectID Label lblExcavationArea Text: Excavation Area Label lblExcavationDepth Text: Excavation Depth Label lblTreeCount Text: Number of Trees Label lblTotalCost Text: Total Excavation Cost Text txtDesignID Read-Only: True Text txtExcavationOID Read-Only: True Text txtExcavationArea Read-Only: True Text txtExcavationDepth Read-Only: False Text txtTreeCount Read-Only: True Text txtTotalCost Read-Only: True Button btnSave Text: Save Your form should look like the following screenshot: One last change before we build our solution: we need to change the default icons of our tools. To do that, double-click on tlExcavationEditor.bmp to open the picture editor and replace the picture with yharnam.bmp, which can be found under C:\ArcGISByExample\yharnam\icons\excavation_editor.bmp, and save tlExcavationEditor.bmp. Change tlNewExcavation.bmp to C:\ArcGISByExample\yharnam\icons\new_excavation.bmp. Save your project and move to the next step to assign the command to the toolbar. Adding the excavation manager toolbar Now that we have our two tools, we will add a toolbar to group them together. Follow these steps to add the Yharnam Excavation Planning Manager Toolbar to your project: If necessary, open Visual Studio Express in administrator mode; we need to do this since our project is actually writing to the registry this time, so it needs administrator permissions. To do that, right-click on Visual Studio and click on Run as administrator. Go to File, then click on Open Project, browse to the Yharnam project from C:\ArcGISByExample\yharnam\Code, and click on Open. Click on the Yharnam project from Solution Explorer to activate it. From the Project menu, click on Add Class. Expand the ArcGIS node and then click on the Extending ArcObjects node. Select Base Toolbar and name it tbYharnam.vb. Click on Add to open ArcGIS New Item Wizard Options. From ArcGIS New Item Wizard Options, select Desktop ArcMap since we will be programming against ArcMap. Click on OK. The property Caption is what is displayed when the toolbar loads. It currently defaults to MY VB.Net Toolbar, so change it to Yharnam Excavation Planning Manager Toolbar as follows: Public Overrides ReadOnly Property Caption() As String Get 'TODO: Replace bar caption Return "YharnamExcavation Planning Manager" End Get End Property Your toolbar is currently empty, which means it doesn't have buttons or tools. Go to the New method and add your tools prog ID, as shown in the following code: Public Sub New() AddItem("Yharnam.tlNewExcavation") AddItem("Yharnam.tlExcavationEditor") End Sub Now it is time to test our new toolbar. Go to Build and then click on Build Solution; make sure ArcMap is not running. If you get an error, make sure you have run the Visual Studio as administrator. For a list of all ArcMap commands, you can refer to http://bit.ly/b04748_arcmapids. Check the commands with namespace esriArcMapUI. Run yharnam.mxd in C:\ArcGISByExample\yharnam\Data\yharnam.mxd. From ArcMap, go to the Customize menu, then to Toolbars, and then select Yharnam Excavation Planning Manager Toolbar, which we just created. You should see the toolbar pop up on ArcMap with the two added commands, as shown in the following screenshot: Close ArcMap and choose not to save any changes. In the next section, we will do the real work of editing. Creating excavation features Features are nothing but records in a table. However, these special records cannot be created without a geometry shape attribute. To create a feature, we need first to learn how to draw and create geometries. We will be using the rubber band ArcObjects interface to create a polygon geometry. We can use it to create other types of geometries as well, but since our excavations are polygons, we will use the polygon rubber band. Using the rubber band to draw geometries on the map In this exercise, we will use the rubber band object to create a polygon geometry by clicking on multiple points on the map. We will import libraries as we need them. Follow these steps: If necessary, open Visual Studio Express in administrator mode; we need to do this since our project is actually writing to the registry this time, so it needs administrator permissions. To do that, right-click on Visual Studio and click on Run as administrator. Go to File, then click on Open Project, browse to the Yharnam project from C:\ArcGISByExample\yharnam\Code, and click on Open. Double-click on tlNewExcavation.vb and write the following code in onMouseDown that is when we click on the map: Public Overrides Sub OnMouseDown(ByVal Button As Integer, ByVal Shift As Integer, ByVal X As Integer, ByVal Y As Integer) Dim pRubberBand As IRubberBand = New RubberPolygonClass End Sub You will get an error under rubber band and that is because the library that this class is located in is not imported. Simply hover over the error and import the library; in this case, it is Esri.ArcGIS.Display, as illustrated in the following screenshot: We now have to call the TrackNew method on the pRubberband object that will allow us to draw. This requires two pieces of parameters. First, the screen on which you are drawing and the symbol you are drawing with, which have the color, size, and so on. By now we are familiar with how we can get these objects. The symbol needs to be of type FillShapeSymbol since we are dealing with polygons. We will go with a simple black symbol for starters. Write the following code: Public Overrides Sub OnMouseDown(ByVal Button As Integer, ByVal Shift As Integer, ByVal X As Integer, ByVal Y As Integer) Dim pDocument As IMxDocument = m_application.Document Dim pRubberBand As IRubberBand = New RubberPolygonClass Dim pFillSymbol As ISimpleFillSymbol = New SimpleFillSymbol Dim pPolygon as IGeometry=pRubberBand.TrackNew(pDocument.ActiveView.ScreenDisplay, pFillSymbol) End Sub Build your solution. If it fails, make sure you have run the solution as administrator. Run yharnam.mxd. Click on the New Excavation tool to activate it, and then click on three different locations on the map. You will see that a polygon is forming as you click; double-click to finish drawing, as illustrated in the following screenshot: The polygon disappears when you finish drawing and the reason is that we didn't actually persist the polygon into a feature or a graphic. As a start, we will draw the polygon on the screen and we will also change the color of the polygon to red. Write the following code to do so: Public Overrides Sub OnMouseDown(ByVal Button As Integer, ByVal Shift As Integer, ByVal X As Integer, ByVal Y As Integer) Dim pDocument As IMxDocument = m_application.Document Dim pRubberBand As IRubberBand = New RubberPolygonClass Dim pFillSymbol As ISimpleFillSymbol = New SimpleFillSymbol Dim pColor As IColor = New RgbColor pColor.RGB = RGB(255, 0, 0) pFillSymbol.Color = pColor Dim pPolygon As IGeometry = pRubberBand.TrackNew(pDocument.ActiveView.ScreenDisplay, pFillSymbol) Dim pDisplay As IScreenDisplay = pDocument.ActiveView.ScreenDisplay pDisplay.StartDrawing(pDisplay.hDC, ESRI.ArcGIS.Display.esriScreenCache.esriNoScreenCache) pDisplay.SetSymbol(pFillSymbol) pDisplay.DrawPolygon(pPolygon) pDisplay.FinishDrawing() End Sub Build your solution and run yharnam.mxd. Activate the New Excavation tool and draw an excavation; you should see a red polygon is displayed on the screen after you finish drawing, as shown in the following screenshot: Close ArcMap and choose not to save the changes. Converting geometries into features Now that we have learned how to draw a polygon, we will convert that polygon into an excavation feature. Follow these steps to do so: If necessary, open Visual Studio Express in administrator mode; we need to do this since our project is actually writing to the registry this time, so it needs administrator permissions. To do that, right-click on Visual Studio and click on Run as administrator. Go to File, then click on Open Project, browse to the Yharnam project from C:\ArcGISByExample\yharnam\Code, and click on Open. Double-click on tlNewExcavation.vb to edit the code. Remove the code that draws the polygon on the map. Your new code should look like the following: Public Overrides Sub OnMouseDown(ByVal Button As Integer, ByVal Shift As Integer, ByVal X As Integer, ByVal Y As Integer) Dim pDocument As IMxDocument = m_application.Document Dim pRubberBand As IRubberBand = New RubberPolygonClass Dim pFillSymbol As ISimpleFillSymbol = New SimpleFillSymbol Dim pPolygon As IGeometry = pRubberBand.TrackNew(pDocument.ActiveView.ScreenDisplay, pFillSymbol) End Sub First we need to open the Yharnam geodatabase located on C:\ArcGISByExample\yharnam\Data\Yharnam.gdb by establishing a workspace connection and then we will open the Excavation feature class. Write the following two functions in tlNewExcavation.vb, getYharnamWorkspace, and getExcavationFeatureclass: Public Function getYharnamWorkspace() As IWorkspace Dim pWorkspaceFactory As IWorkspaceFactory = New FileGDBWorkspaceFactory Return pWorkspaceFactory.OpenFromFile("C:\ArcGISByExample\yharnam\Data\Yharnam.gdb", m_application.hWnd) End Function Public Function getExcavationFeatureClass(pWorkspace As IWorkspace) As IFeatureClass Dim pFWorkspace As IFeatureWorkspace = pWorkspace Return pFWorkspace.OpenFeatureClass("Excavation") End Function To create the feature, we need first to start an editing session and a transaction, and wrap and code between the start and the end of the session. To use editing, we utilize the IWorkspaceEdit interface. Write the following code in the onMouseDown method: Public Overrides Sub OnMouseDown(ByVal Button As Integer, ByVal Shift As Integer, ByVal X As Integer, ByVal Y As Integer) Dim pDocument As IMxDocument = m_application.Document Dim pRubberBand As IRubberBand = New RubberPolygonClass Dim pFillSymbol As ISimpleFillSymbol = New SimpleFillSymbol Dim pPolygon As IGeometry = pRubberBand.TrackNew(pDocument.ActiveView.ScreenDisplay, pFillSymbol) Dim pWorkspaceEdit As IWorkspaceEdit = getYharnamWorkspace() pWorkspaceEdit.StartEditing(True) pWorkspaceEdit.StartEditOperation() pWorkspaceEdit.StopEditOperation() pWorkspaceEdit.StopEditing(True) End Sub Now we will use the CreateFeature method in order to create a new feature and then populate it with the attributes. The only attribute that we care about now is the geometry or the shape. The shape is actually the polygon we just drew. Write the following code to create the feature: Dim pWorkspaceEdit As IWorkspaceEdit = getYharnamWorkspace() pWorkspaceEdit.StartEditing(True) pWorkspaceEdit.StartEditOperation() Dim pExcavationFeatureClass As IFeatureClass = getExcavationFeatureClass(pWorkspaceEdit) Dim pFeature As IFeature = pExcavationFeatureClass.CreateFeature() pFeature.Shape = pPolygon pFeature.Store() pWorkspaceEdit.StopEditOperation() pWorkspaceEdit.StopEditing(True) Build and run yharnam.mxd. Click on the New Excavation tool and draw a polygon on the map. Refresh the map. You will see that a new excavation feature is added to the map, as shown in the following screenshot: Close ArcMap and choose not to save any changes. Reopen yharnam.mxd and you will see that the features you created are still there because they are stored in the geodatabase. Close ArcMap and choose not to save any changes. We have learned how to create features. Now we will learn how to edit excavations as well. View and edit the excavation information We have created some excavation features on the map; however, these are merely polygons and we need to extract useful information from them, and display and edit these excavations. For that, we will use the Excavation Editor tool to click on an excavation and display the Excavation Editor form with the excavation information. Then we will give the ability to edit this information. Follow these steps: If necessary, open Visual Studio Express in administrator mode; we need to do this since our project is actually writing to the registry this time, so it needs administrator permissions. To do that, right-click on Visual Studio and click on Run as administrator. Go to File, then click on Open Project, browse to the Yharnam project from C:\ArcGISByExample\yharnam\Code, and click on Open. Right-click on frmExcavationEditor.vb and click on View Code to view the class code. Add the ArcMapApplication property as shown in the following code so that we can set it from the tool, we will need this at a later stage: Private _application As IApplication Public Property ArcMapApplication() As IApplication Get Return _application End Get Set(ByVal value As IApplication) _application = value End Set End Property Add another method called PopulateExcavation that takes a feature. This method will populate the form fields with the information we get from the excavation feature. We will pass the feature from the Excavation Editor tool at a later stage: Public Sub PopulateExcavation(pFeature As IFeature) End Sub According to the following screenshot of the Excavation feature class from ArcCatalog, we can populate three fields from the excavation attributes, these are the design ID, the depth of excavation, and the object ID of the feature: Write the following code to populate the design ID, the depth, and the object ID. Note that we used the isDBNull function to check if there is any value stored in those fields. Note that we don't have to do that check for the OBJECTID field since it should never be null: Public Sub PopulateExcavation(pFeature As IFeature) Dim designID As Long = 0 Dim dDepth As Double = 0 If IsDBNull(pFeature.Value(pFeature.Fields.FindField("DESIGNID"))) = False Then designID = pFeature.Value(pFeature.Fields.FindField("DESIGNID")) End If If IsDBNull(pFeature.Value(pFeature.Fields.FindField("DEPTH"))) = False Then dDepth = pFeature.Value(pFeature.Fields.FindField("DEPTH")) End If txtDesignID.Text = designID txtExcavationDepth.Text = dDepth txtExcavationOID.Text= pFeature.OID End Sub What left is the excavation area, which is a bit tricky. To do that, we need to get it from the Shape property of the feature by casting it to the IAreaarcobjects interface and use the area property as follows: ….. txtExcavationOID.Text = pFeature.OID Dim pArea As IArea = pFeature.Shape txtExcavationArea.Text = pArea.Area End Sub Now our viewing capability is ready, we need to execute it. Double-click on tlExcavationEditor.vb in order to open the code. We will need the getYharnamWorkspace and getExcavationFeatureClass methods that are in tlNewExcavation.vb. Copy them in tlExcavationEditor.vb. In the onMouseDown event, write the following code to get the feature from the mouse location. This will convert the x, y mouse coordinate into a map point and then does a spatial query to find the excavation under this point. After that, we will basically call our excavation editor form and send it the feature to do the work as follows: Public Overrides Sub OnMouseDown(ByVal Button As Integer, ByVal Shift As Integer, ByVal X As Integer, ByVal Y As Integer) 'TODO: Add tlExcavationEditor.OnMouseDown implementation Dim pMxdoc As IMxDocument = m_application.Document Dim pPoint As IPoint = pMxdoc.ActiveView.ScreenDisplay.DisplayTransformation.ToMapPoint(X, Y) Dim pSFilter As ISpatialFilter = New SpatialFilter pSFilter.Geometry = pPoint Dim pFeatureClass As IFeatureClass = getExcavationFeatureClass(getYharnamWorkspace()) Dim pFCursor As IFeatureCursor = pFeatureClass.Search(pSFilter, False) Dim pFeature As IFeature = pFCursor.NextFeature If pFeatureIs Nothing Then Return Dim pExcavationEditor As New frmExcavationEditor pExcavationEditor.ArcMapApplication = m_application pExcavationEditor.PopulateExcavation(pFeature) pExcavationEditor.Show() End Sub Build and run yharnam.mxd. Click on the Excavation Editor tool and click on one of the excavations you drew before. You should see that the Excavation Editor form pops up with the excavation information; no design ID or depth is currently set, as you can see in the following screenshot: Close ArcMap and choose not to save any changes. We will do the final trick to edit the excavation; there is not much to edit here, only the depth. To do that, copy the getYharnamWorkspace and getExcavationFeatureClass methods that are in tlNewExcavation.vb. Copy them in frmExcavationEditor.vb. You will get an error in the m_application.hwnd, so replace it with _application.hwnd, which is the property we set. Right-click on frmExcavationEditor and select View Designer. Double-click on the Save button to generate the btnSave_Click method. The user will enter the new depth for the excavation in the txtExcavationDepthtextbox. We will use this value and store it in the feature. But before that, we need to retrieve that feature using the object ID, start editing, save the feature, and close the session. Write the following code to do so. Note that we have closed the form at the end of the code, so we can open it again to get the new value: Private Sub btnSave_Click(sender As Object, e As EventArgs) Handles btnSave.Click Dim pWorkspaceEdit As IWorkspaceEdit = getYharnamWorkspace() Dim pFeatureClass As IFeatureClass = getExcavationFeatureClass(pWorkspaceEdit) Dim pFeature As IFeature = pFeatureClass.GetFeature(txtExcavationOID.Text) pWorkspaceEdit.StartEditing(True) pWorkspaceEdit.StartEditOperation() pFeature.Value(pFeature.Fields.FindField("DEPTH")) = txtExcavationDepth.Text pFeature.Store() pWorkspaceEdit.StopEditOperation() pWorkspaceEdit.StopEditing(True) Me.Close End Sub Build and run yharnam.mxd. Click on the Excavation Editor tool and click on one of the excavations you drew before. Type a numeric depth value and click on Save; this will close the form. Use the Excavation Editor tool again to open back the excavation and check if your depth value has been stored successfully. Summary In this article, you started writing the excavation planning manager, code named Yharnam. In the first part of the article, you spent time learning to use the geodatabase editing and preparing the project. You then learned how to use the rubber band tool, which allows you to draw geometries on the map. Using this drawn geometry, you edited the workspace and created a new excavation feature with that geometry. You then learned how to view and edit the excavation feature with attributes. Resources for Article: Further resources on this subject: ArcGIS Spatial Analyst[article] Enterprise Geodatabase[article] Server Logs [article]

 In the following article by Austin Scott, the author of Learning RSLogix 5000 Programming, you will be introduced to the high performance, asynchronous nature of the Logix family of controllers and the requirement for the buffering I/O module data it drives. You will learn various techniques for the buffering I/O module values in RSLogix 5000 and Studio 5000 Logix Designer. You will also learn about the IEC Languages that do not require the input or output module buffering techniques to be applied to them. In order to understand the need for buffering, let's start by exploring the evolution of the modern line of Rockwell Automation Controllers. (For more resources related to this topic, see here.) ControlLogix controllers The ControlLogix controller was first launched in 1997 as a replacement for Allen Bradley's previous large-scale control platform. The PLC-5. ControlLogix represented a significant technological step forward, which included a 32-bit ARM-6 RISC-core microprocessor and the ABrisc Boolean processor combined with a bus interface on the same silicon chip. At launch, the Series 5 ControlLogix controllers (also referred to as L5 and ControlLogix 5550, which has now been superseded by the L6 and L7 series controllers) were able to execute code three times faster than PLC-5. The following is an illustration of the original ControlLogix L5 Controller: ControlLogix Logix L5 Controller The L5 controller is considered to be a PAC (Programmable Automation Controller) rather than a traditional PLC (Programmable Logic Controller), due to its modern design, power, and capabilities beyond a traditional PLC (such as motion control, advanced networking, batching and sequential control). ControlLogix represented a significant technological step forward for Rockwell Automation, but this new technology also presented new challenges for automation professionals. ControlLogix was built using a modern asynchronous operating model rather than the more traditional synchronous model used by all the previous generations of controllers. The asynchronous operating model requires a different approach to real-time software development in RSLogix 5000 (now known in version 20 and higher as Studio 5000 Logix Designer). Logix operating cycle The entire Logix family of controllers (ControlLogix and CompactLogix) have diverged from the traditional synchronous PLC scan architecture in favor of a more efficient asynchronous operation. Like most modern computer systems, asynchronous operation allows the Logix controller to handle multiple Tasks at the same time by slicing the processing time between each task. The continuous update of information in an asynchronous processor creates some programming challenges, which we will explore in this article. The following diagram illustrates the difference between synchronous and asynchronous operation. Synchronous versus Asynchronous Processor Operation Addressing module I/O data Individual channels on a module can be referenced in your Logix Designer / RSLogix 5000 programs using it's address. An address gives the controller directions to where it can find a particular piece of information about a channel on one of your modules. The following diagram illustrates the components of an address in RsLogix 5000 or Studio 5000 Logix Designer: The components of an I/O Module Address in Logix Module I/O tags can be viewed using the Controller Tags window, as the following screen shot illustrates. I/O Module Tags in Studio 5000 Logix Designer Controller Tags Window Using the module I/O tags, input and output module data can be directly accessed anywhere within a logic routine. However, it is recommended that we buffer module I/O data before we evaluate it in Logic. Otherwise, due to the asynchronous tag value updates in our I/O modules, the state of our process values could change part way through logic execution, thus creating unpredictable results. In the next section, we will introduce the concept of module I/O data buffering. Buffering module I/O data In the olden days of PLC5s and SLC500s, before we had access to high-performance asynchronous controllers like the ControlLogix, SoftLogix and CompactLogix families, program execution was sequential (synchronous) and very predictable. In asynchronous controllers, there are many activities happening at the same time. Input and output values can change in the middle of a program scan and put the program in an unpredictable state. Imagine a program starting a pump in one line of code and closing a valve directly in front of that pump in the next line of code, because it detected a change in process conditions. In order to address this issue, we use a technique call buffering and, depending on the version of Logix you are developing on, there are a few different methods of achieving this. Buffering is a technique where the program code does not directly access the real input or output tags on the modules during the execution of a program. Instead, the input and output module tags are copied at the beginning of a programs scan to a set of base tags that will not change state during the program's execution. Think of buffering as taking a snapshot of the process conditions and making decisions on those static values rather than live values that are fluctuating every millisecond. Today, there is a rule in most automation companies that require programmers to write code that "Buffers I/O" data to base tags that will not change during a programs execution. The two widely accepted methods of buffering are: Buffering to base tags Program parameter buffering (only available in the Logix version 24 and higher) Do not underestimate the importance of buffering a program's I/O. I worked on an expansion project for a process control system where the original programmers had failed to implement buffering. Once a month, the process would land in a strange state, which the program could not recover from. The operators had attributed these problem to "Gremlins" for years, until I identified and corrected the issue. Buffering to base tags Logic can be organized into manageable pieces and executed based on different intervals and conditions. The buffering to base tags practice takes advantage of Logix's ability to organize code into routines. The default ladder logic routine that is created in every new Logix project is called MainRoutine. The recommended best practice for buffering tags in ladder logic is to create three routines: One for reading input values and buffering them One for executing logic One for writing the output values from the buffered values The following ladder logic excerpt is from MainRoutine of a program that implements Input and Output Buffering: MainRoutine Ladder Logic Routine with Input and Output Buffering Subroutine Calls The following ladder logic is taken from the BufferInputs routine and demonstrates the buffering of digital input module tag values to Local tags prior to executing our PumpControl routine: Ladder Logic Routine with Input Module Buffering After our input module values have been buffered to Local tags, we can execute our processlogic in our PumpControl routine without having to worry about our values changing in the middle of the routine's execution. The following ladder logic code determines whether all the conditions are met to run a pump: Pump Control Ladder Logic Routine Finally, after all of our Process Logic has finished executing, we can write the resulting values to our digital output modules. The following ladder logic BufferOutputs, routine copies the resulting RunPump value to the digital output module tag. Ladder Logic Routine with Output Module Buffering We have now buffered our module inputs and module outputs in order to ensure they do not change in the middle of a program execution and potentially put our process into an undesired state. Buffering Structured Text I/O module values Just like ladder logic, Structured Text I/O module values should be buffered at the beginning of a routine or prior to executing a routine in order to prevent the values from changing mid-execution and putting the process into a state you could not have predicted. Following is an example of the ladder logic buffering routines written in Structured Text (ST)and using the non-retentive assignment operator: (* I/O Buffering in Structured Text Input Buffering *) StartPump [:=] Local:2:I.Data[0].0; HighPressure [:=] Local:2:I.Data[0].1; PumpStartManualOverride [:=] Local:2:I.Data[0].2; (* I/O Buffering in Structured Text Output Buffering *) Local:3:O.Data[0].0 [:=] RunPump; Function Block Diagram (FBD) and Sequential Function Chart (SFC) I/O module buffering Within Rockwell Automation's Logix platform, all of the supported IEC languages (ladder logic, structured text, function block, and sequential function chart) will compile down to the same controller bytecode language. The available functions and development interface in the various Logix programming languages are vastly different. Function Block Diagrams (FBD) and Sequential Function Charts (SFC) will always automatically buffer input values prior to executing Logic. Once a Function Block Diagram or a Sequential Function Chart has completed the execution of their logic, they will write all Output Module values at the same time. There is no need to perform Buffering on FBD or SFC routines, as it is automatically handled. Buffering using program parameters A program parameter is a powerful new feature in Logix that allows the association of dynamic values to tags and programs as parameters. The importance of program parameters is clear by the way they permeate the user interface in newer versions of Logix Designer (version 24 and higher). Program parameters are extremely powerful, but the key benefit to us for using them is that they are automatically buffered. This means that we could have effectively created the same result in one ladder logic rung rather than the eight we created in the previous exercise. There are four types of program parameters: Input: This program parameter is automatically buffered and passed into a program on each scan cycle. Output: This program parameter is automatically updated at the end of a program (as a result of executing that program) on each scan cycle. It is similar to the way we buffered our output module value in the previous exercise. InOut: This program parameter is updated at the start of the program scan and the end of the program scan. It is also important to note that, unlike the input and output parameters; the InOut parameter is passed as a pointer in memory. A pointer shares a piece of memory with other processes rather than creating a copy of it, which means that it is possible for an InOut parameter to change its value in the middle of a program scan. This makes InOut program parameters unsuitable for buffering when used on their own. Public: This program parameterbehaves like a normal controller tag and can be connected to input, output, and InOut parameters. it is similar to the InOut parameter, public parameters that are updated globally as their values are changed. This makes program parameters unsuitable for buffering, when used on their own. Primarily public program parameters are used for passing large data structures between programs on a controller. In Logix Designer version 24 and higher, a program parameter can be associated with a local tag using the Parameters and Local Tags in the Control Organizer (formally called "Program Tags"). The module input channel can be associated with a base tag within your program scope using the Parameter Connections. Add the module input value as a parameter connection. The previous screenshot demonstrates how we would associate the input module channel with our StartPump base tag using the Parameter Connection value. Summary In this article, we explored the asynchronous nature of the Logix family of controllers. We learned the importance of buffering input module and output module values for ladder logic routines and structured text routines. We also learned that, due to the way Function Block Diagrams (FBD/FB) and Sequential Function Chart (SFC) Routines execute, there is no need to buffer input module or output module tag values. Finally, we introduced the concept of buffering tags using program parameters in version 24 and high of Studio 5000 Logix Designer. Resources for Article: Further resources on this subject: vROps – Introduction and Architecture[article] Ensuring Five-star Rating in the MarketPlace[article] Building Ladder Diagram programs (Simple) [article]

In this article by Narayan Prusty, author of Learning ECMAScript 6, you will learn how ES6 introduces classes that provide a much simpler and clearer syntax to creating constructors and dealing with inheritance. JavaScript never had the concept of classes, although it's an object-oriented programming language. Programmers from the other programming language background often found it difficult to understand JavaScript's object-oriented model and inheritance due to lack of classes. In this article, we will learn about the object-oriented JavaScript using the ES6 classes: Creating objects the classical way What are classes in ES6 Creating objects using classes The inheritance in classes The features of classes (For more resources related to this topic, see here.) Understanding the Object-oriented JavaScript Before we proceed with the ES6 classes, let's refresh our knowledge on the JavaScript data types, constructors, and inheritance. While learning classes, we will be comparing the syntax of the constructors and prototype-based inheritance with the syntax of the classes. Therefore, it is important to have a good grip on these topics. Creating objects There are two ways of creating an object in JavaScript, that is, using the object literal, or using a constructor. The object literal is used when we want to create fixed objects, whereas constructor is used when we want to create the objects dynamically on runtime. Let's consider a case where we may need to use the constructors instead of the object literal. Here is a code example: var student = { name: "Eden", printName: function(){ console.log(this.name); } } student.printName(); //Output "Eden" Here, we created a student object using the object literal, that is, the {} notation. This works well when you just want to create a single student object. But the problem arises when you want to create multiple student objects. Obviously, you don't want to write the previous code multiple times to create multiple student objects. This is where constructors come into use. A function acts like a constructor when invoked using the new keyword. A constructor creates and returns an object. The this keyword, inside a function, when invoked as a constructor, points to the new object instance, and once the constructor execution is finished, the new object is automatically returned. Consider this example: function Student(name) { this.name = name; } Student.prototype.printName = function(){ console.log(this.name); } var student1 = new Student("Eden"); var student2 = new Student("John"); student1.printName(); //Output "Eden" student2.printName(); //Output "John" Here, to create multiple student objects, we invoked the constructor multiple times instead of creating multiple student objects using the object literals. To add methods to the instances of the constructor, we didn't use the this keyword, instead we used the prototype property of constructor. We will learn more on why we did it this way, and what the prototype property is, in the next section. Actually, every object must belong to a constructor. Every object has an inherited property named constructor, pointing to the object's constructor. When we create objects using the object literal, the constructor property points to the global Object constructor. Consider this example to understand this behavior: var student = {} console.log(student.constructor == Object); //Output "true" Understanding inheritance Each JavaScript object has an internal [[prototype]] property pointing to another object called as its prototype. This prototype object has a prototype of its own, and so on until an object is reached with null as its prototype. null has no prototype, and it acts as a final link in the prototype chain. When trying to access a property of an object, and if the property is not found in the object, then the property is searched in the object's prototype. If still not found, then it's searched in the prototype of the prototype object. It keeps on going until null is encountered in the prototype chain. This is how inheritance works in JavaScript. As a JavaScript object can have only one prototype, JavaScript supports only a single inheritance. While creating objects using the object literal, we can use the special __proto__ property or the Object.setPrototypeOf() method to assign a prototype of an object. JavaScript also provides an Object.create() method, with which we can create a new object with a specified prototype as the __proto__ lacked browser support, and the Object.setPrototypeOf() method seemed a little odd. Here is code example that demonstrates different ways to set the prototype of an object while creating, using the object literal: var object1 = { name: "Eden", __proto__: {age: 24} } var object2 = {name: "Eden"} Object.setPrototypeOf(object2, {age: 24}); var object3 = Object.create({age: 24}, {name: {value: "Eden"}}); console.log(object1.name + " " + object1.age); console.log(object2.name + " " + object2.age); console.log(object3.name + " " + object3.age); The output is as follows: Eden 24 Eden 24 Eden 24 Here, the {age:24} object is referred as base object, superobject, or parent object as its being inherited. And the {name:"Eden"} object is referred as the derived object, subobject, or the child object, as it inherits another object. If you don't assign a prototype to an object while creating it using the object literal, then the prototype points to the Object.prototype property. The prototype of Object.prototype is null therefore, leading to the end of the prototype chain. Here is an example to demonstrate this: var obj = { name: "Eden" } console.log(obj.__proto__ == Object.prototype); //Output "true" While creating objects using a constructor, the prototype of the new objects always points to a property named prototype of the function object. By default, the prototype property is an object with one property named as constructor. The constructor property points to the function itself. Consider this example to understand this model: function Student() { this.name = "Eden"; } var obj = new Student(); console.log(obj.__proto__.constructor == Student); //Output "true" console.log(obj.__proto__ == Student.prototype); //Output "true" To add new methods to the instances of a constructor, we should add them to the prototype property of the constructor, as we did earlier. We shouldn't add methods using the this keyword in a constructor body, because every instance of the constructor will have a copy of the methods, and this isn't very memory efficient. By attaching methods to the prototype property of a constructor, there is only one copy of each function that all the instances share. To understand this, consider this example: function Student(name) { this.name = name; } Student.prototype.printName = function(){ console.log(this.name); } var s1 = new Student("Eden"); var s2 = new Student("John"); function School(name) { this.name = name; this.printName = function(){ console.log(this.name); } } var s3 = new School("ABC"); var s4 = new School("XYZ"); console.log(s1.printName == s2.printName); console.log(s3.printName == s4.printName); The output is as follows: true false Here, s1 and s2 share the same printName function that reduces the use of memory, whereas s3 and s4 contain two different functions with the name as printName that makes the program use more memory. This is unnecessary, as both the functions do the same thing. Therefore, we add methods for the instances to the prototype property of the constructor. Implementing the inheritance hierarchy in the constructors is not as straightforward as we did for object literals. Because the child constructor needs to invoke the parent constructor for the parent constructor's initialization logic to take place and we need to add the methods of the prototype property of the parent constructor to the prototype property of the child constructor, so that we can use them with the objects of child constructor. There is no predefined way to do all this. The developers and JavaScript libraries have their own ways of doing this. I will show you the most common way of doing it. Here is an example to demonstrate how to implement the inheritance while creating the objects using the constructors: function School(schoolName) { this.schoolName = schoolName; } School.prototype.printSchoolName = function(){ console.log(this.schoolName); } function Student(studentName, schoolName) { this.studentName = studentName; School.call(this, schoolName); } Student.prototype = new School(); Student.prototype.printStudentName = function(){ console.log(this.studentName); } var s = new Student("Eden", "ABC School"); s.printStudentName(); s.printSchoolName(); The output is as follows: Eden ABC School Here, we invoked the parent constructor using the call method of the function object. To inherit the methods, we created an instance of the parent constructor, and assigned it to the child constructor's prototype property. This is not a foolproof way of implementing inheritance in the constructors, as there are lots of potential problems. For example—in case the parent constructor does something else other than just initializing properties, such as DOM manipulation, then while assigning a new instance of the parent constructor, to the prototype property, of the child constructor, can cause problems. Therefore, the ES6 classes provide a better and easier way to inherit the existing constructors and classes. Using classes We saw that JavaScript's object-oriented model is based on the constructors and prototype-based inheritance. Well, the ES6 classes are just new a syntax for the existing model. Classes do not introduce a new object-oriented model to JavaScript. The ES6 classes aim to provide a much simpler and clearer syntax for dealing with the constructors and inheritance. In fact, classes are functions. Classes are just a new syntax for creating functions that are used as constructors. Creating functions using the classes that aren't used as constructors doesn't make any sense, and offer no benefits. Rather, it makes your code difficult to read, as it becomes confusing. Therefore, use classes only if you want to use it for constructing objects. Let's have a look at classes in detail. Defining a class Just as there are two ways of defining functions, function declaration and function expression, there are two ways to define a class: using the class declaration and the class expression. The class declaration For defining a class using the class declaration, you need to use the class keyword, and a name for the class. Here is a code example to demonstrate how to define a class using the class declaration: class Student { constructor(name) { this.name = name; } } var s1 = new Student("Eden"); console.log(s1.name); //Output "Eden" Here, we created a class named Student. Then, we defined a constructor method in it. Finally, we created a new instance of the class—an object, and logged the name property of the object. The body of a class is in the curly brackets, that is, {}. This is where we need to define methods. Methods are defined without the function keyword, and a comma is not used in between the methods. Classes are treated as functions, and internally the class name is treated as the function name, and the body of the constructor method is treated as the body of the function. There can only be one constructor method in a class. Defining more than one constructor will throw the SyntaxError exception. All the code inside a class body is executed in the strict mode, by default. The previous code is the same as this code when written using function: function Student(name) { this.name = name; } var s1 = new Student("Eden"); console.log(s1.name); //Output "Eden" To prove that a class is a function, consider this code: class Student { constructor(name) { this.name = name; } } function School(name) { this.name = name; } console.log(typeof Student); console.log(typeof School == typeof Student); The output is as follows: function true Here, we can see that a class is a function. It's just a new syntax for creating a function. The class expression A class expression has a similar syntax to a class declaration. However, with class expressions, you are able to omit the class name. Class body and behavior remains the same in both the ways. Here is a code example to demonstrate how to define a class using a class expression: var Student = class { constructor(name) { this.name = name; } } var s1 = new Student("Eden"); console.log(s1.name); //Output "Eden" Here, we stored a reference of the class in a variable, and used it to construct the objects. The previous code is the same as this code when written using function: var Student = function(name) { this.name = name; } var s1 = new Student("Eden"); console.log(s1.name); //Output "Eden" The prototype methods All the methods in the body of the class are added to the prototype property of the class. The prototype property is the prototype of the objects created using class. Here is an example that shows how to add methods to the prototype property of a class: class Person { constructor(name, age) { this.name = name; this.age = age; } printProfile() { console.log("Name is: " + this.name + " and Age is: " + this.age); } } var p = new Person("Eden", 12) p.printProfile(); console.log("printProfile" in p.__proto__); console.log("printProfile" in Person.prototype); The output is as follows: Name is: Eden and Age is: 12 true true Here, we can see that the printProfile method was added to the prototype property of the class. The previous code is the same as this code when written using function: function Person(name, age) { this.name = name; this.age = age; } Person.prototype.printProfile = function() { console.log("Name is: " + this.name + " and Age is: " + this.age); } var p = new Person("Eden", 12) p.printProfile(); console.log("printProfile" in p.__proto__); console.log("printProfile" in Person.prototype); The output is as follows: Name is: Eden and Age is: 12 true true The get and set methods In ES5, to add accessor properties to the objects, we had to use the Object.defineProperty() method. ES6 introduced the get and set prefixes for methods. These methods can be added to the object literals and classes for defining the get and set attributes of the accessor properties. When get and set methods are used in a class body, they are added to the prototype property of the class. Here is an example to demonstrate how to define the get and set methods in a class: class Person { constructor(name) { this._name_ = name; } get name(){ return this._name_; } set name(name){ this._name_ = name; } } var p = new Person("Eden"); console.log(p.name); p.name = "John"; console.log(p.name); console.log("name" in p.__proto__); console.log("name" in Person.prototype); console.log(Object.getOwnPropertyDescriptor(p.__proto__, "name").set); console.log(Object.getOwnPropertyDescriptor(Person.prototype, "name").get); console.log(Object.getOwnPropertyDescriptor(p, "_name_").value); The output is as follows: Eden John true true function name(name) { this._name_ = name; } function name() { return this._name_; } John Here, we created an accessor property to encapsulate the _name_ property. We also logged some other information to prove that name is an accessor property, which is added to the prototype property of the class. The generator method To treat a concise method of an object literal as the generator method, or to treat a method of a class as the generator method, we can simply prefix it with the * character. The generator method of a class is added to the prototype property of the class. Here is an example to demonstrate how to define a generator method in class: class myClass { * generator_function() { yield 1; yield 2; yield 3; yield 4; yield 5; } } var obj = new myClass(); let generator = obj.generator_function(); console.log(generator.next().value); console.log(generator.next().value); console.log(generator.next().value); console.log(generator.next().value); console.log(generator.next().value); console.log(generator.next().done); console.log("generator_function" in myClass.prototype); The output is as follows: 1 2 3 4 5 true true Implementing inheritance in classes Earlier in this article, we saw how difficult it was to implement inheritance hierarchy in functions. Therefore, ES6 aims to make it easy by introducing the extends clause, and the super keyword for classes. By using the extends clause, a class can inherit static and non-static properties of another constructor (which may or may not be defined using a class). The super keyword is used in two ways: It's used in a class constructor method to call the parent constructor When used inside methods of a class, it references the static and non-static methods of the parent constructor Here is an example to demonstrate how to implement the inheritance hierarchy in the constructors using the extends clause, and the super keyword: function A(a) { this.a = a; } A.prototype.printA = function(){ console.log(this.a); } class B extends A { constructor(a, b) { super(a); this.b = b; } printB() { console.log(this.b); } static sayHello() { console.log("Hello"); } } class C extends B { constructor(a, b, c) { super(a, b); this.c = c; } printC() { console.log(this.c); } printAll() { this.printC(); super.printB(); super.printA(); } } var obj = new C(1, 2, 3); obj.printAll(); C.sayHello(); The output is as follows: 3 2 1 Hello Here, A is a function constructor; B is a class that inherits A; C is a class that inherits B; and as B inherits A, therefore C also inherits A. As a class can inherit a function constructor, we can also inherit the prebuilt function constructors, such as String and Array, and also the custom function constructors using the classes instead of other hacky ways that we used to use. The previous example also shows how and where to use the super keyword. Remember that inside the constructor method, you need to use super before using the this keyword. Otherwise, an exception is thrown. If a child class doesn't have a constructor method, then the default behavior will invoke the constructor method of the parent class. Summary In this article, we have learned about the basics of the object-oriented programming using ES5. Then, we jumped into ES6 classes, and learned how it makes easy for us to read and write the object-oriented JavaScript code. We also learned miscellaneous features, such as the accessor methods. Resources for Article: Further resources on this subject: An Introduction to Mastering JavaScript Promises and Its Implementation in Angular.js[article] Finding Peace in REST[article] Scaling influencers [article]

 In this article by Jonathan Hayward author of the book Reactive Programming with JavaScript he explains that Google Maps were a big hit when it came out, and it remains quite important, but the new functionality it introduced was pretty much nothing. The contribution Google made with its maps site was taking things previously only available with a steep learning cliff and giving them its easy trademark simplicity. And that was quite a lot. (For more resources related to this topic, see here.) Similar things might be said about ReactJS. No one at Facebook invented functional reactive programming. No one at Facebook appears to have significantly expanded functional reactive programming. But ReactJS markedly lowered the bar to entry. Previously, with respect to functional reactive programming, there were repeated remarks among seasoned C++ programmers; they said, "I guess I'm just stupid, or at least, I don't have a PhD in computational mathematics." And it might be suggested that proficiency in C++ is no mean feat; getting something to work in Python is less of a feat than getting the same thing to work in C++, just as scaling the local park's winter sledding hill is less of an achievement than scaling Mount Everest. Also, ReactJS introduces enough of changes so that competent C++ programmers who do not have any kind of degree in math, computational or otherwise, stand a fair chance of using ReactJS and being productive in it. Perhaps they may be less effective than pure JavaScript programmers who are particularly interested in functional programming. But learning to effectively program C++ is a real achievement, and most good C++ programmers have a fair chance of usefully implementing functional reactive programming with ReactJS. However, the same cannot be said for following the computer math papers on Wikipedia and implementing something in the academic authors' generally preferred language of Haskell. Here we'll explore a very important topic that is ReactJS as just a view—but what a view! ReactJS is just a view, but what a view! Charles Cézanne famously said, "Monet is just an eye, but what an eye!" Monet didn't try to show off his knowledge of structure and anatomy, but just copy what his eye saw. The consensus judgment of his work holds on to both "just an eye," and "what an eye!" And indeed, the details may be indistinct in Monet, who rebelled against artistry that tried to impress with deep knowledge of anatomy and knowledge of structure that is far beyond what jumps out to the eye. ReactJS is a framework rather than a library, which means that you are supposed to build a solution within the structure provided by ReactJS instead of plugging ReactJS into a solution that you structure yourself. The canonical example of a library is jQuery, where you build a solution your way, and call on jQuery as it fits into a structure that you design. However, ReactJS is specialized as a view. It's not that this is necessarily good or bad, but ReactJS is not a complete web development framework, and does not have even the intension of being the only tool you will ever need. It focuses on being a view, and in Facebook's offering, this does not include any form of AJAX call. This is not a monumental oversight in developing ReactJS; the expectation is that you use ReactJS as a View to provide the user interface functionality, and other tools to meet other needs as appropriate. This text hasn't covered using ReactJS together with your favorite tools, but do combine your favorite tools with ReactJS if they are not going to step on each other's feet. ReactJS may or may not collide with other Views, but it is meant to work with non-View technologies. Summary In this article, we looked at ReactJS as a view and also learned that ReactJS is not a complete web development framework. Resources for Article: Further resources on this subject: An Introduction to Reactive Programming[article] Kivy – An Introduction to Mastering JavaScript Promises and Its Implementation in Angular.js[article] Object-Oriented JavaScript with Backbone Classes [article]

 In this article by Giancarlo Zaccone, the author of the book Python Parallel Programming Cookbook, we will cover the following topics: Introducing Asyncio GPU programming with Python Introducing PyCUDA Introducing PyOpenCL (For more resources related to this topic, see here.) An asynchronous model is of fundamental importance along with the concept of event programming. The execution model of asynchronous activities can be implemented using a single stream of main control, both in uniprocessor systems and multiprocessor systems. In the asynchronous model of a concurrent execution, various tasks intersect with each other along the timeline, and all of this happens under the action of a single flow of control (single-threaded). The execution of a task can be suspended and then resumed alternating in time with any other task. The asynchronous programming model As you can see in the preceding figure, the tasks (each with a different color) are interleaved with one another, but they are in a single thread of control. This implies that when one task is in execution, the other tasks are not. A key difference between a multithreaded programming model and single-threaded asynchronous concurrent model is that in the first case, the operating system decides on the timeline whether to suspend the activity of a thread and start another, while in the second case, the programmer must assume that a thread may be suspended and replaced with another at almost any time. Introducing Asyncio The Python module Asyncio provides facilities to manage events, coroutines, tasks and threads, and synchronization primitives to write concurrent code. When a program becomes very long and complex, it is convenient to divide it into subroutines, each of which realizes a specific task, for which the program implements a suitable algorithm. The subroutine cannot be executed independently but only at the request of the main program, which is then responsible for coordinating the use of subroutines. Coroutines are a generalization of the subroutine. Like a subroutine, the coroutine computes a single computational step, but unlike subroutines, there is no main program that is used to coordinate the results. This is because the coroutines link themselves together to form a pipeline without any supervising function responsible for calling them in a particular order. In a coroutine, the execution point can be suspended and resumed later, having kept track of its local state in the intervening time. In this example, we see how to use the coroutine mechanism of Asyncio to simulate a finite state machine of five states. A Finite-state automaton (FSA) is a mathematical model that is not only widely used in engineering disciplines but also in sciences, such as mathematics and computer science. The automata we want to simulate the behavior is as follows: Finite State Machine We have indicated with S0, S1, S2, S3, and S4 the states of the system, with 0 and 1 as the values for which the automata can pass from one state to the next state (this operation is called a transition). So for example, the state S0 can be passed to the state S1 only for the value 1 and S0 can pass the state S2 only to the value 0. The Python code that follows simulates a transition of the automaton from the state S0, the so-called Start State, up to the state S4, the End State: #Asyncio Finite State Machine import asyncio import time from random import randint @asyncio.coroutine def StartState(): print ("Start State called n") input_value = randint(0,1) time.sleep(1) if (input_value == 0): result = yield from State2(input_value) else : result = yield from State1(input_value) print("Resume of the Transition : nStart State calling " + result) @asyncio.coroutine def State1(transition_value): outputValue = str(("State 1 with transition value = %s n" %(transition_value))) input_value = randint(0,1) time.sleep(1) print("...Evaluating...") if (input_value == 0): result = yield from State3(input_value) else : result = yield from State2(input_value) result = "State 1 calling " + result return (outputValue + str(result)) @asyncio.coroutine def State2(transition_value): outputValue = str(("State 2 with transition value = %s n" %(transition_value))) input_value = randint(0,1) time.sleep(1) print("...Evaluating...") if (input_value == 0): result = yield from State1(input_value) else : result = yield from State3(input_value) result = "State 2 calling " + result return (outputValue + str(result)) @asyncio.coroutine def State3(transition_value): outputValue = str(("State 3 with transition value = %s n" %(transition_value))) input_value = randint(0,1) time.sleep(1) print("...Evaluating...") if (input_value == 0): result = yield from State1(input_value) else : result = yield from EndState(input_value) result = "State 3 calling " + result return (outputValue + str(result)) @asyncio.coroutine def EndState(transition_value): outputValue = str(("End State with transition value = %s n" %(transition_value))) print("...Stop Computation...") return (outputValue ) if __name__ == "__main__": print("Finite State Machine simulation with Asyncio Coroutine") loop = asyncio.get_event_loop() loop.run_until_complete(StartState()) After running the code, we have an output similar to this: C:Python CookBookChapter 4- Asynchronous Programmingcodes - Chapter 4>python asyncio_state_machine.py Finite State Machine simulation with Asyncio Coroutine Start State called ...Evaluating... ...Evaluating... ...Evaluating... ...Evaluating... ...Evaluating... ...Evaluating... ...Evaluating... ...Evaluating... ...Evaluating... ...Evaluating... ...Evaluating... ...Evaluating... ...Stop Computation... Resume of the Transition : Start State calling State 1 with transition value = 1 State 1 calling State 3 with transition value = 0 State 3 calling State 1 with transition value = 0 State 1 calling State 2 with transition value = 1 State 2 calling State 3 with transition value = 1 State 3 calling State 1 with transition value = 0 State 1 calling State 2 with transition value = 1 State 2 calling State 1 with transition value = 0 State 1 calling State 3 with transition value = 0 State 3 calling State 1 with transition value = 0 State 1 calling State 2 with transition value = 1 State 2 calling State 3 with transition value = 1 State 3 calling End State with transition value = 1 Each state of the automata has been defined with the annotation @asyncio.coroutine. For example, the state S0 is: @asyncio.coroutine def StartState(): print ("Start State called n") input_value = randint(0,1) time.sleep(1) if (input_value == 0): result = yield from State2(input_value) else : result = yield from State1(input_value) The transition to the next state is determined by input_value, which is defined by the randint(0,1) function of Python's module random. This function randomly provides the value 0 or 1, where it randomly determines to which state the finite-state machine will be passed: input_value = randint(0,1) After determining the value at which state the finite state machine will be passed, the coroutine calls the next coroutine using the command yield from: if (input_value == 0): result = yield from State2(input_value) else : result = yield from State1(input_value) The variable result is the value that each coroutine returns. It is a string, and at the end of the computation, we can reconstruct [NV1] the transition from the initial state of the automaton, the Start State, up to the final state, the End State. The main program starts the evaluation inside the event loop: if __name__ == "__main__": print("Finite State Machine simulation with Asyncio Coroutine") loop = asyncio.get_event_loop() loop.run_until_complete(StartState()) GPU programming with Python A graphics processing unit (GPU) is an electronic circuit that specializes in processing data to render images from polygonal primitives. Although they were designed to carry out rendering images, GPUs have continued to evolve, becoming more complex and efficient in serving both real-time and offline rendering community. GPUs have continued to evolve, becoming more complex and efficient in performing any scientific computation. Each GPU is indeed composed of several processing units called streaming multiprocessor (SM), representing the first logic level of parallelism; each SM in fact, works simultaneously and independently from the others. The GPU architecture Each SM is in turn divided into a group of Stream Processors (SP), each of which has a core of real execution and can run a thread sequentially. SP represents the smallest unit of execution logic and the level of finer parallelism. The division in SM and SP is structural in nature, but it is possible to outline a further logical organization of the SP of a GPU, which are grouped together in logical blocks characterized by a particular mode of execution—all cores that make up a group run at the same time with the same instructions. This is just the SIMD (Single Instruction, Multiple Data) model. The programming paradigm that characterizes GPU computing is also called stream processing because the data can be viewed as a homogeneous flow of values that are applied synchronously to the same operations. Currently, the most efficient solutions to exploit the computing power provided by the GPU cards are the software libraries CUDA and OpenCL. Introducing PyCUDA PyCUDA is a Python wrapper for CUDA (Compute Unified Device Architecture), the software library developed by NVIDIA for GPU programming. The PyCuda programming model is designed for the common execution of a program on the CPU and GPU so as to allow you to perform the sequential parts on the CPU and the numeric parts that are more intensive on the GPU. The phases to be performed in the sequential mode are implemented and executed on the CPU (host), while the steps to be performed in parallel are implemented and executed on the GPU (device). The functions to be performed in parallel on the device are called kernels. The skeleton general for the execution of a generic function kernel on the device is as follows: Allocation of memory on the device. Transfer of data from the host memory to that allocated on the device. Running the device: Running the configuration. Invocation of the kernel function. Transfer of the results from the memory on the device to the host memory. Release of the memory allocated on the device. The PyCUDA programming model To show the PyCuda workflow, let's consider a 5 × 5 random array and the following procedure: Create the array 5×5 on the CPU. Transfer the array to the GPU. Perform a Task[NV2]  on the array in the GPU (double all the items in the array). Transfer the array from the GPU to the CPU. Print the results. The code for this is as follows: import pycuda.driver as cuda import pycuda.autoinit from pycuda.compiler import SourceModule import numpy a = numpy.random.randn(5,5) a = a.astype(numpy.float32) a_gpu = cuda.mem_alloc(a.nbytes) cuda.memcpy_htod(a_gpu, a) mod = SourceModule(""" __global__ void doubleMatrix(float *a) { int idx = threadIdx.x + threadIdx.y*4; a[idx] *= 2; } """) func = mod.get_function("doubleMatrix") func(a_gpu, block=(5,5,1)) a_doubled = numpy.empty_like(a) cuda.memcpy_dtoh(a_doubled, a_gpu) print ("ORIGINAL MATRIX") print a print ("DOUBLED MATRIX AFTER PyCUDA EXECUTION") print a_doubled The example output should be like this : C:Python CookBookChapter 6 - GPU Programming with Python >python PyCudaWorkflow.py ORIGINAL MATRIX [[-0.59975582 1.93627465 0.65337795 0.13205571 -0.46468592] [ 0.01441949 1.40946579 0.5343408 -0.46614054 -0.31727529] [-0.06868593 1.21149373 -0.6035406 -1.29117763 0.47762445] [ 0.36176383 -1.443097 1.21592784 -1.04906416 -1.18935871] [-0.06960868 -1.44647694 -1.22041082 1.17092752 0.3686313 ]] DOUBLED MATRIX AFTER PyCUDA EXECUTION [[-1.19951165 3.8725493 1.3067559 0.26411143 -0.92937183] [ 0.02883899 2.81893158 1.0686816 -0.93228108 -0.63455057] [-0.13737187 2.42298746 -1.2070812 -2.58235526 0.95524889] [ 0.72352767 -1.443097 1.21592784 -1.04906416 -1.18935871] [-0.06960868 -1.44647694 -1.22041082 1.17092752 0.3686313 ]] The code starts with the following imports: import pycuda.driver as cuda import pycuda.autoinit from pycuda.compiler import SourceModule The pycuda.autoinit import automatically picks a GPU to run on based on the availability and number. It also creates a GPU context for subsequent code to run in. Both the chosen device and the created context are available from pycuda.autoinit as importable symbols if needed. While the SourceModule component is the object where a C-like code for the GPU must be written. The first step is to generate the input 5 × 5 matrix. Since most GPU computations involve large arrays of data, the NumPy module must be imported: import numpy a = numpy.random.randn(5,5) Then, the items in the matrix are converted in a single precision mode, many NVIDIA cards support only single precision: a = a.astype(numpy.float32) The first operation to be done in order to implement a GPU loads the input array from the host memory (CPU) to the device (GPU). This is done at the beginning of the operation and consists two steps that are performed by invoking two functions provided PyCuda[NV3] . Memory allocation on the device is done via the cuda.mem_alloc function. The device and host memory may not ever communicate while performing a function kernel. This means that to run a function in parallel on the device, the data relating to it must be present in the memory of the device itself. Before you copy data from the host memory to the device memory, you must allocate the memory required on the device: a_gpu = cuda.mem_alloc(a.nbytes). Copy of the matrix from the host memory to that of the device with the function: call cuda.memcpy_htod : cuda.memcpy_htod(a_gpu, a). We also note that a_gpu is one dimensional, and on the device, we need to handle it as such. All these operations do not require the invocation of a kernel and are made directly by the main processor. The SourceModule entity serves to define the (C-like) kernel function doubleMatrix that multiplies each array entry by 2: mod = SourceModule(""" __global__ void doubleMatrix(float *a) { int idx = threadIdx.x + threadIdx.y*4; a[idx] *= 2; } """) The __global__ qualifier is a directive that indicates that the doubleMatrix function will be processed on the device. It will be just the compiler Cuda nvcc that will be used to perform this task. Let's look at the function's body, which is as follows: int idx = threadIdx.x + threadIdx.y*4; The idx parameter is the matrix index that is identified by the thread coordinates threadIdx.x and threadIdx.y. Then, the element matrix with the index idx is multiplied by 2: a[idx] *= 2; We also note that this kernel function will be executed once in 16 different threads. Both the variables threadIdx.x and threadIdx.y contain indices between 0 and 3 , and the pair[NV4]  is different for each thread. Threads scheduling is directly linked to the GPU architecture and its intrinsic parallelism. A block of threads is assigned to a single SM. Here, threads are further divided into groups called warps. The size of a warp depends on the architecture under consideration. The threads of the same warp are managed by the control unit called the warp scheduler. To take full advantage of the inherent parallelism of the SM, the threads of the same warp must execute the same instruction. If this condition does not occur, we speak of divergence of threads. If the same warp threads execute different instructions, the control unit cannot handle all the warps. It must follow the sequences of instructions for every single thread (or for homogeneous subsets of threads) in a serial mode. Let's observe how the thread block is divided in various warps—threads are divided by the value of threadIdx. The threadIdx structure consists of 3 fields: threadIdx.x, threadIdx.y, and threadIdx.z. Thread blocks subdivision: T(x,y), where x = threadIdx.x and y = threadIdx.y We can see again that the code in the kernel function will be automatically compiled by the nvcc cuda compiler. If there are no errors, a pointer to this compiled function will be created. In fact, the mod.get_function[NV5] ("doubleMatrix") returns an identifier to the function created func: func = mod.get_function("doubleMatrix ") To perform a function on the device, you must first configure the execution appropriately. This means that we need to determine the size of the coordinates to identify and distinguish the thread belonging to different blocks. This will be done using the block parameter inside the func call: func(a_gpu, block = (5, 5, 1)) The block = (5, 5, 1) tells us that we are calling a kernel function with a_gpu linearized input matrix and a single thread block of size, 5 threads in the x direction, 5 threads in the y direction, and 1 thread in the z direction, 16 threads in total. This structure is designed with parallel implementation of the algorithm of interest. The division of the workload results is an early form of parallelism that is sufficient and necessary to make use of the computing resources provided by the GPU. Once you've configured the kernel's invocation, you can invoke the kernel function that executes instructions in parallel on the device. Each thread executes the same code kernel. After the computation in the gpu device, we use an array to store the results: a_doubled = numpy.empty_like(a) cuda.memcpy_dtoh(a_doubled, a_gpu) Introducing PyOpenCL As for programming with PyCuda, the first step to build a program for PyOpenCL is the encoding of the host application. In fact, this is performed on the host computer (typically, the user's PC) and then, it dispatches the kernel application on the connected devices (GPU cards). The host application must contain five data structures, which are as follows: Device: This identifies the hardware where the kernel code must be executed. A PyOpenCL application can be executed not only on CPU and GPU cards but also on embedded devices such as FPGA (Field Programmable Gate Array). Program: This is a group of kernels. A program selects which kernel must be executed on the device. Kernel: This is the code to be executed on the device. A kernel is essentially a (C-like) function that enables it to be compiled for an execution on any device that supports OpenCL drivers. The kernel is the only way the host can call a function that will run on a device. When the host invokes a kernel, many work items start running on the device. Each work item runs the code of the kernel but works on a different part of the dataset. Command queue: Each device receives kernels through this data structure. A command queue orders the execution of kernels on the device. Context: This is a group of devices. A context allows devices to receive kernels and transfer data. PyOpenCL programming The preceding figure shows how these data structures can work in a host application. Let's remember again that a program can contain multiple functions that are to be executed on the device and each kernel encapsulates only a single function from the program. In this example, we show you the basic steps to build a PyOpenCL program. The task to be executed is the parallel sum of two vectors. In order to maintain a readable output, let's consider two vectors, each of one out of 100 elements. The resulting vector will be for each element's i[NV6] th, which is the sum of the ith element vector_a plus the ith element vector_b. Of course, to be able to appreciate the parallel execution of this code, you can also increase some orders of magnitude the size of the input vector_dimension:[NV7]  import numpy as np import pyopencl as cl import numpy.linalg as la vector_dimension = 100 vector_a = np.random.randint(vector_dimension, size=vector_dimension) vector_b = np.random.randint(vector_dimension, size=vector_dimension) platform = cl.get_platforms()[0] device = platform.get_devices()[0] context = cl.Context([device]) queue = cl.CommandQueue(context) mf = cl.mem_flags a_g = cl.Buffer(context, mf.READ_ONLY | mf.COPY_HOST_PTR, hostbuf=vector_a) b_g = cl.Buffer(context, mf.READ_ONLY | mf.COPY_HOST_PTR, hostbuf=vector_b) program = cl.Program(context, """ __kernel void vectorSum(__global const int *a_g, __global const int *b_g, __global int *res_g) { int gid = get_global_id(0); res_g[gid] = a_g[gid] + b_g[gid]; } """).build() res_g = cl.Buffer(context, mf.WRITE_ONLY, vector_a.nbytes) program.vectorSum(queue, vector_a.shape, None, a_g, b_g, res_g) res_np = np.empty_like(vector_a) cl.enqueue_copy(queue, res_np, res_g) print ("PyOPENCL SUM OF TWO VECTORS") print ("Platform Selected = %s" %platform.name ) print ("Device Selected = %s" %device.name) print ("VECTOR LENGTH = %s" %vector_dimension) print ("INPUT VECTOR A") print vector_a print ("INPUT VECTOR B") print vector_b print ("OUTPUT VECTOR RESULT A + B ") print res_np assert(la.norm(res_np - (vector_a + vector_b))) < 1e-5 The output from Command Prompt should be like this: C:Python CookBook Chapter 6 - GPU Programming with PythonChapter 6 - codes>python PyOpenCLParallellSum.py Platform Selected = NVIDIA CUDA Device Selected = GeForce GT 240 VECTOR LENGTH = 100 INPUT VECTOR A [ 0 29 88 46 68 93 81 3 58 44 95 20 81 69 85 25 89 39 47 29 47 48 20 86 59 99 3 26 68 62 16 13 63 28 77 57 59 45 52 89 16 6 18 95 30 66 19 29 31 18 42 34 70 21 28 0 42 96 23 86 64 88 20 26 96 45 28 53 75 53 39 83 85 99 49 93 23 39 1 89 39 87 62 29 51 66 5 66 48 53 66 8 51 3 29 96 67 38 22 88] INPUT VECTOR B [98 43 16 28 63 1 83 18 6 58 47 86 59 29 60 68 19 51 37 46 99 27 4 94 5 22 3 96 18 84 29 34 27 31 37 94 13 89 3 90 57 85 66 63 8 74 21 18 34 93 17 26 9 88 38 28 14 68 88 90 18 6 40 30 70 93 75 0 45 86 15 10 29 84 47 74 22 72 69 33 81 31 45 62 81 66 69 14 71 96 91 51 35 4 63 36 28 65 10 41] OUTPUT VECTOR RESULT A + B [ 98 72 104 74 131 94 164 21 64 102 142 106 140 98 145 93 108 90 84 75 146 75 24 180 64 121 6 122 86 146 45 47 90 59 114 151 72 134 55 179 73 91 84 158 38 140 40 47 65 111 59 60 79 109 66 28 56 164 111 176 82 94 60 56 166 138 103 53 120 139 54 93 114 183 96 167 45 111 70 122 120 118 107 91 132 132 74 80 119 149 157 59 86 7 92 132 95 103 32 129] In the first line of the code after the required module import, we defined the input vectors: vector_dimension = 100 vector_a = np.random.randint(vector_dimension, size= vector_dimension) vector_b = np.random.randint(vector_dimension, size= vector_dimension) Each vector contain 100 integers items that are randomly selected thought the NumPy function: np.random.randint(max integer , size of the vector) Then, we must select the device to run the kernel code. To do this, we must first select the platform using the get_platform() PyOpenCL statement: platform = cl.get_platforms()[0] This platform, as you can see from the output, corresponds to the NVIDIA CUDA platform. Then, we must select the device using the get_device() platform's method: device = platform.get_devices()[0] In the following steps, the context and the queue are defined, PyOpenCL provides the method context (device selected) and queue (context selected): context = cl.Context([device]) queue = cl.CommandQueue(context) To perform the computation in the device, the input vector must be transferred to the device's memory. So, two input buffers in the device memory must be created: mf = cl.mem_flags a_g = cl.Buffer(context, mf.READ_ONLY | mf.COPY_HOST_PTR, hostbuf=vector_a) b_g = cl.Buffer(context, mf.READ_ONLY | mf.COPY_HOST_PTR, hostbuf=vector_b) Also, we prepare the buffer for the resulting vector: res_g = cl.Buffer(context, mf.WRITE_ONLY, vector_a.nbytes) Finally, the core of the script, the kernel code is defined inside a program as follows: program = cl.Program(context, """ __kernel void vectorSum(__global const int *a_g, __global const int *b_g, __global int *res_g) { int gid = get_global_id(0); res_g[gid] = a_g[gid] + b_g[gid]; } """).build() The kernel's name is vectorSum. The parameter list defines the data types of the input arguments (vectors of integers) and output data type (a vector of integer again). Inside the kernel, the sum of the two vectors is simply defined as: Initialize the vector index: int gid = get_global_id(0) Sum the vector's components: res_g[gid] = a_g[gid] + b_g[gid]; In OpenCL and PyOpenCL, buffers are attached to a context[NV8]  and are only moved to a device once the buffer is used on that device. Finally, we execute vectorSum in the device: program.vectorSum(queue, vector_a.shape, None, a_g, b_g, res_g) To visualize the results, an empty vector is built: res_np = np.empty_like(vector_a) Then, the result is copied into this vector: cl.enqueue_copy(queue, res_np, res_g) Finally, the results are displayed: print ("VECTOR LENGTH = %s" %vector_dimension) print ("INPUT VECTOR A") print vector_a print ("INPUT VECTOR B") print vector_b print ("OUTPUT VECTOR RESULT A + B ") print res_np To check the result, we use the assert statement. It tests the result and triggers an error if the condition is false: assert(la.norm(res_np - (vector_a + vector_b))) < 1e-5 Summary In this article we discussed about Asyncio, GPU programming with Python, PyCUDA, and PyOpenCL. Resources for Article: Further resources on this subject: Bizarre Python[article] Scientific Computing APIs for Python[article] Optimization in Python [article]

In this article by Joydip Kanjilal, author of the book Entity Framework Tutorial - Second Edition explains how Entity Framework contains a powerful client-side query engine that allows you to execute queries against the conceptual model of data, irrespective of the underlying data store in use. This query engine works with a rich functional language called Entity SQL (or E-SQL for short), a derivative of Transact SQL (T-SQL), that enables you to query entities or a collection of entities. (For more resources related to this topic, see here.) An overview of the E-SQL language Entity Framework allows you to write programs against the EDM and also add a level of abstraction on top of the relational model. This isolation of the logical view of data from the Object Model is accomplished by expressing queries in terms of abstractions using an enhanced query language called E-SQL. This language is specially designed to query data from the EDM. E-SQL was designed to address the need for a language that can query data from its conceptual view, rather than its logical view. From T-SQL to E-SQL SQL is the primary language that has been in use for years for querying databases. Remember, SQL is a standard and not owned by any particular database vendor. SQL-92 is a standard, and is the most popular SQL standard currently in use. This standard was released in 1992. The 92 in the name reflects this fact. Different database vendors implemented their own flavors of the SQL-92 standard. The T-SQL language was designed by Microsoft as an SQL Server implementation of the SQL-92 standard. Similar to other SQL languages implemented by different database vendors, the E-SQL language is Entity Framework implementation of the SQL-92 standard that can be used to query data from the EDM. E-SQL is a text-based, provider independent, query language used by Entity Framework to express queries in terms of EDM abstractions and to query data from the conceptual layer of the EDM. One of the major differences between E-SQL and T-SQL is in nested queries. Note that you should always enclose your nested queries in E-SQL using parentheses as seen here: SELECT d, (SELECT DEREF (e) FROM NAVIGATE (d, PayrollEntities.FK_Employee_Department) AS e) AS Employees FROM PayrollEntities.Department AS d; The Select VALUE... statement is used to retrieve singleton values. It is also used to retrieve values that don't have any column names. However, the Select ROW... statement is used to select one or more rows. As an example, if you want a value as a collection from an entity without the column name, you can use the VALUE keyword in the SELECT statement as shown here: SELECT VALUE emp.EmployeeName FROM PayrollEntities.Employee as emp The preceding statement will return the employee names from the Employee entity as a collection of strings. In T-SQL, you can have the ORDER BY clause at the end of the last query when using UNION ALL. SELECT EmployeeID, EmployeeName From Employee UNION ALL SELECT EmployeeID, Basic, Allowances FROM Salary ORDER BY EmployeeID On the contrary, you do not have the ORDER BY clause in the UNION ALL operator in E-SQL. Why E-SQL when I already have LINQ to Entities? LINQ to Entities is a new version of LINQ, well suited for Entity Framework. But why do you need E-SQL when you already have LINQ to Entities available to you? LINQ to Entities queries are verified at the time of compilation. Therefore, it is not at all suited for building and executing dynamic queries. On the contrary, E-SQL queries are verified at runtime, so they can be used for building and executing dynamic queries. You now have a new ADO.NET provider in E-SQL, which is a sophisticated query engine that can be used to query your data from the conceptual model. It should be noted, however, that both LINQ and E-SQL queries are converted into canonical command trees that are in turn translated into database-specific query statements based on the underlying database provider in use, as shown in the following diagram: We will now take a quick look at the features of E-SQL before we delve deep into this language. Features of E-SQL These are the features of E-SQL: Provider neutrality: E-SQL is independent of the underlying ADO.NET data provider in use because it works on top of the conceptual model. SQL like: The syntax of E-SQL statements resemble T-SQL. Expressive with support for entities and types: You can write your E-SQL queries in terms of EDM abstractions. Composable and orthogonal: You can use a subquery wherever you have support for an expression of that type. The subqueries are all treated uniformly regardless of where they have been used. In the sections that follow, we will take a look at the E-SQL language in depth. We will discuss the following points: Operators Expressions Identifiers Variables Parameters Canonical functions Operators in E-SQL An operator is one that operates on a particular operand to perform an operation. Operators in E-SQL can broadly be classified into the following categories: Arithmetic operators: These are used to perform arithmetic operations. Comparison operators: You can use these to compare the values of two operands. Logical operators: These are used to perform logical operations. Reference operators: These act as logical pointers to a particular entity belonging to a particular entity set. Type operators: These can operate on the type of an expression. Case operators: These operate on a set of Boolean expressions. Set operators: These operate on set operations. Arithmetic operators Here is an example of an arithmetic operator: SELECT VALUE s FROM PayrollEntities.Salary AS s where s.Basic = 5000 + 1000 The following arithmetic operators are available in E-SQL: + (add) - (subtract) / (divide) % (modulo) * (multiply) Comparison operators Here is an example of a comparison operator: SELECT VALUE e FROM PayrollEntities.Employee AS e where e.EmployeeID = 1 The following is a list of the comparison operators available in E-SQL: = (equals) != (not equal to) <> (not equal to) > (greater than) < (less than) >= (greater than or equal to) <= (less than or equal to) Logical operators Here is an example of using logical operators in E-SQL: SELECT VALUE s FROM PayrollEntities.Salary AS s where s.Basic > 5000 && s.Allowances > 3000 This is a list of the logical operators available in E-SQL: && (And) ! (Not) || (Or) Reference operators The following is an example of how you can use a reference operator in E-SQL: SELECT VALUE REF(e).FirstName FROM PayrollEntities.Employee as e The following is a list of the reference operators available in E-SQL: Key Ref CreateRef DeRef Type operators Here is an example of a type operator that returns a collection of employees from a collection of persons: SELECT VALUE e FROM OFTYPE(PayrollEntities.Person, PayrollEntities.Employee) AS e The following is a list of the type operators available in E-SQL: OfType Cast Is [Not] Of Treat Set operators This is an example of how you can use a set operator in E-SQL: (Select VALUE e from PayrollEntities.Employee as e where e.FirstName Like 'J%') Union All ( select VALUE s from PayrollEntities.Employee as s where s.DepartmentID = 1) Here is a list of the set operators available in E-SQL: Set Union Element AnyElement Except [Not] Exists [Not] In Overlaps Intersect Operator precedence When you have multiple operators operating in a sequence, the order in which the operators will be executed will be determined by the operator precedence. The following table shows the operator, operator type, and their precedence levels in E-SQL language: Operators Operator type Precedence level . , [] () Primary Level 1 ! not Unary Level 2 * / % Multiplicative Level 3 + and - Additive Level 4 < > <= >= Relational Level 5 = != <> Equality Level 6 && Conditional And Level 7 || Conditional Or Level 8 Expressions in E-SQL Expressions are the building blocks of the E-SQL language. Here are some examples of how expressions are represented: 1; //This represents one scalar item {2}; //This represents a collection of one element {3, 4, 5} //This represents a collection of multiple elements Query expressions in E-SQL Query expressions are used in conjunction with query operators to perform a certain operation and return a result set. Query expressions in E-SQL are actually a series of clauses that are represented using one or more of the following: SELECT: This clause is used to specify or limit the number of elements that are returned when a query is executed in E-SQL. FROM: This clause is used to specify the source or collection for retrieval of the elements in a query. WHERE: This clause is used to specify a particular expression. HAVING: This clause is used to specify a filter condition for retrieval of the result set. GROUP BY: This clause is used to group the elements returned by a query. ORDER BY: This clause is used to order the elements returned in either ascending or descending order. Here is the complete syntax of query expressions in E-SQL: SELECT VALUE [ ALL | DISTINCT ] FROM expression [ ,...n ] as C [ WHERE expression ] [ GROUP BY expression [ ,...n ] ] [ HAVING search_condition ] [ ORDER BY expression] And here is an example of a typical E-SQL query with all clause types being used: SELECT emp.FirstName FROM PayrollEntities.Employee emp, PayrollEntities.Department dept Group By dept.DepartmentName Where emp.DepartmentID = dept.DepartmentID Having emp.EmployeeID > 5 Identifiers, variables, parameters, and types in E-SQL Identifiers in E-SQL are of the following two types: Simple identifiers Quoted identifiers Simple identifiers are a sequence of alphanumeric or underscore characters. Note that an identifier should always begin with an alphabetical character. As an example, the following are valid identifiers: a12_ab M_09cd W0001m However, the following are invalid identifiers: 9abcd _xyz 0_pqr Quoted identifiers are those that are enclosed within square brackets ([]). Here are some examples of quoted identifiers: SELECT emp.EmployeeName AS [Employee Name] FROM Employee as emp SELECT dept.DepartmentName AS [Department Name] FROM Department as dept Quoted identifiers cannot contain a new line, tab, backspace, or carriage return characters. In E-SQL, a variable is a reference to a named expression. Note that the naming conventions for variables follow the same rules for an identifier. In other words, a valid variable reference to a named expression in E-SQL should be a valid identifier too. Here is an example: SELECT emp FROM Employee as emp; In the preceding example, emp is a variable reference. Types can be of three versions: Primitive types like integers and strings Nominal types such as entity types, entity sets, and relationships Transient types like rows, collections, and references The E-SQL language supports the following type categories: Rows Collections References Row A row, which is also known as a tuple, has no identity or behavior and cannot be inherited. The following statement returns one row that contains six elements: ROW (1, 'Joydip'); Collections Collections represent zero or more instances of other instances. You can use SET () to retrieve unique values from a collection of values. Here is an example: SET({1,1,2,2,3,3,4,4,5,5,6,6}) The preceding example will return the unique values from the set. Specifically, 2, 3, 4, 5, and 6. This is equivalent to the following statement: Select Value Distinct x from {1,1,2,2,3,3,4,4,5,5,6,6} As x; You can create collections using MULTISET () or even using {} as shown in the following examples: MULTISET (1, 2, 3, 4, 5, 6) The following represents the same as the preceding example: {1, 2, 3, 4, 5, 6} Here is how you can return a collection of 10 identical rows each with six elements in them: SELECT ROW(1,'Joydip') from {1,2,3,4,5,6,7,8,9,10} To return a collection of all rows from the employee set, you can use the following: Select emp from PayrollEntities.Employee as emp; Similarly, to select all rows from the department set, you use the following: Select dept from PayrollEntities.Department as dept; Reference A reference denotes a logical pointer or reference, to a particular entity. In essence, it is a foreign key to a specific entity set. Operators are used to perform operations on one or more operands. In E-SQL, the following operators are available to construct, deconstruct, and also navigate through references: KEY REF CREATEREF DEREF To create a reference to an instance of Employee, you can use REF() as shown here: SELECT REF (emp) FROM PayrollEntities.Employee as emp Once you have created a reference to an entity using REF(), you can also dereference the entity using DREF() as shown: DEREF (CREATEREF(PayrollEntities.Employee, ROW(@EmployeeID))) Summary In this article, we explored E-SQL and how it can be used with the Entity Client provider to perform CRUD operations in our applications. We discussed the differences between E-SQL and T-SQL and the differences between E-SQL and LINQ. We also discussed when one should choose E-SQL instead of LINQ to query data in applications. Resources for Article: Further resources on this subject: Hosting the service in IIS using the TCP protocol [article] Entity Framework Code-First: Accessing Database Views and Stored Procedures [article] Entity Framework DB First – Inheritance Relationships between Entities [article]

In this article by Steve Ivie, the author of the book, Building Dynamics CRM 2015 Dashboards with Power BI, we will explore the functionality of using Microsoft Power BI integrated with Microsoft Dynamics CRM 2015. Microsoft Dynamics CRM 2015 is a powerful sales automation and relationship management tool with fantastic built-in reporting features. However, when it comes to analyzing data, there is now a more powerful option available with Microsoft Power BI. We will show you how to build an interactive sales dashboard, which can be used by everyone (from a salesperson to the CEO). We will build an interactive sales productivity dashboard that will answer the common salesperson's question: "How is my team doing?" We will build this dashboard with native Microsoft Power BI functionality, including charts, graphs, maps, summaries, and tiles that will be viewable in Microsoft Dynamics CRM 2015 and mobile apps. This article will take you through the following topics: How to set up and configure Microsoft Power BI for Office 365 Connect and access Microsoft Dynamics CRM 2015 datasets Explore methods to connect to Dynamics CRM data with Power BI (For more resources related to this topic, see here.) Preparation To build the sales productivity dashboard, we must first have the data and tool sets in place in Microsoft Dynamics CRM 2015 and Microsoft Power BI. Toward the end of this article, you should be able to set up and use the following environments to get data for your sales productivity dashboard: Microsoft Office 365 Microsoft Dynamics CRM 2015 Online Microsoft Power BI for Office 365 After we connect Power BI with Microsoft Dynamics CRM, we will look at the options to load and query the Dynamics CRM sales data using the Power BI Designer. Setting up Office 365 Before we start building dashboards with Microsoft Power BI, we have a little setup work to do in Microsoft Office 365, Microsoft Power BI sites, and Microsoft Dynamics CRM Online. The good thing is they live inside the Microsoft Office 365 platform. To use these applications, we first need to set up a Microsoft Office 365 instance and user account. Once we establish the Microsoft Office 365 instance, we can access application subscriptions and manage users who use the Microsoft Office 365 Admin Portal. Here is how it it's done: Navigate to Microsoft Office 365 website using the following link: http://products.office.com/en-us/business/explore-office-365-for-business Go to Plans and Pricing and select the plan type that fits your business. There are a few different plans that can be used with Dynamics CRM Online and Power BI, but in this article, we will use the Office 365 Enterprise E3 30-day free trial. Once in the Microsoft Office 365 account setup window, enter your company information and create an account. The account provision process will kick off, and you will be logged in to your Microsoft Office 365 Admin Portal shortly after it is provisioned: The interface as seen after signing in Adding Dynamics CRM 2015 Online Now that we have an active Microsoft Office 365 account, we need to add a Dynamics CRM Online subscription. Dynamics CRM On-Premise deployments will integrate with Power BI using an Internet-facing deployment (IFD) configuration, but in this article, we will use the online version of Dynamics CRM. To add Dynamics CRM Online to the Office 365 instance, perform the following steps: Navigate to Purchase Services in Admin Portal and locate the Microsoft Dynamics CRM subscription offering. In this article, we will use the Microsoft Dynamics CRM Online Professional 30-day trial. Giving user access Before users can connect to a Dynamics CRM Online instance, a license needs to be assigned to a user account. After you assign this license to the user account, you must also assign a security role so that the user can access your Dynamics CRM Organization. Here is how it's done: From the Office 365 Admin Portal, select the Dynamics CRM pane from the list of apps: Once in Dynamics CRM, select Setting | Security | Users and then navigate to Users, who need a role assigned: Navigate to the user submenu and select MANAGE ROLES: Once the user role is assigned, you should now see the data in Dynamics CRM: Data as seen in the Sales Activity Dashboard Importing the sample data In this article, we will build datasets for a sales productivity dashboard using data from Dynamics CRM the Lead, Account, Opportunity entities. To add the sample data, download the .csv files and import them into Dynamics CRM Online with the native import tool. Here is how you import the sample data: Download the sample .csv file from ContactLead.csv, Accounts.csv, and Opportunities.csv. In Dynamics CRM, open the import tool located under any list view: Upload the sample .csv files and begin the import: Verify mapping and initiate the import data. Finding the OData connection Dynamics CRM is a web-based application built on a set of web services. For this article, we will focus on the Organizational Data Service, using the Protocol OData (REST) protocol to connect Power BI to Dynamics CRM. Here is how we locate the OData URL in Dynamics CRM to use with Power BI later: In Dynamics CRM, select Setting | Customizations in the top navigation bar to access the Customizations area. Once in the Customizations area, select Developer Resources and navigate to Organizational Data Service located at the bottom of the browser window: In Developer Resources, scroll down to the bottom of the window and copy the OData (REST) URL link. This URL will be used later when you configure the Power BI connection: Setting up Power BI for the Office 365 site The new Power BI for Office 365 now includes a secure website portal used to store dashboards, reports, and datasets. We need to set up and configure a new Power BI site. Here is how it is done: Navigate to Microsoft Power BI for the Office 365 website using the following link: http://www.powerbi.com Once in the website, enter the e-mail address that was used when you set up the Office 365 account and then submit a request to Power BI for the Office 365 free trial. Shortly after requesting the trial, you will receive an e-mail with a link to access your Power BI site. Once you receive the e-mail, click on the link to the Power BI site and sign in with your specifically created Office 365 user e-mail account. Sales Productivity Dashboard as seen in Power BI Installing the Power BI Designer Power BI along with PowerQuery, PowerMap, and PowerView—used to be only included as a Microsoft Excel 2013 add-in. Although these add-ins are still available, there is now a new tool dedicated to Power BI report and dashboard authoring called Power BI Designer. The Power BI Designer offers a lot of the same functionalities as its predecessor in Excel add-in, but without the Excel requirements. The benefit of using the Power BI Designer is that it is a standalone program that can provide self-service data connectivity, transformation, modeling, visualizations, and collaboration. The Power BI Designer is a standalone 64-bit application that can be deployed together with a 32-bit version of Office, using the same functionality that was used to create interactive charts, maps, graphs, and data transformations without the requirement of Microsoft Excel 2013. Here is how you install it: In the Power BI site, navigate to the down arrow icon located in the top-right corner of the navigation area: Download Power BI Designer Preview. Then, install the PBIDesignr_x64.msi file. Open Power BI Designer from the desktop icon: Now that you have Power BI Designer installed and open, you can begin leveraging the tool for dashboard, report creation, and data transformation. Power BI Designer help videos are available at startup or by navigating to File -> Getting Started in the main menu. The Power BI Designer toolset is based on two views: Query: This connects, shapes, and combines data to data models Report: This builds reports from the queried information to shareable reports Power BI Designer preview Once you build your dashboards and reports with Power BI Designer, you will want to save your work. Using Power BI Designer, you can now save it as a Power BI Designer file. Reviewing authentication methods Now that the Power BI Designer is installed, we are ready to connect to the Dynamics CRM data and start building our sales productivity dashboards and reports, but before we do this, we need to understand the various OData (REST) authentication methods provided by Power BI. Each method is briefly explained here: Anonymous: This authentication allows any user to access any public content without providing a username and password challenge to the client browser. Windows: This authentication occurs when your user runs on a corporate network that uses Microsoft Active Directory service domain identities or other Windows accounts to identify users. Basic: This access authentication is a method for an HTTP user agent to provide a username and password when making a request. The Web API: This authentication takes place in the host. For web hosting, the host is IIS, which uses HTTP modules for authentication. The marketplace key: This authentication is based on the subscription-based account key secured through SSL. The Organizational account: This authentication is based on the users of Dynamics CRM Online, cloud applications, or users who run modern LOB applications on-premises that may leverage a web service such as Azure behind the scenes. Connecting to Dynamics CRM You just learned how to set up and configure Power BI sites and the Power BI Designer. Now you will learn how to connect the Power BI Designer to the Dynamics CRM Online instance and put data entities into Power BI. For our sales productivity dashboard, we will use the following Dynamics CRM entities: Users Leads Accounts Opportunities Checking requirements Before we connect to Dynamic CRM with the Power BI Designer, let's quickly review the general requirements: A user must specify a valid OData URL for a Dynamics CRM Online instance. The connector will not work with an on-premise CRM version. Enable the OData endpoint in the site settings with Dynamics CRM. Then, select Settings | Customizations | Developer Resources. The OData URL is listed under Service Endpoints. The user account that you use to access Dynamics CRM Online must be the same as the one you will use for Power BI. Accessing data Earlier, we downloaded and installed the Power BI Designer, which allows read-only access to the Dynamics CRM Online instance in order to make it easy for users to get the sales data they want. To see how easy it is to access data: Open Power BI Designer and select Query from the bottom-left corner of the Power BI Designer window. In the top-left corner of the Power BI Designer window, select Get Data. In the Get Data Window, select All | Dynamics CRM Online to access the Dynamics CRM Online OData Feed window: Once the Microsoft Dynamics CRM Online window opens, enter the Dynamics CRM Online OData (REST) URL previously captured during setup: The Access Dynamics CRM OnlineOData Feed window may or may not appear to log in to the Dynamics CRM Online instance. If the window does appear, use Organizational account to sign in. For this article, we will select the first URL to connect the OData feed. If the Access Dynamics CRM OnlineOdata Feed window does not appear, you are already connected and authenticated to the Dynamic CRM instance: Loading data Once you have successfully connected to your Dynamics CRM organization, the Query functionality of the Power BI Designer runs against the Dynamics CRM Online instance, and the navigator window returns a list of Dynamics CRM data entities to include in your dashboard. By default, when you load the data to Power BI, all the items will be selected in the navigator window. To individually select multiple items, you will have to check the box to select multiple items. Here's how you do it: Navigate to the top-left corner of the navigator screen and locate the checkbox labeled Select Multiple items. Once this checkbox is ticked, the subarea will include additional checkboxes to individually select the Dynamics CRM data entities. Select the following items from the navigator window:      AccountSet      LeadSet      OpportunitySet      SystemUserSet The right-hand side of the navigator window will show you a preview of the fields included and data currently in the dataset. Select Show Selected to see a queried list of just the dataset selected: Querying the data Our next step is to query the Dynamics CRM data that we will load to Power BI. We can do this by editing the query before we load the data or come back later and update the query. By querying only the data we need from Dynamics CRM before loading to Power BI, we can enhance the performance of our reports and dashboards. Here is how it is done: Select Edit Query from the bottom-right hand side of the window; a view of the entity data is presented in Query view: A view of the entity data To modify the query after you load the data, in the top ribbon select View | Show | Query Settings to access the Query Settings pane. Select Source in the Query Settings window to update the query entity data: In the left-hand side pane, queries are listed and available for selection, viewing, and shaping. In the main pane, data from the selected query is displayed and available for shaping. Summary In this article, we looked at how to set up our Office 365, Dynamics CRM, and Power BI environments. We deployed the Power BI Designer and connected Dynamics CRM to Power BI in order to retrieve the sales entity data for our sales productivity dashboard. Resources for Article: Further resources on this subject: Customization in Microsoft Dynamics CRM [article] Getting Started with Microsoft Dynamics CRM 2013 Marketing [article] Overview of Microsoft Dynamics CRM 2011 [article]

In this article by Rafik Naccache, author of the book Clojure Data Structures and Algorithms Cookbook, we will be implementing a reusable mini-firewall using transducers. (For more resources related to this topic, see here.) Clojure's unique way of describing data transformations, reminiscent of its lisp and functional heritage, has set a new standard in the art of designing highly expressive algorithms. Clojure makes you address your problems in terms of highly declarative multi-stage transformations, and more often than not, you'll find your self alternating map, reduce, filter, and likely operations on the powerful seq abstraction to express the solution you came with as if you were explaining it in plain English to some non IT-savvy person. This declarative way of thinking yields much expressive power, and just looking at how SQL is ruling the database industry nowadays confirms that. But there was room for improvement in what Clojure provided for defining compoundable data transformations. First of all, the transformations you were writing were not reusable. Indeed, you'd catch yourself writing the same map operations over and over again, for instance, had you to do that same operation on different types of collections. Of course, you could encapsulate these in functions, but that will not avoid the second problem we wanted to highlight: the intermediate seq structure generated between the transformation stages. As a matter of fact, each and every step your threading macro passes through yields a new seq function, which is not the most efficient possible processing sequence. Lastly, the transformations you specified were closely tied to the type of input or output they operated on, and that gave the Clojure programming language designers the hassle of redefining all of these operations for any new data abstraction they'd come with while evolving the language. For all of these reasons, transducers were introduced starting from Clojure 1.7. As the official documentation put it, they are composable algorithmic transformations. They are independent from the context of their input and output sources and specify only the essence of the transformation in terms of an individual element. Transducers compose directly, without awareness of input or creation of intermediate aggregates. In a nutshell, a transducer is a "transformation from one reducing function to another", that is, a means to modify how to handle the way receiving the elements of some previous step is carried out. For instance, say we define the following transducer (note that transducers are, for the most part, the usual seq manipulation functions, but with an arity decremented by one): (map inc) We created a transformation that increments all the elements that some previous stage handed over to this transformation. Let's see an example of that (note the use of transduce function to apply a transducer on a seq function): (transduce (map inc) conj (range 0 3)) (range 0 3) hands a seq function of integers over to the (map inc) transducer, which on receiving them, gets them incremented and then, as per the transduce function specification, reduces them using the conj function. Transducers are also compoundable. You can achieve this using the traditional comp operator, used for traditional function composition, but there's one subtlety you must be aware of. Generally, comp is applied left to right, like so: (comp f g ) => f(g(x)) In the case of transducers, however, comp yields a transformation that looks as if composition was applied from right to left. This is not a change in the way comp works, but just how transducers are built. Composing transducers yields a function that changes the inner reducing function, as in the following example: (comp tr1 tr2)=> (fn[] ….. tr2(tr1)) So tr1 is applied on the input before tr2, although comp is still applied from left to right! To get a deeper understanding of transducer internals and how exactly they happen to be composed from right to left, watch Rich Hickey's talk about the subject at https://www.youtube.com/watch?v=6mTbuzafcII. We are going to use transducers to build a mini-firewall, implementing a two-stage data-transformation, one map and one filter stages, and are going to see how this mini-firewall can be interchangeably used on a vector or on a core.async channel. How to do it… First of all, be sure to use Clojure 1.7. Here is the Leiningen dependency to include: [org.clojure/clojure "1.7.0-RC1"] Besides, we are going to use some core.async channels in this recipe. Here is the namespace declaration: (ns recipe24.core (:require [clojure.java.io :refer :all] [clojure.core.async :refer [chan go >! <! >!! <!!]])) Now, let's define our first transducer, applying a source NAT on the TCP frames that pass through our firewall: (defn source-nat [pub-ip tcp-frame] ;; Change source ip into the public IP Interface (assoc tcp-frame :source-ip pub-ip)) (def public-interface "1.2.3.4") ;; This is our public interface IP (def tr-source-nat (map (partial source-nat public-interface))) ;; A transducer transforming tcp frames in such a way ;; That Source IP contains the public interface's. After that, let's concern ourselves with discarding or accepting TCP frames whose source, destination IPs, and ports are invalidated according to some connection whitelist: (defn accepted? [accept-rules tcp-frame] (not (nil? (some #{tcp-frame} accept-rules)))) ;; Verify if this TCP frame exists within ;; the accept-rules ACL (def sample-accept-rules [{:source-ip "4.5.3.8" :dest-ip "4.4.3.5" :dest-port 80} {:source-ip "4.5.3.9" :dest-ip "4.4.3.4" :dest-port 80}]) (def tr-filter-rules (filter (partial accepted? sample-accept-rules))) ;; A transducer dropping tcp frames not present ;; in our sample ACL Now we build our mini firewall, a transducer resulting from the composition of the previous two transducers. As we'll verify conformity of the connections before proceeding to the source IP NAT on them, we make sure that tr-filter-rules comes first in our composition: (def firewall(comp tr-filter-rules tr-source-nat)) Let's try our mini-firewall on a vector of TCP frames: (def sample-frames [{:source-ip "1.1.1.1" :dest-ip "2.3.2.2" :dest-port 10} {:source-ip "2.2.2.2" :dest-ip "4.3.4.1" :dest-port 20} {:source-ip "4.5.3.8" :dest-ip "4.4.3.5" :dest-port 30} {:source-ip "4.5.3.9" :dest-ip "4.4.3.4" :dest-port 80}]) (transduce firewall conj sample-frames) ;; => [{:source-ip "1.2.3.4", :dest-ip "4.4.3.4", :dest-port 80}] And now we'll show how our mini-firewall is reusable. For this, we are going to create a random-TCP frame generator that'll throw some traffic into a core.async channel. But this would not be just any channel; it would be one with a transducer, our firewall, attached to it. We'll see what happens when we try to read from it. First of all, let's write the random TCP frames generator: (def source-hosts [ "4.5.3.8" "4.5.3.9" "1.1.1.1" "2.2.2.2" ]) (def dest-hosts [ "4.4.3.5" "4.4.3.4" "2.3.2.2" "4.3.4.1" ]) (def ports [ 80]) (defn get-random-elt [v] (get v (rand-int (dec (count v))))) (defn random-frame [] {:source-ip (get-random-elt source-hosts) :dest-ip (get-random-elt dest-hosts) :dest-port (get-random-elt ports)}) Now it's time to create a core.async channel, to which we'll attach the firewall transducer. Note that when we attach a transducer to a core.async channel, the former must be buffered: (def from-net (chan 10 firewall)) We now need to throw, from time to time, a random TCP frame inside that channel: (defn gen-frames! [] (go (while true (let [fr (random-frame)] (>! from-net fr) (Thread/sleep 1000))))) We also have to build some function to print the TCP frames that were allowed to pass by the firewall: (defn show-fw-output! [] (while true (println "accepted & NAT'ted : " (<!! from-net)))) Let's see what happens when we launch the last two functions: (gen-frames!) (show-fw-output!) accepted & NATted : {:source-ip 1.2.3.4, :dest-ip 4.4.3.4, :dest-port 80} accepted & NATted : {:source-ip 1.2.3.4, :dest-ip 4.4.3.5, :dest-port 80} accepted & NATted : {:source-ip 1.2.3.4, :dest-ip 4.4.3.5, :dest-port 80} accepted & NATted : {:source-ip 1.2.3.4, :dest-ip 4.4.3.5, :dest-port 80} ... As you can see, we could only read through the channel "NAT'ted" TCP frames that have been granted access according to sample-access-rules. Actually, the transducer attached to the core.async channel transforms the data as they are flowing in, and we were able to do that by reusing our transducer specification without having to re-implement it specifically for that particular input type, that is, the channel. Summary In this article, we used transducers to build a mini-firewall, by implementing a two-stage data-transformation; one map and one filter stages, and saw how this mini-firewall can be interchangeably used on a vector or on a core.async channel. Resources for Article: Further resources on this subject: The Observer Pattern [article] Working with Incanter Datasets [article] Clojure for Domain-specific Languages - Design Concepts with Clojure [article]

The path to mastering performance in Python has just started. Profiling only takes us half way there. Measuring how our program is using the resources at its disposal only tells us where the problem is, not how to fix it. In this article by Fernando Doglio, author of the book Mastering Python High Performance, we will cover the process of optimization, and to do that, we need to start with the basics. We'll keep it inside the language for now: no external tools, just Python and the right way to use it. We will cover the following topics in this article: Memoization / lookup tables Usage of default arguments (For more resources related to this topic, see here.) Memoization / lookup tables This is one of the most common techniques used to improve the performance of a piece of code (namely a function). We can save the results of expensive function calls associated to a specific set of input values and return the saved result (instead of redoing the whole computation) when the function is called with the remembered input. It might be confused with caching, since it is one case of it, although this term refers also, to other types of optimization (such as HTTP caching, buffering, and so on) This methodology is very powerful, because in practice, it'll turn what should have been a potentially very expensive call into a O(1) function call if the implementation is right. Normally, the parameters are used to create a unique key, which is then used on a dictionary to either save the result or obtain it if it's been already saved. There is, of course, a trade-off to this technique. If we're going to be remembering the returned values of a memoized function, then we'll be exchanging memory space for speed. This is a very acceptable trade-off, unless the saved data becomes more than what the system can handle. Classic use cases for this optimization are function calls that repeat the input parameters often. This will assure that most of the times, the memoized results are returned. If there are many function calls but with different parameters, we'll only store results and spend our memory without any real benefit, as seen in the following diagram: You can clearly see how the blue bar (Fixed params, memoized) is clearly the fastest use case, while the others are all similar due to their nature. Here is the code that generates values for the preceding chart. To generate some sort of time-consuming function, the code will call either the twoParams function or the twoParamsMemoized function several hundred times under different conditions, and it will log the execution time: import math import time import random class Memoized: def __init__(self, fn): self.fn = fn self.results = {} def __call__(self, *args): key = ''.join(map(str, args[0])) try: return self.results[key] except KeyError: self.results[key] = self.fn(*args) return self.results[key] @Memoized def twoParamsMemoized(values, period): totalSum = 0 for x in range(0, 100): for v in values: totalSum = math.pow((math.sqrt(v) * period), 4) + totalSum return totalSum def twoParams(values, period): totalSum = 0 for x in range(0, 100): for v in values: totalSum = math.pow((math.sqrt(v) * period), 4) + totalSum return totalSum def performTest(): valuesList = [] for i in range(0, 10): valuesList.append(random.sample(xrange(1, 101), 10)) start_time = time.clock() for x in range(0, 10): for values in valuesList: twoParamsMemoized(values, random.random()) end_time = time.clock() - start_time print "Fixed params, memoized: %s" % (end_time) start_time = time.clock() for x in range(0, 10): for values in valuesList: twoParams(values, random.random()) end_time = time.clock() - start_time print "Fixed params, without memoizing: %s" % (end_time) start_time = time.clock() for x in range(0, 10): for values in valuesList: twoParamsMemoized(random.sample(xrange(1,2000), 10), random.random()) end_time = time.clock() - start_time print "Random params, memoized: %s" % (end_time) start_time = time.clock() for x in range(0, 10): for values in valuesList: twoParams(random.sample(xrange(1,2000), 10), random.random()) end_time = time.clock() - start_time print "Random params, without memoizing: %s" % (end_time) performTest() The main insight to take from the preceding chart is that just like with every aspect of programming, there is no silver bullet algorithm that will work for all cases. Memoization is clearly a very basic way of optimizing code, but clearly, it won't optimize anything given the right circumstances. As for the code, there is not much to it. It is a very simple, non real-world example of the point I was trying to send across. The performTest function will take care of running a series of 10 tests for every use case and measure the total time each use case takes. Notice that we're not really using profilers at this point. We're just measuring time in a very basic and ad-hoc way, which works for us. The input for both functions is simply a set of numbers on which they will run some math functions, just for the sake of doing something. The other interesting bit about the arguments is that, since the first argument is a list of numbers, we can't just use the args parameter as a key inside the Memoized class' methods. This is why we have the following line: key = ''.join(map(str, args[0])) This line will concatenate all the numbers from the first parameter into a single value, which will act as the key. The second parameter is not used here, because it's always random, which would imply that the key would never be the same. Another variation of the preceding method is to pre-calculate all values from the function (assuming we have a limited number of inputs of course) during initialization and then refer to the lookup table during execution. This approach has several preconditions: The number of input values must be finite; otherwise it's impossible to precalculate everything The lookup table with all of its values must fit into memory Just like before, the input must be repeated, at least once, so the optimization both makes sense and is worth the extra effort There are different approaches when it comes to architecting the lookup table, all offering different types of optimizations. It all depends on the type of application and solution that you're trying to optimize. Here is a set of examples. Lookup on a list or linked list This solution works by iterating over an unsorted list and checking the key against each element, with the associated value as the result we're looking for. This is obviously a very slow method of implementation, with a big O notation of O(n) for both the average and worst case scenarios. Still, given the right circumstances, it could prove to be faster than calling the actual function every time. In this case, using a linked list would improve the performance of the algorithm over using a simple list. However, it would still depend heavily on the type of linked list it is (doubly linked list, simple linked list with direct access to the first and last elements, and so on). Simple lookup on a dictionary This method works using a one-dimensional dictionary lookup, indexed by a key consisting of the input parameters (enough of them create a unique key). In particular cases (like we covered earlier), this is probably one of the fastest lookups, even faster than binary search in some cases with a constant execution time (big O notation of O(1)). Note that this approach is efficient as long as the key-generation algorithm is capable of generating unique keys every time. Otherwise, the performance could degrade over time due to the many collisions on the dictionaries. Binary search This particular method is only possible if the list is sorted. This could potentially be an option depending on the values to sort. Yet, sorting them would require an extra effort that would hurt the performance of the entire effort. However, it presents very good results even in long lists (average big O notation of O(log n)). It works by determining in which half of the list the value is and repeating until either the value is found or the algorithm is able to determine that the value is not in the list. To put all of this into perspective, looking at the Memoized class mentioned earlier, it implements a simple lookup on a dictionary. However, this would be the place to implement either of the other algorithms. Use cases for lookup tables There are some classic example use cases for this type of optimization, but the most common one is probably the optimization of trigonometric functions. Based on the computing time, these functions are really slow. When used repeatedly, they can cause some serious damage to your program's performance. This is why it is normally recommended to precalculate the values of these functions. For functions that deal with an infinite domain universe of possible input values, this task becomes impossible. So, the developer is forced to sacrifice accuracy for performance by precalculating a discrete subdomain of the possible input values (that is, going from floating points down to integer numbers). This approach might not be ideal in some cases, since some systems require both performance and accuracy. So, the solution is to meet in the middle and use some form of interpolation to calculate the wanted value, based on the ones that have been precalculated. It will provide better accuracy. Even though it won't be as performant as using the lookup table directly, it should prove to be faster than doing the trigonometric calculation every time. Let's look at some examples of this, for instance, for the following trigonometric function: def complexTrigFunction(x): return math.sin(x) * math.cos(x)**2 We'll take a look at how simple precalculation won't be accurate enough and how some form of interpolation will result in a better level of accuracy. The following code will precalculate the values for the function on a range from -1000 to 1000 (only integer values). Then, it'll try to do the same calculation (only for a smaller range) for floating point numbers: import math import time from collections import defaultdict import itertools trig_lookup_table = defaultdict(lambda: 0) def drange(start, stop, step): assert(step != 0) sample_count = math.fabs((stop - start) / step) return itertools.islice(itertools.count(start, step), sample_count) def complexTrigFunction(x): return math.sin(x) * math.cos(x)**2 def lookUpTrig(x): return trig_lookup_table[int(x)] for x in range(-1000, 1000): trig_lookup_table[x] = complexTrigFunction(x) trig_results = [] lookup_results = [] init_time = time.clock() for x in drange(-100, 100, 0.1): trig_results.append(complexTrigFunction(x)) print "Trig results: %s" % (time.clock() - init_time) init_time = time.clock() for x in drange(-100, 100, 0.1): lookup_results.append(lookUpTrig(x)) print "Lookup results: %s" % (time.clock() - init_time) for idx in range(0, 200): print "%st%s" % (trig_results [idx], lookup_results[idx]) The results from the preceding code will help demonstrate how the simple lookup table approach is not accurate enough (see the following chart). However, it compensates for it on speed since the original function takes 0.001526 seconds to run while the lookup table only takes 0.000717 seconds. The preceding chart shows how the lack of interpolation hurts the accuracy. You can see how even though both plots are quite similar, the results from the lookup table execution aren't as accurate as the trig function used directly. So, now, let's take another look at the same problem. However, this time, we'll add some basic interpolation (we'll limit the rage of values from -PI to PI): import math import time from collections import defaultdict import itertools trig_lookup_table = defaultdict(lambda: 0) def drange(start, stop, step): assert(step != 0) sample_count = math.fabs((stop - start) / step) return itertools.islice(itertools.count(start, step), sample_count) def complexTrigFunction(x): return math.sin(x) * math.cos(x)**2 reverse_indexes = {} for x in range(-1000, 1000): trig_lookup_table[x] = complexTrigFunction(math.pi * x / 1000) complex_results = [] lookup_results = [] init_time = time.clock() for x in drange(-10, 10, 0.1): complex_results .append(complexTrigFunction(x)) print "Complex trig function: %s" % (time.clock() - init_time) init_time = time.clock() factor = 1000 / math.pi for x in drange(-10 * factor, 10 * factor, 0.1 * factor): lookup_results.append(trig_lookup_table[int(x)]) print "Lookup results: %s" % (time.clock() - init_time) for idx in range(0, len(lookup_results )): print "%st%s" % (complex_results [idx], lookup_results [idx]) As you might've noticed in the previous chart, the resulting plot is periodic (specially because we've limited the range from -PI to PI). So, we'll focus on a particular range of values that will generate one single segment of the plot. The output of the preceding script also shows that the interpolation solution is still faster than the original trigonometric function, although not as fast as it was earlier: Interpolation Solution Original function 0.000118 seconds 0.000343 seconds The following chart is a bit different from the previous one, especially because it shows (in green) the error percentage between the interpolated value and the original one: The biggest error we have is around 12 percent (which represents the peaks we see on the chart). However, it's for the smallest values, such as -0.000852248551417 versus -0.000798905501416. This is a case where the error percentage needs to be contextualized to see if it really matters. In our case, since the values related to that error are so small, we can ignore that error in practice. There are other use cases for lookup tables, such as in image processing. However, for the sake of this article, the preceding example should be enough to demonstrate their benefits and trade-off implied in their usage. Usage of default arguments Another optimization technique, one that is contrary to memoization, is not particularly generic. Instead, it is directly tied to how the Python interpreter works. Default arguments can be used to determine values once at function creation time instead of at run time. This can only be done for functions or objects that will not be changed during program execution. Let's look at an example of how this optimization can be applied. The following code shows two versions of the same function, which does some random trigonometric calculation: import math #original function def degree_sin(deg): return math.sin(deg * math.pi / 180.0) #optimized function, the factor variable is calculated during function creation time, #and so is the lookup of the math.sin method. def degree_sin(deg, factor=math.pi/180.0, sin=math.sin): return sin(deg * factor) This optimization can be problematic if not correctly documented. Since it uses attributes to precompute terms that should not change during the program's execution, it could lead to the creation of a confusing API. With a quick and simple test, we can double-check the performance gain from this optimization: import time import math def degree_sin(deg): return math.sin(deg * math.pi / 180.0) * math.cos(deg * math.pi / 180.0) def degree_sin_opt(deg, factor=math.pi/180.0, sin=math.sin, cos = math.cos): return sin(deg * factor) * cos(deg * factor) normal_times = [] optimized_times = [] for y in range(100): init = time.clock() for x in range(1000): degree_sin(x) normal_times.append(time.clock() - init) init = time.clock() for x in range(1000): degree_sin_opt(x) optimized_times.append(time.clock() - init) print "Normal function: %s" % (reduce(lambda x, y: x + y, normal_times, 0) / 100) print "Optimized function: %s" % (reduce(lambda x, y: x + y, optimized_times, 0 ) / 100) The preceding code measures the time it takes for the script to finish each of the versions of the function to run its code 1000 times. It saves those measurements, and finally, it does an average for each case. The result is displayed in the following chart: It clearly isn't an amazing optimization. However, it does shave off some microseconds from our execution time, so we'll keep it in mind. Just remember that this optimization could cause problems if you're working as part of an OS developer team. Summary In this article, we covered several optimization techniques. Some of them are meant to provide big boosts on speed, save memory. Some of them are just meant to provide minor speed improvements. Most of this article covered Python-specific techniques, but some of them can be translated into other languages as well. Resources for Article: Further resources on this subject: How to do Machine Learning with Python [article] The Essentials of Working with Python Collections [article] Symbolizers [article]

The strange relationship between objects, functions, generators and coroutines In this article, I’d like to investigate some relationships between functions, objects, generators and coroutines in Python. At a theoretical level, these are very different concepts, but because of Python’s dynamic nature, many of them can appear to be used interchangeably. I discuss useful applications of all of these in my book, Python 3 Object-oriented Programming - Second Edition. In this essay, we’ll examine their relationship in a more whimsical light; most of the code examples below are ridiculous and should not be attempted in a production setting! Let’s start with functions, which are simplest. A function is an object that can be executed. When executed, the function is entered at one place accepting a group of (possibly zero) objects as parameters. The function exits at exactly one place and always returns a single object. Already we see some complications; that last sentence is true, but you might have several counter-questions: What if a function has multiple return statements? Only one of them will be executed in any one call to the function. What if the function doesn’t return anything? Then it it will implicitly return the None object. Can’t you return multiple objects separated by a comma? Yes, but the returned object is actually a single tuple Here’s a look at a function: def average(sequence): avg = sum(sequence) / len(sequence) return avg print(average([3, 5, 8, 7])) which outputs: That’s probably nothing you haven’t seen before. Similarly, you probably know what an object and a class are in Python. I define an object as a collection of data and associated behaviors. A class represents the “template” for an object. Usually the data is represented as a set of attributes and the behavior is represented as a collection of method functions, but this doesn’t have to be true. Here’s a basic object: class Statistics: def __init__(self, sequence): self.sequence = sequence def calculate_average(self): return sum(self.sequence) / len(self.sequence) def calculate_median(self): length = len(self.sequence) is_middle = int(not length % 2) return ( self.sequence[length // 2 - is_middle] + self.sequence[-length // 2]) / 2 statistics = Statistics([5, 2, 3]) print(statistics.calculate_average()) which outputs: This object has one piece of data attached to it: the sequence. It also has two methods besides the initializer. Only one of these methods is used in this particular example, but as Jack Diederich said in his famous Stop Writing Classes talk, a class with only one function besides the initializer should just be a function. So I included a second one to make it look like a useful class (It’s not. The new statistics module in Python 3.4 should be used instead. Never define for yourself that which has been defined, debugged, and tested by someone else). Classes like this are also things you’ve seen before, but with this background in place, we can now look at some bizarre things you might not expect (or indeed, want) to be able to do with a function. For example, did you know that functions are objects? In fact, anything that you can interact with in Python is defined in the source code for the CPython interpreter as a PyObject structure. This includes functions, objects, basic primitives, containers, classes, modules, you name it. This means we can attach attributes to a function just as with any standard object. Ah, but if functions are objects, can you attach functions to functions? Don’t try this at home (and especially don’t do it at work): def statistics(sequence): statistics.sequence = sequence return statistics def calculate_average(): return sum(statistics.sequence) / len(statistics.sequence) statistics.calculate_average = calculate_average print(statistics([1, 5, 8, 4]).calculate_average()) which outputs: This is a pretty crazy example (but we’re just getting started). The statistics function is being set up as an object that has two attributes: sequence is a list and calculate_average is another function object. For fun, the function returns itself so that the print function can call the calculate_average function all in one line. Note that the statistics function here is an object, not a class. Rather than emulating the Statistics class in the previous example, it is more similar to the statistics instance of that class. It is hard to imagine any reason that you would want to write code like this in real life. Perhaps it could be used to implement the Singleton (anti-)pattern popular with some other languages. Since there can only ever be one statistics function, it is not possible to create two distinct instances with two distinct sequence attributes the way we can with the Statistics class. There is generally little need for such code in Python, though, because of its ‘consenting adults’ nature. We can more closely simulate a class by using a function like a constructor: def Statistics(sequence): def self(): return self.average() self.sequence = sequence def average(): return sum(self.sequence) / len(self.sequence) self.average = average return self statistics = Statistics([2, 1, 1]) print(Statistics([1, 4, 6, 2]).average()) print(statistics()) which outputs: That looks an awful lot like Javascript, doesn’t it? The Statistics function acts like a constructor that returns an object (that happens to be a function, named self). That function object has had a couple attributes attached to it, so our function is now an object with both data and behavior. The last three lines show that we can instantiate two separate Statistics “objects” just as if it were a class. Finally, since the statistics object in the last line really is a function, we can even call it directly. It proxies the call through to the average function (or is it a method at this point? I can’t tell anymore) defined on itself. Before we go on, note that this simulated overlapping of functionality does not mean that we are getting exactly the same behavior out of the Python interpreter. While functions are objects, not all objects are functions. The underlying implementation is different, and if you try to do this in production code, you’ll quickly find confusing anomalies. In normal code, the fact that functions can have attributes attached to them is rarely useful. I’ve seen it used for interesting diagnostics or testing, but it’s generally just a hack. However, knowing that functions are objects allows us to pass them around to be called at a later time. Consider this basic partial implementation of an observer pattern: class Observers(list): register_observer = list.append def notify(self): for observer in self: observer() observers = Observers() def observer_one(): print('one was called') def observer_two(): print('two was called') observers.register_observer(observer_one) observers.register_observer(observer_two) observers.notify() which outputs: At line 2, I’ve intentionally reduced the comprehensibility of this code to conform to my initial ‘most of the examples in this article are ridiculous’ thesis. This line creates a new class attribute named register_observer which points to the list.append function. Since the Observers class inherits from the list class, this line essentially creates a shortcut to a method that would look like this: def register_observer(self, item): self.append(item) And this is how you should do it in your code. Nobody’s going to understand what’s going on if you follow my version. The part of this code that you might want to use in real life is the way the callback functions are passed into the registration function at lines 16 and 17. Passing functions around like this is quite common in Python. The alternative, if functions were not objects, would be to create a bunch of classes that have a single method with an uninformative name like execute and pass those around instead. Observers are a bit too useful, though, so in the spirit of keeping things ridiculous, let’s make a silly function that returns a function object: def silly(): print("silly") return silly silly()()()()()() which outputs: Since we’ve seen some ways that functions can (sort of) imitate objects, let’s now make an object that can behave like a function: class Function: def __init__(self, message): self.message = message def __call__(self, name): return "{} says '{}'".format(name, self.message) function = Function("I am a function") print(function('Cheap imitation function')) which outputs: I don’t use this feature often, but it can be useful in a few situations. For example, if you write a function and call it from many different places, but later discover that it needs to maintain some state, you can change the function to an object and implement the __call__ method without changing all the call sites. Or if you have a callback implementation that normally passes functions around, you can use a callable object when you need to store more complicated state. I’ve also seen Python decorators made out of objects when additional state or behavior is required. Now, let’s talk about generators. As you might expect by now, we’ll start with the silliest way to implement generation code. We can use the idea of a function that returns an object to create a rudimentary generatorish object that calculates the Fibonacci sequence: def FibFunction(): a = b = 1 def next(): nonlocal a, b a, b = b, a + b return b return next fib = FibFunction() for i in range(8): print(fib(), end=' ') which outputs: This is a pretty wacky thing to do, but the point is that it is possible to build functions that are able to maintain state between calls. The state is stored in the surrounding closure; we can access that state by referencing them with Python 3’s nonlocal keyword. It is kind of like global except it accesses the state from the surrounding function, not the global namespace. We can, of course, build a similar construct using classic (or classy) object notation: class FibClass(): def __init__(self): self.a = self.b = 1 def __call__(self): self.a, self.b = self.b, self.a + self.b return self.b fib = FibClass() for i in range(8): print(fib(), end=' ') which outputs: Of course, neither of these obeys the iterator protocol. No matter how I wrangle it, I was not able to get FibFunction to work with Python’s builtin next() function, even after looking through the CPython source code for a couple hours. As I mentioned earlier, using the function syntax to build pseudo-objects quickly leads to frustration. However, it’s easy to tweak the object based FibClass to fulfill the iterator protocol: class FibIterator(): def __init__(self): self.a = self.b = 1 def __next__(self): self.a, self.b = self.b, self.a + self.b return self.b def __iter__(self): return self fib = FibIterator() for i in range(8): print(next(fib), end=' ') which outputs: This class is a standard implementation of the iterator pattern. But it’s kind of ugly and verbose. Luckily, we can get the same effect in Python with a function that includes a yield statement to construct a generator. Here’s the Fibonacci sequence as a generator: ef FibGenerator(): a = b = 1 while True: a, b = b, a + b yield b fib = FibGenerator() for i in range(8): print(next(fib), end=' ') print('n', fib) which outputs: The generator version is a bit more readable than the other two implementations. The thing to pay attention to here is that a generator is not a function. The FibGenerator function returns an object as illustrated by the words “generator object” in the last line of output above. Unlike a normal function, a generator function does not execute any code inside it when we call the function. Instead it constructs a generator object and returns that. You could think of this as like an implicit Python decorator; the Python interpreter sees the yield keyword and wraps it in a decorator that returns an object instead. To start the function code executing, we have to use the next function (either explicitly as in the examples, or implicitly by using a for loop or yield from). While a generator is technically an object, it is often convenient to think of the function that creates it as a function that can have data passed in at one place and can return values multiple times. It’s sort of like a generic version of a function (which can have data passed in at one place and return a value at only one place). It is easy to make a generator that behaves not completely unlike a function, by yielding only one value: def average(sequence): yield sum(sequence) / len(sequence) print(next(average([1, 2, 3]))) which outputs: Unfortunately, the call site at line 4 is less readable than a normal function call, since we have to throw that pesky next() in there. The obvious way around this would be to add a __call__ method to the generator but this fails if we try to use attribute assignment or inheritance. There are optimizations that make generators run quickly in C code and also don’t let us assign attributes to them. We can, however, wrap the generator in a function-like object using a ludicrous decorator: def gen_func(func): def wrapper(*args, **kwargs): gen = func(*args, **kwargs) return next(gen) return wrapper @gen_func def average(sequence): yield sum(sequence) / len(sequence) print(average([1, 6, 3, 4])) which outputs: Of course this is an absurd thing to do. I mean, just write a normal function for pity’s sake! But taking this idea a little further, it could be tempting to create a slightly different wrapper: def callable_gen(func): class CallableGen: def __init__(self, *args, **kwargs): self.gen = func(*args, **kwargs) def __next__(self): return self.gen.__next__() def __iter__(self): return self def __call__(self): return next(self) return CallableGen @callable_gen def FibGenerator(): a = b = 1 while True: a, b = b, a + b yield b fib = FibGenerator() for i in range(8): print(fib(), end=' ') which outputs: To completely wrap the generator, we’d need to proxy a few other methods through to the underlying generator including send, close, and throw. This generator wrapper can be used to call a generator any number of times without calling the next function. I’ve been tempted to do this to make my code look cleaner if there are a lot of next calls in it, but I recommend not yielding into this temptation. Coders reading your code, including yourself, will go berserk trying to figure out what that “function call” is doing. Just get used to the next function and ignore this decorator business. So we’ve drawn some parallels between generators, objects and functions. Let’s talk now about one of the more confusing concepts in Python: coroutines. In Python, coroutines are usually defined as “generators that you can send values into”. At an implementation level, this is probably the most sensible definition. In the theoretical sense, however, it is more accurate to define coroutines as constructs that can accept values at one or more locations and return values at one or more locations. Therefore, while in Python it is easy to think of a generator as a special type of function that has yield statements and a coroutine as a special type of generator that we can send data into a different points, a better taxonomy is to think of a coroutine that can accept and return values at multiple locations as a general case, and generators and functions as special types of coroutines that are restricted in where they can accept or return values. So let’s see a coroutine: def LineInserter(lines): out = [] for line in lines: to_append = yield line out.append(line) if to_append is not None: out.append(to_append) return out emily = """I died for beauty, but was scarce Adjusted in the tomb, When one who died for truth was lain In an adjoining room. He questioned softly why I failed? “For beauty,” I replied. “And I for truth,—the two are one; We brethren are,” he said. And so, as kinsmen met a night, We talked between the rooms, Until the moss had reached our lips, And covered up our names. """ inserter = LineInserter(iter(emily.splitlines())) count = 1 try: line = next(inserter) while True: line = next(inserter) if count % 4 else inserter.send('-------') count += 1 except StopIteration as ex: print('n' + 'n'.join(ex.value)) which outputs: The LineInserter object is called a coroutine rather than a generator only because the yield statement is placed on the right side of an assignment operator. Now whenever we yield a line, it stores any value that might have been sent back into the coroutine in the to_append variable. As you can see in the driver code, we can send a value back in using inserter.send. if you instead just use next, the to_append variable gets a value of None. Don’t ask me why next is a function and send is a method when they both do nearly the same thing! In this example, we use the send call to insert a ruler every four lines to separate stanzas in Emily Dickinson’s famous poem. But I used the exact same coroutine in a program that parses the source file for this article. It checks if any line contains the string !#python, and if so, it executes the subsequent code block and inserts the output (see the ‘which outputs’ lines throughout this article) into the article. Coroutines can provide that little extra something when normal ‘one way’ iteration doesn’t quite cut it. The coroutine in the last example is really nice and elegant, but I find the driver code a bit annoying. I think it’s just me, but something about the indentation of a try…catch statement always frustrates me. Recently, I’ve been emulating Python 3.4’s contextlib.suppress context manager to replace except clauses with a callback. For example: def LineInserter(lines): out = [] for line in lines: to_append = yield line out.append(line) if to_append is not None: out.append(to_append) return out from contextlib import contextmanager @contextmanager def generator_stop(callback): try: yield except StopIteration as ex: callback(ex.value) def lines_complete(all_lines): print('n' + 'n'.join(all_lines)) emily = """I died for beauty, but was scarce Adjusted in the tomb, When one who died for truth was lain In an adjoining room. He questioned softly why I failed? “For beauty,” I replied. “And I for truth,—the two are one; We brethren are,” he said. And so, as kinsmen met a night, We talked between the rooms, Until the moss had reached our lips, And covered up our names. """ inserter = LineInserter(iter(emily.splitlines())) count = 1 with generator_stop(lines_complete): line = next(inserter) while True: line = next(inserter) if count % 4 else inserter.send('-------') count += 1 which outputs: The generator_stop now encapsulates all the ugliness, and the context manager can be used in a variety of situations where StopIteration needs to be handled. Since coroutines are undifferentiated from generators, they can emulate functions just as we saw with generators. We can even call into the same coroutine multiple times as if it were a function: def IncrementBy(increment): sequence = yield while True: sequence = yield [i + increment for i in sequence] sequence = [10, 20, 30] increment_by_5 = IncrementBy(5) increment_by_8 = IncrementBy(8) next(increment_by_5) next(increment_by_8) print(increment_by_5.send(sequence)) print(increment_by_8.send(sequence)) print(increment_by_5.send(sequence)) which outputs: Note the two calls to next at lines 9 and 10. These effectively “prime” the generator by advancing it to the first yield statement. Then each call to send essentially looks like a single call to a function. The driver code for this coroutine doesn’t look anything like calling a function, but with some evil decorator magic, we can make it look less disturbing: def evil_coroutine(func): def wrapper(*args, **kwargs): gen = func(*args, **kwargs) next(gen) def gen_caller(arg=None): return gen.send(arg) return gen_caller return wrapper @evil_coroutine def IncrementBy(increment): sequence = yield while True: sequence = yield [i + increment for i in sequence] sequence = [10, 20, 30] increment_by_5 = IncrementBy(5) increment_by_8 = IncrementBy(8) print(increment_by_5(sequence)) print(increment_by_8(sequence)) print(increment_by_5(sequence)) which outputs: The decorator accepts a function and returns a new wrapper function that gets assigned to the IncrementBy variable. Whenever this new IncrementBy is called, it constructs a generator using the original function, and advances it to the first yield statement using next (the priming action from before). It returns a new function that calls send on the generator each time it is called. This function makes the argument default to None so that it can also work if we call next instead of send. The new driver code is definitely more readable, but once again, I would not recommend using this coding style to make coroutines behave like hybrid object/functions. The argument that other coders aren’t going to understand what is going through your head still stands. Plus, since send can only accept one argument, the callable is quite restricted. Before we leave our discussion of the bizarre relationships between these concepts, let’s look at how the stanza processing code could look without an explicit coroutine, generator, function, or object: emily = """I died for beauty, but was scarce Adjusted in the tomb, When one who died for truth was lain In an adjoining room. He questioned softly why I failed? “For beauty,” I replied. “And I for truth,—the two are one; We brethren are,” he said. And so, as kinsmen met a night, We talked between the rooms, Until the moss had reached our lips, And covered up our names. """ for index, line in enumerate(emily.splitlines(), start=1): print(line) if not index % 4: print('------') which outputs: This code is so simple and elegant! This happens to me nearly every time I try to use coroutines. I keep refactoring and simplifying it until I discover that coroutines are making my code less, not more, readable. Unless I am explicitly modeling a state-transition system or trying to do asynchronous work using the terrific asyncio library (which wraps all the possible craziness with StopIteration, cascading exceptions, etc), I rarely find that coroutines are the right tool for the job. That doesn’t stop me from attempting them though, because they are fun. For the record, the LineInserter coroutine actually is useful in the markdown code executor I use to parse this source file. I need to keep track of more transitions between states (am I currently looking for a code block? am I in a code block? Do I need to execute the code block and record the output?) than in the stanza marking example used here. So, it has become clear that in Python, there is more than one way to do a lot of things. Luckily, most of these ways are not very obvious, and there is usually “one, and preferably only one, obvious way to do things”, to quote The Zen Of Python. I hope by this point that you are more confused about the relationship between functions, objects, generators and coroutines than ever before. I hope you now know how to write much code that should never be written. But mostly I hope you’ve enjoyed your exploration of these topics. If you’d like to see more useful applications of these and other Python concepts, grab a copy of Python 3 Object-oriented Programming, Second Edition.