How-To Tutorials

In this article by Ciro Cardoso, author of the book Lumion3D Best Practices, we will see materials, and why they are essential. In the 3D world, materials and textures are nearly as important as the 3D geometry that composes the scene. A material defines the optical properties of an object when hit by a ray of light. In other words, a material defines how the light interacts with the surface, and textures can help not only to control the color (diffuse), but also the reflections and glossiness. (For more resources related to this topic, see here.) It's not difficult to understand that textures are another essential part of a good material, and if your goal is to achieve believable results, you need textures or images of real elements like stone, wood, brick, and other natural elements. Textures can bring detail to your surface that otherwise would require geometry to look good. In that case, how can Lumion help you and, most importantly, what are the best practices to work with materials? Let's have a look at the following section which will provide the answer. A quick overview of Lumion's materials Lumion always had a good library of materials to assign to your 3D model, The reality is that Physically-Based Rendering (PBR) is more of a concept than a set of rules, and each render engine will implement slightly differently. The good news for us as users is that these materials follow realistic shading and lighting systems to accurately represent real-world materials. You can find excellent information regarding PBR on the following sites: http://www.marmoset.co/toolbag/learn/pbr-theory http://www.marmoset.co/toolbag/learn/pbr-practice https://www.allegorithmic.com/pbr-guide More than 600 materials are already prepared to be assigned directly to your 3D model and, by default, they should provide a realistic and appropriate material. The Lumion team has also made an effort to create a better and simpler interface, as you can see in the following screenshot: The interface was simplified, showing only the most common and essential settings. If you need more control over the material, click on the More… button to have access to extra functionalities. One word of caution: the material preview, which in this case is the sphere, will not reflect the changes you perform using the settings available. For example, if you change the main texture, the sphere will continue to show the previous material. A good practice to tweak materials is to assign the material to the surface, use the viewport to check how the settings are affecting the material, and then do a quick render. The viewport will try to show the final result, but there's nothing like a quick render to see how the material really looks when Lumion does all the lighting and shading calculations. Working with materials in Lumion – three options There are three options to work with materials in Lumion: Using Lumion's materials Using the imported materials that you can create on your favorite modeling application Creating materials using Lumion's Standard material Let's have a look at each one of these options and see how they can help you and when they best suit your project. Using Lumion's materials The first option is obvious; you are using Lumion and it makes sense using Lumion's materials, but you may feel constrained by what is available at Lumion's material library. However, instead of thinking, "I only have 600 materials and I cannot find what I need!", you need to look at the materials library also as a template to create other materials. For example, if none of the brick materials is similar to what you need, nothing stops you from using a brick material, changing the Gloss and Reflection values, and loading a new texture, creating an entirely new material. This is made possible by using the Choose Color Map button, as shown in the following screenshot: When you click on the Choose Color Map button, a new window appears where you can navigate to the folder where the texture is saved. What about the second square? The one with a purple color? Let's see the answer in the following section. Normal maps and their functionality The purple square you just saw is where you can load the normal map. And what is a normal map? Firstly, a normal map is not a bump map. A bump map uses a color range from black to white and in some ways is more limited than a normal map. The following screenshots show the clear difference between these two maps: The map on the left is a bump map and you can see that the level of detail is not the same that we can get with a normal map. A normal map consists of red, green, and blue colors that represent the x, y, and z coordinates. This allows a 2D image to represent depth and Lumion uses this depth information to fake lighting details based on the color associated with the 3D coordinate. The perks of using a normal map Why should you use normal maps? Keep in mind that Lumion is a real-time rendering engine and, as you saw previously, there is the need to keep a balance between detail and geometry. If you add too much detail, the 3D model will look gorgeous but Lumion's performance will suffer drastically. On the other hand, you can have a low-poly 3D model and fake detail with a normal map. Using a normal map for each material has a massive impact on the final quality you can get with Lumion. Since these maps are so important, how can you create one? Tips to create normal maps As you will understand, we cannot cover all the different techniques to create normal maps. However, you may find something to suit your workflow in the following list: Photoshop using an action script called nDo: Teddy Bergsman is the author of this fantastic script. It is a free script that creates a very accurate normal map of any texture you load in Photoshop in seconds. To download and see how to install this script, visit the following link: http://www.philipk.net/ndo.html Here you can find a more detailed tutorial on how to use this nDo script: http://www.philipk.net/tutorials/ndo/ndo.html This script has three options to create normal maps. The default option is Smooth, which gives you a blurry normal map. Then you have the Chisel Hard option to generate a very sharp and subtle normal map but you don't have much control over the final result. The Chisel Soft option is similar to the Chisel Hard except that you have full control over the intensity and bevel radius. This script also allows you to sculpt and combine several normal maps. Using the Quixel NDO application: From the same creator, we have a more capable and optimized application called Quixel NDO. With this application, you can sculpt normal maps in real-time, build your own normal maps without using textures, and preview everything with the 3DO material preview. This is quite useful because you don't have to save the normal map and see how it looks in Lumion. 3DO (which comes free with NDO) has a physically based renderer and lets you load a 3D model to see how the texture looks. Find more information including a free trial here: http://quixel.se/dev/ndo GIMP with the normalmap plugin: If you want to use free software, a good alternative is GIMP. There is a great plugin called normalmap, which does good work not only by creating a normal map but also by providing a preview window to see the tweaks you are making. To download this plugin, visit the following link: https://code.google.com/p/gimp-normalmap/ Do it online with NormalMap-Online: In case you don't want to install another application, the best option is doing it online. In that case, you may want to have a look at NormalMap-Online, as shown in the following screenshot: The process is extremely simple as you can see from the preceding screenshot. You load the image and automatically get a normal map, and on the right-hand side there is a preview to show how the normal map and the texture work together. Christian Petry is the man behind this tool that will help to create sharp and accurate normal maps. He is a great guy and if you like this online application, please consider supporting an application that will save you time and money. Find this online tool here: http://cpetry.github.io/NormalMap-Online/ Don't forget to use a good combination of Strength and Blur/Sharp to create a well-balanced map. You need the correct amount of detail; otherwise your normal map will be too noisy in terms of detail. However, Lumion being a user-friendly application gives you a hand on this topic by providing a tool to create a normal map automatically from a texture you import. Creating a normal map with Lumion's relief feature By now, creating a normal map from a texture is not something too technical or even complex, but it can be time consuming if you need to create a normal map for each texture. This is a wise move because it will remove the need for extra detail for the model to look good. With this in mind, Lumion's team created a new feature that allows you to create a normal map for any texture you import. After loading the new texture, the next step is to click on the Create Normal Map button, as highlighted in the following screenshot: Lumion then creates a normal map based on the texture imported, and you have the ability to invert the map by clicking on the Flip Normal Map direction button, as highlighted in the preceding screenshot. Once Lumion creates the normal map, you need a way to control how the normal map affects the material and the light. For that, you need to use the Relief slider, as shown in the following screenshot: Using this slider is very intuitive; you only need to move the slider and see the adjustments on the viewport, since the material preview will not be updated. The previous screenshot is a good example of that, because even when we loaded a wood texture, the preview still shows a concrete material. Again, this means you can easily use the settings from one material and use that as a base to create something completely new. But how good is the normal map that Lumion creates for you? Have a look for yourself in the following screenshot: On the left hand side, we have a wood floor material with a normal map that Lumion created. The right-hand side image is the same material but the normal map was created using the free nDo script for Photoshop. There is a big difference between the image on the left and the image on the right, and that is related to the normal maps used in this case. You can see clearly that the normal map used for the image on the right achieves the goal of bringing more detail to the surface. The difference is that the normal map that Lumion creates in some situations is too blurry, and for that reason we end up losing detail. Before we explore a few more things regarding creating custom materials in Lumion, let's have a look at another useful feature in Lumion. Summary Physically based rendering materials aren't that scary, don't you agree? In reality, Lumion makes this feature almost unnoticeable by making it so simple. You learned what this feature involves and how you can take full advantage of materials that make your render more believable. You learned the importance of using normal maps and how to create them using a variety of tools for all flavors. You also saw how we can easily improve material reflections without compromising the speed and quality of the render. You also learned another key aspect of Lumion: flexibility to create your own materials using the Standard material. The Standard material, although slightly different from the other materials available in Lumion, lets you play with the reflections, glossiness, and other settings that are essential. On top of all of this, you learned how to create textures. Resources for Article: Further resources on this subject: Unleashing the powers of Lumion [article] Mastering Lumion 3D [article] What is Lumion? [article]

In this article Integrating Google Play Services by Raul Portales, author of the book Mastering Android Game Development, we will cover the tools that Google Play Services offers for game developers. We'll see the integration of achievements and leaderboards in detail, take an overview of events and quests, save games, and use turn-based and real-time multiplaying. Google provides Google Play Services as a way to use special features in apps. Being the game services subset the one that interests us the most. Note that Google Play Services are updated as an app that is independent from the operating system. This allows us to assume that most of the players will have the latest version of Google Play Services installed. (For more resources related to this topic, see here.) More and more features are being moved from the Android SDK to the Play Services because of this. Play Services offer much more than just services for games, but there is a whole section dedicated exclusively to games, Google Play Game Services (GPGS). These features include achievements, leaderboards, quests, save games, gifts, and even multiplayer support. GPGS also comes with a standalone app called "Play Games" that shows the user the games he or she has been playing, the latest achievements, and the games his or her friends play. It is a very interesting way to get exposure for your game. Even as a standalone feature, achievements and leaderboards are two concepts that most games use nowadays, so why make your own custom ones when you can rely on the ones made by Google? GPGS can be used on many platforms: Android, iOS and web among others. It is more used on Android, since it is included as a part of Google apps. There is extensive step-by-step documentation online, but the details are scattered over different places. We will put them together here and link you to the official documentation for more detailed information. For this article, you are supposed to have a developer account and have access to the Google Play Developer Console. It is also advisable for you to know the process of signing and releasing an app. If you are not familiar with it, there is very detailed official documentation at http://developer.android.com/distribute/googleplay/start.html. There are two sides of GPGS: the developer console and the code. We will alternate from one to the other while talking about the different features. Setting up the developer console Now that we are approaching the release state, we have to start working with the developer console. The first thing we need to do is to get into the Game services section of the console to create and configure a new game. In the left menu, we have an option labeled Game services. This is where you have to click. Once in the Game services section, click on Add new game: This bring us to the set up dialog. If you are using other Google services like Google Maps or Google Cloud Messaging (GCM) in your game, you should select the second option and move forward. Otherwise, you can just fill in the fields for I don't use any Google APIs on my game yet and continue. If you don't know whether you are already using them, you probably aren't. Now, it is time to link a game to it. I recommend you publish your game beforehand as an alpha release. This will let you select it from the list when you start typing the package name. Publishing the game to the alpha channel before adding it to Game services makes it much easier to configure. If you are not familiar with signing and releasing your app, check out the official documentation at http://developer.android.com/tools/publishing/app-signing.html. Finally, there are only two steps that we have to take when we link the first app. We need to authorize it and provide branding information. The authorization will generate an OAuth key—that we don't need to use since it is required for other platforms—and also a game ID. This ID is unique to all the linked apps and we will need it to log in. But there is no need to write it down now, it can be found easily in the console at anytime. Authorizing the app will generate the game ID, which is unique to all linked apps. Note that the app we have added is configured with the release key. If you continue and try the login integration, you will get an error telling you that the app was signed with the wrong certificate: You have two ways to work with this limitation: Always make a release build to test GPGS integration Add your debug-signed game as a linked app I recommend that you add the debug signed app as a linked app. To do this, we just need to link another app and configure it with the SHA1 fingerprint of the debug key. To obtain it, we have to open a terminal and run the keytool utility: keytool -exportcert -alias androiddebugkey -keystore <path-to-debug-keystore> -list -v Note that in Windows, the debug keystore can be found at C:Users<USERNAME>.androiddebug.keystore. On Mac and Linux, the debug keystore is typically located at ~/.android/debug.keystore. Dialog to link the debug application on the Game Services console Now, we have the game configured. We could continue creating achievements and leaderboards in the console, but we will put it aside and make sure that we can sign in and connect with GPGS. The only users who can sign in to GPGS while a game is not published are the testers. You can make the alpha and/or beta testers of a linked app become testers of the game services, and you can also add e-mail addresses by hand for this. You can modify this in the Testing tab. Only test accounts can access a game that is not published. The e-mail of the owner of the developer console is prefilled as a tester. Just in case you have problems logging in, double-check the list of testers. A game service that is not published will not appear in the feed of the Play Services app, but it will be possible to test and modify it. This is why it is a good idea to keep it in draft mode until the game itself is ready and publish both the game and the game services at the same time. Setting up the code The first thing we need to do is to add the Google Play Services library to our project. This should already have been done by the wizard when we created the project, but I recommend you to double-check it now. The library needs to be added to the build.gradle file of the main module. Note that Android Studio projects contain a top-level build.gradle and a module-level build.gradle for each module. We will modify the one that is under the mobile module. Make sure that the play services' library is listed under dependencies: apply plugin: 'com.android.application'     dependencies { compile 'com.android.support:appcompat-v7:22.1.1' compile 'com.google.android.gms:play-services:7.3.0' } At the point of writing, the latest version is 7.3.0. The basic features have not changed much and they are unlikely to change. You could force Gradle to use a specific version of the library, but in general I recommend you use the latest version. Once you have it, save the changes and click on Sync Project with Gradle Files. To be able to connect with GPGS, we need to let the game know what the game ID is. This is done through the <meta-data> tag on AndroidManifest.xml. You could hardcode the value here, but it is highly recommended that you set it as a resource in your Android project. We are going to create a new file for this under res/values, which we will name play_services.xml. In this file we will put the game ID, but later we will also have the achievements and leaderboard IDs in it. Using a separate file for these values is recommended because they are constants that do not need to be translated: <application> <meta-data android_name="com.google.android.gms.games.APP_ID" android_value="@string/app_id" /> <meta-data android_name="com.google.android.gms.version" android_value="@integer/google_play_services_version"/> [...] </application> Adding this metadata is extremely important. If you forget to update the AndroidManifest.xml, the app will crash when you try to sign in to Google Play services. Note that the integer for the gms version is defined in the library and we do not need to add it to our file. If you forget to add the game ID to the strings the app will crash. Now, it is time to proceed to sign in. The process is quite tedious and requires many checks, so Google has released an open source project named BaseGameUtils, which makes it easier. Unfortunately this project is not a part of the play services' library and it is not even available as a library. So, we have to get it from GitHub (either check it out or download the source as a ZIP file). BaseGameUtils abstracts us from the complexity of handling the connection with Play Services. Even more cumbersome, BaseGameUtils is not available as a standalone download and has to be downloaded together with another project. The fact that this significant piece of code is not a part of the official library makes it quite tedious to set up. Why it has been done like this is something that I do not comprehend myself. The project that contains BaseGameUtils is called android-basic-samples and it can be downloaded from https://github.com/playgameservices/android-basic-samples. Adding BaseGameUtils is not as straightforward as we would like it to be. Once android-basic-samples is downloaded, open your game project in Android Studio. Click on File > Import Module and navigate to the directory where you downloaded android-basic-samples. Select the BaseGameUtils module in the BasicSamples/libraries directory and click on OK. Finally, update the dependencies in the build.gradle file for the mobile module and sync gradle again: dependencies { compile project(':BaseGameUtils') [...] } After all these steps to set up the project, we are finally ready to begin the sign in. We will make our main Activity extend from BaseGamesActivity, which takes care of all the handling of the connections, and sign in with Google Play Services. One more detail: until now, we were using Activity and not FragmentActivity as the base class for YassActivity (BaseGameActivity extends from FragmentActivity) and this change will mess with the behavior of our dialogs while calling navigateBack. We can change the base class of BaseGameActivity or modify navigateBack to perform a pop-on fragment navigation hierarchy. I recommend the second approach: public void navigateBack() { // Do a pop on the navigation history getFragmentManager().popBackStack(); } This util class has been designed to work with single-activity games. It can be used in multiple activities, but it is not straightforward. This is another good reason to keep the game in a single activity. The BaseGameUtils is designed to be used in single-activity games. The default behavior of BaseGameActivity is to try to log in each time the Activity is started. If the user agrees to sign in, the sign in will happen automatically. But if the user rejects doing so, he or she will be asked again several times. I personally find this intrusive and annoying, and I recommend you to only prompt to log in to Google Play services once (and again, if the user logs out). We can always provide a login entry point in the app. This is very easy to change. The default number of attempts is set to 3 and it is a part of the code of GameHelper: // Should we start the flow to sign the user in automatically on   startup? If // so, up to // how many times in the life of the application? static final int DEFAULT_MAX_SIGN_IN_ATTEMPTS = 3; int mMaxAutoSignInAttempts = DEFAULT_MAX_SIGN_IN_ATTEMPTS; So, we just have to configure it for our activity, adding one line of code during onCreate to change the default behavior with the one we want: just try it once: getGameHelper().setMaxAutoSignInAttempts(1); Finally, there are two methods that we can override to act when the user successfully logs in and when there is a problem: onSignInSucceeded and onSignInFailed. We will use them when we update the main menu at the end of the article. Further use of GPGS is to be made via the GameHelper and/or the GoogleApiClient, which is a part of the GameHelper. We can obtain a reference to the GameHelper using the getGameHelper method of BaseGameActivity. Now that the user can sign into Google Play services we can continue with achievements and leaderboards. Let's go back to the developer console. Achievements We will first define a few achievements in the developer console and then see how to unlock them in the game. Note that to publish any game with GPGS, you need to define at least five achievements. No other feature is mandatory, but achievements are. We need to define at least five achievements to publish a game with Google Play Game services. If you want to use GPGS with a game that has no achievements, I recommend you to add five dummy secret achievements and let them be. To add an achievement, we just need to navigate to the Achievements tab on the left and click on Add achievement: The menu to add a new achievement has a few fields that are mostly self-explanatory. They are as follows: Name: the name that will be shown (can be localized to different languages). Description: the description of the achievement to be shown (can also be localized to different languages). Icon: the icon of the achievement as a 512x512 px PNG image. This will be used to show the achievement in the list and also to generate the locked image and the in-game popup when it is unlocked. Incremental achievements: if the achievement requires a set of steps to be completed, it is called an incremental achievement and can be shown with a progress bar. We will have an incremental achievement to illustrate this. Initial state: Revealed/Hidden depending on whether we want the achievement to be shown or not. When an achievement is shown, the name and description are visible, players know what they have to do to unlock it. A hidden achievement, on the other hand, is a secret and can be a funny surprise when unlocked. We will have two secret achievements. Points: GPGS allows each game to have 1,000 points to give for unlocking achievements. This gets converted to XP in the player profile on Google Play games. This can be used to highlight that some achievements are harder than others, and therefore grant a bigger reward. You cannot change these once they are published, so if you plan to have more achievements in the future, plan ahead with the points. List order: The order of the achievements is shown. It is not followed all the time, since on the Play Games app the unlocked ones are shown before the locked ones. It is still handy to rearrange them. Dialog to add an achievement on the developer console As we already decided, we will have five achievements in our game and they will be as follows: Big Score: score over 100,000 points in one game. This is to be granted while playing. Asteroid killer: destroy 100 asteroids. This will count them across different games and is an incremental achievement. Survivor: survive for 60 seconds. Target acquired: a hidden achievement. Hit 20 asteroids in a row without missing a hit. This is meant to reward players that only shoot when they should. Target lost: this is supposed to be a funny achievement, granted when you miss with 10 bullets in a row. It is also hidden, because otherwise it would be too easy to unlock. So, we created some images for them and added them to the console. The developer console with all the configured achievements Each achievement has a string ID. We will need these ids to unlock the achievements in our game, but Google has made it easy for us. We have a link at the bottom named Get resources that pops up a dialog with the string resources we need. We can just copy them from there and paste them in our project in the play_services.xml file we have already created. Architecture For our game, given that we only have five achievements, we are going to add the code for achievements directly into the ScoreObject. This will make it less code for you to read so we can focus on how it is done. However, for a real production code I recommend you define a dedicated architecture for achievements. The recommended architecture is to have an AchievementsManager class that loads all the achievements when the game starts and stores them in three lists: All achievements Locked achievements Unlocked achievements Then, we have an Achievement base class with an abstract check method that we implement for each one of them: public boolean check (GameEngine gameEngine, GameEvent gameEvent) { } This base class takes care of loading the achievement state from local storage (I recommend using SharedPreferences for this) and modify it as per the result of check. The achievements check is done at AchievementManager level using a checkLockedAchievements method that iterates over the list of achievements that can be unlocked. This method should be called as a part of onEventReceived of GameEngine. This architecture allows you to check only the achievements that are yet to be unlocked and also all the achievements included in the game in a specific dedicated place. In our case, since we are keeping the score inside the ScoreGameObject, we are going to add all achievements code there. Note that making the GameEngine take care of the score and having it as a variable that other objects can read are also recommended design patterns, but it was simpler to do this as a part of ScoreGameObject. Unlocking achievements To handle achievements, we need to have access to an object of the class GoogleApiClient. We can get a reference to it in the constructor of ScoreGameObject: private final GoogleApiClient mApiClient;   public ScoreGameObject(YassBaseFragment parent, View view, int viewResId) { […] mApiClient =  parent.getYassActivity().getGameHelper().getApiClient(); } The parent Fragment has a reference to the Activity, which has a reference to the GameHelper, which has a reference to the GoogleApiClient. Unlocking an achievement requires just a single line of code, but we also need to check whether the user is connected to Google Play services or not before trying to unlock an achievement. This is necessary because if the user has not signed it, an exception is thrown and the game crashes. Unlocking an achievement requires just a single line of code. But this check is not enough. In the edge case, when the user logs out manually from Google Play services (which can be done in the achievements screen), the connection will not be closed and there is no way to know whether he or she has logged out. We are going to create a utility method to unlock the achievements that does all the checks and also wraps the unlock method into a try/catch block and make the API client disconnect if an exception is raised: private void unlockSafe(int resId) { if (mApiClient.isConnecting() || mApiClient.isConnected()) {    try {      Games.Achievements.unlock(mApiClient, getString(resId));    } catch (Exception e) {      mApiClient.disconnect();    } } } Even with all the checks, the code is still very simple. Let's work on the particular achievements we have defined for the game. Even though they are very specific, the methodology to track game events and variables and then check for achievements to unlock is in itself generic, and serves as a real-life example of how to deal with achievements. The achievements we have designed require us to count some game events and also the running time. For the last two achievements, we need to make a new GameEvent for the case when a bullet misses, which we have not created until now. The code in the Bullet object to trigger this new GameEvent is as follows: @Override public void onUpdate(long elapsedMillis, GameEngine gameEngine) { mY += mSpeedFactor * elapsedMillis; if (mY < -mHeight) {    removeFromGameEngine(gameEngine);    gameEngine.onGameEvent(GameEvent.BulletMissed); } } Now, let's work inside ScoreGameObject. We are going to have a method that checks achievements each time an asteroid is hit. There are three achievements that can be unlocked when that event happens: Big score, because hitting an asteroid gives us points Target acquired, because it requires consecutive asteroid hits Asteroid killer, because it counts the total number of asteroids that have been destroyed The code is like this: private void checkAsteroidHitRelatedAchievements() { if (mPoints > 100000) {    // Unlock achievement    unlockSafe(R.string.achievement_big_score); } if (mConsecutiveHits >= 20) {    unlockSafe(R.string.achievement_target_acquired); } // Increment achievement of asteroids hit if (mApiClient.isConnecting() || mApiClient.isConnected()) {    try {      Games.Achievements.increment(mApiClient, getString(R.string.achievement_asteroid_killer), 1);    } catch (Exception e) {      mApiClient.disconnect();    } } } We check the total points and the number of consecutive hits to unlock the corresponding achievements. The "Asteroid killer" achievement is a bit of a different case, because it is an incremental achievement. These type of achievements do not have an unlock method, but rather an increment method. Each time we increment the value, progress on the achievement is updated. Once the progress is 100 percent, it is unlocked automatically. Incremental achievements are automatically unlocked, we just have to increment their value. This makes incremental achievements much easier to use than tracking the progress locally. But we still need to do all the checks as we did for unlockSafe. We are using a variable named mConsecutiveHits, which we have not initialized yet. This is done inside onGameEvent, which is the place where the other hidden achievement target lost is checked. Some initialization for the "Survivor" achievement is also done here: public void onGameEvent(GameEvent gameEvent) { if (gameEvent == GameEvent.AsteroidHit) {    mPoints += POINTS_GAINED_PER_ASTEROID_HIT;    mPointsHaveChanged = true;    mConsecutiveMisses = 0;    mConsecutiveHits++;    checkAsteroidHitRelatedAchievements(); } else if (gameEvent == GameEvent.BulletMissed) {    mConsecutiveMisses++;    mConsecutiveHits = 0;    if (mConsecutiveMisses >= 20) {      unlockSafe(R.string.achievement_target_lost);    } } else if (gameEvent == GameEvent.SpaceshipHit) {    mTimeWithoutDie = 0; } […] } Each time we hit an asteroid, we increment the number of consecutive asteroid hits and reset the number of consecutive misses. Similarly, each time we miss a bullet, we increment the number of consecutive misses and reset the number of consecutive hits. As a side note, each time the spaceship is destroyed we reset the time without dying, which is used for "Survivor", but this is not the only time when the time without dying should be updated. We have to reset it when the game starts, and modify it inside onUpdate by just adding the elapsed milliseconds that have passed: @Override public void startGame(GameEngine gameEngine) { mTimeWithoutDie = 0; […] }   @Override public void onUpdate(long elapsedMillis, GameEngine gameEngine) { mTimeWithoutDie += elapsedMillis; if (mTimeWithoutDie > 60000) {    unlockSafe(R.string.achievement_survivor); } } So, once the game has been running for 60,000 milliseconds since it started or since a spaceship was destroyed, we unlock the "Survivor" achievement. With this, we have all the code we need to unlock the achievements we have created for the game. Let's finish this section with some comments on the system and the developer console: As a rule of thumb, you can edit most of the details of an achievement until you publish it to production. Once your achievement has been published, it cannot be deleted. You can only delete an achievement in its prepublished state. There is a button labeled Delete at the bottom of the achievement screen for this. You can also reset the progress for achievements while they are in draft. This reset happens for all players at once. There is a button labeled Reset achievement progress at the bottom of the achievement screen for this. Also note that GameBaseActivity does a lot of logging. So, if your device is connected to your computer and you run a debug build, you may see that it lags sometimes. This does not happen in a release build for which the log is removed. Leaderboards Since YASS has only one game mode and one score in the game, it makes sense to have only one leaderboard on Google Play Game Services. Leaderboards are managed from their own tab inside the Game services area of the developer console. Unlike achievements, it is not mandatory to have any leaderboard to be able to publish your game. If your game has different levels of difficulty, you can have a leaderboard for each of them. This also applies if the game has several values that measure player progress, you can have a leaderboard for each of them. Managing leaderboards on Play Games console Leaderboards can be created and managed in the Leaderboards tag. When we click on Add leaderboard, we are presented with a form that has several fields to be filled. They are as follows: Name: the display name of the leaderboard, which can be localized. We will simply call it High Scores. Score formatting: this can be Numeric, Currency, or Time. We will use Numeric for YASS. Icon: a 512x512 px icon to identify the leaderboard. Ordering: Larger is better / Smaller is better. We are going to use Larger is better, but other score types may be Smaller is better as in a racing game. Enable tamper protection: this automatically filters out suspicious scores. You should keep this on. Limits: if you want to limit the score range that is shown on the leaderboard, you can do it here. We are not going to use this List order: the order of the leaderboards. Since we only have one, it is not really important for us. Setting up a leaderboard on the Play Games console Now that we have defined the leaderboard, it is time to use it in the game. As happens with achievements, we have a link where we can get all the resources for the game in XML. So, we proceed to get the ID of the leaderboard and add it to the strings defined in the play_services.xml file. We have to submit the scores at the end of the game (that is, a GameOver event), but also when the user exits a game via the pause button. To unify this, we will create a new GameEvent called GameFinished that is triggered after a GameOver event and after the user exits the game. We will update the stopGame method of GameEngine, which is called in both cases to trigger the event: public void stopGame() { if (mUpdateThread != null) {    synchronized (mLayers) {      onGameEvent(GameEvent.GameFinished);    }    mUpdateThread.stopGame();  mUpdateThread = null; } […] } We have to set the updateThread to null after sending the event, to prevent this code being run twice. Otherwise, we could send each score more than once. Similarly, as happens for achievements, submitting a score is very simple, just a single line of code. But we also need to check that the GoogleApiClient is connected and we still have the same edge case when an Exception is thrown. So, we need to wrap it in a try/catch block. To keep everything in the same place, we will put this code inside ScoreGameObject: @Override public void onGameEvent(GameEvent gameEvent) { […] else if (gameEvent == GameEvent.GameFinished) {    // Submit the score    if (mApiClient.isConnecting() || mApiClient.isConnected()) {      try {        Games.Leaderboards.submitScore(mApiClient,          getLeaderboardId(), mPoints);      }      catch (Exception e){        mApiClient.disconnect();      }    } } }   private String getLeaderboardId() { return mParent.getString(R.string.leaderboard_high_scores); } This is really straightforward. GPGS is now receiving our scores and it takes care of the timestamp of the score to create daily, weekly, and all time leaderboards. It also uses your Google+ circles to show the social score of your friends. All this is done automatically for you. The final missing piece is to let the player open the leaderboards and achievements UI from the main menu as well as trigger a sign in if they are signed out. Opening the Play Games UI To complete the integration of achievements and leaderboards, we are going to add buttons to open the native UI provided by GPGS to our main menu. For this, we are going to place two buttons in the bottom–left corner of the screen, opposite the music and sound buttons. We will also check whether we are connected or not; if not, we will show a single sign-in button. For these buttons we will use the official images of GPGS, which are available for developers to use. Note that you must follow the brand guidelines while using the icons and they must be displayed as they are and not modified. This also provides a consistent look and feel across all the games that support Play Games. Since we have seen a lot of layouts already, we are not going to include another one that is almost the same as something we already have. The main menu with the buttons to view achievements and leaderboards. To handle these new buttons we will, as usual, set the MainMenuFragment as OnClickListener for the views. We do this in the same place as the other buttons, that is, inside onViewCreated: @Override public void onViewCreated(View view, Bundle savedInstanceState) { super.onViewCreated(view, savedInstanceState); [...] view.findViewById(    R.id.btn_achievements).setOnClickListener(this); view.findViewById(    R.id.btn_leaderboards).setOnClickListener(this); view.findViewById(R.id.btn_sign_in).setOnClickListener(this); } As happened with achievements and leaderboards, the work is done using static methods that receive a GoogleApiClient object. We can get this object from the GameHelper that is a part of the BaseGameActivity, like this: GoogleApiClient apiClient = getYassActivity().getGameHelper().getApiClient(); To open the native UI, we have to obtain an Intent and then start an Activity with it. It is important that you use startActivityForResult, since some data is passed back and forth. To open the achievements UI, the code is like this: Intent achievementsIntent = Games.Achievements.getAchievementsIntent(apiClient); startActivityForResult(achievementsIntent, REQUEST_ACHIEVEMENTS); This works out of the box. It automatically grays out the icons for the unlocked achievements, adds a counter and progress bar to the one that is in progress, and a padlock to the hidden ones. Similarly, to open the leaderboards UI we obtain an intent from the Games.Leaderboards class instead: Intent leaderboardsIntent = Games.Leaderboards.getLeaderboardIntent( apiClient, getString(R.string.leaderboard_high_scores)); startActivityForResult(leaderboardsIntent, REQUEST_LEADERBOARDS); In this case, we are asking for a specific leaderboard, since we only have one. We could use getLeaderboardsIntent instead, which will open the Play Games UI for the list of all the leaderboards. We can have an intent to open the list of leaderboards or a specific one. What remains to be done is to replace the buttons for the login one when the user is not connected. For this, we will create a method that reads the state and shows and hides the views accordingly: private void updatePlayButtons() { GameHelper gameHelper = getYassActivity().getGameHelper(); if (gameHelper.isConnecting() || gameHelper.isSignedIn()) {    getView().findViewById(      R.id.btn_achievements).setVisibility(View.VISIBLE);    getView().findViewById(      R.id.btn_leaderboards).setVisibility(View.VISIBLE);    getView().findViewById(      R.id.btn_sign_in).setVisibility(View.GONE); } else {    getView().findViewById(      R.id.btn_achievements).setVisibility(View.GONE);    getView().findViewById(      R.id.btn_leaderboards).setVisibility(View.GONE);    getView().findViewById(      R.id.btn_sign_in).setVisibility(View.VISIBLE); } } This method decides whether to remove or make visible the views based on the state. We will call it inside the important state-changing methods: onLayoutCompleted: the first time we open the game to initialize the UI. onSignInSucceeded: when the user successfully signs in to GPGS. onSignInFailed: this can be triggered when we auto sign in and there is no connection. It is important to handle it. onActivityResult: when we come back from the Play Games UI, in case the user has logged out. But nothing is as easy as it looks. In fact, when the user signs out and does not exit the game, GoogleApiClient keeps the connection open. Therefore the value of isSignedIn from GameHelper still returns true. This is the edge case we have been talking about all through the article. As a result of this edge case, there is an inconsistency in the UI that shows the achievements and leaderboards buttons when it should show the login one. When the user logs out from Play Games, GoogleApiClient keeps the connection open. This can lead to confusion. Unfortunately, this has been marked as work as expected by Google. The reason is that the connection is still active and it is our responsibility to parse the result in the onActivityResult method to determine the new state. But this is not very convenient. Since it is a rare case we will just go for the easiest solution, which is to wrap it in a try/catch block and make the user sign in if he or she taps on leaderboards or achievements while not logged in. This is the code we have to handle the click on the achievements button, but the one for leaderboards is equivalent: else if (v.getId() == R.id.btn_achievements) { try {    GoogleApiClient apiClient =      getYassActivity().getGameHelper().getApiClient();    Intent achievementsIntent =      Games.Achievements.getAchievementsIntent(apiClient);    startActivityForResult(achievementsIntent,      REQUEST_ACHIEVEMENTS); } catch (Exception e) {    GameHelper gameHelper = getYassActivity().getGameHelper();    gameHelper.disconnect();    gameHelper.beginUserInitiatedSignIn(); } } Basically, we have the old code to open the achievements activity, but we wrap it in a try/catch block. If an exception is raised, we disconnect the game helper and begin a new login using the beginUserInitiatedSignIn method. It is very important to disconnect the gameHelper before we try to log in again. Otherwise, the login will not work. We must disconnect from GPGS before we can log in using the method from the GameHelper. Finally, there is the case when the user clicks on the login button, which just triggers the login using the beginUserInitiatedSignIn method from the GameHelper: if (v.getId() == R.id.btn_sign_in) { getYassActivity().getGameHelper().beginUserInitiatedSignIn(); } Once you have published your game and the game services, achievements and leaderboards will not appear in the game description on Google Play straight away. It is required that "a fair amount of users" have used them. You have done nothing wrong, you just have to wait. Other features of Google Play services Google Play Game Services provides more features for game developers than achievements and leaderboards. None of them really fit the game we are building, but it is useful to know they exist just in case your game needs them. You can save yourself lots of time and effort by using them and not reinventing the wheel. The other features of Google Play Games Services are: Events and quests: these allow you to monitor game usage and progression. Also, they add the possibility of creating time-limited events with rewards for the players. Gifts: as simple as it sounds, you can send a gift to other players or request one to be sent to you. Yes, this is seen in the very mechanical Facebook games popularized a while ago. Saved games: the standard concept of a saved game. If your game has progression or can unlock content based on user actions, you may want to use this feature. Since it is saved in the cloud, saved games can be accessed across multiple devices. Turn-based and real-time multiplayer: Google Play Game Services provides an API to implement turn-based and real-time multiplayer features without you needing to write any server code. If your game is multiplayer and has an online economy, it may be worth making your own server and granting virtual currency only on the server to prevent cheating. Otherwise, it is fairly easy to crack the gifts/reward system and a single person can ruin the complete game economy. However, if there is no online game economy, the benefits of gifts and quests may be more important than the fact that someone can hack them. Let's take a look at each of these features. Events The event's APIs provides us with a way to define and collect gameplay metrics and upload them to Google Play Game Services. This is very similar to the GameEvents we are already using in our game. Events should be a subset of the game events of our game. Many of the game events we have are used internally as a signal between objects or as a synchronization mechanism. These events are not really relevant outside the engine, but others could be. Those are the events we should send to GPGS. To be able to send an event from the game to GPGS, we have to create it in the developer console first. To create an event, we have to go to the Events tab in the developer console, click on Add new event, and fill in the following fields: Name: a short name of the event. The name can be up to 100 characters. This value can be localized. Description: a longer description of the event. The description can be up to 500 characters. This value can also be localized. Icon: the icon for the event of the standard 512x512 px size. Visibility: as for achievements, this can be revealed or hidden. Format: as for leaderboards, this can be Numeric, Currency, or Time. Event type: this is used to mark events that create or spend premium currency. This can be Premium currency sink, Premium currency source, or None. While in the game, events work pretty much as incremental achievements. You can increment the event counter using the following line of code: Games.Events.increment(mGoogleApiClient, myEventId, 1); You can delete events that are in the draft state or that have been published as long as the event is not in use by a quest. You can also reset the player progress data for the testers of your events as you can do for achievements. While the events can be used as an analytics system, their real usefulness appears when they are combined with quests. Quests A quest is a challenge that asks players to complete an event a number of times during a specific time frame to receive a reward. Because a quest is linked to an event, to use quests you need to have created at least one event. You can create a quest from the quests tab in the developer console. A quest has the following fields to be filled: Name: the short name of the quest. This can be up to 100 characters and can be localized. Description: a longer description of the quest. Your quest description should let players know what they need to do to complete the quest. The description can be up to 500 characters. The first 150 characters will be visible to players on cards such as those shown in the Google Play Games app. Icon: a square icon that will be associated with the quest. Banner: a rectangular image that will be used to promote the quest. Completion Criteria: this is the configuration of the quest itself. It consists of an event and the number of times the event must occur. Schedule: the start and end date and time for the quest. GPGS uses your local time zone, but stores the values as UTC. Players will see these values appear in their local time zone. You can mark a checkbox to notify users when the quest is about to end. Reward Data: this is specific to each game. It can be a JSON object, specifying the reward. This is sent to the client when the quest is completed. Once configured in the developer console, you can do two things with the quests: Display the list of quests Process a quest completion To get the list of quests, we start an activity with an intent that is provided to us via a static method as usual: Intent questsIntent = Games.Quests.getQuestsIntent(mGoogleApiClient,    Quests.SELECT_ALL_QUESTS); startActivityForResult(questsIntent, QUESTS_INTENT); To be notified when a quest is completed, all we have to do is register a listener: Games.Quests.registerQuestUpdateListener(mGoogleApiClient, this); Once we have set the listener, the onQuestCompleted method will be called once the quest is completed. After completing the processing of the reward, the game should call claim to inform Play Game services that the player has claimed the reward. The following code snippet shows how you might override the onQuestCompleted callback: @Override public void onQuestCompleted(Quest quest) { // Claim the quest reward. Games.Quests.claim(mGoogleApiClient, quest.getQuestId(),    quest.getCurrentMilestone().getMilestoneId()); // Process the RewardData to provision a specific reward. String reward = new    String(quest.getCurrentMilestone().getCompletionRewardData(),    Charset.forName("UTF-8")); } The rewards themselves are defined by the client. As we mentioned before, this will make the game quite easy to crack and get rewards. But usually, avoiding the hassle of writing your own server is worth it. Gifts The gifts feature of GPGS allows us to send gifts to other players and to request them to send us one as well. This is intended to make the gameplay more collaborative and to improve the social aspect of the game. As for other GPGS features, we have a built-in UI provided by the library that can be used. In this case, to send and request gifts for in-game items and resources to and from friends in their Google+ circles. The request system can make use of notifications. There are two types of requests that players can send using the game gifts feature in Google Play Game Services: A wish request to ask for in-game items or some other form of assistance from their friends A gift request to send in-game items or some other form of assistance to their friends A player can specify one or more target request recipients from the default request-sending UI. A gift or wish can be consumed (accepted) or dismissed by a recipient. To see the gifts API in detail, you can visit https://developers.google.com/games/services/android/giftRequests. Again, as for quest rewards, this is done entirely by the client, which makes the game susceptible to piracy. Saved games The saved games service offers cloud game saving slots. Your game can retrieve the saved game data to allow returning players to continue a game at their last save point from any device. This service makes it possible to synchronize a player's game data across multiple devices. For example, if you have a game that runs on Android, you can use the saved games service to allow a player to start a game on their Android phone and then continue playing the game on a tablet without losing any of their progress. This service can also be used to ensure that a player's game play continues from where it was left off even if their device is lost, destroyed, or traded in for a newer model or if the game was reinstalled The saved games service does not know about the game internals, so it provides a field that is an unstructured binary blob where you can read and write the game data. A game can write an arbitrary number of saved games for a single player subjected to user quota, so there is no hard requirement to restrict players to a single save file. Saved games are done in an unstructured binary blob. The API for saved games also receives some metadata that is used by Google Play Games to populate the UI and to present useful information in the Google Play Game app (for example, last updated timestamp). Saved games has several entry points and actions, including how to deal with conflicts in the saved games. To know more about these check out the official documentation at https://developers.google.com/games/services/android/savedgames. Multiplayer games If you are going to implement multiplayer, GPGS can save you a lot of work. You may or may not use it for the final product, but it will remove the need to think about the server-side until the game concept is validated. You can use GPGS for turn-based and real-time multiplayer games. Although each one is completely different and uses a different API, there is always an initial step where the game is set up and the opponents are selected or invited. In a turn-based multiplayer game, a single shared state is passed among the players and only the player that owns the turn has permission to modify it. Players take turns asynchronously according to an order of play determined by the game. A turn is finished explicitly by the player using an API call. Then the game state is passed to the other players, together with the turn. There are many cases: selecting opponents, creating a match, leaving a match, canceling, and so on. The official documentation at https://developers.google.com/games/services/android/turnbasedMultiplayer is quite exhaustive and you should read through it if you plan to use this feature. In a real-time multiplayer there is no concept of turn. Instead, the server uses the concept of room: a virtual construct that enables network communication between multiple players in the same game session and lets players send data directly to one another, a common concept for game servers. Real-time multiplayer service is based on the concept of Room. The API of real-time multiplayer allows us to easily: Manage network connections to create and maintain a real-time multiplayer room Provide a player-selection user interface to invite players to join a room, look for random players for auto-matching, or a combination of both Store participant and room-state information on the Play Game services' servers while the game is running Send room invitations and updates to players To check the complete documentation for real-time games, please visit the official web at https://developers.google.com/games/services/android/realtimeMultiplayer. Summary We have added Google Play services to YASS, including setting up the game in the developer console and adding the required libraries to the project. Then, we defined a set of achievements and added the code to unlock them. We have used normal, incremental, and hidden achievement types to showcase the different options available. We have also configured a leaderboard and submitted the scores, both when the game is finished and when it is exited via the pause dialog. Finally, we have added links to the native UI for leaderboards and achievements to the main menu. We have also introduced the concepts of events, quests, and gifts and the features of saved games and multiplayer that Google Play Game services offers. The game is ready to publish now. Resources for Article: Further resources on this subject: SceneKit [article] Creating Games with Cocos2d-x is Easy and 100 percent Free [article] SpriteKit Framework and Physics Simulation [article]

 In this article by, Dharmesh Kakadia, author of the book Apache Mesos Essentials, explains how Mesos works internally in detail. We will start off with cluster scheduling and fairness concepts, understanding the Mesos architecture, and we will move on towards resource isolation and fault tolerance implementation in Mesos. In this article, we will cover the following topics: The Mesos architecture Resource allocation (For more resources related to this topic, see here.) The Mesos architecture Modern organizations have a lot of different kinds of applications for different business needs. Modern applications are distributed and they are deployed across commodity hardware. Organizations today run different applications in siloed environments, where separate clusters are created for different applications. This static partitioning of cluster leads to low utilization, and all the applications will duplicate the effort of dealing with distributed infrastructures. Not only is this a wasted effort, but it also undermines the fact that distributed systems are hard to build and maintain. This is challenging for both developers and operators. For developers, it is a challenge to build applications that scale elastically and can handle faults that are inevitable in large-scale environment. Operators, on the other hand, have to manage and scale all of these applications individually in siloed environments. The preceding situation is like trying to develop applications without having an operating system and managing all the devices in a computer. Mesos solves the problems mentioned earlier by providing a data center kernel. Mesos provides a higher-level abstraction to develop applications that treat distributed infrastructure just like a large computer. Mesos abstracts the hardware infrastructure away from the applications from the physical infrastructure. Mesos makes developers more productive by providing an SDK to easily write data center scale applications. Now, developers can focus on their application logic and do not have to worry about the infrastructure that runs it. Mesos SDK provides primitives to build large-scale distributed systems, such as resource allocation, deployment, and monitoring isolation. They only need to know and implement what resources are needed, and not how they get the resources. Mesos allows you to treat the data center just as a computer. Mesos makes the infrastructure operations easier by providing elastic infrastructure. Mesos aggregates all the resources in a single shared pool of resources and avoids static partitioning. This makes it easy to manage and increases the utilization. The data center kernel has to provide resource allocation, isolation, and fault tolerance in a scalable, robust, and extensible way. We will discuss how Mesos fulfills these requirements, as well as some other important considerations of modern data center kernel: Scalability: The kernel should be scalable in terms of the number of machines and number of applications. As the number of machines and applications increase, the response time of the scheduler should remain acceptable. Flexibility: The kernel should support a wide range of applications. It should also support diverse frameworks currently running on the cluster and future frameworks as well. The framework should also be able to cope up with the heterogeneity in the hardware, as most clusters are built over time and have a variety of hardware running. Maintainability: The kernel would be one of the very important pieces of modern infrastructure. As the requirements evolve, the scheduler should be able to accommodate new requirements. Utilization and dynamism: The kernel should adapt to the changes in resource requirements and available hardware resources and utilize resources in an optimal manner. Fairness: The kernel should be fair in allocating resources to the different users and/or frameworks. We will see what it means to be fair in detail in the next section. The design philosophy behind Mesos was to define a minimal interface to enable efficient resource sharing across frameworks and defer the task scheduling and execution to the frameworks. This allows the frameworks to implement diverse approaches toward scheduling and fault tolerance. It also makes the Mesos core simple, and the frameworks and core can evolve independently. The preceding figure shows the overall architecture (http://mesos.apache.org/documentation/latest/mesos-architecture) of a Mesos cluster. It has the following entities: The Mesos masters The Mesos slaves Frameworks Communication Auxiliary services We will describe each of these entities and their role, followed by how Mesos implements different requirements of the data center kernel. The Mesos slave The Mesos slaves are responsible for executing tasks from frameworks using the resources they have. The slave has to provide proper isolation while running multiple tasks. The isolation mechanism should also make sure that the tasks get resources that they are promised, and not more or less. The resources on slaves that are managed by Mesos can be described using slave resources and slave attributes. Resources are elements of slaves that can be consumed by a task, while we use attributes to tag slaves with some information. Slave resources are managed by the Mesos master and are allocated to different frameworks. Attributes identify something about the node, such as the slave having a specific OS or software version, it's part of a particular network, or it has a particular hardware, and so on. The attributes are simple key-value pairs of strings that are passed along with the offers to frameworks. Since attributes cannot be consumed by a running task, they will always be offered for that slave. Mesos doesn't understand the slave attribute, and interpretation of the attributes is left to the frameworks. More information about resource and attributes in Mesos can be found at https://mesos.apache.org/documentation/attributes-resources. A Mesos resource or attribute can be described as one of the following types: Scalar values are floating numbers Range values are a range of scalar values; they are represented as [minValue-maxValue] Set values are arbitrary strings Text values are arbitrary strings; they are applicable only to attributes Names of the resources can be an arbitrary string consisting of alphabets, numbers, "-", "/", ".", "-". The Mesos master handles the cpus, mem, disk, and ports resources in a special way. A slave without the cpus and mem resources will not be advertised to the frameworks. The mem and disk scalars are interpreted in MB. The ports resource is represented as ranges. The list of resources a slave has to offer to various frameworks can be specified as the resources flag. Resources and attributes are separated by a semicolon. For example: --resources='cpus:30;mem:122880;disk:921600;ports:[21000-29000];bugs:{a,b,c}' --attributes='rack:rack-2;datacenter:europe;os:ubuntuv14.4' This slave offers 30 cpus, 102 GB mem, 900 GB disk, ports from 21000 to 29000, and have bugs a, b, and c. The slave has three attributes: rack with value rack-2, datacenter with value europe, and os with value ubuntu14.4. Mesos does not yet provide direct support for GPUs, but does support custom resource types. This means that if we specify gpu(*):8 as part of --resources, then it will be part of the resource that offers to frameworks. Frameworks can use it just like other resources. Once some of the GPU resources are in use by a task, only the remaining resources will be offered. Mesos does not yet have support for GPU isolation, but it can be extended by implementing a custom isolator. Alternately, we can also specify which slaves have GPUs using attributes, such as --attributes="hasGpu:true". The Mesos master The Mesos master is primarily responsible for allocating resources to different frameworks and managing the task life cycle for them. The Mesos master implements fine-grained resource sharing using resource offers. The Mesos master acts as a resource broker for frameworks using pluggable policies. The master decides to offer cluster resources to frameworks in the form of resource offers based on them. Resources offer represents a unit of allocation in the Mesos world. It's a vector of resource available on a node. An offer represents some resources available on a slave being offered to a particular framework. Frameworks Distributed applications that run on top of Mesos are called frameworks. Frameworks implement the domain requirements using the general resource allocation API of Mesos. A typical framework wants to run a number of tasks. Tasks are the consumers of resources and they do not have to be the same. A framework in Mesos consists of two components: a framework scheduler and executors. Framework schedulers are responsible for coordinating the execution. An executor provides the ability to control the task execution. Executors can realize a task execution in many ways. An executor can choose to run multiple tasks, by spawning multiple threads, in an executor, or it can run one task in each executor. Apart from the life cycle and task management-related functions, the Mesos framework API also provides functions to communicate with framework schedulers and executors. Communication Mesos currently uses an HTTP-like wire protocol to communicate with the Mesos components. Mesos uses the libprocess library to implement the communication that is located in 3rdparty/libprocess. The libprocess library provides asynchronous communication with processes. The communication primitives have an actor message passing, such as semantics. The libprocess messages are immutable, which makes parallelizing the libprocess internals easier. Mesos communication happens along with the following APIs: Scheduler API: This is used to communicate with the framework scheduler and master. The internal communication is intended to be used only by the SchedulerDriver API. Executor API: This is used to communicate with an executor and the Mesos slave. Internal API: This is used to communicate with the Mesos master and slave. Operator API: This is the API exposed by Mesos for operators and is used by web UI, among other things. Unlike most Mesos API, the operator API is a synchronous API. To send a message, the actor does an HTTP POST request. The path is composed by the name of the actor followed by the name of the message. The User-Agent field is set to "libprocess/…" to distinguish from the normal HTTP requests. The message data is passed as the body of the HTTP request. Mesos uses protocol buffers to serialize all the messages (defined in src/messages/messages.proto). The parsing and interpretation of the message is left to the receiving actor. Here is an example header of a message sent to master to register the framework by scheduler(1) running at 10.0.1.7:53523 address: POST /master/mesos.internal.RegisterFrameworkMessage HTTP/1.1 User-Agent: libprocess/scheduler(1)@10.0.1.7:53523 The reply message header from the master that acknowledges the framework registration might look like this: POST /scheduler(1)/mesos.internal.FrameworkRegisteredMessage HTTP/1.1 User-Agent: libprocess/master@10.0.1.7:5050 At the time of writing, there is a very early discussion about rewiring the Mesos Scheduler API and Executor API as a pure HTTP API (https://issues.apache.org/jira/browse/MESOS-2288). This will make the API standard and integration with Mesos for various tools much easier without the need to be dependent on native libmesos. Also, there is an ongoing effort to convert all the internal messages into a standardized JSON or protocol buffer format (https://issues.apache.org/jira/browse/MESOS-1127). Auxiliary services Apart from the preceding main components, a Mesos cluster also needs some auxiliary services. These services are not part of Mesos itself, and are not strictly required, but they form a basis for operating the Mesos cluster in production environments. These services include, but are not limited to, the following: Shared filesystem: Mesos provides a view of the data center as a single computer and allows developers to develop for the data center scale application. With this unified view of resources, clusters need a shared filesystem to truly make the data center a computer. HDFS, NFS (Network File System), and Cloud-based storage options, such as S3, are popular among various Mesos deployments. Consensus service: Mesos uses a consensus service to be resilient in face of failure. Consensus services, such as ZooKeeper or etcd, provide a reliable leader election in a distributed environment. Service fabric: Mesos enables users to run a number of frameworks on unified computing resources. With a large number of applications and services running, it's important for users to be able to connect to them in a seamless manner. For example, how do users connect to Hive running on Mesos? How does the Ruby on Rails application discover and connect to the MongoDB database instances when one or both of them are running on Mesos? How is the website traffic routed to web servers running on Mesos? Answering these questions mainly requires service discovery and load balancing mechanisms, but also things such as IP/port management and security infrastructure. We are collectively referring to these services that connect frameworks to other frameworks and users as service fabric. Operational services: Operational services are essential in managing operational aspects of Mesos. Mesos deployments and upgrades, monitoring cluster health and alerting when human intervention is required, logging, and security are all part of the operational services that play a very important role in a Mesos cluster. Resource allocation As a data center kernel, Mesos serves a large variety of workloads and no single scheduler will be able to satisfy the needs of all different frameworks. For example, the way in which a real-time processing framework schedules its tasks will be very different from how a long running service will schedule its task, which, in turn, will be very different from how a batch processing framework would like to use its resources. This observation leads to a very important design decision in Mesos: separation of resource allocation and task scheduling. Resource allocation is all about deciding who gets what resources, and it is the responsibility of the Mesos master. Task scheduling, on the other hand, is all about how to use the resources. This is decided by various framework schedulers according to their own needs. Another way to understand this would be that Mesos handles coarse-grain resource allocation across frameworks, and then each framework does fine-grain job scheduling via appropriate job ordering to achieve its needs. The Mesos master gets information on the available resources from the Mesos slaves, and based on resource policies, the Mesos master offers these resources to different frameworks. Different frameworks can choose to accept or reject the offer. If the framework accepts a resource offer, the framework allocates the corresponding resources to the framework, and then the framework is free to use them to launch tasks. The following image shows the high-level flow of Mesos resource allocation: Mesos two level scheduler Here is the typical flow of events for one framework in Mesos: The framework scheduler registers itself with the Mesos master. The Mesos master receives the resource offers from slaves. It invokes the allocation module and decides which frameworks should receive the resource offers. The framework scheduler receives the resource offers from the Mesos master. On receiving the resource offers, the framework scheduler inspects the offer to decide whether it's suitable. If it finds it satisfactory, the framework scheduler accepts the offer and replies to the master with the list of executors that should be run on the slave, utilizing the accepted resource offers. Alternatively, the framework can reject the offer and wait for a better offer. The slave allocates the requested resources and launches the task executors. The executor is launched on slave nodes and runs the framework's tasks. It is up to the framework scheduler to accept or reject the resource offers. Here is an example of events that can happen when allocating resources. The framework scheduler gets notified about the task's completion or failure. The framework scheduler will continue receiving the resource offers and task reports and launch tasks as it sees fit. The framework unregisters with the Mesos master and will not receive any further resource offers. Note that this is optional and a long running service, and meta-framework will not unregister during the normal operation. Because of this design, Mesos is also known as a two-level scheduler. Mesos' two-level scheduler design makes it simpler and more scalable, as the resource allocation process does not need to know how scheduling happens. This makes the Mesos core more stable and scalable. Frameworks and Mesos are not tied to each other and each can iterate independently. Also, this makes porting frameworks easier. The choice of a two-level scheduler means that the scheduler does not have a global knowledge about resource utilization and the resource allocation decisions can be nonoptimal. One potential concern could be about the preferences that the frameworks have about the kind of resources needed for execution. Data locality, special hardware, and security constraints can be a few of the constraints on which tasks can run. In the Mesos realm, these preferences are not explicitly specified by a framework to the Mesos master, instead the framework rejects all the offers that do not meet its constraints. The Mesos scheduler Mesos was the first cluster scheduler to allow the sharing of resources to multiple frameworks. Mesos resource allocation is based on online Dominant Resource Fairness (DRF) called HierarchicalDRF. In a world of single resource static partitioning, fairness is easy to define. DRF extends this concept of fairness to multi-resource settings without the need for static partitioning. Resource utilization and fairness are equally important, and often conflicting, goals for a cluster scheduler. The fairness of resource allocation is important in a shared environment, such as data centers, to ensure that all the users/processes of the cluster get nearly an equal amount of resources. Min-max fairness provides a well-known mechanism to share a single resource among multiple users. Min-max fairness algorithm maximizes the minimum resources allocated to a user. In its simplest form, it allocates 1/Nth of the resource to each of the users. The weighted min-max fairness algorithm can also support priorities and reservations. Min-max resource fairness has been a basis for many well-known schedulers in operating systems and distributed frameworks, such as Hadoop's fair scheduler (http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html), capacity scheduler (https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html), Quincy scheduler (http://dl.acm.org/citation.cfm?id=1629601), and so on. However, it falls short when the cluster has multiple types of resources, such as the CPU, memory, disk, and network. When jobs in a distributed environment use different combinations of these resources to achieve the outcome, the fairness has to be redefined. For example, the two requests <1 CPU, 3 GB> and <3 CPU, 1 GB> come to the scheduler. How do they compare and what is the fair allocation? DRF generalizes the min-max algorithm for multiple resources. A user's dominant resource is the resource for which the user has a biggest share. For example, if the total resources are <8 CPU, 5 GB>, then for the user allocation of <2 CPU, 1 GB>, the user's dominant share is maximumOf(2/8,1/5) means CPU. A user's dominant share is the fraction of the dominant resource that's allocated to the user. In our example, it would be 25 percent (2/8). DRF applies the min-max algorithm to the dominant share of each user. It has many provable properties: Strategy proofness: A user cannot gain any advantage by lying about the demands. Sharing incentive: DRF has a minimum allocation guarantee for each user, and no user will prefer exclusive partitioned cluster of size 1/N over DRF allocation. Single resource fairness: In case of only one resource, DRF is equivalent to the min-max algorithm. Envy free: Every user prefers his allocation over any other allocation of other users. This also means that the users with the same requests get equivalent allocations. Bottleneck fairness: When one resource becomes a bottleneck, and each user has a dominant demand for it, DRF is equivalent to min-max. Monotonicity: Adding resources or removing users can only increase the allocation of the remaining users. Pareto efficiency: The allocation achieved by DRF will be pareto efficient, so it would be impossible to make a better allocation for any user without making allocation for some other user worse. We will not further discuss DRF but will encourage you to refer to the DRF paper for more details at http://static.usenix.org/event/nsdi11/tech/full_papers/Ghodsi.pdf. Mesos uses role specified in FrameworkInfo for resource allocation decision. A role can be per user or per framework or can be shared by multiple users and frameworks. If it's not set, Mesos will set it to the current user that runs the framework scheduler. An optimization is to use deny resource offers from particular slaves for a specified time period. Mesos can revoke tasks allocation killing those tasks. Before killing a task, Mesos gives the framework a grace period to clean up. Mesos asks the executor to kill the task, but if it does not oblige the request, it will kill the executor and all of its tasks. Weighted DRF DRF calculates each role's dominant share and allocates the available resources to the user with the smallest dominant share. In practice, an organization rarely wants to assign resources in a complete fair manner. Most organizations want to allocate resources in a weighted manner, such as 50 percent resources to ads team, 30 percent to QA, and 20 percent to R&D teams. To satisfy this command functionality, Mesos implements weighted DRF, where masters can be configured with weights for different roles. When weights are specified, a client's DRF share will be divided by the weight. For example, a role that has a weight of two will be offered twice as many resources as a role with weight of one. Mesos can be configured to use weighted DRF using the --weights and --roles flags on the master startup. The --weights flag expects a list of role/weight pairs in the form of role1=weight1 and role2=weight2. Weights do not need to be integers. We must provide weights for each role that appear in --roles on the master startup. Reservation One of the other most asked questions for requirement is the ability to reserve resources. For example, persistent or stateful services, such as memcache, or a database running on Mesos, would need a reservation mechanism to avoid being negatively affected on restart. Without reservation, memcache is not guaranteed to get a resource offer from the slave, which has all the data and would incur significant time in initialization and downtime for the service. Reservation can also be used to limit the resource per role. Reservation provides guaranteed resources for roles, but improper usage might lead to resource fragmentation and lower utilization of resources. Note that all the reservation requests go through a Mesos authorization mechanism to ensure that the operator or framework requesting the operation has the proper privileges. Reservation privileges are specified to the Mesos master through ACL along with the rest of the ACL configuration. Mesos supports the following two kinds of reservation: Static reservation Dynamic reservation Static reservation In static reservation, resources are reserved for a particular role. The restart of the slave after removing the checkpointed state is required to change static reservation. Static reservation is thus typically managed by operators using the --resources flag on the slave. The flag expects a list of name(role):value for different resources. If a resource is assigned to role A, then only frameworks with role A are eligible to get an offer for that resource. Any resources that do not include a role or resources that are not included in the --resources flag will be included in the default role (default *). For example, --resources="cpus:4;mem:2048;cpus(ads):8;mem(ads):4096" specifies that the slave has 8 CPUs and 4096 MB memory reserved for "ads" role and has 4 CPUs and 2048 MB memory unreserved. Nonuniform static reservation across slaves can quickly become difficult to manage. Dynamic reservation Dynamic reservation allows operators and frameworks to manage reservation more dynamically. Frameworks can use dynamic reservations to reserve offered resources, allowing those resources to only be reoffered to the same framework. At the time of writing, dynamic reservation is still being actively developed and is targeted toward the next release of Mesos (https://issues.apache.org/jira/browse/MESOS-2018). When asked for a reservation, Mesos will try to convert the unreserved resources to reserved resources. On the other hand, during the unreserve operation, the previously reserved resources are returned to the unreserved pool of resources. To support dynamic reservation, Mesos allows a sequence of Offer::Operations to be performed as a response to accepting resource offers. A framework manages reservation by sending Offer::Operations::Reserve and Offer::Operations::Unreserve as part of these operations, when receiving resource offers. For example, consider the framework that receives the following resource offer with 32 CPUs and 65536 MB memory: {   "id" : <offer_id>,   "framework_id" : <framework_id>,   "slave_id" : <slave_id>,   "hostname" : <hostname>,   "resources" : [     {       "name" : "cpus",       "type" : "SCALAR",       "scalar" : { "value" : 32 },       "role" : "*",     },     {       "name" : "mem",       "type" : "SCALAR",       "scalar" : { "value" : 65536 },       "role" : "*",     }   ] } The framework can decide to reserve 8 CPUs and 4096 MB memory by sending the Operation::Reserve message with resources field with the desired resources state: [   {     "type" : Offer::Operation::RESERVE,     "resources" : [       {         "name" : "cpus",         "type" : "SCALAR",         "scalar" : { "value" : 8 },         "role" : <framework_role>,         "reservation" : {           "framework_id" : <framework_id>,           "principal" : <framework_principal>         }       }       {         "name" : "mem",         "type" : "SCALAR",         "scalar" : { "value" : 4096 },         "role" : <framework_role>,         "reservation" : {           "framework_id" : <framework_id>,           "principal" : <framework_principal>         }       }     ]   } ] After a successful execution, the framework will receive resource offers with reservation. The next offer from the slave might look as follows: {   "id" : <offer_id>,   "framework_id" : <framework_id>,   "slave_id" : <slave_id>,   "hostname" : <hostname>,   "resources" : [     {       "name" : "cpus",       "type" : "SCALAR",       "scalar" : { "value" : 8 },       "role" : <framework_role>,       "reservation" : {         "framework_id" : <framework_id>,         "principal" : <framework_principal>       }     },     {       "name" : "mem",       "type" : "SCALAR",       "scalar" : { "value" : 4096 },       "role" : <framework_role>,       "reservation" : {         "framework_id" : <framework_id>,         "principal" : <framework_principal>       }     },     {       "name" : "cpus",       "type" : "SCALAR",       "scalar" : { "value" : 24 },       "role" : "*",     },     {       "name" : "mem",       "type" : "SCALAR",       "scalar" : { "value" : 61440 },       "role" : "*",     }   ] } As shown, the framework has 8 CPUs and 4096 MB memory reserved resources and 24 CPUs and 61440 MB memory underserved in the resource offer. The unreserve operation is similar. The framework on receiving the resource offer can send the unreserve operation message, and subsequent offers will not have reserved resources. The operators can use/reserve and/unreserve HTTP endpoints of the operator API to manage the reservation. The operator API allows operators to change the reservation specified when the slave starts. For example, the following command will reserve 4 CPUs and 4096 MB memory on slave1 for role1 with the operator authentication principal ops: ubuntu@master:~ $ curl -d slaveId=slave1 -d resources="{          {            "name" : "cpus",            "type" : "SCALAR",            "scalar" : { "value" : 4 },            "role" : "role1",            "reservation" : {              "principal" : "ops"            }          },          {            "name" : "mem",            "type" : "SCALAR",            "scalar" : { "value" : 4096 },            "role" : "role1",            "reservation" : {              "principal" : "ops"            }          },        }"        -X POST http://master:5050/master/reserve Before we end this discussion on resource allocation, it would be important to note that the Mesos community continues to innovate on the resource allocation front by incorporating interesting ideas, such as oversubscription (https://issues.apache.org/jira/browse/MESOS-354), from academic literature and other systems. Summary In this article, we looked at the Mesos architecture in detail and learned how Mesos deals with resource allocation, resource isolation, and fault tolerance. We also saw the various ways in which we can extend Mesos. Resources for Article: Further resources on this subject: Recommender systems dissected Tuning Solr JVM and Container [article] Transformation [article] Getting Started [article]

In this article by Nitish Misra, author of the book, Learning Unreal Engine Android Game Development, mentions about the Blueprint class. You would need to do all the scripting and everything else only once. A Blueprint class is an entity that contains actors (static meshes, volumes, camera classes, trigger box, and so on) and functionalities scripted in it. Looking at our example once again of the lamp turning on/off, say you want to place 10 such lamps. With a Blueprint class, you would just have to create and script once, save it, and duplicate it. This is really an amazing feature offered by UE4. (For more resources related to this topic, see here.) Creating a Blueprint class To create a Blueprint class, click on the Blueprints button in the Viewport toolbar, and in the dropdown menu, select New Empty Blueprint Class. A window will then open, asking you to pick your parent class, indicating the kind of Blueprint class you wish to create. At the top, you will see the most common classes. These are as follows: Actor: An Actor, as already discussed, is an object that can be placed in the world (static meshes, triggers, cameras, volumes, and so on, all count as actors) Pawn: A Pawn is an actor that can be controlled by the player or the computer Character: This is similar to a Pawn, but has the ability to walk around Player Controller: This is responsible for giving the Pawn or Character inputs in the game, or controlling it Game Mode: This is responsible for all of the rules of gameplay Actor Component: You can create a component using this and add it to any actor Scene Component: You can create components that you can attach to other scene components Apart from these, there are other classes that you can choose from. To see them, click on All Classes, which will open a menu listing all the classes you can create a Blueprint with. For our key cube, we will need to create an Actor Blueprint Class. Select Actor, which will then open another window, asking you where you wish to save it and what to name it. Name it Key_Cube, and save it in the Blueprint folder. After you are satisfied, click on OK and the Actor Blueprint Class window will open. The Blueprint class user interface is similar to that of Level Blueprint, but with a few differences. It has some extra windows and panels, which have been described as follows: Components panel: The Components panel is where you can view, and add components to the Blueprint class. The default component in an empty Blueprint class is DefaultSceneRoot. It cannot be renamed, copied, or removed. However, as soon as you add a component, it will replace it. Similarly, if you were to delete all of the components, it will come back. To add a component, click on the Add Component button, which will open a menu, from where you can choose which component to add. Alternatively, you can drag an asset from the Content Browser and drop it in either the Graph Editor or the Components panel, and it will be added to the Blueprint class as a component. Components include actors such as static or skeletal meshes, light actors, camera, audio actors, trigger boxes, volumes, particle systems, to name a few. When you place a component, it can be seen in the Graph Editor, where you can set its properties, such as size, position, mobility, material (if it is a static mesh or a skeletal mesh), and so on, in the Details panel. Graph Editor: The Graph Editor is also slightly different from that of Level Blueprint, in that there are additional windows and editors in a Blueprint class. The first window is the Viewport, which is the same as that in the Editor. It is mainly used to place actors and set their positions, properties, and so on. Most of the tools you will find in the main Viewport (the editor's Viewport) toolbar are present here as well. Event Graph: The next window is the Event Graph window, which is the same as a Level Blueprint window. Here, you can script the components that you added in the Viewport and their functionalities (for example, scripting the toggling of the lamp on/off when the player is in proximity and moves away respectively). Keep in mind that you can script the functionalities of the components only present within the Blueprint class. You cannot use it directly to script the functionalities of any actor that is not a component of the Class. Construction Script: Lastly, there is the Construction Script window. This is also similar to the Event Graph, as in you can set up and connect nodes, just like in the Event Graph. The difference here is that these nodes are activated when you are constructing the Blueprint class. They do not work during runtime, since that is when the Event Graph scripts work. You can use the Construction Script to set properties, create and add your own property of any of the components you wish to alter during the construction, and so on. Let's begin creating the Blueprint class for our key cubes. Viewport The first thing we need are the components. We require three components: a cube, a trigger box, and a PostProcessVolume. In the Viewport, click on the Add Components button, and under Rendering, select Static Mesh. It will add a Static Mesh component to the class. You now need to specify which Static Mesh you want to add to the class. With the Static Mesh actor selected in the Components panel, in the actor's Details panel, under the Static Mesh section, click on the None button and select TemplateCube_Rounded. As soon as you set the mesh, it will appear in the Viewport. With the cube selected, decrease its scale (located in the Details panel) from 1 to 0.2 along all three axes. The next thing we need is a trigger box. Click on the Add Component button and select Box Collision in the Collision section. Once added, increase its scale from 1 to 9 along all three axes, and place it in such a way that its bottom is in line with the bottom of the cube. The Construction Script You could set its material in the Details panel itself by clicking on the Override Materials button in the Rendering section, and selecting the key cube material. However, we are going to assign its material using Construction Script. Switch to the Construction Script tab. You will see a node called Construction Script, which is present by default. You cannot delete this node; this is where the script starts. However, before we can script it in, we will need to create a variable of the type Material. In the My Blueprint section, click on Add New and select Variable in the dropdown menu. Name this variable Key Cube Material, and change its type from Bool (which is the default variable type) to Material in the Details panel. Also, be sure to check the Editable box so that we can edit it from outside the Blueprint class. Next, drag the Key Cube Material variable from the My Blueprint panel, drop it in the Graph Editor, and select Set when the window opens up. Connect this to the output pin of the Construction Script node. Repeat this process, only this time, select Get and connect it to the input pin of Key Cube Material. Right-click in the Graph Editor window and type in Set Material in the search bar. You should see Set Material (Static Mesh). Click on it and add it to the scene. This node already has a reference of the Static Mesh actor (TemplateCube_Rounded), so we will not have to create a reference node. Connect this to the Set node. Finally, drag Key Cube Material from My Blueprint, drop it in the Graph Editor, select Get, and connect it to the Material input pin. After you are done, hit Compile. We will now be able to set the cube's material outside of the Blueprint class. Let's test it out. Add the Blueprint class to the level. You will see a TemplateCube_Rounded actor added to the scene. In its Details panel, you will see a Key Cube Material option under the Default section. This is the variable we created inside our Construction Script. Any material we add here will be added to the cube. So, click on None and select KeyCube_Material. As soon as you select it, you will see the material on the cube. This is one of the many things you can do using Construction Script. For now, only this will do. The Event Graph We now need to script the key cube's functionalities. This is more or less the same as what we did in the Level Blueprint with our first key cube, with some small differences. In the Event Graph panel, the first thing we are going to script is enabling and disabling input when the player overlaps and stops overlapping the trigger box respectively. In the Components section, right-click on Box. This will open a menu. Mouse over Add Event and select Add OnComponentBeginOverlap. This will add a Begin Overlap node to the Graph Editor. Next, we are going to need a Cast node. A Cast node is used to specify which actor you want to use. Right-click in the Graph Editor and add a Cast to Character node. Connect this to the OnComponentBeginOverlap node and connect the other actor pin to the Object pin of the Cast to Character node. Finally, add an Enable Input node and a Get Player Controller node and connect them as we did in the Level Blueprint. Next, we are going to add an event for when the player stops overlapping the box. Again, right-click on Box and add an OnComponentEndOverlap node. Do the exact same thing you did with the OnComponentBeginOverlap node; only here, instead of adding an Enable Input node, add a Disable Input node. The setup should look something like this: You can move the key cube we had placed earlier on top of the pedestal, set it to hidden, and put the key cube Blueprint class in its place. Also, make sure that you set the collision response of the trigger actor to Ignore. The next step is scripting the destruction of the key cube when the player touches the screen. This, too, is similar to what we had done in Level Blueprint, with a few differences. Firstly, add a Touch node and a Sequence node, and connect them to each other. Next, we need a Destroy Component node, which you can find under Components | Destroy Component (Static Mesh). This node already has a reference to the key cube (Static Mesh) inside it, so you do not have to create an external reference and connect it to the node. Connect this to the Then 0 node. We also need to activate the trigger after the player has picked up the key cube. Now, since we cannot call functions on actors outside the Blueprint class directly (like we could in Level Blueprint), we need to create a variable. This variable will be of the type Trigger Box. The way this works is, when you have created a Trigger Box variable, you can assign it to any trigger in the level, and it will call that function to that particular trigger. With that in mind, in the My Blueprint panel, click on Add New and create a variable. Name this variable Activated Trigger Box, and set its type to Trigger Box. Finally, make sure you tick on the Editable box; otherwise, you will not be able to assign any trigger to it. After doing that, create a Set Collision Response to All Channels node (uncheck the Context Sensitive box), and set the New Response option to Overlap. For the target, drag the Activated Trigger Box variable, drop it in the Graph Editor, select Get, and connect it to the Target input. Finally, for the Post Process Volume, we will need to create another variable of the type PostProcessVolume. You can name this variable Visual Indicator, again, while ensuring that the Editable box is checked. Add this variable to the Graph Editor as well. Next, click on its pin, drag it out, and release it, which will open the actions menu. Here, type in Enabled, select Set Enabled, and check Enabled. Finally, add a Delay node and a Destroy Actor and connect them to the Set Enabled node, in that order. Your setup should look something like this: Back in the Viewport, you will find that under the Default section of the Blueprint class actor, two more options have appeared: Activated Trigger Box and Visual Indicator (the variables we had created). Using this, you can assign which particular trigger box's collision response you want to change, and which exact post process volume you want to activate and destroy. In front of both variables, you will see a small icon in the shape of an eye dropper. You can use this to choose which external actor you wish to assign the corresponding variable. Anything you scripted using those variables will take effect on the actor you assigned in the scene. This is one of the many amazing features offered by the Blueprint class. All we need to do now for the remaining key cubes is: Place them in the level Using the eye dropper icon that is located next to the name of the variables, pick the trigger to activate once the player has picked up the key cube, and which post process volume to activate and destroy. In the second room, we have two key cubes: one to activate the large door and the other to activate the door leading to the third room. The first key cube will be placed on the pedestal near the big door. So, with the first key cube selected, using the eye dropper, select the trigger box on the pedestal near the big door for the Activated Trigger Box variable. Then, pick the post process volume inside which the key cube is placed for the Visual Indicator variable. The next thing we need to do is to open Level Blueprint and script in what happens when the player places the key cube on the pedestal near the big door. Doing what we did in the previous room, we set up nodes that will unhide the hidden key cube on the pedestal, and change the collision response of the trigger box around the big door to Overlap, ensuring that it was set to Ignore initially. Test it out! You will find that everything is working as expected. Now, do the same with the remaining key cubes. Pick which trigger box and which post process volume to activate when you touch on the screen. Then, in the Level Blueprint, script in which key cube to unhide, and so on (place the key cubes we had placed earlier on the pedestals and set it to Hidden), and place the Blueprint class key cube in its place. This is one of the many ways you can use Blueprint class. You can see it takes a lot of work and hassle. Let us now move on to Artificial intelligence. Scripting basic AI Coming back to the third room, we are now going to implement AI in our game. We have an AI character in the third room which, when activated, moves. The main objective is to make a path for it with the help of switches and prevent it from falling. When the AI character reaches its destination, it will unlock the key cube, which the player can then pick up and place on the pedestal. We first need to create another Blueprint class of the type Character, and name it AI_Character. When created, double-click on it to open it. You will see a few components already set up in the Viewport. These are the CapsuleComponent (which is mainly used for collision), ArrowComponent (to specify which side is the front of the character, and which side is the back), Mesh (used for character animation), and CharacterMovement. All four are there by default, and cannot be removed. The only thing we need to do here is add a StaticMesh for our character, which will be TemplateCube_Rounded. Click on Add Components, add a StaticMesh, and assign it TemplateCube_Rounded (in its Details panel). Next, scale this cube to 0.2 along all three axes and move it towards the bottom of the CapsuleComponent, so that it does not float in midair. This is all we require for our AI character. The rest we will handle in Level Blueprints. Next, place AI_Character into the scene on the Player side of the pit, with all of the switches. Place it directly over the Target Point actor. Next, open up Level Blueprint, and let's begin scripting it. The left-most switch will be used to activate the AI character, and the remaining three will be used to draw the parts of a path on which it will walk to reach the other side. To move the AI character, we will need an AI Move To node. The first thing we need is an overlapping event for the trigger over the first switch, which will enable the input, otherwise the AI character will start moving whenever the player touches the screen, which we do not want. Set up an Overlap event, an Enable Input node, and a Gate event. Connect the Overlap event to the Enable Input event, and then to the Gate node's Open input. The next thing is to create a Touch node. To this, we will attach an AI Move To node. You can either type it in or find it under the AI section. Once created, attach it to the Gate node's Exit pin. We now need to specify to the node which character we want to move, and where it should move to. To specify which character we want to move, select the AI character in the Viewport, and in the Level Blueprint's Graph Editor, right-click and create a reference for it. Connect it to the Pawn input pin. Next, for the location, we want the AI character to move towards the second Target Point actor, located on the other side of the pit. But first, we need to get its location in the world. With it selected, right-click in the Graph Editor, and type in Get Actor Location. This node returns an actor's location (coordinates) in the world (the one connected to it). This will create a Get Actor Location, with the Target Point actor connect to its input pin. Finally, connect its Return Value to the Destination input of the AI Move To node. If you were to test it out, you would find that it works fine, except for one thing: the AI character stops when it reaches the edge of the pit. We want it to fall off the pit if there is no path. For that, we will need a Nav Proxy Link actor. A Nav Proxy Link actor is used when an AI character has to step outside the Nav Mesh temporarily (for example, jump between ledges). We will need this if we want our AI character to fall off the ledge. You can find it in the All Classes section in the Modes panel. Place it in the level. The actor is depicted by two cylinders with a curved arrow connecting them. We want the first cylinder to be on one side of the pit and the other cylinder on the other side. Using the Scale tool, increase the size of the Nav Proxy Link actor. When placing the Nav Proxy Link actor, keep two things in mind: Make sure that both cylinders intersect in the green area; otherwise, the actor will not work Ensure that both cylinders are in line with the AI character; otherwise, it will not move in a straight line but instead to where the cylinder is located Once placed, you will see that the AI character falls off when it reaches the edge of the pit. We are not done yet. We need to bring the AI character back to its starting position so that the player can start over (or else the player will not be able to progress). For that, we need to first place a trigger at the bottom of the pit, making sure that if the AI character does fall into it, it overlaps the trigger. This trigger will perform two actions: first, it will teleport the AI character to its initial location (with the help of the first Target Point); second, it will stop the AI Move To node, or it will keep moving even after it has been teleported. After placing the trigger, open Level Blueprint and create an Overlap event for the trigger box. To this, we will add a Sequence node, since we are calling two separate functions for when the player overlaps the trigger. The first node we are going to create is a Teleport node. Here, we can specify which actor to teleport, and where. The actor we want to teleport is the AI character, so create a reference for it and connect it to the Target input pin. As for the destination, first use the Get Actor Location function to get the location of the first Target Point actor (upon which the AI character is initially placed), and connect it to the Dest Location input. To stop the AI character's movement, right-click anywhere in the Graph Editor, and first uncheck the Context Sensitive box, since we cannot use this function directly on our AI character. What we need is a Stop Active Movement node. Type it into the search bar and create it. Connect this to the Then 1 output node, and attach a reference of the AI character to it. It will automatically convert from a Character Reference into Character Movement component reference. This is all that we need to script for our AI in the third room. There is one more thing left: how to unlock the key cube. In the fourth room, we are going to use the same principle. Here, we are going to make a chain of AI Move To nodes, each connected to the previous one's On Success output pin. This means that when the AI character has successfully reached the destination (Target Point actor), it should move to the next, and so on. Using this, and what we have just discussed about AI, script the path that the AI will follow. Packaging the project Another way of packaging the game and testing it on your device is to first package the game, import it to the device, install it, and then play it. But first, we should discuss some settings regarding packaging, and packaging for Android. The Maps & Modes settings These settings deal with the maps (scenes) and the game mode of the final game. In the Editor, click on Edit and select Project settings. In the Project settings, Project category, select Maps & Modes. Let's go over the various sections: Default Maps: Here, you can set which map the Editor should open when you open the Project. You can also set which map the game should open when it is run. The first thing you need to change is the main menu map we had created. To do this, click on the downward arrow next to Game Default Map and select Main_Menu. Local Multiplayer: If your game has local multiplayer, you can alter a few settings regarding whether the game should have a split screen. If so, you can set what the layout should be for two and three players. Default Modes: In this section, you can set the default game mode the game should run with. The game mode includes things such as the Default Pawn class, HUD class, Controller class, and the Game State Class. For our game, we will stick to MyGame. Game Instance: Here, you can set the default Game Instance Class. The Packaging settings There are settings you can tweak when packaging your game. To access those settings, first go to Edit and open the Project settings window. Once opened, under the Project section click on Packaging. Here, you can view and tweak the general settings related to packaging the project file. There are two sections: Project and Packaging. Under the Project section, you can set options such as the directory of the packaged project, the build configuration to either debug, development, or shipping, whether you want UE4 to build the whole project from scratch every time you build, or only build the modified files and assets, and so on. Under the Packaging settings, you can set things such as whether you want all files to be under one .pak file instead of many individual files, whether you want those .pak files in chunks, and so on. Clicking on the downward arrow will open the advanced settings. Here, since we are packaging our game for distribution check the For Distribution checkbox. The Android app settings In the preceding section, we talked about the general packaging settings. We will now talk about settings specific to Android apps. This can be found in Project Settings, under the Platforms section. In this section, click on Android to open the Android app settings. Here you will find all the settings and properties you need to package your game. At the top the first thing you should do is configure your project for Android. If your project is not configured, it will prompt you to do so (since version 4.7, UE4 automatically creates the androidmanifest.xml file for you). Do this before you do anything else. Here you have various sections. These are: APKPackaging: In this section, you can find options such as opening the folder where all of the build files are located, setting the package's name, setting the version number, what the default orientation of the game should be, and so on. Advanced APKPackaging: This section contains more advanced packaging options, such as one to add extra settings to the .apk files. Build: To tweak settings in the Build section, you first need the source code which is available from GitHub. Here, you can set things like whether you want the build to support x86, OpenGL ES2, and so on. Distribution Signing: This section deals with signing your app. It is a requirement on Android that all apps have a digital signature. This is so that Android can identify the developers of the app. You can learn more about digital signatures by clicking on the hyperlink at the top of the section. When you generate the key for your app, be sure to keep it in a safe and secure place since if you lose it you will not be able to modify or update your app on Google Play. Google Play Service: Android apps are downloaded via the Google Play store. This section deals with things such as enabling/disabling Google Play support, setting your app's ID, the Google Play license key, and so on. Icons: In this section, you can set your game's icons. You can set various sizes of icons depending upon the screen density of the device you are aiming to develop on. You can get more information about icons by click on the hyperlink at the top of the section. Data Cooker: Finally, in this section, you can set how you want the audio in the game to be encoded. For our game, the first thing you need to set is the Android Package Name which is found in the APKPackaging section. The format of the naming is com.YourCompany.[PROJECT]. Here, replace YourCompany with the name of the company and [PROJECT] with the name of your project. Building a package To package your project, in the Editor go to File | Package Project | Android. You will see different types of formats to package the project in. These are as follows: ATC: Use this format if you have a device that has a Qualcomm Snapdragon processor. DXT: Use this format if your device has a Tegra graphical processing unit (GPU). ETC1: You can use this for any device. However, this format does not accept textures with alpha channels. Those textures will be uncompressed, making your game requiring more space. ETC2: Use this format is you have a MALI-based device. PVRTC: Use this format if you have a device with a PowerVR GPU. Once you have decided upon which format to use, click on it to begin the packaging process. A window will open up asking you to specify which folder you want the package to be stored in. Once you have decided where to store the package file, click OK and the build process will commence. When started, just like with launching the project, a small window will pop up at the bottom-right corner of the screen notifying the user that the build process has begun. You can open the output log and cancel the build process. Once the build process is complete, go the folder you set. You will find a .bat file of the game. Providing you have checked the packaged game data inside .apk? option (which is located in the Project settings in the Android category under the APKPackaging section), you will also find an .apk file of the game. The .bat file directly installs the game from the system onto your device. To do so, first connect your device to the system. Then double-click on the .bat file. This will open a command prompt window.   Once it has opened, you do not need to do anything. Just wait until the installation process finishes. Once the installation is done, the game will be on your device ready to be executed. To use the .apk file, you will have to do things a bit differently. An .apk file installs the game when it is on the device. For that, you need to perform the following steps: Connect the device. Create a copy of the .apk file. Paste it in the device's storage. Execute the .apk file from the device. The installation process will begin. Once completed, you can play the game. Summary In this article, we covered with Blueprints and discussed how they work. We also discussed Level Blueprints and the Blueprint class, and covered how to script AI. We discussed how to package the final product and upload the game onto the Google Play Store for people to download. Resources for Article: Further resources on this subject: Flash Game Development: Creation of a Complete Tetris Game [article] Adding Finesse to Your Game [article] Saying Hello to Unity and Android [article]

In this article by Isaac Strack, the author of the book, Getting Started with Meteor.js JavaScript Framework - Second Edition, has discussed some really amazing features of Meteor that has contributed a lot to the success of Meteor. Meteor is a disruptive (in a good way!) technology. It enables a new type of web application that is faster, easier to build, and takes advantage of modern techniques, such as Full Stack Reactivity, Latency Compensation, and Data On The Wire. (For more resources related to this topic, see here.) This article explains how web applications have changed over time, why that matters, and how Meteor specifically enables modern web apps through the above-mentioned techniques. By the end of this article, you will have learned: What a modern web application is What Data On The Wire means and how it's different How Latency Compensation can improve your app experience Templates and Reactivity—programming the reactive way! Modern web applications Our world is changing. With continual advancements in displays, computing, and storage capacities, things that weren't even possible a few years ago are now not only possible but are critical to the success of a good application. The Web in particular has undergone significant change. The origin of the web app (client/server) From the beginning, web servers and clients have mimicked the dumb terminal approach to computing where a server with significantly more processing power than a client will perform operations on data (writing records to a database, math calculations, text searches, and so on), transform the data and render it (turn a database record into HTML and so on), and then serve the result to the client, where it is displayed for the user. In other words, the server does all the work, and the client acts as more of a display, or a dumb terminal. This design pattern for this is called…wait for it…the client/server design pattern. The diagrammatic representation of the client-server architecture is shown in the following diagram: This design pattern, borrowed from the dumb terminals and mainframes of the 60s and 70s, was the beginning of the Web as we know it and has continued to be the design pattern that we think of when we think of the Internet. The rise of the machines (MVC) Before the Web (and ever since), desktops were able to run a program such as a spreadsheet or a word processor without needing to talk to a server. This type of application could do everything it needed to, right there on the big and beefy desktop machine. During the early 90s, desktop computers got even more beefy. At the same time, the Web was coming alive, and people started having the idea that a hybrid between the beefy desktop application (a fat app) and the connected client/server application (a thin app) would produce the best of both worlds. This kind of hybrid app—quite the opposite of a dumb terminal—was called a smart app. Many business-oriented smart apps were created, but the easiest examples can be found in computer games. Massively Multiplayer Online games (MMOs), first-person shooters, and real-time strategies are smart apps where information (the data model) is passed between machines through a server. The client in this case does a lot more than just display the information. It performs most of the processing (or acts as a controller) and transforms the data into something to be displayed (the view). This design pattern is simple but very effective. It's called the Model View Controller (MVC) pattern. The model is essentially the data for an application. In the context of a smart app, the model is provided by a server. The client makes requests to the server for data and stores that data as the model. Once the client has a model, it performs actions/logic on that data and then prepares it to be displayed on the screen. This part of the application (talking to the server, modifying the data model, and preparing data for display) is called the controller. The controller sends commands to the view, which displays the information. The view also reports back to the controller when something happens on the screen (a button click, for example). The controller receives the feedback, performs the logic, and updates the model. Lather, rinse, repeat! Since web browsers were built to be "dumb clients", the idea of using a browser as a smart app back then was out of question. Instead, smart apps were built on frameworks such as Microsoft .NET, Java, or Macromedia (now Adobe) Flash. As long as you had the framework installed, you could visit a web page to download/run a smart app. Sometimes, you could run the app inside the browser, and sometimes, you would download it first, but either way, you were running a new type of web app where the client application could talk to the server and share the processing workload. The browser grows up Beginning in the early 2000s, a new twist on the MVC pattern started to emerge. Developers started to realize that, for connected/enterprise "smart apps", there was actually a nested MVC pattern. The server code (controller) was performing business logic against the database (model) through the use of business objects and then sending processed/rendered data to the client application (a "view"). The client was receiving this data from the server and treating it as its own personal "model". The client would then act as a proper controller, perform logic, and send the information to the view to be displayed on the screen. So, the "view" for the server MVC was the "model" for the client MVC. As browser technologies (HTML and JavaScript) matured, it became possible to create smart apps that used the Nested MVC design pattern directly inside an HTML web page. This pattern makes it possible to run a full-sized application using only JavaScript. There is no longer any need to download multiple frameworks or separate apps. You can now get the same functionality from visiting a URL as you could previously by buying a packaged product. A giant Meteor appears! Meteor takes modern web apps to the next level. It enhances and builds upon the nested MVC design pattern by implementing three key features: Data On The Wire through the Distributed Data Protocol (DDP) Latency Compensation with Mini Databases Full Stack Reactivity with Blaze and Tracker Let's walk through these concepts to see why they're valuable, and then, we'll apply them to our Lending Library application. Data On The Wire The concept of Data On The Wire is very simple and in tune with the nested MVC pattern; instead of having a server process everything, render content, and then send HTML across the wire, why not just send the data across the wire and let the client decide what to do with it? This concept is implemented in Meteor using the Distributed Data Protocol, or DDP. DDP has a JSON-based syntax and sends messages similar to the REST protocol. Additions, deletions, and changes are all sent across the wire and handled by the receiving service/client/device. Since DDP uses WebSockets rather than HTTP, the data can be pushed whenever changes occur. But the true beauty of DDP lies in the generic nature of the communication. It doesn't matter what kind of system sends or receives data over DDP—it can be a server, a web service, or a client app—they all use the same protocol to communicate. This means that none of the systems know (or care) whether the other systems are clients or servers. With the exception of the browser, any system can be a server, and without exception, any server can act as a client. All the traffic looks the same and can be treated in a similar manner. In other words, the traditional concept of having a single server for a single client goes away. You can hook multiple servers together, each serving a discreet purpose, or you can have a client connect to multiple servers, interacting with each one differently. Think about what you can do with a system like that: Imagine multiple systems all coming together to create, for example, a health monitoring system. Some systems are built with C++, some with Arduino, some with…well, we don't really care. They all speak DDP. They send and receive data on the wire and decide individually what to do with that data. Suddenly, very difficult and complex problems become much easier to solve. DDP has been implemented in pretty much every major programming language, allowing you true freedom to architect an enterprise application. Latency Compensation Meteor employs a very clever technique called Mini Databases. A mini database is a "lite" version of a normal database that lives in the memory on the client side. Instead of the client sending requests to a server, it can make changes directly to the mini database on the client. This mini database then automatically syncs with the server (using DDP of course), which has the actual database. Out of the box, Meteor uses MongoDB and Minimongo: When the client notices a change, it first executes that change against the client-side Minimongo instance. The client then goes on its merry way and lets the Minimongo handlers communicate with the server over DDP. If the server accepts the change, it then sends out a "changed" message to all connected clients, including the one that made the change. If the server rejects the change, or if a newer change has come in from a different client, the Minimongo instance on the client is corrected, and any affected UI elements are updated as a result. All of this doesn't seem very groundbreaking, but here's the thing—it's all asynchronous, and it's done using DDP. This means that the client doesn't have to wait until it gets a response back from the server. It can immediately update the UI based on what is in the Minimongo instance. What if the change was illegal or other changes have come in from the server? This is not a problem as the client is updated as soon as it gets word from the server. Now, what if you have a slow internet connection or your connection goes down temporarily? In a normal client/server environment, you couldn't make any changes, or the screen would take a while to refresh while the client waits for permission from the server. However, Meteor compensates for this. Since the changes are immediately sent to Minimongo, the UI gets updated immediately. So, if your connection is down, it won't cause a problem: All the changes you make are reflected in your UI, based on the data in Minimongo. When your connection comes back, all the queued changes are sent to the server, and the server will send authorized changes to the client. Basically, Meteor lets the client take things on faith. If there's a problem, the data coming in from the server will fix it, but for the most part, the changes you make will be ratified and broadcast by the server immediately. Coding this type of behavior in Meteor is crazy easy (although you can make it more complex and therefore more controlled if you like): lists = new Mongo.Collection("lists"); This one line declares that there is a lists data model. Both the client and server will have a version of it, but they treat their versions differently. The client will subscribe to changes announced by the server and update its model accordingly. The server will publish changes, listen to change requests from the client, and update its model (its master copy) based on these change requests. Wow, one line of code that does all that! Of course, there is more to it, but that's beyond the scope of this article, so we'll move on. To better understand Meteor data synchronization, see the Publish and subscribe section of the meteor documentation at http://docs.meteor.com/#/full/meteor_publish. Full Stack Reactivity Reactivity is integral to every part of Meteor. On the client side, Meteor has the Blaze library, which uses HTML templates and JavaScript helpers to detect changes and render the data in your UI. Whenever there is a change, the helpers re-run themselves and add, delete, and change UI elements, as appropriate, based on the structure found in the templates. These functions that re-run themselves are called reactive computations. On both the client and the server, Meteor also offers reactive computations without having to use a UI. Called the Tracker library, these helpers also detect any data changes and rerun themselves accordingly. Because both the client and the server are JavaScript-based, you can use the Tracker library anywhere. This is defined as isomorphic or full stack reactivity because you're using the same language (and in some cases the same code!) on both the client and the server. Re-running functions on data changes has a really amazing benefit for you, the programmer: you get to write code declaratively, and Meteor takes care of the reactive part automatically. Just tell Meteor how you want the data displayed, and Meteor will manage any and all data changes. This declarative style is usually accomplished through the use of templates. Templates work their magic through the use of view data bindings. Without getting too deep, a view data binding is a shared piece of data that will be displayed differently if the data changes. Let's look at a very simple data binding—one for which you don't technically need Meteor—to illustrate the point. Let's perform the following set of steps to understand the concept in detail: In LendLib.html, you will see an HTML-based template expression: <div id="categories-container">      {{> categories}}   </div> This expression is a placeholder for an HTML template that is found just below it: <template name="categories">    <h2 class="title">my stuff</h2>.. So, {{> categories}} is basically saying, "put whatever is in the template categories right here." And the HTML template with the matching name is providing that. If you want to see how data changes will affect the display, change the h2 tag to an h4 tag and save the change: <template name="categories">    <h4 class="title">my stuff</h4> You'll see the effect in your browser. (my stuff will become itsy bitsy.) That's view data binding at work. Change the h4 tag back to an h2 tag and save the change, unless you like the change. No judgment here...okay, maybe a little bit of judgment. It's ugly, and tiny, and hard to read. Seriously, you should change it back before someone sees it and makes fun of you! Alright, now that we know what a view data binding is, let's see how Meteor uses it. Inside the categories template in LendLib.html, you'll find even more templates: <template name="categories"> <h4 class="title">my stuff</h4> <div id="categories" class="btn-group">    {{#each lists}}      <div class="category btn btn-primary">        {{Category}}      </div>    {{/each}} </div> </template> Meteor uses a template language called Spacebars to provide instructions inside templates. These instructions are called expressions, and they let us do things like add HTML for every record in a collection, insert the values of properties, and control layouts with conditional statements. The first Spacebars expression is part of a pair and is a for-each statement. {{#each lists}} tells the interpreter to perform the action below it (in this case, it tells it to make a new div element) for each item in the lists collection. lists is the piece of data, and {{#each lists}} is the placeholder. Now, inside the {{#each lists}} expression, there is one more Spacebars expression: {{Category}} Since the expression is found inside the #each expression, it is considered a property. That is to say that {{Category}} is the same as saying this.Category, where this is the current item in the for-each loop. So, the placeholder is saying, "add the value of the Category property for the current record." Now, if we look in LendLib.js, we will see the reactive values (called reactive contexts) behind the templates: lists : function () { return lists.find(... Here, Meteor is declaring a template helper named lists. The helper, lists, is found inside the template helpers belonging to categories. The lists helper happens to be a function that returns all the data in the lists collection, which we defined previously. Remember this line? lists = new Mongo.Collection("lists"); This lists collection is returned by the above-mentioned helper. When there is a change to the lists collection, the helper gets updated and the template's placeholder is changed as well. Let's see this in action. On your web page pointing to http://localhost:3000, open the browser console and enter the following line: > lists.insert({Category:"Games"}); This will update the lists data collection. The template will see this change and update the HTML code/placeholder. Each of the placeholders will run one additional time for the new entry in lists, and you'll see the following screen: When the lists collection was updated, the Template.categories.lists helper detected the change and reran itself (recomputed). This changed the contents of the code meant to be displayed in the {{> categories}} placeholder. Since the contents were changed, the affected part of the template was re-run. Now, take a minute here and think about how little we had to do to get this reactive computation to run: we simply created a template, instructing Blaze how we want the lists data collection to be displayed, and we put in a placeholder. This is simple, declarative programming at its finest! Let's create some templates We'll now see a real-life example of reactive computations and work on our Lending Library at the same time. Adding categories through the console has been a fun exercise, but it's not a long-term solution. Let's make it so that we can do that on the page instead as follows: Open LendLib.html and add a new button just before the {{#each lists}} expression: <div id="categories" class="btn-group"> <div class="category btn btn-primary" id="btnNewCat">    <span class="glyphicon glyphicon-plus"></span> </div> {{#each lists}} This will add a plus button on the page, as follows: Now, we want to change the button into a text field when we click on it. So let's build that functionality by using the reactive pattern. We will make it based on the value of a variable in the template. Add the following {{#if…else}} conditionals around our new button: <div id="categories" class="btn-group"> {{#if new_cat}} {{else}}    <div class="category btn btn-primary" id="btnNewCat">      <span class="glyphicon glyphicon-plus"></span>    </div> {{/if}} {{#each lists}} The first line, {{#if new_cat}}, checks to see whether new_cat is true or false. If it's false, the {{else}} section is triggered, and it means that we haven't yet indicated that we want to add a new category, so we should be displaying the button with the plus sign. In this case, since we haven't defined it yet, new_cat will always be false, and so the display won't change. Now, let's add the HTML code to display when we want to add a new category: {{#if new_cat}} <div class="category form-group" id="newCat">      <input type="text" id="add-category" class="form-control" value="" />    </div> {{else}} ... {{/if}} There's the smallest bit of CSS we need to take care of as well. Open ~/Documents/Meteor/LendLib/LendLib.css and add the following declaration: #newCat { max-width: 250px; } Okay, so now we've added an input field, which will show up when new_cat is true. The input field won't show up unless it is set to true; so, for now, it's hidden. So, how do we make new_cat equal to true? Save your changes if you haven't already done so, and open LendLib.js. First, we'll declare a Session variable, just below our Meteor.isClient check function, at the top of the file: if (Meteor.isClient) { // We are declaring the 'adding_category' flag Session.set('adding_category', false); Now, we'll declare the new template helper new_cat, which will be a function returning the value of adding_category. We need to place the new helper in the Template.categories.helpers() method, just below the declaration for lists: Template.categories.helpers({ lists: function () {    ... }, new_cat: function(){    //returns true if adding_category has been assigned    //a value of true    return Session.equals('adding_category',true); } }); Note the comma (,) on the line above new_cat. It's important that you add that comma, or your code will not execute. Save these changes, and you'll see that nothing has changed. Ta-da! In reality, this is exactly as it should be because we haven't done anything to change the value of adding_category yet. Let's do this now: First, we'll declare our click event handler, which will change the value in our Session variable. To do this, add the following highlighted code just below the Template.categories.helpers() block: Template.categories.helpers({ ... }); Template.categories.events({ 'click #btnNewCat': function (e, t) {    Session.set('adding_category', true);    Tracker.flush();    focusText(t.find("#add-category")); } }); Now, let's take a look at the following line of code: Template.categories.events({ This line declares that events will be found in the category template. Now, let's take a look at the next line: 'click #btnNewCat': function (e, t) { This tells us that we're looking for a click event on the HTML element with an id="btnNewCat" statement (which we already created in LendLib.html). Session.set('adding_category', true); Tracker.flush(); focusText(t.find("#add-category")); Next, we set the Session variable, adding_category = true, flush the DOM (to clear up anything wonky), and then set the focus onto the input box with the id="add-category" expression. There is one last thing to do, and that is to quickly add the focusText(). helper function. To do this, just before the closing tag for the if (Meteor.isClient) function, add the following code: /////Generic Helper Functions///// //this function puts our cursor where it needs to be. function focusText(i) { i.focus(); i.select(); }; } //<------closing bracket for if(Meteor.isClient){} Now, when you save the changes and click on the plus button, you will see the input box: Fancy! However, it's still not useful, and we want to pause for a second and reflect on what just happened; we created a conditional template in the HTML page that will either show an input box or a plus button, depending on the value of a variable. This variable is a reactive variable, called a reactive context. This means that if we change the value of the variable (like we do with the click event handler), then the view automatically updates because the new_cat helpers function (a reactive computation) will rerun. Congratulations, you've just used Meteor's reactive programming model! To really bring this home, let's add a change to the lists collection (which is also a reactive context, remember?) and figure out a way to hide the input field when we're done. First, we need to add a listener for the keyup event. Or, to put it another way, we want to listen when the user types something in the box and hits Enter. When this happens, we want to add a category based on what the user typed. To do this, let's first declare the event handler. Just after the click handler for #btnNewCat, let's add another event handler: 'click #btnNewCat': function (e, t) {    ... }, 'keyup #add-category': function (e,t){    if (e.which === 13)    {      var catVal = String(e.target.value || "");      if (catVal)      {        lists.insert({Category:catVal});        Session.set('adding_category', false);      }    } } We add a "," character at the end of the first click handler, and then add the keyup event handler. Now, let's check each of the lines in the preceding code: This line checks to see whether we hit the Enter/Return key. if (e.which === 13) This line of code checks to see whether the input field has any value in it: var catVal = String(e.target.value || ""); if (catVal) If it does, we want to add an entry to the lists collection: lists.insert({Category:catVal}); Then, we want to hide the input box, which we can do by simply modifying the value of adding_category: Session.set('adding_category', false); There is one more thing to add and then we'll be done. When we click away from the input box, we want to hide it and bring back the plus button. We already know how to do this reactively, so let's add a quick function that changes the value of adding_category. To do this, add one more comma after the keyup event handler and insert the following event handler: 'keyup #add-category': function (e,t){ ... }, 'focusout #add-category': function(e,t){    Session.set('adding_category',false); } Save your changes, and let's see this in action! In your web browser on http://localhost:3000, click on the plus sign, add the word Clothes, and hit Enter. Your screen should now resemble the following screenshot: Feel free to add more categories if you like. Also, experiment by clicking on the plus button, typing something in, and then clicking away from the input field. Summary In this article, you learned about the history of web applications and saw how we've moved from a traditional client/server model to a nested MVC design pattern. You learned what smart apps are, and you also saw how Meteor has taken smart apps to the next level with Data On The Wire, Latency Compensation, and Full Stack Reactivity. You saw how Meteor uses templates and helpers to automatically update content, using reactive variables and reactive computations. Lastly, you added more functionality to the Lending Library. You made a button and an input field to add categories, and you did it all using reactive programming rather than directly editing the HTML code. Resources for Article: Further resources on this subject: Building the next generation Web with Meteor [article] Quick start - creating your first application [article] Meteor.js JavaScript Framework: Why Meteor Rocks! [article]

In this article by Mohamed Taman, authors of the book JavaFX Essentials, we will learn how to develop a JavaFX, Apple has a great market share in the mobile and PC/Laptop world, with many different devices, from mobile phones such as the iPhone to musical devices such as the iPod and tablets such as the iPad. (For more resources related to this topic, see here.) It has a rapidly growing application market, called the Apple Store, serving its community, where the number of available apps increases daily. Mobile application developers should be ready for such a market. Mobile application developers targeting both iOS and Android face many challenges. By just comparing the native development environments of these two platforms, you will find that they differ substantially. iOS development, according to Apple, is based on the Xcode IDE (https://developer.apple.com/xcode/) and its programming languages. Traditionally, it was Objetive-C and, in June 2014, Apple introduced Swift (https://developer.apple.com/swift/); on the other hand, Android development, as defined by Google, is based on the Intellij IDEA IDE and the Java programming language. Not many developers are proficient in both environments. In addition, these differences rule out any code reuse between the platforms. JavaFX 8 is filling the gap for reusable code between the platforms, as we will see in this article, by sharing the same application in both platforms. Here are some skills that you will have gained by the end of this article: Installing and configuring iOS environment tools and software Creating iOS JavaFX 8 applications Simulating and debugging JavaFX mobile applications Packaging and deploying applications on iOS mobile devices Using RoboVM to run JavaFX on iOS RoboVM is the bridge from Java to Objetive-C. Using this, it becomes easy to develop JavaFX 8 applications that are to be run on iOS-based devices, as the ultimate goal of the RoboVM project is to solve this problem without compromising on developer experience or app user experience. As we saw in the article about Android, using JavaFXPorts to generate APKs was a relatively easy task due to the fact that Android is based on Java and the Dalvik VM. On the contrary, iOS doesn't have a VM for Java, and it doesn't allow dynamic loading of native libraries. Another approach is required. The RoboVM open source project tries to close the gap for Java developers by creating a bridge between Java and Objective-C using an ahead-of-time compiler that translates Java bytecode into native ARM or x86 machine code. Features Let's go through the RoboVM features: Brings Java and other JVM languages, such as Scala, Clojure, and Groovy, to iOS-based devices Translates Java bytecode into machine code ahead of time for fast execution directly on the CPU without any overhead The main target is iOS and the ARM processor (32- and 64-bit), but there is also support for Mac OS X and Linux running on x86 CPUs (both 32- and 64-bit) Does not impose any restrictions on the Java platform features accessible to the developer, such as reflection or file I/O Supports standard JAR files that let the developer reuse the vast ecosystem of third-party Java libraries Provides access to the full native iOS APIs through a Java-to-Objective-C bridge, enabling the development of apps with truly native UIs and with full hardware access Integrates with the most popular tools such as NetBeans, Eclipse, Intellij IDEA, Maven, and Gradle App Store ready, with hundreds of apps already in the store Limitations Mainly due to the restrictions of the iOS platform, there are a few limitations when using RoboVM: Loading custom bytecode at runtime is not supported. All class files comprising the app have to be available at compile time on the developer machine. The Java Native Interface technology as used on the desktop or on servers usually loads native code from dynamic libraries, but Apple does not permit custom dynamic libraries to be shipped with an iOS app. RoboVM supports a variant of JNI based on static libraries. Another big limitation is that RoboVM is an alpha-state project under development and not yet recommended for production usage. RoboVM has full support for reflection. How it works Since February 2015 there has been an agreement between the companies behind RoboVM and JavaFXPorts, and now a single plugin called jfxmobile-plugin allows us to build applications for three platforms—desktop, Android, and iOS—from the same codebase. The JavaFXMobile plugin adds a number of tasks to your Java application that allow you to create .ipa packages that can be submitted to the Apple Store. Android mostly uses Java as the main development language, so it is easy to merge your JavaFX 8 code with it. On iOS, the situation is internally totally different—but with similar Gradle commands. The plugin will download and install the RoboVM compiler, and it will use RoboVM compiler commands to create an iOS application in build/javafxports/ios. Getting started In this section, you will learn how to install the RoboVM compiler using the JavaFXMobile plugin, and make sure the tool chain works correctly by reusing the same application, Phone Dial version 1.0. Prerequisites In order to use the RoboVM compiler to build iOS apps, the following tools are required: Gradle 2.4 or higher is required to build applications with the jfxmobile plugin. A Mac running Mac OS X 10.9 or later. Xcode 6.x from the Mac App Store (https://itunes.apple.com/us/app/xcode/id497799835?mt=12). The first time you install Xcode, and every time you update to a new version, you have to open it once to agree to the Xcode terms. Preparing a project for iOS We will reuse the project we developed before, for the Android platform, since there is no difference in code, project structure, or Gradle build script when targeting iOS. They share the same properties and features, but with different Gradle commands that serve iOS development, and a minor change in the Gradle build script for the RoboVM compiler. Therefore, we will see the power of WORA Write Once, Run Everywhere with the same application. Project structure Based on the same project structure from the Android, the project structure for our iOS app should be as shown in the following figure: The application We are going to reuse the same application from the Phone DialPad version 2.0 JavaFX 8 application: As you can see, reusing the same codebase is a very powerful and useful feature, especially when you are developing to target many mobile platforms such as iOS and Android at the same time. Interoperability with low-level iOS APIs To have the same functionality of natively calling the default iOS phone dialer from our application as we did with Android, we have to provide the native solution for iOS as the following IosPlatform implementation: import org.robovm.apple.foundation.NSURL; import org.robovm.apple.uikit.UIApplication; import packt.taman.jfx8.ch4.Platform;   public class IosPlatform implements Platform {   @Override public void callNumber(String number) {    if (!number.equals("")) {      NSURL nsURL = new NSURL("telprompt://" + number);      UIApplication.getSharedApplication().openURL(nsURL);    } } } Gradle build files We will use the Gradle build script file, but with a minor change by adding the following lines to the end of the script: jfxmobile { ios {    forceLinkClasses = [ 'packt.taman.jfx8.ch4.**.*' ] } android {    manifest = 'lib/android/AndroidManifest.xml' } } All the work involved in installing and using robovm compilers is done by the jfxmobile plugin. The purpose of those lines is to give the RoboVM compiler the location of the main application class that has to be loaded at runtime is, as it is not visible by default to the compiler. The forceLinkClasses property ensures that those classes are linked in during RoboVM compilation. Building the application After we have added the necessary configuration set to build the script for iOS, its time to build the application in order to deploy it to different iOS target devices. To do so, we have to run the following command: $ gradle build We should have the following output: BUILD SUCCESSFUL   Total time: 44.74 secs We have built our application successfully; next, we need to generate the .ipa and, in the case of production, you have to test it by deploying it to as many iOS versions as you can. Generating the iOS .ipa package file In order to generate the final .ipa iOS package for our JavaFX 8 application, which is necessary for the final distribution to any device or the AppStore, you have to run the following gradle command: gradle ios This will generate the .ipa file in the directory build/javafxports/ios. Deploying the application During development, we need to check our application GUI and final application prototype on iOS simulators and measure the application performance and functionality on different devices. These procedures are very useful, especially for testers. Let's see how it is a very easy task to run our application on either simulators or on real devices. Deploying to a simulator On a simulator, you can simply run the following command to check if your application is running: $ gradle launchIPhoneSimulator This command will package and launch the application in an iPhone simulator as shown in the following screenshot: DialPad2 JavaFX 8 application running on the iOS 8.3/iPhone 4s simulator This command will launch the application in an iPad simulator: $ gradle launchIPadSimulator Deploying to an Apple device In order to package a JavaFX 8 application and deploy it to an Apple device, simply run the following command: $ gradle launchIOSDevice This command will launch the JavaFX 8 application in the device that is connected to your desktop/laptop. Then, once the application is launched on your device, type in any number and then tap Call. The iPhone will ask for permission to dial using the default mobile dialer; tap on Ok. The default mobile dialer will be launched and will the number as shown in the following figure: To be able to test and deploy your apps on your devices, you will need an active subscription with the Apple Developer Program. Visit the Apple Developer Portal, https://developer.apple.com/register/index.action, to sign up. You will also need to provision your device for development. You can find information on device provisioning in the Apple Developer Portal, or follow this guide: http://www.bignerdranch.com/we-teach/how-to-prepare/ios-device-provisioning/. Summary This article gave us a very good understanding of how JavaFX-based applications can be developed and customized using RoboVM for iOS to make it possible to run your applications on Apple platforms. You learned about RoboVM features and limitations, and how it works; you also gained skills that you can use for developing. You then learned how to install the required software and tools for iOS development and how to enable Xcode along with the RoboVM compiler, to package and install the Phone Dial JavaFX-8-based application on OS simulators. Finally, we provided tips on how to run and deploy your application on real devices. Resources for Article: Further resources on this subject: Function passing [article] Creating Java EE Applications [article] Contexts and Dependency Injection in NetBeans [article]

In this article by Jurie-Jan Botha, author of the book Grunt Cookbook, has covered the following recipes: Minifying HTML Minifying CSS Optimizing images Linting JavaScript code Uglifying JavaScript code Setting up RequireJS (For more resources related to this topic, see here.) Once our web application is built and its stability ensured, we can start preparing it for deployment to its intended market. This will mainly involve the optimization of the assets that make up the application. Optimization in this context mostly refers to compression of one kind or another, some of which might lead to performance increases too. The focus on compression is primarily due to the fact that the smaller the asset, the faster it can be transferred from where it is hosted to a user's web browser. This leads to a much better user experience, and can sometimes be essential to the functioning of an application. Minifying HTML In this recipe, we make use of the contrib-htmlmin (0.3.0) plugin to decrease the size of some HTML documents by minifying them. Getting ready In this example, we'll work with the a basic project structure. How to do it... The following steps take us through creating a sample HTML document and configuring a task that minifies it: We'll start by installing the package that contains the contrib-htmlmin plugin. Next, we'll create a simple HTML document called index.html in the src directory, which we'd like to minify, and add the following content in it: <html> <head>    <title>Test Page</title> </head> <body>    <!-- This is a comment! -->    <h1>This is a test page.</h1> </body> </html> Now, we'll add the following htmlmin task to our configuration, which indicates that we'd like to have the white space and comments removed from the src/index.html file, and that we'd like the result to be saved in the dist/index.html file: htmlmin: { dist: {    src: 'src/index.html',    dest: 'dist/index.html',    options: {      removeComments: true,      collapseWhitespace: true    } } } The removeComments and collapseWhitespace options are used as examples here, as using the default htmlmin task will have no effect. Other minification options can be found at the following URL: https://github.com/kangax/html-minifier#options-quick-reference We can now run the task using the grunt htmlmin command, which should produce output similar to the following: Running "htmlmin:dist" (htmlmin) task Minified dist/index.html 147 B ? 92 B If we now take a look at the dist/index.html file, we will see that all white space and comments have been removed: <html> <head>    <title>Test Page</title> </head> <body>    <h1>This is a test page.</h1> </body> </html> Minifying CSS In this recipe, we'll make use of the contrib-cssmin (0.10.0) plugin to decrease the size of some CSS documents by minifying them. Getting ready In this example, we'll work with a basic project structure. How to do it... The following steps take us through creating a sample CSS document and configuring a task that minifies it. We'll start by installing the package that contains the contrib-cssmin plugin. Then, we'll create a simple CSS document called style.css in the src directory, which we'd like to minify, and provide it with the following contents: body { /* Average body style */ background-color: #ffffff; color: #000000; /*! Black (Special) */ } Now, we'll add the following cssmin task to our configuration, which indicates that we'd like to have the src/style.css file compressed, and have the result saved to the dist/style.min.css file: cssmin: { dist: {    src: 'src/style.css',    dest: 'dist/style.min.css' } } We can now run the task using the grunt cssmin command, which should produce the following output: Running "cssmin:dist" (cssmin) taskFile dist/style.css created: 55 B ? 38 B If we take a look at the dist/style.min.css file that was produced, we will see that it has the compressed contents of the original src/style.css file: body{background-color:#fff;color:#000;/*! Black (Special) */} There's more... The cssmin task provides us with several useful options that can be used in conjunction with its basic compression feature. We'll look at prefixing a banner, removing special comments, and reporting gzipped results. Prefixing a banner In the case that we'd like to automatically include some information about the compressed result in the resulting CSS file, we can do so in a banner. A banner can be prepended to the result by supplying the desired banner content to the banner option, as shown in the following example: cssmin: { dist: {    src: 'src/style.css',    dest: 'dist/style.min.css',    options: {      banner: '/* Minified version of style.css */'    } } } Removing special comments Comments that should not be removed by the minification process are called special comments and can be indicated using the "/*! comment */" markers. By default, the cssmin task will leave all special comments untouched, but we can alter this behavior by making use of the keepSpecialComments option. The keepSpecialComments option can be set to either the *, 1, or 0 value. The * value is the default and indicates that all special comments should be kept, 1 indicates that only the first comment that is found should be kept, and 0 indicates that none of them should be kept. The following configuration will ensure that all comments are removed from our minified result: cssmin: { dist: {    src: 'src/style.css',    dest: 'dist/style.min.css',    options: {      keepSpecialComments: 0    } } } Reporting on gzipped results Reporting is useful to see exactly how well the cssmin task has compressed our CSS files. By default, the size of the targeted file and minified result will be displayed, but if we'd also like to see the gzipped size of the result, we can set the report option to gzip, as shown in the following example: cssmin: { dist: {    src: 'src/main.css',    dest: 'dist/main.css',    options: {      report: 'gzip'    } } } Optimizing images In this recipe, we'll make use of the contrib-imagemin (0.9.4) plugin to decrease the size of images by compressing them as much as possible without compromising on their quality. This plugin also provides a plugin framework of its own, which is discussed at the end of this recipe. Getting ready In this example, we'll work with the basic project structure. How to do it... The following steps take us through configuring a task that will compress an image for our project. We'll start by installing the package that contains the contrib-imagemin plugin. Next, we can ensure that we have an image called image.jpg in the src directory on which we'd like to perform optimizations. Now, we'll add the following imagemin task to our configuration and indicate that we'd like to have the src/image.jpg file optimized, and have the result saved to the dist/image.jpg file: imagemin: { dist: {    src: 'src/image.jpg',    dest: 'dist/image.jpg' } } We can then run the task using the grunt imagemin command, which should produce the following output: Running "imagemin:dist" (imagemin) task Minified 1 image (saved 13.36 kB) If we now take a look at the dist/image.jpg file, we will see that its size has decreased without any impact on the quality. There's more... The imagemin task provides us with several options that allow us to tweak its optimization features. We'll look at how to adjust the PNG compression level, disable the progressive JPEG generation, disable the interlaced GIF generation, specify SVGO plugins to be used, and use the imagemin plugin framework. Adjusting the PNG compression level The compression of a PNG image can be increased by running the compression algorithm on it multiple times. By default, the compression algorithm is run 16 times. This number can be changed by providing a number from 0 to 7 to the optimizationLevel option. The 0 value means that the compression is effectively disabled and 7 indicates that the algorithm should run 240 times. In the following configuration we set the compression level to its maximum: imagemin: { dist: {    src: 'src/image.png',    dest: 'dist/image.png',    options: {      optimizationLevel: 7    } } } Disabling the progressive JPEG generation Progressive JPEGs are compressed in multiple passes, which allows a low-quality version of them to quickly become visible and increase in quality as the rest of the image is received. This is especially helpful when displaying images over a slower connection. By default, the imagemin plugin will generate JPEG images in the progressive format, but this behavior can be disabled by setting the progressive option to false, as shown in the following example: imagemin: { dist: {    src: 'src/image.jpg',    dest: 'dist/image.jpg',    options: {      progressive: false    } } } Disabling the interlaced GIF generation An interlaced GIF is the equivalent of a progressive JPEG in that it allows the contained image to be displayed at a lower resolution before it has been fully downloaded, and increases in quality as the rest of the image is received. By default, the imagemin plugin will generate GIF images in the interlaced format, but this behavior can be disabled by setting the interlaced option to false, as shown in the following example: imagemin: { dist: {    src: 'src/image.gif',    dest: 'dist/image.gif',    options: {      interlaced: false    } } } Specifying SVGO plugins to be used When optimizing SVG images, the SVGO library is used by default. This allows us to specify the use of various plugins provided by the SVGO library that each performs a specific function on the targeted files. Refer to the following URL for more detailed instructions on how to use the svgo plugins options and the SVGO library: https://github.com/sindresorhus/grunt-svgmin#available-optionsplugins Most of the plugins in the library are enabled by default, but if we'd like to specifically indicate which of these should be used, we can do so using the svgoPlugins option. Here, we can provide an array of objects, where each contain a property with the name of the plugin to be affected, followed by a true or false value to indicate whether it should be activated. The following configuration disables three of the default plugins: imagemin: { dist: {    src: 'src/image.svg',    dest: 'dist/image.svg',    options: {      svgoPlugins: [        {removeViewBox:false},        {removeUselessStrokeAndFill:false},        {removeEmptyAttrs:false}      ]    } } } Using the 'imagemin' plugin framework In order to provide support for the various image optimization projects, the imagemin plugin has a plugin framework of its own that allows developers to easily create an extension that makes use of the tool they require. You can get a list of the available plugin modules for the imagemin plugin's framework at the following URL: https://www.npmjs.com/browse/keyword/imageminplugin The following steps will take us through installing and making use of the mozjpeg plugin to compress an image in our project. These steps start where the main recipe takes off. We'll start by installing the imagemin-mozjpeg package using the npm install imagemin-mozjpeg command, which should produce the following output: imagemin-mozjpeg@4.0.0 node_modules/imagemin-mozjpeg With the package installed, we need to import it into our configuration file, so that we can make use of it in our task configuration. We do this by adding the following line at the top of our Gruntfile.js file: var mozjpeg = require('imagemin-mozjpeg'); With the plugin installed and imported, we can now change the configuration of our imagemin task by adding the use option and providing it with the initialized plugin: imagemin: { dist: {    src: 'src/image.jpg',    dest: 'dist/image.jpg',    options: {      use: [mozjpeg()]    } } } Finally, we can test our setup by running the task using the grunt imagemin command. This should produce an output similar to the following: Running "imagemin:dist" (imagemin) task Minified 1 image (saved 9.88 kB) Linting JavaScript code In this recipe, we'll make use of the contrib-jshint (0.11.1) plugin to detect errors and potential problems in our JavaScript code. It is also commonly used to enforce code conventions within a team or project. As can be derived from its name, it's basically a Grunt adaptation for the JSHint tool. Getting ready In this example, we'll work with the basic project structure. How to do it... The following steps take us through creating a sample JavaScript file and configuring a task that will scan and analyze it using the JSHint tool. We'll start by installing the package that contains the contrib-jshint plugin. Next, we'll create a sample JavaScript file called main.js in the src directory, and add the following content in it: sample = 'abc'; console.log(sample); With our sample file ready, we can now add the following jshint task to our configuration. We'll configure this task to target the sample file and also add a basic option that we require for this example: jshint: { main: {    options: {      undef: true    },    src: ['src/main.js'] } } The undef option is a standard JSHint option used specifically for this example and is not required for this plugin to function. Specifying this option indicates that we'd like to have errors raised for variables that are used without being explicitly defined. We can now run the task using the grunt jshint command, which should produce output informing us of the problems found in our sample file: Running "jshint:main" (jshint) task      src/main.js      1 |sample = 'abc';          ^ 'sample' is not defined.      2 |console.log(sample);          ^ 'console' is not defined.      2 |console.log(sample);                      ^ 'sample' is not defined.   >> 3 errors in 1 file There's more... The jshint task provides us with several options that allow us to change its general behavior, in addition to how it analyzes the targeted code. We'll look at how to specify standard JSHint options, specify globally defined variables, send reported output to a file, and prevent task failure on JSHint errors. Specifying standard JSHint options The contrib-jshint plugin provides a simple way to pass all the standard JSHint options from the task's options object to the underlying JSHint tool. A list of all the options provided by the JSHint tool can be found at the following URL: http://jshint.com/docs/options/ The following example adds the curly option to the task we created in our main recipe to enforce the use of curly braces wherever they are appropriate: jshint: { main: {    options: {      undef: true,      curly: true    },    src: ['src/main.js'] } } Specifying globally defined variables Making use of globally defined variables is quite common when working with JavaScript, which is where the globals option comes in handy. Using this option, we can define a set of global values that we'll use in the targeted code, so that errors aren't raised when JSHint encounters them. In the following example, we indicate that the console variable should be treated as a global, and not raise errors when encountered: jshint: { main: {    options: {      undef: true,      globals: {        console: true      }    },    src: ['src/main.js'] } } Sending reported output to a file If we'd like to store the resulting output from our JSHint analysis, we can do so by specifying a path to a file that should receive it using the reporterOutput option, as shown in the following example: jshint: { main: {    options: {      undef: true,      reporterOutput: 'report.dat'    },    src: ['src/main.js'] } } Preventing task failure on JSHint errors The default behavior for the jshint task is to exit the running Grunt process once a JSHint error is encountered in any of the targeted files. This behavior becomes especially undesirable if you'd like to keep watching files for changes, even when an error has been raised. In the following example, we indicate that we'd like to keep the process running when errors are encountered by giving the force option a true value: jshint: { main: {    options: {      undef: true,      force: true    },    src: ['src/main.js'] } } Uglifying JavaScript Code In this recipe, we'll make use of the contrib-uglify (0.8.0) plugin to compress and mangle some files containing JavaScript code. For the most part, the process of uglifying just removes all the unnecessary characters and shortens variable names in a source code file. This has the potential to dramatically reduce the size of the file, slightly increase performance, and make the inner workings of your publicly available code a little more obscure. Getting ready In this example, we'll work with the basic project structure. How to do it... The following steps take us through creating a sample JavaScript file and configuring a task that will uglify it. We'll start by installing the package that contains the contrib-uglify plugin. Then, we can create a sample JavaScript file called main.js in the src directory, which we'd like to uglify, and provide it with the following contents: var main = function () { var one = 'Hello' + ' '; var two = 'World';   var result = one + two;   console.log(result); }; With our sample file ready, we can now add the following uglify task to our configuration, indicating the sample file as the target and providing a destination output file: uglify: { main: {    src: 'src/main.js',    dest: 'dist/main.js' } } We can now run the task using the grunt uglify command, which should produce output similar to the following: Running "uglify:main" (uglify) task >> 1 file created. If we now take a look at the resulting dist/main.js file, we should see that it contains the uglified contents of the original src/main.js file. There's more... The uglify task provides us with several options that allow us to change its general behavior and see how it uglifies the targeted code. We'll look at specifying standard UglifyJS options, generating source maps, and wrapping generated code in an enclosure. Specifying standard UglifyJS options The underlying UglifyJS tool can provide a set of options for each of its separate functional parts. These parts are the mangler, compressor, and beautifier. The contrib-plugin allows passing options to each of these parts using the mangle, compress, and beautify options. The available options for each of the mangler, compressor, and beautifier parts can be found at each of following URLs (listed in the order mentioned): https://github.com/mishoo/UglifyJS2#mangler-options https://github.com/mishoo/UglifyJS2#compressor-options https://github.com/mishoo/UglifyJS2#beautifier-options The following example alters the configuration of the main recipe to provide a single option to each of these parts: uglify: { main: {    src: 'src/main.js',    dest: 'dist/main.js',    options: {      mangle: {        toplevel: true      },      compress: {        evaluate: false      },      beautify: {        semicolons: false      }    } } } Generating source maps As code gets mangled and compressed, it becomes effectively unreadable to humans, and therefore, nearly impossible to debug. For this reason, we are provided with the option of generating a source map when uglifying our code. The following example makes use of the sourceMap option to indicate that we'd like to have a source map generated along with our uglified code: uglify: { main: {    src: 'src/main.js',    dest: 'dist/main.js',    options: {      sourceMap: true    } } } Running the altered task will now, in addition to the dist/main.js file with our uglified source, generate a source map file called main.js.map in the same directory as the uglified file. Wrapping generated code in an enclosure When building your own JavaScript code modules, it's usually a good idea to have them wrapped in a wrapper function to ensure that you don't pollute the global scope with variables that you won't be using outside of the module itself. For this purpose, we can use the wrap option to indicate that we'd like to have the resulting uglified code wrapped in a wrapper function, as shown in the following example: uglify: { main: {    src: 'src/main.js',    dest: 'dist/main.js',    options: {      wrap: true    } } } If we now take a look at the result dist/main.js file, we should see that all the uglified contents of the original file are now contained within a wrapper function. Setting up RequireJS In this recipe, we'll make use of the contrib-requirejs (0.4.4) plugin to package the modularized source code of our web application into a single file. For the most part, this plugin just provides a wrapper for the RequireJS tool. RequireJS provides a framework to modularize JavaScript source code and consume those modules in an orderly fashion. It also allows packaging an entire application into one file and importing only the modules that are required while keeping the module structure intact. Getting ready In this example, we'll work with the basic project structure. How to do it... The following steps take us through creating some files for a sample application and setting up a task that bundles them into one file. We'll start by installing the package that contains the contrib-requirejs plugin. First, we'll need a file that will contain our RequireJS configuration. Let's create a file called config.js in the src directory and add the following content in it: require.config({ baseUrl: 'app' }); Secondly, we'll create a sample module that we'd like to use in our application. Let's create a file called sample.js in the src/app directory and add the following content in it: define(function (require) { return function () {    console.log('Sample Module'); } }); Lastly, we'll need a file that will contain the main entry point for our application, and also makes use of our sample module. Let's create a file called main.js in the src/app directory and add the following content in it: require(['sample'], function (sample) { sample(); }); Now that we've got all the necessary files required for our sample application, we can setup a requirejs task that will bundle it all into one file: requirejs: { app: {    options: {      mainConfigFile: 'src/config.js',      name: 'main',      out: 'www/js/app.js'    } } } The mainConfigFile option points out the configuration file that will determine the behavior of RequireJS. The name option indicates the name of the module that contains the application entry point. In the case of this example, our application entry point is contained in the app/main.js file, and app is the base directory of our application in the src/config.js file. This translates the app/main.js filename into the main module name. The out option is used to indicate the file that should receive the result of the bundled application. We can now run the task using the grunt requirejs command, which should produce output similar to the following: Running "requirejs:app" (requirejs) task We should now have a file named app.js in the www/js directory that contains our entire sample application. There's more... The requirejs task provides us with all the underlying options provided by the RequireJS tool. We'll look at how to use these exposed options and generate a source map. Using RequireJS optimizer options The RequireJS optimizer is quite an intricate tool, and therefore, provides a large number of options to tweak its behavior. The contrib-requirejs plugin allows us to easily set any of these options by just specifying them as options of the plugin itself. A list of all the available configuration options for the RequireJS build system can be found in the example configuration file at the following URL: https://github.com/jrburke/r.js/blob/master/build/example.build.js The following example indicates that the UglifyJS2 optimizer should be used instead of the default UglifyJS optimizer by using the optimize option: requirejs: { app: {    options: {      mainConfigFile: 'src/config.js',      name: 'main',      out: 'www/js/app.js',      optimize: 'uglify2'    } } } Generating a source map When the source code is bundled into one file, it becomes somewhat harder to debug, as you now have to trawl through miles of code to get to the point you're actually interested in. A source map can help us with this issue by relating the resulting bundled file to the modularized structure it is derived from. Simply put, with a source map, our debugger will display the separate files we had before, even though we're actually using the bundled file. The following example makes use of the generateSourceMap option to indicate that we'd like to generate a source map along with the resulting file: requirejs: { app: {    options: {      mainConfigFile: 'src/config.js',      name: 'main',      out: 'www/js/app.js',      optimize: 'uglify2',      preserveLicenseComments: false,      generateSourceMaps: true    } } } In order to use the generateSourceMap option, we have to indicate that UglifyJS2 is to be used for optimization, by setting the optimize option to uglify2, and that license comments should not be preserved, by setting the preserveLicenseComments option to false. Summary This article covers the optimization of images, minifying of CSS, ensuring the quality of our JavaScript code, compressing it, and packaging it all together into one source file. Resources for Article: Further resources on this subject: Grunt in Action [article] So, what is Node.js? [article] Exploring streams [article]

In this article Jean-Baptiste Onofré, author of the book Mastering Apache Camel, we will see how Apache Camel originated in Apache ServiceMix. Apache ServiceMix 3 was powered by the Spring framework and implemented in the JBI specification. The Java Business Integration (JBI) specification proposed a Plug and Play approach for integration problems. JBI was based on WebService concepts and standards. For instance, it directly reusesthe Message Exchange Patterns (MEP) concept that comes from WebService Description Language (WSDL). Camel reuses some of these concepts, for instance, you will see that we have the concept of MEP in Camel. However, JBI suffered mostly from two issues: In JBI, all messages between endpoints are transported in the Normalized Messages Router (NMR). In the NMR, a message has a standard XML format. As all messages in the NMR havethe same format, it's easy to audit messages and the format is predictable. However, the JBI XML format has an important drawback for performances: it needs to marshall and unmarshall the messages. Some protocols (such as REST or RMI) are not easy to describe in XML. For instance, REST can work in stream mode. It doesn't make sense to marshall streamsin XML. Camel is payload-agnostic. This means that you can transport any kind of messages with Camel (not necessary XML formatted). JBI describes a packaging. We distinguish the binding components (responsible for the interaction with the system outside of the NMR and the handling of the messages in the NMR), and the service engines (responsible for transforming the messages inside the NMR). However, it's not possible to directly deploy the endpoints based on these components. JBI requires a service unit (a ZIP file) per endpoint, and for each package in a service assembly (another ZIP file). JBI also splits the description of the endpoint from its configuration. It does not result in a very flexible packaging: with definitions and configurations scattered in different files, not easy to maintain. In Camel, the configuration and definition of the endpoints are gatheredin a simple URI. It's easier to read. Moreover, Camel doesn't force any packaging; the same definition can be packaged in a simple XML file, OSGi bundle, andregular JAR file. In addition to JBI, another foundation of Camel is the book Enterprise Integration Patterns by Gregor Hohpe and Bobby Woolf. It describes design patterns answering classical problems while dealing with enterprise application integration and message oriented middleware. The book describes the problems and the patterns to solve them. Camel strives to implement the patterns described in the book to make them easy to use and let the developer concentrate on the task at hand. This is what Camel is: an open source framework that allows you to integrate systems and that comes with a lot of connectors and Enterprise Integration Patterns (EIP) components out of the box. And if that is not enough, one can extend and implement custom components. Components and bean support Apache Camel ships with a wide variety of components out of the box; currently, there are more than 100 components available. We can see: The connectivity components that allow exposure of endpoints for external systems or communicate with external systems. For instance, the FTP, HTTP, JMX, WebServices, JMS, and a lot more components are connectivity components. Creating an endpoint and the associated configuration for these components is easy, by directly using a URI. The internal components applying rules to the messages internally to Camel. These kinds of components apply validation or transformation rules to the inflight message. For instance, validation or XSLT are internal components. Camel brings a very powerful connectivity and mediation framework. Moreover, it's pretty easy to create new custom components, allowing you to extend Camel if the default components set doesn't match your requirements. It's also very easy to implement complex integration logic by creating your own processors and reusing your beans. Camel supports beans frameworks (IoC), such as Spring or Blueprint. Predicates and expressions As we will see later, most of the EIP need a rule definition to apply a routing logic to a message. The rule is described using an expression. It means that we have to define expressions or predicates in the Enterprise Integration Patterns. An expression returns any kind of value, whereas a predicate returns true or false only. Camel supports a lot of different languages to declare expressions or predicates. It doesn't force you to use one, it allows you to use the most appropriate one. For instance, Camel supports xpath, mvel, ognl, python, ruby, PHP, JavaScript, SpEL (Spring Expression Language), Groovy, and so on as expression languages. It also provides native Camel prebuilt functions and languages that are easy to use such as header, constant, or simple languages. Data format and type conversion Camel is payload-agnostic. This means that it can support any kind of message. Depending on the endpoints, it could be required to convert from one format to another. That's why Camel supports different data formats, in a pluggable way. This means that Camel can marshall or unmarshall a message in a given format. For instance, in addition to the standard JVM serialization, Camel natively supports Avro, JSON, protobuf, JAXB, XmlBeans, XStream, JiBX, SOAP, and so on. Depending on the endpoints and your need, you can explicitly define the data format during the processing of the message. On the other hand, Camel knows the expected format and type of endpoints. Thanks to this, Camel looks for a type converter, allowing to implicitly transform a message from one format to another. You can also explicitly define the type converter of your choice at some points during the processing of the message. Camel provides a set of ready-to-use type converters, but, as Camel supports a pluggable model, you can extend it by providing your own type converters. It's a simple POJO to implement. Easy configuration and URI Camel uses a different approach based on URI. The endpoint itself and its configuration are on the URI. The URI is human readable and provides the details of the endpoint, which is the endpoint component and the endpoint configuration. As this URI is part of the complete configuration (which defines what we name a route, as we will see later), it's possible to have a complete overview of the integration logic and connectivity in a row. Lightweight and different deployment topologies Camel itself is very light. The Camel core is only around 2 MB, and contains everythingrequired to run Camel. As it's based on a pluggable architecture, all Camel components are provided as external modules, allowing you to install only what you need, without installing superfluous and needlessly heavy modules. Camel is based on simple POJO, which means that the Camel core doesn't depend on other frameworks: it's an atomic framework and is ready to use. All other modules (components, DSL, and so on) are built on top of this Camel core. Moreover, Camel is not tied to one container for deployment. Camel supports a wide range of containers to run. They are as follows: A J2EE application server such as WebSphere, WebLogic, JBoss, and so on A Web container such as Apache Tomcat An OSGi container such as Apache Karaf A standalone application using frameworks such as Spring Camel gives a lot of flexibility, allowing you to embed it into your application or to use an enterprise-ready container. Quick prototyping and testing support In any integration project, it's typical that we have some part of the integration logic not yet available. For instance: The application to integrate with has not yet been purchased or not yet ready The remote system to integrate with has a heavy cost, not acceptable during the development phase Multiple teams work in parallel, so we may have some kinds of deadlocks between the teams As a complete integration framework, Camel provides a very easy way to prototype part of the integration logic. Even if you don't have the actual system to integrate, you can simulate this system (mock), as it allows you to implement your integration logic without waiting for dependencies. The mocking support is directly part of the Camel core and doesn't require any additional dependency. Along the same lines, testing is also crucial in an integration project. In such a kind of project, a lot of errors can happen and most are unforeseen. Moreover, a small change in an integration process might impact a lot of other processes. Camel provides the tools to easily test your design and integration logic, allowing you to integrate this in a continuous integration platform. Management and monitoring using JMX Apache Camel uses the Java Management Extension (JMX) standard and provides a lot of insights into the system using MBeans (Management Beans), providing a detailed view of the following current system: The different integration processes with the associated metrics The different components and endpoints with the associated metrics Moreover, these MBeans provide more insights than metrics. They also provide the operations to manage Camel. For instance, the operations allow you to stop an integration process, to suspend an endpoint, and so on. Using a combination of metrics and operations, you can configure a very agile integration solution. Active community The Apache Camel community is very active. This means that potential issues are identified very quickly and a fix is available soon after. However, it also means that a lot of ideas and contributions are proposed, giving more and more features to Camel. Another big advantage of an active community is that you will never be alone; a lot of people are active on the mailing lists who are ready to answer your question and provide advice. Summary Apache Camel is an enterprise integration solution used in many large organizations with enterprise support available through RedHat or Talend. Resources for Article: Further resources on this subject: Getting Started [article] A Quick Start Guide to Flume [article] Best Practices [article]

In this article by Mayank Joshi, the author of Mastering Chef, we'll learn how to go about building custom Knife plugins and we'll also see how we can write custom handlers that can help us extend the functionality provided by a chef-client run to report any issues with a chef-client run. (For more resources related to this topic, see here.) Custom Knife plugins Knife is one of the most widely used tools in the Chef ecosystem. Be it managing your clients, nodes, cookbooks, environments, roles, users, or handling stuff such as provisioning machines in Cloud environments such as Amazon AWS, Microsoft Azure, and so on, there is a way to go about doing all of these things through Knife. However, Knife, as provided during installation of Chef, isn't capable of performing all these tasks on its own. It comes with a basic set of functionalities, which helps provide an interface between the local Chef repository, workstation and the Chef server. The following are the functionalities, which is provided, by default, by the Knife executable: Management of nodes Management of clients and users Management of cookbooks, roles, and environments Installation of chef-client on the nodes through bootstrapping Searching for data that is indexed on the Chef server. However, apart from these functions, there are plenty more functions that can be performed using Knife; all this is possible through the use of plugins. Knife plugins are a set of one (or more) subcommands that can be added to Knife to support an additional functionality that is not built into the base set of Knife subcommands. Most of the Knife plugins are initially built by users such as you, and over a period of time, they are incorporated into the official Chef code base. A Knife plugin is usually installed into the ~/.chef/plugins/knife directory, from where it can be executed just like any other Knife subcommand. It can also be loaded from the .chef/plugins/knife directory in the Chef repository or if it's installed through RubyGems, it can be loaded from the path where the executable is installed. Ideally, a plugin should be kept in the ~/.chef/plugins/knife directory so that it's reusable across projects, and also in the .chef/plugins/knife directory of the Chef repository so that its code can be shared with other team members. For distribution purpose, it should ideally be distributed as a Ruby gem. The skeleton of a Knife plugin A Knife plugin is structured somewhat like this: require 'chef/knife'   module ModuleName class ClassName < Chef::Knife      deps do      require 'chef/dependencies'    end      banner "knife subcommand argument VALUE (options)"      option :name_of_option      :short => "-l value",      :long => "--long-option-name value",      :description => "The description of the option",      :proc => Proc.new { code_to_be_executed },      :boolean => true | false,      :default => default_value      def run      #Code    end end end Let's look at this skeleton, one line at a time: require: This is used to require other Knife plugins required by a new plugin. module ModuleName: This defines the namespace in which the plugin will live. Every Knife plugin lives in its own namespace. class ClassName < Chef::Knife: This declares that a plugin is a subclass of Knife. deps do: This defines a list of dependencies. banner: This is used to display a message when a user enters Knife subcommand –help. option :name_of_option: This defines all the different command line options available for this new subcommand. def run: This is the place in which we specify the Ruby code that needs to be executed. Here are the command-line options: :short defines the short option name :long defines the long option name :description defines a description that is displayed when a user enters knife subclassName –help :boolean defines whether an option is true or false; if the :short and :long names define value, then this attribute should not be used :proc defines the code that determines the value for this option :default defines a default value The following example shows a part of a Knife plugin named knife-windows: require 'chef/knife' require 'chef/knife/winrm_base'base'   class Chef class Knife    class Winrm < Knife        include Chef::Knife::WinrmBase        deps do        require 'readline'        require 'chef/search/query'        require 'em-winrm'      end        attr_writer :password        banner "knife winrm QUERY COMMAND (options)"        option :attribute,        :short => "-a ATTR",        :long => "--attribute ATTR",        :description => "The attribute to use for opening the connection - default is fqdn",        :default => "fqdn"        ... # more options        def session        session_opts = {}        session_opts[:logger] = Chef::Log.logger if Chef::Log.level == :debug        @session ||= begin          s = EventMachine::WinRM::Session.new(session_opts)          s.on_output do |host, data|            print_data(host, data)          end          s.on_error do |host, err|            print_data(host, err, :red)          end          s.on_command_complete do |host|             host = host == :all ? 'All Servers' : host            Chef::Log.debug("command complete on #{host}")          end          s        end        end        ... # more def blocks      end end end Namespace As we saw with skeleton, the Knife plugin should have its own namespace and the namespace is declared using the module method as follows: require 'chef/knife' #Any other require, if needed   module NameSpace class SubclassName < Chef::Knife Here, the plugin is available under the namespace called NameSpace. One should keep in mind that Knife loads the subcommand irrespective of the namespace to which it belongs. Class name The class name declares a plugin as a subclass of both Knife and Chef. For example: class SubclassName < Chef::Knife The capitalization of the name is very important. The capitalization pattern can be used to define the word grouping that makes the best sense for the use of a plugin. For example, if we want our plugin subcommand to work as follows: knife bootstrap hdfs We should have our class name as: BootstrapHdfs. If, say, we used a class name such as BootStrapHdfs, then our subcommand would be as follows: knife boot strap hdfs It's important to remember that a plugin can override an existing Knife subcommand. For example, we already know about commands such as knife cookbook upload. If you want to override the current functionality of this command, all you need to do is create a new plugin with the following name: class CookbookUpload < Chef::Knife Banner Whenever a user enters the knife –help command, he/she is presented with a list of available subcommands. For example: knife --help Usage: knife sub-command (options)    -s, --server-url URL             Chef Server URL Available subcommands: (for details, knife SUB-COMMAND --help)   ** BACKUP COMMANDS ** knife backup export [COMPONENT [COMPONENT ...]] [-D DIR] (options) knife backup restore [COMPONENT [COMPONENT ...]] [-D DIR] (options)   ** BOOTSTRAP COMMANDS ** knife bootstrap FQDN (options) .... Let us say we are creating a new plugin and we would want Knife to be able to list it when a user enters the knife –help command. To accomplish this, we would need to make use of banner. For example, let's say we've a plugin called BootstrapHdfs with the following code: module NameSpace class BootstrapHdfs < Chef::Knife    ...    banner "knife bootstrap hdfs (options)"    ... end end Now, when a user enters the knife –help command, he'll see the following output: ** BOOTSTRAPHDFS COMMANDS ** knife bootstrap hdfs (options) Dependencies Reusability is one of the key paradigms in development and the same is true for Knife plugins. If you want a functionality of one Knife plugin to be available in another, you can use the deps method to ensure that all the necessary files are available. The deps method acts like a lazy loader, and it ensures that dependencies are loaded only when a plugin that requires them is executed. This is one of the reasons for using deps over require, as the overhead of the loading classes is reduced, thereby resulting in code with a lower memory footprint; hence, faster execution. One can use the following syntax to specify dependencies: deps do require 'chef/knife/name_of_command' require 'chef/search/query' #Other requires to fullfill dependencies end Requirements One can acquire the functionality available in other Knife plugins using the require method. This method can also be used to require the functionality available in other external libraries. This method can be used right at the beginning of the plugin script, however, it's always wise to use it inside deps, or else the libraries will be loaded even when they are not being put to use. The syntax to use require is fairly simple, as follows: require 'path_from_where_to_load_library' Let's say we want to use some functionalities provided by the bootstrap plugin. In order to accomplish this, we will first need to require the plugin: require 'chef/knife/bootstrap' Next, we'll need to create an object of that plugin: obj = Chef::Knife::Bootstrap.new Once we've the object with us, we can use it to pass arguments or options to that object. This is accomplished by changing the object's config and the name_arg variables. For example: obj.config[:use_sudo] = true Finally, we can run the plugin using the run method as follows: obj.run Options Almost every other Knife plugin accepts some command line option or other. These options can be added to a Knife subcommand using the option method. An option can have a Boolean value, string value, or we can even write a piece of code to determine the value of an option. Let's see each of them in action once: An option with a Boolean value (true/false): option :true_or_false, :short => "-t", :long => "—true-or-false", :description => "True/False?", :boolean => true | false, :default => true Here is an option with a string value: option :some_string_value, :short => "-s VALUE", :long => "—some-string-value VALUE", :description => "String value", :default => "xyz" An option where a code is used to determine the option's value: option :tag, :short => "-T T=V[,T=V,...]", :long => "—tags Tag=Value[,Tag=Value,...]", :description => "A list of tags", :proc => Proc.new { |tags| tag.split(',') } Here the proc attribute will convert a list of comma-separated values into an array. All the options that are sent to the Knife subcommand through a command line are available in form of a hash, which can be accessed using the config method. For example, say we had an option: option :option1 :short => "-s VALUE", :long => "—some-string-value VALUE", :description => "Some string value for option1", :default => "option1" Now, while issuing the Knife subcommand, say a user entered something like this: $ knife subcommand –option1 "option1_value" We can access this value for option1 in our Knife plugin run method using config[:option1] When a user enters the knife –help command, the description attributes are displayed as part of help. For example: **EXAMPLE COMMANDS** knife example -s, --some-type-of-string-value    This is not a random string value. -t, --true-or-false                 Is this value true? Or is this value false? -T, --tags                         A list of tags associated with the virtual machine. Arguments A Knife plugin can also accept the command-line arguments that aren't specified using the option flag, for example, knife node show NODE. These arguments are added using the name_args method: require 'chef/knife' module MyPlugin class ShowMsg << Chef::Knife    banner 'knife show msg MESSAGE'    def run      unless name_args.size == 1      puts "You need to supply a string as an argument."        show_usage        exit 1      end      msg = name_args.join(" ")      puts msg    end end end Let's see this in action: knife show msg You need to supply a string as an argument. USAGE: knife show msg MESSAGE    -s, --server-url URL             Chef Server URL        --chef-zero-host HOST       Host to start chef-zero on ... Here, we didn't pass any argument to the subcommand and, rightfully, Knife sent back a message saying You need to supply a string as an argument. Now, let's pass a string as an argument to the subcommand and see how it behaves: knife show msg "duh duh" duh duh Under the hood what's happening is that name_args is an array, which is getting populated by the arguments that we have passed in the command line. In the last example, the name_args array would've contained two entries ("duh","duh"). We use the join method of the Array class to create a string out of these two entities and, finally, print the string. The run method Every Knife plugin will have a run method, which will contain the code that will be executed when the user executes the subcommand. This code contains the Ruby statements that are executed upon invocation of the subcommand. This code can access the options values using the config[:option_hash_symbol_name] method. Search inside a custom Knife plugin Search is perhaps one of the most powerful and most used functionalities provided by Chef. By incorporating a search functionality in our custom Knife plugin, we can accomplish a lot of tasks, which would otherwise take a lot of efforts to accomplish. For example, say we have classified our infrastructure into multiple environments and we want a plugin that can allow us to upload a particular file or folder to all the instances in a particular environment on an ad hoc basis, without invoking a full chef-client run. This kind of stuff is very much doable by incorporating a search functionality into the plugin and using it to find the right set of nodes in which you want to perform a certain operation. We'll look at one such plugin in the next section. To be able to use Chef's search functionality, all you need to do is to require the Chef's query class and use an object of the Chef::Search::Query class to execute a query against the Chef server. For example: require 'chef/search/query' query_object = Chef::Search::Query.new query = 'chef_environment:production' query_object.search('node',query) do |node| puts "Node name = #{node.name}" end Since the name of a node is generally FQDN, you can use the values returned in node.name to connect to remote machines and use any library such as net-scp to allow users to upload their files/folders to a remote machine. We'll try to accomplish this task when we write our custom plugin at the end of this article. We can also use this information to edit nodes. For example, say we had a set of machines acting as web servers. Initially, all these machines were running Apache as a web server. However, as the requirements changed, we wanted to switch over to Nginx. We can run the following piece of code to accomplish this task: require 'chef/search/query'   query_object = Chef::Search::Query.new query = 'run_list:*recipe\[apache2\]*' query_object.search('node',query) do |node| ui.msg "Changing run_list to recipe[nginx] for #{node.name}" node.run_list("recipe[nginx]") node.save ui.msg "New run_list: #{node.run_list}" end knife.rb settings Some of the settings defined by a Knife plugin can be configured so that they can be set inside the knife.rb script. There are two ways to go about doing this: By using the :proc attribute of the option method and code that references Chef::Config[:knife][:setting_name] By specifying the configuration setting directly within the def Ruby blocks using either Chef::Config[:knife][:setting_name] or config[:setting_name] An option that is defined in this way can be configured in knife.rb by using the following syntax: knife [:setting_name] This approach is especially useful when a particular setting is used a lot. The precedence order for the Knife option is: The value passed via a command line. The value saved in knife.rb The default value. The following example shows how the Knife bootstrap command uses a value in knife.rb using the :proc attribute: option :ssh_port :short => '-p PORT', :long => '—ssh-port PORT', :description => 'The ssh port', :proc => Proc.new { |key| Chef::Config[:knife][:ssh_port] = key } Here Chef::Config[:knife][:ssh_port] tells Knife to check the knife.rb file for a knife[:ssh_port] setting. The following example shows how the Knife bootstrap command calls the knife ssh subcommand for the actual SSH part of running a bootstrap operation: def knife_ssh ssh = Chef::Knife::Ssh.new ssh.ui = ui ssh.name_args = [ server_name, ssh_command ] ssh.config[:ssh_user] = Chef::Config[:knife][:ssh_user] || config[:ssh_user] ssh.config[:ssh_password] = config[:ssh_password] ssh.config[:ssh_port] = Chef::Config[:knife][:ssh_port] || config[:ssh_port] ssh.config[:ssh_gateway] = Chef::Config[:knife][:ssh_gateway] || config[:ssh_gateway] ssh.config[:identity_file] = Chef::Config[:knife][:identity_file] || config[:identity_file] ssh.config[:manual] = true ssh.config[:host_key_verify] = Chef::Config[:knife][:host_key_verify] || config[:host_key_verify] ssh.config[:on_error] = :raise ssh end Let's take a look at the preceding code: ssh = Chef::Knife::Ssh.new creates a new instance of the Ssh subclass named ssh A series of settings in Knife ssh are associated with a Knife bootstrap using the ssh.config[:setting_name] syntax Chef::Config[:knife][:setting_name] tells Knife to check the knife.rb file for various settings It also raises an exception if any aspect of the SSH operation fails User interactions The ui object provides a set of methods that can be used to define user interactions and to help ensure a consistent user experience across all different Knife plugins. One should make use of these methods, rather than handling user interactions manually. Method Description ui.ask(*args, &block) The ask method calls the corresponding ask method of the HighLine library. More details about the HighLine library can be found at http://www.rubydoc.info/gems/highline/1.7.2. ui.ask_question(question, opts={}) This is used to ask a user a question. If :default => default_value is passed as a second argument, default_value will be used if the user does not provide any answer. ui.color (string, *colors) This method is used to specify a color. For example: server = connections.server.create(server_def)   puts "#{ui.color("Instance ID", :cyan)}: #{server.id}"   puts "#{ui.color("Flavor", :cyan)}: #{server.flavor_id}"   puts "#{ui.color("Image", :cyan)}: #{server.image_id}"   ...   puts "#{ui.color("SSH Key", :cyan)}: #{server.key_name}" print "n#{ui.color("Waiting for server", :magenta)}" ui.color?() This indicates that the colored output should be used. This is only possible if an output is sent across to a terminal. ui.confirm(question,append_instructions=true) This is used to ask (Y/N) questions. If a user responds back with N, the command immediately exits with the status code 3. ui.edit_data(data,parse_output=true) This is used to edit data. This will result in firing up of an editor. ui.edit_object(class,name) This method provides a convenient way to download an object, edit it, and save it back to the Chef server. It takes two arguments, namely, the class of object to edit and the name of object to edit. ui.error This is used to present an error to a user. ui.fatal This is used to present a fatal error to a user. ui.highline This is used to provide direct access to a highline object provided by many ui methods. ui.info This is used to present information to a user. ui.interchange This is used to determine whether the output is in a data interchange format such as JSON or YAML. ui.list(*args) This method is a way to quickly and easily lay out lists. This method is actually a wrapper to the list method provided by the HighLine library. More details about the HighLine library can be found at http://www.rubydoc.info/gems/highline/1.7.2. ui.msg(message) This is used to present a message to a user. ui.output(data) This is used to present a data structure to a user. This makes use of a generic default presenter. ui.pretty_print(data) This is used to enable the pretty_print output for JSON data. ui.use_presenter(presenter_class) This is used to specify a custom output presenter. ui.warn(message) This is used to present a warning to a user. For example, to show a fatal error in a plugin in the same way that it would be shown in Knife, do something similar to the following: unless name_args.size == 1    ui.fatal "Fatal error !!!"    show_usage    exit 1 end Exception handling In most cases, the exception handling available within Knife is enough to ensure that the exception handling for a plugin is consistent across all the different plugins. However, if the required one can handle exceptions in the same way as any other Ruby program, one can make use of the begin-end block, along with rescue clauses, to tell Ruby which exceptions we want to handle. For example: def raise_and_rescue begin    puts 'Before raise'    raise 'An error has happened.'    puts 'After raise' rescue    puts 'Rescued' end puts 'After begin block' end   raise_and_rescue If we were to execute this code, we'd get the following output: ruby test.rb Before raise Rescued After begin block A simple Knife plugin With the knowledge about how Knife's plugin system works, let's go about writing our very own custom Knife plugin, which can be quite useful for some users. Before we jump into the code, let's understand the purpose that this plugin is supposed to serve. Let's say we've a setup where our infrastructure is distributed across different environments and we've also set up a bunch of roles, which are used while we try to bootstrap the machines using Chef. So, there are two ways in which a user can identify machines: By environments By roles Actually, any valid Chef search query that returns a node list can be the criteria to identify machines. However, we are limiting ourselves to these two criteria for now. Often, there are situations where a user might want to upload a file or folder to all the machines in a particular environment, or to all the machines belonging to a particular role. This plugin will help users accomplish this task with lots of ease. The plugin will accept three arguments. The first one will be a key-value pair with the key being chef_environment or a role, the second argument will be a path to the file or folder that is required to be uploaded, and the third argument will be the path on a remote machine where the files/folders will be uploaded to. The plugin will use Chef's search functionality to find the FQDN of machines, and eventually make use of the net-scp library to transfer the file/folder to the machines. Our plugin will be called knife-scp and we would like to use it as follows: knife scp chef_environment:production /path_of_file_or_folder_locally /path_on_remote_machine Here is the code that can help us accomplish this feat: require 'chef/knife'   module CustomPlugins class Scp < Chef::Knife    banner "knife scp SEARCH_QUERY PATH_OF_LOCAL_FILE_OR_FOLDER PATH_ON_REMOTE_MACHINE"      option :knife_config_path,      :short => "-c PATH_OF_knife.rb",      :long => "--config PATH_OF_knife.rb",      :description => "Specify path of knife.rb",      :default => "~/.chef/knife.rb"      deps do      require 'chef/search/query'      require 'net/scp'      require 'parallel'    end      def run      if name_args.length != 3        ui.msg "Missing arguments! Unable to execute the command successfully."        show_usage        exit 1      end                  Chef::Config.from_file(File.expand_path("#{config[:knife_config_path]}"))      query = name_args[0]      local_path = name_args[1]      remote_path = name_args[2]      query_object = Chef::Search::Query.new      fqdn_list = Array.new      query_object.search('node',query) do |node|        fqdn_list << node.name      end      if fqdn_list.length < 1        ui.msg "No valid servers found to copy the files to"      end      unless File.exist?(local_path)        ui.msg "#{local_path} doesn't exist on local machine"        exit 1      end        Parallel.each((1..fqdn_list.length).to_a, :in_processes => fqdn_list.length) do |i|        puts "Copying #{local_path} to #{Chef::Config[:knife][:ssh_user]}@#{fqdn_list[i-1]}:#{remote_path} "        Net::SCP.upload!(fqdn_list[i-1],"#{Chef::Config[:knife][:ssh_user]}","#{local_path}","#{remote_path}",:ssh => { :keys => ["#{Chef::Config[:knife][:identity_file]}"] }, :recursive => true)      end    end end end This plugin uses the following additional gems: The parallel gem to execute statements in parallel. More information about this gem can be found at https://github.com/grosser/parallel. The net-scp gem to do the actual transfer. This gem is a pure Ruby implementation of the SCP protocol. More information about the gem can be found at https://github.com/net-ssh/net-scp. Both these gems and the Chef search library are required in the deps block to define the dependencies. This plugin accepts three command line arguments and uses knife.rb to get information about which user to connect over SSH and also uses knife.rb to fetch information about the SSH key file to use. All these command line arguments are stored in the name_args array. A Chef search is then used to find a list of servers that match the query, and eventually a parallel gem is used to parallely SCP the file from a local machine to a list of servers returned by a Chef query. As you can see, we've tried to handle a few error situations, however, there is still a possibility of this plugin throwing away errors as the Net::SCP.upload function can error out at times. Let's see our plugin in action: Case1: The file that is supposed to be uploaded doesn't exist locally. We expect the script to error out with an appropriate message: knife scp 'chef_environment:ft' /Users/mayank/test.py /tmp /Users/mayank/test.py doesn't exist on local machine Case2: The /Users/mayank/test folder is: knife scp 'chef_environment:ft' /Users/mayank/test /tmp Copying /Users/mayank/test to ec2-user@host02.ft.sychonet.com:/tmp Copying /Users/mayank/test to ec2-user@host01.ft.sychonet.com:/tmp Case3: A config other than /etc/chef/knife.rb is specified: knife scp -c /Users/mayank/.chef/knife.rb 'chef_environment:ft' /Users/mayank/test /tmp Copying /Users/mayank/test to ec2-user@host02.ft.sychonet.com:/tmp Copying /Users/mayank/test to ec2-user@host01.ft.sychonet.com:/tmp Distributing plugins using gems As you must have noticed, until now we've been creating our plugins under ~/.chef/plugins/knife. Though this is sufficient for plugins that are meant to be used locally, it's just not good enough to be distributed to a community. The most ideal way of distributing a Knife plugin is by packaging your plugin as a gem and distributing it via a gem repository such as rubygems.org. Even if publishing your gem to a remote gem repository sounds like a far-fetched idea, at least allowing people to install your plugin by building a gem locally and installing it via gem install. This is a far better way than people downloading your code from an SCM repository and copying it over to either ~/.chef/plugins/knife or any other folder they've configured for the purpose of searching for custom Knife plugins. With distributing your plugin using gems, you ensure that the plugin is installed in a consistent way and you can also ensure that all the required libraries are preinstalled before a plugin is ready to be consumed by users. All the details required to create a gem are contained in a file known as Gemspec, which resides at the root of your project's directory and is typically named the <project_name>.gemspec. Gemspec file that consists of the structure, dependencies, and metadata required to build your gem. The following is an example of a .gemspec file: Gem::Specification.new do |s| s.name = 'knife-scp' s.version = '1.0.0' s.date = '2014-10-23' s.summary = 'The knife-scp knife plugin' s.authors = ["maxcoder"] s.email = 'maxcoder@sychonet.com" s.files = ["lib/chef/knife/knife-scp.rb"] s.homepage = "https://github.com/maxc0d3r/knife-plugins" s.add_runtime_dependency "parallel","~> 1.2", ">= 1.2.0" s.add_runtime_dependency "net-scp","~> 1.2", ">= 1.2.0" end The s.files variable contains the list of files that will be deployed by a gem install command. Knife can load the files from gem_path/lib/chef/knife/<file_name>.rb, and hence we've kept the knife-scp.rb script in that location. The s.add_runtime_dependency dependency is used to ensure that the required gems are installed whenever a user tries to install our gem. Once the file is there, we can just run a gem build to build our gem file as follows: knife-scp git:(master) x gem build knife-scp.gemspec WARNING: licenses is empty, but is recommended. Use a license abbreviation from: http://opensource.org/licenses/alphabetical WARNING: See http://guides.rubygems.org/specification-reference/ for help Successfully built RubyGem Name: knife-scp Version: 1.0.0 File: knife-scp-1.0.0.gem The gem file is created and now, we can just use gem install knife-scp-1.0.0.gem to install our gem. This will also take care of the installation of any dependencies such as parallel, net-scp gems, and so on. You can find a source code for this plugin at the following location: https://github.com/maxc0d3r/knife-plugins. Once the gem has been installed, the user can run it as mentioned earlier. For the purpose of distribution of this gem, it can either be pushed using a local gem repository, or it can be published to https://rubygems.org/. To publish it to https://rubygems.org/, create an account there. Run the following command to log in using a gem: gem push This will ask for your email address and password. Next, push your gem using the following command: gem push your_gem_name.gem That's it! Now you should be able to access your gem at the following location: http://www.rubygems.org/gems/your_gem_name. As you might have noticed, we've not written any tests so far to check the plugin. It's always a good idea to write test cases before submitting your plugin to the community. It's useful both to the developer and consumers of the code, as both know that the plugin is going to work as expected. Gems support adding test files into the package itself so that tests can be executed when a gem is downloaded. RSpec is a popular choice to test a framework, however, it really doesn't matter which tool you use to test your code. The point is that you need to test and ship. Some popular Knife plugins, built by a community, and their uses, are as follows: knife-elb: This plugin allows the automation of the process of addition and deletion of nodes from Elastic Load Balancers on AWS. knife-inspect: This plugin allows you to see the difference between what's on a Chef server versus what's on a local Chef repository. knife-community: This plugin helps to deploy Chef cookbooks to Chef Supermarket. knife-block: This plugin allows you to configure and manage multiple Knife configuration files against multiple Chef servers. knife-tagbulk: This plugin allows bulk tag operations (creation or deletion) using standard Chef search queries. More information about the plugin can be found at: https://github.com/priestjim/knife-tagbulk. You can find a lot of other useful community-written plugins at: https://docs.chef.io/community_plugin_knife.html. Custom Chef handlers A Chef handler is used to identify different situations that might occur during a chef-client run, and eventually it instructs the chef-client on what it should do to handle these situations. There are three types of handlers in Chef: The exception handler: This is used to identify situations that have caused a chef-client run to fail. This can be used to send out alerts over an email or dashboard. The report handler: This is used to report back when a chef-client run has successfully completed. This can report details about the run, such as the number of resources updated, time taken for a chef-client run to complete, and so on. The start handler: This is used to run events at the beginning of a chef-client run. Writing custom Chef handlers is nothing more than just inheriting your class from Chef::Handler and overriding the report method. Let's say we want to send out an email every time a chef-client run breaks. Chef provides a failed? method to check the status of a chef-client run. The following is a very simple piece of code that will help us accomplish this: require 'net/smtp' module CustomHandler class Emailer < Chef::Handler    def send_email(to,opts={})      opts[:server] ||= 'localhost'      opts[:from] ||='maxcoder@sychonet.com'      opts[:subject] ||='Error'      opts[:body] ||= 'There was an error running chef-client'        msg = <<EOF      From: <#{opts[:from]}>      To: #{to}      Subject: #{opts[:subject]}        #{opts[:body]}      EOF        Net::SMTP.start(opts[:server]) do |smtp|        smtp.send_message msg, opts[:from], to      end    end      def report      name = node.name      subject = "Chef run failure on #{name}"      body = [run_status.formatted_exception]      body += ::Array(backtrace).join("n")      if failed?        send_email(          "ops@sychonet.com",          :subject => subject,          :body => body        )      end    end end end If you don't have the required libraries already installed on your machine, you'll need to make use of chef_gem to install them first before you actually make use of this code. With your handler code ready, you can make use of the chef_handler cookbook to install this custom handler. To do so, create a new cookbook, email-handler, and copy the file emailer.rb created earlier to the file's source. Once done, add the following recipe code: include_recipe 'chef_handler'   handler_path = node['chef_handler']['handler_path'] handler = ::File.join handler_path, 'emailer'   cookbook_file "#{handler}.rb" do source "emailer.rb" end   chef_handler "CustomHandler::Emailer" do source handler    action :enable end Now, just include this handler into your base role, or at the start of run_list and during the next chef-client run, if anything breaks, an email will be sent across to ops@sychonet.com. You can configure many different kinds of handlers like the ones that push notifications over to IRC, Twitter, and so on, or you may even write them for scenarios where you don't want to leave a component of a system in a state that is undesirable. For example, say you were in a middle of a chef-client run that adds/deletes collections from Solr. Now, you might not want to leave the Solr setup in a messed-up state if something were to go wrong with the provisioning process. In order to ensure that a system is in the right state, you can write your own custom handlers, which can be used to handle such situations and revert the changes done until now by the chef-client run. Summary In this article, we learned about how custom Knife plugins can be used. We also learned how we can write our own custom Knife plugin and distribute it by packaging it as a gem. Finally, we learned about custom Chef handlers and how they can be used effectively to communicate information and statistics about a chef-client run to users/admins, or handle any issues with a chef-client run. Resources for Article: Further resources on this subject: An Overview of Automation and Advent of Chef [article] Testing Your Recipes and Getting Started with ChefSpec [article] Chef Infrastructure [article]

In this article by Alexandru Vaduva, author of the book Learning Embedded Linux Using the Yocto Project, you will be presented with information about various concepts that appeared in the Linux virtualization article. As some of you might know, this subject is quite vast and selecting only a few components to be explained is also a challenge. I hope my decision would please most of you interested in this area. The information available in this article might not fit everyone's need. For this purpose, I have attached multiple links for more detailed descriptions and documentation. As always, I encourage you to start reading and finding out more, if necessary. I am aware that I cannot put all the necessary information in only a few words. In any Linux environment today, Linux virtualization is not a new thing. It has been available for more than ten years and has advanced in a really quick and interesting manner. The question now does not revolve around virtualization as a solution for me, but more about what virtualization solutions to deploy and what to virtualize. (For more resources related to this topic, see here.) Linux virtualization The first benefit everyone sees when looking at virtualization is the increase in server utilization and the decrease in energy costs. Using virtualization, the workloads available on a server are maximized, which is very different from scenarios where hardware uses only a fraction of the computing power. It can reduce the complexity of interaction with various environments and it also offers an easier-to-use management system. Today, working with a large number of virtual machines is not as complicated as interaction with a few of them because of the scalability most tools offer. Also, the time of deployment has really decreased. In a matter of minutes, you can deconfigure and deploy an operating system template or create a virtual environment for a virtual appliance deploy. One other benefit virtualization brings is flexibility. When a workload is just too big for allocated resources, it can be easily duplicated or moved on another environment that suit its needs better on the same hardware or on a more potent server. For a cloud-based solution regarding this problem, the sky is the limit here. The limit may be imposed by the cloud type on the basis of whether there are tools available for a host operating system. Over time, Linux was able to provide a number of great choices for every need and organization. Whether your task involves server consolidation in an enterprise data centre, or improving a small nonprofit infrastructure, Linux should have a virtualization platform for your needs. You simply need to figure out where and which project you should chose. Virtualization is extensive, mainly because it contains a broad range of technologies, and also since large portions of the terms are not well defined. In this article, you will be presented with only components related to the Yocto Project and also to a new initiative that I personally am interested in. This initiative tries to make Network Function Virtualization (NFV) and Software Defined Networks (SDN) a reality and is called Open Platform for NFV (OPNFV). It will be explained here briefly. SDN and NFV I have decided to start with this topic because I believe it is really important that all the research done in this area is starting to get traction with a number of open source initiatives from all sorts of areas and industries. Those two concepts are not new. They have been around for 20 years since they were first described, but the last few years have made possible it for them to resurface as real and very possible implementations. The focus of this article will be on the NFV article since it has received the most amount of attention, and also contains various implementation proposals. NFV NFV is a network architecture concept used to virtualize entire categories of network node functions into blocks that can be interconnected to create communication services. It is different from known virtualization techniques. It uses Virtual Network Functions (VNF) that can be contained in one or more virtual machines, which execute different processes and software components available on servers, switches, or even a cloud infrastructure. A couple of examples include virtualized load balancers, intrusion detected devices, firewalls, and so on. The development product cycles in the telecommunication industry were very rigorous and long due to the fact that the various standards and protocols took a long time until adherence and quality meetings. This made it possible for fast moving organizations to become competitors and made them change their approach. In 2013, an industry specification group published a white paper on software-defined networks and OpenFlow. The group was part of European Telecommunications Standards Institute (ETSI) and was called Network Functions Virtualisation. After this white paper was published, more in-depth research papers were published, explaining things ranging from terminology definitions to various use cases with references to vendors that could consider using NFV implementations. ETSI NFV The ETSI NFV workgroup has appeared useful for the telecommunication industry to create more agile cycles of development and also make it able to respond in time to any demands from dynamic and fast changing environments. SDN and NFV are two complementary concepts that are key enabling technologies in this regard and also contain the main ingredients of the technology that are developed by both telecom and IT industries. The NFV framework consist of six components: NFV Infrastructure (NFVI): It is required to offer support to a variety of use cases and applications. It comprises of the totality of software and hardware components that create the environment for which VNF is deployed. It is a multitenant infrastructure that is responsible for the leveraging of multiple standard virtualization technologies use cases at the same time. It is described in the following NFV Industry Specification Groups (NFV ISG) documents: NFV Infrastructure Overview NFV Compute NFV Hypervisor Domain NFV Infrastructure Network Domain The following image presents a visual graph of various use cases and fields of application for the NFV Infrastructure NFV Management and Orchestration (MANO): It is the component responsible for the decoupling of the compute, networking, and storing components from the software implementation with the help of a virtualization layer. It requires the management of new elements and the orchestration of new dependencies between them, which require certain standards of interoperability and a certain mapping. NFV Software Architecture: It is related to the virtualization of the already implemented network functions, such as proprietary hardware appliances. It implies the understanding and transition from a hardware implementation into a software one. The transition is based on various defined patterns that can be used in a process. NFV Reliability and Availability: These are real challenges and the work involved in these components started from the definition of various problems, use cases, requirements, and principles, and it has proposed itself to offer the same level of availability as legacy systems. It relates to the reliability component and the documentation only sets the stage for future work. It only identifies various problems and indicates the best practices used in designing resilient NFV systems. NFV Performance and Portability: The purpose of NFV, in general, is to transform the way it works with networks of future. For this purpose, it needs to prove itself as wordy solution for industry standards. This article explains how to apply the best practices related to performance and portability in a general VNF deployment. NFV Security: Since it is a large component of the industry, it is concerned about and also dependent on the security of networking and cloud computing, which makes it critical for NFV to assure security. The Security Expert Group focuses on those concerns. An architectural of these components is presented here: After all the documentation is in place, a number of proof of concepts need to be executed in order to test the limitation of these components and accordingly adjust the theoretical components. They have also appeared to encourage the development of the NFV ecosystem. SDN Software-Defined Networking (SDN) is an approach to networking that offers the possibility to manage various services using the abstraction of available functionalities to administrators. This is realized by decoupling the system into a control plane and data plane and making decisions based on the network traffic that is sent; this represents the control plane realm, and where the traffic is forwarded is represented by the data plane. Of course, some method of communication between the control and data plane is required, so the OpenFlow mechanism entered into the equation at first; however other components could as well take its place. The intention of SDN was to offer an architecture that was manageable, cost-effective, adaptable, and dynamic, as well as suitable for the dynamic and high-bandwidth scenarios that are available today. The OpenFlow component was the foundation of the SDN solution. The SDN architecture permitted the following: Direct programming: The control plane is directly programmable because it is completely decoupled by the data plane. Programmatically configuration: SDN permitted management, configuration, and optimization of resources though programs. These programs could also be written by anyone because they were not dependent on any proprietary components. Agility: The abstraction between two components permitted the adjustment of network flows according to the needs of a developer. Central management: Logical components could be centered on the control plane, which offered a viewpoint of a network to other applications, engines, and so on. Opens standards and vendor neutrality: It is implemented using open standards that have simplified the SDN design and operations because of the number of instructions provided to controllers. This is smaller compared to other scenarios in which multiple vendor-specific protocols and devices should be handled. Also, meeting market requirements with traditional solutions would have been impossible, taking into account newly emerging markets of mobile device communication, Internet of Things (IoT), Machine to Machine (M2M), Industry 4.0, and others, all require networking support. Taking into consideration the available budgets for further development in various IT departments, were all faced to make a decision. It seems that the mobile device communication market all decided to move toward open source in the hope that this investment would prove its real capabilities, and would also lead to a brighter future. OPNFV The Open Platform for the NFV Project tries to offer an open source reference platform that is carrier-graded and tightly integrated in order to facilitate industry peers to help improve and move the NFV concept forward. Its purpose is to offer consistency, interoperability, and performance among numerous blocks and projects that already exist. This platform will also try to work closely with a variety of open source projects and continuously help with integration, and at the same time, fill development gaps left by any of them. This project is expected to lead to an increase in performance, reliability, serviceability, availability, and power efficiency, but at the same time, also deliver an extensive platform for instrumentation. It will start with the development of an NFV infrastructure and a virtualized infrastructure management system where it will combine a number of already available projects. Its reference system architecture is represented by the x86 architecture. The project's initial focus point and proposed implementation can be consulted in the following image. From this image, it can be easily seen that the project, although very young since it was started in November 2014, has had an accelerated start and already has a few implementation propositions. There are already a number of large companies and organizations that have started working on their specific demos. OPNFV has not waited for them to finish and is already discussing a number of proposed project and initiatives. These are intended both to meet the needs of their members as well as assure them of the reliability various components, such as continuous integration, fault management, test-bed infrastructure, and others. The following figure describes the structure of OPNFV: The project has been leveraging as many open source projects as possible. All the adaptations made to these project can be done in two places. Firstly, they can be made inside the project, if it does not require substantial functionality changes that could cause divergence from its purpose and roadmap. The second option complements the first and is necessary for changes that do not fall in the first category; they should be included somewhere in the OPNFV project's codebase. None of the changes that have been made should be up streamed without proper testing within the development cycle of OPNFV. Another important element that needs to be mentioned is that OPNFV does not use any specific or additional hardware. It only uses available hardware resources as long the VI-Ha reference point is supported. In the preceding image, it can be seen that this is already done by having providers, such as Intel for the computing hardware, NetApp for storage hardware, and Mellanox for network hardware components. The OPNFV board and technical steering committee have a quite large palette of open source projects. They vary from Infrastructure as a Service (IaaS) and hypervisor to the SDN controller and the list continues. This only offers the possibility for a large number of contributors to try some of the skills that maybe did not have the time to work on, or wanted to learn but did not have the opportunity to. Also, a more diversified community offers a broader view of the same subject. There are a large variety of appliances for the OPNFV project. The virtual network functions are diverse for mobile deployments where mobile gateways (such as Serving Gateway (SGW), Packet Data Network Gateway (PGW), and so on) and related functions (Mobility Management Entity (MME) and gateways), firewalls or application-level gateways and filters (web and e-mail traffic filters) are used to test diagnostic equipment (Service-Level Agreement (SLA) monitoring). These VNF deployments need to be easy to operate, scale, and evolve independently from the type of VNF that is deployed. OPNFV sets out to create a platform that has to support a set of qualities and use-cases as follows: A common mechanism is needed for the life-cycle management of VNFs, which include deployment, instantiation, configuration, start and stop, upgrade/downgrade, and final decommissioning A consistent mechanism is used to specify and interconnect VNFs, VNFCs, and PNFs; these are indepedant of the physical network infrastructure, network overlays, and so on, that is, a virtual link A common mechanism is used to dynamically instantiate new VNF instances or decommission sufficient ones to meet the current performance, scale, and network bandwidth needs A mechanism is used to detect faults and failure in the NFVI, VIM, and other components of an infrastructure as well as recover from these failures A mechanism is used to source/sink traffic from/to a physical network function to/from a virtual network function NFVI as a Service is used to host different VNF instances from different vendors on the same infrastructure There are some notable and easy-to-grasp use case examples that should be mentioned here. They are organized into four categories. Let's start with the first category: the Residential/Access category. It can be used to virtualize the home environment but it also provides fixed access to NFV. The next one is data center: it has the virtualization of CDN and provides use cases that deal with it. The mobile category consists of the virtualization of mobile core networks and IMS as well as the virtualization of mobile base stations. Lastly, there are cloud categories that include NFVIaaS, VNFaaS, the VNF forwarding graph (Service Chains), and the use cases of VNPaaS. More information about this project and various implementation components is available at https://www.opnfv.org/. For the definitions of missing terminologies, please consult http://www.etsi.org/deliver/etsi_gs/NFV/001_099/003/01.02.01_60/gs_NFV003v010201p.pdf. Virtualization support for the Yocto Project The meta-virtualization layer tries to create a long and medium term production-ready layer specifically for an embedded virtualization. This roles that this has are: Simplifying the way collaborative benchmarking and researching is done with tools, such as KVM/LxC virtualization, combined with advance core isolation and other techniques Integrating and contributing with projects, such as OpenFlow, OpenvSwitch, LxC, dmtcp, CRIU and others, which can be used with other components, such as OpenStack or Carrier Graded Linux. To summarize this in one sentence, this layer tries to provide support while constructing OpenEmbedded and Yocto Project-based virtualized solutions. The packages that are available in this layer, which I will briefly talk about are as follows: CRIU Docker LXC Irqbalance Libvirt Xen Open vSwitch This layer can be used in conjunction with the meta-cloud-services layer that offer cloud agents and API support for various cloud-based solutions. In this article, I am referring to both these layers because I think it is fit to present these two components together. Inside the meta-cloud-services layer, there are also a couple of packages that will be discussed and briefly presented, as follows: openLDAP SPICE Qpid RabbitMQ Tempest Cyrus-SASL Puppet oVirt OpenStack Having mentioned these components, I will now move on with the explanation of each of these tools. Let's start with the content of the meta-virtualization layer, more exactly with CRIU package, a project that implements Checkpoint/Restore In Userspace for Linux. It can be used to freeze an already running application and checkpoint it to a hard drive as a collection of files. These checkpoints can be used to restore and execute the application from that point. It can be used as part of a number of use cases, as follows: Live migration of containers: It is the primary use case for a project. The container is check pointed and the resulting image is moved into another box and restored there, making the whole experience almost unnoticeable by the user. Upgrading seamless kernels: The kernel replacement activity can be done without stopping activities. It can be check pointed, replaced by calling kexec, and all the services can be restored afterwards. Speeding up slow boot services: It is a service that has a slow boot procedure, can be check pointed after the first start up is finished, and for consecutive starts, can be restored from that point. Load balancing of networks: It is a part of the TCP_REPAIR socket option and switches the socket in a special state. The socket is actually put into the state expected from it at the end of the operation. For example, if connect() is called, the socket will be put in an ESTABLISHED state as requested without checking for acknowledgment of communication from the other end, so offloading could be at the application level. Desktop environment suspend/resume: It is based on the fact that the suspend/restore action for a screen session or an X application is by far faster than the close/open operation. High performance and computing issues: It can be used for both load balancing of tasks over a cluster and the saving of cluster node states in case a crash occurs. Having a number of snapshots for application doesn't hurt anybody. Duplication of processes: It is similar to the remote fork() operation. Snapshots for applications: A series of application states can be saved and reversed back if necessary. It can be used both as a redo for the desired state of an application as well as for debugging purposes. Save ability in applications that do not have this option: An example of such an application could be games in which after reaching a certain level, the establishment of a checkpoint is the thing you need. Migrate a forgotten application onto the screen: If you have forgotten to include an application onto the screen and you are already there, CRIU can help with the migration process. Debugging of applications that have hung: For services that are stuck because of git and need a quick restart, a copy of the services can be used to restore. A dump process can also be used and through debugging, the cause of the problem can be found. Application behavior analysis on a different machine: For those applications that could behave differently from one machine to another, a snapshot of the application in question can be used and transferred into the other. Here, the debugging process can also be an option. Dry running updates: Before a system or kernel update on a system is done, its services and critical applications could be duplicated onto a virtual machine and after the system update and all the test cases pass, the real update can be done. Fault-tolerant systems: It can be used successfully for process duplication on other machines. The next element is irqbalance, a distributed hardware interrupt system that is available across multiple processors and multiprocessor systems. It is, in fact, a daemon used to balance interrupts across multiple CPUs, and its purpose is to offer better performances as well as better IO operation balance on SMP systems. It has alternatives, such as smp_affinity, which could achieve maximum performance in theory, but lacks the same flexibility that irqbalance provides. The libvirt toolkit can be used to connect with the virtualization capabilities available in the recent Linux kernel versions that have been licensed under the GNU Lesser General Public License. It offers support for a large number of packages, as follows: KVM/QEMU Linux supervisor Xen supervisor LXC Linux container system OpenVZ Linux container system Open Mode Linux a paravirtualized kernel Hypervisors that include VirtualBox, VMware ESX, GSX, Workstation and player, IBM PowerVM, Microsoft Hyper-V, Parallels, and Bhyve Besides these packages, it also offers support for storage on a large variety of filesystems, such as IDE, SCSI or USB disks, FiberChannel, LVM, and iSCSI or NFS, as well as support for virtual networks. It is the building block for other higher-level applications and tools that focus on the virtualization of a node and it does this in a secure way. It also offers the possibility of a remote connection. For more information about libvirt, take a look at its project goals and terminologies at http://libvirt.org/goals.html. The next is Open vSwitch, a production-quality implementation of a multilayer virtual switch. This software component is licensed under Apache 2.0 and is designed to enable massive network automations through various programmatic extensions. The Open vSwitch package, also abbreviated as OVS, provides a two stack layer for hardware virtualizations and also supports a large number of the standards and protocols available in a computer network, such as sFlow, NetFlow, SPAN, CLI, RSPAN, 802.1ag, LACP, and so on. Xen is a hypervisor with a microkernel design that provides services offering multiple computer operating systems to be executed on the same architecture. It was first developed at the Cambridge University in 2003, and was developed under GNU General Public License version 2. This piece of software runs on a more privileged state and is available for ARM, IA-32, and x86-64 instruction sets. A hypervisor is a piece of software that is concerned with the CPU scheduling and memory management of various domains. It does this from the domain 0 (dom0), which controls all the other unprivileged domains called domU; Xen boots from a bootloader and usually loads into the dom0 host domain, a paravirtualized operating system. A brief look at the Xen project architecture is available here: Linux Containers (LXC) is the next element available in the meta-virtualization layer. It is a well-known set of tools and libraries that offer virtualization at the operating system level by offering isolated containers on a Linux control host machine. It combines the functionalities of kernel control groups (cgroups) with the support for isolated namespaces to provide an isolated environment. It has received a fair amount of attention mostly due to Docker, which will be briefly mentioned a bit later. Also, it is considered a lightweight alternative to full machine virtualization. Both of these options, containers and machine virtualization, have a fair amount of advantages and disadvantages. If the first option, containers offer low overheads by sharing certain components, and it may turn out that it does not have a good isolation. Machine virtualization is exactly the opposite of this and offers a great solution to isolation at the cost of a bigger overhead. These two solutions could also be seen as complementary, but this is only my personal view of the two. In reality, each of them has its particular set of advantages and disadvantages that could sometimes be uncomplementary as well. More information about Linux containers is available at https://linuxcontainers.org/. The last component of the meta-virtualization layer that will be discussed is Docker, an open source piece of software that tries to automate the method of deploying applications inside Linux containers. It does this by offering an abstraction layer over LXC. Its architecture is better described in this image: As you can see in the preceding diagram, this software package is able to use the resources of the operating system. Here, I am referring to the functionalities of the Linux kernel and have isolated other applications from the operating system. It can do this either through LXC or other alternatives, such as libvirt and systemd-nspawn, which are seen as indirect implementations. It can also do this directly through the libcontainer library, which has been around since the 0.9 version of Docker. Docker is a great component if you want to obtain automation for distributed systems, such as large-scale web deployments, service-oriented architectures, continuous deployment systems, database clusters, private PaaS, and so on. More information about its use cases is available at https://www.docker.com/resources/usecases/. Make sure you take a look at this website; interesting information is often here. After finishing with the meta-virtualization layer, I will move next to the meta-cloud-services layer that contains various elements. I will start with Simple Protocol for Independent Computing Environments (Spice). This can be translated into a remote-display system for virtualized desktop devices. It initially started as a closed source software, and in two years it was decided to make it open source. It then became an open standard to interaction with devices, regardless of whether they are virtualized one not. It is built on a client-server architecture, making it able to deal with both physical and virtualized devices. The interaction between backend and frontend is realized through VD-Interfaces (VDI), and as shown in the following diagram, its current focus is the remote access to QEMU/KVM virtual machines: Next on the list is oVirt, a virtualization platform that offers a web interface. It is easy to use and helps in the management of virtual machines, virtualized networks, and storages. Its architecture consists of an oVirt Engine and multiple nodes. The engine is the component that comes equipped with a user-friendly interface to manage logical and physical resources. It also runs the virtual machines that could be either oVirt nodes, Fedora, or CentOS hosts. The only downfall of using oVirt is that it only offers support for a limited number of hosts, as follows: Fedora 20 CentOS 6.6, 7.0 Red Hat Enterprise Linux 6.6, 7.0 Scientific Linux 6.6, 7.0 As a tool, it is really powerful. It offers integration with libvirt for Virtual Desktops and Servers Manager (VDSM) communications with virtual machines and also support for SPICE communication protocols that enable remote desktop sharing. It is a solution that was started and is mainly maintained by Red Hat. It is the base element of their Red Hat Enterprise Virtualization (RHEV), but one thing is interesting and should be watched out for is that Red Hat now is not only a supporter of projects, such as oVirt and Aeolus, but has also been a platinum member of the OpenStack foundation since 2012. For more information on projects, such as oVirt, Aeolus, and RHEV, the following links can be useful to you: http://www.redhat.com/promo/rhev3/?sc_cid=70160000000Ty5wAAC&offer_id=70160000000Ty5NAAS, http://www.aeolusproject.org/ and http://www.ovirt.org/Home. I will move on to a different component now. Here, I am referring to the open source implementation of the Lightweight Directory Access Protocol, simply called openLDAP. Although it has a somewhat controverted license called OpenLDAP Public License, which is similar in essence to the BSD license, it is not recorded at opensource.org, making it uncertified by Open Source Initiative (OSI). This software component comes as a suite of elements, as follows: A standalone LDAP daemon that plays the role of a server called slapd A number of libraries that implement the LDAP protocol Last but not the least, a series of tools and utilities that also have a couple of clients samples between them There are also a number of additions that should be mentioned, such as ldapc++ and libraries written in C++, JLDAP and the libraries written in Java; LMDB, a memory mapped database library; Fortress, a role-based identity management; SDK, also written in Java; and a JDBC-LDAP Bridge driver that is written in Java and called JDBC-LDAP. Cyrus-SASL is a generic client-server library implementation for Simple Authentication and Security Layer (SASL) authentication. It is a method used for adding authentication support for connection-based protocols. A connection-based protocol adds a command that identifies and authenticates a user to the requested server and if negotiation is required, an additional security layer is added between the protocol and the connection for security purposes. More information about SASL is available in the RFC 2222, available at http://www.ietf.org/rfc/rfc2222.txt. For a more detailed description of Cyrus SASL, refer to http://www.sendmail.org/~ca/email/cyrus/sysadmin.html. Qpid is a messaging tool developed by Apache, which understands Advanced Message Queueing Protocol (AMQP) and has support for various languages and platforms. AMQP is an open source protocol designed for high-performance messaging over a network in a reliable fashion. More information about AMQP is available at http://www.amqp.org/specification/1.0/amqp-org-download. Here, you can find more information about the protocol specifications as well as about the project in general. Qpid projects push the development of AMQP ecosystems and this is done by offering message brokers and APIs that can be used in any developer application that intends to use AMQP messaging part of their product. To do this, the following can be done: Letting the source code open source. Making AMQP available for a large variety of computing environments and programming languages. Offering the necessary tools to simplify the development process of an application. Creating a messaging infrastructure to make sure that other services can integrate well with the AMQP network. Creating a messaging product that makes integration with AMQP trivial for any programming language or computing environment. Make sure that you take a look at Qpid Proton at http://qpid.apache.org/proton/overview.html for this. More information about the the preceding functionalities can be found at http://qpid.apache.org/components/index.html#messaging-apis. RabbitMQ is another message broker software component that implements AMQP, which is also available as open source. It has a number of components, as follows: The RabbitMQ exchange server Gateways for HTTP, Streaming Text Oriented Message Protocol (STOMP) and Message Queue Telemetry Transport (MQTT) AMQP client libraries for a variety of programming languages, most notably Java, Erlang, and .Net Framework A plugin platform for a number of custom components that also offer a collection of predefined one: Shovel: It is a plugin that executes the copy/move operation for messages between brokers Management: It enables the control and monitoring of brokers and clusters of brokers Federation: It enables sharing at the exchange level of messages between brokers You can find out more information regarding RabbitMQ by referring to the RabbitMQ documentation article at http://www.rabbitmq.com/documentation.html. Comparing the two, Qpid and RabbitMQ, it can be concluded that RabbitMQ is better and also that it has a fantastic documentation. This makes it the first choice for the OpenStack Foundation as well as for readers interested in benchmarking information for more than these frameworks. It is also available at http://blog.x-aeon.com/2013/04/10/a-quick-message-queue-benchmark-activemq-rabbitmq-hornetq-qpid-apollo/. One such result is also available in this image for comparison purposes: The next element is puppet, an open source configuration management system that allows IT infrastructure to have certain states defined and also enforce these states. By doing this, it offers a great automation system for system administrators. This project is developed by the Puppet Labs and was released under GNU General Public License until version 2.7.0. After this, it moved to the Apache License 2.0 and is now available in two flavors: The open source puppet version: It is mostly similar to the preceding tool and is capable of configuration management solutions that permit for definition and automation of states. It is available for both Linux and UNIX as well as Max OS X and Windows. The puppet enterprise edition: It is a commercial version that goes beyond the capabilities of the open source puppet and permits the automation of the configuration and management process. It is a tool that defines a declarative language for later use for system configuration. It can be applied directly on the system or even compiled as a catalogue and deployed on a target using a client-server paradigm, which is usually the REST API. Another component is an agent that enforces the resources available in the manifest. The resource abstraction is, of course, done through an abstraction layer that defines the configuration through higher lever terms that are very different from the operating system-specific commands. If you visit http://docs.puppetlabs.com/, you will find more documentation related to Puppet and other Puppet Lab tools. With all this in place, I believe it is time to present the main component of the meta-cloud-services layer, called OpenStack. It is a cloud operating system that is based on controlling a large number of components and together it offers pools of compute, storage, and networking resources. All of them are managed through a dashboard that is, of course, offered by another component and offers administrators control. It offers users the possibility of providing resources from the same web interface. Here is an image depicting the Open Source Cloud operating System, which is actually OpenStack: It is primarily used as an IaaS solution, its components are maintained by the OpenStack Foundation, and is available under Apache License version 2. In the Foundation, today, there are more than 200 companies that contribute to the source code and general development and maintenance of the software. At the heart of it, all are staying its components Also, each component has a Python module used for simple interaction and automation possibilities: Compute (Nova): It is used for the hosting and management of cloud computing systems. It manages the life cycles of the compute instances of an environment. It is responsible for the spawning, decommissioning, and scheduling of various virtual machines on demand. With regard to hypervisors, KVM is the preferred option but other options such as Xen and VMware are also viable. Object Storage (Swift): It is used for storage and data structure retrieval via RESTful and the HTTP API. It is a scalable and fault-tolerant system that permits data replication with objects and files available on multiple disk drives. It is developed mainly by an object storage software company called SwiftStack. Block Storage (Cinder): It provides persistent block storage for OpenStack instances. It manages the creation and attach and detach actions for block devices. In a cloud, a user manages its own devices, so a vast majority of storage platforms and scenarios should be supported. For this purpose, it offers a pluggable architecture that facilitates the process. Networking (Neutron): It is the component responsible for network-related services, also known as Network Connectivity as a Service. It provides an API for network management and also makes sure that certain limitations are prevented. It also has an architecture based on pluggable modules to ensure that as many networking vendors and technologies as possible are supported. Dashboard (Horizon): It provides web-based administrators and user graphical interfaces for interaction with the other resources made available by all the other components. It is also designed keeping extensibility in mind because it is able to interact with other components responsible for monitoring and billing as well as with additional management tools. It also offers the possibility of rebranding according to the needs of commercial vendors. Identity Service (Keystone): It is an authentication and authorization service It offers support for multiple forms of authentication and also existing backend directory services such as LDAP. It provides a catalogue for users and the resources they can access. Image Service (Glance): It is used for the discovery, storage, registration, and retrieval of images of virtual machines. A number of already stored images can be used as templates. OpenStack also provides an operating system image for testing purposes. Glance is the only module capable of adding, deleting, duplicating, and sharing OpenStack images between various servers and virtual machines. All the other modules interact with the images using the available APIs of Glance. Telemetry (Ceilometer): It is a module that provides billing, benchmarking, and statistical results across all current and future components of OpenStack with the help of numerous counters that permit extensibility. This makes it a very scalable module. Orchestrator (Heat): It is a service that manages multiple composite cloud applications with the help of various template formats, such as Heat Orchestration Templates (HOT) or AWS CloudFormation. The communication is done both on a CloudFormation compatible Query API and an Open Stack REST API. Database (Trove): It provides Cloud Database as service functionalities that are both reliable and scalable. It uses relational and nonrelational database engines. Bare Metal Provisioning (Ironic): It is a components that provides virtual machine support instead of bare metal machines support. It started as a fork of the Nova Baremetal driver and grew to become the best solution for a bare-metal hypervisor. It also offers a set of plugins for interaction with various bare-metal hypervisors. It is used by default with PXE and IPMI, but of course, with the help of the available plugins it can offer extended support for various vendor-specific functionalities. Multiple Tenant Cloud Messaging (Zaqar): It is, as the name suggests, a multitenant cloud messaging service for the web developers who are interested in Software as a Service (SaaS). It can be used by them to send messages between various components by using a number of communication patterns. However, it can also be used with other components for surfacing events to end users as well as communication in the over-cloud layer. Its former name was Marconi and it also provides the possibility of scalable and secure messaging. Elastic Map Reduce (Sahara): It is a module that tries to automate the method of providing the functionalities of Hadoop clusters. It only requires the defines for various fields, such as Hadoop versions, various topology nodes, hardware details, and so on. After this, in a few minutes, a Hadoop cluster is deployed and ready for interaction. It also offers the possibility of various configurations after deployment. Having mentioned all this, maybe you would not mind if a conceptual architecture is presented in the following image to present to you with ways in which the above preceding components are interacted with. To automate the deployment of such an environment in a production environment, automation tools, such as the previously mentioned Puppet tool, can be used. Take a look at this diagram: Now, let's move on and see how such a system can be deployed using the functionalities of the Yocto Project. For this activity to start, all the required metadata layers should be put together. Besides the already available Poky repository, other ones are also required and they are defined in the layer index on OpenEmbedded's website because this time, the README file is incomplete: git clone –b dizzy git://git.openembedded.org/meta-openembedded git clone –b dizzy git://git.yoctoproject.org/meta-virtualization git clone –b icehouse git://git.yoctoproject.org/meta-cloud-services source oe-init-build-env ../build-controller After the appropriate controller build is created, it needs to be configured. Inside the conf/layer.conf file, add the corresponding machine configuration, such as qemux86-64, and inside the conf/bblayers.conf file, the BBLAYERS variable should be defined accordingly. There are extra metadata layers, besides the ones that are already available. The ones that should be defined in this variable are: meta-cloud-services meta-cloud-services/meta-openstack-controller-deploy meta-cloud-services/meta-openstack meta-cloud-services/meta-openstack-qemu meta-openembedded/meta-oe meta-openembedded/meta-networking meta-openembedded/meta-python meta-openembedded/meta-filesystem meta-openembedded/meta-webserver meta-openembedded/meta-ruby After the configuration is done using the bitbake openstack-image-controller command, the controller image is built. The controller can be started using the runqemu qemux86-64 openstack-image-controller kvm nographic qemuparams="-m 4096" command. After finishing this activity, the deployment of the compute can be started in this way: source oe-init-build-env ../build-compute With the new build directory created and also since most of the work of the build process has already been done with the controller, build directories such as downloads and sstate-cache, can be shared between them. This information should be indicated through DL_DIR and SSTATE_DIR. The difference between the two conf/bblayers.conf files is that the second one for the build-compute build directory replaces meta-cloud-services/meta-openstack-controller-deploy with meta-cloud-services/meta-openstack-compute-deploy. This time the build is done with bitbake openstack-image-compute and should be finished faster. Having completed the build, the compute node can also be booted using the runqemu qemux86-64 openstack-image-compute kvm nographic qemuparams="-m 4096 –smp 4" command. This step implies the image loading for OpenStack Cirros as follows: wget download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-disk.img scp cirros-0.3.2-x86_64-disk.img root@<compute_ip_address>:~ ssh root@<compute_ip_address> ./etc/nova/openrc glance image-create –name "TestImage" –is=public true –container-format bare –disk-format qcow2 –file /home/root/cirros-0.3.2-x86_64-disk.img Having done all of this, the user is free to access the Horizon web browser using http://<compute_ip_address>:8080/ The login information is admin and the password is password. Here, you can play and create new instances, interact with them, and, in general, do whatever crosses your mind. Do not worry if you've done something wrong to an instance; you can delete it and start again. The last element from the meta-cloud-services layer is the Tempest integration test suite for OpenStack. It is represented through a set of tests that are executed on the OpenStack trunk to make sure everything is working as it should. It is very useful for any OpenStack deployments. More information about Tempest is available at https://github.com/openstack/tempest. Summary In this article, you were not only presented with information about a number of virtualization concepts, such as NFV, SDN, VNF, and so on, but also a number of open source components that contribute to everyday virtualization solutions. I offered you examples and even a small exercise to make sure that the information remains with you even after reading this book. I hope I made some of you curious about certain things. I also hope that some of you documented on projects that were not presented here, such as the OpenDaylight (ODL) initiative, that has only been mentioned in an image as an implementation suggestion. If this is the case, I can say I fulfilled my goal. Resources for Article: Further resources on this subject: Veil-Evasion [article] Baking Bits with Yocto Project [article] An Introduction to the Terminal [article]

In this article by Tiago Antao, author of Bioinformatics with Python Cookbook, you will process next-generation sequencing datasets using Python. If you work in life sciences, you are probably aware of the increasing importance of computational methods to analyze increasingly larger datasets. There is a massive need for bioinformaticians to process this data, and one the main tools is, of course, Python. Python is probably the fastest growing language in the field of data sciences. It includes a rich ecology of software libraries to perform complex data analysis. Another major point in Python is its great community, which is always ready to help and produce great documentation and high-quality reliable software. In this article, we will use Python to process next-generation sequencing datasets. This is one of the many examples of Python usability in bioinformatics; chances are that if you have a biological dataset to analyze, Python can help you. This is surely the case with population genetics, genomics, phylogenetics, proteomics, and many other fields. Next-generation Sequencing (NGS) is one of the fundamental technological developments of the decade in the field of life sciences. Whole Genome Sequencing (WGS), RAD-Seq, RNA-Seq, Chip-Seq, and several other technologies are routinely used to investigate important biological problems. These are also called high-throughput sequencing technologies with good reason; they generate vast amounts of data that need to be processed. NGS is the main reason why computational biology is becoming a "big data" discipline. More than anything else, this is a field that requires strong bioinformatics techniques. There is very strong demand for professionals with these skillsets. Here, we will not discuss each individual NGS technique per se (this will be a massive undertaking). We will use two existing WGS datasets: the Human 1000 genomes project (http://www.1000genomes.org/) and the Anopheles 1000 genomes dataset (http://www.malariagen.net/projects/vector/ag1000g). The code presented will be easily applicable for other genomic sequencing approaches; some of them can also be used for transcriptomic analysis (for example, RNA-Seq). Most of the code is also species-independent, that is, you will be able to apply them to any species in which you have sequenced data. As this is not an introductory text, you are expected to at least know what FASTA, FASTQ, BAM, and VCF files are. We will also make use of basic genomic terminology without introducing it (things such as exomes, nonsynonym mutations, and so on). You are required to be familiar with basic Python, and we will leverage that knowledge to introduce the fundamental libraries in Python to perform NGS analysis. Here, we will concentrate on analyzing VCF files. Preparing the environment You will need Python 2.7 or 3.4. You can use many of the available distributions, including the standard one at http://www.python.org, but we recommend Anaconda Python from http://continuum.io/downloads. We also recommend the IPython Notebook (Project Jupyter) from http://ipython.org/. If you use Anaconda, this and many other packages are available with a simple conda install. There are some amazing libraries to perform data analysis in Python; here, we will use NumPy (http://www.numpy.org/) and matplotlib (http://matplotlib.org/), which you may already be using in your projects. We will also make use of the less widely used seaborn library (http://stanford.edu/~mwaskom/software/seaborn/). For bioinformatics, we will use Biopython (http://biopython.org) and PyVCF (https://pyvcf.readthedocs.org). The code used here is available on GitHub at https://github.com/tiagoantao/bioinf-python. In your realistic pipeline, you will probably be using other tools, such as bwa, samtools, or GATK to perform your alignment and SNP calling. In our case, tabix and bgzip (http://www.htslib.org/) is needed. Analyzing variant calls After running a genotype caller (for example, GATK or samtools mpileup), you will have a Variant Call Format (VCF) file reporting on genomic variations, such as SNPs (Single-Nucleotide Polymorphisms), InDels (Insertions/Deletions), CNVs (Copy Number Variation) among others. In this recipe, we will discuss VCF processing with the PyVCF module over the human 1000 genomes project to analyze SNP data. Getting ready I am to believe that 2 to 20 GB of data for a tutorial is asking too much. Although, the 1000 genomes' VCF files with realistic annotations are in that order of magnitude, we will want to work with much less data here. Fortunately, the bioinformatics community has developed tools that allow partial download of data. As part of the samtools/htslib package (http://www.htslib.org/), you can download tabix and bgzip, which will take care of data management. For example: tabix -fh ftp://ftp- trace.ncbi.nih.gov/1000genomes/ftp/release/20130502/supporting/vcf_ with_sample_level_annotation/ALL.chr22.phase3_shapeit2_mvncall_ integrated_v5_extra_anno.20130502.genotypes.vcf.gz 22:1-17000000 |bgzip -c > genotypes.vcf.gz tabix -p vcf genotypes.vcf.gz The first line will perform a partial download of the VCF file for chromosome 22 (up to 17 Mbp) of the 1000 genomes project. Then, bgzip will compress it. The second line will create an index, which we will need for direct access to a section of the genome. The preceding code is available at https://github.com/tiagoantao/bioinf-python/blob/master/notebooks/01_NGS/Working_with_VCF.ipynb. How to do it… Take a look at the following steps: Let's start inspecting the information that we can get per record, as shown in the following code: import vcf v = vcf.Reader(filename='genotypes.vcf.gz')   print('Variant Level information') infos = v.infos for info in infos:    print(info)   print('Sample Level information') fmts = v.formats for fmt in fmts:    print(fmt)     We start by inspecting the annotations that are available for each record (remember that each record encodes variants, such as SNP, CNV, and InDel, and the state of that variant per sample). At the variant (record) level, we will find AC: the total number of ALT alleles in called genotypes, AF: the estimated allele frequency, NS: the number of samples with data, AN: the total number of alleles in called genotypes, and DP: the total read depth. There are others, but they are mostly specific to the 1000 genomes project (here, we are trying to be as general as possible). Your own dataset might have much more annotations or none of these.     At the sample level, there are only two annotations in this file: GT: genotype and DP: the per sample read depth. Yes, you have the per variant (total) read depth and the per sample read depth; be sure not to confuse both. Now that we know which information is available, let's inspect a single VCF record with the following code: v = vcf.Reader(filename='genotypes.vcf.gz') rec = next(v) print(rec.CHROM, rec.POS, rec.ID, rec.REF, rec.ALT, rec.QUAL, rec.FILTER) print(rec.INFO) print(rec.FORMAT) samples = rec.samples print(len(samples)) sample = samples[0] print(sample.called, sample.gt_alleles, sample.is_het, sample.phased) print(int(sample['DP']))     We start by retrieving standard information: the chromosome, position, identifier, reference base (typically, just one), alternative bases (can have more than one, but it is not uncommon as the first filtering approach to only accept a single ALT, for example, only accept bi-allelic SNPs), quality (PHRED scaled—as you may expect), and the FILTER status. Regarding the filter, remember that whatever the VCF file says, you may still want to apply extra filters (as in the next recipe).     Then, we will print the additional variant-level information (AC, AS, AF, AN, DP, and so on), followed by the sample format (in this case, DP and GT). Finally, we will count the number of samples and inspect a single sample checking if it was called for this variant. If available, the reported alleles, heterozygosity, and phasing status (this dataset happens to be phased, which is not that common). Let's check the type of variant and the number of nonbiallelic SNPs in a single pass with the following code: from collections import defaultdict f = vcf.Reader(filename='genotypes.vcf.gz')   my_type = defaultdict(int) num_alts = defaultdict(int)   for rec in f:    my_type[rec.var_type, rec.var_subtype] += 1    if rec.is_snp:        num_alts[len(rec.ALT)] += 1 print(num_alts) print(my_type)     We use the Python defaultdict collection type. We find that this dataset has InDels (both insertions and deletions), CNVs, and, of course, SNPs (roughly two-third being transitions with one-third transversions). There is a residual number (79) of triallelic SNPs. There's more… The purpose of this recipe is to get you up to speed on the PyVCF module. At this stage, you should be comfortable with the API. We do not delve much here on usage details because that will be the main purpose of the next recipe: using the VCF module to study the quality of your variant calls. It will probably not be a shocking revelation that PyVCF is not the fastest module on earth. This file format (highly text-based) makes processing a time-consuming task. There are two main strategies of dealing with this problem: parallel processing or converting to a more efficient format. Note that VCF developers will perform a binary (BCF) version to deal with part of these problems at http://www.1000genomes.org/wiki/analysis/variant-call-format/bcf-binary-vcf-version-2. See also The specification for VCF is available at http://samtools.github.io/hts-specs/VCFv4.2.pdf GATK is one of the most widely used variant callers; check https://www.broadinstitute.org/gatk/ samtools and htslib are both used for variant calling and SAM/BAM management; check http://htslib.org Studying genome accessibility and filtering SNP data If you are using NGS data, the quality of your VCF calls may need to be assessed and filtered. Here, we will put in place a framework to filter SNP data. More than giving filtering rules (an impossible task to be performed in a general way), we give you procedures to assess the quality of your data. With this, you can then devise your own filters. Getting ready In the best-case scenario, you have a VCF file with proper filters applied; if this is the case, you can just go ahead and use your file. Note that all VCF files will have a FILTER column, but this does not mean that all the proper filters were applied. You have to be sure that your data is properly filtered. In the second case, which is one of the most common, your file will have unfiltered data, but you have enough annotations. Also, you can apply hard filters (that is, no need for programmatic filtering). If you have a GATK annotated file, refer, for instance, to http://gatkforums.broadinstitute.org/discussion/2806/howto-apply-hard-filters-to-a-call-set. In the third case, you have a VCF file that has all the annotations that you need, but you may want to apply more flexible filters (for example, "if read depth > 20, then accept; if mapping quality > 30, accept if mapping quality > 40"). In the fourth case, your VCF file does not have all the necessary annotations, and you have to revisit your BAM files (or even other sources of information). In this case, the best solution is to find whatever extra information you have and create a new VCF file with the needed annotations. Some genotype callers like GATK allow you to specify with annotations you want; you may also want to use extra programs to provide more annotations, for example, SnpEff (http://snpeff.sourceforge.net/) will annotate your SNPs with predictions of their effect (for example, if they are in exons, are they coding on noncoding?). It is impossible to provide a clear-cut recipe; it will vary with the type of your sequencing data, your species of study, and your tolerance to errors, among other variables. What we can do is provide a set of typical analysis that is done for high-quality filtering. In this recipe, we will not use data from the Human 1000 genomes project; we want "dirty" unfiltered data that has a lot of common annotations that can be used to filter it. We will use data from the Anopheles 1000 genomes project (Anopheles is the mosquito vector involved in the transmission of the parasite causing malaria), which makes available filtered and unfiltered data. You can find more information about this project at http://www.malariagen.net/projects/vector/ag1000g. We will get a part of the centromere of chromosome 3L for around 100 mosquitoes, followed by a part somewhere in the middle of that chromosome (and index both), as shown in the following code: tabix -fh ftp://ngs.sanger.ac.uk/production/ag1000g/phase1/preview/ag1000g.AC. phase1.AR1.vcf.gz 3L:1-200000 |bgzip -c > centro.vcf.gz tabix -fh ftp://ngs.sanger.ac.uk/production/ag1000g/phase1/preview/ag1000g.AC. phase1.AR1.vcf.gz 3L:21000001-21200000 |bgzip -c > standard.vcf.gz tabix -p vcf centro.vcf.gz tabix -p vcf standard.vcf.gz As usual, the code to download this data is available at the https://github.com/tiagoantao/bioinf-python/blob/master/notebooks/01_NGS/Filtering_SNPs.ipynb notebook. Finally, a word of warning about this recipe: the level of Python here will be slightly more complicated than before. The more general code that we will write may be easier to reuse in your specific case. We will perform extensive use of functional programming techniques (lambda functions) and the partial function application. How to do it… Take a look at the following steps: Let's start by plotting the distribution of variants across the genome in both files as follows: %matplotlib inline from collections import defaultdict   import seaborn as sns import matplotlib.pyplot as plt   import vcf   def do_window(recs, size, fun):    start = None    win_res = []    for rec in recs:        if not rec.is_snp or len(rec.ALT) > 1:            continue        if start is None:            start = rec.POS        my_win = 1 + (rec.POS - start) // size        while len(win_res) < my_win:            win_res.append([])        win_res[my_win - 1].extend(fun(rec))    return win_res   wins = {} size = 2000 vcf_names = ['centro.vcf.gz', 'standard.vcf.gz'] for vcf_name in vcf_names:    recs = vcf.Reader(filename=vcf_name)    wins[name] = do_window(recs, size, lambda x: [1])     We start by performing the required imports (as usual, remember to remove the first line if you are not on the IPython Notebook). Before I explain the function, note what we will do.     For both files, we will compute windowed statistics: we will divide our file that includes 200,000 bp of data in windows of size 2,000 (100 windows). Every time we find a bi-allelic SNP, we will add one to the list related to that window in the window function. The window function will take a VCF record (an SNP—rec.is_snp—that is not bi-allelic—len(rec.ALT) == 1), determine the window where that record belongs (by performing an integer division of rec.POS by size), and extend the list of results of that window by the function that is passed to it as the fun parameter (which in our case is just one).     So, now we have a list of 100 elements (each representing 2,000 base pairs). Each element will be another list, which will have 1 for each bi-allelic SNP found. So, if you have 200 SNPs in the first 2,000 base pairs, the first element of the list will have 200 ones. Let's continue: def apply_win_funs(wins, funs):    fun_results = []    for win in wins:        my_funs = {}        for name, fun in funs.items():            try:                my_funs[name] = fun(win)            except:                my_funs[name] = None        fun_results.append(my_funs)    return fun_results   stats = {} fig, ax = plt.subplots(figsize=(16, 9)) for name, nwins in wins.items():    stats[name] = apply_win_funs(nwins, {'sum': sum})    x_lim = [i * size for i in range(len(stats[name]))]    ax.plot(x_lim, [x['sum'] for x in stats[name]], label=name) ax.legend() ax.set_xlabel('Genomic location in the downloaded segment') ax.set_ylabel('Number of variant sites (bi-allelic SNPs)') fig.suptitle('Distribution of MQ0 along the genome', fontsize='xx-large')     Here, we will perform a plot that contains statistical information for each of our 100 windows. The apply_win_funs will calculate a set of statistics for every window. In this case, it will sum all the numbers in the window. Remember that every time we find an SNP, we will add one to the window list. This means that if we have 200 SNPs, we will have 200 1s; hence; summing them will return 200.     So, we are able to compute the number of SNPs per window in an apparently convoluted way. Why we are doing things with this strategy will become apparent soon, but for now, let's check the result of this computation for both files (refer to the following figure): Figure 1: The number of bi-allelic SNPs distributed of windows of 2, 000 bp of size for an area of 200 Kbp near the centromere (blue) and in the middle of chromosome (green). Both areas come from chromosome 3L for circa 100 Ugandan mosquitoes from the Anopheles 1000 genomes project     Note that the amount of SNPs in the centromere is smaller than the one in the middle of the chromosome. This is expected because calling variants in chromosomes is more difficult than calling variants in the middle and also because probably there is less genomic diversity in centromeres. If you are used to humans or other mammals, you may find the density of variants obnoxiously high, that is, mosquitoes for you! Let's take a look at the sample-level annotation. We will inspect Mapping Quality Zero (refer to https://www.broadinstitute.org/gatk/guide/tooldocs/org_broadinstitute_gatk_tools_walkers_annotator_MappingQualityZeroBySample.php for details), which is a measure of how well all the sequences involved in calling this variant map clearly to this position. Note that there is also an MQ0 annotation at the variant-level: import functools   import numpy as np mq0_wins = {} vcf_names = ['centro.vcf.gz', 'standard.vcf.gz'] size = 5000 def get_sample(rec, annot, my_type):    res = []    samples = rec.samples    for sample in samples:        if sample[annot] is None: # ignoring nones            continue        res.append(my_type(sample[annot]))    return res   for vcf_name in vcf_names:    recs = vcf.Reader(filename=vcf_name)    mq0_wins[vcf_name] = do_window(recs, size, functools.partial(get_sample, annot='MQ0', my_type=int))     Start by inspecting this by looking at the last for; we will perform a windowed analysis by getting the MQ0 annotation from each record. We perform this by calling the get_sample function in which we return our preferred annotation (in this case, MQ0) cast with a certain type (my_type=int). We will use the partial application function here. Python allows you to specify some parameters of a function and wait for other parameters to be specified later. Note that the most complicated thing here is the functional programming style. Also, note that it makes it very easy to compute other sample-level annotations; just replace MQ0 with AB, AD, GQ, and so on. You will immediately have a computation for that annotation. If the annotation is not of type integer, no problem; just adapt my_type. This is a difficult programming style if you are not used to it, but you will reap the benefits very soon. Let's now print the median and top 75 percent percentile for each window (in this case, with a size of 5,000) as follows: stats = {} colors = ['b', 'g'] i = 0 fig, ax = plt.subplots(figsize=(16, 9)) for name, nwins in mq0_wins.items():    stats[name] = apply_win_funs(nwins, {'median': np.median, '75': functools.partial(np.percentile, q=75)})    x_lim = [j * size for j in range(len(stats[name]))]    ax.plot(x_lim, [x['median'] for x in stats[name]], label=name, color=colors[i])    ax.plot(x_lim, [x['75'] for x in stats[name]], '--', color=colors[i])    i += 1 ax.legend() ax.set_xlabel('Genomic location in the downloaded segment') ax.set_ylabel('MQ0') fig.suptitle('Distribution of MQ0 along the genome', fontsize='xx-large')     Note that we now have two different statistics on apply_win_funs: percentile and median. Again, we will pass function names as parameters (np.median) and perform the partial function application (np.percentile). The result can be seen in the following figure: Figure 2: Median (continuous line) and 75th percentile (dashed) of MQ0 of sample SNPs distributed on windows of 5,000 bp of size for an area of 200 Kbp near the centromere (blue) and in the middle of chromosome (green); both areas come from chromosome 3L for circa 100 Ugandan mosquitoes from the Anopheles 1000 genomes project     For the "standard" file, the median MQ0 is 0 (it is plotted at the very bottom, which is almost unseen); this is good as it suggests that most sequences involved in the calling of variants map clearly to this area of the genome. For the centromere, MQ0 is of poor quality. Furthermore, there are areas where the genotype caller could not find any variants at all; hence, the incomplete chart. Let's compare heterozygosity with the DP sample-level annotation:     Here, we will plot the fraction of heterozygosity calls as a function of the sample read depth (DP) for every SNP. We will first explain the result and only then the code that generates it.     The following screenshot shows the fraction of calls that are heterozygous at a certain depth: Figure 3: The continuous line represents the fraction of heterozygosite calls computed at a certain depth; in blue is the centromeric area, in green is the "standard" area; the dashed lines represent the number of sample calls per depth; both areas come from chromosome 3L for circa 100 Ugandan mosquitoes from the Anopheles 1000 genomes project In the preceding screenshot, there are two considerations to be taken into account:     At a very low depth, the fraction of heterozygote calls is biased low; this makes sense because the number of reads per position does not allow you to make a correct estimate of the presence of both alleles in a sample. So, you should not trust calls at a very low depth.     As expected, the number of calls in the centromere is way lower than calls outside it. The distribution of SNPs outside the centromere follows a common pattern that you can expect in many datasets. Here is the code: def get_sample_relation(recs, f1, f2):    rel = defaultdict(int)    for rec in recs:        if not rec.is_snp:              continue        for sample in rec.samples:            try:                 v1 = f1(sample)                v2 = f2(sample)                if v1 is None or v2 is None:                    continue # We ignore Nones                rel[(v1, v2)] += 1            except:                pass # This is outside the domain (typically None)    return rel   rels = {} for vcf_name in vcf_names:    recs = vcf.Reader(filename=vcf_name)    rels[vcf_name] = get_sample_relation(recs, lambda s: 1 if s.is_het else 0, lambda s: int(s['DP'])) Let's start by looking at the for loop. Again, we will use functional programming: the get_sample_relation function will traverse all the SNP records and apply the two functional parameters; the first determines heterozygosity, whereas the second gets the sample DP (remember that there is also a variant DP).     Now, as the code is complex as it is, I opted for a naive data structure to be returned by get_sample_relation: a dictionary where the key is the pair of results (in this case, heterozygosity and DP) and the sum of SNPs, which share both values. There are more elegant data structures with different trade-offs for this: scipy spare matrices, pandas' DataFrames, or maybe, you want to consider PyTables. The fundamental point here is to have a framework that is general enough to compute relationships among a couple of sample annotations.     Also, be careful with the dimension space of several annotations, for example, if your annotation is of float type, you might have to round it (if not, the size of your data structure might become too big). Now, let's take a look at all the plotting codes. Let's perform it in two parts; here is part 1: def plot_hz_rel(dps, ax, ax2, name, rel):    frac_hz = []    cnt_dp = []    for dp in dps:        hz = 0.0        cnt = 0          for khz, kdp in rel.keys():             if kdp != dp:                continue            cnt += rel[(khz, dp)]            if khz == 1:                hz += rel[(khz, dp)]        frac_hz.append(hz / cnt)        cnt_dp.append(cnt)    ax.plot(dps, frac_hz, label=name)    ax2.plot(dps, cnt_dp, '--', label=name)     This function will take a data structure (as generated by get_sample_relation) expecting that the first parameter of the key tuple is the heterozygosity state (0 = homozygote, 1 = heterozygote) and the second is the DP. With this, it will generate two lines: one with the fraction of samples (which are heterozygotes at a certain depth) and the other with the SNP count. Let's now call this function, as shown in the following code: fig, ax = plt.subplots(figsize=(16, 9)) ax2 = ax.twinx() for name, rel in rels.items():    dps = list(set([x[1] for x in rel.keys()]))    dps.sort()    plot_hz_rel(dps, ax, ax2, name, rel) ax.set_xlim(0, 75) ax.set_ylim(0, 0.2) ax2.set_ylabel('Quantity of calls') ax.set_ylabel('Fraction of Heterozygote calls') ax.set_xlabel('Sample Read Depth (DP)') ax.legend() fig.suptitle('Number of calls per depth and fraction of calls which are Hz',,              fontsize='xx-large')     Here, we will use two axes. On the left-hand side, we will have the fraction of heterozygozite SNPs, whereas on the right-hand side, we will have the number of SNPs. Then, we will call our plot_hz_rel for both data files. The rest is standard matplotlib code. Finally, let's compare variant DP with the categorical variant-level annotation: EFF. EFF is provided by SnpEFF and tells us (among many other things) the type of SNP (for example, intergenic, intronic, coding synonymous, and coding nonsynonymous). The Anopheles dataset provides this useful annotation. Let's start by extracting variant-level annotations and the functional programming style, as shown in the following code: def get_variant_relation(recs, f1, f2):    rel = defaultdict(int)    for rec in recs:        if not rec.is_snp:              continue        try:            v1 = f1(rec)            v2 = f2(rec)            if v1 is None or v2 is None:                continue # We ignore Nones            rel[(v1, v2)] += 1        except:            pass    return rel     The programming style here is similar to get_sample_relation, but we do not delve into the samples. Now, we will define the types of effects that we will work with and convert the effect to an integer as it would allow you to use it as in index, for example, matrices. Think about coding a categorical variable: accepted_eff = ['INTERGENIC', 'INTRON', 'NON_SYNONYMOUS_CODING', 'SYNONYMOUS_CODING']   def eff_to_int(rec):    try:        for annot in rec.INFO['EFF']:            #We use the first annotation            master_type = annot.split('(')[0]            return accepted_eff.index(master_type)    except ValueError:        return len(accepted_eff) We will now traverse the file; the style should be clear to you now: eff_mq0s = {} for vcf_name in vcf_names:    recs = vcf.Reader(filename=vcf_name)    eff_mq0s[vcf_name] = get_variant_relation(recs, lambda r: eff_to_int(r), lambda r: int(r.INFO['DP'])) Finally, we will plot the distribution of DP using the SNP effect, as shown in the following code: fig, ax = plt.subplots(figsize=(16,9)) vcf_name = 'standard.vcf.gz' bp_vals = [[] for x in range(len(accepted_eff) + 1)] for k, cnt in eff_mq0s[vcf_name].items():    my_eff, mq0 = k    bp_vals[my_eff].extend([mq0] * cnt) sns.boxplot(bp_vals, sym='', ax=ax) ax.set_xticklabels(accepted_eff + ['OTHER']) ax.set_ylabel('DP (variant)') fig.suptitle('Distribution of variant DP per SNP type',              fontsize='xx-large') Here, we will just print a box plot for the noncentromeric file (refer to the following screenshot). The results are as expected: SNPs in code areas will probably have more depth if they are in more complex regions (that is easier to call) than intergenic SNPs: Figure 4: Boxplot for the distribution of variant read depth across different SNP effects There's more… The approach would depend on the type of sequencing data that you have, the number of samples, and potential extra information (for example, pedigree among samples). This recipe is very complex as it is, but parts of it are profoundly naive (there is a limit of complexity that I could force on you on a simple recipe). For example, the window code does not support overlapping windows; also, data structures are simplistic. However, I hope that they give you an idea of the general strategy to process genomic high-throughput sequencing data. See also There are many filtering rules, but I would like to draw your attention to the need of reasonably good coverage (clearly more than 10 x), for example, refer to. Meynet et al "Variant detection sensitivity and biases in whole genome and exome sequencing" at http://www.biomedcentral.com/1471-2105/15/247/ Brad Chapman is one of the best known specialist in sequencing analysis and data quality with Python and the main author of Blue Collar Bioinformatics, a blog that you may want to check at https://bcbio.wordpress.com/ Brad is also the main author of bcbio-nextgen, a Python-based pipeline for high-throughput sequencing analysis. Refer to https://bcbio-nextgen.readthedocs.org Peter Cock is the main author of Biopython and is heavily involved in NGS analysis; be sure to check his blog, "Blasted Bionformatics!?" at http://blastedbio.blogspot.co.uk/ Summary In this article, we prepared the environment, analyzed variant calls and learned about genome accessibility and filtering SNP data.

This article by Matjaz B. Juric, Sven Bernhardt, Hajo Normann, Danilo Schmiedel, Guido Schmutz, Mark Simpson, and Torsten Winterberg, authors of the book Design Principles for Process-driven Architectures Using Oracle BPM and SOA Suite 12c, describes the strategies and a methodology that can help us realize the benefits of BPM as a successful enterprise modernization strategy. In this article, we will do the following: Provide the reader with a set of actions in the course of a complete methodology that they can incorporate in order to create the desired attractiveness towards broader application throughout the enterprise Describe organizational and cultural barriers to applying enterprise BPM and discuss ways to overcome them (For more resources related to this topic, see here.) The postmature birth of enterprise BPM When enterprise architects discuss the future of the software landscape of their organization, they map the functional capabilities, such as customer relationship management and order management, to existing or new applications—some packaged and some custom. Then, they connect these applications by means of middleware. They typically use the notion of an integration middleware, such as an enterprise service bus (ESB), to depict the technical integration between these applications, exposing functionality as services, APIs, or, more trendy, "micro services". These services are used by modern, more and more mobile frontends and B2B partners. For several years now, it has been hard to find a PowerPoint slide that discusses future enterprise middleware without the notion of a BPM layer that sits on top of the frontend and the SOA service layer. So, in most organizations, we find a slide deck that contains this visual box named BPM, signifying the aim to improve process excellence by automating business processes along the management discipline known as business process management (BPM). Over the years, we have seen that the frontend layer often does materialize as a portal or through a modern mobile application development platform. The envisioned SOA services can be found living on an ESB or API gateway. Yet, the component called BPM and the related practice of modeling executable processes has failed to finally incubate until now—at least in most organizations. BPM still waits for morphing from an abstract item on a PowerPoint slide and in a shelved analyst report to some automated business processes that are actually deployed to a business-critical production machine. When we look closer—yes—there is a license for a BPM tool, and yes, some processes have even been automated, but those tend to be found rather in the less visible corners of the enterprise, seldom being of the concern for higher management and the hot project teams that work day and night on the next visible release. In short, BPM remains the hobby of some enterprise architects and the expensive consultants they pay. Will BPM ever take the often proposed lead role in the middleware architect's tool box? Will it lead to a better, more efficient, and productive organization? To be very honest, at the moment the answer to that question is rather no than yes. There is a good chance that BPM remains just one more of the silver bullets that fill some books and motivate some great presentations at conferences, yet do not have an impact on the average organization. But there is still hope for enterprise BPM as opposed to a departmental approach to process optimization. There is a good chance that BPM, next to other enabling technologies, will indeed be the driver for successful enterprise modernization. Large organizations all over the globe reengage with smaller and larger system integrators to tackle the process challenge. Maybe BPM as a practice needs more time than other items found in Gardner hype curves to mature before it's widely applied. This necessary level of higher maturity encompasses both the tools and the people using them. Ultimately, this question of large-scale BPM adoption will be answered individually in each organization. Only when a substantive set of enterprises experience tangible benefits from BPM will they talk about it, thus growing a momentum that leads to the success of enterprise BPM as a whole. This positive marketing based on actual project and program success will be the primary way to establish a force of attraction towards BPM that will raise curiosity and interest in the minds of the bulk of the organizations that are still rather hesitant or ignorant about using BPM. Oracle BPM Suite 12c – new business architecture features New tools in Oracle BPM Suite 12c put BPM in the mold of business architecture (BA). This new version contains new BA model types and features that help companies to move out of the IT-based, rather technical view of business processes automation and into strategic process improvement. Thus, these new model types help us to embark on the journey towards enterprise BPM. This is an interesting step in evolution of enterprise middleware—Oracle is the first vendor of a business process automation engine that moved up from concrete automated processes to strategic views on end-to-end processes, thus crossing the automation/strategic barrier. BPM Suite 12c introduces cross-departmental business process views. Thereby, it allows us to approach an enterprise modeling exercise through top-down modeling. It has become an end-to-end value chain model that sits on top of processes. It chains separated business processes together into one coherent end-to-end view. The value chain describes a bird's-eye view of the steps needed to achieve the most critical business goals of an organization. These steps comprise of business processes, of which some are automated in a BPMN engine and others actually run in packaged applications or are not automated at all. Also, BPM Suite 12c allows the capturing of the respective goals and provides the tools to measure them as KPIs and service-level agreements. In order to understand the path towards these yet untackled areas, it is important to understand where we stand today with BPM and what this new field of end-to-end process transparency is all about. Yet, before we get there, we will leave enterprise IT for a moment and take a look at the nature of a game (any game) in order to prepare for a deeper understanding of the mechanisms and cultural impact that underpin the move from a departmental to an enterprise approach to BPM. Football games – same basic rules, different methodology Any game, be it a physical sport, such as football, or a mental sport, such as chess, is defined through a set of common rules. How the game is played will look very different depending on the level of the league it is played in. A Champions League football game is so much more refined than your local team playing at the nearby stadium, not to mention the neighboring kids kicking the ball in the dirt ground. These kids will show creativity and pleasure in the game, yet the level of sophistication is a completely different ball game in the big stadium. You can marvel at the effort made to ensure that everybody plays their role in a well-trained symbiosis with their peers, all sharing a common set of collaboration rules and patterns. The team spent so many hours training the right combinations, establishing a deep trust. There is no time to discuss the meaning of an order shouted out by the trainer. They have worked on this common understanding of how to do things in various situations. They have established one language that they share. As an observer of a great match, you appreciate an elaborate art, not of one artist but of a coherent team. No one would argue the prevalence of this continuum in the refinement and rising sophistication in rough physical sports. It is puzzling to know to which extent we, as players in the games of enterprise IT, often tend to neglect the needs and forces that are implied by such a continuum of rising sophistication. The next sections will take a closer look at these BPM playgrounds and motivate you to take the necessary steps toward team excellence when moving from a small, departmental level BPM/SOA project to a program approach that is centered on a BPM and SOA paradigm. Which BPM game do we play? Game Silo BPM is the workflow or business process automation in organizational departments. It resembles the kids playing soccer on the neighborhood playground. After a few years of experience with automated processes, the maturity rises to resemble your local football team—yes, they play in a stadium, and it is often not elegant. Game Silo BPM is a tactical game in which work gets done while management deals with reaching departmental goals. New feature requests lead to changed or new applications and the people involved know each other very well over many years under established leadership. Workflows are automated to optimize performance. Game Enterprise BPM thrives for process excellence at Champions League. It is a strategic game in which higher management and business departments outline the future capability maps and cross-departmental business process models. In this game, players tend to regard the overall organization as a set of more or less efficient functional capabilities. One or a group of functional capabilities make up a department. Game Silo BPM – departmental workflows Today, most organizations use BPM-based process automation based on tools such as Oracle BPM Suite to improve the efficiency of the processes within departments. These processes often support the functional capability that this particular department owns by increasing its efficiency. Increasing efficiency is to do more with less. It is about automating manual steps, removing bottlenecks, and making sure that the resources are optimally allocated across your process. The driving force is typically the team manager, who is measured by the productivity of his team. The key factor to reach this goal is the automation of redundant or unnecessary human/IT interactions. Through process automation, we gain insights into the performance of the process. This insight can be called process transparency, allowing for the constant identification of areas of improvement. A typical example of the processes found in Game Silo BPM is an approval process that can be expressed as a clear path among process participants. Often, these processes work on documents, thus having a tight relationship with content management. We also find complex backend integration processes that involve human interaction only in the case of an exception. The following figure depicts this siloed approach to process automation. Recognizing a given business unit as a silo indicates its closed and self-contained world. A word of caution is needed: The term "silo" is often used in a negative connotation. It is important, though, to recognize that a closed and coherent team with no dependencies on other teams is a preferred working environment for many employees and managers. In more general terms, it reflects an archaic type of organization that we lean towards. It allows everybody in the team to indulge in what can be called the siege mentality we as humans gravitate to some coziness along the notion of a well-defined and manageable island of order. Figure 1: Workflow automation in departmental silos As discussed in the introduction, there is a chance that BPM never leaves this departmental context in many organizations. If you are curious to find out where you stand today, it is easy to depict whether a business process is stuck in the silo or whether it's part of an enterprise BPM strategy. Just ask whether the process is aligned with corporate goals or whether it's solely associated with departmental goals and KPIs. Another sign is the lack of a methodology and of a link to cross-departmental governance that defines the mode of operation and the means to create consistent models that speak one common language. To impose the enterprise architecture tools of Game Enterprise BPM on Game Silo BPM would be an overhead; you don't need them if your organization does not strive for cross-departmental process transparency. Oracle BPM Suite 11g is made for playing Game Silo BPM Oracle BPM Suite 11g provides all the tools and functionalities necessary to automate a departmental workflow. It is not sufficient to model business processes that span departmental barriers and require top-down process hierarchies. The key components are the following: The BPMN process modeling tool and the respective execution engine The means to organize logical roles and depict them as swimlanes in BPMN process models The human-task component that involves human input in decision making The business rule for supporting automated decision making The technical means to call backend SOA services The wizards to create data mappings The process performance can be measured by means of business activity monitoring (BAM) Oracle BPM Suite models processes in BPMN Workflows are modeled on a pretty fine level of granularity using the standard BPMN 2.0 version (and later versions). BPMN is both business ready and technically detailed enough to allow model processes to be executed in a process engine. Oracle fittingly expresses the mechanism as what you see is what you execute. Those workflows typically orchestrate human interaction through human tasks and functionality through SOA services. In the next sections of this article, we will move our head out of the cocoon of the silo, looking higher and higher along the hierarchy of the enterprise until we reach the world of workshops and polished PowerPoint slides in which the strategy of the organization is defined. Game Enterprise BPM Enterprise BPM is a management discipline with a methodology typically found in business architecture (BA) and the respective enterprise architecture (EA) teams. Representatives of higher management and of business departments and EA teams meet management consultants in order to understand the current situation and the desired state of the overall organization. Therefore, enterprise architects define process maps—a high-level view of the organization's business processes, both for the AS-IS and various TO-BE states. In the next step, they define the desired future state and depict strategies and means to reach it. Business processes that generate value for the organization typically span several departments. The steps in these end-to-end processes can be mapped to the functional building blocks—the departments. Figure 2: Cross-departmental business process needs an owner The goal of Game Enterprise BPM is to manage enterprise business processes, making sure they realize the corporate strategy and meet the respective goals, which are ideally measured by key performance indicators, such as customer satisfaction, reduction of failure, and cost reduction. It is a good practice to reflect on the cross-departmental efforts through the notion of a common or shared language that is spoken across departmental boundaries. A governance board is a great means to reach this capability in the overall organization. Still wide open – the business/IT divide Organizational change in the management structure is a prerequisite for the success of Game Enterprise BPM but is not a sufficient condition. Several business process management books describe the main challenge in enterprise BPM as the still-wide-open business/IT divide. There is still a gap between process understanding and ownership in Game Enterprise BPM and how automated process are modeled and perceived in departmental workflows of Game Silo BPM. Principles, goals, standards, and best practices defined in Game Enterprise BPM do not trickle down into everyday work in Game Silo BPM. One of the biggest reasons for this divide is the fact that there is no direct link between the models and tools used in top management to depict the business strategy, IT strategy, and business architecture and the high-level value chain and between the process models and the models and artifacts used in enterprise architecture and from there, even software architecture. Figure 3: Gap between business architecture and IT enterprise architecture in strategic BPM So, traditionally, organizations use specific business architecture or enterprise architecture tools in order to depict AS-IS and TO-BE high-level reflections of value chains, hierarchical business processes, and capability maps alongside application heat maps. These models kind of hang in the air, they are not deeply grounded in real life. Business process models expressed in event process chains (EPCs), vision process models and other modeling types often don't really reflect the flows and collaboration structures of actual procedures of the office. This leads to the perception of business architecture departments as ivory towers with no, or weak, links to the realities of the organization. On the other side, the tools and models of the IT enterprise architecture and software architecture speak a language not understood by the members of business departments. Unified Modeling Language (UML) is the most prominent set of model types that stuck in IT. However, while the UML class and activity diagrams promised to be valid to depict the nouns and stories of the business processes, their potential to allow a shared language and approach to depict joint vocabulary and views on requirements rarely materialized. Until now, there has been no workflow tool vendor approaching these strategic enterprise-level models, bridging the business/IT gap. Oracle BPM Suite 12c tackles Game Enterprise BPM With BPM Suite 12c, Oracle is starting to engage in this domain. The approach Oracle took can be summarized as applying the Pareto principle: 80 percent of the needed features for strategic enterprise modeling can be found in just 20 percent of the functionality of those high-end, enterprise-level modeling tools. So, Oracle implemented these 20 percent of business architecture models: Enterprise maps to define the organizational and application context Value chains to establish a root for process hierarchies The strategy model to depict focus areas and assign technical capabilities and optimization strategies The following figure represents the new features in Oracle BPM Suite 12c in the context of the Game Enterprise BPM methodology: Figure 4: Oracle BPM Suite 12c new features in the context of the Game Enterprise BPM methodology The preceding figure is based on the BPTrends Process Change Methodology introduced in the Business Process Change book by Paul Harmon. These new features promise to create a link from higher-level process models and other strategic depictions into executable processes. This link could not be established until now since there are too many interface mismatches between enterprise tooling and workflow automation engines. The new model types in Oracle BPM Suite 12c are, as discussed, a subset of all the features and model types in the EA tools. For this subset, these interface mismatches have been made obsolete: there is a clear trace with no tool disruption from the enterprise map to the value chain and associated KPIs down to the BPMN process that are automated. These features have been there in the EA and BPM tools before. What is new is this undisrupted trace. Figure 5: Undisrupted trace from the business architecture to executable processes The preceding figure is based on the BPTrends Process Change Methodology introduced in the Business Process Change book by Paul Harmon. This holistic set of tools that brings together aspects from modeling time, design time, and runtime makes it more likely to succeed in finally bridging the business/IT gap. Figure 6: Tighter links from business and strategy to executable software and processes Today, we do not live in a perfect world. To understand to which extent this gap is closed, it helps to look at how people work. If there is a tool to depict enterprise strategy, end-to-end business processes, business capabilities, and KPIs that are used in daily work and that have the means to navigate to lower-level models, then we have come quite far. The features of Oracle BPM Suite 12c, which are discussed below, are a step in this direction but are not the end of the journey. Using business architect features The process composer is a business user-friendly web application. From the login page, it guides the user in the spirit of a business architecture methodology. All the models that we create are part of a "space". On its entry page, the main structure is divided into business architecture models in BA projects, which are described in this article. It is a good practice to start with an enterprise map that depicts the business capabilities of our organization. The rationale is that the functions the organization is made of tend to be more stable and less a matter of interpretation and perspective than any business process view. Enterprise maps can be used to put those value chains and process models into the organizational and application landscape context. Oracle suggests organizing the business capabilities into three segments. Thus, the default enterprise map model is prepopulated through three default lanes: core, management, and support. In many organizations, this structure is feasible as it is up to you to either use them or create your own lanes. Then we can define within each lane the key business capabilities that make up the core business processes. Figure 7: Enterprise map of RYLC depicting key business capabilities Properties of BA models Each element (goal, objective, strategy) within the model can be enriched with business properties, such as actual cost, actual time, proposed cost and proposed time. These properties are part of the impact analysis report that can be generated to analyze the BA project. Figure 8: Use properties to specify SLAs and other BA characteristics Depicting organizational units Within RYLC as an organization, we now depict its departments as organizational units. We can adorn goals to each of the units, which depict its function and the role it plays in the concert of the overall ecosystem. This association of a unit to a goal is expressed via links to goals defined in the strategy model. These goals will be used for the impact analysis reports that show the impact of changes on the organizational or procedural changes. It is possible to create several organization units as shown in the following screenshot: Figure 9: Define a set of organization units Value chains A value chain model forms the root of a process hierarchy. A value chain consists of one direct line of steps, no gateways, and no exceptions. The modeler in Oracle BPM Suite allows each step in the chain to depict associated business goals and key performance indicators (KPIs) that can be used to measure the organization's performance rather than the details of departmental performance. The value chain model is a very simple one depicting the flow of the most basic business process steps. Each step is a business process in its own right. On the level of the chain, there is no decision making expressed. This resembles a business process expressed in BPMN that has only direct lines and no gateways. Figure 10: Creation of a new value chain model called "Rental Request-to-Delivery" The value chain model type allows the structuring of your business processes into a hierarchy with a value chain forming the topmost level. Strategy models Strategy models that can be used to further motivate their KPIs are depicted at the value chain level. Figure 11: Building the strategy model for the strategy "Become a market leader" These visual maps leverage existing process documentation and match it with current business trends and existing reference models. These models depict processes and associated organizational aspects encompassing cross-departmental views on higher levels and concrete processes down to level 3. From there, they prioritize distinct processes and decide on one of several modernization strategies—process automation being just one of several! So, in the proposed methodology in the following diagram, in the Define strategy and performance measures (KPIs), the team down-selects for each business process or even subprocess one or several means to improve process efficiency or transparency. Typically, these means are defined as "supporting capabilities". These are a technique, a tool, or an approach that helps to modernize a business process. A few typical supporting capabilities are mentioned here: Explicit process automation Implicit process handling and automation inside a packaged application, such as SAP ERP or Oracle Siebel Refactored COTS existing application Business process outsourcing Business process retirement The way toward establishing these supporting capabilities is defined through a list of potential modernization strategies. Several of the modernization strategies relate to the existing applications that support a business process, such as refactoring, replacement, retirement, or re-interfacing of the respective supporting applications. The application modernization strategy that we are most interested in this article is establish explicit automated process. It is a best practice to create a high-level process map and define for each of the processes whether to leave it as it is or to depict one of several modernization strategies. When several modernization strategies are found for one process, we can drill down into the process through a hierarchy and stop at the level on which there is a disjunct modernization strategy. Figure 12: Business process automation as just one of several process optimization strategies The preceding figure is based on the BPTrends Process Change Methodology introduced in the Business Process Change book by Paul Harmon. Again, just one of these technical capabilities in strategic BPM is the topic of this article: process automation. It is not feasible to suggest automating all the business processes of any organization. Key performance indicators Within a BA project (strategy and value chain models), there are three different types of KPIs that can be defined: Manual KPI: This allows us to enter a known value Rollup KPI: This evaluates an aggregate of the child KPIs External KPI: This provides a way to include KPI data from applications other than BPM Suite, such as SAP, E-Business Suite, PeopleSoft, and so on. Additionally, KPIs can be defined on a BPMN process level, which is not covered in this article. KPIs in the value chain step level The following are the steps to configure the KPIs in the value chain step level: Open the value chain model Rental Request-to-Delivery. Right-click on the Vehicle Reservation & Allocation chain step, and select KPI. Click on the + (plus) sign to create a manual KPI, as illustrated in the next screenshot. The following image shows the configuration of the KPIs: Figure 13: Configuring a KPI Why we need a new methodology for Game Enterprise BPM Now, Game Enterprise BPM needs to be played everywhere. This implies that Game Silo BPM needs to diminish, meaning it needs to be replaced, gradually, through managed evolution, league by league, aiming for excellence at Champions League. We can't play Game Enterprise BPM with the same culture of ad hoc, joyful creativity, which we find in Game Silo BPM. We can't just approach our colleague; let's call him Ingo Maier, who we know has drawn the process model for a process we are interested in. We can't just walk over to the other desk to him, asking him about the meaning of an unclear section in the process. That is because in Game Enterprise BPM, Ingo Maier, as a person whom we know as part of our team Silo, does not exist anymore. We deal with process models, with SOA services, with a language defined somewhere else, in another department. This is what makes it so hard to move up in BPM leagues. Hiding behind the buzz term "agile" does not help. In order to raise BPM maturity up, when we move from Game Silo BPM to Game Enterprise BPM, the organization needs to establish a set of standards, guidelines, tools, and modes of operations that allow playing and succeeding at Champions League. Additionally, we have to define the modes of operations and the broad steps that lead to a desired state. This formalization of collaboration in teams should be described, agreed on, and lived as our BPM methodology. The methodology thrives for a team in which each player contributes to one coherent game along well-defined phases. Political change through Game Enterprise BPM The political problem with a cross-departmental process view becomes apparent if we look at the way organizations distribute political power in Game Silo BPM. The heads of departments form the most potent management layer while the end-to-end business process has no potent stakeholder. Thus, it is critical for any Game Enterprise BPM to establish a good balance of de facto power with process owners acting as stakeholders for a cross-departmental process view. This fills the void in Game Silo BPM of end-to-end process owners. With Game Enterprise BPM, the focus shifts from departmental improvements to the KPIs and improvement of the core business processes. Pair modeling the value chains and business processes Value chains and process models down to a still very high-level layer, such as a layer 3, can be modeled by process experts without involving technically skilled people. They should omit all technical details. To provide the foundation for automated processes, we need to add more details about domain knowledge and some technical details. Therefore, these domain process experts meet with BPM tool experts to jointly define the next level of detail in BPMN. In an analogy to the practice of pair development in agile methodologies, you could call this kind of collaboration pair modeling. Ideally, the process expert(s) and the tool expert look at the same screen and discuss how to improve the flow of the process model, while the visual representation evolves into variances, exceptions, and better understanding of the involved business objects. For many organizations that are used to a waterfall process, this is a fundamentally new way of requirement gathering that might be a challenge for some. The practice is an analogy of the customer on site practice in agile methodologies. This new way of close collaboration for process modeling is crucial for the success of BPM projects since it allows us to establish a deep and shared understanding in a very pragmatic and productive way. Figure 14: Roles and successful modes of collaboration When the process is modeled in sufficient detail to clearly depict an algorithmic definition of the flow of the process and all its variances, the model can be handed over to BPM developers. They add all the technical bells and whistles, such as data mapping, decision rules, service calls, and exception handling. Portal developers will work on their implementation of the use cases. SOA developers will use Oracle SOA Suite to integrate with backend systems, therefore implementing SOA services. The discussed notion of a handover from higher-level business process models to development teams can also be used to depict the line at which it might make sense to outsource parts of the overall development. Summary In this article, we saw how BPM as an approach to model, automate, and optimize business process is typically applied rather on a departmental level. We saw how BPM Suite 12c introduced new features that allow us to cross the bridge towards the top-down, cross-departmental, enterprise-level BPM. We depicted the key characteristics of the enterprise BPM methodology, which aligns corporate or strategic activities with actual process automation projects. We learned the importance of modeling standards and guidelines, which should be used to gain business process insight and understanding on broad levels throughout the enterprise. The goal is to establish a shared language to talk about the capabilities and processes of the overall organization and the services it provides to its customers. The role of data in SOA that will support business processes was understood with a critical success factor being the definition of the business data model that, along with services, will form the connection between the process layer, user interface layer, and the services layer. We understood how important it is to separate application logic from service logic and process logic to ensure the benefits of a process-driven architecture are realized. Resources for Article: Further resources on this subject: Introduction to Oracle BPM [article] Oracle B2B Overview [article] Event-driven BPEL Process [article]

The Banana Pi is a single-board computer, which enables you to build your own individual and versatile system. In fact, it is a complete computer, including all the required elements such as a processor, memory, network, and other interfaces, which we are going to explore. It provides enough power to run even relatively complex applications suitably. In this article by, Ryad El-Dajani, author of the book, Banana Pi Cookbook, we are going to get to know the Banana Pi device. The available distributions are mentioned, as well as how to download and install these distributions. We will also examine Android in contrast to our upcoming Linux adventure. (For more resources related to this topic, see here.) Thus, you are going to transform your little piece of hardware into a functional, running computer with a working operating system. You will master the whole process of doing the required task from connecting the cables, choosing an operating system, writing the image to an SD card, and successfully booting up and shutting down your device for the first time. Banana Pi Overview In the following picture, you see a Banana Pi on the left-hand side and a Banana Pro on the right-hand side: As you can see, there are some small differences that we need to notice. The Banana Pi provides a dedicated composite video output besides the HDMI output. However, with the Banana Pro, you can connect your display via composite video output using a four-pole composite audio/video cable on the jack. In contrast to the Banana Pi, which has 26 pin headers, the Banana Pro provides 40 pins. Also the pins for the UART port interface are located below the GPIO headers on the Pi, while they are located besides the network interface on the Pro. The other two important differences are not clearly visible on the previous picture. The operating system for your device comes in the form of image files that need to be written (burned) to an SD card. The Banana Pi uses normal SD cards while the Banana Pro will only accept Micro SD cards. Moreover, the Banana Pro provides a Wi-Fi interface already on board. Therefore, you are also able to connect the Banana Pro to your wireless network, while the Pi would require an external wireless USB device. Besides the mentioned differences, the devices are very similar. You will find the following hardware components and interfaces on your device. On the back side, you will find: A20 ARM Cortex-A7 dual core central processing unit (CPU) ARM Mali400 MP2 graphics processing unit (GPU) 1 gigabyte of DDR3 memory (that is shared with the GPU) On the front side, you will find: Ethernet network interface adapter Two USB 2.0 ports A 5V micro USB power with DC in and a micro USB OTG port A SATA 2.0 port and SATA power output Various display outputs [HDMI, LVDS, and composite (integrated into jack on the Pro)] A CSI camera input connector An infrared (IR) receiver A microphone Various hardware buttons on board (power key, reset key, and UBoot key) Various LEDs (red for power status, blue for Ethernet status, and green for user defined) As you can see, you have a lot of opportunities for letting your device interact with various external components. Operating systems for the Banana Pi The Banana Pi is capable of running any operating system that supports the ARM Cortex-A7 architecture. There are several operating systems precompiled, so you are able to write the operating system to an SD card and boot your system flawlessly. Currently, there are the following operating systems provided officially by LeMaker, the manufacturer of the Banana Pi. Android Android is a well-known operating system for mobile phones, but it is also runnable on various other devices such as smart watches, cars, and, of course, single-board computers such as the Banana Pi. The main advantage of running Android on a single-board computer is its convenience. Anybody who uses an Android-based smartphone will recognize the graphical user interface (GUI) and may have less initial hurdles. Also, setting up a media center might be easier to do on Android than on a Linux-based system. However, there are also a few disadvantages, as you are limited to software that is provided by an Android store such as Google Play. As most apps are optimized for mobile use at the moment, you will not find a lot of usable software for your Banana Pi running Android, except some Games and Multimedia applications. Moreover, you are required to use special Windows software called PhoenixCard to be able to prepare an Android SD card. In this article, we are going to ignore the installing of Android. For further information, please see Installing the Android OS image (LeMaker Wiki) at http://wiki.lemaker.org/BananaPro/Pi:SD_card_installation. Linux Most of the Linux users never realize that they are actually using Linux when operating their phones, appliances, routers, and many more products, as most of its magic happens in the background. We are going to dig into this adventure to discover its possibilities when running on our Banana Pi device. The following Linux-based operating systems—so-called distributions—are used by the majority of the Banana Pi user base and are supported officially by the manufacturer: Lubuntu: This is a lightweight distribution based on the well-known Ubuntu using the LXDE desktop, which is principally a good choice, if you are a Windows user. Raspbian: This is a distribution based on Debian, which was initially produced for the Raspberry Pi (hence the name). As a lot of Raspberry Pi owners are running Raspbian on their devices while also experimenting with the Banana Pi, LeMaker ported the original Raspbian distribution to the Banana Pi. Raspbian also comes with an LXDE desktop by default. Bananian: This too is a Debian-based Linux distribution optimized exclusively for the Banana Pi and its siblings. All of the aforementioned distributions are based on the well-known distribution, Debian. Besides the huge user base, all Debian-based distributions use the same package manager Apt (Advanced Packaging Tool) to search for and install new software, and all are similar to use. There are still more distributions that are officially supported by LeMaker, such as Berryboot, LeMedia, OpenSUSE, Fedora, Gentoo, Scratch, ArchLinux, Open MediaVault, and OpenWrt. All of them have their pros and cons or their specific use cases. If you are an experienced Linux user, you may choose your preferred distribution from the mentioned list, as most of the recipes are similar to, or even equally usable on, most of the Linux-based operating systems. Moreover, the Banana Pi community publishes various customized Linux distributions for the Banana Pi regularly. The possible advantages of a customized distribution may include enabled and optimized hardware acceleration capabilities, supportive helper scripts, fully equipped desktop environments, and much more. However, when deciding to use a customized distribution, there is no official support by LeMaker and you have to contact the publisher in case you encounter bugs, or need help. You can also check the customized Arch Linux image that author have built (http://blog.eldajani.net/banana-pi-arch-linux-customized-distribution/) for the Banana Pi and Banana Pro, including several useful applications. Downloading an operating system for the Banana Pi The following two recipes will explain how to set up the SD card with the desired operating system and how to get the Banana Pi up and running for the first time. This recipe is a predecessor. Besides the device itself, you will need at least a source for energy, which is usually a USB power supply and an SD card to boot your Banana Pi. Also, a network cable and connection is highly recommended to be able to interact with your Banana Pi from another computer via a remote shell using the application. You might also want to actually see something on a display. Then, you will need to connect your Banana Pi via HDMI, composite, or LVDS to an external screen. It is recommended that you use an HDMI Version 1.4 cable since lower versions can possibly cause issues. Besides inputting data using a remote shell, you can directly connect an USB keyboard and mouse to your Banana Pi via the USB ports. After completing the required tasks in the upcoming recipes, you will be able to boot your Banana Pi. Getting ready The following components are required for this recipe: Banana Pi SD card (minimum class 4; class 10 is recommended) USB power supply (5V 2A recommended) A computer with an SD card reader/writer (to write the image to the SD card) Furthermore, you are going to need an Internet connection to download a Linux distribution or Android. A few optional but highly recommended components are: Connection to a display (via HDMI or composite) Network connection via Ethernet USB keyboard and mouse You can acquire these items from various retailers. All items shown in the previous two pictures were bought from an online retailer that is known for originally selling books. However, the Banana Pi and the other products can be acquired from a large number of retailers. It is recommended to get a USB power supply with 2000mA (2A) output. How to do it… To download an operating system for Banana Pi, follow these steps: Download an image of your desired operating system. We are going to download Android and Raspbian from the official LeMaker image files website: http://www.lemaker.org/resources/9-38/image_files.html. The following screenshot shows the LeMaker website where you can download the official images: If you are clicking on one of the mirrors (such as Google Drive, Dropbox, and so on), you will be redirected to the equivalent file-hosting service. From there, you are actually able to download the archive file. Once your archive containing the image is downloaded, you are ready to unpack the downloaded archive, which we will do in the upcoming recipes. Setting up the SD card on Windows This recipe will explain how to set up the SD card using a Windows operating system. How to do it… In the upcoming steps, we will unpack the archive containing the operating system image for the Banana Pi and write the image to the SD card: Open the downloaded archive with 7-Zip. The following screenshot shows the 7-Zip application opening a compressed .tgz archive: Unpack the archive to a directory until you get a file with the file extension .img. If it is .tgz or .tar.gz file, you will need to unpack the archive twice Create a backup of the contents of the SD card as everything on the SD card is going to be erased unrecoverablely. Open SD Formatter (https://www.sdcard.org/downloads/formatter_4/) and check the disk letter (E: in the following screenshot). Choose Option to open the Option Setting window and choose: FORMAT TYPE: FULL (Erase) FORMAT SIZE ADJUSTMENT: ON When everything is configured correctly, check again to see whether you are using the correct disk and click Format to start the formatting process. Writing a Linux distribution image to the SD card on Windows The following steps explain how to write a Linux-based distribution to the SD card on Windows: Format the SD card using SD Formatter, which we covered in the previous section. Open the Win32 Disk Imager (http://sourceforge.net/projects/win32diskimager/). Choose the image file by clicking on the directory button. Check, whether you are going to write to the correct disk and then click on Write. Once the burning process is done, you are ready to insert the freshly prepared SD card containing your Linux operating system into the Banana Pi and boot it up for the first time. Booting up and shutting down the Banana Pi This recipe will explain how to boot up and shut down the Banana Pi. As the Banana Pi is a real computer, these tasks are as equally important as tasks on your desktop computer. The booting process starts the Linux kernel and several important services. The shutting down stops them accordingly and does not power off the Banana Pi until all data is synchronized with the SD card or external components correctly. How to do it… We are going to boot up and shut down the Banana Pi. Booting up Do the following steps to boot up your Banana Pi: Attach the Ethernet cable to your local network. Connect your Banana Pi to a display. Plug in an USB keyboard and mouse. Insert the SD card to your device. Power your Banana Pi by plugging in the USB power cable. The next screenshot shows the desktop of Raspbian after a successful boot: Shutting down Linux To shut down your Linux-based distribution, you either use the shutdown command or do it via the desktop environment (in case of Raspbian, it is called LXDE). For the latter method, these are the steps: Click on the LXDE icon in the lower-left corner. Click on Logout. Click on Shutdown in the upcoming window. To shut down your operating system via the shell, type in the following command: $ sudo shutdown -h now Connecting via SSH on Windows using PuTTY The following recipe shows you how to connect to your Banana Pi remotely using an open source application called PuTTY. Getting ready For this recipe, you will need the following ingredients: A booted up Linux operating system on your Banana Pi connected to your local network The PuTTY application on your Windows PC that is also connected to your local area network How to do it… To connect to your Banana Pi via SSH on Windows, perform the following: Run putty.exe. You will see the PuTTY Configuration dialog. Enter the IP address of the Banana Pi and leave the Port as number 22 as. Click on the Open button. A new terminal will appear, attempting a connection to the Banana Pi. When connecting to the Banana Pi for the first time, you will see a PuTTY security alert. The following screenshot shows the PuTTY Security Alert window: Trust the connection by clicking on Yes. You will be requested to enter the login credentials. Use the default username bananapi and password bananapi. When you are done, you should be welcomed by the shell of your Banana Pi. The following screenshot shows the shell of your Banana Pi accessed via SSH using PuTTY on Windows: To quit your SSH session, execute the command exit or press Ctrl + D. Searching, installing, and removing the software Once you have your decent operating system on the Banana Pi, sooner or later you are going to require a new software. As most software for Linux systems is published as open source, you can obtain the source code and compile it for yourself. One alternative is to use a package manager. A lot of software is precompiled and provided as installable packages by the so-called repositories. In case of Debian-based distributions (for example, Raspbian, Bananian, and Lubuntu), the package manager that uses these repositories is called Advanced Packaging Tool (Apt). The two most important tools for our requirements will be apt-get and apt-cache. In this recipe, we will cover the searching, the installing, and removing of software using the Apt utilities. Getting ready The following ingredients are required for this recipe. A booted Debian-based operating system on your Banana Pi An Internet connection How to do it… We will separate this recipe into searching for, installing and removing of packages. Searching for packages In the upcoming example, we will search for a solitaire game: Connect to your Banana Pi remotely or open a terminal on the desktop. Type the following command into the shell: $ apt-cache search solitaire You will get a list of packages that contain the string solitaire in their package name or description. Each line represents a package and shows the package name and description separated by a dash (-). Now we have obtained a list of solitaire games: The preceding screenshot shows the output after searching for packages containing the string solitaire using the apt-cache command. Installing a package We are going to install a package by using its package name. From the previous received list, we select the package ace-of-penguins. Type the following command into the shell: $ sudo apt-get install ace-of-penguins If asked to type the password for sudo, enter the user's password. If a package requires additional packages (dependencies), you will be asked to confirm the additional packages. In this case, enter Y. After downloading and installing, the desired package is installed: Removing a package When you want to uninstall (remove) a package, you also use the apt-get command: Type the following command into a shell: $ sudo apt-get remove ace-of-penguins If asked to type the password for sudo, enter the user's password. You will be asked to confirm the removal. Enter Y. After this process, the package is removed from your system. You will have uninstalled the package ace-of-penguins. Summary In this article, we discovered the installation of a Linux operating system on the Banana Pi. Furthermore, we connected to the Banana Pi via the SSH protocol using PuTTY. Moreover, we discussed how to install new software using the Advanced Packaging Tool. This article is a combination of parts from the first two chapters of the Banana Pi Cookbook. In the Banana Pi Cookbook, we are diving more into detail and explain the specifics of the Banana Pro, for example, how to connect to the local network via WLAN. If you are using a Linux-based desktop computer, you will also learn how to set up the SD card and connect via SSH to your Banana Pi on your Linux computer. Resources for Article: Further resources on this subject: Color and motion finding [article] Controlling the Movement of a Robot with Legs [article] Develop a Digital Clock [article]

The containerization paradigm has been there in the IT industry for quite a long time in different forms. However, the streamlined stuffing and packaging mission-critical software applications inside highly optimized and organized containers to be efficiently shipped, deployed and delivered from any environment (virtual machines or bare-metal servers, local or remote, and so on) has got a new lease of life with the surging popularity of the Docker platform. The open source Docker project has resulted in an advanced engine for automating most of the activities associated with Docker image formation and the creation of self-defined and self-sufficient, yet collaborative, application-hosted containers. Further on, the Docker platform has come out with additional modules to simplify container discovery, networking, orchestration, security, registry, and so on. In short, the end-to-end life cycle management of Docker containers is being streamlined sagaciously, signaling a sharp reduction of workloads of application developers and system administrators. In this article, we want to highlight the Docker paradigm and its illustrating and insightful capabilities for bringing forth newer possibilities and fresh opportunities. In this article by Pethuru Raj, Jeeva S. Chelladhurai, Vinod Singh, authors of the book Learning Docker, we will cover the reason behind the Docker platform's popularity. Docker is an open platform for developers and system administrators of distributed applications. Building and managing distributed applications is beset with a variety of challenges, especially in the context of pervasive cloud environments. Docker brings forth a bevy of automation through the widely leveraged abstraction technique in successfully enabling distributed applications. The prime point is that the much-talked simplicity is being easily achieved through the Docker platform. With just a one-line command, you can install any application. For example, popular applications and platforms, such as WordPress, MySQL, Nginx, and Memcache, can all be installed and configured with just one command. Software and platform vendors are systematically containerizing their software packages to be readily found, subscribed, and used by many. Performing changes on applications and putting the updated and upgraded images in the repository is made easier. That's the real and tectonic shift brought in by the Docker-sponsored containerization movement. There are other advantages with containerization as well, and they are explained in the subsequent sections. Explaining the key drivers for Docker containerization We have been fiddling with the virtualization technique and tools for quite a long time now in order to establish the much-demanded software portability. The inhibiting dependency factor between software and hardware needs to be decimated with the leverage of virtualization, a kind of beneficial abstraction, through an additional layer of indirection. The idea is to run any software on any hardware. This is done by creating multiple virtual machines (VMs) out of a single physical server, and each VM has its own operating system (OS). Through this isolation, which is enacted through automated tools and controlled resource sharing, heterogeneous applications are accommodated in a physical machine. With virtualization, IT infrastructures become open; programmable; and remotely monitorable, manageable, and maintainable. Business workloads can be hosted in appropriately sized virtual machines and delivered to the outside world, ensuring broader and higher utilization. On the other side, for high-performance applications, virtual machines across multiple physical machines can be readily identified and rapidly combined to guarantee any kind of high-performance needs. However, the virtualization paradigm has its own drawbacks. Because of the verbosity and bloatedness (every VM carries its own operating system), VM provisioning typically takes some minutes; the performance goes down due to excessive usage of compute resources; and so on. Furthermore, the much-published portability need is not fully met by virtualization. The hypervisor software from different vendors comes in the way of ensuring application portability. The OS and application distribution, version, edition, and patch differences hinder smooth portability. Compute virtualization has flourished, whereas the other closely associated network and storage virtualization concepts are just taking off. Building distributed applications through VM interactions invites and involves some practical difficulties. Let's move over to containerization. All of these barriers easily contribute to the unprecedented success of the containerization idea. A container generally contains an application, and all of the application's libraries, binaries, and other dependencies are stuffed together to be presented as a comprehensive, yet compact, entity for the outside world. Containers are exceptionally lightweight, highly portable, easily and quickly provisionable, and so on. Docker containers achieve native system performance. The greatly articulated DevOps goal gets fully fulfilled through application containers. As best practice, it is recommended that every container host one application or service. The popular Docker containerization platform has come out with an enabling engine to simplify and accelerate life cycle management of containers. There are industry-strength and open automated tools made freely available to facilitate the needs of container networking and orchestration. Thereby, producing and sustaining business-critical distributed applications is becoming easy. Business workloads are methodically containerized to be easily taken to cloud environments, and they are exposed for container crafters and composers to bring forth cloud-based software solutions and services. Precisely speaking, containers are turning out to be the most featured, favored, and fine-tuned runtime environment for IT and business services. Reflecting the containerization journey The concept of containerization for building software containers that bundle and sandbox software applications has been there for years at different levels. Consolidating all constituting and contributing modules together into a single software package is a grand way out for faster and error-free provisioning, deployment, and delivery of software applications. Web application development and operations teams have been extensively handling a variety of open source as well as commercial-grade containers (for instance, servlet containers). These are typically deployment and execution containers or engines that suffer from the ignominious issue of portability. However, due to its unfailing versatility and ingenuity, the idea of containerization has picked up with a renewed verve at a different level altogether, and today it is being positioned as the game-changing OS-level virtualization paradigm in achieving the elusive goal of software portability. There are a few OS-level virtualization technologies on different operating environments (For example on Linux such as OpenVZ and Linux-VServer, FreeBSD jails, AIX workload partitions, and Solaris containers). Containers have the innate potential to transform the way we run and scale applications systematically. They isolate and encapsulate our applications and platforms from the host system. A container is typically an OS within your host OS, and you can install and run applications on it. For all practical purposes, a container behaves like a virtual machine (VM), and especially Linux-centric containers are greatly enabled by various kernel-level features (cgroups and namespaces). Because applications are logically isolated, users can run multiple versions of PHP, Python, Ruby, Apache software, and so on coexisting in a host and cleanly tucked away in their containers. Applications are methodically being containerized using the sandbox aspect (a subtle and smart isolation technique) to eliminate all kinds of constricting dependencies. A typical present-day container contains not only an application but also its dependencies and binaries to make it self-contained to run everywhere (laptop, on-premise systems as well as off-premise systems), without any code tweaking and twisting. Such comprehensive and compact sandboxed applications within containers are being prescribed as the most sought-after way forward to fully achieve the elusive needs of IT agility, portability, extensibility, scalability, maneuverability, and security. In a cloud environment, various workloads encapsulated inside containers can easily be moved across systems—even across cloud centers—and cloned, backed up, and deployed rapidly. Live-in modernization and migration of applications is possible with containerization-backed application portability. The era of adaptive, instant-on, and mission-critical application containers that elegantly embed software applications inside itself is set to flourish in the days to come, with a stream of powerful technological advancements. Describing the Docker Platform However, containers are hugely complicated and not user-friendly. Following the realization of the fact that several complexities were coming in the way of massively producing and fluently using containers, an open source project was initiated, with the goal of deriving a sophisticated and modular platform that consists of an enabling engine for simplifying and streamlining containers' life cycles. In other words, the Docker platform was built to automate crafting, packaging, shipping, deployment, and delivery of any software application embedded in a lightweight, extensible, and self-sufficient container that can run virtually anywhere. Docker is being considered as the most flexible and futuristic containerization technology in realizing highly competent and enterprise-class distributed applications. This is meant to make deft and decisive impacts, as the brewing trend in the IT industry is that instead of large, monolithic applications distributed on a single physical or virtual server, companies are building smaller, self-defined, sustainable, easily manageable, and discrete applications. Docker is emerging as an open solution for developers and system administrators to quickly develop, ship, and run any applications. It lets you quickly assemble applications from disparate and distributed components, and eliminates any kind of friction that could come when shipping code. Docker, through a host of tools, simplifies the isolation of individual processes and makes them self-sustainable by running them in lightweight containers. It essentially isolates the "process" in question from the rest of your system, rather than adding a thick layer of virtualization overhead between the host and the guest. As a result, Docker containers run much faster, and its images can be shared easily for bigger and better application containers. The Docker-inspired container technology takes the concept of declarative resource provisioning a step further to bring in the critical aspects of simplification, industrialization, standardization, and automation. Enlightening the major components of the Docker platform The Docker platform architecture comprises several modules, tools, and technologies in order to bring in the originally envisaged benefits of the blossoming journey of containerization. The Docker idea is growing by leaps and bounds, as the open source community is putting in enormous efforts and contributing in plenty in order to make Docker prominent and dominant in production environments. Docker containers internally use a standardized execution environment called Libcontainer, which is an interface for various Linux kernel isolation features, such as namespaces and cgroups. This architecture allows multiple containers to be run in complete isolation from one another while optimally sharing the same Linux kernel. A Docker client doesn't communicate directly with the running containers. Instead, it communicates with the Docker daemon via the venerable sockets, or through a RESTful API. The daemon communicates directly with the containers running on the host. The Docker client can either be installed local to the daemon or on a different host altogether. The Docker daemon runs as root and is capable of orchestrating all the containers running on the host. A Docker image is the prime build component for any Docker container. It is a read-only template from which one or more container instances can be launched programmatically. Just as virtual machines are based on images, Docker Containers are also based on Docker images. These images are typically tiny compared to VM images, and are conveniently stackable thanks to the property of AUFS, a prominent Docker filesystem. Of course, there are other filesystems being supported by the Docker platform. Docker Images can be exchanged with others and versioned like source code in private or public Docker registries. Registries are used to store images and can be local or remote. When a container gets launched, Docker first searches in the local registry for the launched image. If it is not found locally, then it searches in the public remote registry (DockerHub). If the image is there, Docker downloads it on the local registry and uses it to launch the container. This makes it easy to distribute images efficiently and securely. One of Docker's killer features is the ability to find, download, and start container images created by other developers quickly. In addition to the various base images, which you can use to create your own Docker containers, the public Docker registry features images of ready-to-run software, including databases, middleware solutions, content management systems, development environments, web servers, and so on. While the Docker command-line client searches in the public registry by default, it is also possible to maintain private registries. This is a great option for distributing images with proprietary code or components internally in your company. Pushing images to the registry is just as easy as downloading images from them. Docker Inc.'s registry has a web-based interface for searching, reading, commenting, and recommending (also known as starring) images. A Docker container is a running instance of a Docker image. You can create Docker images from a running container as well. For example, you can launch a container, install a bunch of software packages using a package manager such as APT or yum, and then commit those changes to a new Docker image. But a more powerful and extensible way of creating Docker images is through a Dockerfile, the widely used build-mechanism of Docker images through declarations. The Dockerfile mechanism gets introduced to automate the build process. You can put a Dockerfile under version control and have a perfectly repeatable way of creating images. Expounding the Containerization-enabling Technologies Linux has offered a surfeit of ways to containerize applications, from its own Linux containers (LXC) to infrastructure-based technologies. Due to the tight integration, a few critical drawbacks are frequently associated with Linux containers. Libcontainer is, therefore, the result of collaborative and concerted attempts by many established vendors to standardize the way applications are packed up, delivered, and run in isolation. Libcontainer provides a standard interface to make sandboxes or containers inside an OS. With it, a container can interface in a predictable way with the host OS resources. This enables the application inside it to be controlled as expected. Libcontainer came into being as an important ingredient of the Docker platform, and uses kernel-level namespaces to isolate the container from the host. The user namespace separates the container's and host's user databases. This succinct arrangement ensures that the container's root user does not have root privileges on the host. The process namespace is responsible for displaying and managing processes running only in the container, not the host. And the network namespace provides the container with its own network device and IP address. Control Groups (cgroups) is another important contributor. While namespaces are responsible for isolation of the host from the container, control groups implement resource monitoring, accounting, and auditing. While allowing Docker to limit the resources being consumed by a container, such as memory, disk space, and I/O, cgroups also outputs a lot of useful metrics about these resources. These metrics allow Docker to monitor resource consumption of the various processes within the containers, and make sure that each of them gets only its fair share of the available resources. Let's move on to Docker Filesystem now. In a traditional Linux boot, the kernel first mounts the root filesystem (rootfs) as read-only, checks its integrity, and then switches the whole rootfs volume to read-write mode. When Docker mounts rootfs, it starts as read-only, but instead of changing the filesystem to read-write mode, it takes advantage of a union mount to add a read-write filesystem over the read-only file system. In fact, there may be multiple read-only filesystems stacked on top of each other. Each of these filesystems is considered as a layer. Docker has been using AuFS (short for Advanced union File system) as a filesystem for containers. When a process needs to modify a file, AuFS creates a copy of that file and is capable of merging multiple layers into a single representation of a filesystem. This process is called copy-on-write. The really cool thing is that AuFS allows Docker to use certain images as the basis for containers. That is, only one copy of the image is required to be stored, resulting in saving of storage and memory, as well as faster deployment of containers. AuFS controls the version of container images, and each new version is simply a difference of changes from the previous version, effectively keeping image files to a minimum. Container orchestration is an important ingredient for sustaining the journey of containerization towards the days of smarter containers. Docker has added a few commands to facilitate safe and secure networking of local as well as remote containers. There are orchestration tools and tricks emerging and evolving in order to establish the much-needed dynamic and seamless linkage between containers, and thus produce composite containers that are more aligned and tuned for specific IT as well as business purposes. Nested containers are bound to flourish in the days to come, with continued focus on composition technologies. There are additional composition services being added by the Docker team in order to make container orchestration pervasive. Exposing the distinct benefits of the Docker platform These are the direct benefits of using Docker containers: Enabling Docker containers for micro services: Micro service architecture (MSA) is an architectural concept that typically aims to disentangle a software-intensive solution by decomposing its distinct functionality into a set of discrete services. These services are built around business capabilities, and are independently deployable through automated tools. Each micro service can be deployed without interrupting other micro services. With the monolithic era all set to fade away, micro service architecture is slowly, yet steadily, emerging as a championed way to design, build, and sustain large-scale software systems. It not only solves the problem of loose coupling and software modularity, but is a definite boon for continuous integration and deployment, which are the hallmarks of the agile world. As we all know, introducing changes in one part of an application, which mandates changes across the application, has been a bane to the goal of continuous deployment. MSA needs lightweight mechanisms for small and independently deployable services, scalability, and portability. These requirements can easily be met by smartly using containers. Containers provide an ideal environment for service deployment in terms of speed, isolation management, and life cycle. It is easy to deploy new versions of services inside containers. They are also better suited for micro services than virtual machines because micro services can start up and shut down more quickly. In addition, computing, memory, and other resources can scale independently. Synchronizing well with cloud environments : With the faster maturity of the Docker platform, there is a new paradigm of Containers as a Service (CaaS) emerging and evolving; that is, containers are being readied, hosted, and delivered as a service from the cloud (private, public, or hybrid) over the public Web. All the necessary service enablement procedures are being meticulously enacted on containers to make them ready for the forthcoming service era. In other words, knowledge-filled, service-oriented, cloud-based, composable, and cognitive containers are being proclaimed as one of the principal ingredients for the establishment and sustenance of a vision of a smarter planet. Precisely speaking, applications are containerized and exposed as services to be discovered and used by a variety of consumers for a growing set of use cases. Eliminating the dependency hell: With Docker containers (the lightweight and nimble cousin of VMs), packaging an application along with all of its dependencies compactly and run it smoothly in development, test, and production environments get automated. The Docker engine resolves the problem of the dependency hell. Modern applications are often assembled from existing components and rely on other services and applications. For example, your Python application might use PostgreSQL as a data store, Redis for caching, and Apache as a web server. Each of these components comes with its own set of dependencies, which may conflict with those of the other components. By precisely packaging each component and its dependencies, Docker solves the dependency hell. The brewing and beneficial trend is to decompose applications into decoupled, minimalist, and specialized containers that are designed to do a single thing really well. Bridging Development and Operations Gaps: There are configuration tools and other automation tools, such as Puppet, Chef, Salt, and so on, pioneered for the DevOps movement. However, they are more tuned for operations teams. The Docker idea is the first and foremost in bringing development as well as operations team members together meticulously. Developers can work inside the containers and operations engineers can work outside the containers simultaneously. This is because Docker encapsulates everything for an application to run independently and the prevailing gaps between development and operation are decimated completely. Guaranteeing Consistency for the Agile World: We know that the aspects of continuous integration (CI) and continuous delivery (CD) are very vital for agile production, as they substantially reduce the number of bugs in any software solution getting released to the market. We also know that the traditional CI and CD tools build application slugs by pulling source code repositories. Although these tools work relatively well for many applications, there can be binary dependencies or OS-level variations that make code runs slightly different in production environments than in development and staging environments. Furthermore, CI was built with the assumption that an application was located in single code repository. Now, with the arrival of Docker containers, there is a fresh set of CI tools that can be used to start testing multi-container applications that draw from multiple code repositories. Bringing in collaboration for the best-of-breed containers: Instead of tuning your own service containers for various software packages, Docker encourages the open source community to collaborate and fine-tune containers in Docker Hub, the public Docker repository. Thus, Docker is changing the cloud development practices by allowing anyone to leverage the proven and potential best practices in the form of stitching together other people's containers. It is like having a Lego set of cloud components to stick them together. Accelerating resource provisioning: In the world of the cloud, faster provisioning of IT resources and business applications is very important. Typically, VM provisioning takes 5 or more minutes, whereas Docker containers, because of their lightweight nature, can be provisioned in a few seconds through Docker tools. Similarly, live-in migration of containers inside as well as across cloud centers is being facilitated. A bare-metal (BM) server can accommodate hundreds of Docker containers that are easily being deployed on the cloud. With a series of advancements, containers are expected to be nested. A software component that makes up a tier in one container might be called by another in a remote location. An update of the same tier might be passed on to any other container that uses the same component to ensure utmost consistency. Envisaging application-aware containers - Docker promotes an application-centric container model. That is to say, containers should run individual applications or services rather than a slew of them. Containers are fast and consume lesser amount of resources to create and run applications. Following the single-responsibility principle and running one main process per container results in loose coupling of the components of your system. Empowering IT agility, autonomy, and affordability: Docker containers are bringing down IT costs significantly, simplifying IT operations and making OS patching easier, as there are fewer OSes to manage. This ensures greater application mobility, easier application patching, and better workload visibility and controllability. This in turn guarantees higher resource utilization and faster responses for newer workload requirements, delivering newer kinds of sophisticated services quickly. Summary Docker harnesses some powerful kernel-level technologies smartly and provides a simple tool set and a unified API for managing the namespace, cgroups, and a copy-on-write filesystem. The guys at Docker Inc. has created a compact tool that is greater than the sum of all its parts. The result is a potential game changer for DevOps, system administrators, and developers. Docker provides tools to make creating and working with containers as easy as possible. Containers sandbox processes from each other. Docker does a nice job of harnessing the benefits of containerization for a focused purpose, namely lightweight packaging and deployment of applications. Precisely speaking, virtualization and other associated technologies coexist with containerization to ensure software-defined, self-servicing, and composite clouds. Resources for Article: Further resources on this subject: Network and Data Management for Containers [article] Giving Containers Data and Parameters [article] Unboxing Docker [article]

In this article by Vinoo Das author of the book Learning Redis, we will see how Redis as a NOSQL data store, provides a loose sense of transaction. As in a traditional RDBMS, the transaction starts with a BEGIN and ends with either COMMIT or ROLLBACK. All these RDBMS servers are multithreaded, so when a thread locks a resource, it cannot be manipulated by another thread unless and until the lock is released. Redis by default has MULTI to start and EXEC to execute the commands. In case of a transaction, the first command is always MULTI, and after that all the commands are stored, and when EXEC command is received, all the stored commands are executed in sequence. So inside the hood, once Redis receives the EXEC command, all the commands are executed as a single isolated operation. Following are the commands that can be used in Redis for transaction: MULTI: This marks the start of a transaction block EXEC: This executes all the commands in the pipeline after MULTI WATCH: This watches the keys for conditional execution of a transaction UNWATCH: This removes the WATCH keys of a transaction DISCARD: This flushes all the previously queued commands in the pipeline (For more resources related to this topic, see here.) The following figure represents how transaction in Redis works: Transaction in Redis Pipeline versus transaction As we have seen for many generic terms in pipeline the commands are grouped and executed, and the responses are queued in a block and sent. But in transaction, until the EXEC command is received, all the commands received after MULTI are queued and then executed. To understand that, it is important to take a case where we have a multithreaded environment and see the outcome. In the first case, we take two threads firing pipelined commands at Redis. In this sample, the first thread fires a pipelined command, which is going to change the value of a key multiple number of times, and the second thread will try to read the value of that key. Following is the class which is going to fire the two threads at Redis: MultiThreadedPipelineCommandTest.java: package org.learningRedis.chapter.four.pipelineandtx; public class MultiThreadedPipelineCommandTest { public static void main(String[] args) throws InterruptedException {    Thread pipelineClient = new Thread(new PipelineCommand());    Thread singleCommandClient = new Thread(new SingleCommand());    pipelineClient.start();    Thread.currentThread().sleep(50);    singleCommandClient.start(); } } The code for the client which is going to fire the pipeline commands is as follows: package org.learningRedis.chapter.four.pipelineandtx; import java.util.Set; import Redis.clients.jedis.Jedis; import Redis.clients.jedis.Pipeline; public class PipelineCommand implements Runnable{ Jedis jedis = ConnectionManager.get(); @Override public void run() {      long start = System.currentTimeMillis();      Pipeline commandpipe = jedis.pipelined();      for(int nv=0;nv<300000;nv++){        commandpipe.sadd("keys-1", "name"+nv);      }      commandpipe.sync();      Set<String> data= jedis.smembers("keys-1");      System.out.println("The return value of nv1 after pipeline [ " + data.size() + " ]");    System.out.println("The time taken for executing client(Thread-1) "+ (System.currentTimeMillis()-start));    ConnectionManager.set(jedis); } } The code for the client which is going to read the value of the key when pipeline is executed is as follows: package org.learningRedis.chapter.four.pipelineandtx; import java.util.Set; import Redis.clients.jedis.Jedis; public class SingleCommand implements Runnable { Jedis jedis = ConnectionManager.get(); @Override public void run() {    Set<String> data= jedis.smembers("keys-1");    System.out.println("The return value of nv1 is [ " + data.size() + " ]");    ConnectionManager.set(jedis); } } The result will vary as per machine configuration but by changing the thread sleep time and running the program couple of times, the result will be similar to the one shown as follows: The return value of nv1 is [ 3508 ] The return value of nv1 after pipeline [ 300000 ] The time taken for executing client(Thread-1) 3718 Please fire FLUSHDB command every time you run the test, otherwise you end up seeing the value of the previous test run, that is 300,000 Now we will run the sample in a transaction mode, where the command pipeline will be preceded by MULTI keyword and succeeded by EXEC command. This client is similar to the previous sample where two clients in separate threads will fire commands to a single key on Redis. The following program is a test client that gives two threads one with commands in transaction mode and the second thread will try to read and modify the same resource: package org.learningRedis.chapter.four.pipelineandtx; public class MultiThreadedTransactionCommandTest { public static void main(String[] args) throws InterruptedException {    Thread transactionClient = new Thread(new TransactionCommand());    Thread singleCommandClient = new Thread(new SingleCommand());    transactionClient.start();    Thread.currentThread().sleep(30);    singleCommandClient.start(); } } This program will try to modify the resource and read the resource while the transaction is going on: package org.learningRedis.chapter.four.pipelineandtx; import java.util.Set; import Redis.clients.jedis.Jedis; public class SingleCommand implements Runnable { Jedis jedis = ConnectionManager.get(); @Override public void run() {    Set<String> data= jedis.smembers("keys-1");    System.out.println("The return value of nv1 is [ " + data.size() + " ]");    ConnectionManager.set(jedis); } } This program will start with MULTI command, try to modify the resource, end it with EXEC command, and later read the value of the resource: package org.learningRedis.chapter.four.pipelineandtx; import java.util.Set; import Redis.clients.jedis.Jedis; import Redis.clients.jedis.Transaction; import chapter.four.pubsub.ConnectionManager; public class TransactionCommand implements Runnable { Jedis jedis = ConnectionManager.get(); @Override public void run() {      long start = System.currentTimeMillis();      Transaction transactionableCommands = jedis.multi();      for(int nv=0;nv<300000;nv++){        transactionableCommands.sadd("keys-1", "name"+nv);      }      transactionableCommands.exec();      Set<String> data= jedis.smembers("keys-1");      System.out.println("The return value nv1 after tx [ " + data.size() + " ]");    System.out.println("The time taken for executing client(Thread-1) "+ (System.currentTimeMillis()-start));    ConnectionManager.set(jedis); } } The result of the preceding program will vary as per machine configuration but by changing the thread sleep time and running the program couple of times, the result will be similar to the one shown as follows: The return code is [ 1 ] The return value of nv1 is [ null ] The return value nv1 after tx [ 300000 ] The time taken for executing client(Thread-1) 7078 Fire the FLUSHDB command every time you run the test. The idea is that the program should not pick up a value obtained because of a previous run of the program. The proof that the single command program is able to write to the key is if we see the following line: The return code is [1]. Let's analyze the result. In case of pipeline, a single command reads the value and the pipeline command sets a new value to that key as evident in the following result: The return value of nv1 is [ 3508 ] Now compare this with what happened in case of transaction when a single command tried to read the value but it was blocked because of the transaction. Hence the value will be NULL or 300,000. The return value of nv1 after tx [0] or The return value of nv1 after tx [300000] So the difference in output can be attributed to the fact that in a transaction, if we have started a MULTI command, and are still in the process of queueing commands (that is, we haven't given the server the EXEC request yet), then any other client can still come in and make a request, and the response would be sent to the other client. Once the client gives the EXEC command, then all other clients are blocked while all of the queued transaction commands are executed. Pipeline and transaction To have a better understanding, let's analyze what happened in case of pipeline. When two different connections made requests to the Redis for the same resource, we saw a result where client-2 picked up the value while client-1 was still executing: Pipeline in Redis in a multi connection environment What it tells us is that requests from the first connection which is pipeline command is stacked as one command in its execution stack, and the command from the other connection is kept in its own stack specific to that connection. The Redis execution thread time slices between these two executions stacks, and that is why client-2 was able to print a value when the client-1 was still executing. Let's analyze what happened in case of transaction here. Again the two commands (transaction commands and GET commands) were kept in their own execution stacks, but when the Redis execution thread gave time to the GET command, and it went to read the value, seeing the lock it was not allowed to read the value and was blocked. The Redis execution thread again went back to executing the transaction commands, and again it came back to GET command where it was again blocked. This process kept happening until the transaction command released the lock on the resource and then the GET command was able to get the value. If by any chance, the GET command was able to reach the resource before the transaction lock, it got a null value. Please bear in mind that Redis does not block execution to other clients while queuing transaction commands but blocks only during executing them. Transaction in Redis multi connection environment This exercise gave us an insight into what happens in the case of pipeline and transaction. Summary In this article, we saw in brief how to use Redis, not simply as a datastore, but also as pipeline the commands which is so much more like bulk processing. Apart from that, we covered areas such as transaction, messaging, and scripting. We also saw how to combine messaging and scripting, and create reliable messaging in Redis. This capability of Redis makes it different from some of the other datastore solutions. Resources for Article: Further resources on this subject: Implementing persistence in Redis (Intermediate) [article] Using Socket.IO and Express together [article] Exploring streams [article]