Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7019 Articles
article-image-getting-started-sorting-algorithms-java
Packt
16 Nov 2016
9 min read
Save for later

Getting Started with Sorting Algorithms in Java

Packt
16 Nov 2016
9 min read
In this article by Peter Verhas author of the book Java 9 Programming By Example, we will develop a simple sort program. Using this code as an example, we will look at different build tools, which are frequently used for Java projects, and learn the basic features of the Java language. (For more resources related to this topic, see here.) The problem we will solve The sorting problem is one of the oldest programming tasks that an engineer solves. We will have a set of records and we know that we will want to find a specific one sometime later, and we will want to find that one fast. To find it, we will sort the records in a specific order that helps finding the record we want fast. As an example, we can have the names of the students with some marks on cards. When students will come to the office asking for the result, we can turn all pages one after the other to find the name of the enquiring student. However, it is better if we sort the papers by the name of the students lexicographically. When a student comes, we can search the mark attached to the name much faster. We can look at the middle card; if it shows the name of the student, then we are happy to have found the name and the mark. If the card precedes the name of the student lexicographically, then we will continue searching in the second half, otherwise the first half. Following that approach, we can find the name of the student in no more steps than as many times the pack of cards can be halved. If we have two cards, then it is two steps at most. If it is four, then we will need three steps at most. If there are eight cards, then we may need four steps, but not more. If there are 1000 cards, then we may need at most 11 steps, while the original, non-sorted set will need 1000 steps, worst case. That is, approximately, it speeds up the search 100 times, so this is worth sorting the cards, unless the sorting itself takes too much time. In many cases, it is worth sorting the dataset and there are many sorting algorithms to do that. There are simpler and more complex algorithms, and as in many cases, more complex algorithms are the one that run faster. As we are focusing on the Java programming part and not the algorithm forging, in this article, we will develop a Java code that implements a simple and not-that-fast algorithm. Bubble sort The algorithm that we will implement in this article is well known as bubble sort. The approach is very simple. Begin at the start of the cards and compare the first and the second card. If the first card is later in lexicographic order than the second one, then swap the two cards. Then, repeat this for the card that is at the second place now, then the third, and so on. There is a card that is lexicographically the latest, say Wilson, and sometime later, we will get to this card as we go on swapping, the cards going from start to end. When we get this card and start to compare it with the next one, we will always swap them; this way, Wilson's card will travel to the last place where it has to be after the sort. All we have to do is repeat this travelling from the start and the occasional swapping of cards again, but this time only to the last but one element. This time, the second latest element will get to its place—say Wilkinson will be right before Wilson. If we have n cards, and we repeat this n-1 times, all cards will get to their place. Project structure and build tools When a project is more complex than a single class, and it usually is, then it is wise to define a project structure. We will have to decide where we store the source files, where the resource files (those that contain some resource for the program, but are not Java source) are, where should the .class files be written by the compiler, and so on. Generally, the structure is mainly the directory setup and configuring the tools that perform the build that use these tools. The compilation of complex programs cannot be feasibly done using the command line issuing javac commands. If we have a 100 Java source files, the compilation will require that many javac commands to be issued. We can write a simple bash script that does that. First, it will be just 100 lines, each compiling one source Java file to class file. Then, we will realize that this is only time, CPU, and power consuming to compile the files that are not changed since the last compilation. So, we can add some bash programming that checks the time stamp on the source and generated files. Then, we will probably realize that… whatever. At the end, we will end up with a tool that is essentially a build tool. And, this is already done. Instead of creating one, we will use a build tool that is ready. There are a few of them that can be found at https://en.wikipedia.org/wiki/List_of_build_automation_software Make The Make program was originally created in April 1976, so this is not a new tool. It is included in the Unix system so this tool is available without any extra installation on Linux, Mac OS X, or any other Unix-based system. Additionally, there are numerous ports of the tool on Windows and some version is/was included in the Visual C compiler toolset. The Make is not tied to Java. It was created when the major programming language was C, but it is not tied to C or any other language. Make is a dependency description language that has a very simple syntax. The Make, just like any other build tool, works controlled by a project description file. In case of make, this file contains a rule set. The description file is usually named Makefile, but in case the name of the description file is different, it can be specified as a command-line option to the make command. Rules in Makefile follow each other and a it is one or more lines. The first line starts at the first position (there is no tab or space at the start of the line) and the following lines start with a tab character. Thus, Makefile may look something like the following code: run : hello.jar java -cp hello.jar HelloWorld hello.jar : HelloWorld.class jar -cf hello.jar HelloWorld.class HelloWorld.class : HelloWorld.java javac HelloWorld.java The file defines three so-called targets: run, hello.jar, and HelloWorld.class. To create HelloWorld.class, type the following line at the Command Prompt: make HelloWorld.class The make will look at the rule and see that it depends on HelloWorld.java. If the HelloWorld.class file does not exist, or HelloWorld.java is newer than the Java source file, make will execute the command that is written on the next line and it will compile the Java source file. If the class file was created following the last modification of HelloWorld.java, then make knows that there is no need to run the command. In case of creating HelloWorld.class,the make program has an easy task. The source file was already there. If you issue the make hello.jar command, the procedure is more complex. The make command sees that in order to create hello.jar, it needs HelloWorld.class, which itself is also a target on another rule. Thus, it may need to be created. First, it starts the problem the same way as before. If HelloWorld.class is there, and is older than hello.jar, there is nothing to do. If it is not there, or is newer than hello.jar, then the jar -cf hello.jar HelloWorld.class command needs to be executed, but not yet. It remembers that this command has to be executed sometime in the future when all the commands that are needed to create HelloWorld.class are already executed successfully. Thus, it continues to create the class file exactly the same way as I already described earlier. In general, a rule can have the following format: target : dependencies command The make command can create any target using the make target command by first calculating which commands to execute and then executing them one by one. The commands are shell commands executing in a different process and may pose problems under Windows, which may render the Makefile files operating system dependent. Note that the run target is not an actual file that make creates. A target can be a file name or just a name for the target. In the latter case, make will never consider the readily available target. As we do not use make for Java project, there is no room to get into more details. Additionally, I cheated a bit by making the description of a rule simpler than it should be. The make tool has many powerful features out of the scope of this book. There are also several implementations that differ a little from each other. You will most probably meet the one made by the Free Software Foundation—the GNU make. And, of course, just in case of any Unix command-line tool, man is your friend. The man make command will display the documentation of the tool on the screen. The main points that you should remember about make are as follows: It defines the dependencies of the individual artifacts (targets) in a declarative way It defines the actions to create the missing artifacts in an imperative way. Summary In this article, we have developed a very basic sort algorithm. It was made purposefully simple so that we could reiterate on the basic and most important Java language elements, classes, packages, variables, methods, and so on. Resources for Article: Further resources on this subject: Algorithm Analysis [article] Introduction to C# and .NET [article] Parallel Computing [article]
Read more
  • 0
  • 0
  • 4613

article-image-encrypt-and-hash-passwords
Pedro Narciso
16 Nov 2016
7 min read
Save for later

Encrypt and hash passwords

Pedro Narciso
16 Nov 2016
7 min read
A couple of days ago, Dropbox posted an explanation on how they store your passwords. In this post, we are going to implement a library that encrypts and hashes a password in a similar way. Disclaimer: Security is a complex AND important topic. It is YOUR responsability to educate yourself on it. As you may have read on the Dropbox blog post, they: Get the hash product of the SHA512 function over the password. Apply the bcrypt function over the previous result. Encrypt the bcrypt product with a symmetric encryption algorithm (AES256). The above transformation can be written in pseudocode as: hashed_password_to_store = encrypt_aes256(bcrypt(sha512('my password'))) Remember: You should not keep the AES256 key in your database nor hardcoded in the application code. Okay, let's start with it! First, we are going to create a project: $ mkdir password-encryption-tool $ cd password-encryption-tool $ npm init // init our node.js project Edit index.js and write the skeleton of the function. 'use strict'; function hashAndEncryptPassword(input){ let sha512hash = sha512(input); let bcryptHash = bcryptHash(sha512hash); let encryptedPassword = encrypt(bcryptHash); return encryptedPassword; } Good! Of course, that function does not work yet. The functions sha512, bcryptHash and encrypt are not yet implemented, but it gives us a starting point. The #sha512(input) function recives a password as input and returns its SHA512 product. To implement that function, we are going to use the Hash class from the crypto module. Docs here. function sha512(input){ let hasher = crypto.createHash('SHA512'); hasher.update(input); return hasher.digest('base64'); //Returns the hash in base64 format } Done! The bcrypt function requires a third party library. There are many options but here are my personal recommendations – bcrypt.js and bcrypt. The former is a pure JavaScript implementation with 0 dependencies. The latter is a native one and therefore faster (2.7 times), which is what I use personally. But, for this example, we are going to use bcrypt.js. Why? Because if you do not have the build environment needed for the native version, I don't want you to stop following this tutorial and start pulling out your hair because there is some dependency not compiling on your machine. Anyway, your code will work with both of them because bcrypt.js API is compatible with bcrypt. Okay, that was a long explanation. Let's install bcryptjs: $ npm install bcryptjs --save This installs bcryptjs and updates our package.json file with the new dependency. Bcrypt implementations expose both synchronous and asynchronous APIs. Given that the bcrypt algorithm was made to be slow, I am going to use the asynchronous API. After all, you don't want to block the event loop, do you? This decision forces us to refactor our code a little bit. The asynchronous version of #hashAndEncryptPassword(input) looks like: //... const crypto = require('crypto'), bcrypt = require('bcryptjs'); const BCRYPT_SALT_ROUNDS=10; function hashAndEncryptPassword(key, input, callback){ let sha512hash ; try{ sha512hash = sha512(input); }catch(err){ return callback(err); } bcrypt.hash(sha512hash, BCRYPT_SALT_ROUNDS, function(err, result){ var encryptedHash; if(err){ return callback(err); } try{ encryptedHash = encrypt(key, result); }catch(err){ return callback(err); } callback( null, encryptedHash); }); } //... Note the function signature change with the inclusion of the callback parameter, plus the bcrypt integration. Also, the SHA512 is wrapped in a try/catch block. Any error happening here will be properly handled and returned to the callback function. Time to implement the encrypt function. Node.js Crypto API exposes the Cipher class. Its usage is similar to the Hash class. We just need to provide the algorithm, key and IV. The IV is used for randomization at the encryption stage; for that reason, it is very important that you use a new random IV for each different value you encrypt. Later, when we want to decrypt a value, you need to provide the same key and IV. Because they do not need to be secret, we are going to append them along the encrypted hash. IVs are just a bunch of random bytes and their generation is quite simple. const IV = new Buffer(crypto.randomBytes(16)); // A buffer with 16 random bytes And here is our encrypt function: ... const IV_LEN = 16, ALGORITHM = 'AES-256-CTR'; function encrypt (key, input) { const IV = new Buffer(crypto.randomBytes(IV_LEN)); let cipher = crypto.createCipheriv(ALGORITHM, key, IV); cipher.setEncoding('base64'); cipher.write(input); cipher.end(); const cipherText = cipher.read(); // IV is not a secret. We can store it along the password return cipherText + '$' + IV.toString('base64'); } ... The encrypt function accepts key and input as arguments, and: Generates an IV.. Creates a cipher using the AES-256-CTR algorithm, our key and the IV. Sets cipherText constant with the encrypted version of input. Returns the cipherText plus $ plus the IV. We are almost done. We now have a function that allows us to safely store a user password in our database. Now, as you've probably guessed, when we want to check if a given password is valid, we need to compare it with our stored password. But, you can just simple compare them. You need to apply some transformations in order to get comparable versions. A simple approach could be to just apply the same transformation to a plain password and then they will be comparable. AES256(bcrypt(SHA512(plain text))) === hashed-and-encrypted But we are going to make it slightly different. Here is our recipe: Get the SHA512 of the password we want to validate. Decrypt our stored password. Use bcrypt.compare to compare them. The SHA512 part was already implemented as the sha512 function. To decrypt the stored password we need to create a decipher with crypto.createDecipheriv with the previously used algorithm, key, and IV. function decrypt(key, input){ var result; let [cipherText, IV] = input.split('$'); let buffIV = new Buffer(IV, 'base64'); let decipher = crypto.createDecipheriv(ALGORITHM, key, buffIV); result = decipher.update(cipherText, 'base64', 'utf8'); result += decipher.final('utf8'); return result; } Now we can implement our compare function. function compare(key, clearPassword, encryptedPassword, callback){ var hash; try{ hash = decrypt(key, encryptedPassword); }catch(err){ return callback(err); } bcrypt.compare(sha512(clearPassword), hash, callback); } And do not forget to export compare and hashAndEncryptPassword functions. exports.hashAndEncryptPassword = hashAndEncryptPassword; exports.compare = compare; We can now use our module to hash and encrypt passwords: var passwordTool = require('.'); // Remember to not include your key in your code! // better load it from an environment variable var key = 'BKYHAT11zlXUiXE3iZfzSEWfvwjdbfPK'; var password = 'super secret'; passwordTool.hashAndEncryptPassword(key, password, function(err, hash){ if(err){ throw err; } console.log('Password ready to store', hash); }); // prints: Password ready to store ZGmzcPy29oYjZj5+P/wg4nS0Bs64... And to match a user provided password with our stored copy: passwordTool.compare(key, password, storedPassword, function (err, result){ if(err){ throw err; } console.log('match?', result); }); You can find the complete source code of this article at https://github.com/revington/password-tool. About the author Pedro Narciso García Revington is a senior full-stack developer with 10+ years of experience in high scalability and availability, microservices, automated deployments, data processing, CI, (T,B,D)DD, and polyglot persistence. He is a self-taught, highly motivated problem solver with a focus on delivering quick and elegant solutions.
Read more
  • 0
  • 0
  • 3147

article-image-setting-development-environment-android-wear-applications
Packt
16 Nov 2016
8 min read
Save for later

Setting up Development Environment for Android Wear Applications

Packt
16 Nov 2016
8 min read
"Give me six hours to chop down a tree and I will spend the first four sharpening the axe." -Abraham Lincoln In this article by Siddique Hameed, author of the book, Mastering Android Wear Application Development, they have discussed the steps, topics and process involved in setting up the development environment using Android Studio. If you have been doing Android application development using Android Studio, some of the items discussed here might already be familiar to you. However, there are some Android Wear platform specific items that may be of interest to you. (For more resources related to this topic, see here.) Android Studio Android Studio Integrated Development Environment (IDE) is based on IntelliJ IDEA platform. If you had done Java development using IntelliJ IDEA platform, you'll be feeling at home working with Android Studio IDE. Android Studio platform comes bundled with all the necessary tools and libraries needed for Android application development. If this is the first time you are setting up Android Studio on your development system, please make sure that you have satisfied all the requirements before installation. Please refer developer site (http://developer.android.com/sdk/index.html#Requirements) for checking on the items needed for the operating system of your choice. Please note that you need at least JDK version 7 installed on your machine for Android Studio to work. You can verify your JDK's version that by typing following commands shown in the following terminal window: If your system does not meet that requirement, please upgrade it using the method that is specific to your operating system. Installation Android Studio platform includes Android Studio IDE, SDK Tools, Google API Libraries and systems images needed for Android application development. Visit the http://developer.android.com/sdk/index.html URL for downloading Android Studio for your corresponding operating system and following the installation instruction. Git and GitHub Git is a distributed version control system that is used widely for open-source projects. We'll be using Git for sample code and sample projects as we go along the way. Please make sure that you have Git installed on your system by typing the command as shown in the following a terminal window. If you don't have it installed, please download and install it using this link for your corresponding operating system by visting https://git-scm.com/downloads. If you are working on Apple's Macintosh OS like El Capitan or Yosemite or Linux distributions like Ubuntu, Kubuntu or Mint, chances are you already have Git installed on it. GitHub (http://github.com) is a free and popular hosting service for Git based open-source projects. They make checking out and contributing to open-source projects easier than ever. Sign up with GitHub for a free account if you don't have an account already. We don't need to be an expert on Git for doing Android application development. But, we do need to be familiar with basic usages of Git commands for working with the project. Android Studio comes by default with Git and GitHub integration. It helps importing sample code from Google's GitHub repository and helps you learn by checking out various application code samples. Gradle Android application development uses Gradle (http://gradle.org/)as the build system. It is used to build, test, run and package the apps for running and testing Android applications. Gradle is declarative and uses convention over configuration for build settings and configurations. It manages all the library dependencies for compiling and building the code artifacts. Fortunately, Android Studio abstracts most of the common Gradle tasks and operations needed for development. However, there may be some cases where having some extra knowledge on Gradle would be very helpful. We won't be digging into Gradle now, we'll be discussing about it as and when needed during the course of our journey. Android SDK packages When you install Android Studio, it doesn't include all the Android SDK packages that are needed for development. The Android SDK separates tools, platforms and other components & libraries into packages that can be downloaded, as needed using the Android SDK Manager. Before we start creating an application, we need to add some required packages into the Android SDK. Launch SDK Manager from Android Studio Tools | Android | SDK Manager. Let's quickly go over a few items from what's displayed in the preceding screenshot. As you can see, the Android SDK's location is /opt/android-sdk on my machine, it may very well be different on your machine depending on what you selected during Android Studio installation setup. The important point to note is that the Android SDK is installed on a different location than the Android Studio's path (/Applications/Android Studio.app/). This is considered a good practice because the Android SDK installation can be unaffected depending on a new installation or upgrade of Android Studio or vice versa. On the SDK Platforms tab, select some recent Android SDK versions like Android 6.0, 5.1.1, and 5.0.1. Depending on the Android versions you are planning on supporting in your wearable apps, you can select other older Android versions. Checking on Show Package Details option on the bottom right, the SDK Manager will show all the packages that will be installed for a given Android SDK version. To be on the safer side, select all the packages. As you may have noticed already Android Wear ARM and Intel system images are included in the package selection. Now when you click on SDK Tools tab, please make sure the following items are selected: Android SDK Build Tools Android SDK Tools 24.4.1 (Latest version) Android SDK Platform-Tools Android Support Repository, rev 25 (Latest version) Android Support Library, rev 23.1.1 (Latest version) Google Play services, rev 29 (Latest version) Google Repository, rev 24 (Latest version) Intel X86 Emulator Accelerator (HAXM installer), rev 6.0.1 (Latest version) Documentation for Android SDK (Optional) Please do not change anything on SDK Update Sites. Keep the update sites as it was configured by default. Clicking on OK will take some time downloading and installing all the components and packages selected. Android Virtual Device Android Virtual Devices will enable us to test the code using Android Emulators. It lets us pick and choose various Android system target versions and form factors needed for testing. Launch Android Virtual Device Manager from Tools | Android | AVD Manager From Android Virtual Device Manager window, click on Create New Virtual Device button on the bottom left and proceed to the next screen and select Wear category Select Marshmallow API Level 23 on x86 and everything else as default, as shown in the following screenshot: Note that the current latest Android version is Marshmallow of API level 23 at the time of this writing. It may or may not be the latest version while you are reading this article. Feel free to select the latest version that is available during that time. Also, if you'd like to support or test in earlier Android versions, please feel free to do so in that screen. After the virtual device is selected successfully, you should see that listed on the Android Virtual Devices list as show in the following screenshot: Although it's not required to use real Android Wear device during development, sometimes it may be convenient and faster developing it in a real physical device. Let's build a skeleton App Since we have all the components and configurations needed for building wearable app, let's build a skeleton app and test out what we have so far. From Android Studio's Quick Start menu, click on Import an Android code sample tab: Select the Skeleton Wearable App from Wearable category as shown in following screenshot: Click Next and select your preferred project location. As you can see the skeleton project is cloned from Google's sample code repository from GitHub. Clicking on Finish button will pull the source code and Android Studio will compile and build the code and get it ready for execution. The following screenshot indicates that the Gradle build finished successfully without any errors. Click on Run configuration to run the app: When the app starts running, Android Studio will prompt us to select the deployment targets, we can select the emulator we created earlier and click OK. After the code compiles and uploaded to the emulator, the main activity of the Skeleton App will be launched as shown below: Clicking on SHOW NOTIFICATION will show the notification as below: Clicking on START TIMER will start the timer and run for five seconds and clicking on Finish Activity will close the activity take the emulator to the home screen. Summary We discussed the process involved in setting up the Android Studio development environment by covering the installation instruction, requirements and SDK tools, packages and other components needed for Android Wear development. We also checked out source code for Skeleton Wearable App from Google's sample code repository and successfully ran and tested it on Android device emulator. Resources for Article: Further resources on this subject: Building your first Android Wear Application [Article] Getting started with Android Development [Article] The Art of Android Development Using Android Studio [Article]
Read more
  • 0
  • 0
  • 7841

article-image-multithreading-qt
Packt
16 Nov 2016
13 min read
Save for later

Multithreading with Qt

Packt
16 Nov 2016
13 min read
Qt has its own cross-platform implementation of threading. In this article by Guillaume Lazar and Robin Penea, authors of the book Mastering Qt 5, we will study how to use Qt and the available tools provided by the Qt folks. (For more resources related to this topic, see here.) More specifically, we will cover the following: Understanding the QThread framework in depth The worker model and how you can offload a process from the main thread An overview of all the available threading technologies in Qt Discovering QThread Qt provides a sophisticated threading system. We assume that you already know threading basics and the associated issues (deadlocks, threads synchronization, resource sharing, and so on) and we will focus on how Qt implements it. The QThread is the central class for of the Qt threading system. A QThread instance manages one thread of execution within the program. You can subclass QThread to override the run() function, which will be executed in the QThread class. Here is how you can create and start a QThread: QThread thread; thread.start(); The start() function calling will automatically call the run() function of thread and emit the started() signal. Only at this point, the new thread of execution will be created. When run() is completed, thread will emit the finished() signal. This brings us to a fundamental aspect of QThread: it works seamlessly with the signal/slot mechanism. Qt is an event-driven framework, where a main event loop (or the GUI loop) processes events (user input, graphical, and so on) to refresh the UI. Each QThread comes with its own event loop that can process events outside the main loop. If not overridden, run() calls the QThread::exec() function, which starts the thread's event loop. You can also override QThread and call exec(), as follows: class Thread : public QThread { Q_OBJECT protected: void run() { Object* myObject = new Object(); connect(myObject, &Object::started, this, &Thread::doWork); exec(); } private slots: void doWork(); }; The started()signal will be processed by the Thread event loop only upon the exec() call. It will block and wait until QThread::exit() is called. A crucial thing to note is that a thread event loop delivers events for all QObject classes that are living in that thread. This includes all objects created in that thread or moved to that thread. This is referred to as the thread affinity of an object. Here's an example: class Thread : public QThread { Thread() : mObject(new QObject()) { } private : QObject* myObject; }; // Somewhere in MainWindow Thread thread; thread.start(); In this snippet, myObject is constructed in the Thread constructor, which is created in turn in MainWindow. At this point, thread is living in the GUI thread. Hence, myObject is also living in the GUI thread. An object created before a QCoreApplication object has no thread affinity. As a consequence, no event will be dispatched to it. It is great to be able to handle signals and slots in our own QThread, but how can we control signals across multiple threads? A classic example is a long running process that is executed in a separate thread that has to notify the UI to update some state: class Thread : public QThread { Q_OBJECT void run() { // long running operation emit result("I <3 threads"); } signals: void result(QString data); }; // Somewhere in MainWindow Thread* thread = new Thread(this); connect(thread, &Thread::result, this, &MainWindow::handleResult); connect(thread, &Thread::finished, thread, &QObject::deleteLater); thread->start(); Intuitively, we assume that the first connect function sends the signal across multiple threads (to have a result available in MainWindow::handleResult), whereas the second connect function should work on thread's event loop only. Fortunately, this is the case due to a default argument in the connect() function signature: the connection type. Let's see the complete signature: QObject::connect( const QObject *sender, const char *signal, const QObject *receiver, const char *method, Qt::ConnectionType type = Qt::AutoConnection) The type variable takes Qt::AutoConnection as a default value. Let's review the possible values of Qt::ConectionType enum as the official Qt documentation states: Qt::AutoConnection: If the receiver lives in the thread that emits the signal, Qt::DirectConnection is used. Otherwise, Qt::QueuedConnection is used. The connection type is determined when the signal is emitted. Qt::DirectConnection: This slot is invoked immediately when the signal is emitted. The slot is executed in the signaling thread. Qt::QueuedConnection: The slot is invoked when control returns to the event loop of the receiver's thread. The slot is executed in the receiver's thread. Qt::BlockingQueuedConnection: This is the same as Qt::QueuedConnection, except that the signaling thread blocks until the slot returns. This connection must not be used if the receiver lives in the signaling thread or else the application will deadlock. Qt::UniqueConnection: This is a flag that can be combined with any one of the preceding connection types, using a bitwise OR element. When Qt::UniqueConnection is set, QObject::connect() will fail if the connection already exists (that is, if the same signal is already connected to the same slot for the same pair of objects). When using Qt::AutoConnection, the final ConnectionType is resolved only when the signal is effectively emitted. If you look again at our example, the first connect(): connect(thread, &Thread::result, this, &MainWindow::handleResult); When the result() signal will be emitted, Qt will look at the handleResult() thread affinity, which is different from the thread affinity of the result() signal. The thread object is living in MainWindow (remember that it has been created in MainWindow), but the result() signal has been emitted in the run() function, which is running in a different thread of execution. As a result, a Qt::QueuedConnection function will be used. We will now take a look at the second connect(): connect(thread, &Thread::finished, thread, &QObject::deleteLater); Here, deleteLater() and finished() live in the same thread, therefore, a Qt::DirectConnection will be used. It is crucial that you understand that Qt does not care about the emitting object thread affinity, it looks only at the signal's "context of execution." Loaded with this knowledge, we can take another look at our first QThread example to have a complete understanding of this system: class Thread : public QThread { Q_OBJECT protected: void run() { Object* myObject = new Object(); connect(myObject, &Object::started, this, &Thread::doWork); exec(); } private slots: void doWork(); }; When Object::started() is emitted, a Qt::QueuedConnection function will be used. his is where your brain freezes. The Thread::doWork() function lives in another thread than Object::started(), which has been created in run(). If the Thread has been instantiated in the UI Thread, this is where doWork() would have belonged. This system is powerful but complex. To make things more simple, Qt favors the worker model. It splits the threading plumbing from the real processing. Here is an example: class Worker : public QObject { Q_OBJECT public slots: void doWork() { emit result("workers are the best"); } signals: void result(QString data); }; // Somewhere in MainWindow QThread* thread = new Thread(this); Worker* worker = new Worker(); worker->moveToThread(thread); connect(thread, &QThread::finished, worker, &QObject::deleteLater); connect(this, &MainWindow::startWork, worker, &Worker::doWork); connect(worker, &Worker::resultReady, this, handleResult); thread->start(); // later on, to stop the thread thread->quit(); thread->wait(); We start by creating a Worker class that has the following: A doWork()slot that will have the content of our old QThread::run() function A result()signal that will emit the resulting data Next, in MainWindow, we create a simple thread and an instance of Worker. The worker->moveToThread(thread) function is where the magic happens. It changes the affinity of the worker object. The worker now lives in the thread object. You can only push an object from your current thread to another thread. Conversely, you cannot pull an object that lives in another thread. You cannot change the thread affinity of an object if the object does not live in your thread. Once thread->start() is executed, we cannot call worker->moveToThread(this) unless we are doing it from this new thread. After that, we will use three connect() functions: We handle worker life cycle by reaping it when the thread is finished. This signal will use a Qt::DirectConnection function. We start the Worker::doWork() upon a possible UI event. This signal will use a Qt::QueuedConnection. We process the resulting data in the UI thread with handleResult(). This signal will use a Qt::QueuedConnection. To sum up, QThread can be either subclassed or used in conjunction with a worker class. Generally, the worker approach is favored because it separates more cleanly the threading affinity plumbing from the actual operation you want to execute in parallel. Flying over Qt multithreading technologies Built upon QThread, several threading technologies are available in Qt. First, to synchronize threads, the usual approach is to use a mutual exclusion (mutex) for a given resource. Qt provides it by the mean of the QMutex class. Its usage is straightforward: QMutex mutex; int number = 1; mutex.lock(); number *= 2; mutex.unlock(); From the mutex.lock() instruction, any other thread trying to lock the mutex object will wait until mutex.unlock() has been called. The locking/unlocking mechanism is error prone in complex code. You can easily forget to unlock a mutex in a specific exit condition, causing a deadlock. To simplify this situation, Qt provides a QMutexLocker that should be used where the QMutex needs to be locked: QMutex mutex; QMutexLocker locker(&mutex); int number = 1; number *= 2; if (overlyComplicatedCondition) { return; } else if (notSoSimple) { return; } The mutex is locked when the locker object is created, and it will be unlocked when locker is destroyed, for example, when it goes out of scope. This is the case for every condition we stated where the return statement appears. It makes the code simpler and more readable. If you need to create and destroy threads frequently, managing QThread instances by hand can become cumbersome. For this, you can use the QThreadPool class, which manages a pool of reusable QThreads. To execute code within threads managed by a QThreadPool, you will use a pattern very close to the worker we covered earlier. The main difference is that the processing class has to extend the QRunnable class. Here is how it looks: class Job : public QRunnable { void run() { // long running operation } } Job* job = new Job(); QThreadPool::globalInstance()->start(job); Just override the run() function and ask QThreadPool to execute your job in a separate thread. The QThreadPool::globalInstance() function is a static helper function that gives you access to an application global instance. You can create your own QThreadPool class if you need to have a finer control over the QThreadPool life cycle. Note that QThreadPool::start() takes the ownership of the job object and will automatically delete it when run() finishes. Watch out, this does not change the thread affinity like QObject::moveToThread() does with workers! A QRunnable class cannot be reused, it has to be a freshly baked instance. If you fire up several jobs, QThreadPool automatically allocates the ideal number of threads based on the core count of your CPU. The maximum number of threads that the QThreadPool class can start can be retrieved with QThreadPool::maxThreadCount(). If you need to manage threads by hand, but you want to base it on the number of cores of your CPU, you can use the handy static function, QThreadPool::idealThreadCount(). Another approach to multithreaded development is available with the Qt Concurrent framework. It is a higher level API that avoids the use of mutexes/locks/wait conditions and promotes the distribution of the processing among CPU cores. Qt Concurrent relies of the QFuture class to execute a function and expect a result later on: void longRunningFunction(); QFuture<void> future = QtConcurrent::run(longRunningFunction); The longRunningFunction() will be executed in a separated thread obtained from the default QThreadPool class. To pass parameters to a QFuture class and retrieve the result of the operation, use the following code: QImage processGrayscale(QImage& image); QImage lenna; QFuture<QImage> future = QtConcurrent::run(processGrayscale, lenna); QImage grayscaleLenna = future.result(); Here, we pass lenna as a parameter to the processGrayscale() function. Because we want a QImage as a result, we declare QFuture with the template type QImage. After that, future.result() blocks the current thread and waits for the operation to be completed to return the final QImage template type. To avoid blocking, QFutureWatcher comes to the rescue: QFutureWatcher<QImage> watcher; connect(&watcher, &QFutureWatcher::finished, this, &QObject::handleGrayscale); QImage processGrayscale(QImage& image); QImage lenna; QFuture<QImage> future = QtConcurrent::run(processImage, lenna); watcher.setFuture(future); We start by declaring a QFutureWatcher with the template argument matching the one used for QFuture. Then, simply connect the QFutureWatcher::finished signal to the slot you want to be called when the operation has been completed. The last step is to the tell the watcher to watch the future object with watcher.setFuture(future). This statement looks almost like it's coming from a science fiction movie. Qt Concurrent also provides a MapReduce and FilterReduce implementation. MapReduce is a programming model that basically does two things: Map or distribute the processing of datasets among multiple cores of the CPU Reduce or aggregate the results to provide it to the caller check styleThis technique has been first promoted by Google to be able to process huge datasets within a cluster of CPU. Here is an example of a simple Map operation: QList images = ...; QImage processGrayscale(QImage& image); QFuture<void> future = QtConcurrent::mapped( images, processGrayscale); Instead of QtConcurrent::run(), we use the mapped function that takes a list and the function to apply to each element in a different thread each time. The images list is modified in place, so there is no need to declare QFuture with a template type. The operation can be made a blocking operation using QtConcurrent::blockingMapped() instead of QtConcurrent::mapped(). Finally, a MapReduce operation looks like this: QList images = ...; QImage processGrayscale(QImage& image); void combineImage(QImage& finalImage, const QImage& inputImage); QFuture<void> future = QtConcurrent::mappedReduced( images, processGrayscale, combineImage); Here, we added a combineImage() that will be called for each result returned by the map function, processGrayscale(). It will merge the intermediate data, inputImage, into the finalImage. This function is called only once at a time per thread, so there is no need to use a mutex object to lock the result variable. The FilterReduce reduce follows exactly the same pattern, the filter function simply allows to filter the input list instead of transforming it. Summary In this article, we discovered how a QThread works and you learned how to efficiently use tools provided by Qt to create a powerful multi-threaded application. Resources for Article: Further resources on this subject: QT Style Sheets [article] GUI Components in Qt 5 [article] DOM and QTP [article]
Read more
  • 0
  • 1
  • 42447

article-image-css-animation-animatecss
Jabran Rafique
15 Nov 2016
6 min read
Save for later

CSS animation with Animate.css

Jabran Rafique
15 Nov 2016
6 min read
CSS animation is one of the best and easiest ways to make a web page interactive. These animations were made available as part of CSS3 specifications. Animate.css is one of the best libraries available for CSS animations, with a collection of more than 50 types of animations. It was built and maintained by designer Daniel Eden and many other contributors. Animate.css is an open source library and available at GitHub under MIT License. Installation There are a few ways to install the library: Download directly from Github Use directly from a Content Delivery Network (CDN) Install using Bower $ bower install animate.css --save Install using npm $ npm install animate.css --save Usage Now that we have Animate.css installed, using one of the above methods, we can start using it straight away in our app. For simplicity we will use the CDN URL to consume the Animate.css in our tutorial app. Here is a basic structure of our index.html: <!DOCTYPE html> <html> <head> <title>Learning Animate.css</title> <link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/animate.css/3.5.2/animate.min.css"> </head> <body> <div class="container"> <h1>Learning Animate.css</h1> <p>Animate this text!</p> </div> </body> </html> We will use Animate.css effects on paragraph tag in this HTML document. Animate.css has a default class, .animated, that triggers the animation. Other animation effects are used in addition to this default class. For example, let's apply a bounce animation to our paragraph text: ... <p class="animated bounce">Animate this text!</p> ... Here is the demo for this: See the Pen Learning Animate.css – Demo by Jabran Rafique (@jabranr) on CodePen. Similarly, we can replace the bounce class with any other class from Animate.css and it will get the specific animation. The animations will run only once; they will run again only after we reload the browser. To make it run infinitely add the .infinite class so that it becomes: ... <p class="animated infinite bounce">This text will animate infinite times.</p> ... Here is the demo for this: See the Pen Learning Animate.css – Demo (Infinite animation) by Jabran Rafique (@jabranr) on CodePen. Animate.css works best when used dynamically as it sets more control over animations. To use it dynamically, we will make use of JavaScript. Let's try that with the following basic example: First we will add another class to the element, so we can manipulate the element via JavaScript. ... <p class="js-animate">Animate this text dynamically!</p> ... Now we can manipulate this DOM element using JavaScript to add/remove Animate.css animations when clicked on text. var $elementToAnimate = document.querySelector('.js-animate'); $elementToAnimate.addEventListener('click', function(e) { this.classList.toggle('animated'); if (this.classList.contains('bounce')) { this.classList.remove('bounce'); } else { this.classList.add('bounce'); } }, false); Here is the demo for this: See the Pen Learning Animate.css – Demo (Dynamic animations) – part 3 by Jabran Rafique (@jabranr) on CodePen. Similarly, you can also use a DOM manipulation library such as jQuery to apply Animate.css classes. In fact, the official documentation provides a short helper method that registers itself with jQuery as a plugin and can easily be used by any element in the DOM. Here is the code for this: $.fn.extend({ animateCss: function (animationName) { var animationEnd = 'webkitAnimationEnd mozAnimationEnd MSAnimationEnd oanimationend animationend'; this.addClass('animated ' + animationName).one(animationEnd, function() { $(this).removeClass('animated ' + animationName); }); } }); Now we can apply Animate.css animation to any element as follows: $('.js-animate').animateCss('bounce'); Customize Animate.css Using more than 50 different animations in a single project may not be an ideal scenario. Also, if we are looking at the performance side, then adding the complete library in order to use one or few animations may not be useful. Therefore, it is best to customize the library and use only those parts that are needed by the project. Animate.css comes with a built-in Gulp workflow that makes it easier to customize the library to our needs. In order to use the Gulp workflow, the following must be installed already: Node Gulp npm install -g gulp To customize the Animate.css, simply open the animate-config.json file and comment out the animations that are NOT required by the project. Then run the gulp command in terminal to generate a customized CSS file. The terminal window will inform on how many animations were activated into generated customized CSS file. Customizing the Animate.css not only boosts performance, but also keeps the code clean by including only parts that are actually being used in the project. Sass version Sass is a CSS preprocessor language that lets us use variables and methods to reuse code inside our project. Sass files are ultimately converted into CSS files before they are used by any web platform. A Sass-only version is under development. The in-development source code can be found under sass branch of the project at GitHub. Once released, this will provide more robust control for customized and personalized animations by relying less on JavaScript, that is Gulp workflow. Contribute Contributions to any open source project make it robust and more useful with bugfixes and new features. Just like any other open source project, Animate.css also welcomes contributions. To contribute to Animate.css project just head over to GitHub project and fork the repository to work on an existing issue, or by adding a new one. All of the demos in this tutorial can be found at CodePen as Collection. Hope this quick get started guide will make it easier to use the Animate.css library in your next project to create awesome animated effects. About the author Jabran Rafique is a London-based web engineer. He currently works as a Front-End Web Developer at Rated People. He has a Masters in Computer Science from Staffordshire University and more than 6 years of professional experience in web systems. He had also served as Regional Lead and Advocate at Google Map Maker in 2008, where he contributed to building digital maps in order to make them available to millions of people worldwide. He has organized and spoken at several international events as well. He writes on his website/blog about different things, and shares code at GitHub and thoughts on Twitter.
Read more
  • 0
  • 0
  • 11971

article-image-manual-and-automated-testing
Packt
15 Nov 2016
10 min read
Save for later

Manual and Automated Testing

Packt
15 Nov 2016
10 min read
In this article by Claus Führer the author of the book Scientific Computing with Python 3, we focus on two aspects of testing for scientific programming: Manual and Automated testing. Manual testing is what is done by every programmer to quickly check that an implementation is working. Automated testing is the refined, automated variant of that idea. We will introduce some tools available for automatic testing in general, with a view on the particular case of scientific computing. (For more resources related to this topic, see here.) Manual Testing During the development of code you do a lot of small tests in order to test its functionality. This could be called Manual Testing. Typically, you would test that a given function does what it is supposed to do, by manually testing the function in an interactive environment. For instance, suppose that you implement the Bisection algorithm. It is an algorithm that finds a zero (root) of a scalar nonlinear function. To start the algorithm an interval has to be given with the property, that the function takes different signs on the interval boundaries. You would then test an implementation of that algorithm typically by checking: That a solution is found when the function has opposite signs at the interval boundaries that an exception is raised when the function has the same sign at the interval boundaries Manual testing, as necessary as may seem to be, is unsatisfactory. Once you convinced yourself that the code does what it is supposed to do, you formulate a relatively small number of demonstration examples to convince others of the quality of the code. At that stage one often loses interest in the tests made during development and they are forgotten or even deleted. As soon as you change a detail and things no longer work correctly you might regret that your earlier tests are no longer available. Automatic Testing The correct way to develop any piece of code is to use automatic testing. The advantages are The automated repetition of a large number of tests after every code refactoring and before new versions are launched A silent documentation of the use of the code A documentation of the test coverage of your code: Did things work before a change or was a certain aspect never tested? We suggest to develop tests in parallel to the code. Good design of tests is an art of its own and there is rarely an investment which guarantees such a good pay-off in development time savings as the investment in good tests. Now we will go through the implementation of a simple algorithm with the automated testing methods in mind. Testing the bisection algorithm Let us examine automated testing for the bisection algorithm. With this algorithm a zero of a real valued function is found. An implementation of the algorithm can have the following form: def bisect(f,a,b,tol=1.e-8): """ Implementation of the bisection algorithm f real valued function a,b interval boundaries (float) with the property f(a)*f(b)<=0 tol tolerance ( float ) """ if f(a)*f(b)>0: raise ValueError ("Incorrect initial interval [a,b]") for i in range (100): c = (a + b)/2 . if f (a)*f(c) <= 0: b=c else: a=c if abs (a - b)<tol: return (a + b)/2 raise Exception (’ No root found within the given tolerance { }’.format (tol) We assume this to be stored in a file bisection.py. As a first test case we test that the zero of the function F(x) = x is found: def test_identity(): result = bisect(lambda x: x, -1., 1.) #(for lambda) expected = 0. assert allclose(result, expected),’expected zero not found’ text_identity() In this code you meet the Python keyword assert for the first time. It raises an exception AssertionError if its first argument returns the value False. Its optional second argument is a string with additional information. We use the function allclose in order to test for equality for float. Let us comment on some of the features of the test function. We use an assertion to make sure that an exception will be raised if the code does not behave as expected. We have to manually run the test in the line test_identity(). There are many tools to automate this kind of call. Let us now setup a test that checks if bisect raises an exception when the function has the same sign on both ends of the interval. For now, we will suppose that the exception raised is a ValueError exception. Example: Checking the sign for the bisection algorithm. def test_badinput(): try: bisect(lambda x: x,0.5,1) except ValueError: pass else: raise AssertionError() test_badinput() In this case an AssertionError is raised if the exception is not of type ValueError. There are tools to simplify the above construction to check that an exception is raised. Another useful kind of tests is the edge case test. Here we test arguments or user input which is likely to create mathematically undefined situations or states of the program not foreseen by the programmer. For instance, what happens if both bounds are equal? What happens if a>b? We easily setup up such a test by using for instance def test_equal_boundaries(): result = bisect(lambda x: x, 1., 1.) expected = 0. assert allclose(result, expected), ‘test equal interval bounds failed’ def test_reverse_boundaries(): result = bisect(lambda x: x, 1., -1.) expected = 0. assert allclose(result, expected), ‘test reverse interval bounds failed’ test_equal_boundaries() test_reverse_boundaries() Using unittest The standard Python package unittest greatly facilitates automated testing. That package requires that we rewrite our tests a little to be compatible. The first test would have to be rewritten in a class, as follows: from bisection import bisect import unittest class TestIdentity(unittest.TestCase): def test(self): result = bisect(lambda x: x, -1.2, 1.,tol=1.e-8) expected = 0. self.assertAlmostEqual(result, expected) if __name__==‘__main__’: unittest.main() Let us examine the differences to the previous implementation. First, the test is now a method and a part of a class. The class must inherit from unittest,TestCase. The test method’s name must start with test. Note that we may now use one of the assertion tools of the package, namely       . Finally, the tests are run using unittest.main. We recommend to write the tests in a file separate from the code to be tested. That’s why it starts with an import. The test passes and returns Ran 1 test in 0.002s OK If we would have run it with a loose tolerance parameter, e.g., 1.e-3, a failure of the test would have been reported: F ========================================================== FAIL: test (__main__.TestIdentity) ---------------------------------------------------------------------- Traceback (most recent call last): File “<ipython-input-11-e44778304d6f>“, line 5, in test self.assertAlmostEqual(result, expected) AssertionError: 0.00017089843750002018 != 0.0 within 7 places --------------------------------------------------------------------- Ran 1 test in 0.004s FAILED (failures=1) Tests can and should be grouped together as methods of a test class: Example: import unittest from bisection import bisect class TestIdentity(unittest.TestCase): def identity_fcn(self,x): return x def test_functionality(self): result = bisect(self.identity_fcn, -1.2, 1.,tol=1.e-8) expected = 0. self.assertAlmostEqual(result, expected) def test_reverse_boundaries(self): result = bisect(self.identity_fcn, 1., -1.) expected = 0. self.assertAlmostEqual(result, expected) def test_exceeded_tolerance(self): tol=1.e-80 self.assertRaises(Exception, bisect, self.identity_fcn, -1.2, 1.,tol) if __name__==‘__main__’: unittest.main() Here, the last test needs some comments: We used the method unittest.TestCase.assertRaises. It tests whether an exception is correctly raised. Its first parameter is the exception type, for example,ValueError, Exception, and its second argument is a the name of the function, which is expected to raise the exception. The remaining arguments are the arguments for this function. The command unittest.main() creates an instance of the class TestIdentity and executes those methods starting by test. Test setUp and tearDown The class unittest.TestCase provides two special methods, setUp and tearDown, which are run before and after every call to a test method. This is needed when testing generators, which are exhausted after every test. We demonstrate this here by testing a program which checks in which line in a file a given string occurs for the first time: class NotFoundError(Exception): pass def find_string(file, string): for i,lines in enumerate(file.readlines()): if string in lines: return i raise NotFoundError(‘String {} not found in File {}‘. format(string,file.name)) We assume, that this code is saved in a file find_string.py. A test has to prepare a file and open it and remove it after the test: import unittest import os # used for, e.g., deleting files from find_in_file import find_string, NotFoundError class TestFindInFile(unittest.TestCase): def setUp(self): file = open(‘test_file.txt’, ‘w’) file.write(‘aha’) file.close() self.file = open(‘test_file.txt’, ‘r’) def tearDown(self): os.remove(self.file.name) def test_exists(self): line_no=find_string(self.file, ‘aha’) self.assertEqual(line_no, 0) def test_not_exists(self): self.assertRaises(NotFoundError, find_string,self.file, ‘bha’) if __name__==‘__main__’: unittest.main() Before each test setUp is run and afterwards tearDown is executed. Parametrizing Tests One frequently wants to repeat the same test set-up with different data sets. When using the functionalities of unittests this requires to automatically generate test cases with the corresponding methods injected: To this end we first construct a test case with one or several methods that will be used, when we later set up test methods. Let us consider the bisection method again and let us check if the values it returns are really zeros of the given function. We first build the test case and the method which will use for the tests: class Tests(unittest.TestCase): def checkifzero(self,fcn_with_zero,interval): result = bisect(fcn_with_zero,*interval,tol=1.e-8) function_value=fcn_with_zero(result) expected=0. self.assertAlmostEqual(function_value, expected) Then we dynamically create test functions as attributes of this class: test_data=[‘name’:’identity’, ‘function’:lambda x: x, ‘interval’:[-1.2, 1.], ‘name’:’parabola’, ‘function’:lambda x: x**2-1, ’interval’:[0, 10.], ‘name’:’cubic’, ‘function’:lambda x: x**3-2*x** 2,‘interval’:[0.1, 5.],] def make_test_function(dic): return lambda self:self.checkifzero(dic[‘function’],dic [‘interval’]) for data in test_data: setattr(Tests, “test_name”.format(name=data[‘name’]), make_test_function(data)) if __name__==‘__main__’: unittest.main() In this example the data is provided as a list of dictionaries. A function make_test_function dynamically generates a test function which uses a particular data dictionary to perform the test with the previously defined method checkifzero. This test function is made a method of the TestCase class by using the Python command settattr. Summary No program development without testing! In this article we showed the importance of well organized and documented tests. Some professionals even start development by first specifying tests. A useful tool for automatic testing is unittest, which we explained in detail. While testing improves the reliability of a code, profiling is needed to improve the performance. Alternative ways to code may result in large performance differences. We showed how to measure computation time and how to localize bottlenecks in your code. Resources for Article: Further resources on this subject: Python Data Analysis Utilities [article] Machine Learning with R [article] Storage Scalability [article]
Read more
  • 0
  • 0
  • 2277
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-vector-representation-words
Janu Verma
15 Nov 2016
6 min read
Save for later

Vector Representation of Words

Janu Verma
15 Nov 2016
6 min read
In natural language processing (NLP) tasks, the first step is to represent a document as an element in a vector space. Standard machine learning and data mining algorithms expect a data instance as a vector; in fact, when we say data, we mean a matrix (a row/vector for each data point). There are various ways to express a textual document as a vector, depending on the problem and the assumptions of the model. In traditional Vector Space Models (VSMs), a word is represented as a vector of dimension equal to the size of vocabulary, with each word in the vocabulary corresponding to an entry in the vector. For example, if the text is "Friends work and play together", then our vocabulary has 5 words. We can represent the words as: Friends = [1,0,0,0,0] Work = [0,1,0,0,0] And = [0,0,1,0,0] Play = [0,0,0,1,0] Together = [0,0,0,0,1] Such a representation, called one-hot encoding, is very useful because we can merge these encodings to achieve a vector representation of the entire textual document, which is very central to modern search engines. The above vectors are binary, and there are other encodings possible, such as employing frequency or some other variant. If you are more curious, you can read about TF-IDF. This type of representation has obvious limitations, most importantly, it treats a word as atomic and provides useful information about the relationships that may exist between individual words. We can't perform any meaningful comparision between words other than equality. Furthermore, such a representation results in word vectors that are extremely sparse. Distributed Representations To overcome some of the limitations of the one-hot scheme, a distributed assumption is adapted, which states that words that appear in the same context are semantically closer than the words that do not share the same context. Using this principle, a word can be represented as points in a continous vector space, where semantically similar words correspond to nearby points. This represenation is also called word embeddings, since we are embedding word vectors in the distributed vector space. Essentially, the weight of each word in the vector is distributed across many dimensions. So, instead of a one-to-one mapping between a word and a basic vector (dimension), the word contribution is spread across all of the dimensions of the vector. The dimensions are believed to capture the semantic properties of the words. For example, for our text "Friends work and play together", each word can be represented as something like: Friends = [0.73,0.34,0.52,0.01] Work = [0.65,0.79,0.22,0.1] And = [0.87,0.94,0.14,0.7] Play = [0.73, 0.69, 0.89, 0.4] Together = [0.87,0.79,0.22,0.09] You can see that the words 'Friends' and 'Together' are closer to each other, and the words 'Work' and 'Play' have a higher similarity. Note that these vectors are chosen arbitrarily and do not show an actual representation. The sole purpose here is to give an example. Learning Distributed Representations: Word2Vec Distributed representations of words can be learned by training a model on a corpus of textual data. Mikolov, et. al. proposed an efficient method to learn these embeddings, making it feasible to learn high-quality word vectors on a huge corpus of data. The model is called word2vec, which uses a neural network to learn representations. Two architectures for neural network were proposed – Continuous Bag-of-Words (CBOW) and Skip-Gram. CBOW predicts the current word from a window of neighboring words, while skip-gram uses the current word to predict the surrounding words. Word2Vec models use a sliding window to quantify context. In each sliding window, there is a central word that is under attention, with a few words preceding and following the central word. One of the important parameters of the word2vec models is the length of the sliding window. Consider a textual document: "The goal of the manifold learning techniques is to 'learn' the low dimensional manifold." If we consider 4 words preceding and following the central word, the context of 'manifold' is: ['the', 'goal', 'of', 'the', 'learning', 'techniques', 'is', 'to']. These context words form the input layer of the neural network, and each word is represented as a vector using the one-hot schema. There is one hidden layer and one output layer. The output layer is formed by the central words ,that is, each element in the vocabulary. This way, we learn a representation for each word in terms of the context words. The actual ordering of the context words is irrelevant, which is called a bag-of-words assumption. The skip-gram method is completely opposite of the CBOW method. Here the central word is the input layer, and the context words are now at the output layer. CBOW is faster, but skip-gram does a better job for not-so-frequent words. The implementation of both the architectures can be found at Google Code. Google has also made public pre-trained word vectors that are trained on about 100 billion words from Google News dataset. Several other data, such as Wikipedia, have been used to compute word vectors. And modern neural network packages like Tensorflow have word2vec support. Refer to the word2vec tutorial of Tensorflow. It is important to understand that word2vec is not deep learning; in fact, both the CBOW and skip-gram architectures are shallow neural models with only one hidden layer. Applications Distributed representation of words has been successfully applied in many applications. Machine Translation has been shown to achieve much higher accuracy using distributed representations, so you can make the following assertions: - ```Distance(France, Germany) < Distance(France, Spain)``` - ```Vector('Paris') - Vector('France') + Vector('Italy') ~ Vector(Rome)``` - ```Vector('king') - Vector('man') + Vector('woman') ~ Vector('queen')``` - The odd one in [staple, hammer, saw, drill] is staple. Item2vec: word2vec for collborative filtering and recommendation system, so you can infer: Vector(David Guetta) - Vector(Avicii) + Vector(Beyonce) -> Vector(Rihanna) BioVectors: Word vectors for Bioinformations. BioVectors can characterize biological sequences in terms of biochemical and biophysical interpretations of the underlying patterns References Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. In Proceedings of Workshop at ICLR, 2013. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality. In Proceedings of NIPS, 2013. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic Regularities in Continuous Space Word Representations. In Proceedings of NAACL HLT, 2013 Word2Vec Implementation Tensorflow Example Python Implementation Tomas Mikolov, Quoc V. Le and Ilya Sutskever. Exploiting Similarities among Languages for Machine Translation. We show how the word vectors can be applied to machine translation. Barkan, O; Koenigstein, N (2016).Item2Vec: Neural Item Embedding for Collaborative Filtering Asgari, Ehsaneddin; Mofrad, Mohammad R.K. (2015). Continuous Distributed Representation of Biological Sequences for Deep Proteomics and Genomics. PloS one. 10 (11): e0141287.
Read more
  • 0
  • 0
  • 14953

article-image-running-your-project-using-kanban
Packt
14 Nov 2016
5 min read
Save for later

Running Your Project Using Kanban

Packt
14 Nov 2016
5 min read
In this article by Patrick Li, author of the book JIRA 7 Essentials - Fourth Edition, we will focus on the project template that is exclusive in JIRA Software, called Kanban. By the end of this article, you will learn the following: (For more resources related to this topic, see here.) JIRA Software project template How to run a project using JIRA's Kanban support Identifying inefficiencies in your process with Kanban Kanban Kanban, which runs in iterations, focuses more on the actual execution of delivery. It has a heavy emphasis on visualizing the delivery workflow from start to finish, places limits on different stages of the workflow by controlling how many work items are allowed to be in each stage, and measures the lead time. With Kanban, the importance is to be able to visually see the work items going through the workflow, identify areas of inefficiency and bottlenecks, and correct them. It is a continuous process with work coming in from one end, and going out from the other, and making sure that things go through as efficiently as possible. Running a project with Kanban Compared to Scrum, Kanban is a much simpler methodology. Unlike Scrum, which has a backlog and requires the team to prioritize and plan their delivery in sprints, Kanban focuses purely on the execution and measurement of throughput. In JIRA, a typical Kanban board will have the following differences compared to a Scrum board: There is no backlog view. Since Kanban does not have a sprint-planning phase, your board acts as the backlog. There are not active sprints. The idea behind Kanban is that you have a continuous flow of work. Columns can have minimum and maximum constraints. Columns will be highlighted if the constraints are violated. As shown in the following screenshot, both the Selected for Development and In Progress columns are highlighted due to constraint violation: Creating a Kanban project The first step to work with Kanban in JIRA is to create a project with the Kanban template: Select the Create project option from the Projects drop-down menu. Choose the Kanban software development template and click on Next. Accept the settings and click on Next. Enter the name and key for the new project and click on Submit. After you have created a Kanban project, you will be taken to the Kanban board view, which looks very similar to the active sprint view of a Scrum board. Remember, with Kanban, it is like you are running a sprint that does not end or ends when the entire project is completed. So the agile board itself focuses on helping you and your team to execute on delivery. Using the Kanban board As we have mentioned earlier, with Kanban, there is no planning phase, so you go straight to the issues board. Working with the Kanban board is actually very simple; new issues created by you and your team go straight onto the board and into the first column, called Backlog by default. Members of the team will then grab issues from the Backlog column, assign the issue to them, and move them through the workflow. During various stages, issues may need to be re-assigned to other users, for example, when an issue leaves the development stage and enters testing, it may be re-assigned to a test engineer. Once enough issues have been completed (moved to the last column, called Done by default), a release can be made. Once released, all issues in the Done column will be removed from the board (still in the system), so the team can continue to focus on the tasks at hand. Let's look at an example of the Kanban board shown in the following screenshot, in which we can clearly see that we have problems in both the In Development and In Testing phases of our process. In Development is highlighted in red, meaning we have enough work there, which is a sign of bottleneck. In Testing is highlighted in yellow, which means that we do not enough work and is a sign of efficiency: With this, the board is able to visually tell us where we are having problems and allows us to focus on these problem areas. The bottleneck in the In Development phase could mean we do not have enough developers, which causes the efficiency in the In Testing phase, where our testers are simply sitting around waiting for work to come. So, this raises a common question: what should be the correct constraints for my columns? The quick answer is, try and experiment as you go. The long answer is, there is no single correct, silver bullet answer. What you need to understand is, there are many factors that can influence the throughput of your team, such as the size of your team, team member leaving and joining, as well as the tasks at hand. In our example, the easy solution will be to lower the limit for both columns and we are done. But often, it is just as important for you to find the root cause of the problem rather than trying to simply fix the board itself. Perhaps, what you should try to do is get more developers onto your team so you can keep up the pace that is required for delivery. The take away here is that the Kanban board can help you pinpoint areas of problem, and it is up to you and your team to figure out the cause and find the appropriate solution. Summary In this article, we introduced the software project template that come with JIRA Software, and the agile methodology it supports, namely Kanban. We talked about how you can run projects using JIRA and the features it provides. Resources for Article: Further resources on this subject: JIRA 101 [article] Working with JIRA [article] JIRA Workflows [article]
Read more
  • 0
  • 0
  • 2308

article-image-digital-and-mobile-forensics
Packt
14 Nov 2016
14 min read
Save for later

Digital and Mobile Forensics

Packt
14 Nov 2016
14 min read
In this article, Mattia Epifani and Pasquale Stirparo, co-authors of the book Learning iOS Forensics - Second Edition, would be talking mainly, if not solely, about computer forensics and computer crimes, such as when an attacker breaks into a computer network system and steals data. This would involve two types of offenses—unlawful/unauthorized access and data theft. As mobile phones became more popular, the new field of mobile forensics developed. (For more resources related to this topic, see here.) Nowadays, things have changed radically and they are still changing at quite a fast pace as technology evolves. Digital forensics, which includes all disciplines dealing with electronic evidence, is also being applied to common crimes, to those that, at least by definition, are not strictly IT crimes. Today, more than ever, we live in a society that is fully digitalized and people are equipped with all kinds of devices, which have different types of capabilities, but all of them process, store, and transmit information (mainly over the Internet). This means that forensic investigators have to be able to deal with all these devices. As defined at the first Digital Forensics Research Workshop (DFRWS) in 2001, digital forensics is: "The use of scientifically derived and proven methods toward the preservation, collection, validation, identification, analysis, interpretation, documentation, and presentation of digital evidence derived from digital sources for the purpose of facilitating or furthering the reconstruction of events found to be criminal, or helping to anticipate unauthorized actions shown to be disruptive to planned operations." As Casey asserted (Casey, 2011): "In this modern age, it is hard to imagine a crime that does not have a digital dimension." Criminals of all kinds use technology to facilitate their offenses, communicate with their peers, recruit other criminals, launder money, commit credit card fraud, gather information on their victims, and so on. This obviously creates new challenges for all the different actors involved, such as attorneys, judges, law enforcement agents, and forensic examiners. Among the cases solved in recent years, there were kidnappings where the kidnapper was caught—thanks to a request for ransom sent by e-mail from his mobile phone. There have been many cases of industrial espionage in which unfaithful employees were hiding projects in the memory cards of their smartphones, cases of drug dealing solved—thanks to the evidence found in the backup of mobile phones that was on computer, and many other such cases. Even the largest robberies of our time are now being conducted via computer networks. In this article, you will learn the following: Definition and principles of mobile forensics How to properly handle digital evidence Mobile forensics Mobile forensics is a field of study in digital forensics that focuses on mobile devices. Among the different digital forensics fields, mobile forensics is without doubt the fastest growing and evolving area of study, having an impact on many different situations from corporate to criminal investigations and intelligence gathering, which are on the rise. Moreover, the importance of mobile forensics is increasing exponentially due to the continuous fast growth of the mobile market. One of the most interesting peculiarities of mobile forensics is that mobile devices, particularly mobile phones, usually belong to a single individual, while this is not always the case with a computer that may be shared among employees of a company or members of a family. For this reason, the analysis of mobile phones gives access to plenty of personal information. Another important and interesting aspect that comes with mobile forensics, which is both challenging and frustrating at the same time for the analyst, is the multitude of different device models and the customized flavors of their operating systems available in the market. This makes it very difficult to have a single solution (either a tool or process) to address them all. Just think of all the applications people have installed on their smartphones: IM clients, web browsers, social network clients, password managers, navigation systems, and much more, other than the classic default ones, such as an address book, which can provide a lot more information than just the phone number for each contact that has been saved. Moreover, syncing such devices with a computer has become a very easy and smooth process, and all user activities, schedules, to-do lists, and everything else is stored inside a smartphone. Aren't these enough to profile a person and reconstruct all their recent activities, than building the network of contacts? Finally, in addition to a variety of smartphones and operating systems, such as Apple iOS, Google Android, Microsoft Windows Phone, and Blackberry OS, there is a massive number of so-called feature phones that use older mobile OS systems. Therefore, it's pretty clear that when talking about mobile/smartphone forensics, there is so much more than just printouts of phone calls. In fact, with a complete examination, we can retrieve SMSes/MMSes, pictures, videos, installed applications, e-mails, geolocation data, and so on—both present and deleted information. Digital evidence As mentioned earlier, on one hand the increasing involvement of mobile devices in digital forensics cases has brought a whole new series of challenges and complexities. However, on the other hand, this has also resulted in a much greater amount of evidence from criminals that it is now being used to reconstruct their activities with a more comprehensive level of detail. Moreover, while classical physical evidence may be destroyed, digital evidence, most of the time, leaves traces. Over the years, there have been several definitions of what digital evidence actually is, some of them focusing particularly on the evidentiary aspects of proof to be used in court, such as the one proposed by the Standard Working Group on Digital Evidence (SWGDE), stating that: "Digital evidence is any information of probative value that is either stored or transmitted in a digital form." The definition proposed by the International Organization of Computer Evidence (IOCE) states: "Digital evidence is information stored or transmitted in binary form that may be relied on in court." The definition given by E. Casey (Casey, 2000), refers to digital evidence as: "Physical objects that can establish that a crime has been committed, can provide a link between a crime and its victim, or can provide a link between a crime and its perpetrator." While all of these are correct, as previously said, all of these definitions focus mostly on proofs and tend to disregard data that is extremely useful for an investigation. For this reason, and for the purpose of this book, we will refer to the definition given by Carrier (Carrier, 2006), where digital evidence is defined as: "Digital data that supports or refutes a hypothesis about digital events or the state of digital data." This definition is a more general one, but better matches the current state of digital evidence and its value within the entire investigation process. Also from a standardization point of view, there have been, and still are, many attempts to define guidelines and best practices for digital forensics on how to handle digital evidence. Other than the several guidelines and special publications from NIST, there is a standard from ISO/IEC that was released in 2012, the ISO 27037 guidelines for identification, collection and/or acquisition, and preservation of digital evidence, which is not specific to mobile forensics, but is related to digital forensics in general, aiming to build a standard procedure for collecting and handling digital evidence, which will be legally recognized and accepted in court in different countries. This is a really important goal if you consider the lack of borders in the Internet era, particularly when it comes to digital crimes, where illicit actions can be perpetrated by attackers from anywhere in the world. Handling of mobile evidence In order to be useful not only in court but also during the entire investigation phase, digital evidence must be collected, preserved, and analyzed in a forensically sound manner. This means that each step, from the identification to the reporting, has to be carefully and strictly followed. Historically, we are used to referring to a methodology as forensically sound if, and only if, it would imply that the original source of evidence remains unmodified and unaltered. This was mostly true when talking about classical computer forensics, in scenarios where the forensic practitioner found the computer switched off or had to deal with external hard drives, although not completely true even in these situations. However, since the rise of live forensics, this concept has become more and more untrue. In fact, methods and tools for acquiring memory from live systems inevitably alter, even if just a little bit, the target system they are run on. The advent of mobile forensics stresses this concept even more, because mobile devices, and smartphones in particular, are networked devices that continuously exchange data through several communication protocols, such as GSM/CDMA, Wi-Fi, Bluetooth, and so on. Moreover, in order to acquire a mobile device, forensic practitioners need to have some degree of interaction with the device. Based on the type, a smartphone can need more or less interaction, altering in this way the original state of the device. All of this does not mean that preservation of the source evidence is useless, but that it is nearly impossible in the field of mobile devices. Therefore, it becomes a matter of extreme importance to thoroughly document every step taken during the collection, preservation, and acquisition phases. Using this approach, forensic practitioners will be able to demonstrate that they have been as unintrusive as possible. As Casey states (Casey, 2011): "One of the keys to forensic soundness is documentation. A solid case is built on supporting documentation that reports on where the evidence originated and how it was handled. From a forensic standpoint, the acquisition process should change the original evidence as little as possible and any changes should be documented and assessed in the context of the final analytical results." When in the presence of mobile devices to be collected, it is a good practice for the forensic practitioner to consider the following points: Take note of the current location where the device has been found. Report the device status (switched on or off, broken screen, and so on). Report date, time, and other information visible on the screen if the device is switched on, for example, by taking a picture of the screen. Look very carefully for the presence of memory cards. Although it is not the case with iOS devices, generally many mobile phones have a slot for an external memory card, where pictures, chat databases, and many other types of user data are usually stored. Look very carefully for the presence of cables related to the mobile phone that is being collected, especially if you don't have a full set of cables in your lab. Many mobile phones have their own cables to connect to the computer and to recharge the battery. Search for the original Subscriber Identity Module (SIM) package, because that is where the PIN and PIN unblocking key (PUK) codes are written. Take pictures of every item before collection. Modifications to mobile devices can happen not only because of interaction with the forensic practitioner, but also due to interaction with the network, voluntarily or not. In fact, digital evidence in mobile devices can be lost completely as they are susceptible to being overwritten by new data, for example, with the smartphone receiving an SMS while it is being collected, thus overwriting possible evidence previously stored in the same area of memory as the newly arrived SMS, or upon receiving a remote wiping command over a wireless network. Most of today's smartphones and iOS devices can be configured to be completely wiped remotely. From a real case: While searching inside the house of a person under investigation, law enforcement agents found and seized, among other things, computers and a smartphone. After cataloguing and documenting everything, they put all the material into boxes to bring them back to the laboratory. Once back in their laboratory, when acquiring the smart phone in order to proceed with the forensics analysis, they noticed that the smartphone was empty and it appeared to be brand new. The owner had wiped it remotely. Therefore, isolating the mobile device from all radio networks is a fundamental step in the process of preservation of evidence. There are several ways to achieve this, all with their own pros and cons, as follows: Airplane mode: Enabling Airplane mode on a device requires some sort of interaction, which may pose some risks of modification by the forensic practitioner. This is one of the best possible options since it implies that all wireless communication chips are switched off. In this case, it is always good to document the action taken with pictures and/or videos. Normally, this is possible only if the phone is not password-protected or the password is known. However, for devices with iOS 7 or higher, it is also possible to enable airplane mode by lifting the dock from the bottom, where there will be a button with the shape of a plane. This is possible only if the Access on Lock Screen option is enabled from Settings | Control Center. Faraday's bag: This item is a sort of envelope made of conducting material, which blocks out static electric fields and electromagnetic radiation completely isolating the device from communicating with external networks. It is based, as the name suggests, on Faraday's law. This is the most common solution, particularly useful when the device is being carried from the crime scene to the lab after seizure. However, the use of Faraday's bag will make the phone continuously search for a network, which will cause the battery to quickly drain. Unfortunately, it is also risky to plug the phone to a power cable outside that will go inside the bag, because this may act as antenna. Moreover, it is important to keep in mind that when you remove the phone from the bag (once arrived in the lab) it will again be exposed to the network. So, you would need either a shielded lab environment or a Faraday solution that would allow you to access the phone while it is still inside the shielded container, without the need for external power cables. Jamming: A jammer is used to prevent a wireless device from communicating by sending out radio waves along the same frequencies as that device. In our case, it would jam the GSM/UMTS/LTE frequencies that mobile phones use to connect with cellular base stations to send/receive data. Be aware that this practice may be considered illegal in some countries, since it will also interfere with any other mobile device in the range of the jammer, disrupting their communications too. Switching off the device: This is a very risky practice because it may activate authentication mechanisms, such as PIN codes or passcodes, that are not available to the forensic practitioner, or other encryption mechanisms that carry the risk of delaying or even blocking the acquisition of the mobile device. Removing the SIM card: In most mobile devices, this operation implies removing the battery and therefore all the risks and consequences we just mentioned regarding switching off the device; however, in iOS devices this task is quite straightforward and easy, and it does not imply removing the battery (in iOS devices this is not possible). Moreover, SIM cards can have PIN protection enabled; removing it from the phone may lock the SIM card, preventing its content from being displayed. However, bear in mind that removing the SIM card will isolate the device only from the cellular network, while other networks, such as Wi-Fi or Bluetooth, may still be active and therefore need to be addressed. The following image shows a SIM card extracted from an iPhone with just a clip; image taken from http://www.maclife.com/: Summary In this article, we gave a general introduction to digital forensics for those relatively new to this area of study and a good recap to those already in the field, keeping the mobile forensics field specifically in mind. We have shown what digital evidence is and how it should be handled, presenting several techniques to isolate the mobile device from the network. Resources for Article: Further resources on this subject: Mobile Forensics [article] Mobile Forensics and Its Challanges [article] Forensics Recovery [article]
Read more
  • 0
  • 0
  • 23716

article-image-face-detection-and-tracking-using-ros-open-cv-and-dynamixel-servos
Packt
14 Nov 2016
13 min read
Save for later

Face Detection and Tracking Using ROS, Open-CV and Dynamixel Servos

Packt
14 Nov 2016
13 min read
In this article by Lentin Joseph, the author of the book ROS Robotic Projects, we learn how one of the capability in most of the service and social robots is face detection and tracking. The robot can identify faces and it can move its head according to the human face move around it. There are numerous implementation of face detection and tracking system in web. Most of the trackers are having a pan and tilt mechanism and a camera is mounted on the top of the servos. In this article, we are going to see a simple tracker which is having only pan mechanism. We are going to use a USB webcam which is mounted on AX-12 Dynamixel servo. (For more resources related to this topic, see here.) You can see following topics on this article: Overview of the project Hardware and software prerequisites Overview of the project The aim of the project is to build a simple face tracker which can track face only in the horizontal axis of camera. The tracker is having a webcam, Dynamixel servo called AX-12 and a supporting bracket to mount camera on the servo. The servo tracker will follow the face until it align to the center of the image which is getting from webcam. Once it reaches the center, it will stop and wait for the face movement. The face detection is done using OpenCV and ROS interface, and controlling the servo is done using Dynamixel motor driver in ROS. We are going to create two ROS packages for this complete tracking system, one is for face detection and finding centroid of face and next is for sending commands to servo to track the face using the centroid values. Ok!! Let's start discussing the hardware and software prerequisites of this project. Hardware and software prerequisites Following table of hardware components which can be used for building this project. You can also see a rough price of each component and purchase link of the same. List of hardware components: No Component name Estimated price (USD) Purchase link 1 Webcam 32 https://amzn.com/B003LVZO8S 2 Dynamixel AX -12 A servo with mounting bracket 76 https://amzn.com/B0051OXJXU 3 USB To Dynamixel Adapter 50 http://www.robotshop.com/en/robotis-usb-to-dynamixel-adapter.html 4 Extra 3 pin cables for AX-12 servos 12 http://www.trossenrobotics.com/p/100mm-3-Pin-DYNAMIXEL-Compatible-Cable-10-Pack 5 Power adapter 5 https://amzn.com/B005JRGOCM 6 6 Port AX/MX Power Hub 5 http://www.trossenrobotics.com/6-port-ax-mx-power-hub 7 USB extension cable 1 https://amzn.com/B00YBKA5Z0   Total Cost + Shipping + Tax ~ 190 - 200   The URLs and price can vary. If the links are not available, you can do a google search may do the job. The shipping charges and tax are excluded from the price. If you are thinking that, the total cost is not affordable for you, then there are cheap alternatives to do this project too. The main heart of this project is Dynamixel servo. We may can replace this servo with RC servos which only cost around $10 and using an Arduino board cost around $20 can be used to control the servo too, so you may can think about porting the face tracker project work using Arduino and RC servo Ok, let's look on to the software prerequisites of the project. The prerequisites include ROS framework, OS version and ROS packages: No Name of software Estimated price (USD) Download link 1 Ubuntu 16.04 L.T.S Free http://releases.ubuntu.com/16.04/ 2 ROS Kinetic L.T.S Free http://wiki.ros.org/kinetic/Installation/Ubuntu 3 ROS usb_cam package Free http://wiki.ros.org/usb_cam 3 ROS cv_bridge package Free http://wiki.ros.org/cv_bridge 4 ROS Dynamixel controller Free https://github.com/arebgun/dynamixel_motor 5 Windows 7 or higher ~ $120 https://www.microsoft.com/en-in/software-download/windows7 7 RoboPlus (Windows application) Free http://www.robotis.com/download/software/RoboPlusWeb%28v1.1.3.0%29.exe The above table will gives you an idea about which all are the software we are going to be used for this project. We may need both Windows and Ubuntu for doing this project. It will be great if you have dual operating system on your computer Let's see how to install these software first Installing dependent ROS packages We have already installed and configured Ubuntu 16.04 and ROS Kinetic on it. Here are the dependent packages we need to install for this project. Installing usb_cam ROS package Let's see the use of usb_cam package in ROS first. The usb_cam package is ROS driver for Video4Linux (V4L) USB camera. The V4L is a collection of devices drivers in Linux for real time video capture from webcams. The usb_cam ROS package work using the V4L devices and publish the video stream from devices as ROS image messages. We can subscribe it and do our own processing using it. The official ROS page of this package is given in the above table. You may can check this page for different settings and configuration this package offers. Creating ROS workspace for dependencies Before starting installing usb_cam package, let's create a ROS workspace for keeping the dependencies of the entire projects mentioned in the book. We may can create another workspace for keeping the project code. Create a ROS workspace called ros_project_dependencies_ws in home folder. Clone the usb_cam package into the src folder: $ git clone https://github.com/bosch-ros-pkg/usb_cam.git Build the workspace using catkin_make After building the package, install v4l-util Ubuntu package. It is a collection of command line V4L utilities which is using by usb_cam package: $ sudo apt-get install v4l-utils Configuring webcam on Ubuntu 16.04 After installing these two, we can connect the webcam to PC to check it properly detected in our PC. Take a terminal and execute dmesg command to check the kernel logs. If your camera is detected in Linux, it may give logs like this{ $ dmesg Kernels logs of webcam device You can use any webcam which has driver support in Linux. In this project, iBall Face2Face (http://www.iball.co.in/Product/Face2Face-C8-0--Rev-3-0-/90) webcam is used for tracking. You can also go for a popular webcam which is mentioned as a hardware prerequisite. You can opt that for better performance and tracking. If our webcam has support in Ubuntu, we may can open the video device using a tool called cheese. Cheese is simply a webcam viewer. Enter the command cheese in the terminal, if it is not available you can install it using following command: $ sudo apt-get install cheese If the driver and device are proper, you may get the video stream from webcam like this: Webcam video streaming using cheese Congratulation!!, your webcam is working well in Ubuntu, but are we done with everything? No. The next thing is to test the ROS usb_cam package. We have to make sure that is working well in ROS!! Interfacing Webcam to ROS Let's test the webcam using usb_cam package. The following command is used to launch the usb_cam nodes to display images from webcam and publishing ROS image topics at the same time: $ roslaunch usb_cam usb_cam-test.launch If everything works fine, you will get the image stream and logs in the terminal as shown below: Working of usb_cam package in ROS The image is displayed using image_view package in ROS, which is subscribing the topic called /usb_cam/image_raw Here are the topics, that usb_cam node is publishing: Figure 4: The topics publishing by usb_cam node We have just done with interfacing a webcam in ROS. So what's next? We have to interface AX-12 Dynamixel servo to ROS. Before proceeding to interfacing, we have to do something to configure this servo. Next we are going to see how to configure a Dynamixel servo AX-12A. Configuring a Dynamixel servo using RoboPlus The configuring of Dynamixel servo can be done using a software called RoboPlus providing by ROBOTIS INC (http://en.robotis.com/index/), the manufacturer of Dynamixel servos. For configuring Dynamixel, you have to switch your operating system to Windows. The tool RoboPlus will work on Windows. In this project, we are going to configure the servo in Windows 7. Here is the link to download RoboPlus: http://www.robotis.com/download/software/RoboPlusWeb%28v1.1.3.0%29.exe. If the link is not working, you can just search in google to get the RoboPlus 1.1.3 version. After installing the software, you will get the following window, navigate to Expert tab in the software for getting the application for configuring Dynamixel: Dynamixel Manager in RoboPlus Before taking the Dynamixel Wizard and do configuring, we have to connect the Dynamixel and properly powered. Following image of AX-12A servo that we are using for this project and its pin connection. AX-12A Dynamixel and its connection diagram Unlike other RC servos, AX-12 is an intelligent actuator which is having a microcontroller which can monitoring every parameters of servo and customize all the servo parameters. It is having a geared drive and the output of the servo is connected to servo horn. We may can connect any links on this servo horn. There are two connection ports behind each servo. Each port is having pins such as VCC, GND and Data. The ports of Dynamixel are daisy chained so that we can connect another servo from one servo. Here is the connection diagram of Dynamixel with PC. AX-12A Dynamixel and its connection diagram The main hardware component which interfacing Dynamixel to PC is called USB to Dynamixel. This is a USB to serial adapter which can convert USB to RS232, RS 484 and TTL. In AX-12 motors, the data communication is using TTL. From the Figure AX 12A Dynamixel and its connection diagram, we can seen that there are three pins in each port. The data pin is used to send and receive from AX-12 and power pins are used to power the servo. The input voltage range of AX-12A Dynamixel is from 9V to 12V. The second port in each Dynamixel can be used for daisy chaining. We can connect up to 254 servos using this chaining Official links of AX-12A servo and USB to Dynamixel AX-12A: http://www.trossenrobotics.com/dynamixel-ax-12-robot-actuator.aspx USB to Dynamixel: http://www.trossenrobotics.com/robotis-bioloid-usb2dynamixel.aspx For working with Dynamixel, we should know some more things. Let's have a look on some of the important specification of AX-12A servo. The specifications are taken from the servo manual. Figure 8: AX-12A Specification The Dynamixel servos can communicate to PC to a maximum speed of 1 Mbps. It can also give feedback of various parameters such as its position, temperature and current load. Unlike RC servos, this can rotate up to 300 degrees and communication is mainly using digital packets. Powering and connecting Dynamixel to PC Now we are going to connect Dynamixel to PC. Given below a standard way of connecting Dynamixel to PC: Connecting Dynamixel to PC The three pin cable can be first connected to any of the port of AX-12 and other side have to connect to the way to connect 6 port power hub. From the 6-port power hub, connect another cable to the USB to Dynamixel. We have to select the switch of USB to Dynamixel to TTL mode. The power can be either be connected through a 12V adapter or through battery. The 12V adapter is having 2.1X5.5 female barrel jack, so you should check the specification of male adapter plug while purchasing. Setting USB to Dynamixel driver on PC As we have already discussed the USB to Dynamixel adapter is a USB to serial convertor, which is having an FTDI chip (http://www.ftdichip.com/) on it. We have to install a proper FTDI driver on the PC for detecting the device. The driver may need for Windows but not for Linux, because FTDI drivers are built in the Linux kernel. If you install the RoboPlus software, the driver may be already installed along with it. If it is not, you can manually install from the RoboPlus installation folder. Plug the USB to Dynamixel to the Windows PC, and check the device manager. (Right click on My Computer | Properties | Device Manager). If the device is properly detected, you can see like following figure: Figure 10: COM Port of USB to Dynamixel If you are getting a COM port for USB to Dynamixel, then you can start the Dynamixel Manager from RoboPlus. You can connect to the serial port number from the list and click the Search button to scan for Dynamixel as shown in following figure. Select the COM port from the list and connecting to the port is marked as 1. After connecting to the COM port, select the default baud rate as 1 Mbps and click the Start searching button: COM Port of USB to Dynamixel If you are getting a list of servo in the left side panel, it means that your PC have detected a Dynamixel servo. If the servo is not detecting, you can do following steps to debug: Make sure that supply is proper and connections are proper using a multi meter. Make sure that servo LED on the back is blinking when power on. If it is not coming, it can be a problem with servo or power supply. Upgrade the firmware of servo using Dynamixel Manager from the option marked as 6. The wizard is shown in the following figure. During wizard, you may need power off the supply and ON it again for detecting the servo. After detecting the servo, you have to select the servo model and install the new firmware. This may help to detect the servo in the Dynamixel manager if the existing servo firmware is out dated. Dynamixel recovery wizard If the servos are listing on the Dynamixel manager, click on a servo and you can see its complete configuration. We have to modify some values inside the configurations for our current face tracker project. Here are the parameters: ID : Set the ID as 1 Baud rate: 1 Moving Speed: 100 Goal Position: 512 The modified servo settings are shown in the following figure: Modified Dynamixel firmware settings After doing these settings, you can check the servo is working good or not by changing its Goal position. Yes!! Now you are done with Dynamixel configuration, Congratulation!! What's next? We want to interface Dynamixel to ROS. Summary This article was about building a face tracker using webcam and Dynamixel motor. The software we have used was ROS and OpenCV. Initially you can see how to configure the webcam and Dynamixel motor and after configuring, we were trying to build two package for tracking. One package does the face detection and second package is a controller which can send position command to Dynamixel to track the face. We have discussed the use of all files inside the packages and did a final run to show the complete working of the system. Resources for Article: Further resources on this subject: Using ROS with UAVs [article] Hardware Overview [article] Arduino Development [article]
Read more
  • 0
  • 0
  • 21351
article-image-tensorflow-toolbox
Packt
14 Nov 2016
6 min read
Save for later

The TensorFlow Toolbox

Packt
14 Nov 2016
6 min read
In this article by Saif Ahmed, author of the book Machine Learning with TensorFlow, we learned how most machine learning platforms are focused toward scientists and practitioners in academic or industrial settings. Accordingly, while quite powerful, they are often rough around the edges and have few user-experience features. (For more resources related to this topic, see here.) Quite a bit of effort goes into peeking at the model at various stages and viewing and aggregating performance across models and runs. Even viewing the neural network can involve far more effort than expected. While this was acceptable when neural networks were simple and only a few layers deep, today's networks are far deeper. In 2015, Microsoft won the annual ImageNet competition using a deep network with 152 layers. Visualizing such networks can be difficult, and peeking at weights and biases can be overwhelming. Practitioners started using home-built visualizers and bootstrapped tools to analyze their networks and run performance. TensorFlow changed this by releasing TensorBoard directly alongside their overall platform release. TensorBoard runs out of box with no additional installations or setup. Users just need to instrument their code according to what they wish to capture. It features plotting of events, learning rate and loss over time; histograms, for weights and biases; and images. The Graph Explorer allows interactive reviews of the neural network. A quick preview You can follow along with the code here: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/models/image/cifar10/cifar10_train.py The example uses the CIFAR-10 image set. The CIFAR-10 dataset consists of 60,000 images in ten classes compiled by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The dataset has become one of several standard learning tools and benchmarks for machine learning efforts. Let's start with the Graph Explorer. We can immediately see a convolutional network being used. This is not surprising as we're trying to classify images here. This is just one possible view of the graph. You can try the Graph Explorer as well. It allows deep dives into individual components. Our next stop on the quick preview is the EVENTS tab. This tab shows scalar data over time. The different statistics are grouped into individual tabs on the right-hand side. The following screenshot shows a number of popular scalar statistics, such as loss, learning rate, cross entropy, and sparsity across multiple parts of the network. The HISTOGRAMS tab is a close cousin as it shows tensor data over time. Despite the name, as of TensorFlow v0.7, it does not actually display histograms. Rather, it shows summaries of tensor data using percentiles. The summary view is shown in the following figure. Just like with the EVENTS tab, the data is grouped into tabs on the right-hand side. Different runs can be toggled on and off and runs can be shown overlaid, allowing interesting comparisons. It features three runs, which we can see on the left side, and we'll look at just the softmax function and associated parameters. For now, don't worry too much about what these mean, we're just looking at what we can achieve for our own classifiers. However, the summary view does not do justice to the utility of the HISTOGRAMS tab. Instead, we will zoom into a single graph to observe what is going on. This is shown in the following figure: Notice that each histogram chart shows a time series of nine lines. The top is the maximum, the middle the median, and the bottom the minimum. The three lines directly above and below the median are one and half standard deviation, one standard deviation, and half standard deviation marks. Obviously, this does represent multimodal distributions as it is not a histogram. However, it does provide a quick gist of what would otherwise be a mountain of data to sift through. A couple of things to note are how data can be collected and segregated by runs, how different data streams can be collected, how we can enlarge the views, and how we can zoom into each of the graphs. Enough of graphics, lets jump into code so we can run this for ourselves! Installing TensorBoard TensorFlow comes prepackaged with TensorBoard, so it will already be installed. It runs as a locally served web application accessible via the browser at http://0.0.0.0:6006. Conveniently, there is no server-side code or configurations required. Depending on where your paths are, you may be able to run it directly, as follows: tensorboard --logdir=/tmp/tensorlogs If your paths are not correct, you may need to prefix the application accordingly, as shown in the following command line: tf_install_dir/ tensorflow/tensorboard --logdir=/tmp/tensorlogs On Linux, you can run it in the background and just let it keep running, as follows: nohup tensorboard --logdir=/tmp/tensorlogs & Some thought should be put into the directory structure though. The Runs list on the left side of the dashboard is driven by subdirectories in the logdir location. The following image shows two runs: MNIST_Run1 and MNIST_Run2. Having an organized runs folder will allow plotting successive runs side by side to see differences. When initializing the writer, you will pass in the log_location as the first parameter, as follows: writer = tf.train.SummaryWriter(log_location, sess.graph_def) Consider saving a base location and appending run-specific subdirectories for each run. This will help organize outputs without expending more thought on it. We’ll discuss more about this later. Incorporating hooks into our code The best way to get started with TensorBoard is by taking existing working examples and instrument them with the code required for TensorBoard. We will do this for several common training scripts. Summary In this article, we covered the major areas of TensorBoard—EVENTS, HISTOGRAMS, and viewing GRAPH. We modified popular models to see the exact changes required before TensorBoard could be up and running. This should have demonstrated the fairly minimal effort required to get started with TensorBoard. Resources for Article: Further resources on this subject: Supervised Machine Learning [article] Implementing Artificial Neural Networks with TensorFlow [article] Why we need Design Patterns? [article]
Read more
  • 0
  • 0
  • 2584

article-image-turn-your-life-gamified-experience-unity
Packt
14 Nov 2016
29 min read
Save for later

Turn Your Life into a Gamified Experience with Unity

Packt
14 Nov 2016
29 min read
In this article for by Lauren S. Ferro from the book Gamification with Unity 5.x we will look into a Gamified experience with Unity. In a world full of work, chores, and dull things, we all must find the time to play. We must allow ourselves to be immersed in enchanted world of fantasy and to explore faraway and uncharted exotic islands that form the mysterious worlds. We may also find hidden treasure while confronting and overcoming some of our worst fears. As we enter these utopian and dystopian worlds, mesmerized by the magic of games, we realize anything and everything is possible and all that we have to do is imagine. Have you ever wondered what Gamification is? Join us as we dive into the weird and wonderful world of gamifying real-life experiences, where you will learn all about game design, motivation, prototyping, and bringing all your knowledge together to create an awesome application. Each chapter in this book is designed to guide you through the process of developing your own gamified application, from the initial idea to getting it ready and then published. The following is just a taste of what to expect from the journey that this book will take you on. (For more resources related to this topic, see here.) Not just pixels and programming The origins of gaming have an interesting and ancient history. It stems as far back as the ancient Egyptians with the game Sennet; and long since the reign of great Egyptian Kings, we have seen games as a way to demonstrate our strength and stamina, with the ancient Greeks and Romans. However, as time elapsed, games have not only developed from the marble pieces of Sennet or the glittering swords of battles, they have also adapted to changes in the medium: from stone to paper, and from paper to technology. We saw the rise and development of physical games (such as table top and card games) to games that require us to physically move our characters using our bodies and peripherals (Playstaton Move and WiiMote), in order to interact with the gaming environment (Wii Sports and Heavy Rain). So, now we not only have the ability to create 3D virtual worlds with virtual reality, but also can enter these worlds and have them enter ours with augmented reality. Therefore, it is important to remember that, just as the following image, (Dungeons and Dragons), games don't have to take on a digital form, they can also be physical: Dungeons and Dragons board with figurines and dice Getting contextual At the beginning of designing a game or game-like experience, designers need to consider the context for which the experience is to take place. Context is an important consideration for how it may influence the design and development of the game (such as hardware, resources, and target group). The way in which a designer may create a game-like experience varies. For example, a game-like experience aimed to encourage students to submit assessments on time will be designed differently from the one promoting customer loyalty. In this way, the designer should be more context aware, and as a result, it may be more likely to keep it in view during the design process. Education: Games can be educational, and they may be designed specifically to teach or to have elements of learning entwined into them to support learning materials. Depending on the type of learning game, it may include formal (educational institutions) or informal educational environments (learning a language for a business trip). Therefore, if you are thinking about creating an educational game, you might need to think about these considerations in more detail. Business: Maybe your intention is get your employees to arrive on time or to finish reports in the afternoon rather than right before they go home. Designing content for use within a business context targets situations that occur within the workplace. It can include objectives such as increasing employee productivity (individual/group). Personal: Getting personal with game-like applications can relate specifically to creating experiences to achieve personal objectives. These may include personal development, personal productivity, organization, and so on. Ultimately, only one person maintains these experiences; however, other social elements, such as leaderboards and group challenges, can bring others into the personal experience as well. Game: If it is not just educational, business, or personal development, chances are that you probably want to create a game to be a portal into lustrous worlds of wonder or to pass time on the evening commute home. Pure gaming contexts have no personal objectives (other than to overcome challenges of course). Who is our application targeting and where do they come from? Understanding the user is one of the most important considerations for any approach to be successful. User considerations not only include the demographics of the user (for example, who they are and where they are from), but also the aim of the experience, the objectives that you aim to achieve, and outcomes that the objectives lead to. In this book, this section considers the real-life consequences that your application/game will have on its audience. For example, will a loyalty application encourage people to engage with your products/store in the areas that you're targeting it toward. Therefore, we will explore ways that your application can obtain demographic data in Unity. Are you creating a game to teach Spanish to children, teenagers, or adults? This will change the way that you will need to think about your audience. For example, children tend to be users who are encouraged to play by their parents, teenagers tend to be a bit more autonomous but may still be influenced by their parents, and adults are usually completely autonomous. Therefore, this can influence the amount and the type of feedback that you can give and how often. Where are your audience from? For example, are you creating an application for a global reward program or a local one? This will have an effect on whether or not you will incorporate things like localization features so that the application adapts to your audience automatically or whether it's embedded into the design. What kind of devices does your audience use? Do they live in an area where they have access to a stable Internet connection? Do they need to have a powerful system to run your game or application? Chances are if the answer is yes for the latter question then you should probably take a look at how you will optimize your application. What is game design? Many types of games exist and so do design approaches. There are different ways that you can design and implement games. Now, let's take a brief look at how games are made, and more importantly, what they are made of: Generating ideas: This involves thinking about the story that we want to tell, or a trip that we may want the player to go on. At this stage, we're just getting everything out of our head and onto the paper. Everything and anything should be written; the stranger and abstract the idea, the better. It's important at this stage not to feel trapped that an idea may not be suitable. Often, the first few ideas that we create are the worst, and the great stuff comes from iterating all the ideas that we put down in this stage. Talk about your ideas with friends and family, and even online forums are a great place to get feedback on your initial concepts. One of the first things that any aspiring game designer can begin with is to look at what is already out there. A lot is learned when we succeed—or fail—especially why and how. Therefore, at this stage, you will want to do a bit of research about what you are designing. For instance, if you're designing an application to teach English, not only should you see other similar applications that are out there but also how English is actually taught, even in an educational environment. While you are generating ideas, it is also useful to think about the technology and materials that you will use along the way. What game engine is better for your game's direction? Do you need to purchase licenses if you are intending to make your game commercial? Answering these kinds of questions earlier can save many headaches later on when you have your concept ready to go. Especially, if you will need to learn how to use the software, as some have steep learning curves. Defining your idea: This is not just a beautiful piece of art that we see when a game is created; it can be rough, messy, and downright simple, but it communicates the idea. Not just this; it also communicates the design of the game's space and how a player may interact and even traverse it. Concept design is an art in itself and includes concepts on environments, characters puzzles, and even the quest itself. We will take the ideas that we had during the idea generation and flesh them out. We begin to refine it, to see what works and what doesn't. Again, get feedback. The importance of feedback is vital. When you design games, you often get caught up; you are so immersed in your ideas, and they make sense to you. You have sorted out every details (at least for the most part, it feels like that). However, you aren't designing for you, you are designing for your audience, and getting an outsiders opinion can be crucial and even offer a perspective that you may not necessarily would have thought of. This stage also includes the story. A game without a story is like a life without existence. What kind of story do you want your player to be a part of? Can they control it, or is it set in stone? Who are the characters? The answers to these questions will breathe soul into your ideas. While you design your story, keep referring to the concept that you created, the atmosphere, the characters, and the type of environment that you envision. Some other aspects of your game that you will need to consider at this stage are as follows: How will your players learn how to play your game? How will the game progress? This may include introducing different abilities, challenges, levels, and so on. Here is where you will need to observe the flow of the game. Too much happening and you will have a recipe for chaos, not enough and your player will get bored. What is the number of players that you envision playing your game, even if you intend for a co-op or online mode? What are the main features that will be in your game? How will you market your game? Will there be an online blog that documents the stages of development? Will it include interviews with different members of the team? Will there be different content that is tailored for each network (for example, Twitter, Facebook, Instagram, and so on). Bringing it together: This involves thinking about how all your ideas will come together and how they will work, or won't. Think of this stage as creating a painting. You may have all pieces, but you need to know how to use them to create the piece of art. Some brushes (for example, story, characters) work better with some paints (for example, game elements, mechanics), and so on. This stage is about bringing your ideas and concepts into reality. This stage features design processes, such as the following: Storyboards that will give an overview of how the story and the gameplay evolve throughout the game. Character design sheets that will outline characteristics about your characters and how they fit into the story. Game User Interfaces (GUIs) that will provide information to the player during gameplay. This may include elements, such as progress bars, points, and items that they will collect along the way. Prototyping: This is where things get real…well, relatively. It may be something as simple as a piece of paper or something more complex as a 3D model. You then begin to create the environments or the levels that your player will explore. As you develop your world, you will take your content and populate the levels. Prototyping is where we take what was in our head and sketched out on paper and use it to sculpt the gameful beast. The main purpose of this stage is to see how everything works, or doesn't. For example, the fantastic idea of a huge mech-warrior with flames shooting out of an enormous gun on its back was perhaps not the fantastic idea that was on paper, at least not in the intended part of the game. Rapid prototyping is fast and rough. Remember when you were in school and you had things, such as glue, scissors, pens, and pencils; well, that is what you will need for this. It gets the game to a functioning point before you spend tireless hours in a game engine trying to create your game. A few bad rapid prototypes early on can save a lot of time instead of a single digital one. Lastly, rapid prototyping isn't just for the preliminary prototyping phase. It can be used before you add in any new features to your game once it's already set up. Iteration: This is to the game what an iron is to a creased shirt. You want your game to be on point and iterating it gets it to that stage. For instance, that awesome mech-warrior that you created for the first level was perhaps better as the final boss. Iteration is about fine-tuning the game, that is, to tweak it so that it not only flows better overall, but also improves the gameplay. Playtesting: This is the most important part of the whole process once you have your game to a relatively functioning level. The main concept here is to playtest, playtest, and playtest. The importance of this stage cannot be emphasized enough. More often than not, games are buggy when finally released, with problems and issues that could be avoided during this stage. As a result, players lose interest and reviews contain frustration and disappointment, which—let's face it—we don't want after hours and hours of blood, sweat, and tears. The key here is not only to playtest your game but also to do it in multiple ways and on multiple devices with a range of different people. If you release your game on PC, test it on a high performance one and a low performance one. The same process should be applied for mobile devices (phones, tablets) and operating systems. Evaluate: Evaluateyour game based on the playtesting. Iterating, playtesting, and evaluating are three steps that you will go through on a regular basis, more so as you implement a new feature or tweak an existing one. This cycle is important. You wouldn't buy a car that has parts added without being tested first so why should a player buy a game with untested features? Build: Build your game and get it ready for distribution, albeit on CD or online as a digital download Publish: Publish your game! Your baby has come of age and is ready to be released out into the wild where it will be a portal for players around the world to enter the world that you (and your team) created from scratch. Getting gamified When we merge everyday objectives with games, we create gamified experiences. The aim of these experiences is to improve something about ourselves in ways that are ideally more motivating than how we perceive them in real life. For example, think of something that you find difficult to stay motivated with. This may be anything from managing your finances, to learning a new language, or even exercising. Now, if you make a deal with yourself to buy a new dress once you finish managing your finances or to go on a trip once you have learned a new language, you are turning the experience into a game. The rules are simply to finish the task; the condition of finishing it results in a reward—in the preceding example, either a dress or the trip. The fundamental thing to remember is that gamified experiences aim to make ordinary tasks extraordinary and enjoyable for the player. Games, gaming, and game-like experiences can give rise to many types of opportunities for us to play or even escape reality. To finish this brief exploration into the design of games, we must realize that games are not solely about sitting in front of the TV, playing on the computer, or being glued to the seat transfixed on a digital character dodging bullets. The game mechanics that make a task more engaging and fun is defined as "Gamification." Gamification relates to games, and not play; while the term has become popular, the concept is not entirely new. Think about loyalty cards, not just frequent flyer mile programs, but maybe even at your local butcher or café. Do you get a discount after a certain amount of purchases? For example, maybe, the tenth coffee is free. It's been a while since various reward schemes have already been in place; giving children a reward for completing household chores or for good behavior and rewarding "gold stars" for academic excellence is gamification. If you consider some social activities, such as Scouts, they utilize "gamification" as part of their procedures. Scouts learn new skills and cooperate and through doing so, they achieve status and receive badges of honor that demonstrate levels of competency. Gamification has become a favorable approach to "engaging" clients with new and exciting design schemes to maintain interest and promote a more enjoyable and ideally "fun" product. The product in question does not have to be "digital." Therefore, "gamification" can exist both in a physical realm (as mentioned before with the rewarding of gold stars) as well as in a more prominent digital sense (for example, badge and point reward systems) as an effective way to motivate and engage users. Some common examples of gamification include the following: Loyalty programs: Each time you engage with the company in a particular way, such as buying certain products or amount of, you are rewarded. These rewards can include additional products, points toward items, discounts, and even free items. School House points: A pastime that some of us may remember, especially fans of Harry Potter is that each time you do the right thing, such as follow the school rules, you get some points. Alternatively, you do the wrong thing, and you lose points. Scouts: It rewards levels of competency with badges and ranks. The more skilled you are, the more badges you collect, wear, and ultimately, the faster you work your way up the hierarchy. Rewarding in general: This will often be associated with some rules, and these rules determine whether or not you will get a reward. Eat your vegetables, you will get dessert; do your math homework, you will get to play. Both have winning conditions. Tests: As horrifying as it might sound, tests can be considered as a game. For example, we're on a quest to learn about history. Each assignment you get is like a task, preparing you for the final battle—the exam. At the end of all these assessments, you get a score or a grade that indicates to you your progress as you pass from one concept to the next. Ultimately, your final exam will determine your rank among your peers and whether or not you made it to the next level (that being anywhere from your year level to a university). It may be also worth noting that just as in games, you also have those trying to work the system, searching for glitches in the system that they can exploit. However, just as in games, they too eventually are kicked. One last thing to remember when you design anything targeted toward kids is that they can be a lot more perceptive than what we sometimes give them credit for. Therefore, if you "disguise" educational content with gameplay, it is likely that they will see through it. It's the same with adults; they know that they are monitoring their health or spending habits, it's your job to make it a little less painful. Therefore, be upfront, transparent, and cut through the "disguise." Of course, kids don't want to be asked to "play a game about maths" but they will be more interested in "going on adventures to beat the evil dragon with trigonometry." The same goes for adults; creating an awesome character that can be upgraded to a level-80 warrior for remembering to take out the trash, keep hydrated, and eat healthier is a lot better than telling them this is a "fun" application to become a better person. There is no I in Team Working on our own can be good, sometimes working with others can be better! However, the problem with working in a team is that we're all not equal. Some of us are driven by the project, with the aim to get the best possible outcome, whereas, others are driven by fame, reward, money, and the list goes on. If you ever worked on a group project in school, then you know exactly what it's like. Agile gamification is, to put simply, getting teams to work better together. Often, large complex projects encounter a wide range of problems from keeping on top of schedules, different perspectives, undefined roles, and a lack of overall motivation. Agile frameworks in this context are associated with the term Scrum. This describes an overall framework used to formalize software development projects. The Scrum process works as follows: The owner of the product will create a wish list known as the product backlog. Once the sprint planning begins, members of the team (between 3-9 people) will take sections from the top of the product backlog. Sprint planning involves the following: It involves listing all of the items that are needed to be completed for the project (in a story format—who, what, and why). This list needs to be prioritized. It includes estimating each task relatively (using the Fibonacci system). It involves planning the work sprint (1-2 week long, but less than 1 month long) and working toward a demo. It also involves making the work visible using a storyboard that contains the following sections: To do, Doing, and Done. Items begin in the To do section; once they have begun, they move to the Doing section; and once they are completed, they are then put in the Done section. The idea is that the team works through tasks in the burn down chart. Ideally, the amount of points that the sprint began with (in terms of tasks to be done) decreases in value each day you get closer to finishing the sprint. The team engages with daily meetings (preferably standing up) run by the Sprint/Scrum master. These meetings discuss what was done, what is planned to be done during the day, any issues that come up or might come up, and how can improvements be made. It provides a demonstration of the product's basic (working) features. During this stage, feedback is provided by the product owner as to whether or not they are happy with what has been done, the direction that it is going, and how it will relate to the remaining parts of the project. At this stage, the owner may ask you to improve it, iterate it, and so forth, for the next sprint. Lastly, the idea is to get the team together and to review the development of the project as a whole: what went well and what didn't go so well and what are the areas of improvement that can then be used to make the next Scrum better? Next, they will decide on how to implement each section. They will meet each day to not only assess the overall progress made for the development of each section but also to ensure that the work will be achieved within the time frame. Throughout the process, the team leader known as the Scrum/Sprint Master has the job of ensuring that the team stays focused and completes sections of the product backlog on time. Once the sprint is finished, the work should be at a level to be shipped, sold to the customer, or to at least show to a stakeholder. At the end of the sprint, the team and Scrum/Sprint Master assess the completed work and determine whether it is at an acceptable level. If the work is approved, the next sprint begins. Just as the first sprint, the team chooses another chunk of the product backlog and begins the process again.An overview of the Scrum process However, in the modern world, Scrum is adopted and applied to a range of different contexts outside of software development. As a result, it has gone through some iterations, including gamification. Agile Gamification, as it is more commonly known as, takes the concept of Scrum and turns it into a playful experience. Adding an element of fun to agile frameworks To turn the concept of Scrum into something a bit more interesting and at the same time to boost the overall motivation of your team, certain parts of it can be transformed with game elements. For example, implementing leaderboards based on the amount of tasks that each team member is able to complete (and on time) results in a certain number of points. By the end of the spring, the team member with the most number of points may be able to obtain a reward, such as a bonus in their next pay or an extended lunch break. It is also possible to make the burn down chart a bit more exciting by placing various bonuses if certain objectives are met within certain time frame or at a certain point during the burn down;as a result, giving added incentive to team members to get things delivered on time. In addition, to ensure that quality standards are also maintained, Scrum/Sprint Masters can also provide additional rewards if there is few or no feedback regarding things, such as quality or the overall cohesiveness of the output from the sprint. An example of a gamified framework can be seen in the image below. While setting up a DuoLingo Classroom account, users are presented with various game elements (for example, progress bar) and a checklist to ensure that everything that needs to be completed is done. Playtesting This is one of the most important parts of your game design. In fact, you cannot expect to have a great game without it. Playtesting is not just about checking whether your game works, or if there are bugs, it is also about finding out what people really think about it before you put it out in the world to see. In some cases, playtesting can make the difference between succeeding of failing epically. Consider this scenario: you have spent the last year, your blood, sweat and tears, and even your soul to create something fantastic. You probably think it's the best thing out there. Then, after you release it, you realize that only half the game was balanced, or worst, half interesting. At this stage, you will feel pretty down, but all these could have been avoided if you had taken the time to get some feedback. As humans, we don't necessarily like to hear our greatest love being criticized, especially if we have committed so much of our lives to it. However, the thing to keep in mind is, this stage shapes the final details. Playtesting is not meant for the final stages, when your game is close to being finished. At each stage, even when you begin to get a basic prototype completed, it should be play tested. During these stages, it does not have to be a large-scale testing, it can be done by a few colleagues, friends, or even family who can give you an idea of whether or not you're heading in the right direction. Of course, the other important thing to keep in mind is that the people who are testing your game are as close, if not the target audience. For instance, image that you're creating for your gamified application to encourage people to take medication on a regular basis is not ideal to test with people who do not take medication. Sure, they may be able to cover general feedback, such as user interface elements or even interaction, but in terms of its effectiveness, you're better off taking the time to recruit more specific people. Iterating After we have done all the playtesting is the time to re-plan another development cycle. In fact, the work of tuning your application doesn't stop after the first tests. On the contrary, it goes through different iterations many times. The iteration cycle starts with the planning stage, which include brainstorming, organizing the work (as we saw for instance in Scrum), and so on. In the next phase, development, we actually create the application, as we did in the previous chapter. Then, there is the playtesting, which we saw earlier in this chapter. In the latter stage, we tune and tweak values and fix bugs from our application. Afterward, we iterate the whole cycle again, by entering in the planning stage again. Here, we will need to plan the next iteration: what should be left and what should be done better or what to remove. All these decisions should be based on what we have collected in the playtesting stage. The cycle is well represented in the following diagram as a spiral that goes on and on through the process: The point of mentioning it now is because after you finish playtesting your game, you will need to repeat the stages that we have done previously, again. You will have to modify your design; you may need to even redesign things again. So, it is better to think of this as upgrading your design, rather than a tedious and repetitive process. When to stop? In theory, there is no stopping; the more the iteration, the better the application will be. Usually, the iterations stop when the application is well enough for your standards or when external constrains, such as the market or deadlines, don't allow you to perform any more iteration. The question when to stop? is tricky, and the answer really depends on many factors. You will need to take into account the resources needed to perform another iteration and time constraints. Of course, remember that your final goal is to deliver a quality product to your audience and each iteration is a step closer. Taking in the view with dashboards Overviews, summaries, and simplicity make life easier. Dashboards are a great way for keeping a lot of information relatively concise and contained, without being too overwhelming to a player. Of course, if the players want to obtain more detailed information, perhaps statistics about their accuracy since they began, they will have the ability to do so. So, what exactly is a dashboard? A dashboard is a central hub to view all of your progress, achievements, points, and rewards. If we take a look at the following screenshot, we can get a rough idea about what kind of information that they display. The image on the left is the dashboard for Memrise and displays current language courses, in this case, German; the players' achievements and streak; and the progress that they are making in the course. On the right is the dashboard for DuoLingo. Similar to Memrise, it also features information about daily streaks, amount of time committed, and the strength of each category learned for the new language, in this case, Italian. By just looking at these dashboards, the player can get a very quick idea about how well or bad they are doing.   Different dashboards (left) Memrise (right) DuoLingo Different approaches to dashboards can encourage different behaviors depending on the data displayed and how it is displayed. For example, you can have a dashboard that provides reflective information more dominantly, such as progress bars and points. Others can provide a more social approach by displaying the players rank among friends and comparing their statistics to others who are also engaged with the application. Some dashboards may even suggest friends that have similar elements in common, such as the language that is being learned. Ideally, the design of dashboards can be as simple or as complicated as the designer decides, but typically, the less is more approach is better. Summary Everything that we discussed in this chapter is just a taste of what this book offers. Each aspect of the design process is explained in more detail, giving you not only the information, but also the practical skills that you can use to build upon and develop any gamified application from start to finish. If you want to find out about gamification, how to use it, and more importantly how to implement it into Unity, then this book is a great foundation to get you going. In particular, you will learn how to apply all these concepts into Unity and create gamified experiences. Furthermore, the book will bring you to create a gamified application starting from the basic pieces, with a particular focus to your audience and your goals. Learning about the uses of gamification does not have to stop with this book. In fact, there are many ways that you can develop the knowledge that you have gained and apply it to other tasks. Some other Packt books, such as the Unity UI Cookbook by Francesco Sapio, which you can obtain at https://www.packtpub.com/game-development/unity-ui-cookbook features a range of different recipes to implement a range of different UI elements that can even be featured in your dashboard. In fact, UIs are the key for the development of gamifed experiences and applications. The main thing is that you continue to learn, adapt, and to apply your knowledge in many different types of contexts. Resources for Article: Further resources on this subject: Buildbox 2 Game Development: peek-a-boo [article] Customizing the Player Character [article] Sprites in Action [article]
Read more
  • 0
  • 0
  • 24636

article-image-software-defined-data-center
Packt
14 Nov 2016
33 min read
Save for later

The Software-defined Data Center

Packt
14 Nov 2016
33 min read
In this article by Valentin Hamburger, author of the book Building VMware Software-Defined Data Centers, we are introduced and briefed about the software-defined data center (SDDC) that has been introduced by VMware, to further describe the move to a cloud like IT experience. The term software-defined is the important bit of information. It basically means that every key function in the data center is performed and controlled by software, instead of hardware. This opens a whole new way of operating, maintaining but also innovating in a modern data center. (For more resources related to this topic, see here.) But how does a so called SDDC look like – and why is a whole industry pushing so hard towards its adoption? This question might also be a reason why you are reading this article, which is meant to provide a deeper understanding of it and give practical examples and hints how to build and run such a data center. Meanwhile it will also provide the knowledge of mapping business challenges with IT solutions. This is a practice which becomes more and more important these days. IT has come a long way from a pure back office, task oriented role in the early days, to a business relevant asset, which can help organizations to compete with their competition. There has been a major shift from a pure infrastructure provider role to a business enablement function. Today, most organizations business is just as good as their internal IT agility and ability to innovate. There are many examples in various markets where a whole business branch was built on IT innovations such as Netflix, Amazon Web Services, Uber, Airbnb – just to name a few. However, it is unfair to compare any startup with a traditional organization. A startup has one application to maintain and they have to build up a customer base. A traditional organization has a proven and wide customer base and many applications to maintain. So they need to adapt their internal IT to become a digital enterprise, with all the flexibility and agility of a startup, but also maintaining the trust and control over their legacy services. This article will cover the following points: Why is there a demand for SDDC in IT What is SDDC Understand the business challenges and map it to SDDC deliverables The relation of a SDDC and an internal private cloud Identify new data center opportunities and possibilities Become a center of innovation to empower your organizations business The demand for change Today organizations face different challenges in the market to stay relevant. The biggest move was clearly introduced by smartphones and tablets. It was not just a computer in a smaller device, they changed the way IT is delivered and consumed by end users. These devices proved that it can be simple to consume and install applications. Just search in an app store – choose what you like – use it as long as you like it. If you do not need it any longer, simply remove it. All with very simplistic commands and easy to use gestures. More and more people relying on IT services by using a smartphone as their terminal to almost everything. These devices created a demand for fast and easy application and service delivery. So in a way, smartphones have not only transformed the whole mobile market, they also transformed how modern applications and services are delivered from organizations to their customers. Although it would be quite unfair to compare a large enterprise data center with an app store or enterprise service delivery with any app installs on a mobile device, there are startups and industries which rely solely on the smartphone as their target for services, such as Uber or WhatsApp. On the other side, smartphone apps also introduce a whole new way of delivering IT services, since any company never knows how many people will use the app simultaneously. But in the backend they still have to use web servers and databases to continuously provide content and data for these apps. This also introduces a new value model for all other companies. People start to judge a company by the quality of their smartphone apps available. Also people started to migrate to companies which might offer a better smartphone integration as the previous one used. This is not bound to a single industry, but affects a broad spectrum of industries today such as the financial industry, car manufacturers, insurance groups, and even food retailers, just to name a few. A classic data center structure might not be ideal for quick and seamless service delivery. These architectures are created by projects to serve a particular use case for a couple of years. An example of this bigger application environments are web server farms, traditional SAP environments, or a data warehouse. Traditionally these were designed with an assumption about their growth and use. Special project teams have set them up across the data center pillars, as shown in the following figure. Typically, those project teams separate after such the application environment has been completed. All these pillars in the data center are required to work together, but every one of them also needs to mind their own business. Mostly those different divisions also have their own processes which than may integrate in a data center wide process. There was a good reason to structure a data center in this way, the simple fact that nobody can be an expert for every discipline. Companies started to create groups to operate certain areas in a data center, each building their own expertise for their own subject. This was evolving and became the most applied model for IT operations within organizations. Many, if not all, bigger organizations have adopted this approach and people build their careers on this definitions. It served IT well for decades and ensured that each party was adding its best knowledge to any given project. However, this setup has one flaw, it has not been designed for massive change and scale. The bigger these divisions get, the slower they can react to request from other groups in the data center. This introduces a bi-directional issue – since all groups may grow in a similar rate, the overall service delivery time might also increase exponentially. Unfortunately, this also introduces a cost factor when it comes to service deployments across these pillars. Each new service, an organization might introduce or develop, will require each area of IT to contribute. Traditionally, this is done by human hand overs from one department to the other. Each of these hand overs will delay the overall project time or service delivery time, which is also often referred to as time to market. It reflects the needed time interval from the request of a new service to its actual delivery. It is important to mention that this is a level of complexity every modern organization has to deal with, when it comes to application deployment today. The difference between organizations might be in the size of the separate units, but the principle is always the same. Most organizations try to bring their overall service delivery time down to be quicker and more agile. This is often related to business reasons as well as IT cost reasons. In some organizations the time to deliver a brand new service from request to final roll out may take 90 working days. This means – a requestor might wait 18 weeks or more than four and a half month from requesting a new business service to its actual delivery. Do not forget that this reflects the complete service delivery – over all groups until it is ready for production. Also, after these 90 days the requirement of the original request might have changed which would lead into repeating the entire process. Often a quicker time to market is driven by the lines of business (LOB) owners to respond to a competitor in the market, who might already deliver their services faster. This means that today's IT has changed from a pure internal service provider to a business enabler supporting its organization to fight the competition with advanced and innovative services. While this introduces a great chance to the IT department to enable and support their organizations business, it also introduces a threat at the same time. If the internal IT struggles to deliver what the business is asking for, it may lead to leverage shadow IT within the organization. The term shadow IT describes a situation where either the LOBs of an organization or its application developers have grown so disappointed with the internal IT delivery times, that they actually use an external provider for their requirements. This behavior is not agreed with the IT security and can lead to heavy business or legal troubles. This happens more often than one might expect, and it can be as simple as putting some internal files on a public cloud storage provider. These services grant quick results. It is as simple as Register – Download – Use. They are very quick in enrolling new users and sometimes provide a limited use for free. The developer or business owner might not even be aware that there is something non-compliant going on while using this services. So besides the business demand for a quicker service delivery and the security aspect, there an organizations IT department has now also the pressure of staying relevant. But SDDC can provide much more value to the IT than just staying relevant. The automated data center will be an enabler for innovation and trust and introduce a new era of IT delivery. It can not only provide faster service delivery to the business, it can also enable new services or offerings to help the whole organization being innovative for their customers or partners. Business challenges—the use case Today's business strategies often involve a digital delivery of services of any kind. This implies that the requirements a modern organization has towards their internal IT have changed drastically. Unfortunately, the business owners and the IT department tend to have communication issues in some organizations. Sometimes they even operate completely disconnected from each other, as if each of them where their own small company within the organization. Nevertheless, a lot of data center automation projects are driven by enhanced business requirements. In some of these cases, the IT department has not been made aware of what these business requirements look like, or even what the actual business challenges are. Sometimes IT just gets as little information as: We are doing cloud now. This is a dangerous simplification since, the use case is key when it comes to designing and identifying the right solution to solve the organizations challenges. It is important to get the requirements from both sides, the IT delivery side as well as the business requirements and expectations. Here is a simple example how a use case might be identified and mapped to technical implementation. The business view John works as a business owner in an insurance company. He recognizes that their biggest competitor in the market started to offer a mobile application to their clients. The app is simple and allows to do online contract management and tells the clients which products the have enrolled as well as rich information about contract timelines and possible consolidation options. He asks his manager to start a project to also deliver such an application to their customers. Since it is only a simple smartphone application, he expects that it's development might take a couple of weeks and than they can start a beta phase. To be competitive he estimates that they should have something useable for their customers within maximum of 5 months. Based on these facts, he got approval from his manager to request such a product from the internal IT. The IT view Tom is the data center manager of this insurance company. He got informed that the business wants to have a smartphone application to do all kinds of things for the new and existing customers. He is responsible to create a project and bring all necessary people on board to support this project and finally deliver the service to the business. The programming of the app will be done by an external consulting company. Tom discusses a couple of questions regarding this request with his team: How many users do we need to serve? How much time do we need to create this environment? What is the expected level of availability? How much compute power/disk space might be required? After a round of brainstorming and intense discussion, the team still is quite unsure how to answer these questions. For every question there is a couple of variables the team cannot predict. Will only a few of their thousands of users adopt to the app, what if they undersize the middleware environment? What if the user adoption rises within a couple of days, what if it lowers and the environment is over powered and therefor the cost is too high? Tom and his team identified that they need a dynamic solution to be able to serve the business request. He creates a mapping to match possible technical capabilities to the use case. After this mapping was completed, he is using it to discuss with his CIO if and how it can be implemented. Business challenge Question IT capability Easy to use app to win new customers/keep existing How many users do we need to server? Dynamic scale of an environment based on actual performance demand. How much time do we need to create this environment? To fulfill the expectations the environment needs to be flexible. Start small – scale big. What is the expected level of availability? Analytics and monitoring over all layers. Including possible self healing approach. How much compute power/disk space might be required? Create compute nodes based on actual performance requirements on demand. Introduce a capacity on demand model for required resources. Given this table, Tom revealed that with their current data center structure it is quite difficult to deliver what the business is asking for. Also, he got a couple of requirements from other departments, which are going in a similar direction. Based on these mappings, he identified that they need to change their way of deploying services and applications. They will need to use a fair amount of automation. Also, they have to span these functionalities across each data center department as a holistic approach, as shown in the following diagram: In this example, Tom actually identified a very strong use case for SDDC in his company. Based on the actual business requirements of a "simple" application, the whole IT delivery of this company needs to adopt. While this may sound like pure fiction, these are the challenges modern organizations need to face today. It is very important to identify the required capabilities for the entire data center and not just for a single department. You will also have to serve the legacy applications and bring them onto the new model. Therefore it is important to find a solution, which is serving the new business case as well as the legacy applications either way. In the first stage of any SDDC introduction in an organization, it is key to keep always an eye on the big picture. Tools to enable SDDC There is a basic and broadly accepted declaration of what a SDDC needs to offer. It can be considered as the second evolutionary step after server virtualization. It offers an abstraction layer from the infrastructure components such as compute, storage, and network by using automation and tools as such as a self service catalog In a way, it represents a virtualization of the whole data center with the purpose to simplify the request and deployment of complex services. Other capabilities of an SDDC are: Automated infrastructure/service consumption Policy based services and applications deployment Changes to services can be made easily and instantly All infrastructure layers are automated (storage, network, and compute) No human intervention is needed for infrastructure/service deployment High level of standardization is used Business logic is for chargeback or show back functionality All of the preceding points define a SDDC technically. But it is important to understand that a SDDC is considered to solve the business challenges of the organization running it. That means based on the actual business requirements, each SDDC will serve a different use case. Of course there is a main setup you can adopt and roll out – but it is important to understand your organizations business challenges in order to prevent any planning or design shortcomings. Also, to realize this functionality, SDDC needs a couple of software tools. These are designed to work together to deliver a seamless environment. The different parts can be seen like gears in a watch where each gear has an equally important role to make the clockwork function correctly. It is important to remember this when building your SDDC, since missing on one part can make another very complex or even impossible afterwards. This is a list of VMware tools building a SDDC: vRealize Business for Cloud vRealize Operations Manager vRealize Log Insight vRealize Automation vRealize Orchestrator vRealize Automation Converged Blueprint vRealize Code Stream VMware NSX VMware vSphere vRealize Business for Cloud is a charge back/show back tool. It can be used to track cost of services as well as the cost of a whole data center. Since the agility of a SDDC is much higher than for a traditional data center, it is important to track and show also the cost of adding new services. It is not only important from a financial perspective, it also serves as a control mechanism to ensure users are not deploying uncontrolled services and leaving them running even if they are not required anymore. vRealize Operations Manager is serving basically two functionalities. One is to help with the troubleshooting and analytics of the whole SDDC platform. It has an analytics engine, which applies machine learning to the behavior of its monitored components. The other important function is capacity management. It is capable of providing what-if analysis and informs about possible shortcomings of resources way before they occur. These functionalities also use the machine learning algorithms and get more accurate over time. This becomes very important in an dynamic environment where on-demand provisioning is granted. vRealize Log Insight is a unified log management. It offers rich functionality and can search and profile a large amount of log files in seconds. It is recommended to use it as a universal log endpoint for all components in your SDDC. This includes all OSes as well as applications and also your underlying hardware. In an event of error, it is much simpler to have a central log management which is easy searchable and delivers an outcome in seconds. vRealize Automation (vRA) is the base automation tool. It is providing the cloud portal to interact with your SDDC. The portal it provides offers the business logic such as service catalogs, service requests, approvals, and application life cycles. However, it relies strongly on vRealize Orchestrator for its technical automation part. vRA can also tap into external clouds to extend the internal data center. Extending a SDDC is mostly referred to as hybrid cloud. There are a couple of supported cloud offerings vRA can manage. vRealize Orchestrator (vRO) is providing the workflow engine and the technical automation part of the SDDC. It is literally the orchestrator of your new data center. vRO can be easily bound together with vRA to form a very powerful automation suite, where anything with an application programming interface (API) can be integrated. Also it is required to integrate third-party solutions into your deployment workflows, such as configuration management database (CMDB), IP address management (IPAM), or ticketing systems via IT service management (ITSM). vRealize Automation Converged Blueprint was formally known as vRealize Automation Application Services and is an add-on functionality to vRA, which takes care of application installations. It can be used with pre-existing scripts (like Windows PowerShell or Bash on Linux) – but also with variables received from vRA. This makes it very powerful when it comes to on demand application installations. This tool can also make use of vRO to provide even better capabilities for complex application installations. vRealize Code Stream is an addition to vRA and serves specific use cases in the DevOps area of the SDDC. It can be used with various development frameworks such as Jenkins. Also it can be used as a tool for developers to build and operate their own software test, QA and deployment environment. Not only can the developer build these separate stages, the migration from one stage into another can also be fully automated by scripts. This makes it a very powerful tool when it comes to stage and deploy modern and traditional applications within the SDDC. VMware NSX is the network virtualization component. Given the complexity some applications/services might introduce, NSX will provide a good and profound solution to help solving it. The challenges include: Dynamic network creation Microsegmentation Advanced security Network function virtualization VMware vSphere is mostly the base infrastructure and used as the hypervisor for server virtualization. You are probably familiar with vSphere and its functionalities. However, since the SDDC is introducing a change to you data center architecture, it is recommended to re-visit some of the vSphere functionalities and configurations. By using the full potential of vSphere it is possible to save effort when it comes to automation aspects as well as the service/application deployment part of the SDDC. This represents your toolbox required to build the platform for an automated data center. All of them will bring tremendous value and possibilities, but they also will introduce change. It is important that this change needs to be addressed and is a part of the overall SDDC design and installation effort. Embrace the change. The implementation journey While a big part of this article focuses on building and configuring the SDDC, it is important to mention that there are also non-technical aspects to consider. Creating a new way of operating and running your data center will always involve people. It is important to also briefly touch this part of the SDDC. Basically there are three major players when it comes to a fundamental change in any data center, as shown in the following image: Basically there are three major topics relevant for every successful SDDC deployment. Same as for the tools principle, these three disciplines need to work together in order to enable the change and make sure that all benefits can be fully leveraged. These three categories are: People Process Technology The process category Data center processes are as established and settled as IT itself. Beginning with the first operator tasks like changing tapes or starting procedures up to highly sophisticated processes to ensure that the service deployment and management is working as expected they have already come a long way. However, some of these processes might not be fit for purpose anymore, once automation is applied to a data center. To build a SDDC it is very important to revisit data center processes and adopt them to work with the new automation tasks. The tools will offer integration points into processes, but it is equally important to remove bottle necks for the processes as well. However, keep in mind that if you automate a bad process, the process will still be bad – but fully automated. So it is also necessary to re-visit those processes so that they can become slim and effective as well. Remember Tom, the data center manager. He has successfully identified that they need a SDDC to fulfill the business requirements and also did a use case to IT capabilities mapping. While this mapping is mainly talking about what the IT needs to deliver technically, it will also imply that the current IT processes need to adopt to this new delivery model. The process change example in Tom's organization If the compute department works on a service involving OS deployment, they need to fill out an Excel sheet with IP addresses and server names and send it to the networking department. The network admins will ensure that there is no double booking by reserving the IP address and approve the requested host name. After successfully proving the uniqueness of this data, name and IP gets added to the organizations DNS server. The manual part of this process is not longer feasible once the data center enters the automation era – imagine that every time somebody orders a service involving a VM/OS deploy, the network department gets an e-mail containing the Excel with the IP and host name combination. The whole process will have to stop until this step is manually finished. To overcome this, the process has to be changed to use an automated solution for IPAM. The new process has to track IP and host names programmatically to ensure there is no duplication within the entire data center. Also, after successfully checking the uniqueness of the data, it has to be added to the Domain Name System (DNS). While this is a simple example on one small process, normally there is a large number of processes involved which need to be re-viewed for a fully automated data center. This is a very important task and should not be underestimated since it can be a differentiator for success or failure of an SDDC. Think about all other processes in place which are used to control the deploy/enable/install mechanics in your data center. Here is a small example list of questions to ask regarding established processes: What is our current IPAM/DNS process? Do we need to consider a CMDB integration? What is our current ticketing process? (ITSM) What is our process to get resources from network, storage, and compute? What OS/VM deployment process is currently in place? What is our process to deploy an application (hand overs, steps, or departments involved)? What does our current approval process look like? Do we need a technical approval to deliver a service? Do we need a business approval to deliver a service? What integration process do we have for a service/application deployment? DNS, Active Directory (AD), Dynamic Host Configuration Protocol (DHCP), routing, Information Technology Infrastructure Library (ITIL), and so on Now for the approval question, normally these are an exception for the automation part, since approvals are meant to be manual in the first place (either technical or business). If all the other answers to this example questions involve human interaction as well, consider to change these processes to be fully automated by the SDDC. Since human intervention creates waiting times, it has to be avoided during service deployments in any automated data center. Think of it as the robotic construction bands todays car manufacturers are using. The processes they have implemented, developed over ages of experience, are all designed to stop the band only in case of an emergency. The same comes true for the SDDC – try to enable the automated deployment through your processes, stop the automation only in case of an emergency. Identifying processes is the simple part, changing them is the tricky part. However, keep in mind that this is an all new model of IT delivery, therefore there is no golden way of doing it. Once you have committed to change those processes, keep monitoring if they truly fulfill their requirement. This leads to another process principle in the SDDC: Continual Service Improvement (CSI). Re-visit what you have changed from time to time and make sure that those processes are still working as expected, if they don't, change them again. The people category Since every data center is run by people, it is important to also consider that a change of technology will also impact those people. There are some claims that a SDDC can be run with only half of the staff or save a couple of employees since all is automated. The truth is, a SDDC will transform IT roles in a data center. This means that some classic roles might vanish, while others will be added by this change. It is unrealistic to say that you can run an automated data center with half the staff than before. But it is realistic to say that your staff can concentrate on innovation and development instead of working a 100% to keep the lights on. And this is the change an automated data center introduces. It opens up the possibilities to evolve into a more architecture and design focused role for current administrators. The people example in Tom's organization Currently there are two admins in the compute department working for Tom. They are managing and maintaining the virtual environment, which is largely VMware vSphere. They are creating VMs manually, deploying an OS by a network install routine (which was a requirement for physical installs – so they kept the process) and than handing the ready VMs over to the next department to finish installing the service they are meant for. Recently they have experienced a lot of demand for VMs and each of them configures 10 to 12 VMs per day. Given this, they cannot concentrate on other aspects of their job, like improving OS deployments or the hand over process. At a first look it seems like the SDDC might replace these two employees since the tools will largely automate their work. But that is like saying a jackhammer will replace a construction worker. Actually their roles will shift to a more architectural aspect. They need to come up with a template for OS installations and an improvement how to further automate the deployment process. Also they might need to add new services/parts to the SDDC in order to fulfill the business needs continuously. So instead of creating all the VMs manually, they are now focused on designing a blueprint, able to be replicated as easy and efficient as possible. While their tasks might have changed, their workforce is still important to operate and run the SDDC. However, given that they focus on design and architectural tasks now, they also have the time to introduce innovative functions and additions to the data center. Keep in mind that an automated data center affects all departments in an IT organization. This means that also the tasks of the network and storage as well as application and database teams will change. In fact, in a SDDC it is quite impossible to still operate the departments disconnected from each other since a deployment will affect all of them. This also implies that all of these departments will have admins shifting to higher-level functions in order to make the automation possible. In the industry, this shift is also often referred to as Operational Transformation. This basically means that not only the tools have to be in place, you also have to change the way how the staff operates the data center. In most cases organizations decide to form a so-called center of excellence (CoE) to administer and operate the automated data center. This virtual group of admins in a data center is very similar to project groups in traditional data centers. The difference is that these people should be permanently assigned to the CoE for a SDDC. Typically you might have one champion from each department taking part in this virtual team. Each person acts as an expert and ambassador for their department. With this principle, it can be ensured that decisions and overlapping processes are well defined and ready to function across the departments. Also, as an ambassador, each participant should advertise the new functionalities within their department and enable their colleagues to fully support the new data center approach. It is important to have good expertise in terms of technology as well as good communication skills for each member of the CoE. The technology category This is the third aspect of the triangle to successfully implement a SDDC in your environment. Often this is the part where people spend most of their attention, sometimes by ignoring one of the other two parts. However, it is important to note that all three topics need to be equally considered. Think of it like a three legged chair, if one leg is missing it can never stand. The term technology does not necessarily only refer to new tools required to deploy services. It also refers to already established technology, which has to be integrated with the automation toolset (often referred to as third-party integration). This might be your AD, DHCP server, e-mail system, and so on. There might be technology which is not enabling or empowering the data center automation, so instead of only thinking about adding tools, there might also be tools to be removed or replaced. This is a normal IT lifecycle task and has been gone through many iterations already. Think of things like a fax machine or the telex – you might not use them anymore, they have been replaced by e-mail and messaging. The technology example in Tom's organization The team uses some tools to make their daily work easier when it comes to new service deployments. One of the tools is a little graphical user interface to quickly add content to AD. The admins use it to insert the host name, Organizational Unit as well as creating the computer account with it. This was meant to save admin time, since they don't have to open all the various menus in the AD configuration to accomplish these tasks. With the automated service delivery, this has to be done programmatically. Once a new OS is deployed it has to be added to the AD including all requirements by the deployment tool. Since AD offers an API this can be easily automated and integrated into the deployment automation. Instead of painfully integrating the graphical tool, this is now done directly by interfacing the organizations AD, ultimately replacing the old graphical tool. The automated deployment of a service across the entire data center requires a fair amount of communication. Not in a traditional way, but machine-to-machine communication leveraging programmable interfaces. Using such APIs is another important aspect of the applied data center technologies. Most of today's data center tools, from backup all the way up to web servers, do come with APIs. The better the API is documented, the easier the integration into the automation tool. In some cases you might need the vendors to support you with the integration of their tools. If you have identified a tool in the data center, which does not offer any API or even command-line interface (CLI) option at all, try to find a way around this software or even consider replacing it with a new tool. APIs are the equivalent of hand overs in the manual world. The better the communication works between tools, the faster and easier the deployment will be completed. To coordinate and control all this communication, you will need far more than scripts to run. This is a task for an orchestrator, which can run all necessary integration workflows from a central point. This orchestrator will act like a conductor for a big orchestra. It will form the backbone of your SDDC. Why are these three topics so important? The technology aspect closes the triangle and brings the people and the processes parts together. If the processes are not altered to fit the new deployment methods – automation will be painful and complex to implement. If the deployment stops at some point, since the processes require manual intervention, the people will have to fill in this gap. This means that they now have new roles, but also need to maintain some of their old tasks to keep the process running. By introducing such an unbalanced implementation of an automated data center, the workload for people can actually increase, while the service delivery times may not dramatically decrease. This may lead to an avoidance of the automated tasks, since the manual intervention might seen as faster by individual admins. So it is very important to accept all three aspects as the main part of the SDDC implementation journey. They all need to be addressed equally and thoughtfully to unveil the benefits and improvements an automated data center has to offer. However, keep in mind that this truly is a journey. A SDDC is not implemented in days but in month. Given this, also the implementation team in the data center has this time to adopt themselves and their process to this new way of delivering IT services. Also all necessary departments and their lead needs to be involved in this procedure. A SDDC implementation is always a team effort. Additional possibilities and opportunities All the previews mentioned topics serve the sole goal to install and use the SDDC within your data center. However, once you have the SDDC running the real fun begins since you can start to introduce additional functionalities impossible for any traditional data center. Lets just briefly touch on some of the possibilities from an IT view. The self-healing data center This is a concept where the automatic deployment of services is connected to a monitoring system. Once the monitoring system detects that a service or environment may be facing constraints, it can automatically trigger an additional deployment for this service to increase the throughput. While this is application dependent, for infrastructure services this can become quite handy. Think of ESXi host auto deployments if compute power is becoming a constraint, or data store deployments if disk space is running low. If this automation is acting to aggressive for your organization, it can be used with an approval function. Once the monitoring detects a shortcoming it will ask for approval to fix it with a deployment action. Instead of getting an e-mail from your monitoring system that there is a constraint identified, you get an e-mail with the constraint and the resolving action. All you need to do is to approve the action. The self-scaling data center A similar principle is to use a capacity management tool to predict the growth of your environment. If it approaches a trigger, the system can automatically generate an order letter, containing all needed components to satisfy the growing capacity demands. This can than be sent to finance or the purchasing management for approval and before you even get into any capacity constraints, the new gear might be available and ready to run. However, consider the regular turnaround time for ordering hardware, which might affect how far in the future you have to set the trigger for such functionality. Both of this opportunities are more than just nice to haves, they enable your data center to be truly flexible and proactive. Due to the fact that a SDDC is offering a high amount of agility, it will also need some self-monitoring to stay flexible and useable and to fulfill unpredicted demand. Summary In this article we discussed the main principles and declarations of an SDDC. It provided an overview of the opportunities and possibilities this new data center architecture provides. Also, it covered the changes which will be introduced by this new approach. Finally it discussed the implementation journey and its involvement with people, processes and technology. Resources for Article: Further resources on this subject: VM, It Is Not What You Think! [article] Introducing vSphere vMotion [article] Creating a VM using VirtualBox - Ubuntu Linux [article]
Read more
  • 0
  • 0
  • 9714
article-image-building-our-first-app-7-minute-workout
Packt
14 Nov 2016
27 min read
Save for later

Building Our First App – 7 Minute Workout

Packt
14 Nov 2016
27 min read
In this article by, Chandermani Arora and Kevin Hennessy, the authors of the book Angular 2 By Example, we will build a new app in Angular, and in the process, develop a better understanding of the framework. This app will also help us explore some new capabilities of the framework. (For more resources related to this topic, see here.) The topics that we will cover in this article include the following: 7 Minute Workout problem description: We detail the functionality of the app that we build. Code organization: For our first real app, we will try to explain how to organize code, specifically Angular code. Designing the model: One of the building blocks of our app is its model. We design the app model based on the app's requirements. Understanding the data binding infrastructure: While building the 7 Minute Workout view, we will look at the data binding capabilities of the framework, which include property, attribute, class, style, and event bindings. Let's get started! The first thing we will do is define the scope of our 7 Minute Workout app. What is 7 Minute Workout? We want everyone reading this article to be physically fit. Our purpose is to simulate your grey matter. What better way to do it than to build an app that targets physical fitness! 7 Minute Workout is an exercise/workout plan that requires us to perform a set of twelve exercises in quick succession within the seven minute time span. 7 Minute Workout has become quite popular due to its benefits and the short duration of the workout. We cannot confirm or refute the claims but doing any form of strenuous physical activity is better than doing nothing at all. If you are interested to know more about the workout, then check out http://well.blogs.nytimes.com/2013/05/09/the-scientific-7-minute-workout/. The technicalities of the app include performing a set of 12 exercises, dedicating 30 seconds for each of the exercises. This is followed by a brief rest period before starting the next exercise. For the app that we are building, we will be taking rest periods of 10 seconds each. So, the total duration comes out be a little more than 7 minutes. Once the 7 Minute Workout app ready, it will look something like this: Downloading the code base The code for this app can be downloaded from the GitHub site https://github.com/chandermani/angular2byexample dedicated to this article. Since we are building the app incrementally, we have created multiple checkpoints that map to GitHub branches such as checkpoint2.1, checkpoint2.2, and so on. During the narration, we will highlight the branch for reference. These branches will contain the work done on the app up to that point in time. The 7 Minute Workout code is available inside the repository folder named trainer. So let's get started! Setting up the build Remember that we are building on a modern platform for which browsers still lack support. Therefore, directly referencing script files in HTML is out of question (while common, it's a dated approach that we should avoid anyway). The current browsers do not understand TypeScript; as a matter of fact, even ES 2015 (also known as ES6) is not supported. This implies that there has to be a process that converts code written in TypeScript into standard JavaScript (ES5), which browsers can work with. Hence, having a build setup for almost any Angular 2 app becomes imperative. Having a build process may seem like overkill for a small application, but it has some other advantages as well. If you are a frontend developer working on the web stack, you cannot avoid Node.js. This is the most widely used platform for Web/JavaScript development. So, no prizes for guessing that the Angular 2 build setup too is supported over Node.js with tools such as Grunt, Gulp, JSPM, and webpack. Since we are building on the Node.js platform, install Node.js before starting. While there are quite elaborate build setup options available online, we go for a minimal setup using Gulp. The reason is that there is no one size fits all solution out there. Also, the primary aim here is to learn about Angular 2 and not to worry too much about the intricacies of setting up and running a build. Some of the notable starter sites plus build setups created by the community are as follows: Start site Location angular2-webpack-starter http://bit.ly/ng2webpack angular2-seed http://bit.ly/ng2seed angular-cli— It allows us to generate the initial code setup, including the build configurations, and has good scaffolding capabilities too. http://bit.ly/ng2-cli A natural question arises if you are very new to Node.js or the overall build process: what does a typical Angular build involve? It depends! To get an idea about this process, it would be beneficial if we look at the build setup defined for our app. Let's set up the app's build locally then. Follow these steps to have the boilerplate Angular 2 app up and running: Download the base version of this app from http://bit.ly/ng2be-base and unzip it to a location on your machine. If you are familiar with how Git works, you can just checkout the branch base: git checkout base This code serves as the starting point for our app. Navigate to the trainer folder from command line and execute these commands: npm i -g gulp npm install The first command installs Gulp globally so that you can invoke the Gulp command line tool from anywhere and execute Gulp tasks. A Gulp task is an activity that Gulp performs during the build execution. If we look at the Gulp build script (which we will do shortly), we realize that it is nothing but a sequence of tasks performed whenever a build occurs. The second command installs the app's dependencies (in the form of npm packages). Packages in the Node.js world are third-party libraries that are either used by the app or support the app's building process. For example, Gulp itself is a Node.js package. The npm is a command-line tool for pulling these packages from a central repository. Once Gulp is installed and npm pulls dependencies from the npm store, we are ready to build and run the application. From the command line, enter the following command: gulp play This compiles and runs the app. If the build process goes fine, the default browser window/tab will open with a rudimentary "Hello World" page (http://localhost:9000/index.html). We are all set to begin developing our app in Angular 2! But before we do that, it would be interesting to know what has happened under the hood. The build internals Even if you are new to Gulp, looking at gulpfile.js gives you a fair idea about what the build process is doing. A Gulp build is a set of tasks performed in a predefined order. The end result of such a process is some form of package code that is ready to be run. And if we are building our apps using TypeScript/ES2015 or some other similar language that browsers do not understand natively, then we need an additional build step, called transpilation. Code transpiling As it stands in 2016, browsers still cannot run ES2015 code. While we are quick to embrace languages that hide the not-so-good parts of JavaScript (ES5), we are still limited by the browser's capabilities. When it comes to language features, ES5 is still the safest bet as all browsers support it. Clearly, we need a mechanism to convert our TypeScript code into plain JavaScript (ES5). Microsoft has a TypeScript compiler that does this job. The TypeScript compiler takes the TypeScript code and converts it into ES5-format code that can run in all browsers. This process is commonly referred to as transpiling, and since the TypeScript compiler does it, it's called a transpiler. Interestingly, transpilation can happen at both build/compile time and runtime: Build-time transpilation: Transpilation as part of the build process takes the script files (in our case, TypeScript .ts files) and compiles them into plain JavaScript. Our build setup uses build-time transpilation. Runtime transpilation: This happens in the browser at runtime. We include the raw language-specific script files (.ts in our case), and the TypeScript compiler—which is loaded in the browser beforehand—compiles these script files on the fly.  While runtime transpilation simplifies the build setup process, as a recommendation, it should be limited to development workflows only, considering the additional performance overheard involved in loading the transpiler and transpiling the code on the fly. The process of transpiling is not limited to TypeScript. Every language targeted towards the Web such as CoffeeScript, ES2015, or any other language that is not inherently understood by a browser needs transpilation. There are transpilers for most languages, and the prominent ones (other than TypeScript) are tracuer and babel. To compile TypeScript files, we can install the TypeScript compiler manually from the command line using this: npm install -g typescript Once installed, we can compile any TypeScript file into ES5 format using the compiler (tsc.exe). But for our build setup, this process is automated using the ts2js Gulp task (check out gulpfile.js). And if you are wondering when we installed TypeScript… well, we did it as part of the npm install step, when setting up the code for the first time. The gulp-typescript package downloads the TypeScript compiler as a dependency. With this basic understanding of transpilation, we can summarize what happens with our build setup: The gulp play command kicks off the build process. This command tells Gulp to start the build process by invoking the play task. Since the play task has a dependency on the ts2js task, ts2js is executed first. The ts2js compiles the TypeScript files (.ts) located in src folder and outputs them to the dist folder at the root. Post build, a static file server is started that serves all the app files, including static files (images, videos, and HTML) and script files (check gulp.play task). Thenceforth, the build process keeps a watch on any script file changes (the gulp.watch task) you make and recompiles the code on the fly. livereload also has been set up for the app. Any changes to the code refresh the browser running the app automatically. In case browser refresh fails, we can always do a manual refresh. This is a rudimentary build setup required to run an Angular app. For complex build requirements, we can always look at the starter/seed projects that have a more complete and robust build setup, or build something of our own. Next let's look at the boilerplate app code already there and the overall code organization. Organizing code This is how we are going to organize our code and other assets for the app: The trainer folder is the root folder for the app and it has a folder (static) for the static content (such as images, CSS, audio files, and others) and a folder (src) for the app's source code. The organization of the app's source code is heavily influenced by the design of Angular and the Angular style guide (http://bit.ly/ng2-style-guide) released by the Angular team. The components folder hosts all the components that we create. We will be creating subfolders in this folder for every major component of the application. Each component folder will contain artifacts related to that component, which includes its template, its implementation and other related item. We will also keep adding more top-level folders (inside the src folder) as we build the application. If we look at the code now, the components/app folder has defined a root level component TrainerAppComponent and root level module AppModule. bootstrap.ts contains code to bootstrap/load the application module (AppModule). 7 Minute Workout uses Just In Time (JIT) compilation to compile Angular views. This implies that views are compiled just before they are rendered in the browser. Angular has a compiler running in the browser that compiles these views. Angular also supports the Ahead Of Time (AoT) compilation model. With AoT, the views are compiled on the server side using a server version of the Angular compiler. The views returned to the browser are precompiled and ready to be used. For 7 Minute Workout, we stick to the JIT compilation model just because it is easy to set up as compared to AoT, which requires server-side tweaks and package installation. We highly recommend that you use AoT compilation for production apps due the numerous benefits it offers. AoT can improve the application's initial load time and reduce its size too. Look at the AoT platform documentation (cookbook) at http://bit.ly/ng2-aot to understand how AoT compilation can benefit you. Time to start working on our first focus area, which is the app's model! The 7 Minute Workout model Designing the model for this app requires us to first detail the functional aspects of the 7 Minute Workout app, and then derive a model that satisfies those requirements. Based on the problem statement defined earlier, some of the obvious requirements are as follows: Being able to start the workout. Providing a visual clue about the current exercise and its progress. This includes the following: Providing a visual depiction of the current exercise Providing step-by-step instructions on how to do a specific exercise The time left for the current exercise Notifying the user when the workout ends. Some other valuable features that we will add to this app are as follows: The ability to pause the current workout. Providing information about the next exercise to follow. Providing audio clues so that the user can perform the workout without constantly looking at the screen. This includes: A timer click sound Details about the next exercise Signaling that the exercise is about to start Showing related videos for the exercise in progress and the ability to play them. As we can see, the central theme for this app is workout and exercise. Here, a workout is a set of exercises performed in a specific order for a particular duration. So, let's go ahead and define the model for our workout and exercise. Based on the requirements just mentioned, we will need the following details about an exercise: The name. This should be unique. The title. This is shown to the user. The description of the exercise. Instructions on how to perform the exercise. Images for the exercise. The name of the audio clip for the exercise. Related videos. With TypeScript, we can define the classes for our model. Create a folder called workout-runner inside the src/components folder and copy the model.ts file from the checkpoint2.1 branch folder workout-runner(http://bit.ly/ng2be-2-1-model-ts) to the corresponding local folder. model.ts contains the model definition for our app. The Exercise class looks like this: export class Exercise { constructor( public name: string, public title: string, public description: string, public image: string, public nameSound?: string, public procedure?: string, public videos?: Array<string>) { } } TypeScript Tips Passing constructor parameters with public or private is a shorthand for creating and initializing class members at one go. The ? suffix after nameSound, procedure, and videos implies that these are optional parameters. For the workout, we need to track the following properties: The name. This should be unique. The title. This is shown to the user. The exercises that are part of the workout. The duration for each exercise. The rest duration between two exercises. So, the model class (WorkoutPlan) looks like this: export class WorkoutPlan { constructor( public name: string, public title: string, public restBetweenExercise: number, public exercises: ExercisePlan[], public description?: string) { } totalWorkoutDuration(): number { … } } The totalWorkoutDuration function returns the total duration of the workout in seconds. WorkoutPlan has a reference to another class in the preceding definition—ExercisePlan. It tracks the exercise and the duration of the exercise in a workout, which is quite apparent once we look at the definition of ExercisePlan: export class ExercisePlan { constructor( public exercise: Exercise, public duration: number) { } } These three classes constitute our base model, and we will decide in the future whether or not we need to extend this model as we start implementing the app's functionality. Since we have started with a preconfigured and basic Angular app, you just need to understand how this app bootstrapping is occurring. App bootstrapping Look at the src folder. There is a bootstrap.ts file with only the execution bit (other than imports): platformBrowserDynamic().bootstrapModule(AppModule); The boostrapModule function call actually bootstraps the application by loading the root module, AppModule. The process is triggered by this call in index.html: System.import('app').catch(console.log.bind(console)); The System.import statement sets off the app bootstrapping process by loading the first module from bootstrap.ts. Modules defined in the context of Angular 2, (using @NgModule decorator) are different from modules SystemJS loads. SystemJS modules are JavaScript modules, which can be in different formats adhering to CommonJS, AMD, or ES2015 specifications. Angular modules are constructs used by Angular to segregate and organize its artifacts. Unless the context of discussion is SystemJS, any reference to module implies Angular module. Now we have details on how SystemJS loads our Angular app. App loading with SystemJS SystemJS starts loading the JavaScript module with the call to System.import('app') in index.html. SystemJS starts by loading bootstrap.ts first. The imports defined inside bootstrap.ts cause SystemJS to then load the imported modules. If these module imports have further import statements, SystemJS loads them too, recursively. And finally the platformBrowserDynamic().bootstrapModule(AppModule); function gets executed once all the imported modules are loaded. For the SystemJS import function to work, it needs to know where the module is located. We define this in the file, systemjs.config.js, and reference it in index.html, before the System.import script: <script src="systemjs.config.js"></script> This configuration file contains all of the necessary configuration for SystemJS to work correctly. Open systemjs.config.js, the app parameter to import function points to a folder dist as defined on the map object: var map = { 'app': 'dist', ... } And the next variable, packages, contains settings that hint SystemJS how to load a module from a package when no filename/extension is specified. For app, the default module is bootstrap.js: var packages = { 'app': { main: 'bootstrap.js', defaultExtension: 'js' }, ... }; Are you wondering what the dist folder has to do with our application? Well, this is where our transpiled scripts end up.  As we build our app in TypeScript, the TypeScript compiler converts these .ts script files in the src folder to JavaScript modules and deposits them into the dist folder. SystemJS then loads these compiled JavaScript modules. The transpiled code location has been configured as part of the build definition in gulpfile.js. Look for this excerpt in gulpfile.ts: return tsResult.js .pipe(sourcemaps.write()) .pipe(gulp.dest('dist')) The module specification used by our app can again be verified in gulpfile.js. Take a look at this line: noImplicitAny: true, module: 'system', target: 'ES5', These are TypeScript compiler options, with one being module, that is, the target module definition format. The system module type is a new module format designed to support the exact semantics of ES2015 modules within ES5. Once the scripts are transpiled and the module definitions created (in the target format), SystemJS can load these modules and their dependencies. It's time to get into the thick of action; let's build our first component. Our first component – WorkoutRunnerComponent To implement the WorkoutRunnerComponent, we need to outline the behavior of the application. What we are going to do in the WorkoutRunnerComponent implementation is as follows: Start the workout. Show the workout in progress and show the progress indicator. After the time elapses for an exercise, show the next exercise. Repeat this process until all the exercises are over. Let's start with the implementation. The first thing that we will create is the WorkoutRunnerComponent implementation. Open workout-runner folder in the src/components folder and add a new code file called workout-runner.component.ts to it. Add this chunk of code to the file: import {WorkoutPlan, ExercisePlan, Exercise} from './model' export class WorkoutRunnerComponent { } The import module declaration allows us to reference the classes defined in the model.ts file in WorkoutRunnerComponent. We first need to set up the workout data. Let's do that by adding a constructor and related class properties to the WorkoutRunnerComponent class: workoutPlan: WorkoutPlan; restExercise: ExercisePlan; constructor() { this.workoutPlan = this.buildWorkout(); this.restExercise = new ExercisePlan( new Exercise("rest", "Relax!", "Relax a bit", "rest.png"), this.workoutPlan.restBetweenExercise); } The buildWorkout on WorkoutRunnerComponent sets up the complete workout, as we will see shortly. We also initialize a restExercise variable to track even the rest periods as exercise (note that restExercise is an object of type ExercisePlan). The buildWorkout function is a lengthy function, so it's better if we copy the implementation from the workout runner's implementation available in Git branch checkpoint2.1 (http://bit.ly/ng2be-2-1-workout-runner-component-ts). The buildWorkout code looks like this: buildWorkout(): WorkoutPlan { let workout = new WorkoutPlan("7MinWorkout", "7 Minute Workout", 10, []); workout.exercises.push( new ExercisePlan( new Exercise( "jumpingJacks", "Jumping Jacks", "A jumping jack or star jump, also called side-straddle hop is a physical jumping exercise.", "JumpingJacks.png", "jumpingjacks.wav", `Assume an erect position, with feet together and arms at your side. …`, ["dmYwZH_BNd0", "BABOdJ-2Z6o", "c4DAnQ6DtF8"]), 30)); // (TRUNCATED) Other 11 workout exercise data. return workout; } This code builds the WorkoutPlan object and pushes the exercise data into the exercises array (an array of ExercisePlan objects), returning the newly built workout. The initialization is complete; now, it's time to actually implement the start workout. Add a start function to the WorkoutRunnerComponent implementation, as follows: start() { this.workoutTimeRemaining = this.workoutPlan.totalWorkoutDuration(); this.currentExerciseIndex = 0; this.startExercise(this.workoutPlan.exercises[this.currentExerciseIndex]); } Then declare the new variables used in the function at the top, with other variable declarations: workoutTimeRemaining: number; currentExerciseIndex: number; The workoutTimeRemaining variable tracks the total time remaining for the workout, and currentExerciseIndex tracks the currently executing exercise index. The call to startExercise actually starts an exercise. This how the code for startExercise looks: startExercise(exercisePlan: ExercisePlan) { this.currentExercise = exercisePlan; this.exerciseRunningDuration = 0; let intervalId = setInterval(() => { if (this.exerciseRunningDuration >= this.currentExercise.duration) { clearInterval(intervalId); } else { this.exerciseRunningDuration++; } }, 1000); } We start by initializing currentExercise and exerciseRunningDuration. The currentExercise variable tracks the exercise in progress and exerciseRunningDuration tracks its duration. These two variables also need to be declared at the top: currentExercise: ExercisePlan; exerciseRunningDuration: number; We use the setInterval JavaScript function with a delay of 1 second (1,000 milliseconds) to track the exercise progress by incrementing exerciseRunningDuration. The setInterval invokes the callback every second. The clearInterval call stops the timer once the exercise duration lapses. TypeScript Arrow functions The callback parameter passed to setInterval (()=>{…}) is a lambda function (or an arrow function in ES 2015). Lambda functions are short-form representations of anonymous functions, with added benefits. You can learn more about them at https://basarat.gitbooks.io/typescript/content/docs/arrow-functions.html. As of now, we have a WorkoutRunnerComponent class. We need to convert it into an Angular component and define the component view. Add the import for Component and a component decorator (highlighted code): import {WorkoutPlan, ExercisePlan, Exercise} from './model' import {Component} from '@angular/core'; @Component({ selector: 'workout-runner', template: ` <pre>Current Exercise: {{currentExercise | json}}</pre> <pre>Time Left: {{currentExercise.duration- exerciseRunningDuration}}</pre>` }) export class WorkoutRunnerComponent { As you already know how to create an Angular component. You understand the role of the @Component decorator, what selector does, and how the template is used. The JavaScript generated for the @Component decorator contains enough metadata about the component. This allows Angular framework to instantiate the correct component at runtime. String enclosed in backticks (` `) are a new addition to ES2015. Also called template literals, such string literals can be multi line and allow expressions to be embedded inside (not to be confused with Angular expressions). Look at the MDN article here at http://bit.ly/template-literals for more details. The preceding template HTML will render the raw ExercisePlan object and the exercise time remaining. It has an interesting expression inside the first interpolation: currentExercise | json. The currentExercise property is defined in WorkoutRunnerComponent, but what about the | symbol and what follows it (json)? In the Angular 2 world, it is called a pipe. The sole purpose of a pipe is to transform/format template data. The json pipe here does JSON data formatting. A general sense of what the json pipe does, we can remove the json pipe plus the | symbol and render the template; we are going to do this next. As our app currently has only one module (AppModule), we add the WorkoutRunnerComponent declaration to it. Update app.module.ts by adding the highlighted code: import {WorkoutRunnerComponent} from '../workout-runner/workout-runner.component'; @NgModule({ imports: [BrowserModule], declarations: [TrainerAppComponent, WorkoutRunnerComponent], Now WorkoutRunnerComponent can be referenced in the root component so that it can be rendered. Modify src/components/app/app.component.ts as highlighted in the following code: @Component({ ... template: ` <div class="navbar ...> ... </div> <div class="container ...> <workout-runner></workout-runner> </div>` }) We have changed the root component template and added the workout-runner element to it. This will render the WorkoutRunnerComponent inside our root component. While the implementation may look complete there is a crucial piece missing. Nowhere in the code do we actually start the workout. The workout should start as soon as we load the page. Component life cycle hooks are going to rescue us! Component life cycle hooks Life of an Angular component is eventful. Components get created, change state during their lifetime and finally they are destroyed. Angular provides some life cycle hooks/functions that the framework invokes (on the component) when such event occurs. Consider these examples: When component is initialized, Angular invokes ngOnInit When a component's input properties change, Angular invokes ngOnChanges When a component is destroyed, Angular invokes ngOnDestroy As developers, we can tap into these key moments and perform some custom logic inside the respective component. Angular has TypeScript interfaces for each of these hooks that can be applied to the component class to clearly communicate the intent. For example: class WorkoutRunnerComponent implements OnInit { ngOnInit (){ ... } ... The interface name can be derived by removing the prefix ng from the function names. The hook we are going to utilize here is ngOnInit. The ngOnInit function gets fired when the component's data-bound properties are initialized but before the view initialization starts. Add the ngOnInit function to the WorkoutRunnerComponent class with a call to start the workout: ngOnInit() { this.start(); } And implement the OnInit interface on WorkoutRunnerComponent; it defines the ngOnInit method: import {Component,OnInit} from '@angular/core'; … export class WorkoutRunnerComponent implements OnInit { There are a number of other life cycle hooks, including ngOnDestroy, ngOnChanges, and ngAfterViewInit, that components support; but we are not going to dwell into any of them here. Look at the developer guide (http://bit.ly/ng2-lifecycle) on Life cycle Hooks to learn more about other such hooks. Time to run our app! Open the command line, navigate to the trainer folder, and type this line: gulp play If there are no compilation errors and the browser automatically loads the app (http://localhost:9000/index.html), we should see the following output: The model data updates with every passing second! Now you'll understand why interpolations ({{ }}) are a great debugging tool. This will also be a good time to try rendering currentExercise without the json pipe (use {{currentExercise}}), and see what gets rendered. We are not done yet! Wait long enough on the index.html page and you will realize that the timer stops after 30 seconds. The app does not load the next exercise data. Time to fix it! Update the code inside the setInterval if condition: if (this.exerciseRunningDuration >= this.currentExercise.duration) { clearInterval(intervalId); let next: ExercisePlan = this.getNextExercise(); if (next) { if (next !== this.restExercise) { this.currentExerciseIndex++; } this.startExercise(next); } else { console.log("Workout complete!"); } } The if condition if (this.exerciseRunningDuration >= this.currentExercise.duration) is used to transition to the next exercise once the time duration of the current exercise lapses. We use getNextExercise to get the next exercise and call startExercise again to repeat the process. If no exercise is returned by the getNextExercise call, the workout is considered complete. During exercise transitioning, we increment currentExerciseIndex only if the next exercise is not a rest exercise. Remember that the original workout plan does not have a rest exercise. For the sake of consistency, we have created a rest exercise and are now swapping between rest and the standard exercises that are part of the workout plan. Therefore, currentExerciseIndex does not change when the next exercise is rest. Let's quickly add the getNextExercise function too. Add the function to the WorkoutRunnerComponent class: getNextExercise(): ExercisePlan { let nextExercise: ExercisePlan = null; if (this.currentExercise === this.restExercise) { nextExercise = this.workoutPlan.exercises[this.currentExerciseIndex + 1]; } else if (this.currentExerciseIndex < this.workoutPlan.exercises.length - 1) { nextExercise = this.restExercise; } return nextExercise; } The WorkoutRunnerComponent.getNextExercise returns the next exercise that needs to be performed. Note that the returned object for getNextExercise is an ExercisePlan object that internally contains the exercise details and the duration for which the exercise runs. The implementation is quite self-explanatory. If the current exercise is rest, take the next exercise from the workoutPlan.exercises array (based on currentExerciseIndex); otherwise, the next exercise is rest, given that we are not on the last exercise (the else if condition check). With this, we are ready to test our implementation. So go ahead and refresh index.html. Exercises should flip after every 10 or 30 seconds. Great! The current build setup automatically compiles any changes made to the script files when the files are saved; it also refreshes the browser post these changes. But just in case the UI does not update or things do not work as expected, refresh the browser window. If you are having a problem with running the code, look at the Git branch checkpoint2.1 for a working version of what we have done thus far. Or if you are not using Git, download the snapshot of checkpoint2.1 (a zip file) from http://bit.ly/ng2be-checkpoint2-1. Refer to the README.md file in the trainer folder when setting up the snapshot for the first time. We have are done with controller. Summary We started this article with the aim of creating an Angular app which is  complex. The 7 Minute Workout app fitted the bill, and you learned a lot about the Angular framework while building this app. We started by defining the functional specifications of the 7 Minute Workout app. We then focused our efforts on defining the code structure for the app. To build the app, we started off by defining the model of the app. Once the model was in place, we started the actual implementation, by building an Angular component. Angular components are nothing but classes that are decorated with a framework-specific decorator, @Component. We now have a basic 7 Minute Workout app. For a better user experience, we have added a number of small enhancements to it too, but we are still missing some good-to-have features that would make our app more usable. Resources for Article: Further resources on this subject: Angular.js in a Nutshell [article] Introduction to JavaScript [article] Create Your First React Element [article] <hr noshade="noshade" size="1"
Read more
  • 0
  • 0
  • 13971

article-image-fundamental-selinux-concepts
Packt
14 Nov 2016
41 min read
Save for later

Fundamental SELinux Concepts

Packt
14 Nov 2016
41 min read
In this article by Sven Vermeulen, the author of the book SELinux System Administration Second Edition, we will see how Security Enhanced Linux (SELinux) brings additional security measures for your Linux system to further protect the resources on the system. This article explains why SELinux has opted to use labels to identify resources, the way SELinux differentiates itself from regular Linux access controls through the enforcement of security rules, how the access control rules enforced by SELinux are provided through policy files, the different SELinux implementations between Linux distributions. (For more resources related to this topic, see here.) Providing more security to Linux Seasoned Linux administrators and security engineers already know that they need to put some trust in the users and processes on their system in order for the system to remain secure. This is partially because users can attempt to exploit vulnerabilities found in the software running on the system, but a large contribution to this trust level is because the secure state of the system depends on the behavior of the users. A Linux user with access to sensitive information could easily leak that out to the public, manipulate the behavior of the applications he or she launches, and do many other things that affect the security of the system. The default access controls that are active on a regular Linux system are discretionary; it is up to the user's how the access controls should behave. The Linux discretionary access control (DAC) mechanism is based on the user and/or group information of the process and is matched against the user and/or group information of the file, directory, or other resource being manipulated. Consider the /etc/shadow file, which contains the password and account information of the local Linux accounts: $ ls -l /etc/shadow -rw------- 1 root root 1010 Apr 25 22:05 /etc/shadow Without additional access control mechanisms in place, this file is readable and writable by any process that is owned by the root user, regardless of the purpose of the process on the system. The shadow file is a typical example of a sensitive file that we don't want to see leaked or abused in any other fashion. Yet, the moment someone has access to the file, they can copy it elsewhere, for example to a home directory, or even mail it to a different computer and attempt to attack the password hashes stored within. Another example of how Linux DAC requires trust from its users is when a database is hosted on the system. Database files themselves are (hopefully) only accessible to runtime users of the database management system (DBMS) and the Linux root user. Properly secured systems will only grant trusted users access to these files (for instance, through sudo) by allowing them to change their effective user ID from their personal user to database runtime user or even root account for a well-defined set of commands. These users too can analyze the database files and gain access to potentially confidential information in the database without going through the DBMS. However, regular users are not the only reason for securing a system. Lots of software daemons run as the Linux root user or have significant privileges on the system. Errors within those daemons can easily lead to information leakage or might even lead to exploitable remote command execution vulnerabilities. Backup software, monitoring software, change management software, scheduling software, and so on: they all often run with the highest privileged account possible on a regular Linux system. Even when the administrator does not allow privileged users, their interaction with daemons induces a potential security risk. As such, the users are still trusted to correctly interact with these applications in order for the system to function properly. Through this, the administrator leaves the security of the system to the discretion of its (many) users. Enter SELinux, which provides an additional access control layer on top of the standard Linux DAC mechanism. SELinux provides a mandatory access control (MAC) system that, unlike its DAC counterpart, gives the administrator full control over what is allowed on the system and what isn't. It accomplishes this by supporting a policy-driven approach over what processes are and aren't allowed to do and by enforcing this policy through the Linux kernel. Mandatory means that access control is enforced by the operating system and defined solely by the administrator. Users and processes do not have permission to change the security rules, so they cannot work around the access controls; security is not left to their discretion anymore. The word mandatory here, just like the word discretionary before, was not chosen by accident to describe the abilities of the access control system: both are known terms in the security research field and have been described in many other publications, including the Trusted Computer System Evaluation Criteria (TCSEC) (http://csrc.nist.gov/publications/history/dod85.pdf) standard (also known as the Orange Book) by the Department of Defense in the United States of America in 1985. This publication has led to the common criteria standard for computer security certification (ISO/IEC 15408), available at http://www.commoncriteriaportal.org/cc/. Using Linux security modules Consider the example of the shadow file again. A MAC system can be configured to only allow a limited number of processes to read and write to the file. A user logged on as root cannot directly access the file or even move it around. He can't even change the attributes of the file: # id uid=0(root) gid=0(root) # cat /etc/shadow cat: /etc/shadow: Permission denied # chmod a+r /etc/shadow chmod: changing permissions of '/etc/shadow': Permission denied This is enforced through rules that describe when the contents of a file can be read. With SELinux, these rules are defined in the SELinux policy and are loaded when the system boots. It is the Linux kernel itself that is responsible for enforcing the rules. Mandatory access control systems such as SELinux can be easily integrated into the Linux kernel through its support for Linux Security Modules (LSM): High-level overview of how LSM is integrated into the Linux kernel LSM has been available in the Linux kernel since version 2.6, sometime in December 2003. It is a framework that provides hooks inside the Linux kernel to various locations, including the system call entry points, and allows a security implementation such as SELinux to provide functions to be called when a hook is triggered. These functions can then do their magic (for instance, checking the policy and other information) and give a go/no-go back to allow the call to go through or not. LSM by itself does not provide any security functionality; instead, it relies on security implementations that do the heavy lifting. SELinux is one of the implementations that use LSM, but there are several others: AppArmor, Smack, TOMOYO Linux, and Yama, to name a few. At the time of writing this, only one main security implementation can be active through the LSM hooks. Work is underway to enable stacking multiple security implementations, allowing system administrators to have more than one implementation active. Recent work has already allowed multiple implementations to be defined (but not simultaneously active). When supported, this will allow administrators to pick the best features of a number of implementations and enforce smaller LSM-implemented security controls on top of the more complete security model implementations, such as SELinux, TOMOYO, Smack, or AppArmor. Extending regular DAC with SELinux SELinux does not change the Linux DAC implementation nor can it override denials made by the Linux DAC permissions. If a regular system (without SELinux) prevents a particular access, there is nothing SELinux can do to override this decision. This is because the LSM hooks are triggered after the regular DAC permission checks have been executed. For instance, if you need to allow an additional user access to a file, you cannot add an SELinux policy to do that for you. Instead, you will need to look into other features of Linux, such as the use of POSIX access control lists. Through the setfacl and getfacl commands (provided by the acl package) the user can set additional permissions on files and directories, opening up the selected resource to additional users or groups. As an example, let's grant user lisa read-write access to a file using setfacl: $ setfacl -m u:lisa:rw /path/to/file Similarly, to view the current POSIX ACLs applied to the file, use this command: $ getfacl /path/to/file # file: file # owner: swift # group: swift user::rw- user:lisa:rw- group::r-- mask::r-- other::r-- Restricting root privileges The regular Linux DAC allows for an all-powerful user: root. Unlike most other users on the system, the logged-on root user has all the rights needed to fully manage the entire system, ranging from overriding access controls to controlling audits, changing user IDs, managing the network, and much more. This is supported through a security concept called capabilities (for an overview of Linux capabilities, check out the capabilities manual page: man capabilities). SELinux is also able to restrict access to these capabilities in a fine-grained manner. Due to this fine-grained authorization aspect of SELinux, even the root user can be confined without impacting the operations on the system. The aforementioned example of accessing /etc/shadow is just one example of a restriction that a powerful user as root still might not be able to make due to the SELinux access controls being in place. When SELinux was added to the mainstream Linux kernel, some security projects even went as far as providing public root shell access to an SELinux-protected system, asking hackers and other security researchers to compromise the box. The ability to restrict root was welcomed by system administrators who sometimes need to pass on the root password or root shell to other users (for example, database administrators) who needed root privileges when their software went haywire. Thanks to SELinux, the administrator can now pass on a root shell while resting assured that the user only has those rights he needs, and not full system-administration rights. Reducing the impact of vulnerabilities If there is one benefit of SELinux that needs to be stressed, while often also being misunderstood, it is its ability to reduce the impact of vulnerabilities. A properly written SELinux policy confines applications so that their allowed activities are reduced to a minimum set. This least-privilege model ensures that abnormal application behavior is not only detected and audited but also prevented. Many application vulnerabilities can be exploited to execute tasks that an application is not meant to do. When this happens, SELinux will prevent this. However, there are two misconceptions about SELinux state and its ability to thwart exploits, namely, the impact of the policy and the exploitation itself. If the policy is not written in a least-privilege model, then SELinux might consider this nonstandard behavior as normal and allow the actions to continue. For policy writers, this means that their policy code has to be very fine-grained. Sadly, that makes writing policies very time-consuming; there are more than 80 classes and over 200 permissions known to SELinux, and policy rules need to take into account all these classes and permissions for each interaction between two objects or resources. As a result, policies tend to become convoluted and harder to maintain. Some policy writers make the policies more permissive than is absolutely necessary, which might result in exploits becoming successful even though the action is not expected behavior from an application point of view. Some application policies are explicitly marked as unconfined (which is discussed later in this article), showing that they are very liberal in their allowed permissions. Red Hat Enterprise Linux even has several application policies as completely permissive, and it starts enforcing access controls for those applications only after a few releases. The second misconception is the exploit itself. If an application's vulnerability allows an unauthenticated user to use the application services as if he were authorized, SELinux will not play a role in reducing the impact of the vulnerability; it only notices the behavior of the application itself and not of the sessions internal to the application. As long as the application itself behaves as expected (accessing its own files and not poking around in other filesystems), SELinux will happily allow the actions to take place. It is only when the application starts behaving erratically that SELinux stops the exploit from continuing. Exploits such as remote command execution (RCE) against applications that should not be executing random commands (such as database management systems or web servers, excluding CGI-like functionality) will be prevented, whereas session hijacking or SQL injection attacks are not controllable through SELinux policies. Enabling SELinux support Enabling SELinux on a Linux system is not just a matter of enabling the SELinux LSM module within the Linux kernel. An SELinux implementation comprises the following: The SELinux kernel subsystem, implemented in the Linux kernel through LSM Libraries, used by applications that need to interact with SELinux Utilities, used by administrators to interact with SELinux Policies, which define the access controls themselves The libraries and utilities are bundled by the SELinux user space project (https://github.com/SELinuxProject/selinux/wiki). Next to the user space applications and libraries, various components on a Linux system are updated with SELinux-specific code, including the init system and several core utilities. Because SELinux isn't just a switch that needs to be toggled, Linux distributions that support SELinux usually come with SELinux predefined and loaded: Fedora and Red Hat Enterprise Linux (with its derivatives, such as CentOS and Oracle Linux) are well-known examples. Other supporting distributions might not automatically have SELinux enabled but can easily support it through the installation of additional packages (which is the case with Debian and Ubuntu), and others have a well-documented approach on how to convert a system to SELinux (for example, Gentoo and Arch Linux). Labeling all resources and objects When SELinux has to decide whether it has to allow or deny a particular action, it makes a decision based on the context of both the subject (which is initiating the action) and the object (which is the target of the action). These contexts (or parts of the context) are mentioned in the policy rules that SELinux enforces. The context of a process is what identifies the process to SELinux. SELinux has no notion of Linux process ownership and, frankly, does not care how the process is called, which process ID it has, and what account the process runs as. All it wants to know is what the context of that process is, which is represented to users and administrators as a label. Label and context are often used interchangeably, and although there is a  technical distinction (one is a representation of the other), we will not dwell on that much. Let's look at an example label: the context of the current user (try it out yourself if you are on an SELinux-enabled system): $ id -Z unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 The id command, which returns information about the current user, is executed here with the -z switch (a commonly agreed-upon switch for displaying SELinux information). It shows us the context of the current user (actually the context of the id process itself when it was executing). As we can see, the context has a string representation and looks as if it has five fields (it doesn't; it has four fields—the last field just happens to contain a :). SELinux developers decided to use labels instead of real process and file (or other resource) metadata for its access controls. This is different to MAC systems such as AppArmor, which use the path of the binary (and thus the process name) and the paths of the resources to handle permission checks. The decision to make SELinux a label-based mandatory access control was taken for various reasons, which are as follows: Using paths might be easier to comprehend for administrators, but this doesn't allow us to keep the context information close to the resource. If a file or directory is moved or remounted or a process has a different namespace view on the files, then the access controls might behave differently. With label-based contexts, this information is retained and the system keeps controlling the resource properly. Contexts reveal the purpose of the process very well. The same binary application can be launched in different contexts depending on how it got started. The context value (such as the one shown in the id -Z output earlier) is exactly what the administrator needs. With it, he knows what the rights are of each of the running instances, but he can also deduce from it how the process might have been launched and what its purpose is. Contexts also make abstractions of the object itself. We are used to talking about processes and files, but contexts are also applicable to less tangible resources such as pipes (interprocess communication) or database objects. Path-based identification only works as long as you can write a path. As an example, consider the following policies: Allow the httpd processes to bind to TCP port 80 Allow the processes labeled with httpd_t to bind to TCP ports labeled with http_port_t In the first example, we cannot easily reuse this policy when the web server process isn't using the httpd binary (perhaps because it was renamed or it isn't Apache but another web server) or when we want to have HTTP access on a different port. With the labeled approach, the binary can be called apache2 or MyWebServer.py; as long as the process is labeled httpd_t, the policy applies. The same happens with the port definition: you can label port 8080 with http_port_t and thus allow the web servers to bind to that port as well. Dissecting the SELinux context To come to a context, SELinux uses at least three, and sometimes four, values. Let's look at the context of an Apache web server as an example: $ ps -eZ | grep httpd system_u:system_r:httpd_t:s0 511 ? 00:00:00 httpd As we can see, the process is assigned a context that contains following fields: system_u: This represents the SELinux user system_r: This represents the SELinux role httpd_t: This represents the SELinux type (also known as the domain in the case of a process) s0: This represents the sensitivity level This structure can be depicted as follows: The structure of a SELinux context, using the id -Z output as an example When we work with SELinux, contexts are all we need. In the majority of cases, it is the third field (called the domain or type) that is most important since the majority of SELinux policy rules (over 99 percent) consist of rules related to the interaction between two types (without mentioning roles, users, or sensitivity levels). SELinux contexts are aligned with LSM security attributes and exposed to the user space, allowing end users and applications to easily query the contexts. An interesting place where these attributes are presented is within the /proc pseudo filesystem. Inside each process's /proc/<pid> location, we find a subdirectory called attr, inside of which the following files can be found: $ ls /proc/$$/attr current fscreate prev exec keycreate sockcreate All these files, if read, display either nothing or an SELinux context. If it is empty, then that means the application has not explicitly set a context for that particular purpose, and the SELinux context will be deduced either from the policy or inherited from its parent. The meaning of the files are as follows: The current file displays the current SELinux context of the process. The exec file displays the SELinux context that will be assigned by the next application execution done through this application. It is usually empty. The fscreate file displays the SELinux context that will be assigned to the next file that is written by the application. It is usually empty. The keycreate file displays the SELinux context that will be assigned to the keys cached in the kernel by this application. It is usually empty. The prev file displays the previous SELinux context for this particular process. This is usually the context of its parent application. The sockcreate file displays the SELinux context that will be assigned to the next socket created by the application. It is usually empty. If an application has multiple subtasks, then the same information is available in each subtask directory at /proc/<pid>/task/<taskid>/attr. Enforcing access through types The SELinux type (the third part of an SELinux context) of a process (called the domain) is the basis of the fine-grained access controls of that process with respect to itself and other types (which can be processes, files, sockets, network interfaces, and more). In most SELinux literature, the SELinux label-based access control mechanism is fine-tuned to say that SELinux is a type enforcement mandatory access control system: when some actions are denied, the fine-grained access controls on the type level are most likely to blame. With type enforcement, SELinux is able to control what an application is allowed to do based on how it got executed in the first place: a web server that is launched interactively by a user will run with a different type than a web server executed through the init system, even though the process binary and path are the same. The web server launched from the init system is most likely trusted (and thus allowed to do whatever web servers are supposed to do), whereas a manually launched web server is less likely to be considered normal behavior and as such will have different privileges. The majority of SELinux resources will focus on types. Even though the SELinux type is just the third part of an SELinux context, it is the most important one for most administrators. Most documentation will even just talk about a type such as httpd_t rather than a full SELinux context. Take a look at the following dbus-daemon processes: # ps -eZ | grep dbus-daemon system_u:system_r:system_dbusd_t 4531 ? 00:00:00 dbus-daemon staff_u:staff_r:staff_dbusd_t 5266 ? 00:00:00 dbus-daemon In this example, one dbus-daemon process is the system D-Bus daemon running with the aptly named system_dbusd_t type, whereas another one is running with the staff_dbusd_t type assigned to it. Even though their binaries are completely the same, they both serve a different purpose on the system and as such have a different type assigned. SELinux then uses this type to govern the actions allowed by the process towards other types, including how system_dbusd_t can interact with staff_dbusd_t. SELinux types are by convention suffixed with _t, although this is not mandatory. Granting domain access through roles SELinux roles (the second part of an SELinux context) allow SELinux to support role-based access controls. Although type enforcement is the most used (and known) part of SELinux, role-based access control is an important method to keep a system secure, especially from malicious user attempts. SELinux roles are used to define which process types (domains) user processes can be in. As such, they help define what a user can and cannot do. By convention, SELinux roles are defined with an _r suffix. On most SELinux-enabled systems, the following roles are made available to be assigned to users: user_r This role is meant for restricted users: the user_r SELinux role is only allowed to have processes with types specific to end-user applications. Privileged types, including those used to switch to another Linux user, are not allowed for this role. staff_r This role is meant for non-critical operations: the SELinux staff_r role is generally restricted to the same applications as the restricted user, but it has the ability to switch roles. It is the default role for operators to be in (so as to keep those users in the least privileged role as long as possible). sysadm_r This role is meant for system administrators: the sysadm_r SELinux role is very privileged, enabling various system-administration tasks. However, certain end-user application types might not be supported (especially if those types are used for potentially vulnerable or untrusted software) to keep the system free from infections. system_r This role is meant for daemons and background processes: the system_r SELinux role is quite privileged, supporting the various daemon and system process types. However, end-user application types and other administrative types are not allowed in this role. unconfined_r This role is meant for end users: the unconfined_r role is allowed a limited number of types, but those types are very privileged as it is meant for running any application launched by a user in a more or less unconfined manner (not restricted by SELinux rules). This role as such is only available if the system administrator wants to protect certain processes (mostly daemons) while keeping the rest of the system operations almost untouched by SELinux. Other roles might be supported as well, such as guest_r and xguest_r, depending on the distribution. It is wise to consult the distribution documentation for more information about the supported roles. An overview of available roles can be obtained through the seinfo command (part of setools-console in RHEL or app-admin/setools in Gentoo): # seinfo --role Roles: 14 auditadm_r dbadm_r ... unconfined_r Limiting roles through users An SELinux user (the first part of an SELinux context) is different from a Linux user. Unlike Linux user information, which can change while the user is working on the system (through tools such as sudo or su), the SELinux policy can (and generally will) enforce that the SELinux user remain the same even when the Linux user itself has changed. Because of the immutable state of the SELinux user, specific access controls can be implemented to ensure that users cannot work around the set of permissions granted to them, even when they get privileged access. An example of such an access control is the user-based access control (UBAC) feature that some Linux distributions (optionally) enable, which prevents users from accessing files of different SELinux users even when those users try to use the Linux DAC controls to open up access to each other's files. The most important feature of SELinux users, however, is that SELinux user definitions restrict which roles the (Linux) user is allowed to be in. A Linux user is first assigned to an SELinux user—multiple Linux users can be assigned to the same SELinux user. Once set, that user cannot switch to an SELinux role he isn't meant to be in. This is the role-based access control implementation of SELinux: Mapping Linux accounts to SELinux users SELinux users are, by convention, defined with a _u suffix, although this is not mandatory. The SELinux users that most distributions have available are named after the role they represent, but instead of ending with _r, they end with _u. For instance, for the sysadm_r role, there is a sysadm_u SELinux user. Controlling information flow through sensitivities The fourth part of an SELinux context, the sensitivity, is not always present (some Linux distributions by default do not enable sensitivity labels). If they are present though, then this part of the label is needed for the multi-level security (MLS) support within SELinux. Sensitivity labels allow classification of resources and restriction of access to those resources based on a security clearance. These labels consist of two parts: a confidentiality value (prefixed with s) and a category value (prefixed with c). In many larger organizations and companies, documents are labeled internal, confidential, or strictly confidential. SELinux can assign processes a certain clearance level towards these resources. With MLS, SELinux can be configured to follow the Bell-LaPadula model, a security model that can be characterized by no read up and no write down: based on a process clearance level, that process cannot read anything with a higher confidentiality level nor write to (or communicate otherwise with) any resource with a lower confidentiality level. SELinux does not use the internal, confidential, and other labels. Instead, it uses numbers from 0 (lowest confidentiality) to whatever the system administrator has defined as the highest value (this is configurable and set when the SELinux policy is built). Categories allow resources to be tagged with one or more categories, on which access controls are also possible. The idea behind categories is to support multitenancy (for example, systems hosting applications for multiple customers) within a Linux system, by having processes and resources belonging to one tenant to be assigned a particular set of categories, whereas the processes and resources of another tenant get a different set of categories. When a process does not have proper categories assigned, it cannot do anything with the resources (or other processes) that have other categories assigned. An unwritten convention in the SELinux world is that (at least) two categories are used to differentiate between tenants. By having services randomly pick two categories for a tenant out of a predefined set of categories, while ensuring each tenant has a unique combination, these services receive proper isolation. The use of two categories is not mandatory but is implemented by services such as sVirt and Docker. In that sense, categories can be seen as tags, allowing access to be granted only when the tags of the process and the target resource match. As multilevel security is not often used, the benefits of only using categories is persisted in what is called multi-category security (MCS). This is a special MLS case, where only a single confidentiality level is supported (s0). Defining and distributing policies Enabling SELinux does not automatically start the enforcement of access. If SELinux is enabled and it cannot find a policy, it will refuse to start. That is because the policy defines the behavior of the system (what SELinux should allow). SELinux policies are generally distributed in a compiled form (just like with software) as policy modules. These modules are then aggregated into a single policy store and loaded in memory to allow SELinux to enforce the policy rules on the system. Gentoo, being a source-based meta-distribution, distributes the SELinux policies as (source) code as well, which is compiled and built at install time, just like it does with other software. The following diagram shows the relationship between policy rules, policy modules, and a policy package (which is often a one-to-one mapping towards a policy store): Relationship between policy rules, policy modules and policy store Writing SELinux policies A SELinux policy writer can write down the policy rules in (currently) three possible languages: In standard SELinux source format—a human-readable and well-established language for writing SELinux policies In reference policy style—this extends the standard SELinux source format with M4 macros to facilitate the development of policies. In the SELinux Common Intermediate Language (CIL)—a computer-readable (and, with some effort, human-readable) format for SELinux policies. Most SELinux supporting distributions base their policy on the reference policy (https://github.com/TresysTechnology/refpolicy/wiki), a fully functional SELinux policy set managed as a free software project. This allows distributions to ship with a functional policy set rather than having to write one themselves. Many project contributors are distribution developers, trying to push changes of their distribution to the reference policy project itself, where the changes are peer-reviewed to make sure no rules are brought into the project that might jeopardize the security of any platform. It easily becomes very troublesome to write reusable policy modules without the extensive set of M4 macros offered by the reference policy project. The SELinux CIL format is quite recent (RHEL 7.2 does not support it yet), and although it is very much in use already (the recent SELinux user space converts everything in CIL in the background), it is not that common yet for policy writers to use it directly. As an example, consider the web server rule we discussed earlier, repeated here for your convenience: Allow the processes labeled with httpd_t to bind to TCP ports labeled with http_port_t. In the standard SELinux source format, this is written down as follows: allow httpd_t http_port_t : tcp_socket { name_bind }; Using reference policy style, this rule is part of the following macro call: corenet_tcp_bind_http_port(httpd_t) In CIL language, the rule would be expressed as follows: (allow httpd_t http_port_t (tcp_socket (name_bind))) In most representations, we can see what the rule is about: The subject (who is taking the action): In this case, it is a processes labeled with the httpd_t type. The target resource or object (the target for the action): In this case, it is a TCP socket (tcp_socket) labeled with the http_port_t type. In reference policy style, this is implied by the function name. The action or permission: In this case, it is binding to a port (name_bind). In reference policy style, this is implied by the function name. The result that the policy will enforce: In this case, it is that the action is allowed (allow). In reference policy style, this is implied by the function name. A policy is generally written for an application or set of applications. So the preceding example will be part of the policy written for web servers. Policy writers will generally create three files per application or application set: A .te file, which contains the type enforcement rules. An .if file, which contains interface and template definitions, allowing policy writers to easily use the newly generated policy rules to enhance other policies with. You can compare this to header files in other programming languages. An .fc file, which contains file context expressions. These are rules that assign labels to resources on the filesystem. A finished policy will then be packaged into an SELinux policy module. Distributing policies through modules Initially, SELinux used a single, monolithic policy approach: all possible access control rules are maintained in a single policy file. It quickly became clear that this is not manageable in the long term, and the idea of developing a modular policy approach was born. Within the modular approach, policy developers can write isolated policy sets for a particular application (or set of applications), roles, and so on. These policies then get built and distributed as policy modules. Platforms that need access controls for a particular application load the SELinux policy module that defines the access rules for that application. The process of building policy modules is shown in the next diagram. It also shows where CIL comes into play, even when the policy rules themselves are not written in CIL. For distributions that do not yet support CIL, semodule will directly go from the .pp file to the policy.## file. Build process from policy rule to policy store With the recent SELinux user space, the *.pp files (which are the SELinux policy modules) are considered to be written in a high-level language (HLL). Do not assume that this means they are human readable: these files are binary files. The consideration here is that SELinux wants to support writing SELinux policies in a number of formats, which it calls high-level languages, as long as it has a parser that can convert the files into CIL. Marking the binary module formats as high-level allowed the SELinux project to introduce the distinction between high-level languages and CIL in a backward-compatible manner. When distributing SELinux policy modules, most Linux distributions place the *.pp SELinux policy modules inside /usr/share/selinux, usually within a subdirectory named after the policy store (such as targeted). There, these modules are ready for administrators to activate them. When activating a module, the semodule command (part of the policycoreutils package) will copy those modules into a dedicated directory: /etc/selinux/targeted/modules/active/modules (RHEL) or /var/lib/selinux/mcs/active/modules (Gentoo). This location is defined by the version of the SELinux user space—more recent versions use the /var/lib location. When all modules are aggregated in a single location, the final policy binary is compiled, resulting in /etc/selinux/targeted/policy/policy.30 (or some other number) and loaded in memory. On RHEL, the SELinux policies are provided by the selinux-policy-targeted (or -minimum or -mls) package. On Gentoo, they are provided by the various sec-policy/selinux-* packages (Gentoo uses separate packages for each module, reducing the number of SELinux policies that are loaded on an average system). Bundling modules in a policy store A policy store contains a single comprehensive policy, and only a single policy can be active on a system at any point in time. Administrators can switch policy stores, although this often requires the system to be rebooted and might even require relabeling the entire system (relabeling is the act of resetting the contexts on all files and resources available on that system). The active policy on the system can be queried using sestatus (SELinux status, provided through the policycoreutils package), as follows: # sestatus | grep Loaded policy Loaded policy name: targeted In this example, the currently loaded policy (store) is named targeted. The policy name that SELinux will use upon its next reboot is defined in the /etc/selinux/config configuration file as the SELINUXTYPE parameter. It is the init system of systems (be it a SysV-compatible init system or systemd) that is generally responsible for loading the SELinux policy, effectively activating SELinux support on the system. The init system reads the configuration, locates the policy store, and loads the policy file in memory. If the init system does not support this (in other words, it is not SELinux-aware) then the policy can be loaded through the load_policy command, part of the policycoreutils package. Distinguishing between policies The most common SELinux policy store names are strict, targeted, mcs, and mls. None of the names assigned to policy stores are fixed, though, so it is a matter of convention. Hence, it is recommended to consult the distribution documentation to verify what should be the proper name of the policy. Still, the name often provides some information about the SELinux options that are enabled through the policy. Supporting MLS One of the options that can be enabled is MLS support. If it's disabled, the SELinux context will not have a fourth field with sensitivity information in it, making the contexts of processes and files look as follows: staff_u:sysadm_r:sysadm_t To check whether MLS is enabled, it is sufficient to see whether the context indeed doesn't contain such a fourth field, but it can also be acquired from the Policy MLS status line in the output of sestatus: # sestatus | grep MLS Policy MLS Status: disabled Another method would be to look into the pseudo file, /sys/fs/selinux/mls. A value of 0 means disabled, whereas a value of 1 means enabled: # cat /sys/fs/selinux/mls 0 Policy stores that have MLS enabled are generally targeted, mcs and mls, whereas strict generally has MLS disabled. Dealing with unknown permissions Permissions (such as read, open, and lock) are defined both in the Linux kernel and in the policy itself. However, sometimes, newer Linux kernels support permissions that the current policy does not yet understand. Take the block_suspend permission (to be able to block system suspension) as an example. If the Linux kernel supports (and checks) this permission but the loaded SELinux policy does not understand that permission yet, then SELinux has to decide how it should deal with the permission. SELinux can be configured to do one of the following actions: allow: assume everything that is not understood is allowed deny: assume no one is allowed to perform this action reject: stop and halt the system This is configured through the deny_unknown value. To see the state for unknown permissions, look for the Policy deny_unknown status line in sestatus: # sestatus | grep deny_unknown Policy deny_unknown status: denied Administrators can set this for themselves in the /etc/selinux/semanage.conf file through the handle-unknown variable (with allow, deny, or reject). RHEL by default allows unknown permissions, whereas Gentoo by default denies them. Supporting unconfined domains An SELinux policy can be very strict, limiting applications as close as possible to their actual behavior, but it can also be very liberal in what applications are allowed to do. One of the concepts available in many SELinux policies is the idea of unconfined domains. When enabled, it means that certain SELinux domains (process contexts) are allowed to do almost anything they want (of course, within the boundaries of the regular Linux DAC permissions, which still hold) and only a select number of domains are truly confined (restricted) in their actions. Unconfined domains have been brought forward to allow SELinux to be active on desktops and servers where administrators do not want to fully restrict the entire system, but only a few of the applications running on it. Generally, these implementations focus on constraining network-facing services (such as web servers and database management systems) while allowing end users and administrators to roam around unrestricted. With other MAC systems, such as AppArmor, unconfinement is inherently part of the design of the system as they only restrict actions for well-defined applications or users. However, SELinux was designed to be a full mandatory access control system and thus needs to provide access control rules even for those applications that shouldn't need any. By marking these applications as unconfined, almost no additional restrictions are imposed by SELinux. We can see whether unconfined domains are enabled on the system through seinfo, which we use to query the policy for the unconfined_t SELinux type. On a system where unconfined domains are supported, this type will be available: # seinfo -tunconfined_t unconfined_t For a system where unconfined domains are not supported, the type will not be part of the policy: # seinfo -tunconfined_t ERROR: could not find datum for type unconfined_t Most distributions that enable unconfined domains call their policy targeted, but this is just a convention that is not always followed. Hence, it is always best to consult the policy using seinfo. RHEL enables unconfined domains, whereas with Gentoo, this is a configurable setting through the unconfined USE flag. Limiting cross-user sharing When UBAC is enabled, certain SELinux types will be protected by additional constraints. This will ensure that one SELinux user cannot access files (or other specific resources) of another user, even when those users are sharing their data through the regular Linux permissions. UBAC provides some additional control over information flow between resources, but it is far from perfect. In its essence, it is made to isolate SELinux users from one another. A constraint in SELinux is an access control rule that uses all parts of a context to make its decision. Unlike type-enforcement rules, which are purely based on the type, constraints can take the SELinux user, SELinux role, or sensitivity label into account. Constraints are generally developed once and then left untouched—most policy writers will not touch constraints during their development efforts. Many Linux distributions, including RHEL, disable UBAC. Gentoo allows users to select whether or not they want UBAC through the Gentoo ubac USE flag (which is enabled by default). Incrementing policy versions While checking the output of sestatus, we see that there is also a notion of policy versions: # sestatus | grep version Max kernel policy version: 28 This version has nothing to do with the versioning of policy rules but with the SELinux features that the currently running kernel supports. In the preceding output, 28 is the highest policy version the kernel supports. Every time a new feature is added to SELinux, the version number is increased. The policy file itself (which contains all the SELinux rules loaded at boot time by the system) can be found in /etc/selinux/targeted/policy (where targeted refers to the policy store used, so if the system uses a policy store named strict, then the path would be /etc/selinux/strict/policy). If multiple policy files exist, we can use the output of seinfo to find out which policy file is used: # seinfo Statistics for policy file: /etc/selinux/targeted/policy/policy.30 Policy Version & Type: v.30 (binary, mls) ... The next table provides the current list of policy feature enhancements and the Linux kernel version in which that feature is introduced. Many of the features are only of concern to the policy developers, but knowing the evolution of the features gives us a good idea about the evolution of SELinux. Version Linux kernel Description 12   The old API for SELinux, now deprecated. 15 2.6.0 Introduced the new API for SELinux. 16 2.6.5 Added support for conditional policy extensions. 17 2.6.6 Added support for IPv6. 18 2.6.8 Added support for fine-grained netlink socket permissions. 19 2.6.12 Added support for MLS. 20 2.6.14 Reduced the size of the access vector table. 21 2.6.19 Added support for MLS range transitions. 22 2.6.25 Introduced policy capabilities. 23 2.6.26 Added support for per-domain permissive mode. 24 2.6.28 Added support for explicit hierarchy (type bounds). 25 2.6.39 Added support for filename-based transitions. 26 3.0 Added support for role transitions for non-process classes. Added support for role attributes. 27 3.5 Added support for flexible inheritance of user and role for newly created objects. 28 3.5 Added support for flexible inheritance of type for newly created objects. 29 3.14 Added support for attributes within SELinux constraints. 30 4.3 Added support for extended permissions and implemented first on IOCTL controls. Enhanced SELinux XEN support. History of SELinux feature evolution By default, when an SELinux policy is built, the highest supported version as defined by the Linux kernel and libsepol (the library responsible for building the SELinux policy binary) is used. Administrators can force a version to be lower using the policy-version parameter in /etc/selinux/semanage.conf. Different policy content Besides the aforementioned policy capabilities, the main difference between policies (and distributions) is the policy content itself. We already covered that most distributions base their policy on the reference policy project. But although that project is considered the master for most distributions, each distribution has its own deviation from the main policy set. Many distributions make extensive additions to the policy without directly passing the policies to the upstream reference policy project. There are several possible reasons why this is not directly done: The policy enhancements or additions are still immature: Red Hat initially starts with policies being active but permissive, meaning the policies are not enforced. Instead, SELinux logs what it would have prevented and, based on those, logs the policies that are enhanced. This ensures that a policy is only ready after a few releases. The policy enhancements or additions are too specific to the distribution: If a policy set is not reusable for other distributions, then some distributions will opt to keep those policies to themselves as the act of pushing changes to upstream projects takes quite some effort. The policy enhancements or additions haven't followed the upstream rules and guidelines: The reference policy has a set of guidelines that policies need to adhere to. If a policy set does not comply with these rules, then it will not be accepted. The policy enhancements or additions are not implementing the same security model as the reference policy project wants: As SELinux is a very extensive mandatory access control system, it is possible to write completely different policies. The distribution does not have the time or resources to push changes upstream. This ensures that SELinux policies between distributions (and even releases of the same distribution) can, content-wise, be quite different. Gentoo for instance aims to follow the reference policy project closely, with changes being merged within a matter of weeks. Summary In this article, we saw that SELinux offers a more fine-grained access control mechanism on top of the Linux access controls. SELinux is implemented through Linux Security Modules and uses labels to identify its resources and processes based on ownership (user), role, type, and even the security sensitivity and categorization of the resource. We covered how SELinux policies are handled within an SELinux-enabled system and briefly touched upon how policy writers structure policies. Linux distributions implement SELinux policies, which might be a bit different from each other based on supporting features, such as sensitivity labels, default behavior for unknown permissions, support for confinement levels, or specific constraints put in place such as UBAC. However, most of the policy rules themselves are similar and are even based on the same upstream reference policy project. Resources for Article: Further resources on this subject: SELinux - Highly Secured Web Hosting for Python-based Web Applications [article] Introduction to Docker [article] Booting the System [article]
Read more
  • 0
  • 0
  • 13507
Modal Close icon
Modal Close icon