Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7008 Articles
article-image-r-statistical-package-interfacing-python
Janu Verma
17 Nov 2016
8 min read
Save for later

The R Statistical Package Interfacing with Python

Janu Verma
17 Nov 2016
8 min read
One of my coding hobbies is to explore different Python packages and libraries. In this post, I'll talk about the package rpy2, which is used to call R inside python. Being an avid user of R and a huge supporter of R graphical packages, I had always desired to call R inside my Python code to be able to produce beautiful visualizations. The R framework offers machinery for a variety of statistical and data mining tasks. Let's review the basics of R before we delve into R-Python interfacing. R is a statistical language which is free, is open source, and has comprehensive support for various statistical, data mining, and visualization tasks. Quick-R describes it as: "R is an elegant and comprehensive statistical and graphical programming language." R is one of the fastest growing languages, mainly due to the surge in interest in statistical learning and data science. The Data Science Specialization on Coursera has all courses taught in R. There are R packages for machine learning, graphics, text mining, bioinformatics, topics modeling, interactive visualizations, markdown, and many others. In this post, I'll give a quick introduction to R. The motivation is to acquire some knowledge of R to be able to follow the discussion on R-Python interfacing. Installing R R can be downloaded from one of the Comprehensive R Archive Network (CRAN) mirror sites. Running R To run R interactively on the command line, type r. Launch the standard GUI (which should have been included in the download) and type R code in it. RStudio is the most popular IDE for R. It is recommended, though not required, to install RStudio and run R on it. To write a file with R code, create a file with the .r extension (for example, myFirstCode.r). And run the code by typing the following on the terminal: Rscript file.r Basics of R The most fundamental data structure in R is a vector; actually everything in R is a vector (even numbers are 1-dimensional vectors). This is one of the strangest things about R. Vectors contain elements of the same type. A vector is created by using the c() function. a = c(1,2,5,9,11) a [1] 1 2 5 9 11 strings = c("aa", "apple", "beta", "down") strings [1] "aa" "apple" "beta" "down" The elements in a vector are indexed, but the indexing starts at 1 instead of 0, as in most major languages (for example, python). strings[1] [1] "aa" The fact that everything in R is a vector and that the indexing starts at 1 are the main reasons for people's initial frustration with R (I forget this all the time). Data Frames A lot of R packages expect data as a data frame, which are essentially matrices but the columns can be accessed by names. The columns can be of different types. Data frames are useful outside of R also. The Python package Pandas was written primarily to implement data frames and to do analysis on them. In R, data frames are created (from vectors) as follows: students = c("Anne", "Bret", "Carl", "Daron", "Emily") scores = c(7,3,4,9,8) grades = c('B', 'D', 'C', 'A', 'A') results = data.frame(students, scores, grades) results students scores grades 1 Anne 7 B 2 Bret 3 D 3 Carl 4 C 4 Daron 9 A 5 Emily 8 A The elements of a data frame can be accessed as: results$students [1] Anne Bret Carl Daron Emily Levels: Anne Bret Carl Daron Emily This gives a vector, the elements of which can be called by indexing. results$students[1] [1] Anne Levels: Anne Bret Carl Daron Emily Reading Files Most of the times the data is given as a comma-separated values (csv) file or a tab-separated values (tsv) file. We will see how to read a csv/tsv file in R and create a data frame from it. (Aside: The datasets in most Kaggle competitions are given as csv files and we are required to do machine learning on them. In Python, one creates a pandas data frame or a numpy array from this csv file.) In R, we use a read.csv or read.table command to load a csv file into memory, for example, for the Titanic competition on Kaggle: training_data <- read.csv("train.csv", header=TRUE) train <- data.frame(survived=train_all$Survived, age=train_all$Age, fare=train_all$Fare, pclass=train_all$Pclass) Similarly, a tsv file can be loaded as: data <- read.csv("file.tsv";, header=TRUE, delimiter="t") Thus given a csv/tsv file with or without headers, we can read it using the read.csv function and create a data frame using: data.frame(vector_1, vector_2, ... vector_n). This should be enough to start exploring R packages. Another command that is very useful in R is head(), which is similar to the less command on Unix. rpy2 First things first, we need to have both Python and R installed. Then install rpy2 from the Python package index (Pypi). To do this, simply type the following on the command line: pip install rpy2 We will use the high-level interface to R, the robjects subpackage of rpy2. import rpy2.robjects as ro We can pass commands to the R session by putting the R commands in the ro.r() method as strings. Recall that everything in R is a vector. Let's create a vector using robjects: ro.r('x=c(2,4,6,8)') print(ro.r('x')) [1] 2 4 6 8 Keep in mind that though x is an R object (vector), ro.r('x') is a Python object (rpy2 object). This can be checked as follows: type(ro.r('x')) <class 'rpy2.robjects.vectors.FloatVector'> The most important data types in R are data frames, which are essentially matrices. We can create a data frame using rpy2: ro.r('x=c(2,4,6,8)') ro.r('y=c(4,8,12,16)') ro.r('rdf=data.frame(x,y)') This created an R data frame, rdf. If we want to manipulate this data frame using Python, we need to convert it to a python object. We will convert the R data frame to a pandas data frame. The Python package pandas contains efficient implementations of data frame objects in python. import pandas.rpy.common as com df = com.load_data('rdf') print type(df) <class 'pandas.core.frame.DataFrame'> df.x = 2*df.x Here we have doubled each of the elements of the x vector in the data frame df. But df is a Python object, which we can convert back to an R data frame using pandas as: rdf = com.convert_to_r_dataframe(df) print type(rdf) <class 'rpy2.robjects.vectors.DataFrame'> Let's use the plotting machinery of R, which is the main purpose of studying rpy2: ro.r('plot(x,y)') Not only R data types, but rpy2 lets us import R packages as well (given that these packages are installed on R) and use them for analysis. Here we will build a linear model on x and y using the R package stats: from rpy2.robjects.packages import importr stats = importr('stats') base = importr('base') fit = stats.lm('y ~ x', data=rdf) print(base.summary(fit)) We get the following results: Residuals: 1 2 3 4 0 0 0 0 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 0 0 NA NA x 2 0 Inf <2e-16 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 0 on 2 degrees of freedom Multiple R-squared: 1, Adjusted R-squared: 1 F-statistic: Inf on 1 and 2 DF, p-value: < 2.2e-16 R programmers will immediately recognize the output as coming from applying linear model function lm() on data. I'll end this discussion with an example using my favorite R package ggplot2. I have written a lot of posts on data visualization using ggplot2. The following example is borrowed from the official documentation of rpy2. import math, datetime import rpy2.robjects.lib.ggplot2 as ggplot2 import rpy2.robjects as ro from rpy2.robjects.packages import importr base = importr('base') datasets = importr('datasets') mtcars = datasets.data.fetch('mtcars')['mtcars'] pp = ggplot2.ggplot(mtcars) + ggplot2.aes_string(x='wt', y='mpg', col='factor(cyl)') + ggplot2.geom_point() + ggplot2.geom_smooth(ggplot2.aes_string(group = 'cyl'), method = 'lm') pp.plot() Author: Janu Verma is a researcher in the IBM T.J. Watson Research Center, New York. His research interests are in mathematics, machine learning, information visualization, computational biology, and healthcare analytics. He has held research positions at Cornell University, Kansas State University, Tata Institute of Fundamental Research, Indian Institute of Science, and the Indian Statistical Institute. He has written papers for IEEE Vis, KDD, International Conference on HealthCare Informatics, Computer Graphics and Applications, Nature Genetics, IEEE Sensors Journals and so on. His current focus is on the development of visual analytics systems for prediction and understanding. He advises start-ups and other companies on data science and machine learning in the Delhi-NCR area. He can be found at Here.
Read more
  • 0
  • 0
  • 16961

article-image-how-build-facebook-messenger-bot
Amit Kothari
17 Nov 2016
9 min read
Save for later

How to build a Facebook Messenger Bot

Amit Kothari
17 Nov 2016
9 min read
One of the trending topics in the tech world this year is conversational interfaces. Everyone from big tech companies to start-ups are putting their bets on it. There is an explosion of virtual assistants and chat bots, and the number is increasing every week. What is conversational interface and why are companies investing in it? In this post we will learn about it and build a simple bot using the Facebook Messenger Platform. Conversational Interface A conversational interface is an interface that allows the users to interact with computers using natural language, instead of commands or clicks and taps. It can be voice-based like Siri, Ok Google, or Microsoft's Cortana, or it can be text-based like many Slack or Facebook Messenger bots. Instead of building apps for different platforms, device types, and sizes, companies can now provide a way to interact with their services using voice or text, without the need to download an app or learn how to use it. Facebook Messenger Bot Many apps like Slack, Telegraph, and Skype now offer platforms to build virtual assistants or bots. But with more than 1 billion users, Facebook Messenger is definitely the most popular choice. A Facebook Messenger bot works in the following way: The user sends a message to the bot using their Facebook Messenger account. The Facebook Messenger Platform will receive and post the message to the bot's server using a webhook. The bot server will receive the message, process it, and send a response back to the Facebook Messenger Platform using HTTP POST. Facebook will forward the message to the user. Let's build a simple Facebook Messenger Bot. Creating a Facebook App and Page First we need to create a Facebook Page and a Facebook app: Create a Facebook account, if you do not have one. Create a Facebook Page. This page will work as the identity of your bot. Create a Facebook App. Enter a display name and contact e-mail, and select a category for your bot. After creating the app, click on 'Add product' and then select the 'Messenger' option. Click on the 'Get started' button, and under the 'Token Generation' section, select the page you just created. This will generate a Page Access Token, which the bot server will need to communicate with the Facebook Messenger Platform. Copy this token for later use. You also need to set up webhook so that Facebook can forward the message to the bot server, but before that, let's build the server. Building the Bot Server You can use any language or framework to build the bot server. For this example, I am using Node.js. Let's start by creating an Express.js hello world app. Before we start, we need Node.js and npm installed. Follow the instructions on the Node.js website if you do not have these installed already: Create a new directory for your application, and inside of the app directory, create the package.json file by using the npm init command. Install Express and add it as a dependency using the npm install --save express command. Create a file called index.js with the following code: const app = require('express')(); app.set('port', (process.env.PORT || 5000)); app.get('/', function (req, res) { res.send('Helle World'); }); app.listen(app.get('port'), function () { console.log('App running on port', app.get('port')) }); Our hello world app is done. Run the node index.js command and open http://localhost:5000 in your browser. If all is good, you will see 'Hello World' text on the page. Now that we have the basic app set up, let's add the code to be able to communicate with the Facebook Messenger Platform: Install body-parser and request npm modules using the npm install --save body-parser request command. Body-parser is a middleware to parse incoming request bodies and request is an HTTP client. Now update index.js with the following code. Replace <fb-page-access-token> with the page access token generated before. You will need the validation token while setting up the webhook. Feel free to change it to anything you like: 'use strict' const app = require('express')(); const bodyParser = require('body-parser'); const request = require('request'); const VALIDATION_TOKEN = 'super_secret_validation_token'; const PAGE_ACCESS_TOKEN = '<fb-page-access-token>'; // replace <fb-page-access-token> with your page access token app.set('port', (process.env.PORT || 5000)); app.use(bodyParser.urlencoded({ extended: false })); app.use(bodyParser.json()); app.get('/', function (req, res) { res.send('Helle World'); }); app.get('/webhook', function (req, res) { if (req.query['hub.mode'] === 'subscribe' && req.query['hub.verify_token'] === VALIDATION_TOKEN) { res.status(200).send(req.query['hub.challenge']); } else { res.sendStatus(403); } }); app.post('/webhook/', function (req, res) { // Iterate over each entry req.body.entry.forEach(function (entry) { // Iterate over each messaging event entry.messaging.forEach(function (event) { let sender = event.sender.id if (event.message && event.message.text) { callSendAPI(sender, `Message received ${event.message.text}`); } }); res.sendStatus(200); // Respond with 200 to notify the message is received by the webhook successfully }); }); function callSendAPI(sender, text) { request({ url: 'https://graph.facebook.com/v2.6/me/messages', qs: { access_token: PAGE_ACCESS_TOKEN }, method: 'POST', json: { recipient: { id: sender }, message: { text: text }, } }, function (error, response, body) { if (error) { console.error('Unable to send message', error); } }); } app.listen(app.get('port'), function () { console.log('App running on port', app.get('port')) }); Here we have defined two new service endpoints. The first one responds to the GET request and will be used for verification when we set up the webhook. The second endpoint will respond to the POST request. This will process the incoming message and send a response using the callSendAPI method. For now, the server will simply send back the message received from the user. You can deploy the server anywhere you like. Because of its simplicity, I am going to use Heroku for this tutorial. Deploying the App to Heroku If you do not have an account, sign up on heroku.com and then install Heroku CLI. Initialize a git repository and commit the code to easily push it to Heroku. Use git init to initialise and git add . followed by git commit -m "Initial commit" to commit the code. Create a Procfile file with the following content. Procfile declares what commands to run on heroku. web: node index.js. Run the heroku create command to create and set up a new app on Heroku. Now you can run the app locally using the heroku local command. To deploy the app on Heroku, use git push heroku master and use heroku open to open the deployed app in the browser. Setting up the Messenger Webhook Now that the server is ready, we can set up the webhook: Go back to the app page; under the Messenger settings page, click on the 'Setup Webhooks' button. Enter the bot server webhook url (https:///webhook), and the validation token, and select all the options under subscription fields. We are ready to test our bot. Go to the Facebook page you created, click on message, and send a message to the bot. If everything is good, you will see a response from the bot. The bot is working, but we can update it to send a different response based on the user message. Let's create a simple bot for Game of Thrones fans, where if a user types in one of the great house's names, the bot will respond with their house words. Update index.js with the following code: 'use strict' const app = require('express')(); const bodyParser = require('body-parser'); const request = require('request'); const VALIDATION_TOKEN = 'super_secret_validation_token'; const PAGE_ACCESS_TOKEN = '<fb-page-access-token>'; // replace <fb-page-access-token> with your page access token const HOUSE_WORDS = { STARK: 'Winter is Coming', LANNISTER: 'Hear Me Roar!', BARATHEON: 'Ours is the Fury', TULLY: 'Family, Duty, Honor', GREYJOY: 'We Do Not Sow', MARTELL: 'Unbowed, Unbent, Unbroken', TYRELL: 'Growing Strong', FREY: 'We Stand Together', } app.set('port', (process.env.PORT || 5000)); app.use(bodyParser.urlencoded({ extended: false })); app.use(bodyParser.json()); app.get('/', function (req, res) { res.send('Helle World'); }); app.get('/webhook', function (req, res) { if (req.query['hub.mode'] === 'subscribe' && req.query['hub.verify_token'] === VALIDATION_TOKEN) { res.status(200).send(req.query['hub.challenge']); } else { res.sendStatus(403); } }); app.post('/webhook/', function (req, res) { // Iterate over each entry req.body.entry.forEach(function (entry) { // Iterate over each messaging event entry.messaging.forEach(function (event) { let sender = event.sender.id if (event.message && event.message.text) { callSendAPI(sender, getHouseWord(event.message.text)); } }); res.sendStatus(200); // Respond with 200 to notify the message is received by the webhook successfully }); }); function callSendAPI(sender, text) { request({ url: 'https://graph.facebook.com/v2.6/me/messages', qs: { access_token: PAGE_ACCESS_TOKEN }, method: 'POST', json: { recipient: { id: sender }, message: { text: text }, } }, function (error, response, body) { if (error) { console.error('Unable to send message', error); } }); } function getHouseWord(text) { const house = Object.keys(HOUSE_WORDS) .find(function (houseName) { return text.toUpperCase().indexOf(houseName) !== -1; }); return house ? HOUSE_WORDS[house] : 'No house word found :('; } app.listen(app.get('port'), function () { console.log('App running on port', app.get('port')) }); Now if you type one of the family names defined in our code, the bot will respond with the family words. So if you type 'stark', the bot will respond 'Winter is Coming'. App Review While you and other page admins can chat with the bot, to make it publicly available, you have to submit the bot for review. This can be done from the app page's 'App Review' option. Fill out the form, and once approved, your bot will be accessible by all Facebook Messenger users. Although our bot is working, it is not truly conversational. It only responds to a fix set of commands. To build a smarter bot, we need natural language processing. Natural language processors convert natural language text or audio to a format that computers can understand. There are many services that allow us to write smarter bot engines, and one of them is Wit. Wit was acquired by Facebook in 2015. You can read more about Wit and how to integrate it with your bot here. I hope you enjoyed this post. If you have built any bot on Facebook or any other platform, please share it with us in the comment section below. Author: Amit Kothari is a full-stack software developer based in Melbourne, Australia. He has 10+ years experience in designing and implementing software, mainly in Java/JEE. His recent experience is in building web applications using JavaScript frameworks like React and AngularJS, and backend micro services / REST API in Java. He is passionate about lean software development and continuous delivery.
Read more
  • 0
  • 0
  • 5500

article-image-kotlin-basics
Packt
16 Nov 2016
7 min read
Save for later

Kotlin Basics

Packt
16 Nov 2016
7 min read
In this article by Stephen Samuel and Stefan Bocutiu, the authors of the book Programming Kotlin, it’s time to discover the fundamental building blocks of Kotlin. This article will cover the basic constructs of the language, such as defining variables, control flow syntax, type inference, and smart casting, and its basic types and their hierarchy. (For more resources related to this topic, see here.) For those coming from a Java background, this article will also highlight some of the key differences between Kotlin and Java and how Kotlin’s language features are able to exist on the JVM. For those who are not existing Java programmers, then those differences can be safely skipped. vals and vars Kotlin has two keywords for declaring variables: val and var. A var is a mutable variable—a variable that can be changed to another value by reassigning it. This is equivalent to declaring a variable in Java. val name = “kotlin” Alternatively, the var can be initialized later: var name: String name = “kotlin” Variables defined with var can be reassigned since they are mutable: var name = “kotlin” name = “more kotlin” The val keyword is used to declare a read-only variable. This is equivalent to declaring a final variable in Java. A val must be initialized when created since it cannot be changed later: val name = “kotlin” A read-only variable does not mean the instance itself is automatically immutable. The instance may still allow its member variables to be changed via functions or properties. But the variable itself cannot change its value or be reassigned to another value. Type inference Did you notice in the previous section that the type of the variable was not included when it was initialized? This is different to Java, where the type of the variable must always accompany its declaration. Even though Kotlin is a strongly typed language, we don’t always need to declare types explicitly. The compiler can attempt to figure out the type of an expression from the information included in the expression. A simple val is an easy case for the compiler because the type is clear from the right-hand side. This mechanism is called type inference. This reduces boilerplate while keeping the type safety we expect of a modern language. Values and variables are not the only places where type inference can be used. It can also be used in closures where the type of the parameter(s) can be inferred from the function signature. It can also be used in single-line functions, where the return value can be inferred from the expression in the function, as this example demonstrates: fun plusOne(x: Int) = x + 1 Sometimes, it is helpful to add type inference if the type inferred by the compiler is not exactly what you want: val explicitType: Number = 12.3 Basic types One of the big differences between Kotlin and Java is that in Kotlin, everything is an object. If you come from a Java background, then you will already be aware that in Java, there are special primitive types, which are treated differently from objects. They cannot be used as generic types, do not support method/function calls, and cannot be assigned null. An example is the boolean primitive type. Java introduced wrapper objects to offer a workaround in which primitive types are wrapped in objects so that java.lang. Boolean wraps a boolean in order to smooth over the distinctions. Kotlin removes this necessity entirely from the language by promoting the primitives to full objects. Whenever possible, the Kotlin compiler will map basic types back to JVM primitives for performance reasons. However, the values must sometimes be boxed, such as when the type is nullable or when it is used in generics. Two different values that are boxed might not use the same instance, so referential equality is not guaranteed on boxed values. Numbers The built-in number types are as follows: Type Width long 64 int 32 short 16 byte 8 double 64 float 32 To create a number literal, use one of the following forms: val int = 123 val long = 123456L val double = 12.34 val float = 12.34F val hexadecimal = 0xAB val binary = 0b01010101 You will notice that a long value requires the suffix L and a float, F. The double type is used as the default for floating point numbers, and int for integral numbers. The hexadecimal and binary use the prefixes 0x and 0b respectively. Kotlin does not support the automatic widening of numbers, so conversion must be invoked explicitly. Each number has a function that will convert the value to one of the other number types: val int = 123 val long = int.toLong() val float = 12.34F val double = float.toDouble() The full set of methods for conversions between types is as follows: toByte() toShort() toInt() toLong() toFloat() toDouble() toChar() Unlike Java, there are no built-in bitwise operators, but named functions instead. This can be invoked like operators (except inverse): val leftShift = 1 shl 2 val rightShift = 1 shr 2 val unsignedRightShift = 1 ushr 2 val and = 1 and 0x00001111 val or = 1 and 0x00001111 val xor = 1 xor 0x00001111 val inv = 1.inv() Booleans Booleans are rather standard and support the usual negation, conjunction and disjunction operations. Conjunction and disjunction are lazily evaluated. So if the left-hand side satisfies the clause, then the right-hand side will not be evaluated: val x = 1 val y = 2 val z = 2 val isTrue = x < y && x < z val alsoTrue = x == y || y == z Chars Chars represent a single character. Character literals use single quotes, such as a or Z. Chars also support escaping for the following characters: t, b, n, r, , , \, $. All Unicode characters can be represented using the respective Unicode number, like so: u1234. Note that the char type is not treated as a number, unlike Java. Strings Just as in Java, strings are immutable. String literals can be created using double or triple quotes. Double quotes create an escaped string. In an escaped string, special characters such as newline must be escaped: val string = “string with n new line” Triple quotes create a raw string. In a raw string, no escaping is necessarily, and all characters can be included. val rawString = “““ raw string is super useful for strings that span many lines “““ Strings also provide an iterator function, so they can be used in a for loop. Arrays In Kotlin, we can create an array using the arrayOf() library function: val array = arrayOf(1, 2, 3) Alternatively, we can create an array from an initial size and a function that is used to generate each element: val perfectSquares = Array(10, { k -> k * k }) Unlike Java, arrays are not treated specially by the language and are regular collection classes. Instances of Array provide an iterator function and a size function as well as a get and set function. The get and set functions are also available through bracket syntax like many C style languages: val element1 = array[0] val element2 = array[1] array[2] = 5 To avoid boxing types that will ultimately be represented as primitives in the JVM, Kotlin provides alternative array classes that are specialized for each of the primitive types. This allows performance-critical code to use arrays as efficiently as they would do in plain Java. The provided classes are ByteArray, CharArray, ShortArray, IntArray, LongArray, BooleanArray, FloatArray, and DoubleArray. Comments Comments in Kotlin will come as no surprise to most programmers as they are the same as Java, Javascript, and C, among other languages. Block comments and line comments are supported: // line comment /* A block comment can span many lines */ Packages Packages allow us to split code into namespaces. Any file may begin with a package declaration: package com.packt.myproject class Foo fun bar(): String = “bar” The package name is used to give us the fully-qualified name (FQN) for a class, object, interface, or function. In the previous example, the Foo class has the FQN com.packt.myproject.Foo, and the top-level function bar has the FQN com.packt.myproject.bar. Summary In Kotlin, everything is an object in the sense that we can call member functions and properties on any variable. Some types are built in because their implementation is optimized, but to the user, they look like ordinary classes. In this article, we described most of these types: numbers, characters, booleans, and arrays. Resources for Article: Further resources on this subject: Responsive Applications with Asynchronous Programming [Article] Asynchronous Programming in F# [Article] Go Programming Control Flow [Article]
Read more
  • 0
  • 0
  • 6944

article-image-encrypt-and-hash-passwords
Pedro Narciso
16 Nov 2016
7 min read
Save for later

Encrypt and hash passwords

Pedro Narciso
16 Nov 2016
7 min read
A couple of days ago, Dropbox posted an explanation on how they store your passwords. In this post, we are going to implement a library that encrypts and hashes a password in a similar way. Disclaimer: Security is a complex AND important topic. It is YOUR responsability to educate yourself on it. As you may have read on the Dropbox blog post, they: Get the hash product of the SHA512 function over the password. Apply the bcrypt function over the previous result. Encrypt the bcrypt product with a symmetric encryption algorithm (AES256). The above transformation can be written in pseudocode as: hashed_password_to_store = encrypt_aes256(bcrypt(sha512('my password'))) Remember: You should not keep the AES256 key in your database nor hardcoded in the application code. Okay, let's start with it! First, we are going to create a project: $ mkdir password-encryption-tool $ cd password-encryption-tool $ npm init // init our node.js project Edit index.js and write the skeleton of the function. 'use strict'; function hashAndEncryptPassword(input){ let sha512hash = sha512(input); let bcryptHash = bcryptHash(sha512hash); let encryptedPassword = encrypt(bcryptHash); return encryptedPassword; } Good! Of course, that function does not work yet. The functions sha512, bcryptHash and encrypt are not yet implemented, but it gives us a starting point. The #sha512(input) function recives a password as input and returns its SHA512 product. To implement that function, we are going to use the Hash class from the crypto module. Docs here. function sha512(input){ let hasher = crypto.createHash('SHA512'); hasher.update(input); return hasher.digest('base64'); //Returns the hash in base64 format } Done! The bcrypt function requires a third party library. There are many options but here are my personal recommendations – bcrypt.js and bcrypt. The former is a pure JavaScript implementation with 0 dependencies. The latter is a native one and therefore faster (2.7 times), which is what I use personally. But, for this example, we are going to use bcrypt.js. Why? Because if you do not have the build environment needed for the native version, I don't want you to stop following this tutorial and start pulling out your hair because there is some dependency not compiling on your machine. Anyway, your code will work with both of them because bcrypt.js API is compatible with bcrypt. Okay, that was a long explanation. Let's install bcryptjs: $ npm install bcryptjs --save This installs bcryptjs and updates our package.json file with the new dependency. Bcrypt implementations expose both synchronous and asynchronous APIs. Given that the bcrypt algorithm was made to be slow, I am going to use the asynchronous API. After all, you don't want to block the event loop, do you? This decision forces us to refactor our code a little bit. The asynchronous version of #hashAndEncryptPassword(input) looks like: //... const crypto = require('crypto'), bcrypt = require('bcryptjs'); const BCRYPT_SALT_ROUNDS=10; function hashAndEncryptPassword(key, input, callback){ let sha512hash ; try{ sha512hash = sha512(input); }catch(err){ return callback(err); } bcrypt.hash(sha512hash, BCRYPT_SALT_ROUNDS, function(err, result){ var encryptedHash; if(err){ return callback(err); } try{ encryptedHash = encrypt(key, result); }catch(err){ return callback(err); } callback( null, encryptedHash); }); } //... Note the function signature change with the inclusion of the callback parameter, plus the bcrypt integration. Also, the SHA512 is wrapped in a try/catch block. Any error happening here will be properly handled and returned to the callback function. Time to implement the encrypt function. Node.js Crypto API exposes the Cipher class. Its usage is similar to the Hash class. We just need to provide the algorithm, key and IV. The IV is used for randomization at the encryption stage; for that reason, it is very important that you use a new random IV for each different value you encrypt. Later, when we want to decrypt a value, you need to provide the same key and IV. Because they do not need to be secret, we are going to append them along the encrypted hash. IVs are just a bunch of random bytes and their generation is quite simple. const IV = new Buffer(crypto.randomBytes(16)); // A buffer with 16 random bytes And here is our encrypt function: ... const IV_LEN = 16, ALGORITHM = 'AES-256-CTR'; function encrypt (key, input) { const IV = new Buffer(crypto.randomBytes(IV_LEN)); let cipher = crypto.createCipheriv(ALGORITHM, key, IV); cipher.setEncoding('base64'); cipher.write(input); cipher.end(); const cipherText = cipher.read(); // IV is not a secret. We can store it along the password return cipherText + '$' + IV.toString('base64'); } ... The encrypt function accepts key and input as arguments, and: Generates an IV.. Creates a cipher using the AES-256-CTR algorithm, our key and the IV. Sets cipherText constant with the encrypted version of input. Returns the cipherText plus $ plus the IV. We are almost done. We now have a function that allows us to safely store a user password in our database. Now, as you've probably guessed, when we want to check if a given password is valid, we need to compare it with our stored password. But, you can just simple compare them. You need to apply some transformations in order to get comparable versions. A simple approach could be to just apply the same transformation to a plain password and then they will be comparable. AES256(bcrypt(SHA512(plain text))) === hashed-and-encrypted But we are going to make it slightly different. Here is our recipe: Get the SHA512 of the password we want to validate. Decrypt our stored password. Use bcrypt.compare to compare them. The SHA512 part was already implemented as the sha512 function. To decrypt the stored password we need to create a decipher with crypto.createDecipheriv with the previously used algorithm, key, and IV. function decrypt(key, input){ var result; let [cipherText, IV] = input.split('$'); let buffIV = new Buffer(IV, 'base64'); let decipher = crypto.createDecipheriv(ALGORITHM, key, buffIV); result = decipher.update(cipherText, 'base64', 'utf8'); result += decipher.final('utf8'); return result; } Now we can implement our compare function. function compare(key, clearPassword, encryptedPassword, callback){ var hash; try{ hash = decrypt(key, encryptedPassword); }catch(err){ return callback(err); } bcrypt.compare(sha512(clearPassword), hash, callback); } And do not forget to export compare and hashAndEncryptPassword functions. exports.hashAndEncryptPassword = hashAndEncryptPassword; exports.compare = compare; We can now use our module to hash and encrypt passwords: var passwordTool = require('.'); // Remember to not include your key in your code! // better load it from an environment variable var key = 'BKYHAT11zlXUiXE3iZfzSEWfvwjdbfPK'; var password = 'super secret'; passwordTool.hashAndEncryptPassword(key, password, function(err, hash){ if(err){ throw err; } console.log('Password ready to store', hash); }); // prints: Password ready to store ZGmzcPy29oYjZj5+P/wg4nS0Bs64... And to match a user provided password with our stored copy: passwordTool.compare(key, password, storedPassword, function (err, result){ if(err){ throw err; } console.log('match?', result); }); You can find the complete source code of this article at https://github.com/revington/password-tool. About the author Pedro Narciso García Revington is a senior full-stack developer with 10+ years of experience in high scalability and availability, microservices, automated deployments, data processing, CI, (T,B,D)DD, and polyglot persistence. He is a self-taught, highly motivated problem solver with a focus on delivering quick and elegant solutions.
Read more
  • 0
  • 0
  • 3147

article-image-commonly-used-data-structures
Packt
16 Nov 2016
23 min read
Save for later

Commonly Used Data Structures

Packt
16 Nov 2016
23 min read
In this article by Erik Azar and Mario Eguiluz Alebicto, authors of the book Swift Data Structure and Algorithms, we will learn that the Swift language is truly powerful, but a powerful language is nothing if it doesn't have a powerful standard library to accompany it. The Swift standard library defines a base layer of functionality that you can use for writing your applications, including fundamental data types, collection types, functions and methods, and a large number of protocols. (For more resources related to this topic, see here.) We're going to take a close look at the Swift standard library, specifically support for collection types, with a very low level examination of Arrays, Dictionaries, Sets, and Tuples. The topics covered in this article are: Using the Swift standard library Implementing subscripting Understanding immutability Interoperability between Swift and Objective-C Swift Protocol-Oriented Programming Using the Swift standard library Users often treat a standard library as part of the language. In reality, philosophies on standard library design very widely, often with opposing views. For example, the C and C++ standard libraries are relatively small, containing only the functionality that "every developer" might reasonably require to develop an application. Conversely, languages such as Python, Java, and .NET have large standard libraries that include features, which tend to be separate in other languages, such as XML, JSON, localization, and e-mail processing. In the Swift programming language, the Swift standard library is separate from the language itself, and is a collection of classes, structures, enumerations, functions, and protocols, which are written in the core language. The Swift standard library is currently very small, even compared to C and C++. It provides a base layer of functionality through a series of generic structures and enums, which also adopt various protocols that are also defined in the library. You'll use these as the building blocks for applications that you develop. The Swift standard library also places specific performance characteristics on functions and methods in the library. These characteristics are guaranteed only when all participating protocols performance expectations are satisfied. An example is the Array.append(), its algorithm complexity is amortized O(1), unless the self's storage is shared with another live array; O(count) if self does not wrap a bridged NSArray; otherwise the efficiency is unspecified. We're going to take a detailed look at the implementations for Arrays, Dictionaries, Sets, and Tuples. You'll want to make sure you have at least Xcode 8.1 installed to work with code in this section Why Structures? If you're coming from a background working in languages such as Objective-C, C++, Java, Ruby, Python, or other object-oriented languages, you traditionally use classes to define the structure of your types. This is not the case in the Swift standard library; structures are used when defining the majority of the types in the library. If you're coming from an Objective-C or C++ background this might seem especially odd and feel wrong, because classes are much more powerful than structures. So why does Swift use structures, which are value types, when it also supports classes, which are reference types, that support inheritance, deinitializers, and reference counting? It's exactly because of the limited functionality of structures over classes why Swift uses structures instead of classes for the building blocks of the standard library. Because structures are value types it means they can have only one owner and are always copied when assigned to a new variable or passed to a function. This can make your code inherently safer because changes to the structure will not affect other parts of your application. The preceding description refers to the "copying" of value types. The behavior you see in your code will always be as if a copy took place. However, Swift only performs an actual copy behind the scenes when it is absolutely necessary to do so. Swift manages all value copying to ensure optimal performance, and you should not avoid assignment to try to preempt this optimization. Structures in Swift are far more powerful than in other C-based languages, they are very similar to classes. Swift structures support the same basic features as C-based structures, but Swift also adds support, which makes them feel more like classes. Features of Swift structures: In addition to an automatically generated memberwise initializer, they can have custom initializers Can have methods Can implement protocols So this may leave you asking, when should I use a class over a structure? Apple has published guidelines you can follow when considering to create a structure. If one or more of the following conditions apply, consider creating a structure: Its primary purpose is to encapsulate a few simple data values You expect the encapsulated values to be copied rather than referenced when you pass around or assign an instance of the structure Any properties stored by the structure are value types, which would be expected to be copied instead of referenced The structure doesn't need to inherit properties or behavior from another existing type In all other cases, create a class which will call instances of that class to be passed by the reference. Apple Guidelines for choosing between classes and structures: https://developer.apple.com/library/content/documentation/Swift/Conceptual/Swift_Programming_Language/ClassesAndStructures.html Declaring arrays in Swift An array stores values of the same type in an ordered list. Arrays in Swift have a few important differences between arrays in Objective-C. The first is elements in Swift arrays must be of the same type. If you're used to working with Objective-C NSArrays prior to Xcode 7 you may feel this is a disadvantage, but it actually is a significant advantage as it allows you to know exactly what type you get back when working with an array, allowing you to write more efficient code that leads to less defects. However, if you find you must have dissimilar types in an array, you can define the array to be of a protocol that is common to the other types, or defines the array of type AnyObject. Another difference is Objective-C array elements have to be a class type. In Swift, arrays are generic type collections, there are no limitations on what that type can be. You can create an array that holds any value type, such as Int, Float, String, or Enums, as well as classes. The last difference is, unlike arrays in Objective-C, Swift arrays are defined as a structure instead of a class. We know the basic constructs for declaring an integer array and performing operations such as adding, removing, and deleting elements from an array. Let's take a closer, more detailed examination of how arrays are implemented in the standard library. There are three Array types available in Swift: Array ContiguousArray ArraySlice Every Array class maintains a region of memory that stores the elements contained in the array. For array element types that are not a class or @objc protocol type, the arrays memory region is stored in contiguous blocks. Otherwise, when an arrays element type is a class or @objc protocol type, the memory region can be a contiguous block of memory, an instance of NSArray, or an instance of an NSArray subclass. If may be more efficient to use a ContiguousArray if you're going to store elements that are a class or an @objc protocol type. The ContiguousArray class shares many of the protocols that Array implements, so most of the same properties are supported. The key differentiator between Array and ContiguousArray is that ContiguousArray' do not provide support for bridging to Objective-C. The ArraySlice class represents a sub-sequence of an Array, ContiguousArray, or another ArraySlice. Like ContiguousArray, ArraySlice instances also use contiguous memory to store elements, and they do not bridge to Objective-C. An ArraySlice represents a sub-sequence from a larger, existing Array type. Because of this, you need to be aware of the side effects if you try storing an ArraySlice after the original Array's lifetime has ended and the elements are no longer accessible. This could lead to memory or object leaks thus Apple recommends that long-term storage of ArraySlice instances is discouraged. When you create an instance of Array, ContiguousArray, or ArraySlice, an extra amount of space is reserved for storage of its elements. The amount of storage reserved is referred to as an array's capacity, which represents the potential amount of storage available without having to reallocate the array. Since Swift arrays share an exponential growth strategy, as elements are appended to an array, the array will automatically resize when it runs out of capacity. When you amortize the append operations over many iterations, the append operations are performed in constant time. If you know ahead of time an array will contain a large number of elements, it may be more efficient to allocate additional reserve capacity at creation time. This will avoid constant reallocation of the array as you add new elements. The following code snippet shows an example of declaring an initial capacity of 500 Integer elements for the array intArray: // Create an array using full array syntax var intArray = Array<Int>() // Create an array using shorthand syntax intArray = [Int]() intArray.capacity // contains 0 intArray.reserveCapacity(500) intArray.capacity // contains 508 You can notice from the preceding example that our reserve capacity is actually larger than 500 integer elements. For performance reasons, Swift may allocate more elements than you requests. But you can be guaranteed that at least the number of elements specified will be created. When you make a copy of an array a separate physical copy is not made during the assignment. Swift implements a feature called copy-on-write, which means that array elements are not copied until a mutating operation is performed when more than one array instances is sharing the same buffer. The first mutating operation may cost O(n) in time and space, where n is the length of the array. Initializing Array The initialization phase prepares a struct, class, or enum for use through a method called init. If you're familiar with other languages such as C++, Java, or C# this type of initialization is performed in their class constructor, which is defined using the class's name. If you're coming from Objective-C, the Swift initializers will behave a little differently from how you're used to. In Objective-C, the init methods will directly return the object they initialize, callers will then check the return value when initializing a class and check for nil to see if the initialization process failed. In Swift, this type of behavior is implemented as a feature called failable initialization, which we'll discuss shortly. There are four initializers provided for the three types of Swift arrays implemented in the standard library. Additionally, you can use a dictionary literal to define a collection of one or more elements to initialize the array, the elements are separated by a comma (,): // Create an array using full array syntax var intArray = Array<Int>() // Create an array using shorthand syntax intArray = [Int]() // Use array literal declaration var intLiteralArray: [Int] = [1, 2, 3] // [1, 2, 3] //: Use shorthand literal declaration intLiteralArray = [1, 2, 3] // [1, 2, 3] //: Create an array with a default value intLiteralArray = [Int](count: 5, repeatedValue: 2) // [2, 2, 2, 2, 2] Adding and Updating Elements in an Array To add a new element to an array you can use the append(_:) method. This will add newElement to the end of the array: var intArray = [Int]() intArray.append(50) // [50] If you have an existing collection type you can use the append(_:) method. This will append elements from newElements to the end of the array: intArray.append([60, 65, 70, 75]) // [50, 60, 65, 70, 75] If you want to add elements at a specific index you can use the insert(newElement:at:) method. This will add newElement at index i: intArray.insert(newElement: 55, at: 1)// [50, 55, 60, 65, 70, 75] Requires i <= count otherwise you will receive a fatal error: Array index out of range message and execution will stop. To replace an element at a specific index you can use the subscript notation, providing the index of the element you want to replace: intArray[2] = 63 // [50, 55, 63, 65, 70, 75] Retrieving and Removing Elements from an Array There are several methods you can use to retrieve elements from an array. If you know the specific array index or sub-range of indexes you can use array subscripting: // Initial intArray elements // [50, 55, 63, 65, 70, 75] // Retrieve an element by index intArray[5] // returns 75 // Retrieve an ArraySlice of elements by subRange intArray[2..<5] // Returns elements between index 2 and less than index 5 // [63, 65, 70] // Retrieve an ArraySlice of elements by subRange intArray[2…5] // Returns elements between index 2 and index 5 // [63, 65, 70, 75] You can also iterate over the array, examining each element in the collection: for element in intArray { print(element) } // 50 // 55 // 63 // 65 // 70 // 75 You can also check if an array has a specific element or pass a closure that will allow you to evaluate each element: intArray.contains(55) // returns true See Apples Swift documentation for a complete list of methods available for arrays:https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/CollectionTypes.html Retrieving and Initializing Dictionaries A dictionary is an unordered collection that stores associations between keys and values of the same type with no defined ordering. Each value is associated with a unique key that acts as an identifier for the value in the dictionary. A dictionary data structure works just like a real world dictionary; to retrieve a definition, reference, translation, or some other type of information for a specific word you open the dictionary and locate the word to find the information you're looking for. With a dictionary data structure, you store and retrieve values by associating them with a key. A dictionary key type must conform to the Hashtable protocol. Initializing a Dictionary Just like arrays, there are two ways you can declare a dictionary using either the full or literal syntax: // Full syntax declaration var myDict = Dictionary<Int, String>() // Shorthand syntax declaration var myDict = [Int: String]() Additionally, you can use a dictionary literal to define a collection of one or more key-value pairs to initialize the dictionary. The key and value are separated by a colon (:) and the pairs are separated by a comma (,). If the keys and values have consistent types you do not need to declare the type of dictionary in the declaration, Swift will infer it based on the key-value pairs used during initialization, which saves a few keystrokes allowing you to use the shorter form: // Use dictionary literal declaration var myDict: [Int: String] = [1: "One", 2: "Two", 3: "Three"] // [2: "Two", 3: "Three", 1: "One"] // Use shorthand literal declaration var myDict = [1: "One", 2: "Two", 3: "Three"] // [2: "Two", 3: "Three", 1: "One"] Adding/Modifying/Removing a Key-Value Pair To add a new key-value pair or to update an existing pair you can use the updateValue(_:forKey) method or subscript notation. If the key is does not exist a new pair will be added, otherwise the existing pair is updated with the new value: // Add a new pair to the dictionary myDict.updateValue("Four", forKey: 4) $R0: String? = nil // [2: "Two", 3: "Three", 1: "One", 4: "Four"] // Add a new pair using subscript notation myDict[5] = "Five" // [5: "Five", 2: "Two", 3: "Three", 1: "One", 4: "Four"] Unlike the subscript method, the updateValue(_:forKey:) method will return the value that was replaced, or nil if a new key-value pair was added. To remove a key-value pair you can use the removeValue(forKey:) method, providing the key to delete or setting the key to nil using subscript notation: // Remove a pair from the dictionary – returns the removed pair let removedPair = myDict.removeValue(forKey: 1) removedPair: String? = "One" // [5: "Five", 2: "Two", 3: "Three", 4: "Four"] // Remove a pair using subscript notation myDict[2] = nil // [5: "Five", 3: "Three", 4: "Four"] Unlike the subscript method, the removeValue(forKey:) method will return the value that was removed, or nil if the key doesn't exist. Retrieving Values from a Dictionary You can retrieve specific key-value pairs from a dictionary using subscript notation. The key is passed to the square bracket subscript notation; it's possible the key may not exist so subscripts will return an Optional. You can use either optional binding or forced unwrapping to retrieve the pair or determine if it doesn't exist. Do not use forced unwrapping though unless you're absolutely sure the key exists or a runtime exception will be thrown: // Example using Optional Binding var myDict = [1: "One", 2: "Two", 3: "Three"] if let optResult = myDict[4] { print(optResult) } else { print("Key Not Found") } // Example using Forced Unwrapping – only use if you know the key will exist let result = myDict[3]! print(result) If instead of getting a specific value, you can iterate over the sequence of a dictionary and return a (key, value) tuple, which can be decomposed into explicitly named constants. In this example, (key, value) are decomposed into (stateAbbr, stateName): // Dictionary of state abbreviations to state names let states = [ "AL" : "Alabama", "CA" : "California", "AK" : "Alaska", "AZ" : "Arizona", "AR" : "Arkansas"] for (stateAbbr, stateName) in states { print("The state abbreviation for (stateName) is (stateAbbr)") } // Output of for...in The state abbreviation for Alabama is AL The state abbreviation for California is CA The state abbreviation for Alaska is AK The state abbreviation for Arizona is AZ The state abbreviation for Arkansas is AR You can see from the output that items in the dictionary are not output in the order they were inserted. Recall that dictionaries are unordered collections, so there is no guarantee that the order pairs will be retrieved when iterating over them. If you want to retrieve only the keys or values independently you can use the keys or values properties on a dictionary. These properties will return a LazyMapCollection instance over the collection. The elements of the result will be computed lazily each time they are read by calling the transform closure function on the base element. The key and value will appear in the same order as they would as a key-value pair, .0 member and .1 member, respectively: for (stateAbbr) in states.keys { print("State abbreviation: (stateAbbr)") } //Output of for...in State abbreviation: AL State abbreviation: CA State abbreviation: AK State abbreviation: AZ State abbreviation: AR for (stateName) in states.values { print("State name: (stateName)") } //Output of for...in State name: Alabama State name: California State name: Alaska State name: Arizona State name: Arkansas You can read more about the LazyMapCollection structure on Apple's developer site at https://developer.apple.com/library/prerelease/ios/documentation/Swift/Reference/Swift_LazyMapCollection_Structure/ There may be occasions when you want to iterate over a dictionary in an ordered manner. For those cases, you can make use of the global sort(_:) method. This will return an array containing the sorted elements of a dictionary as an array: // Sort the dictionary by the value of the key let sortedArrayFromDictionary = states.sort({ $0.0 < $1.0 }) // sortedArrayFromDictionary contains... // [("AK", "Alaska"), ("AL", "Alabama"), ("AR", "Arkansas"), ("AZ", "Arizona"), ("CA", "California")] for (key) in sortedArrayFromDictionary.map({ $0.0}) { print("The key: (key)") } //Output of for...in The key: AK The key: AL The key: AR The key: AZ The key: CA for (value) in sortedArrayFromDictionary.map({ $0.1}) { print("The value: (value)") } //Output of for...in The value: Alaska The value: Alabama The value: Arkansas The value: Arizona The value: California Let's walk through what is happening here. For the sort method, we are passing a closure that will compare if the first arguments key from the key-value pair, $0.0 against the second arguments key from the key-value pair, $1.0, if the first argument is less than the second it will be added to the new array. When the sort method has iterated over and sorted all of the elements, a new array of [(String, String)] containing the key-value pairings is returned. Next, we want to retrieve the list of keys from the sorted array. On the sortedArrayFromDictionary variable we call map({ $0.0}), the transform passed to the map method will add the .0 element of each array element in sortedArrayFromDictionary to the new array returned by the map method. The last step is to retrieve a list of values from the sorted array. We are performing the same call to the map method as we did to retrieve the list of keys, this time we want the .1 element of each array element in sortedArrayFromDictionary, though. Like the preceding example, these values will be added to the new array returned by the map method.     But what if you wanted to base your sorting order on the dictionary value instead of the key? This is simple to do; you would just change the parameter syntax that is passed to the sort method. Changing it to states.sort({ $0.1 < $1.1 }) will now compare the first arguments value from the key-value pair, $0.1, against the second arguments value from its key-value pair, $1.1, and adding the lesser of the two to the new array that will be returned. Declaring Sets A Set is an unordered collection of unique, non-nil elements with no defined ordering. A Set type must conform to the Hashtable protocol to be stored in a set. All of Swift's basic types are hashable by default. Enumeration case values that do not use associate values are also hashable by default. You can store a custom type in a Set, you'll just need to ensure that it conforms to the Hashable protocol, as well as the Equatable protocol since Hashtable conforms to it. Sets can be used anywhere you would use an array when ordering is not important and you want to ensure only unique elements are stored. Additionally, access time has the potential of being more efficient over arrays. With an array, the worst case scenario when searching for an element is O(n) where n is the size of the array. Whereas accessing an element in a Set is always constant time O(1), regardless of its size. Initializing a Set Unlike the other collection types, Sets cannot be inferred from an array literal alone and must be explicitly declared by specifying the Set type: // Full syntax declaration var stringSet = Set<String>() Because of Swift's type inference, you do not have to specify the type of the Set that you're initializing, it will infer the type based on the array literal it is initialized with. Remember though that the array literal must contain the same types: // Initialize a Set from an array literal var stringSet: Set = ["Mary", "John", "Sally"] print(stringSet.debugDescription) // Out of debugDescription shows stringSet is indeed a Set type "Set(["Mary", "John", "Sally"])" Modifying and Retrieving Elements of a Set To add a new element to a Set use the insert(_:) method. You can check if an element is already stored in a Set using the contains(_:) method. Swift provides several methods for removing elements from a Set, if you have an instance of the element you want to remove you can use the remove(_:) method, which takes an element instance. If you know the index of an element in the Set you can use the remove(at:) method, which takes an instance of SetIndex<Element>. If the Set count is greater than 0 you can use the removeFirst() method to remove the element and the starting index. Lastly, if you want to remove all elements from a Set, use the removeAll() method or removeAll(keepCapacity) method, if keepCapacity is true the current capacity will not decrease: var stringSet: Set = ["Erik", "Mary", "Michael", "John", "Sally"] // ["Mary", "Michael", "Sally", "John", "Erik"] stringSet.insert("Patrick") // ["Mary", "Michael", "Sally", "Patrick", "John", "Erik"] if stringSet.contains("Erik") { print("Found element") } else { print("Element not found") } // Found element stringSet.remove("Erik") // ["Mary", "Sally", "Patrick", "John", "Michael"] if let idx = stringSet.index(of: "John") { stringSet.remove(at: idx) } // ["Mary", "Sally", "Patrick", "Michael"] stringSet.removeFirst() // ["Sally", "Patrick", "Michael"] stringSet.removeAll() // [] You can iterate over a Set the same as you would the other collection types by using the for…in loop. The Swift Set type is unordered, so you can use the sort method like we did for the Dictionary type if you want to iterate over elements in a specific order: var stringSet: Set = ["Erik", "Mary", "Michael", "John", "Sally"] // ["Mary", "Michael", "Sally", "John", "Erik"] for name in stringSet { print("name = (name)") } // name = Mary // name = Michael // name = Sally // name = John // name = Erik for name in stringSet.sorted() { print("name = (name)") } // name = Erik // name = John // name = Mary // name = Michael // name = Sally Set Operations The Set type is modeled after the mathematical Set Theory and it implements methods that support basic Set operations for comparing two Sets, as well as operations that perform membership and equality comparisons between two Sets. Comparison Operations The Swift Set type contains four methods for performing common operations on two sets. The operations can be performed either by returning a new set, or using the operations alternate InPlace method to perform the operation in place on the source Set. The union(_:) and formUnion(_:) methods create a new Set or update the source Set with all the values from both Sets, respectively. The intersection(_:) and formIntersection(_:) methods create a new Set or update the source Set with values only common to both Sets, respectively. The symmetricDifference(_:) or formSymmetricDifference(_:) methods create a new Set or update the source Set with values in either Set, but not both, respectively. The subtracting(_:) or subtract(_:) methods create a new Set or update the source Set with values not in the specified Set: let adminRole: Set = [ "READ", "EDIT", "DELETE", "CREATE", "SETTINGS", "PUBLISH_ANY", "ADD_USER", "EDIT_USER", "DELETE_USER"] let editorRole: Set = ["READ", "EDIT", "DELETE", "CREATE", "PUBLISH_ANY"] let authorRole: Set = ["READ", "EDIT_OWN", "DELETE_OWN", "PUBLISH_OWN", "CREATE"] let contributorRole: Set = [ "CREATE", "EDIT_OWN"] let subscriberRole: Set = ["READ"] // Contains values from both Sets let fooResource = subscriberRole.union(contributorRole) // "READ", "EDIT_OWN", "CREATE" // Contains values common to both Sets let commonPermissions = authorRole.intersection(contributorRole) // "EDIT_OWN", "CREATE" // Contains values in either Set but not both let exclusivePermissions = authorRole.symmetricDifference(contributorRole) // "PUBLISH_OWN", "READ", "DELETE_OWN" Membership and Equality Operations Two sets are said to be equal if they contain precisely the same values, and since Sets are unordered, the ordering of the values between Sets does not matter. Use the == operator, which is the "is equal" operator, to determine if two Sets contain all of the same values: // Note ordering of the sets does not matter var sourceSet: Set = [1, 2, 3] var destSet: Set = [2, 1, 3] var isequal = sourceSet == destSet // isequal is true Use the isSubset(of:) method to determine if all of the values of a Set are contained in a specified Set. Use the isStrictSubset(of:) method to determine if a Set is a subset, but not equal to the specified Set. Use the isSuperset(of:) method to determine if a Set contains all of the values of the specified Set. Use the isStrictSuperset(of:) method to determine if a Set is a superset, but not equal to the specified Set. Use the isDisjoint(with:) method to determine if two Sets have the same values in common: let contactResource = authorRole // "EDIT_OWN", "PUBLISH_OWN", "READ", "DELETE_OWN", "CREATE" let userBob = subscriberRole // "READ" let userSally = authorRole // "EDIT_OWN", "PUBLISH_OWN", "READ", "DELETE_OWN", "CREATE" if userBob.isSuperset(of: fooResource){ print("Access granted") } else { print("Access denied") } // "Access denied" if userSally.isSuperset(of: fooResource){ print("Access granted") } else { print("Access denied") } // Access granted authorRole.isDisjoint(with: editorRole) // false editorRole.isSubset(of: adminRole) // true Summary In this article, we've learned about the difference between classes and structures and when you would use one type over another, as well as characteristics of value types and reference types and how each type is allocated at runtime. We went into some of the implementation details for the Array, Dictionary, and Set collection types that are implemented in the Swift standard library. Resources for Article: Further resources on this subject: Python Data Structures [Article] The pandas Data Structures [Article] Python Data Persistence using MySQL Part III: Building Python Data Structures Upon the Underlying Database Data [Article]
Read more
  • 0
  • 0
  • 2750

article-image-getting-started-sorting-algorithms-java
Packt
16 Nov 2016
9 min read
Save for later

Getting Started with Sorting Algorithms in Java

Packt
16 Nov 2016
9 min read
In this article by Peter Verhas author of the book Java 9 Programming By Example, we will develop a simple sort program. Using this code as an example, we will look at different build tools, which are frequently used for Java projects, and learn the basic features of the Java language. (For more resources related to this topic, see here.) The problem we will solve The sorting problem is one of the oldest programming tasks that an engineer solves. We will have a set of records and we know that we will want to find a specific one sometime later, and we will want to find that one fast. To find it, we will sort the records in a specific order that helps finding the record we want fast. As an example, we can have the names of the students with some marks on cards. When students will come to the office asking for the result, we can turn all pages one after the other to find the name of the enquiring student. However, it is better if we sort the papers by the name of the students lexicographically. When a student comes, we can search the mark attached to the name much faster. We can look at the middle card; if it shows the name of the student, then we are happy to have found the name and the mark. If the card precedes the name of the student lexicographically, then we will continue searching in the second half, otherwise the first half. Following that approach, we can find the name of the student in no more steps than as many times the pack of cards can be halved. If we have two cards, then it is two steps at most. If it is four, then we will need three steps at most. If there are eight cards, then we may need four steps, but not more. If there are 1000 cards, then we may need at most 11 steps, while the original, non-sorted set will need 1000 steps, worst case. That is, approximately, it speeds up the search 100 times, so this is worth sorting the cards, unless the sorting itself takes too much time. In many cases, it is worth sorting the dataset and there are many sorting algorithms to do that. There are simpler and more complex algorithms, and as in many cases, more complex algorithms are the one that run faster. As we are focusing on the Java programming part and not the algorithm forging, in this article, we will develop a Java code that implements a simple and not-that-fast algorithm. Bubble sort The algorithm that we will implement in this article is well known as bubble sort. The approach is very simple. Begin at the start of the cards and compare the first and the second card. If the first card is later in lexicographic order than the second one, then swap the two cards. Then, repeat this for the card that is at the second place now, then the third, and so on. There is a card that is lexicographically the latest, say Wilson, and sometime later, we will get to this card as we go on swapping, the cards going from start to end. When we get this card and start to compare it with the next one, we will always swap them; this way, Wilson's card will travel to the last place where it has to be after the sort. All we have to do is repeat this travelling from the start and the occasional swapping of cards again, but this time only to the last but one element. This time, the second latest element will get to its place—say Wilkinson will be right before Wilson. If we have n cards, and we repeat this n-1 times, all cards will get to their place. Project structure and build tools When a project is more complex than a single class, and it usually is, then it is wise to define a project structure. We will have to decide where we store the source files, where the resource files (those that contain some resource for the program, but are not Java source) are, where should the .class files be written by the compiler, and so on. Generally, the structure is mainly the directory setup and configuring the tools that perform the build that use these tools. The compilation of complex programs cannot be feasibly done using the command line issuing javac commands. If we have a 100 Java source files, the compilation will require that many javac commands to be issued. We can write a simple bash script that does that. First, it will be just 100 lines, each compiling one source Java file to class file. Then, we will realize that this is only time, CPU, and power consuming to compile the files that are not changed since the last compilation. So, we can add some bash programming that checks the time stamp on the source and generated files. Then, we will probably realize that… whatever. At the end, we will end up with a tool that is essentially a build tool. And, this is already done. Instead of creating one, we will use a build tool that is ready. There are a few of them that can be found at https://en.wikipedia.org/wiki/List_of_build_automation_software Make The Make program was originally created in April 1976, so this is not a new tool. It is included in the Unix system so this tool is available without any extra installation on Linux, Mac OS X, or any other Unix-based system. Additionally, there are numerous ports of the tool on Windows and some version is/was included in the Visual C compiler toolset. The Make is not tied to Java. It was created when the major programming language was C, but it is not tied to C or any other language. Make is a dependency description language that has a very simple syntax. The Make, just like any other build tool, works controlled by a project description file. In case of make, this file contains a rule set. The description file is usually named Makefile, but in case the name of the description file is different, it can be specified as a command-line option to the make command. Rules in Makefile follow each other and a it is one or more lines. The first line starts at the first position (there is no tab or space at the start of the line) and the following lines start with a tab character. Thus, Makefile may look something like the following code: run : hello.jar java -cp hello.jar HelloWorld hello.jar : HelloWorld.class jar -cf hello.jar HelloWorld.class HelloWorld.class : HelloWorld.java javac HelloWorld.java The file defines three so-called targets: run, hello.jar, and HelloWorld.class. To create HelloWorld.class, type the following line at the Command Prompt: make HelloWorld.class The make will look at the rule and see that it depends on HelloWorld.java. If the HelloWorld.class file does not exist, or HelloWorld.java is newer than the Java source file, make will execute the command that is written on the next line and it will compile the Java source file. If the class file was created following the last modification of HelloWorld.java, then make knows that there is no need to run the command. In case of creating HelloWorld.class,the make program has an easy task. The source file was already there. If you issue the make hello.jar command, the procedure is more complex. The make command sees that in order to create hello.jar, it needs HelloWorld.class, which itself is also a target on another rule. Thus, it may need to be created. First, it starts the problem the same way as before. If HelloWorld.class is there, and is older than hello.jar, there is nothing to do. If it is not there, or is newer than hello.jar, then the jar -cf hello.jar HelloWorld.class command needs to be executed, but not yet. It remembers that this command has to be executed sometime in the future when all the commands that are needed to create HelloWorld.class are already executed successfully. Thus, it continues to create the class file exactly the same way as I already described earlier. In general, a rule can have the following format: target : dependencies command The make command can create any target using the make target command by first calculating which commands to execute and then executing them one by one. The commands are shell commands executing in a different process and may pose problems under Windows, which may render the Makefile files operating system dependent. Note that the run target is not an actual file that make creates. A target can be a file name or just a name for the target. In the latter case, make will never consider the readily available target. As we do not use make for Java project, there is no room to get into more details. Additionally, I cheated a bit by making the description of a rule simpler than it should be. The make tool has many powerful features out of the scope of this book. There are also several implementations that differ a little from each other. You will most probably meet the one made by the Free Software Foundation—the GNU make. And, of course, just in case of any Unix command-line tool, man is your friend. The man make command will display the documentation of the tool on the screen. The main points that you should remember about make are as follows: It defines the dependencies of the individual artifacts (targets) in a declarative way It defines the actions to create the missing artifacts in an imperative way. Summary In this article, we have developed a very basic sort algorithm. It was made purposefully simple so that we could reiterate on the basic and most important Java language elements, classes, packages, variables, methods, and so on. Resources for Article: Further resources on this subject: Algorithm Analysis [article] Introduction to C# and .NET [article] Parallel Computing [article]
Read more
  • 0
  • 0
  • 4613
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at €18.99/month. Cancel anytime
article-image-setting-development-environment-android-wear-applications
Packt
16 Nov 2016
8 min read
Save for later

Setting up Development Environment for Android Wear Applications

Packt
16 Nov 2016
8 min read
"Give me six hours to chop down a tree and I will spend the first four sharpening the axe." -Abraham Lincoln In this article by Siddique Hameed, author of the book, Mastering Android Wear Application Development, they have discussed the steps, topics and process involved in setting up the development environment using Android Studio. If you have been doing Android application development using Android Studio, some of the items discussed here might already be familiar to you. However, there are some Android Wear platform specific items that may be of interest to you. (For more resources related to this topic, see here.) Android Studio Android Studio Integrated Development Environment (IDE) is based on IntelliJ IDEA platform. If you had done Java development using IntelliJ IDEA platform, you'll be feeling at home working with Android Studio IDE. Android Studio platform comes bundled with all the necessary tools and libraries needed for Android application development. If this is the first time you are setting up Android Studio on your development system, please make sure that you have satisfied all the requirements before installation. Please refer developer site (http://developer.android.com/sdk/index.html#Requirements) for checking on the items needed for the operating system of your choice. Please note that you need at least JDK version 7 installed on your machine for Android Studio to work. You can verify your JDK's version that by typing following commands shown in the following terminal window: If your system does not meet that requirement, please upgrade it using the method that is specific to your operating system. Installation Android Studio platform includes Android Studio IDE, SDK Tools, Google API Libraries and systems images needed for Android application development. Visit the http://developer.android.com/sdk/index.html URL for downloading Android Studio for your corresponding operating system and following the installation instruction. Git and GitHub Git is a distributed version control system that is used widely for open-source projects. We'll be using Git for sample code and sample projects as we go along the way. Please make sure that you have Git installed on your system by typing the command as shown in the following a terminal window. If you don't have it installed, please download and install it using this link for your corresponding operating system by visting https://git-scm.com/downloads. If you are working on Apple's Macintosh OS like El Capitan or Yosemite or Linux distributions like Ubuntu, Kubuntu or Mint, chances are you already have Git installed on it. GitHub (http://github.com) is a free and popular hosting service for Git based open-source projects. They make checking out and contributing to open-source projects easier than ever. Sign up with GitHub for a free account if you don't have an account already. We don't need to be an expert on Git for doing Android application development. But, we do need to be familiar with basic usages of Git commands for working with the project. Android Studio comes by default with Git and GitHub integration. It helps importing sample code from Google's GitHub repository and helps you learn by checking out various application code samples. Gradle Android application development uses Gradle (http://gradle.org/)as the build system. It is used to build, test, run and package the apps for running and testing Android applications. Gradle is declarative and uses convention over configuration for build settings and configurations. It manages all the library dependencies for compiling and building the code artifacts. Fortunately, Android Studio abstracts most of the common Gradle tasks and operations needed for development. However, there may be some cases where having some extra knowledge on Gradle would be very helpful. We won't be digging into Gradle now, we'll be discussing about it as and when needed during the course of our journey. Android SDK packages When you install Android Studio, it doesn't include all the Android SDK packages that are needed for development. The Android SDK separates tools, platforms and other components & libraries into packages that can be downloaded, as needed using the Android SDK Manager. Before we start creating an application, we need to add some required packages into the Android SDK. Launch SDK Manager from Android Studio Tools | Android | SDK Manager. Let's quickly go over a few items from what's displayed in the preceding screenshot. As you can see, the Android SDK's location is /opt/android-sdk on my machine, it may very well be different on your machine depending on what you selected during Android Studio installation setup. The important point to note is that the Android SDK is installed on a different location than the Android Studio's path (/Applications/Android Studio.app/). This is considered a good practice because the Android SDK installation can be unaffected depending on a new installation or upgrade of Android Studio or vice versa. On the SDK Platforms tab, select some recent Android SDK versions like Android 6.0, 5.1.1, and 5.0.1. Depending on the Android versions you are planning on supporting in your wearable apps, you can select other older Android versions. Checking on Show Package Details option on the bottom right, the SDK Manager will show all the packages that will be installed for a given Android SDK version. To be on the safer side, select all the packages. As you may have noticed already Android Wear ARM and Intel system images are included in the package selection. Now when you click on SDK Tools tab, please make sure the following items are selected: Android SDK Build Tools Android SDK Tools 24.4.1 (Latest version) Android SDK Platform-Tools Android Support Repository, rev 25 (Latest version) Android Support Library, rev 23.1.1 (Latest version) Google Play services, rev 29 (Latest version) Google Repository, rev 24 (Latest version) Intel X86 Emulator Accelerator (HAXM installer), rev 6.0.1 (Latest version) Documentation for Android SDK (Optional) Please do not change anything on SDK Update Sites. Keep the update sites as it was configured by default. Clicking on OK will take some time downloading and installing all the components and packages selected. Android Virtual Device Android Virtual Devices will enable us to test the code using Android Emulators. It lets us pick and choose various Android system target versions and form factors needed for testing. Launch Android Virtual Device Manager from Tools | Android | AVD Manager From Android Virtual Device Manager window, click on Create New Virtual Device button on the bottom left and proceed to the next screen and select Wear category Select Marshmallow API Level 23 on x86 and everything else as default, as shown in the following screenshot: Note that the current latest Android version is Marshmallow of API level 23 at the time of this writing. It may or may not be the latest version while you are reading this article. Feel free to select the latest version that is available during that time. Also, if you'd like to support or test in earlier Android versions, please feel free to do so in that screen. After the virtual device is selected successfully, you should see that listed on the Android Virtual Devices list as show in the following screenshot: Although it's not required to use real Android Wear device during development, sometimes it may be convenient and faster developing it in a real physical device. Let's build a skeleton App Since we have all the components and configurations needed for building wearable app, let's build a skeleton app and test out what we have so far. From Android Studio's Quick Start menu, click on Import an Android code sample tab: Select the Skeleton Wearable App from Wearable category as shown in following screenshot: Click Next and select your preferred project location. As you can see the skeleton project is cloned from Google's sample code repository from GitHub. Clicking on Finish button will pull the source code and Android Studio will compile and build the code and get it ready for execution. The following screenshot indicates that the Gradle build finished successfully without any errors. Click on Run configuration to run the app: When the app starts running, Android Studio will prompt us to select the deployment targets, we can select the emulator we created earlier and click OK. After the code compiles and uploaded to the emulator, the main activity of the Skeleton App will be launched as shown below: Clicking on SHOW NOTIFICATION will show the notification as below: Clicking on START TIMER will start the timer and run for five seconds and clicking on Finish Activity will close the activity take the emulator to the home screen. Summary We discussed the process involved in setting up the Android Studio development environment by covering the installation instruction, requirements and SDK tools, packages and other components needed for Android Wear development. We also checked out source code for Skeleton Wearable App from Google's sample code repository and successfully ran and tested it on Android device emulator. Resources for Article: Further resources on this subject: Building your first Android Wear Application [Article] Getting started with Android Development [Article] The Art of Android Development Using Android Studio [Article]
Read more
  • 0
  • 0
  • 7841

article-image-multithreading-qt
Packt
16 Nov 2016
13 min read
Save for later

Multithreading with Qt

Packt
16 Nov 2016
13 min read
Qt has its own cross-platform implementation of threading. In this article by Guillaume Lazar and Robin Penea, authors of the book Mastering Qt 5, we will study how to use Qt and the available tools provided by the Qt folks. (For more resources related to this topic, see here.) More specifically, we will cover the following: Understanding the QThread framework in depth The worker model and how you can offload a process from the main thread An overview of all the available threading technologies in Qt Discovering QThread Qt provides a sophisticated threading system. We assume that you already know threading basics and the associated issues (deadlocks, threads synchronization, resource sharing, and so on) and we will focus on how Qt implements it. The QThread is the central class for of the Qt threading system. A QThread instance manages one thread of execution within the program. You can subclass QThread to override the run() function, which will be executed in the QThread class. Here is how you can create and start a QThread: QThread thread; thread.start(); The start() function calling will automatically call the run() function of thread and emit the started() signal. Only at this point, the new thread of execution will be created. When run() is completed, thread will emit the finished() signal. This brings us to a fundamental aspect of QThread: it works seamlessly with the signal/slot mechanism. Qt is an event-driven framework, where a main event loop (or the GUI loop) processes events (user input, graphical, and so on) to refresh the UI. Each QThread comes with its own event loop that can process events outside the main loop. If not overridden, run() calls the QThread::exec() function, which starts the thread's event loop. You can also override QThread and call exec(), as follows: class Thread : public QThread { Q_OBJECT protected: void run() { Object* myObject = new Object(); connect(myObject, &Object::started, this, &Thread::doWork); exec(); } private slots: void doWork(); }; The started()signal will be processed by the Thread event loop only upon the exec() call. It will block and wait until QThread::exit() is called. A crucial thing to note is that a thread event loop delivers events for all QObject classes that are living in that thread. This includes all objects created in that thread or moved to that thread. This is referred to as the thread affinity of an object. Here's an example: class Thread : public QThread { Thread() : mObject(new QObject()) { } private : QObject* myObject; }; // Somewhere in MainWindow Thread thread; thread.start(); In this snippet, myObject is constructed in the Thread constructor, which is created in turn in MainWindow. At this point, thread is living in the GUI thread. Hence, myObject is also living in the GUI thread. An object created before a QCoreApplication object has no thread affinity. As a consequence, no event will be dispatched to it. It is great to be able to handle signals and slots in our own QThread, but how can we control signals across multiple threads? A classic example is a long running process that is executed in a separate thread that has to notify the UI to update some state: class Thread : public QThread { Q_OBJECT void run() { // long running operation emit result("I <3 threads"); } signals: void result(QString data); }; // Somewhere in MainWindow Thread* thread = new Thread(this); connect(thread, &Thread::result, this, &MainWindow::handleResult); connect(thread, &Thread::finished, thread, &QObject::deleteLater); thread->start(); Intuitively, we assume that the first connect function sends the signal across multiple threads (to have a result available in MainWindow::handleResult), whereas the second connect function should work on thread's event loop only. Fortunately, this is the case due to a default argument in the connect() function signature: the connection type. Let's see the complete signature: QObject::connect( const QObject *sender, const char *signal, const QObject *receiver, const char *method, Qt::ConnectionType type = Qt::AutoConnection) The type variable takes Qt::AutoConnection as a default value. Let's review the possible values of Qt::ConectionType enum as the official Qt documentation states: Qt::AutoConnection: If the receiver lives in the thread that emits the signal, Qt::DirectConnection is used. Otherwise, Qt::QueuedConnection is used. The connection type is determined when the signal is emitted. Qt::DirectConnection: This slot is invoked immediately when the signal is emitted. The slot is executed in the signaling thread. Qt::QueuedConnection: The slot is invoked when control returns to the event loop of the receiver's thread. The slot is executed in the receiver's thread. Qt::BlockingQueuedConnection: This is the same as Qt::QueuedConnection, except that the signaling thread blocks until the slot returns. This connection must not be used if the receiver lives in the signaling thread or else the application will deadlock. Qt::UniqueConnection: This is a flag that can be combined with any one of the preceding connection types, using a bitwise OR element. When Qt::UniqueConnection is set, QObject::connect() will fail if the connection already exists (that is, if the same signal is already connected to the same slot for the same pair of objects). When using Qt::AutoConnection, the final ConnectionType is resolved only when the signal is effectively emitted. If you look again at our example, the first connect(): connect(thread, &Thread::result, this, &MainWindow::handleResult); When the result() signal will be emitted, Qt will look at the handleResult() thread affinity, which is different from the thread affinity of the result() signal. The thread object is living in MainWindow (remember that it has been created in MainWindow), but the result() signal has been emitted in the run() function, which is running in a different thread of execution. As a result, a Qt::QueuedConnection function will be used. We will now take a look at the second connect(): connect(thread, &Thread::finished, thread, &QObject::deleteLater); Here, deleteLater() and finished() live in the same thread, therefore, a Qt::DirectConnection will be used. It is crucial that you understand that Qt does not care about the emitting object thread affinity, it looks only at the signal's "context of execution." Loaded with this knowledge, we can take another look at our first QThread example to have a complete understanding of this system: class Thread : public QThread { Q_OBJECT protected: void run() { Object* myObject = new Object(); connect(myObject, &Object::started, this, &Thread::doWork); exec(); } private slots: void doWork(); }; When Object::started() is emitted, a Qt::QueuedConnection function will be used. his is where your brain freezes. The Thread::doWork() function lives in another thread than Object::started(), which has been created in run(). If the Thread has been instantiated in the UI Thread, this is where doWork() would have belonged. This system is powerful but complex. To make things more simple, Qt favors the worker model. It splits the threading plumbing from the real processing. Here is an example: class Worker : public QObject { Q_OBJECT public slots: void doWork() { emit result("workers are the best"); } signals: void result(QString data); }; // Somewhere in MainWindow QThread* thread = new Thread(this); Worker* worker = new Worker(); worker->moveToThread(thread); connect(thread, &QThread::finished, worker, &QObject::deleteLater); connect(this, &MainWindow::startWork, worker, &Worker::doWork); connect(worker, &Worker::resultReady, this, handleResult); thread->start(); // later on, to stop the thread thread->quit(); thread->wait(); We start by creating a Worker class that has the following: A doWork()slot that will have the content of our old QThread::run() function A result()signal that will emit the resulting data Next, in MainWindow, we create a simple thread and an instance of Worker. The worker->moveToThread(thread) function is where the magic happens. It changes the affinity of the worker object. The worker now lives in the thread object. You can only push an object from your current thread to another thread. Conversely, you cannot pull an object that lives in another thread. You cannot change the thread affinity of an object if the object does not live in your thread. Once thread->start() is executed, we cannot call worker->moveToThread(this) unless we are doing it from this new thread. After that, we will use three connect() functions: We handle worker life cycle by reaping it when the thread is finished. This signal will use a Qt::DirectConnection function. We start the Worker::doWork() upon a possible UI event. This signal will use a Qt::QueuedConnection. We process the resulting data in the UI thread with handleResult(). This signal will use a Qt::QueuedConnection. To sum up, QThread can be either subclassed or used in conjunction with a worker class. Generally, the worker approach is favored because it separates more cleanly the threading affinity plumbing from the actual operation you want to execute in parallel. Flying over Qt multithreading technologies Built upon QThread, several threading technologies are available in Qt. First, to synchronize threads, the usual approach is to use a mutual exclusion (mutex) for a given resource. Qt provides it by the mean of the QMutex class. Its usage is straightforward: QMutex mutex; int number = 1; mutex.lock(); number *= 2; mutex.unlock(); From the mutex.lock() instruction, any other thread trying to lock the mutex object will wait until mutex.unlock() has been called. The locking/unlocking mechanism is error prone in complex code. You can easily forget to unlock a mutex in a specific exit condition, causing a deadlock. To simplify this situation, Qt provides a QMutexLocker that should be used where the QMutex needs to be locked: QMutex mutex; QMutexLocker locker(&mutex); int number = 1; number *= 2; if (overlyComplicatedCondition) { return; } else if (notSoSimple) { return; } The mutex is locked when the locker object is created, and it will be unlocked when locker is destroyed, for example, when it goes out of scope. This is the case for every condition we stated where the return statement appears. It makes the code simpler and more readable. If you need to create and destroy threads frequently, managing QThread instances by hand can become cumbersome. For this, you can use the QThreadPool class, which manages a pool of reusable QThreads. To execute code within threads managed by a QThreadPool, you will use a pattern very close to the worker we covered earlier. The main difference is that the processing class has to extend the QRunnable class. Here is how it looks: class Job : public QRunnable { void run() { // long running operation } } Job* job = new Job(); QThreadPool::globalInstance()->start(job); Just override the run() function and ask QThreadPool to execute your job in a separate thread. The QThreadPool::globalInstance() function is a static helper function that gives you access to an application global instance. You can create your own QThreadPool class if you need to have a finer control over the QThreadPool life cycle. Note that QThreadPool::start() takes the ownership of the job object and will automatically delete it when run() finishes. Watch out, this does not change the thread affinity like QObject::moveToThread() does with workers! A QRunnable class cannot be reused, it has to be a freshly baked instance. If you fire up several jobs, QThreadPool automatically allocates the ideal number of threads based on the core count of your CPU. The maximum number of threads that the QThreadPool class can start can be retrieved with QThreadPool::maxThreadCount(). If you need to manage threads by hand, but you want to base it on the number of cores of your CPU, you can use the handy static function, QThreadPool::idealThreadCount(). Another approach to multithreaded development is available with the Qt Concurrent framework. It is a higher level API that avoids the use of mutexes/locks/wait conditions and promotes the distribution of the processing among CPU cores. Qt Concurrent relies of the QFuture class to execute a function and expect a result later on: void longRunningFunction(); QFuture<void> future = QtConcurrent::run(longRunningFunction); The longRunningFunction() will be executed in a separated thread obtained from the default QThreadPool class. To pass parameters to a QFuture class and retrieve the result of the operation, use the following code: QImage processGrayscale(QImage& image); QImage lenna; QFuture<QImage> future = QtConcurrent::run(processGrayscale, lenna); QImage grayscaleLenna = future.result(); Here, we pass lenna as a parameter to the processGrayscale() function. Because we want a QImage as a result, we declare QFuture with the template type QImage. After that, future.result() blocks the current thread and waits for the operation to be completed to return the final QImage template type. To avoid blocking, QFutureWatcher comes to the rescue: QFutureWatcher<QImage> watcher; connect(&watcher, &QFutureWatcher::finished, this, &QObject::handleGrayscale); QImage processGrayscale(QImage& image); QImage lenna; QFuture<QImage> future = QtConcurrent::run(processImage, lenna); watcher.setFuture(future); We start by declaring a QFutureWatcher with the template argument matching the one used for QFuture. Then, simply connect the QFutureWatcher::finished signal to the slot you want to be called when the operation has been completed. The last step is to the tell the watcher to watch the future object with watcher.setFuture(future). This statement looks almost like it's coming from a science fiction movie. Qt Concurrent also provides a MapReduce and FilterReduce implementation. MapReduce is a programming model that basically does two things: Map or distribute the processing of datasets among multiple cores of the CPU Reduce or aggregate the results to provide it to the caller check styleThis technique has been first promoted by Google to be able to process huge datasets within a cluster of CPU. Here is an example of a simple Map operation: QList images = ...; QImage processGrayscale(QImage& image); QFuture<void> future = QtConcurrent::mapped( images, processGrayscale); Instead of QtConcurrent::run(), we use the mapped function that takes a list and the function to apply to each element in a different thread each time. The images list is modified in place, so there is no need to declare QFuture with a template type. The operation can be made a blocking operation using QtConcurrent::blockingMapped() instead of QtConcurrent::mapped(). Finally, a MapReduce operation looks like this: QList images = ...; QImage processGrayscale(QImage& image); void combineImage(QImage& finalImage, const QImage& inputImage); QFuture<void> future = QtConcurrent::mappedReduced( images, processGrayscale, combineImage); Here, we added a combineImage() that will be called for each result returned by the map function, processGrayscale(). It will merge the intermediate data, inputImage, into the finalImage. This function is called only once at a time per thread, so there is no need to use a mutex object to lock the result variable. The FilterReduce reduce follows exactly the same pattern, the filter function simply allows to filter the input list instead of transforming it. Summary In this article, we discovered how a QThread works and you learned how to efficiently use tools provided by Qt to create a powerful multi-threaded application. Resources for Article: Further resources on this subject: QT Style Sheets [article] GUI Components in Qt 5 [article] DOM and QTP [article]
Read more
  • 0
  • 1
  • 42447

article-image-manual-and-automated-testing
Packt
15 Nov 2016
10 min read
Save for later

Manual and Automated Testing

Packt
15 Nov 2016
10 min read
In this article by Claus Führer the author of the book Scientific Computing with Python 3, we focus on two aspects of testing for scientific programming: Manual and Automated testing. Manual testing is what is done by every programmer to quickly check that an implementation is working. Automated testing is the refined, automated variant of that idea. We will introduce some tools available for automatic testing in general, with a view on the particular case of scientific computing. (For more resources related to this topic, see here.) Manual Testing During the development of code you do a lot of small tests in order to test its functionality. This could be called Manual Testing. Typically, you would test that a given function does what it is supposed to do, by manually testing the function in an interactive environment. For instance, suppose that you implement the Bisection algorithm. It is an algorithm that finds a zero (root) of a scalar nonlinear function. To start the algorithm an interval has to be given with the property, that the function takes different signs on the interval boundaries. You would then test an implementation of that algorithm typically by checking: That a solution is found when the function has opposite signs at the interval boundaries that an exception is raised when the function has the same sign at the interval boundaries Manual testing, as necessary as may seem to be, is unsatisfactory. Once you convinced yourself that the code does what it is supposed to do, you formulate a relatively small number of demonstration examples to convince others of the quality of the code. At that stage one often loses interest in the tests made during development and they are forgotten or even deleted. As soon as you change a detail and things no longer work correctly you might regret that your earlier tests are no longer available. Automatic Testing The correct way to develop any piece of code is to use automatic testing. The advantages are The automated repetition of a large number of tests after every code refactoring and before new versions are launched A silent documentation of the use of the code A documentation of the test coverage of your code: Did things work before a change or was a certain aspect never tested? We suggest to develop tests in parallel to the code. Good design of tests is an art of its own and there is rarely an investment which guarantees such a good pay-off in development time savings as the investment in good tests. Now we will go through the implementation of a simple algorithm with the automated testing methods in mind. Testing the bisection algorithm Let us examine automated testing for the bisection algorithm. With this algorithm a zero of a real valued function is found. An implementation of the algorithm can have the following form: def bisect(f,a,b,tol=1.e-8): """ Implementation of the bisection algorithm f real valued function a,b interval boundaries (float) with the property f(a)*f(b)<=0 tol tolerance ( float ) """ if f(a)*f(b)>0: raise ValueError ("Incorrect initial interval [a,b]") for i in range (100): c = (a + b)/2 . if f (a)*f(c) <= 0: b=c else: a=c if abs (a - b)<tol: return (a + b)/2 raise Exception (’ No root found within the given tolerance { }’.format (tol) We assume this to be stored in a file bisection.py. As a first test case we test that the zero of the function F(x) = x is found: def test_identity(): result = bisect(lambda x: x, -1., 1.) #(for lambda) expected = 0. assert allclose(result, expected),’expected zero not found’ text_identity() In this code you meet the Python keyword assert for the first time. It raises an exception AssertionError if its first argument returns the value False. Its optional second argument is a string with additional information. We use the function allclose in order to test for equality for float. Let us comment on some of the features of the test function. We use an assertion to make sure that an exception will be raised if the code does not behave as expected. We have to manually run the test in the line test_identity(). There are many tools to automate this kind of call. Let us now setup a test that checks if bisect raises an exception when the function has the same sign on both ends of the interval. For now, we will suppose that the exception raised is a ValueError exception. Example: Checking the sign for the bisection algorithm. def test_badinput(): try: bisect(lambda x: x,0.5,1) except ValueError: pass else: raise AssertionError() test_badinput() In this case an AssertionError is raised if the exception is not of type ValueError. There are tools to simplify the above construction to check that an exception is raised. Another useful kind of tests is the edge case test. Here we test arguments or user input which is likely to create mathematically undefined situations or states of the program not foreseen by the programmer. For instance, what happens if both bounds are equal? What happens if a>b? We easily setup up such a test by using for instance def test_equal_boundaries(): result = bisect(lambda x: x, 1., 1.) expected = 0. assert allclose(result, expected), ‘test equal interval bounds failed’ def test_reverse_boundaries(): result = bisect(lambda x: x, 1., -1.) expected = 0. assert allclose(result, expected), ‘test reverse interval bounds failed’ test_equal_boundaries() test_reverse_boundaries() Using unittest The standard Python package unittest greatly facilitates automated testing. That package requires that we rewrite our tests a little to be compatible. The first test would have to be rewritten in a class, as follows: from bisection import bisect import unittest class TestIdentity(unittest.TestCase): def test(self): result = bisect(lambda x: x, -1.2, 1.,tol=1.e-8) expected = 0. self.assertAlmostEqual(result, expected) if __name__==‘__main__’: unittest.main() Let us examine the differences to the previous implementation. First, the test is now a method and a part of a class. The class must inherit from unittest,TestCase. The test method’s name must start with test. Note that we may now use one of the assertion tools of the package, namely       . Finally, the tests are run using unittest.main. We recommend to write the tests in a file separate from the code to be tested. That’s why it starts with an import. The test passes and returns Ran 1 test in 0.002s OK If we would have run it with a loose tolerance parameter, e.g., 1.e-3, a failure of the test would have been reported: F ========================================================== FAIL: test (__main__.TestIdentity) ---------------------------------------------------------------------- Traceback (most recent call last): File “<ipython-input-11-e44778304d6f>“, line 5, in test self.assertAlmostEqual(result, expected) AssertionError: 0.00017089843750002018 != 0.0 within 7 places --------------------------------------------------------------------- Ran 1 test in 0.004s FAILED (failures=1) Tests can and should be grouped together as methods of a test class: Example: import unittest from bisection import bisect class TestIdentity(unittest.TestCase): def identity_fcn(self,x): return x def test_functionality(self): result = bisect(self.identity_fcn, -1.2, 1.,tol=1.e-8) expected = 0. self.assertAlmostEqual(result, expected) def test_reverse_boundaries(self): result = bisect(self.identity_fcn, 1., -1.) expected = 0. self.assertAlmostEqual(result, expected) def test_exceeded_tolerance(self): tol=1.e-80 self.assertRaises(Exception, bisect, self.identity_fcn, -1.2, 1.,tol) if __name__==‘__main__’: unittest.main() Here, the last test needs some comments: We used the method unittest.TestCase.assertRaises. It tests whether an exception is correctly raised. Its first parameter is the exception type, for example,ValueError, Exception, and its second argument is a the name of the function, which is expected to raise the exception. The remaining arguments are the arguments for this function. The command unittest.main() creates an instance of the class TestIdentity and executes those methods starting by test. Test setUp and tearDown The class unittest.TestCase provides two special methods, setUp and tearDown, which are run before and after every call to a test method. This is needed when testing generators, which are exhausted after every test. We demonstrate this here by testing a program which checks in which line in a file a given string occurs for the first time: class NotFoundError(Exception): pass def find_string(file, string): for i,lines in enumerate(file.readlines()): if string in lines: return i raise NotFoundError(‘String {} not found in File {}‘. format(string,file.name)) We assume, that this code is saved in a file find_string.py. A test has to prepare a file and open it and remove it after the test: import unittest import os # used for, e.g., deleting files from find_in_file import find_string, NotFoundError class TestFindInFile(unittest.TestCase): def setUp(self): file = open(‘test_file.txt’, ‘w’) file.write(‘aha’) file.close() self.file = open(‘test_file.txt’, ‘r’) def tearDown(self): os.remove(self.file.name) def test_exists(self): line_no=find_string(self.file, ‘aha’) self.assertEqual(line_no, 0) def test_not_exists(self): self.assertRaises(NotFoundError, find_string,self.file, ‘bha’) if __name__==‘__main__’: unittest.main() Before each test setUp is run and afterwards tearDown is executed. Parametrizing Tests One frequently wants to repeat the same test set-up with different data sets. When using the functionalities of unittests this requires to automatically generate test cases with the corresponding methods injected: To this end we first construct a test case with one or several methods that will be used, when we later set up test methods. Let us consider the bisection method again and let us check if the values it returns are really zeros of the given function. We first build the test case and the method which will use for the tests: class Tests(unittest.TestCase): def checkifzero(self,fcn_with_zero,interval): result = bisect(fcn_with_zero,*interval,tol=1.e-8) function_value=fcn_with_zero(result) expected=0. self.assertAlmostEqual(function_value, expected) Then we dynamically create test functions as attributes of this class: test_data=[‘name’:’identity’, ‘function’:lambda x: x, ‘interval’:[-1.2, 1.], ‘name’:’parabola’, ‘function’:lambda x: x**2-1, ’interval’:[0, 10.], ‘name’:’cubic’, ‘function’:lambda x: x**3-2*x** 2,‘interval’:[0.1, 5.],] def make_test_function(dic): return lambda self:self.checkifzero(dic[‘function’],dic [‘interval’]) for data in test_data: setattr(Tests, “test_name”.format(name=data[‘name’]), make_test_function(data)) if __name__==‘__main__’: unittest.main() In this example the data is provided as a list of dictionaries. A function make_test_function dynamically generates a test function which uses a particular data dictionary to perform the test with the previously defined method checkifzero. This test function is made a method of the TestCase class by using the Python command settattr. Summary No program development without testing! In this article we showed the importance of well organized and documented tests. Some professionals even start development by first specifying tests. A useful tool for automatic testing is unittest, which we explained in detail. While testing improves the reliability of a code, profiling is needed to improve the performance. Alternative ways to code may result in large performance differences. We showed how to measure computation time and how to localize bottlenecks in your code. Resources for Article: Further resources on this subject: Python Data Analysis Utilities [article] Machine Learning with R [article] Storage Scalability [article]
Read more
  • 0
  • 0
  • 2277

article-image-vector-representation-words
Janu Verma
15 Nov 2016
6 min read
Save for later

Vector Representation of Words

Janu Verma
15 Nov 2016
6 min read
In natural language processing (NLP) tasks, the first step is to represent a document as an element in a vector space. Standard machine learning and data mining algorithms expect a data instance as a vector; in fact, when we say data, we mean a matrix (a row/vector for each data point). There are various ways to express a textual document as a vector, depending on the problem and the assumptions of the model. In traditional Vector Space Models (VSMs), a word is represented as a vector of dimension equal to the size of vocabulary, with each word in the vocabulary corresponding to an entry in the vector. For example, if the text is "Friends work and play together", then our vocabulary has 5 words. We can represent the words as: Friends = [1,0,0,0,0] Work = [0,1,0,0,0] And = [0,0,1,0,0] Play = [0,0,0,1,0] Together = [0,0,0,0,1] Such a representation, called one-hot encoding, is very useful because we can merge these encodings to achieve a vector representation of the entire textual document, which is very central to modern search engines. The above vectors are binary, and there are other encodings possible, such as employing frequency or some other variant. If you are more curious, you can read about TF-IDF. This type of representation has obvious limitations, most importantly, it treats a word as atomic and provides useful information about the relationships that may exist between individual words. We can't perform any meaningful comparision between words other than equality. Furthermore, such a representation results in word vectors that are extremely sparse. Distributed Representations To overcome some of the limitations of the one-hot scheme, a distributed assumption is adapted, which states that words that appear in the same context are semantically closer than the words that do not share the same context. Using this principle, a word can be represented as points in a continous vector space, where semantically similar words correspond to nearby points. This represenation is also called word embeddings, since we are embedding word vectors in the distributed vector space. Essentially, the weight of each word in the vector is distributed across many dimensions. So, instead of a one-to-one mapping between a word and a basic vector (dimension), the word contribution is spread across all of the dimensions of the vector. The dimensions are believed to capture the semantic properties of the words. For example, for our text "Friends work and play together", each word can be represented as something like: Friends = [0.73,0.34,0.52,0.01] Work = [0.65,0.79,0.22,0.1] And = [0.87,0.94,0.14,0.7] Play = [0.73, 0.69, 0.89, 0.4] Together = [0.87,0.79,0.22,0.09] You can see that the words 'Friends' and 'Together' are closer to each other, and the words 'Work' and 'Play' have a higher similarity. Note that these vectors are chosen arbitrarily and do not show an actual representation. The sole purpose here is to give an example. Learning Distributed Representations: Word2Vec Distributed representations of words can be learned by training a model on a corpus of textual data. Mikolov, et. al. proposed an efficient method to learn these embeddings, making it feasible to learn high-quality word vectors on a huge corpus of data. The model is called word2vec, which uses a neural network to learn representations. Two architectures for neural network were proposed – Continuous Bag-of-Words (CBOW) and Skip-Gram. CBOW predicts the current word from a window of neighboring words, while skip-gram uses the current word to predict the surrounding words. Word2Vec models use a sliding window to quantify context. In each sliding window, there is a central word that is under attention, with a few words preceding and following the central word. One of the important parameters of the word2vec models is the length of the sliding window. Consider a textual document: "The goal of the manifold learning techniques is to 'learn' the low dimensional manifold." If we consider 4 words preceding and following the central word, the context of 'manifold' is: ['the', 'goal', 'of', 'the', 'learning', 'techniques', 'is', 'to']. These context words form the input layer of the neural network, and each word is represented as a vector using the one-hot schema. There is one hidden layer and one output layer. The output layer is formed by the central words ,that is, each element in the vocabulary. This way, we learn a representation for each word in terms of the context words. The actual ordering of the context words is irrelevant, which is called a bag-of-words assumption. The skip-gram method is completely opposite of the CBOW method. Here the central word is the input layer, and the context words are now at the output layer. CBOW is faster, but skip-gram does a better job for not-so-frequent words. The implementation of both the architectures can be found at Google Code. Google has also made public pre-trained word vectors that are trained on about 100 billion words from Google News dataset. Several other data, such as Wikipedia, have been used to compute word vectors. And modern neural network packages like Tensorflow have word2vec support. Refer to the word2vec tutorial of Tensorflow. It is important to understand that word2vec is not deep learning; in fact, both the CBOW and skip-gram architectures are shallow neural models with only one hidden layer. Applications Distributed representation of words has been successfully applied in many applications. Machine Translation has been shown to achieve much higher accuracy using distributed representations, so you can make the following assertions: - ```Distance(France, Germany) < Distance(France, Spain)``` - ```Vector('Paris') - Vector('France') + Vector('Italy') ~ Vector(Rome)``` - ```Vector('king') - Vector('man') + Vector('woman') ~ Vector('queen')``` - The odd one in [staple, hammer, saw, drill] is staple. Item2vec: word2vec for collborative filtering and recommendation system, so you can infer: Vector(David Guetta) - Vector(Avicii) + Vector(Beyonce) -> Vector(Rihanna) BioVectors: Word vectors for Bioinformations. BioVectors can characterize biological sequences in terms of biochemical and biophysical interpretations of the underlying patterns References Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. In Proceedings of Workshop at ICLR, 2013. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality. In Proceedings of NIPS, 2013. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic Regularities in Continuous Space Word Representations. In Proceedings of NAACL HLT, 2013 Word2Vec Implementation Tensorflow Example Python Implementation Tomas Mikolov, Quoc V. Le and Ilya Sutskever. Exploiting Similarities among Languages for Machine Translation. We show how the word vectors can be applied to machine translation. Barkan, O; Koenigstein, N (2016).Item2Vec: Neural Item Embedding for Collaborative Filtering Asgari, Ehsaneddin; Mofrad, Mohammad R.K. (2015). Continuous Distributed Representation of Biological Sequences for Deep Proteomics and Genomics. PloS one. 10 (11): e0141287.
Read more
  • 0
  • 0
  • 14953
article-image-css-animation-animatecss
Jabran Rafique
15 Nov 2016
6 min read
Save for later

CSS animation with Animate.css

Jabran Rafique
15 Nov 2016
6 min read
CSS animation is one of the best and easiest ways to make a web page interactive. These animations were made available as part of CSS3 specifications. Animate.css is one of the best libraries available for CSS animations, with a collection of more than 50 types of animations. It was built and maintained by designer Daniel Eden and many other contributors. Animate.css is an open source library and available at GitHub under MIT License. Installation There are a few ways to install the library: Download directly from Github Use directly from a Content Delivery Network (CDN) Install using Bower $ bower install animate.css --save Install using npm $ npm install animate.css --save Usage Now that we have Animate.css installed, using one of the above methods, we can start using it straight away in our app. For simplicity we will use the CDN URL to consume the Animate.css in our tutorial app. Here is a basic structure of our index.html: <!DOCTYPE html> <html> <head> <title>Learning Animate.css</title> <link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/animate.css/3.5.2/animate.min.css"> </head> <body> <div class="container"> <h1>Learning Animate.css</h1> <p>Animate this text!</p> </div> </body> </html> We will use Animate.css effects on paragraph tag in this HTML document. Animate.css has a default class, .animated, that triggers the animation. Other animation effects are used in addition to this default class. For example, let's apply a bounce animation to our paragraph text: ... <p class="animated bounce">Animate this text!</p> ... Here is the demo for this: See the Pen Learning Animate.css – Demo by Jabran Rafique (@jabranr) on CodePen. Similarly, we can replace the bounce class with any other class from Animate.css and it will get the specific animation. The animations will run only once; they will run again only after we reload the browser. To make it run infinitely add the .infinite class so that it becomes: ... <p class="animated infinite bounce">This text will animate infinite times.</p> ... Here is the demo for this: See the Pen Learning Animate.css – Demo (Infinite animation) by Jabran Rafique (@jabranr) on CodePen. Animate.css works best when used dynamically as it sets more control over animations. To use it dynamically, we will make use of JavaScript. Let's try that with the following basic example: First we will add another class to the element, so we can manipulate the element via JavaScript. ... <p class="js-animate">Animate this text dynamically!</p> ... Now we can manipulate this DOM element using JavaScript to add/remove Animate.css animations when clicked on text. var $elementToAnimate = document.querySelector('.js-animate'); $elementToAnimate.addEventListener('click', function(e) { this.classList.toggle('animated'); if (this.classList.contains('bounce')) { this.classList.remove('bounce'); } else { this.classList.add('bounce'); } }, false); Here is the demo for this: See the Pen Learning Animate.css – Demo (Dynamic animations) – part 3 by Jabran Rafique (@jabranr) on CodePen. Similarly, you can also use a DOM manipulation library such as jQuery to apply Animate.css classes. In fact, the official documentation provides a short helper method that registers itself with jQuery as a plugin and can easily be used by any element in the DOM. Here is the code for this: $.fn.extend({ animateCss: function (animationName) { var animationEnd = 'webkitAnimationEnd mozAnimationEnd MSAnimationEnd oanimationend animationend'; this.addClass('animated ' + animationName).one(animationEnd, function() { $(this).removeClass('animated ' + animationName); }); } }); Now we can apply Animate.css animation to any element as follows: $('.js-animate').animateCss('bounce'); Customize Animate.css Using more than 50 different animations in a single project may not be an ideal scenario. Also, if we are looking at the performance side, then adding the complete library in order to use one or few animations may not be useful. Therefore, it is best to customize the library and use only those parts that are needed by the project. Animate.css comes with a built-in Gulp workflow that makes it easier to customize the library to our needs. In order to use the Gulp workflow, the following must be installed already: Node Gulp npm install -g gulp To customize the Animate.css, simply open the animate-config.json file and comment out the animations that are NOT required by the project. Then run the gulp command in terminal to generate a customized CSS file. The terminal window will inform on how many animations were activated into generated customized CSS file. Customizing the Animate.css not only boosts performance, but also keeps the code clean by including only parts that are actually being used in the project. Sass version Sass is a CSS preprocessor language that lets us use variables and methods to reuse code inside our project. Sass files are ultimately converted into CSS files before they are used by any web platform. A Sass-only version is under development. The in-development source code can be found under sass branch of the project at GitHub. Once released, this will provide more robust control for customized and personalized animations by relying less on JavaScript, that is Gulp workflow. Contribute Contributions to any open source project make it robust and more useful with bugfixes and new features. Just like any other open source project, Animate.css also welcomes contributions. To contribute to Animate.css project just head over to GitHub project and fork the repository to work on an existing issue, or by adding a new one. All of the demos in this tutorial can be found at CodePen as Collection. Hope this quick get started guide will make it easier to use the Animate.css library in your next project to create awesome animated effects. About the author Jabran Rafique is a London-based web engineer. He currently works as a Front-End Web Developer at Rated People. He has a Masters in Computer Science from Staffordshire University and more than 6 years of professional experience in web systems. He had also served as Regional Lead and Advocate at Google Map Maker in 2008, where he contributed to building digital maps in order to make them available to millions of people worldwide. He has organized and spoken at several international events as well. He writes on his website/blog about different things, and shares code at GitHub and thoughts on Twitter.
Read more
  • 0
  • 0
  • 11971

article-image-face-detection-and-tracking-using-ros-open-cv-and-dynamixel-servos
Packt
14 Nov 2016
13 min read
Save for later

Face Detection and Tracking Using ROS, Open-CV and Dynamixel Servos

Packt
14 Nov 2016
13 min read
In this article by Lentin Joseph, the author of the book ROS Robotic Projects, we learn how one of the capability in most of the service and social robots is face detection and tracking. The robot can identify faces and it can move its head according to the human face move around it. There are numerous implementation of face detection and tracking system in web. Most of the trackers are having a pan and tilt mechanism and a camera is mounted on the top of the servos. In this article, we are going to see a simple tracker which is having only pan mechanism. We are going to use a USB webcam which is mounted on AX-12 Dynamixel servo. (For more resources related to this topic, see here.) You can see following topics on this article: Overview of the project Hardware and software prerequisites Overview of the project The aim of the project is to build a simple face tracker which can track face only in the horizontal axis of camera. The tracker is having a webcam, Dynamixel servo called AX-12 and a supporting bracket to mount camera on the servo. The servo tracker will follow the face until it align to the center of the image which is getting from webcam. Once it reaches the center, it will stop and wait for the face movement. The face detection is done using OpenCV and ROS interface, and controlling the servo is done using Dynamixel motor driver in ROS. We are going to create two ROS packages for this complete tracking system, one is for face detection and finding centroid of face and next is for sending commands to servo to track the face using the centroid values. Ok!! Let's start discussing the hardware and software prerequisites of this project. Hardware and software prerequisites Following table of hardware components which can be used for building this project. You can also see a rough price of each component and purchase link of the same. List of hardware components: No Component name Estimated price (USD) Purchase link 1 Webcam 32 https://amzn.com/B003LVZO8S 2 Dynamixel AX -12 A servo with mounting bracket 76 https://amzn.com/B0051OXJXU 3 USB To Dynamixel Adapter 50 http://www.robotshop.com/en/robotis-usb-to-dynamixel-adapter.html 4 Extra 3 pin cables for AX-12 servos 12 http://www.trossenrobotics.com/p/100mm-3-Pin-DYNAMIXEL-Compatible-Cable-10-Pack 5 Power adapter 5 https://amzn.com/B005JRGOCM 6 6 Port AX/MX Power Hub 5 http://www.trossenrobotics.com/6-port-ax-mx-power-hub 7 USB extension cable 1 https://amzn.com/B00YBKA5Z0   Total Cost + Shipping + Tax ~ 190 - 200   The URLs and price can vary. If the links are not available, you can do a google search may do the job. The shipping charges and tax are excluded from the price. If you are thinking that, the total cost is not affordable for you, then there are cheap alternatives to do this project too. The main heart of this project is Dynamixel servo. We may can replace this servo with RC servos which only cost around $10 and using an Arduino board cost around $20 can be used to control the servo too, so you may can think about porting the face tracker project work using Arduino and RC servo Ok, let's look on to the software prerequisites of the project. The prerequisites include ROS framework, OS version and ROS packages: No Name of software Estimated price (USD) Download link 1 Ubuntu 16.04 L.T.S Free http://releases.ubuntu.com/16.04/ 2 ROS Kinetic L.T.S Free http://wiki.ros.org/kinetic/Installation/Ubuntu 3 ROS usb_cam package Free http://wiki.ros.org/usb_cam 3 ROS cv_bridge package Free http://wiki.ros.org/cv_bridge 4 ROS Dynamixel controller Free https://github.com/arebgun/dynamixel_motor 5 Windows 7 or higher ~ $120 https://www.microsoft.com/en-in/software-download/windows7 7 RoboPlus (Windows application) Free http://www.robotis.com/download/software/RoboPlusWeb%28v1.1.3.0%29.exe The above table will gives you an idea about which all are the software we are going to be used for this project. We may need both Windows and Ubuntu for doing this project. It will be great if you have dual operating system on your computer Let's see how to install these software first Installing dependent ROS packages We have already installed and configured Ubuntu 16.04 and ROS Kinetic on it. Here are the dependent packages we need to install for this project. Installing usb_cam ROS package Let's see the use of usb_cam package in ROS first. The usb_cam package is ROS driver for Video4Linux (V4L) USB camera. The V4L is a collection of devices drivers in Linux for real time video capture from webcams. The usb_cam ROS package work using the V4L devices and publish the video stream from devices as ROS image messages. We can subscribe it and do our own processing using it. The official ROS page of this package is given in the above table. You may can check this page for different settings and configuration this package offers. Creating ROS workspace for dependencies Before starting installing usb_cam package, let's create a ROS workspace for keeping the dependencies of the entire projects mentioned in the book. We may can create another workspace for keeping the project code. Create a ROS workspace called ros_project_dependencies_ws in home folder. Clone the usb_cam package into the src folder: $ git clone https://github.com/bosch-ros-pkg/usb_cam.git Build the workspace using catkin_make After building the package, install v4l-util Ubuntu package. It is a collection of command line V4L utilities which is using by usb_cam package: $ sudo apt-get install v4l-utils Configuring webcam on Ubuntu 16.04 After installing these two, we can connect the webcam to PC to check it properly detected in our PC. Take a terminal and execute dmesg command to check the kernel logs. If your camera is detected in Linux, it may give logs like this{ $ dmesg Kernels logs of webcam device You can use any webcam which has driver support in Linux. In this project, iBall Face2Face (http://www.iball.co.in/Product/Face2Face-C8-0--Rev-3-0-/90) webcam is used for tracking. You can also go for a popular webcam which is mentioned as a hardware prerequisite. You can opt that for better performance and tracking. If our webcam has support in Ubuntu, we may can open the video device using a tool called cheese. Cheese is simply a webcam viewer. Enter the command cheese in the terminal, if it is not available you can install it using following command: $ sudo apt-get install cheese If the driver and device are proper, you may get the video stream from webcam like this: Webcam video streaming using cheese Congratulation!!, your webcam is working well in Ubuntu, but are we done with everything? No. The next thing is to test the ROS usb_cam package. We have to make sure that is working well in ROS!! Interfacing Webcam to ROS Let's test the webcam using usb_cam package. The following command is used to launch the usb_cam nodes to display images from webcam and publishing ROS image topics at the same time: $ roslaunch usb_cam usb_cam-test.launch If everything works fine, you will get the image stream and logs in the terminal as shown below: Working of usb_cam package in ROS The image is displayed using image_view package in ROS, which is subscribing the topic called /usb_cam/image_raw Here are the topics, that usb_cam node is publishing: Figure 4: The topics publishing by usb_cam node We have just done with interfacing a webcam in ROS. So what's next? We have to interface AX-12 Dynamixel servo to ROS. Before proceeding to interfacing, we have to do something to configure this servo. Next we are going to see how to configure a Dynamixel servo AX-12A. Configuring a Dynamixel servo using RoboPlus The configuring of Dynamixel servo can be done using a software called RoboPlus providing by ROBOTIS INC (http://en.robotis.com/index/), the manufacturer of Dynamixel servos. For configuring Dynamixel, you have to switch your operating system to Windows. The tool RoboPlus will work on Windows. In this project, we are going to configure the servo in Windows 7. Here is the link to download RoboPlus: http://www.robotis.com/download/software/RoboPlusWeb%28v1.1.3.0%29.exe. If the link is not working, you can just search in google to get the RoboPlus 1.1.3 version. After installing the software, you will get the following window, navigate to Expert tab in the software for getting the application for configuring Dynamixel: Dynamixel Manager in RoboPlus Before taking the Dynamixel Wizard and do configuring, we have to connect the Dynamixel and properly powered. Following image of AX-12A servo that we are using for this project and its pin connection. AX-12A Dynamixel and its connection diagram Unlike other RC servos, AX-12 is an intelligent actuator which is having a microcontroller which can monitoring every parameters of servo and customize all the servo parameters. It is having a geared drive and the output of the servo is connected to servo horn. We may can connect any links on this servo horn. There are two connection ports behind each servo. Each port is having pins such as VCC, GND and Data. The ports of Dynamixel are daisy chained so that we can connect another servo from one servo. Here is the connection diagram of Dynamixel with PC. AX-12A Dynamixel and its connection diagram The main hardware component which interfacing Dynamixel to PC is called USB to Dynamixel. This is a USB to serial adapter which can convert USB to RS232, RS 484 and TTL. In AX-12 motors, the data communication is using TTL. From the Figure AX 12A Dynamixel and its connection diagram, we can seen that there are three pins in each port. The data pin is used to send and receive from AX-12 and power pins are used to power the servo. The input voltage range of AX-12A Dynamixel is from 9V to 12V. The second port in each Dynamixel can be used for daisy chaining. We can connect up to 254 servos using this chaining Official links of AX-12A servo and USB to Dynamixel AX-12A: http://www.trossenrobotics.com/dynamixel-ax-12-robot-actuator.aspx USB to Dynamixel: http://www.trossenrobotics.com/robotis-bioloid-usb2dynamixel.aspx For working with Dynamixel, we should know some more things. Let's have a look on some of the important specification of AX-12A servo. The specifications are taken from the servo manual. Figure 8: AX-12A Specification The Dynamixel servos can communicate to PC to a maximum speed of 1 Mbps. It can also give feedback of various parameters such as its position, temperature and current load. Unlike RC servos, this can rotate up to 300 degrees and communication is mainly using digital packets. Powering and connecting Dynamixel to PC Now we are going to connect Dynamixel to PC. Given below a standard way of connecting Dynamixel to PC: Connecting Dynamixel to PC The three pin cable can be first connected to any of the port of AX-12 and other side have to connect to the way to connect 6 port power hub. From the 6-port power hub, connect another cable to the USB to Dynamixel. We have to select the switch of USB to Dynamixel to TTL mode. The power can be either be connected through a 12V adapter or through battery. The 12V adapter is having 2.1X5.5 female barrel jack, so you should check the specification of male adapter plug while purchasing. Setting USB to Dynamixel driver on PC As we have already discussed the USB to Dynamixel adapter is a USB to serial convertor, which is having an FTDI chip (http://www.ftdichip.com/) on it. We have to install a proper FTDI driver on the PC for detecting the device. The driver may need for Windows but not for Linux, because FTDI drivers are built in the Linux kernel. If you install the RoboPlus software, the driver may be already installed along with it. If it is not, you can manually install from the RoboPlus installation folder. Plug the USB to Dynamixel to the Windows PC, and check the device manager. (Right click on My Computer | Properties | Device Manager). If the device is properly detected, you can see like following figure: Figure 10: COM Port of USB to Dynamixel If you are getting a COM port for USB to Dynamixel, then you can start the Dynamixel Manager from RoboPlus. You can connect to the serial port number from the list and click the Search button to scan for Dynamixel as shown in following figure. Select the COM port from the list and connecting to the port is marked as 1. After connecting to the COM port, select the default baud rate as 1 Mbps and click the Start searching button: COM Port of USB to Dynamixel If you are getting a list of servo in the left side panel, it means that your PC have detected a Dynamixel servo. If the servo is not detecting, you can do following steps to debug: Make sure that supply is proper and connections are proper using a multi meter. Make sure that servo LED on the back is blinking when power on. If it is not coming, it can be a problem with servo or power supply. Upgrade the firmware of servo using Dynamixel Manager from the option marked as 6. The wizard is shown in the following figure. During wizard, you may need power off the supply and ON it again for detecting the servo. After detecting the servo, you have to select the servo model and install the new firmware. This may help to detect the servo in the Dynamixel manager if the existing servo firmware is out dated. Dynamixel recovery wizard If the servos are listing on the Dynamixel manager, click on a servo and you can see its complete configuration. We have to modify some values inside the configurations for our current face tracker project. Here are the parameters: ID : Set the ID as 1 Baud rate: 1 Moving Speed: 100 Goal Position: 512 The modified servo settings are shown in the following figure: Modified Dynamixel firmware settings After doing these settings, you can check the servo is working good or not by changing its Goal position. Yes!! Now you are done with Dynamixel configuration, Congratulation!! What's next? We want to interface Dynamixel to ROS. Summary This article was about building a face tracker using webcam and Dynamixel motor. The software we have used was ROS and OpenCV. Initially you can see how to configure the webcam and Dynamixel motor and after configuring, we were trying to build two package for tracking. One package does the face detection and second package is a controller which can send position command to Dynamixel to track the face. We have discussed the use of all files inside the packages and did a final run to show the complete working of the system. Resources for Article: Further resources on this subject: Using ROS with UAVs [article] Hardware Overview [article] Arduino Development [article]
Read more
  • 0
  • 0
  • 21351

article-image-fundamental-selinux-concepts
Packt
14 Nov 2016
41 min read
Save for later

Fundamental SELinux Concepts

Packt
14 Nov 2016
41 min read
In this article by Sven Vermeulen, the author of the book SELinux System Administration Second Edition, we will see how Security Enhanced Linux (SELinux) brings additional security measures for your Linux system to further protect the resources on the system. This article explains why SELinux has opted to use labels to identify resources, the way SELinux differentiates itself from regular Linux access controls through the enforcement of security rules, how the access control rules enforced by SELinux are provided through policy files, the different SELinux implementations between Linux distributions. (For more resources related to this topic, see here.) Providing more security to Linux Seasoned Linux administrators and security engineers already know that they need to put some trust in the users and processes on their system in order for the system to remain secure. This is partially because users can attempt to exploit vulnerabilities found in the software running on the system, but a large contribution to this trust level is because the secure state of the system depends on the behavior of the users. A Linux user with access to sensitive information could easily leak that out to the public, manipulate the behavior of the applications he or she launches, and do many other things that affect the security of the system. The default access controls that are active on a regular Linux system are discretionary; it is up to the user's how the access controls should behave. The Linux discretionary access control (DAC) mechanism is based on the user and/or group information of the process and is matched against the user and/or group information of the file, directory, or other resource being manipulated. Consider the /etc/shadow file, which contains the password and account information of the local Linux accounts: $ ls -l /etc/shadow -rw------- 1 root root 1010 Apr 25 22:05 /etc/shadow Without additional access control mechanisms in place, this file is readable and writable by any process that is owned by the root user, regardless of the purpose of the process on the system. The shadow file is a typical example of a sensitive file that we don't want to see leaked or abused in any other fashion. Yet, the moment someone has access to the file, they can copy it elsewhere, for example to a home directory, or even mail it to a different computer and attempt to attack the password hashes stored within. Another example of how Linux DAC requires trust from its users is when a database is hosted on the system. Database files themselves are (hopefully) only accessible to runtime users of the database management system (DBMS) and the Linux root user. Properly secured systems will only grant trusted users access to these files (for instance, through sudo) by allowing them to change their effective user ID from their personal user to database runtime user or even root account for a well-defined set of commands. These users too can analyze the database files and gain access to potentially confidential information in the database without going through the DBMS. However, regular users are not the only reason for securing a system. Lots of software daemons run as the Linux root user or have significant privileges on the system. Errors within those daemons can easily lead to information leakage or might even lead to exploitable remote command execution vulnerabilities. Backup software, monitoring software, change management software, scheduling software, and so on: they all often run with the highest privileged account possible on a regular Linux system. Even when the administrator does not allow privileged users, their interaction with daemons induces a potential security risk. As such, the users are still trusted to correctly interact with these applications in order for the system to function properly. Through this, the administrator leaves the security of the system to the discretion of its (many) users. Enter SELinux, which provides an additional access control layer on top of the standard Linux DAC mechanism. SELinux provides a mandatory access control (MAC) system that, unlike its DAC counterpart, gives the administrator full control over what is allowed on the system and what isn't. It accomplishes this by supporting a policy-driven approach over what processes are and aren't allowed to do and by enforcing this policy through the Linux kernel. Mandatory means that access control is enforced by the operating system and defined solely by the administrator. Users and processes do not have permission to change the security rules, so they cannot work around the access controls; security is not left to their discretion anymore. The word mandatory here, just like the word discretionary before, was not chosen by accident to describe the abilities of the access control system: both are known terms in the security research field and have been described in many other publications, including the Trusted Computer System Evaluation Criteria (TCSEC) (http://csrc.nist.gov/publications/history/dod85.pdf) standard (also known as the Orange Book) by the Department of Defense in the United States of America in 1985. This publication has led to the common criteria standard for computer security certification (ISO/IEC 15408), available at http://www.commoncriteriaportal.org/cc/. Using Linux security modules Consider the example of the shadow file again. A MAC system can be configured to only allow a limited number of processes to read and write to the file. A user logged on as root cannot directly access the file or even move it around. He can't even change the attributes of the file: # id uid=0(root) gid=0(root) # cat /etc/shadow cat: /etc/shadow: Permission denied # chmod a+r /etc/shadow chmod: changing permissions of '/etc/shadow': Permission denied This is enforced through rules that describe when the contents of a file can be read. With SELinux, these rules are defined in the SELinux policy and are loaded when the system boots. It is the Linux kernel itself that is responsible for enforcing the rules. Mandatory access control systems such as SELinux can be easily integrated into the Linux kernel through its support for Linux Security Modules (LSM): High-level overview of how LSM is integrated into the Linux kernel LSM has been available in the Linux kernel since version 2.6, sometime in December 2003. It is a framework that provides hooks inside the Linux kernel to various locations, including the system call entry points, and allows a security implementation such as SELinux to provide functions to be called when a hook is triggered. These functions can then do their magic (for instance, checking the policy and other information) and give a go/no-go back to allow the call to go through or not. LSM by itself does not provide any security functionality; instead, it relies on security implementations that do the heavy lifting. SELinux is one of the implementations that use LSM, but there are several others: AppArmor, Smack, TOMOYO Linux, and Yama, to name a few. At the time of writing this, only one main security implementation can be active through the LSM hooks. Work is underway to enable stacking multiple security implementations, allowing system administrators to have more than one implementation active. Recent work has already allowed multiple implementations to be defined (but not simultaneously active). When supported, this will allow administrators to pick the best features of a number of implementations and enforce smaller LSM-implemented security controls on top of the more complete security model implementations, such as SELinux, TOMOYO, Smack, or AppArmor. Extending regular DAC with SELinux SELinux does not change the Linux DAC implementation nor can it override denials made by the Linux DAC permissions. If a regular system (without SELinux) prevents a particular access, there is nothing SELinux can do to override this decision. This is because the LSM hooks are triggered after the regular DAC permission checks have been executed. For instance, if you need to allow an additional user access to a file, you cannot add an SELinux policy to do that for you. Instead, you will need to look into other features of Linux, such as the use of POSIX access control lists. Through the setfacl and getfacl commands (provided by the acl package) the user can set additional permissions on files and directories, opening up the selected resource to additional users or groups. As an example, let's grant user lisa read-write access to a file using setfacl: $ setfacl -m u:lisa:rw /path/to/file Similarly, to view the current POSIX ACLs applied to the file, use this command: $ getfacl /path/to/file # file: file # owner: swift # group: swift user::rw- user:lisa:rw- group::r-- mask::r-- other::r-- Restricting root privileges The regular Linux DAC allows for an all-powerful user: root. Unlike most other users on the system, the logged-on root user has all the rights needed to fully manage the entire system, ranging from overriding access controls to controlling audits, changing user IDs, managing the network, and much more. This is supported through a security concept called capabilities (for an overview of Linux capabilities, check out the capabilities manual page: man capabilities). SELinux is also able to restrict access to these capabilities in a fine-grained manner. Due to this fine-grained authorization aspect of SELinux, even the root user can be confined without impacting the operations on the system. The aforementioned example of accessing /etc/shadow is just one example of a restriction that a powerful user as root still might not be able to make due to the SELinux access controls being in place. When SELinux was added to the mainstream Linux kernel, some security projects even went as far as providing public root shell access to an SELinux-protected system, asking hackers and other security researchers to compromise the box. The ability to restrict root was welcomed by system administrators who sometimes need to pass on the root password or root shell to other users (for example, database administrators) who needed root privileges when their software went haywire. Thanks to SELinux, the administrator can now pass on a root shell while resting assured that the user only has those rights he needs, and not full system-administration rights. Reducing the impact of vulnerabilities If there is one benefit of SELinux that needs to be stressed, while often also being misunderstood, it is its ability to reduce the impact of vulnerabilities. A properly written SELinux policy confines applications so that their allowed activities are reduced to a minimum set. This least-privilege model ensures that abnormal application behavior is not only detected and audited but also prevented. Many application vulnerabilities can be exploited to execute tasks that an application is not meant to do. When this happens, SELinux will prevent this. However, there are two misconceptions about SELinux state and its ability to thwart exploits, namely, the impact of the policy and the exploitation itself. If the policy is not written in a least-privilege model, then SELinux might consider this nonstandard behavior as normal and allow the actions to continue. For policy writers, this means that their policy code has to be very fine-grained. Sadly, that makes writing policies very time-consuming; there are more than 80 classes and over 200 permissions known to SELinux, and policy rules need to take into account all these classes and permissions for each interaction between two objects or resources. As a result, policies tend to become convoluted and harder to maintain. Some policy writers make the policies more permissive than is absolutely necessary, which might result in exploits becoming successful even though the action is not expected behavior from an application point of view. Some application policies are explicitly marked as unconfined (which is discussed later in this article), showing that they are very liberal in their allowed permissions. Red Hat Enterprise Linux even has several application policies as completely permissive, and it starts enforcing access controls for those applications only after a few releases. The second misconception is the exploit itself. If an application's vulnerability allows an unauthenticated user to use the application services as if he were authorized, SELinux will not play a role in reducing the impact of the vulnerability; it only notices the behavior of the application itself and not of the sessions internal to the application. As long as the application itself behaves as expected (accessing its own files and not poking around in other filesystems), SELinux will happily allow the actions to take place. It is only when the application starts behaving erratically that SELinux stops the exploit from continuing. Exploits such as remote command execution (RCE) against applications that should not be executing random commands (such as database management systems or web servers, excluding CGI-like functionality) will be prevented, whereas session hijacking or SQL injection attacks are not controllable through SELinux policies. Enabling SELinux support Enabling SELinux on a Linux system is not just a matter of enabling the SELinux LSM module within the Linux kernel. An SELinux implementation comprises the following: The SELinux kernel subsystem, implemented in the Linux kernel through LSM Libraries, used by applications that need to interact with SELinux Utilities, used by administrators to interact with SELinux Policies, which define the access controls themselves The libraries and utilities are bundled by the SELinux user space project (https://github.com/SELinuxProject/selinux/wiki). Next to the user space applications and libraries, various components on a Linux system are updated with SELinux-specific code, including the init system and several core utilities. Because SELinux isn't just a switch that needs to be toggled, Linux distributions that support SELinux usually come with SELinux predefined and loaded: Fedora and Red Hat Enterprise Linux (with its derivatives, such as CentOS and Oracle Linux) are well-known examples. Other supporting distributions might not automatically have SELinux enabled but can easily support it through the installation of additional packages (which is the case with Debian and Ubuntu), and others have a well-documented approach on how to convert a system to SELinux (for example, Gentoo and Arch Linux). Labeling all resources and objects When SELinux has to decide whether it has to allow or deny a particular action, it makes a decision based on the context of both the subject (which is initiating the action) and the object (which is the target of the action). These contexts (or parts of the context) are mentioned in the policy rules that SELinux enforces. The context of a process is what identifies the process to SELinux. SELinux has no notion of Linux process ownership and, frankly, does not care how the process is called, which process ID it has, and what account the process runs as. All it wants to know is what the context of that process is, which is represented to users and administrators as a label. Label and context are often used interchangeably, and although there is a  technical distinction (one is a representation of the other), we will not dwell on that much. Let's look at an example label: the context of the current user (try it out yourself if you are on an SELinux-enabled system): $ id -Z unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 The id command, which returns information about the current user, is executed here with the -z switch (a commonly agreed-upon switch for displaying SELinux information). It shows us the context of the current user (actually the context of the id process itself when it was executing). As we can see, the context has a string representation and looks as if it has five fields (it doesn't; it has four fields—the last field just happens to contain a :). SELinux developers decided to use labels instead of real process and file (or other resource) metadata for its access controls. This is different to MAC systems such as AppArmor, which use the path of the binary (and thus the process name) and the paths of the resources to handle permission checks. The decision to make SELinux a label-based mandatory access control was taken for various reasons, which are as follows: Using paths might be easier to comprehend for administrators, but this doesn't allow us to keep the context information close to the resource. If a file or directory is moved or remounted or a process has a different namespace view on the files, then the access controls might behave differently. With label-based contexts, this information is retained and the system keeps controlling the resource properly. Contexts reveal the purpose of the process very well. The same binary application can be launched in different contexts depending on how it got started. The context value (such as the one shown in the id -Z output earlier) is exactly what the administrator needs. With it, he knows what the rights are of each of the running instances, but he can also deduce from it how the process might have been launched and what its purpose is. Contexts also make abstractions of the object itself. We are used to talking about processes and files, but contexts are also applicable to less tangible resources such as pipes (interprocess communication) or database objects. Path-based identification only works as long as you can write a path. As an example, consider the following policies: Allow the httpd processes to bind to TCP port 80 Allow the processes labeled with httpd_t to bind to TCP ports labeled with http_port_t In the first example, we cannot easily reuse this policy when the web server process isn't using the httpd binary (perhaps because it was renamed or it isn't Apache but another web server) or when we want to have HTTP access on a different port. With the labeled approach, the binary can be called apache2 or MyWebServer.py; as long as the process is labeled httpd_t, the policy applies. The same happens with the port definition: you can label port 8080 with http_port_t and thus allow the web servers to bind to that port as well. Dissecting the SELinux context To come to a context, SELinux uses at least three, and sometimes four, values. Let's look at the context of an Apache web server as an example: $ ps -eZ | grep httpd system_u:system_r:httpd_t:s0 511 ? 00:00:00 httpd As we can see, the process is assigned a context that contains following fields: system_u: This represents the SELinux user system_r: This represents the SELinux role httpd_t: This represents the SELinux type (also known as the domain in the case of a process) s0: This represents the sensitivity level This structure can be depicted as follows: The structure of a SELinux context, using the id -Z output as an example When we work with SELinux, contexts are all we need. In the majority of cases, it is the third field (called the domain or type) that is most important since the majority of SELinux policy rules (over 99 percent) consist of rules related to the interaction between two types (without mentioning roles, users, or sensitivity levels). SELinux contexts are aligned with LSM security attributes and exposed to the user space, allowing end users and applications to easily query the contexts. An interesting place where these attributes are presented is within the /proc pseudo filesystem. Inside each process's /proc/<pid> location, we find a subdirectory called attr, inside of which the following files can be found: $ ls /proc/$$/attr current fscreate prev exec keycreate sockcreate All these files, if read, display either nothing or an SELinux context. If it is empty, then that means the application has not explicitly set a context for that particular purpose, and the SELinux context will be deduced either from the policy or inherited from its parent. The meaning of the files are as follows: The current file displays the current SELinux context of the process. The exec file displays the SELinux context that will be assigned by the next application execution done through this application. It is usually empty. The fscreate file displays the SELinux context that will be assigned to the next file that is written by the application. It is usually empty. The keycreate file displays the SELinux context that will be assigned to the keys cached in the kernel by this application. It is usually empty. The prev file displays the previous SELinux context for this particular process. This is usually the context of its parent application. The sockcreate file displays the SELinux context that will be assigned to the next socket created by the application. It is usually empty. If an application has multiple subtasks, then the same information is available in each subtask directory at /proc/<pid>/task/<taskid>/attr. Enforcing access through types The SELinux type (the third part of an SELinux context) of a process (called the domain) is the basis of the fine-grained access controls of that process with respect to itself and other types (which can be processes, files, sockets, network interfaces, and more). In most SELinux literature, the SELinux label-based access control mechanism is fine-tuned to say that SELinux is a type enforcement mandatory access control system: when some actions are denied, the fine-grained access controls on the type level are most likely to blame. With type enforcement, SELinux is able to control what an application is allowed to do based on how it got executed in the first place: a web server that is launched interactively by a user will run with a different type than a web server executed through the init system, even though the process binary and path are the same. The web server launched from the init system is most likely trusted (and thus allowed to do whatever web servers are supposed to do), whereas a manually launched web server is less likely to be considered normal behavior and as such will have different privileges. The majority of SELinux resources will focus on types. Even though the SELinux type is just the third part of an SELinux context, it is the most important one for most administrators. Most documentation will even just talk about a type such as httpd_t rather than a full SELinux context. Take a look at the following dbus-daemon processes: # ps -eZ | grep dbus-daemon system_u:system_r:system_dbusd_t 4531 ? 00:00:00 dbus-daemon staff_u:staff_r:staff_dbusd_t 5266 ? 00:00:00 dbus-daemon In this example, one dbus-daemon process is the system D-Bus daemon running with the aptly named system_dbusd_t type, whereas another one is running with the staff_dbusd_t type assigned to it. Even though their binaries are completely the same, they both serve a different purpose on the system and as such have a different type assigned. SELinux then uses this type to govern the actions allowed by the process towards other types, including how system_dbusd_t can interact with staff_dbusd_t. SELinux types are by convention suffixed with _t, although this is not mandatory. Granting domain access through roles SELinux roles (the second part of an SELinux context) allow SELinux to support role-based access controls. Although type enforcement is the most used (and known) part of SELinux, role-based access control is an important method to keep a system secure, especially from malicious user attempts. SELinux roles are used to define which process types (domains) user processes can be in. As such, they help define what a user can and cannot do. By convention, SELinux roles are defined with an _r suffix. On most SELinux-enabled systems, the following roles are made available to be assigned to users: user_r This role is meant for restricted users: the user_r SELinux role is only allowed to have processes with types specific to end-user applications. Privileged types, including those used to switch to another Linux user, are not allowed for this role. staff_r This role is meant for non-critical operations: the SELinux staff_r role is generally restricted to the same applications as the restricted user, but it has the ability to switch roles. It is the default role for operators to be in (so as to keep those users in the least privileged role as long as possible). sysadm_r This role is meant for system administrators: the sysadm_r SELinux role is very privileged, enabling various system-administration tasks. However, certain end-user application types might not be supported (especially if those types are used for potentially vulnerable or untrusted software) to keep the system free from infections. system_r This role is meant for daemons and background processes: the system_r SELinux role is quite privileged, supporting the various daemon and system process types. However, end-user application types and other administrative types are not allowed in this role. unconfined_r This role is meant for end users: the unconfined_r role is allowed a limited number of types, but those types are very privileged as it is meant for running any application launched by a user in a more or less unconfined manner (not restricted by SELinux rules). This role as such is only available if the system administrator wants to protect certain processes (mostly daemons) while keeping the rest of the system operations almost untouched by SELinux. Other roles might be supported as well, such as guest_r and xguest_r, depending on the distribution. It is wise to consult the distribution documentation for more information about the supported roles. An overview of available roles can be obtained through the seinfo command (part of setools-console in RHEL or app-admin/setools in Gentoo): # seinfo --role Roles: 14 auditadm_r dbadm_r ... unconfined_r Limiting roles through users An SELinux user (the first part of an SELinux context) is different from a Linux user. Unlike Linux user information, which can change while the user is working on the system (through tools such as sudo or su), the SELinux policy can (and generally will) enforce that the SELinux user remain the same even when the Linux user itself has changed. Because of the immutable state of the SELinux user, specific access controls can be implemented to ensure that users cannot work around the set of permissions granted to them, even when they get privileged access. An example of such an access control is the user-based access control (UBAC) feature that some Linux distributions (optionally) enable, which prevents users from accessing files of different SELinux users even when those users try to use the Linux DAC controls to open up access to each other's files. The most important feature of SELinux users, however, is that SELinux user definitions restrict which roles the (Linux) user is allowed to be in. A Linux user is first assigned to an SELinux user—multiple Linux users can be assigned to the same SELinux user. Once set, that user cannot switch to an SELinux role he isn't meant to be in. This is the role-based access control implementation of SELinux: Mapping Linux accounts to SELinux users SELinux users are, by convention, defined with a _u suffix, although this is not mandatory. The SELinux users that most distributions have available are named after the role they represent, but instead of ending with _r, they end with _u. For instance, for the sysadm_r role, there is a sysadm_u SELinux user. Controlling information flow through sensitivities The fourth part of an SELinux context, the sensitivity, is not always present (some Linux distributions by default do not enable sensitivity labels). If they are present though, then this part of the label is needed for the multi-level security (MLS) support within SELinux. Sensitivity labels allow classification of resources and restriction of access to those resources based on a security clearance. These labels consist of two parts: a confidentiality value (prefixed with s) and a category value (prefixed with c). In many larger organizations and companies, documents are labeled internal, confidential, or strictly confidential. SELinux can assign processes a certain clearance level towards these resources. With MLS, SELinux can be configured to follow the Bell-LaPadula model, a security model that can be characterized by no read up and no write down: based on a process clearance level, that process cannot read anything with a higher confidentiality level nor write to (or communicate otherwise with) any resource with a lower confidentiality level. SELinux does not use the internal, confidential, and other labels. Instead, it uses numbers from 0 (lowest confidentiality) to whatever the system administrator has defined as the highest value (this is configurable and set when the SELinux policy is built). Categories allow resources to be tagged with one or more categories, on which access controls are also possible. The idea behind categories is to support multitenancy (for example, systems hosting applications for multiple customers) within a Linux system, by having processes and resources belonging to one tenant to be assigned a particular set of categories, whereas the processes and resources of another tenant get a different set of categories. When a process does not have proper categories assigned, it cannot do anything with the resources (or other processes) that have other categories assigned. An unwritten convention in the SELinux world is that (at least) two categories are used to differentiate between tenants. By having services randomly pick two categories for a tenant out of a predefined set of categories, while ensuring each tenant has a unique combination, these services receive proper isolation. The use of two categories is not mandatory but is implemented by services such as sVirt and Docker. In that sense, categories can be seen as tags, allowing access to be granted only when the tags of the process and the target resource match. As multilevel security is not often used, the benefits of only using categories is persisted in what is called multi-category security (MCS). This is a special MLS case, where only a single confidentiality level is supported (s0). Defining and distributing policies Enabling SELinux does not automatically start the enforcement of access. If SELinux is enabled and it cannot find a policy, it will refuse to start. That is because the policy defines the behavior of the system (what SELinux should allow). SELinux policies are generally distributed in a compiled form (just like with software) as policy modules. These modules are then aggregated into a single policy store and loaded in memory to allow SELinux to enforce the policy rules on the system. Gentoo, being a source-based meta-distribution, distributes the SELinux policies as (source) code as well, which is compiled and built at install time, just like it does with other software. The following diagram shows the relationship between policy rules, policy modules, and a policy package (which is often a one-to-one mapping towards a policy store): Relationship between policy rules, policy modules and policy store Writing SELinux policies A SELinux policy writer can write down the policy rules in (currently) three possible languages: In standard SELinux source format—a human-readable and well-established language for writing SELinux policies In reference policy style—this extends the standard SELinux source format with M4 macros to facilitate the development of policies. In the SELinux Common Intermediate Language (CIL)—a computer-readable (and, with some effort, human-readable) format for SELinux policies. Most SELinux supporting distributions base their policy on the reference policy (https://github.com/TresysTechnology/refpolicy/wiki), a fully functional SELinux policy set managed as a free software project. This allows distributions to ship with a functional policy set rather than having to write one themselves. Many project contributors are distribution developers, trying to push changes of their distribution to the reference policy project itself, where the changes are peer-reviewed to make sure no rules are brought into the project that might jeopardize the security of any platform. It easily becomes very troublesome to write reusable policy modules without the extensive set of M4 macros offered by the reference policy project. The SELinux CIL format is quite recent (RHEL 7.2 does not support it yet), and although it is very much in use already (the recent SELinux user space converts everything in CIL in the background), it is not that common yet for policy writers to use it directly. As an example, consider the web server rule we discussed earlier, repeated here for your convenience: Allow the processes labeled with httpd_t to bind to TCP ports labeled with http_port_t. In the standard SELinux source format, this is written down as follows: allow httpd_t http_port_t : tcp_socket { name_bind }; Using reference policy style, this rule is part of the following macro call: corenet_tcp_bind_http_port(httpd_t) In CIL language, the rule would be expressed as follows: (allow httpd_t http_port_t (tcp_socket (name_bind))) In most representations, we can see what the rule is about: The subject (who is taking the action): In this case, it is a processes labeled with the httpd_t type. The target resource or object (the target for the action): In this case, it is a TCP socket (tcp_socket) labeled with the http_port_t type. In reference policy style, this is implied by the function name. The action or permission: In this case, it is binding to a port (name_bind). In reference policy style, this is implied by the function name. The result that the policy will enforce: In this case, it is that the action is allowed (allow). In reference policy style, this is implied by the function name. A policy is generally written for an application or set of applications. So the preceding example will be part of the policy written for web servers. Policy writers will generally create three files per application or application set: A .te file, which contains the type enforcement rules. An .if file, which contains interface and template definitions, allowing policy writers to easily use the newly generated policy rules to enhance other policies with. You can compare this to header files in other programming languages. An .fc file, which contains file context expressions. These are rules that assign labels to resources on the filesystem. A finished policy will then be packaged into an SELinux policy module. Distributing policies through modules Initially, SELinux used a single, monolithic policy approach: all possible access control rules are maintained in a single policy file. It quickly became clear that this is not manageable in the long term, and the idea of developing a modular policy approach was born. Within the modular approach, policy developers can write isolated policy sets for a particular application (or set of applications), roles, and so on. These policies then get built and distributed as policy modules. Platforms that need access controls for a particular application load the SELinux policy module that defines the access rules for that application. The process of building policy modules is shown in the next diagram. It also shows where CIL comes into play, even when the policy rules themselves are not written in CIL. For distributions that do not yet support CIL, semodule will directly go from the .pp file to the policy.## file. Build process from policy rule to policy store With the recent SELinux user space, the *.pp files (which are the SELinux policy modules) are considered to be written in a high-level language (HLL). Do not assume that this means they are human readable: these files are binary files. The consideration here is that SELinux wants to support writing SELinux policies in a number of formats, which it calls high-level languages, as long as it has a parser that can convert the files into CIL. Marking the binary module formats as high-level allowed the SELinux project to introduce the distinction between high-level languages and CIL in a backward-compatible manner. When distributing SELinux policy modules, most Linux distributions place the *.pp SELinux policy modules inside /usr/share/selinux, usually within a subdirectory named after the policy store (such as targeted). There, these modules are ready for administrators to activate them. When activating a module, the semodule command (part of the policycoreutils package) will copy those modules into a dedicated directory: /etc/selinux/targeted/modules/active/modules (RHEL) or /var/lib/selinux/mcs/active/modules (Gentoo). This location is defined by the version of the SELinux user space—more recent versions use the /var/lib location. When all modules are aggregated in a single location, the final policy binary is compiled, resulting in /etc/selinux/targeted/policy/policy.30 (or some other number) and loaded in memory. On RHEL, the SELinux policies are provided by the selinux-policy-targeted (or -minimum or -mls) package. On Gentoo, they are provided by the various sec-policy/selinux-* packages (Gentoo uses separate packages for each module, reducing the number of SELinux policies that are loaded on an average system). Bundling modules in a policy store A policy store contains a single comprehensive policy, and only a single policy can be active on a system at any point in time. Administrators can switch policy stores, although this often requires the system to be rebooted and might even require relabeling the entire system (relabeling is the act of resetting the contexts on all files and resources available on that system). The active policy on the system can be queried using sestatus (SELinux status, provided through the policycoreutils package), as follows: # sestatus | grep Loaded policy Loaded policy name: targeted In this example, the currently loaded policy (store) is named targeted. The policy name that SELinux will use upon its next reboot is defined in the /etc/selinux/config configuration file as the SELINUXTYPE parameter. It is the init system of systems (be it a SysV-compatible init system or systemd) that is generally responsible for loading the SELinux policy, effectively activating SELinux support on the system. The init system reads the configuration, locates the policy store, and loads the policy file in memory. If the init system does not support this (in other words, it is not SELinux-aware) then the policy can be loaded through the load_policy command, part of the policycoreutils package. Distinguishing between policies The most common SELinux policy store names are strict, targeted, mcs, and mls. None of the names assigned to policy stores are fixed, though, so it is a matter of convention. Hence, it is recommended to consult the distribution documentation to verify what should be the proper name of the policy. Still, the name often provides some information about the SELinux options that are enabled through the policy. Supporting MLS One of the options that can be enabled is MLS support. If it's disabled, the SELinux context will not have a fourth field with sensitivity information in it, making the contexts of processes and files look as follows: staff_u:sysadm_r:sysadm_t To check whether MLS is enabled, it is sufficient to see whether the context indeed doesn't contain such a fourth field, but it can also be acquired from the Policy MLS status line in the output of sestatus: # sestatus | grep MLS Policy MLS Status: disabled Another method would be to look into the pseudo file, /sys/fs/selinux/mls. A value of 0 means disabled, whereas a value of 1 means enabled: # cat /sys/fs/selinux/mls 0 Policy stores that have MLS enabled are generally targeted, mcs and mls, whereas strict generally has MLS disabled. Dealing with unknown permissions Permissions (such as read, open, and lock) are defined both in the Linux kernel and in the policy itself. However, sometimes, newer Linux kernels support permissions that the current policy does not yet understand. Take the block_suspend permission (to be able to block system suspension) as an example. If the Linux kernel supports (and checks) this permission but the loaded SELinux policy does not understand that permission yet, then SELinux has to decide how it should deal with the permission. SELinux can be configured to do one of the following actions: allow: assume everything that is not understood is allowed deny: assume no one is allowed to perform this action reject: stop and halt the system This is configured through the deny_unknown value. To see the state for unknown permissions, look for the Policy deny_unknown status line in sestatus: # sestatus | grep deny_unknown Policy deny_unknown status: denied Administrators can set this for themselves in the /etc/selinux/semanage.conf file through the handle-unknown variable (with allow, deny, or reject). RHEL by default allows unknown permissions, whereas Gentoo by default denies them. Supporting unconfined domains An SELinux policy can be very strict, limiting applications as close as possible to their actual behavior, but it can also be very liberal in what applications are allowed to do. One of the concepts available in many SELinux policies is the idea of unconfined domains. When enabled, it means that certain SELinux domains (process contexts) are allowed to do almost anything they want (of course, within the boundaries of the regular Linux DAC permissions, which still hold) and only a select number of domains are truly confined (restricted) in their actions. Unconfined domains have been brought forward to allow SELinux to be active on desktops and servers where administrators do not want to fully restrict the entire system, but only a few of the applications running on it. Generally, these implementations focus on constraining network-facing services (such as web servers and database management systems) while allowing end users and administrators to roam around unrestricted. With other MAC systems, such as AppArmor, unconfinement is inherently part of the design of the system as they only restrict actions for well-defined applications or users. However, SELinux was designed to be a full mandatory access control system and thus needs to provide access control rules even for those applications that shouldn't need any. By marking these applications as unconfined, almost no additional restrictions are imposed by SELinux. We can see whether unconfined domains are enabled on the system through seinfo, which we use to query the policy for the unconfined_t SELinux type. On a system where unconfined domains are supported, this type will be available: # seinfo -tunconfined_t unconfined_t For a system where unconfined domains are not supported, the type will not be part of the policy: # seinfo -tunconfined_t ERROR: could not find datum for type unconfined_t Most distributions that enable unconfined domains call their policy targeted, but this is just a convention that is not always followed. Hence, it is always best to consult the policy using seinfo. RHEL enables unconfined domains, whereas with Gentoo, this is a configurable setting through the unconfined USE flag. Limiting cross-user sharing When UBAC is enabled, certain SELinux types will be protected by additional constraints. This will ensure that one SELinux user cannot access files (or other specific resources) of another user, even when those users are sharing their data through the regular Linux permissions. UBAC provides some additional control over information flow between resources, but it is far from perfect. In its essence, it is made to isolate SELinux users from one another. A constraint in SELinux is an access control rule that uses all parts of a context to make its decision. Unlike type-enforcement rules, which are purely based on the type, constraints can take the SELinux user, SELinux role, or sensitivity label into account. Constraints are generally developed once and then left untouched—most policy writers will not touch constraints during their development efforts. Many Linux distributions, including RHEL, disable UBAC. Gentoo allows users to select whether or not they want UBAC through the Gentoo ubac USE flag (which is enabled by default). Incrementing policy versions While checking the output of sestatus, we see that there is also a notion of policy versions: # sestatus | grep version Max kernel policy version: 28 This version has nothing to do with the versioning of policy rules but with the SELinux features that the currently running kernel supports. In the preceding output, 28 is the highest policy version the kernel supports. Every time a new feature is added to SELinux, the version number is increased. The policy file itself (which contains all the SELinux rules loaded at boot time by the system) can be found in /etc/selinux/targeted/policy (where targeted refers to the policy store used, so if the system uses a policy store named strict, then the path would be /etc/selinux/strict/policy). If multiple policy files exist, we can use the output of seinfo to find out which policy file is used: # seinfo Statistics for policy file: /etc/selinux/targeted/policy/policy.30 Policy Version & Type: v.30 (binary, mls) ... The next table provides the current list of policy feature enhancements and the Linux kernel version in which that feature is introduced. Many of the features are only of concern to the policy developers, but knowing the evolution of the features gives us a good idea about the evolution of SELinux. Version Linux kernel Description 12   The old API for SELinux, now deprecated. 15 2.6.0 Introduced the new API for SELinux. 16 2.6.5 Added support for conditional policy extensions. 17 2.6.6 Added support for IPv6. 18 2.6.8 Added support for fine-grained netlink socket permissions. 19 2.6.12 Added support for MLS. 20 2.6.14 Reduced the size of the access vector table. 21 2.6.19 Added support for MLS range transitions. 22 2.6.25 Introduced policy capabilities. 23 2.6.26 Added support for per-domain permissive mode. 24 2.6.28 Added support for explicit hierarchy (type bounds). 25 2.6.39 Added support for filename-based transitions. 26 3.0 Added support for role transitions for non-process classes. Added support for role attributes. 27 3.5 Added support for flexible inheritance of user and role for newly created objects. 28 3.5 Added support for flexible inheritance of type for newly created objects. 29 3.14 Added support for attributes within SELinux constraints. 30 4.3 Added support for extended permissions and implemented first on IOCTL controls. Enhanced SELinux XEN support. History of SELinux feature evolution By default, when an SELinux policy is built, the highest supported version as defined by the Linux kernel and libsepol (the library responsible for building the SELinux policy binary) is used. Administrators can force a version to be lower using the policy-version parameter in /etc/selinux/semanage.conf. Different policy content Besides the aforementioned policy capabilities, the main difference between policies (and distributions) is the policy content itself. We already covered that most distributions base their policy on the reference policy project. But although that project is considered the master for most distributions, each distribution has its own deviation from the main policy set. Many distributions make extensive additions to the policy without directly passing the policies to the upstream reference policy project. There are several possible reasons why this is not directly done: The policy enhancements or additions are still immature: Red Hat initially starts with policies being active but permissive, meaning the policies are not enforced. Instead, SELinux logs what it would have prevented and, based on those, logs the policies that are enhanced. This ensures that a policy is only ready after a few releases. The policy enhancements or additions are too specific to the distribution: If a policy set is not reusable for other distributions, then some distributions will opt to keep those policies to themselves as the act of pushing changes to upstream projects takes quite some effort. The policy enhancements or additions haven't followed the upstream rules and guidelines: The reference policy has a set of guidelines that policies need to adhere to. If a policy set does not comply with these rules, then it will not be accepted. The policy enhancements or additions are not implementing the same security model as the reference policy project wants: As SELinux is a very extensive mandatory access control system, it is possible to write completely different policies. The distribution does not have the time or resources to push changes upstream. This ensures that SELinux policies between distributions (and even releases of the same distribution) can, content-wise, be quite different. Gentoo for instance aims to follow the reference policy project closely, with changes being merged within a matter of weeks. Summary In this article, we saw that SELinux offers a more fine-grained access control mechanism on top of the Linux access controls. SELinux is implemented through Linux Security Modules and uses labels to identify its resources and processes based on ownership (user), role, type, and even the security sensitivity and categorization of the resource. We covered how SELinux policies are handled within an SELinux-enabled system and briefly touched upon how policy writers structure policies. Linux distributions implement SELinux policies, which might be a bit different from each other based on supporting features, such as sensitivity labels, default behavior for unknown permissions, support for confinement levels, or specific constraints put in place such as UBAC. However, most of the policy rules themselves are similar and are even based on the same upstream reference policy project. Resources for Article: Further resources on this subject: SELinux - Highly Secured Web Hosting for Python-based Web Applications [article] Introduction to Docker [article] Booting the System [article]
Read more
  • 0
  • 0
  • 13507
article-image-turn-your-life-gamified-experience-unity
Packt
14 Nov 2016
29 min read
Save for later

Turn Your Life into a Gamified Experience with Unity

Packt
14 Nov 2016
29 min read
In this article for by Lauren S. Ferro from the book Gamification with Unity 5.x we will look into a Gamified experience with Unity. In a world full of work, chores, and dull things, we all must find the time to play. We must allow ourselves to be immersed in enchanted world of fantasy and to explore faraway and uncharted exotic islands that form the mysterious worlds. We may also find hidden treasure while confronting and overcoming some of our worst fears. As we enter these utopian and dystopian worlds, mesmerized by the magic of games, we realize anything and everything is possible and all that we have to do is imagine. Have you ever wondered what Gamification is? Join us as we dive into the weird and wonderful world of gamifying real-life experiences, where you will learn all about game design, motivation, prototyping, and bringing all your knowledge together to create an awesome application. Each chapter in this book is designed to guide you through the process of developing your own gamified application, from the initial idea to getting it ready and then published. The following is just a taste of what to expect from the journey that this book will take you on. (For more resources related to this topic, see here.) Not just pixels and programming The origins of gaming have an interesting and ancient history. It stems as far back as the ancient Egyptians with the game Sennet; and long since the reign of great Egyptian Kings, we have seen games as a way to demonstrate our strength and stamina, with the ancient Greeks and Romans. However, as time elapsed, games have not only developed from the marble pieces of Sennet or the glittering swords of battles, they have also adapted to changes in the medium: from stone to paper, and from paper to technology. We saw the rise and development of physical games (such as table top and card games) to games that require us to physically move our characters using our bodies and peripherals (Playstaton Move and WiiMote), in order to interact with the gaming environment (Wii Sports and Heavy Rain). So, now we not only have the ability to create 3D virtual worlds with virtual reality, but also can enter these worlds and have them enter ours with augmented reality. Therefore, it is important to remember that, just as the following image, (Dungeons and Dragons), games don't have to take on a digital form, they can also be physical: Dungeons and Dragons board with figurines and dice Getting contextual At the beginning of designing a game or game-like experience, designers need to consider the context for which the experience is to take place. Context is an important consideration for how it may influence the design and development of the game (such as hardware, resources, and target group). The way in which a designer may create a game-like experience varies. For example, a game-like experience aimed to encourage students to submit assessments on time will be designed differently from the one promoting customer loyalty. In this way, the designer should be more context aware, and as a result, it may be more likely to keep it in view during the design process. Education: Games can be educational, and they may be designed specifically to teach or to have elements of learning entwined into them to support learning materials. Depending on the type of learning game, it may include formal (educational institutions) or informal educational environments (learning a language for a business trip). Therefore, if you are thinking about creating an educational game, you might need to think about these considerations in more detail. Business: Maybe your intention is get your employees to arrive on time or to finish reports in the afternoon rather than right before they go home. Designing content for use within a business context targets situations that occur within the workplace. It can include objectives such as increasing employee productivity (individual/group). Personal: Getting personal with game-like applications can relate specifically to creating experiences to achieve personal objectives. These may include personal development, personal productivity, organization, and so on. Ultimately, only one person maintains these experiences; however, other social elements, such as leaderboards and group challenges, can bring others into the personal experience as well. Game: If it is not just educational, business, or personal development, chances are that you probably want to create a game to be a portal into lustrous worlds of wonder or to pass time on the evening commute home. Pure gaming contexts have no personal objectives (other than to overcome challenges of course). Who is our application targeting and where do they come from? Understanding the user is one of the most important considerations for any approach to be successful. User considerations not only include the demographics of the user (for example, who they are and where they are from), but also the aim of the experience, the objectives that you aim to achieve, and outcomes that the objectives lead to. In this book, this section considers the real-life consequences that your application/game will have on its audience. For example, will a loyalty application encourage people to engage with your products/store in the areas that you're targeting it toward. Therefore, we will explore ways that your application can obtain demographic data in Unity. Are you creating a game to teach Spanish to children, teenagers, or adults? This will change the way that you will need to think about your audience. For example, children tend to be users who are encouraged to play by their parents, teenagers tend to be a bit more autonomous but may still be influenced by their parents, and adults are usually completely autonomous. Therefore, this can influence the amount and the type of feedback that you can give and how often. Where are your audience from? For example, are you creating an application for a global reward program or a local one? This will have an effect on whether or not you will incorporate things like localization features so that the application adapts to your audience automatically or whether it's embedded into the design. What kind of devices does your audience use? Do they live in an area where they have access to a stable Internet connection? Do they need to have a powerful system to run your game or application? Chances are if the answer is yes for the latter question then you should probably take a look at how you will optimize your application. What is game design? Many types of games exist and so do design approaches. There are different ways that you can design and implement games. Now, let's take a brief look at how games are made, and more importantly, what they are made of: Generating ideas: This involves thinking about the story that we want to tell, or a trip that we may want the player to go on. At this stage, we're just getting everything out of our head and onto the paper. Everything and anything should be written; the stranger and abstract the idea, the better. It's important at this stage not to feel trapped that an idea may not be suitable. Often, the first few ideas that we create are the worst, and the great stuff comes from iterating all the ideas that we put down in this stage. Talk about your ideas with friends and family, and even online forums are a great place to get feedback on your initial concepts. One of the first things that any aspiring game designer can begin with is to look at what is already out there. A lot is learned when we succeed—or fail—especially why and how. Therefore, at this stage, you will want to do a bit of research about what you are designing. For instance, if you're designing an application to teach English, not only should you see other similar applications that are out there but also how English is actually taught, even in an educational environment. While you are generating ideas, it is also useful to think about the technology and materials that you will use along the way. What game engine is better for your game's direction? Do you need to purchase licenses if you are intending to make your game commercial? Answering these kinds of questions earlier can save many headaches later on when you have your concept ready to go. Especially, if you will need to learn how to use the software, as some have steep learning curves. Defining your idea: This is not just a beautiful piece of art that we see when a game is created; it can be rough, messy, and downright simple, but it communicates the idea. Not just this; it also communicates the design of the game's space and how a player may interact and even traverse it. Concept design is an art in itself and includes concepts on environments, characters puzzles, and even the quest itself. We will take the ideas that we had during the idea generation and flesh them out. We begin to refine it, to see what works and what doesn't. Again, get feedback. The importance of feedback is vital. When you design games, you often get caught up; you are so immersed in your ideas, and they make sense to you. You have sorted out every details (at least for the most part, it feels like that). However, you aren't designing for you, you are designing for your audience, and getting an outsiders opinion can be crucial and even offer a perspective that you may not necessarily would have thought of. This stage also includes the story. A game without a story is like a life without existence. What kind of story do you want your player to be a part of? Can they control it, or is it set in stone? Who are the characters? The answers to these questions will breathe soul into your ideas. While you design your story, keep referring to the concept that you created, the atmosphere, the characters, and the type of environment that you envision. Some other aspects of your game that you will need to consider at this stage are as follows: How will your players learn how to play your game? How will the game progress? This may include introducing different abilities, challenges, levels, and so on. Here is where you will need to observe the flow of the game. Too much happening and you will have a recipe for chaos, not enough and your player will get bored. What is the number of players that you envision playing your game, even if you intend for a co-op or online mode? What are the main features that will be in your game? How will you market your game? Will there be an online blog that documents the stages of development? Will it include interviews with different members of the team? Will there be different content that is tailored for each network (for example, Twitter, Facebook, Instagram, and so on). Bringing it together: This involves thinking about how all your ideas will come together and how they will work, or won't. Think of this stage as creating a painting. You may have all pieces, but you need to know how to use them to create the piece of art. Some brushes (for example, story, characters) work better with some paints (for example, game elements, mechanics), and so on. This stage is about bringing your ideas and concepts into reality. This stage features design processes, such as the following: Storyboards that will give an overview of how the story and the gameplay evolve throughout the game. Character design sheets that will outline characteristics about your characters and how they fit into the story. Game User Interfaces (GUIs) that will provide information to the player during gameplay. This may include elements, such as progress bars, points, and items that they will collect along the way. Prototyping: This is where things get real…well, relatively. It may be something as simple as a piece of paper or something more complex as a 3D model. You then begin to create the environments or the levels that your player will explore. As you develop your world, you will take your content and populate the levels. Prototyping is where we take what was in our head and sketched out on paper and use it to sculpt the gameful beast. The main purpose of this stage is to see how everything works, or doesn't. For example, the fantastic idea of a huge mech-warrior with flames shooting out of an enormous gun on its back was perhaps not the fantastic idea that was on paper, at least not in the intended part of the game. Rapid prototyping is fast and rough. Remember when you were in school and you had things, such as glue, scissors, pens, and pencils; well, that is what you will need for this. It gets the game to a functioning point before you spend tireless hours in a game engine trying to create your game. A few bad rapid prototypes early on can save a lot of time instead of a single digital one. Lastly, rapid prototyping isn't just for the preliminary prototyping phase. It can be used before you add in any new features to your game once it's already set up. Iteration: This is to the game what an iron is to a creased shirt. You want your game to be on point and iterating it gets it to that stage. For instance, that awesome mech-warrior that you created for the first level was perhaps better as the final boss. Iteration is about fine-tuning the game, that is, to tweak it so that it not only flows better overall, but also improves the gameplay. Playtesting: This is the most important part of the whole process once you have your game to a relatively functioning level. The main concept here is to playtest, playtest, and playtest. The importance of this stage cannot be emphasized enough. More often than not, games are buggy when finally released, with problems and issues that could be avoided during this stage. As a result, players lose interest and reviews contain frustration and disappointment, which—let's face it—we don't want after hours and hours of blood, sweat, and tears. The key here is not only to playtest your game but also to do it in multiple ways and on multiple devices with a range of different people. If you release your game on PC, test it on a high performance one and a low performance one. The same process should be applied for mobile devices (phones, tablets) and operating systems. Evaluate: Evaluateyour game based on the playtesting. Iterating, playtesting, and evaluating are three steps that you will go through on a regular basis, more so as you implement a new feature or tweak an existing one. This cycle is important. You wouldn't buy a car that has parts added without being tested first so why should a player buy a game with untested features? Build: Build your game and get it ready for distribution, albeit on CD or online as a digital download Publish: Publish your game! Your baby has come of age and is ready to be released out into the wild where it will be a portal for players around the world to enter the world that you (and your team) created from scratch. Getting gamified When we merge everyday objectives with games, we create gamified experiences. The aim of these experiences is to improve something about ourselves in ways that are ideally more motivating than how we perceive them in real life. For example, think of something that you find difficult to stay motivated with. This may be anything from managing your finances, to learning a new language, or even exercising. Now, if you make a deal with yourself to buy a new dress once you finish managing your finances or to go on a trip once you have learned a new language, you are turning the experience into a game. The rules are simply to finish the task; the condition of finishing it results in a reward—in the preceding example, either a dress or the trip. The fundamental thing to remember is that gamified experiences aim to make ordinary tasks extraordinary and enjoyable for the player. Games, gaming, and game-like experiences can give rise to many types of opportunities for us to play or even escape reality. To finish this brief exploration into the design of games, we must realize that games are not solely about sitting in front of the TV, playing on the computer, or being glued to the seat transfixed on a digital character dodging bullets. The game mechanics that make a task more engaging and fun is defined as "Gamification." Gamification relates to games, and not play; while the term has become popular, the concept is not entirely new. Think about loyalty cards, not just frequent flyer mile programs, but maybe even at your local butcher or café. Do you get a discount after a certain amount of purchases? For example, maybe, the tenth coffee is free. It's been a while since various reward schemes have already been in place; giving children a reward for completing household chores or for good behavior and rewarding "gold stars" for academic excellence is gamification. If you consider some social activities, such as Scouts, they utilize "gamification" as part of their procedures. Scouts learn new skills and cooperate and through doing so, they achieve status and receive badges of honor that demonstrate levels of competency. Gamification has become a favorable approach to "engaging" clients with new and exciting design schemes to maintain interest and promote a more enjoyable and ideally "fun" product. The product in question does not have to be "digital." Therefore, "gamification" can exist both in a physical realm (as mentioned before with the rewarding of gold stars) as well as in a more prominent digital sense (for example, badge and point reward systems) as an effective way to motivate and engage users. Some common examples of gamification include the following: Loyalty programs: Each time you engage with the company in a particular way, such as buying certain products or amount of, you are rewarded. These rewards can include additional products, points toward items, discounts, and even free items. School House points: A pastime that some of us may remember, especially fans of Harry Potter is that each time you do the right thing, such as follow the school rules, you get some points. Alternatively, you do the wrong thing, and you lose points. Scouts: It rewards levels of competency with badges and ranks. The more skilled you are, the more badges you collect, wear, and ultimately, the faster you work your way up the hierarchy. Rewarding in general: This will often be associated with some rules, and these rules determine whether or not you will get a reward. Eat your vegetables, you will get dessert; do your math homework, you will get to play. Both have winning conditions. Tests: As horrifying as it might sound, tests can be considered as a game. For example, we're on a quest to learn about history. Each assignment you get is like a task, preparing you for the final battle—the exam. At the end of all these assessments, you get a score or a grade that indicates to you your progress as you pass from one concept to the next. Ultimately, your final exam will determine your rank among your peers and whether or not you made it to the next level (that being anywhere from your year level to a university). It may be also worth noting that just as in games, you also have those trying to work the system, searching for glitches in the system that they can exploit. However, just as in games, they too eventually are kicked. One last thing to remember when you design anything targeted toward kids is that they can be a lot more perceptive than what we sometimes give them credit for. Therefore, if you "disguise" educational content with gameplay, it is likely that they will see through it. It's the same with adults; they know that they are monitoring their health or spending habits, it's your job to make it a little less painful. Therefore, be upfront, transparent, and cut through the "disguise." Of course, kids don't want to be asked to "play a game about maths" but they will be more interested in "going on adventures to beat the evil dragon with trigonometry." The same goes for adults; creating an awesome character that can be upgraded to a level-80 warrior for remembering to take out the trash, keep hydrated, and eat healthier is a lot better than telling them this is a "fun" application to become a better person. There is no I in Team Working on our own can be good, sometimes working with others can be better! However, the problem with working in a team is that we're all not equal. Some of us are driven by the project, with the aim to get the best possible outcome, whereas, others are driven by fame, reward, money, and the list goes on. If you ever worked on a group project in school, then you know exactly what it's like. Agile gamification is, to put simply, getting teams to work better together. Often, large complex projects encounter a wide range of problems from keeping on top of schedules, different perspectives, undefined roles, and a lack of overall motivation. Agile frameworks in this context are associated with the term Scrum. This describes an overall framework used to formalize software development projects. The Scrum process works as follows: The owner of the product will create a wish list known as the product backlog. Once the sprint planning begins, members of the team (between 3-9 people) will take sections from the top of the product backlog. Sprint planning involves the following: It involves listing all of the items that are needed to be completed for the project (in a story format—who, what, and why). This list needs to be prioritized. It includes estimating each task relatively (using the Fibonacci system). It involves planning the work sprint (1-2 week long, but less than 1 month long) and working toward a demo. It also involves making the work visible using a storyboard that contains the following sections: To do, Doing, and Done. Items begin in the To do section; once they have begun, they move to the Doing section; and once they are completed, they are then put in the Done section. The idea is that the team works through tasks in the burn down chart. Ideally, the amount of points that the sprint began with (in terms of tasks to be done) decreases in value each day you get closer to finishing the sprint. The team engages with daily meetings (preferably standing up) run by the Sprint/Scrum master. These meetings discuss what was done, what is planned to be done during the day, any issues that come up or might come up, and how can improvements be made. It provides a demonstration of the product's basic (working) features. During this stage, feedback is provided by the product owner as to whether or not they are happy with what has been done, the direction that it is going, and how it will relate to the remaining parts of the project. At this stage, the owner may ask you to improve it, iterate it, and so forth, for the next sprint. Lastly, the idea is to get the team together and to review the development of the project as a whole: what went well and what didn't go so well and what are the areas of improvement that can then be used to make the next Scrum better? Next, they will decide on how to implement each section. They will meet each day to not only assess the overall progress made for the development of each section but also to ensure that the work will be achieved within the time frame. Throughout the process, the team leader known as the Scrum/Sprint Master has the job of ensuring that the team stays focused and completes sections of the product backlog on time. Once the sprint is finished, the work should be at a level to be shipped, sold to the customer, or to at least show to a stakeholder. At the end of the sprint, the team and Scrum/Sprint Master assess the completed work and determine whether it is at an acceptable level. If the work is approved, the next sprint begins. Just as the first sprint, the team chooses another chunk of the product backlog and begins the process again.An overview of the Scrum process However, in the modern world, Scrum is adopted and applied to a range of different contexts outside of software development. As a result, it has gone through some iterations, including gamification. Agile Gamification, as it is more commonly known as, takes the concept of Scrum and turns it into a playful experience. Adding an element of fun to agile frameworks To turn the concept of Scrum into something a bit more interesting and at the same time to boost the overall motivation of your team, certain parts of it can be transformed with game elements. For example, implementing leaderboards based on the amount of tasks that each team member is able to complete (and on time) results in a certain number of points. By the end of the spring, the team member with the most number of points may be able to obtain a reward, such as a bonus in their next pay or an extended lunch break. It is also possible to make the burn down chart a bit more exciting by placing various bonuses if certain objectives are met within certain time frame or at a certain point during the burn down;as a result, giving added incentive to team members to get things delivered on time. In addition, to ensure that quality standards are also maintained, Scrum/Sprint Masters can also provide additional rewards if there is few or no feedback regarding things, such as quality or the overall cohesiveness of the output from the sprint. An example of a gamified framework can be seen in the image below. While setting up a DuoLingo Classroom account, users are presented with various game elements (for example, progress bar) and a checklist to ensure that everything that needs to be completed is done. Playtesting This is one of the most important parts of your game design. In fact, you cannot expect to have a great game without it. Playtesting is not just about checking whether your game works, or if there are bugs, it is also about finding out what people really think about it before you put it out in the world to see. In some cases, playtesting can make the difference between succeeding of failing epically. Consider this scenario: you have spent the last year, your blood, sweat and tears, and even your soul to create something fantastic. You probably think it's the best thing out there. Then, after you release it, you realize that only half the game was balanced, or worst, half interesting. At this stage, you will feel pretty down, but all these could have been avoided if you had taken the time to get some feedback. As humans, we don't necessarily like to hear our greatest love being criticized, especially if we have committed so much of our lives to it. However, the thing to keep in mind is, this stage shapes the final details. Playtesting is not meant for the final stages, when your game is close to being finished. At each stage, even when you begin to get a basic prototype completed, it should be play tested. During these stages, it does not have to be a large-scale testing, it can be done by a few colleagues, friends, or even family who can give you an idea of whether or not you're heading in the right direction. Of course, the other important thing to keep in mind is that the people who are testing your game are as close, if not the target audience. For instance, image that you're creating for your gamified application to encourage people to take medication on a regular basis is not ideal to test with people who do not take medication. Sure, they may be able to cover general feedback, such as user interface elements or even interaction, but in terms of its effectiveness, you're better off taking the time to recruit more specific people. Iterating After we have done all the playtesting is the time to re-plan another development cycle. In fact, the work of tuning your application doesn't stop after the first tests. On the contrary, it goes through different iterations many times. The iteration cycle starts with the planning stage, which include brainstorming, organizing the work (as we saw for instance in Scrum), and so on. In the next phase, development, we actually create the application, as we did in the previous chapter. Then, there is the playtesting, which we saw earlier in this chapter. In the latter stage, we tune and tweak values and fix bugs from our application. Afterward, we iterate the whole cycle again, by entering in the planning stage again. Here, we will need to plan the next iteration: what should be left and what should be done better or what to remove. All these decisions should be based on what we have collected in the playtesting stage. The cycle is well represented in the following diagram as a spiral that goes on and on through the process: The point of mentioning it now is because after you finish playtesting your game, you will need to repeat the stages that we have done previously, again. You will have to modify your design; you may need to even redesign things again. So, it is better to think of this as upgrading your design, rather than a tedious and repetitive process. When to stop? In theory, there is no stopping; the more the iteration, the better the application will be. Usually, the iterations stop when the application is well enough for your standards or when external constrains, such as the market or deadlines, don't allow you to perform any more iteration. The question when to stop? is tricky, and the answer really depends on many factors. You will need to take into account the resources needed to perform another iteration and time constraints. Of course, remember that your final goal is to deliver a quality product to your audience and each iteration is a step closer. Taking in the view with dashboards Overviews, summaries, and simplicity make life easier. Dashboards are a great way for keeping a lot of information relatively concise and contained, without being too overwhelming to a player. Of course, if the players want to obtain more detailed information, perhaps statistics about their accuracy since they began, they will have the ability to do so. So, what exactly is a dashboard? A dashboard is a central hub to view all of your progress, achievements, points, and rewards. If we take a look at the following screenshot, we can get a rough idea about what kind of information that they display. The image on the left is the dashboard for Memrise and displays current language courses, in this case, German; the players' achievements and streak; and the progress that they are making in the course. On the right is the dashboard for DuoLingo. Similar to Memrise, it also features information about daily streaks, amount of time committed, and the strength of each category learned for the new language, in this case, Italian. By just looking at these dashboards, the player can get a very quick idea about how well or bad they are doing.   Different dashboards (left) Memrise (right) DuoLingo Different approaches to dashboards can encourage different behaviors depending on the data displayed and how it is displayed. For example, you can have a dashboard that provides reflective information more dominantly, such as progress bars and points. Others can provide a more social approach by displaying the players rank among friends and comparing their statistics to others who are also engaged with the application. Some dashboards may even suggest friends that have similar elements in common, such as the language that is being learned. Ideally, the design of dashboards can be as simple or as complicated as the designer decides, but typically, the less is more approach is better. Summary Everything that we discussed in this chapter is just a taste of what this book offers. Each aspect of the design process is explained in more detail, giving you not only the information, but also the practical skills that you can use to build upon and develop any gamified application from start to finish. If you want to find out about gamification, how to use it, and more importantly how to implement it into Unity, then this book is a great foundation to get you going. In particular, you will learn how to apply all these concepts into Unity and create gamified experiences. Furthermore, the book will bring you to create a gamified application starting from the basic pieces, with a particular focus to your audience and your goals. Learning about the uses of gamification does not have to stop with this book. In fact, there are many ways that you can develop the knowledge that you have gained and apply it to other tasks. Some other Packt books, such as the Unity UI Cookbook by Francesco Sapio, which you can obtain at https://www.packtpub.com/game-development/unity-ui-cookbook features a range of different recipes to implement a range of different UI elements that can even be featured in your dashboard. In fact, UIs are the key for the development of gamifed experiences and applications. The main thing is that you continue to learn, adapt, and to apply your knowledge in many different types of contexts. Resources for Article: Further resources on this subject: Buildbox 2 Game Development: peek-a-boo [article] Customizing the Player Character [article] Sprites in Action [article]
Read more
  • 0
  • 0
  • 24636

article-image-digital-and-mobile-forensics
Packt
14 Nov 2016
14 min read
Save for later

Digital and Mobile Forensics

Packt
14 Nov 2016
14 min read
In this article, Mattia Epifani and Pasquale Stirparo, co-authors of the book Learning iOS Forensics - Second Edition, would be talking mainly, if not solely, about computer forensics and computer crimes, such as when an attacker breaks into a computer network system and steals data. This would involve two types of offenses—unlawful/unauthorized access and data theft. As mobile phones became more popular, the new field of mobile forensics developed. (For more resources related to this topic, see here.) Nowadays, things have changed radically and they are still changing at quite a fast pace as technology evolves. Digital forensics, which includes all disciplines dealing with electronic evidence, is also being applied to common crimes, to those that, at least by definition, are not strictly IT crimes. Today, more than ever, we live in a society that is fully digitalized and people are equipped with all kinds of devices, which have different types of capabilities, but all of them process, store, and transmit information (mainly over the Internet). This means that forensic investigators have to be able to deal with all these devices. As defined at the first Digital Forensics Research Workshop (DFRWS) in 2001, digital forensics is: "The use of scientifically derived and proven methods toward the preservation, collection, validation, identification, analysis, interpretation, documentation, and presentation of digital evidence derived from digital sources for the purpose of facilitating or furthering the reconstruction of events found to be criminal, or helping to anticipate unauthorized actions shown to be disruptive to planned operations." As Casey asserted (Casey, 2011): "In this modern age, it is hard to imagine a crime that does not have a digital dimension." Criminals of all kinds use technology to facilitate their offenses, communicate with their peers, recruit other criminals, launder money, commit credit card fraud, gather information on their victims, and so on. This obviously creates new challenges for all the different actors involved, such as attorneys, judges, law enforcement agents, and forensic examiners. Among the cases solved in recent years, there were kidnappings where the kidnapper was caught—thanks to a request for ransom sent by e-mail from his mobile phone. There have been many cases of industrial espionage in which unfaithful employees were hiding projects in the memory cards of their smartphones, cases of drug dealing solved—thanks to the evidence found in the backup of mobile phones that was on computer, and many other such cases. Even the largest robberies of our time are now being conducted via computer networks. In this article, you will learn the following: Definition and principles of mobile forensics How to properly handle digital evidence Mobile forensics Mobile forensics is a field of study in digital forensics that focuses on mobile devices. Among the different digital forensics fields, mobile forensics is without doubt the fastest growing and evolving area of study, having an impact on many different situations from corporate to criminal investigations and intelligence gathering, which are on the rise. Moreover, the importance of mobile forensics is increasing exponentially due to the continuous fast growth of the mobile market. One of the most interesting peculiarities of mobile forensics is that mobile devices, particularly mobile phones, usually belong to a single individual, while this is not always the case with a computer that may be shared among employees of a company or members of a family. For this reason, the analysis of mobile phones gives access to plenty of personal information. Another important and interesting aspect that comes with mobile forensics, which is both challenging and frustrating at the same time for the analyst, is the multitude of different device models and the customized flavors of their operating systems available in the market. This makes it very difficult to have a single solution (either a tool or process) to address them all. Just think of all the applications people have installed on their smartphones: IM clients, web browsers, social network clients, password managers, navigation systems, and much more, other than the classic default ones, such as an address book, which can provide a lot more information than just the phone number for each contact that has been saved. Moreover, syncing such devices with a computer has become a very easy and smooth process, and all user activities, schedules, to-do lists, and everything else is stored inside a smartphone. Aren't these enough to profile a person and reconstruct all their recent activities, than building the network of contacts? Finally, in addition to a variety of smartphones and operating systems, such as Apple iOS, Google Android, Microsoft Windows Phone, and Blackberry OS, there is a massive number of so-called feature phones that use older mobile OS systems. Therefore, it's pretty clear that when talking about mobile/smartphone forensics, there is so much more than just printouts of phone calls. In fact, with a complete examination, we can retrieve SMSes/MMSes, pictures, videos, installed applications, e-mails, geolocation data, and so on—both present and deleted information. Digital evidence As mentioned earlier, on one hand the increasing involvement of mobile devices in digital forensics cases has brought a whole new series of challenges and complexities. However, on the other hand, this has also resulted in a much greater amount of evidence from criminals that it is now being used to reconstruct their activities with a more comprehensive level of detail. Moreover, while classical physical evidence may be destroyed, digital evidence, most of the time, leaves traces. Over the years, there have been several definitions of what digital evidence actually is, some of them focusing particularly on the evidentiary aspects of proof to be used in court, such as the one proposed by the Standard Working Group on Digital Evidence (SWGDE), stating that: "Digital evidence is any information of probative value that is either stored or transmitted in a digital form." The definition proposed by the International Organization of Computer Evidence (IOCE) states: "Digital evidence is information stored or transmitted in binary form that may be relied on in court." The definition given by E. Casey (Casey, 2000), refers to digital evidence as: "Physical objects that can establish that a crime has been committed, can provide a link between a crime and its victim, or can provide a link between a crime and its perpetrator." While all of these are correct, as previously said, all of these definitions focus mostly on proofs and tend to disregard data that is extremely useful for an investigation. For this reason, and for the purpose of this book, we will refer to the definition given by Carrier (Carrier, 2006), where digital evidence is defined as: "Digital data that supports or refutes a hypothesis about digital events or the state of digital data." This definition is a more general one, but better matches the current state of digital evidence and its value within the entire investigation process. Also from a standardization point of view, there have been, and still are, many attempts to define guidelines and best practices for digital forensics on how to handle digital evidence. Other than the several guidelines and special publications from NIST, there is a standard from ISO/IEC that was released in 2012, the ISO 27037 guidelines for identification, collection and/or acquisition, and preservation of digital evidence, which is not specific to mobile forensics, but is related to digital forensics in general, aiming to build a standard procedure for collecting and handling digital evidence, which will be legally recognized and accepted in court in different countries. This is a really important goal if you consider the lack of borders in the Internet era, particularly when it comes to digital crimes, where illicit actions can be perpetrated by attackers from anywhere in the world. Handling of mobile evidence In order to be useful not only in court but also during the entire investigation phase, digital evidence must be collected, preserved, and analyzed in a forensically sound manner. This means that each step, from the identification to the reporting, has to be carefully and strictly followed. Historically, we are used to referring to a methodology as forensically sound if, and only if, it would imply that the original source of evidence remains unmodified and unaltered. This was mostly true when talking about classical computer forensics, in scenarios where the forensic practitioner found the computer switched off or had to deal with external hard drives, although not completely true even in these situations. However, since the rise of live forensics, this concept has become more and more untrue. In fact, methods and tools for acquiring memory from live systems inevitably alter, even if just a little bit, the target system they are run on. The advent of mobile forensics stresses this concept even more, because mobile devices, and smartphones in particular, are networked devices that continuously exchange data through several communication protocols, such as GSM/CDMA, Wi-Fi, Bluetooth, and so on. Moreover, in order to acquire a mobile device, forensic practitioners need to have some degree of interaction with the device. Based on the type, a smartphone can need more or less interaction, altering in this way the original state of the device. All of this does not mean that preservation of the source evidence is useless, but that it is nearly impossible in the field of mobile devices. Therefore, it becomes a matter of extreme importance to thoroughly document every step taken during the collection, preservation, and acquisition phases. Using this approach, forensic practitioners will be able to demonstrate that they have been as unintrusive as possible. As Casey states (Casey, 2011): "One of the keys to forensic soundness is documentation. A solid case is built on supporting documentation that reports on where the evidence originated and how it was handled. From a forensic standpoint, the acquisition process should change the original evidence as little as possible and any changes should be documented and assessed in the context of the final analytical results." When in the presence of mobile devices to be collected, it is a good practice for the forensic practitioner to consider the following points: Take note of the current location where the device has been found. Report the device status (switched on or off, broken screen, and so on). Report date, time, and other information visible on the screen if the device is switched on, for example, by taking a picture of the screen. Look very carefully for the presence of memory cards. Although it is not the case with iOS devices, generally many mobile phones have a slot for an external memory card, where pictures, chat databases, and many other types of user data are usually stored. Look very carefully for the presence of cables related to the mobile phone that is being collected, especially if you don't have a full set of cables in your lab. Many mobile phones have their own cables to connect to the computer and to recharge the battery. Search for the original Subscriber Identity Module (SIM) package, because that is where the PIN and PIN unblocking key (PUK) codes are written. Take pictures of every item before collection. Modifications to mobile devices can happen not only because of interaction with the forensic practitioner, but also due to interaction with the network, voluntarily or not. In fact, digital evidence in mobile devices can be lost completely as they are susceptible to being overwritten by new data, for example, with the smartphone receiving an SMS while it is being collected, thus overwriting possible evidence previously stored in the same area of memory as the newly arrived SMS, or upon receiving a remote wiping command over a wireless network. Most of today's smartphones and iOS devices can be configured to be completely wiped remotely. From a real case: While searching inside the house of a person under investigation, law enforcement agents found and seized, among other things, computers and a smartphone. After cataloguing and documenting everything, they put all the material into boxes to bring them back to the laboratory. Once back in their laboratory, when acquiring the smart phone in order to proceed with the forensics analysis, they noticed that the smartphone was empty and it appeared to be brand new. The owner had wiped it remotely. Therefore, isolating the mobile device from all radio networks is a fundamental step in the process of preservation of evidence. There are several ways to achieve this, all with their own pros and cons, as follows: Airplane mode: Enabling Airplane mode on a device requires some sort of interaction, which may pose some risks of modification by the forensic practitioner. This is one of the best possible options since it implies that all wireless communication chips are switched off. In this case, it is always good to document the action taken with pictures and/or videos. Normally, this is possible only if the phone is not password-protected or the password is known. However, for devices with iOS 7 or higher, it is also possible to enable airplane mode by lifting the dock from the bottom, where there will be a button with the shape of a plane. This is possible only if the Access on Lock Screen option is enabled from Settings | Control Center. Faraday's bag: This item is a sort of envelope made of conducting material, which blocks out static electric fields and electromagnetic radiation completely isolating the device from communicating with external networks. It is based, as the name suggests, on Faraday's law. This is the most common solution, particularly useful when the device is being carried from the crime scene to the lab after seizure. However, the use of Faraday's bag will make the phone continuously search for a network, which will cause the battery to quickly drain. Unfortunately, it is also risky to plug the phone to a power cable outside that will go inside the bag, because this may act as antenna. Moreover, it is important to keep in mind that when you remove the phone from the bag (once arrived in the lab) it will again be exposed to the network. So, you would need either a shielded lab environment or a Faraday solution that would allow you to access the phone while it is still inside the shielded container, without the need for external power cables. Jamming: A jammer is used to prevent a wireless device from communicating by sending out radio waves along the same frequencies as that device. In our case, it would jam the GSM/UMTS/LTE frequencies that mobile phones use to connect with cellular base stations to send/receive data. Be aware that this practice may be considered illegal in some countries, since it will also interfere with any other mobile device in the range of the jammer, disrupting their communications too. Switching off the device: This is a very risky practice because it may activate authentication mechanisms, such as PIN codes or passcodes, that are not available to the forensic practitioner, or other encryption mechanisms that carry the risk of delaying or even blocking the acquisition of the mobile device. Removing the SIM card: In most mobile devices, this operation implies removing the battery and therefore all the risks and consequences we just mentioned regarding switching off the device; however, in iOS devices this task is quite straightforward and easy, and it does not imply removing the battery (in iOS devices this is not possible). Moreover, SIM cards can have PIN protection enabled; removing it from the phone may lock the SIM card, preventing its content from being displayed. However, bear in mind that removing the SIM card will isolate the device only from the cellular network, while other networks, such as Wi-Fi or Bluetooth, may still be active and therefore need to be addressed. The following image shows a SIM card extracted from an iPhone with just a clip; image taken from http://www.maclife.com/: Summary In this article, we gave a general introduction to digital forensics for those relatively new to this area of study and a good recap to those already in the field, keeping the mobile forensics field specifically in mind. We have shown what digital evidence is and how it should be handled, presenting several techniques to isolate the mobile device from the network. Resources for Article: Further resources on this subject: Mobile Forensics [article] Mobile Forensics and Its Challanges [article] Forensics Recovery [article]
Read more
  • 0
  • 0
  • 23716
Modal Close icon
Modal Close icon