How-To Tutorials

In this article by Andrea Isoni author of the book Machine Learning for the Web, we will the most relevant regression and classification techniques are discussed. All of these algorithms share the same background procedure, and usually the name of the algorithm refers to both a classification and a regression method. The linear regression algorithms, Naive Bayes, decision tree, and support vector machine are going to be discussed in the following sections. To understand how to employ the techniques, a classification and a regression problem will be solved using the mentioned methods. Essentially, a labeled train dataset will be used to train the models, which means to find the values of the parameters, as we discussed in the introduction. As usual, the code is available in the my GitHub folder at https://github.com/ai2010/machine_learning_for_the_web/tree/master/chapter_3/. (For more resources related to this topic, see here.) We will conclude the article with an extra algorithm that may be used for classification, although it is not specifically designed for this purpose (hidden Markov model). We will now begin to explain the general causes of error in the methods when predicting the true labels associated with a dataset. Model error estimation We said that the trained model is used to predict the labels of new data, and the quality of the prediction depends on the ability of the model to generalize, that is, the correct prediction of cases not present in the trained data. This is a well-known problem in literature and related to two concepts: bias and variance of the outputs. The bias is the error due to a wrong assumption in the algorithm. Given a point x(t) with label yt, the model is biased if it is trained with different training sets, and the predicted label ytpred will always be different from yt. The variance error instead refers to the different wrongly predicted labels of the given point x(t). A classic example to explain the concepts is to consider a circle with the true value at the center (true label), as shown in the following figure. The closer the predicted labels are to the center, the more unbiased the model and the lower the variance (top left in the following figure). The other three cases are also shown here: Variance and bias example. A model with low variance and low bias errors will have the predicted labels that is blue dots (as show in the preceding figure) concentrated on the red center (true label). The high bias error occurs when the predictions are far away from the true label, while high variance appears when the predictions are in a wide range of values. We have already seen that labels can be continuous or discrete, corresponding to regression classification problems respectively. Most of the models are suitable for solving both problems, and we are going to use word regression and classification referring to the same model. More formally, given a set of N data points and corresponding labels, a model with a set of parameters with the true parameter values will have the mean square error (MSE), equal to: We will use the MSE as a measure to evaluate the methods discussed in this article. Now we will start describing the generalized linear methods. Generalized linear models The generalized linear model is a group of models that try to find the M parameters that form a linear relationship between the labels yi and the feature vector x(i) that is as follows: Here, are the errors of the model. The algorithm for finding the parameters tries to minimize the total error of the model defined by the cost function J: The minimization of J is achieved using an iterative algorithm called batch gradient descent: Here, a is called learning rate, and it is a trade-off between convergence speed and convergence precision. An alternative algorithm that is called stochastic gradient descent, that is loop for : The qj is updated for each training example i instead of waiting to sum over the entire training set. The last algorithm converges near the minimum of J, typically faster than batch gradient descent, but the final solution may oscillate around the real values of the parameters. The following paragraphs describe the most common model and the corresponding cost function, J. Linear regression Linear regression is the simplest algorithm and is based on the model: The cost function and update rule are: Ridge regression Ridge regression, also known as Tikhonov regularization, adds a term to the cost function J such that: , where l is the regularization parameter. The additional term has the function needed to prefer a certain set of parameters over all the possible solutions penalizing all the parameters qj different from 0. The final set of qj shrank around 0, lowering the variance of the parameters but introducing a bias error. Indicating with the superscript l the parameters from the linear regression, the ridge regression parameters are related by the following formula: This clearly shows that the larger the l value, the more the ridge parameters are shrunk around 0. Lasso regression Lasso regression is an algorithm similar to ridge regression, the only difference being that the regularization term is the sum of the absolute values of the parameters: Logistic regression Despite the name, this algorithm is used for (binary) classification problems, so we define the labels. The model is given the so-called logistic function expressed by: In this case, the cost function is defined as follows: From this, the update rule is formally the same as linear regression (but the model definition,  , is different): Note that the prediction for a point p,  , is a continuous value between 0 and 1. So usually, to estimate the class label, we have a threshold at =0.5 such that: The logistic regression algorithm is applicable to multiple label problems using the techniques one versus all or one versus one. Using the first method, a problem with K classes is solved by training K logistic regression models, each one assuming the labels of the considered class j as +1 and all the rest as 0. The second approach consists of training a model for each pair of labels (  trained models). Probabilistic interpretation of generalized linear models Now that we have seen the generalized linear model, let’s find the parameters qj that satisfy the relationship: In the case of linear regression, we can assume  as normally distributed with mean 0 and variance s2 such that the probability  is  equivalent to: Therefore, the total likelihood of the system can be expressed as follows: In the case of the logistic regression algorithm, we are assuming that the logistic function itself is the probability: Then the likelihood can be expressed by: In both cases, it can be shown that maximizing the likelihood is equivalent to minimizing the cost function, so the gradient descent will be the same. k-nearest neighbours (KNN) This is a very simple classification (or regression) method in which given a set of feature vectors  with corresponding labels yi, a test point x(t) is assigned to the label value with the majority of the label occurrences in the K nearest neighbors  found, using a distance measure such as the following: Euclidean: Manhattan: Minkowski:  (if q=2, this reduces to the Euclidean distance) In the case of regression, the value yt is calculated by replacing the majority of occurrences by the average of the labels .  The simplest average (or the majority of occurrences) has uniform weights, so each point has the same importance regardless of their actual distance from x(t). However, a weighted average with weights equal to the inverse distance from x(t) may be used. Summary In this article, the major classification and regression algorithms, together with the techniques to implement them, were discussed. You should now be able to understand in which situation each method can be used and how to implement it using Python and its libraries (sklearn and pandas). Resources for Article: Further resources on this subject: Supervised Machine Learning [article] Unsupervised Learning [article] Specialized Machine Learning Topics [article]

Managing WordPress via the web interface can be a struggle sometimes. Even Common tasks can take a lot of effort to complete, especially when they are recurring. Luckily, with the WP-CLI (WordPress Command-Line Interface), you have a command line tool to manage your WordPress installations. In this post, I will show you how to install WP-CLI, get a WordPressinstallation up and running, and manage WordPress themes and plugins. Install WP-CLI When it comes to installing WP-CLI, all you have to do is check the system requirements, download the WP-CLI archive, and make it executable. Checking system requirements To use a command-line tool like WP-CLI, you need shell access to your web server’s filesystem. This can be achieved through your terminal application on your local machine or via ssh on a remote host. Most serious hosting plans, and every server plan, will provide you with at least one SSH account. The other requirements are very basic: UNIX-like environment (OS X, Linux, FreeBSD, Cygwin) PHP 5.3.29 or later WordPress 3.7 or later Downloading the WP-CLI PHP archive WP-CLI is distributed in a PHP Archive file (.phar). So at your shell’s prompt, download the wp-cli.phar using wget or curl: $ curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar To verify the download, execute the archive with the php binary: $ phpwp-cli.phar --info If the download issuccessful, you will see an output like this: PHP binary: /usr/bin/php PHP version: 5.5.29 php.ini used: WP-CLI root dir: phar://wp-cli.phar WP-CLI packages dir: WP-CLI global config: WP-CLI project config: WP-CLI version: 0.24.0 Now you are ready to use WP-CLI to manage your WordPress installations, but to save time and typed characters, it’s a commonly used convention to call WP-CLI with the wp command. Set up the wp command for execution To avoid typing phpwp-cli.phar ... every time you want to use WP-CLI, you have to modify the archive’s file permissions and move it to somewhere in your PATH or add an alias to your shell. To make the archive executable, edit its permissions with chmod: $ chmod +x wp-cli.phar After this,move the archive to a directory somewhere within your PATH: $ sudo mv wp-cli.phar /usr/local/bin/wp If you are on shared hosting, you might have not sufficient rights to move an executable to /usr/local/bin/. In that case, you can add a shell alias to your shells configuration. Put the following line in the .bashrc or .profile file in your home directory, depending on your shell and your operating system: aliaswp='~/wp-cli.phar' To make the shell recognize the new alias, you have to reinitialize the configuration file. For example: $ source .bashrc Now you can use the wp command to manage WordPress with WP-CLI. Let’s start with some basic tasks. Managing WordPress with WP-CLI With WP-CLI, you can manage each and every aspect of your WordPress installation that can be managed through the WordPress admin area—and quite a few more. To get a glimpse of the tasks that can be done with WP-CLI, you simply type: $ wp help But let’s start with installing the WordPress core. Install a blog To geta fresh WordPress installation done, you just need your database information and a few commands. To download the latest WordPress core,simply type: $ wp core download You can also specify the version and language you want to download: $ wp core download --locale=de_DE --version=4.5.2 To create the configuration file (wp-config.php) use the core config command: $ wp core download --locale=de_DE --version=4.5.2 To finalize the installation process, you have to configure your blog’s information and the admin user data: $ wp core install --url="http://yourdomain.com" --title="My new blog" --admin_name="admin" --admin_email="admin@yourdomain.com" --admin_password="secretpassword" You can now point your browser to your WordPress admin login and start blogging. Managing WordPress themes and plugins After installing WordPress WP-CLI, you can start using it to manage your themes and plugins. Managing themes Managing themes with WP-CLI is much quicker than using the WordPress admin area. To install and activate a theme from wordpress.org, you type: $ wp theme install twentysixteen --activate You can also install a theme from a local .zip file: $ wp theme install ../mynewtheme.zip Or from aURL: $ wp theme install https://github.com/Automattic/_s/archive/master.zip To list all installed themes,type: $ wp theme list Managing plugins Managing plugins with WP-CLI is nearly the same. You use the wp plugin command instead of wp theme. When you don’t know the slug of a plugin, you can search the wordpress.org plugin repository: $ wp plugin search "yoastseo" You will get an output like this: Success: Showing 10 of 285 plugins. +-------------------------------------------+-------------------------------------+--------+ | name | slug | rating | +-------------------------------------------+-------------------------------------+--------+ | Yoast SEO | wordpress-seo | 80 | | Meta Box for Yoast SEO | meta-box-yoast-seo | 0 | | Remove Yoast SEO comments | remove-yoast-seo-comments | 0 | | ACF-Content Analysis for Yoast SEO | acf-content-analysis-for-yoast-seo | 100 | | qTranslate-X& Yoast SEO | dennisridder-qtx-seo | 0 | | SEO Advanced Custom Fields Analyzer | seo-advanced-custom-fields-analyzer | 0 | | Integration: Yoast SEO & qTranslate-X | wp-seo-qtranslate-x | 72 | | Uninstall Yoast SEO | uninstall-yoast-seo | 100 | | Remove Branding for Yoast SEO | remove-branding-for-yoast-seo | 84 | | All Meta Stats Yoast SEO Addon | all-meta-stats-yoast-seo-addon | 100 | +-------------------------------------------+-------------------------------------+--------+ Find the slug and install it afterward: $ wp plugin install wordpress-seo Then you can activate the plugin with: $ wp plugin activate wordpress-seo Deactivating a plugin is pretty much the same: $ wp plugin deactivate wordpress-seo Conclusion You should now have a basic understanding of how to manage WordPress sites with WP-CLI.You can also use WP-CLI to manage your database, do backups, keep things up to date, handle posts and comments, and maintain WordPress multi-sites. Its full strength is especially exploited when used in shell scripts, or in a more development-style app environment with version control, unit testing, and multi-stage deployment. About the author Marcel Reschke is a developer based out of Germany. He can be found on GitHub .

In this article by Simon Walkowiak author of the book Big Data Analytics with R, we will have the opportunity to learn some most important R functions from base R installation and well-known third party packages used for data crunching, transformation, and analysis. More specifically in this article you will learn to: Understand the landscape of available R data structures Be guided through a number of R operations allowing you to import data from standard and proprietary data formats Carry out essential data cleaning and processing activities such as subsetting, aggregating, creating contingency tables, and so on Inspect the data by implementing a selection of Exploratory Data Analysis techniques such as descriptive statistics Apply basic statistical methods to estimate correlation parameters between two (Pearson's r) or more variables (multiple regressions) or find the differences between means for two (t-tests) or more groups Analysis of variance (ANOVA) Be introduced to more advanced data modeling tasks like logistic and Poisson regressions (For more resources related to this topic, see here.) Learning R This book assumes that you have been previously exposed to R programming language, and this article would serve more as a revision, and an overview, of the most essential operations, rather than a very thorough handbook on R. The goal of this work is to present you with specific R applications related to Big Data and the way you can combine R with your existing Big Data analytics workflows instead of teaching you basics of data processing in R. There is a substantial number of great introductory and beginner-level books on R available at IT specialized bookstores or online, directly from Packt Publishing, and other respected publishers, as well as on the Amazon store. Some of the recommendations include the following: R in Action: Data Analysis and Graphics with R by Robert Kabacoff (2015), 2nd edition, Manning Publications R Cookbook by Paul Teetor (2011), O'Reilly Discovering Statistics Using R by Andy Field, Jeremy Miles, and Zoe Field (2012), SAGE Publications R for Data Science by Dan Toomey (2014), Packt Publishing An alternative route to the acquisition of good practical R skills is through a large number of online resources, or more traditional tutor-led in-class training courses. The first option offers you an almost limitless choice of websites, blogs, and online guides. A good starting point is the main and previously mentioned Comprehensive R Archive Network (CRAN) page (https://cran.r-project.org/), which, apart from the R core software, contains several well-maintained manuals and Task Views—community run indexes of R packages dealing with specific statistical or data management issues. R-bloggers on the other hand (http://www.r-bloggers.com/) deliver regular news on R in the form of R-related blog posts or tutorials prepared by R enthusiasts and data scientists. Other interesting online sources, which you will probably find yourself using quite often, are as follows: http://www.inside-r.org/—news and information from and by R community http://www.rdocumentation.org/—a useful search engine of R packages and functions http://blog.rstudio.org/—a blog run and edited by RStudio engineers http://www.statmethods.net/—a very informative tutorial-laden website based on the popular R book R in Action by Rob Kabacoff However, it is very likely that after some initial reading, and several months of playing with R, your most frequent destinations to seek further R-related information and obtain help on more complex use cases for specific functions will become StackOverflow(http://stackoverflow.com/) and, even better, StackExchange (http://stackexchange.com/). StackExchange is in fact a network of support and question-and-answer community-run websites, which address many problems related to statistical, mathematical, biological, and other methods or concepts, whereas StackOverflow, which is currently one of the sub-sites under the StackExchange label, focuses more on applied programming issues and provides users with coding hints and solutions in most (if not all) programming languages known to developers. Both tend to be very popular amongst R users, and as of late December 2015, there were almost 120,000 R-tagged questions asked on StackOverflow. The http://stackoverflow.com/tags/r/info page also contains numerous links and further references to free interactive R learning resources, online books and manuals and many other. Another good idea is to start your R adventure from user-friendly online training courses available through online-learning providers like Coursera (https://www.coursera.org), DataCamp (https://www.datacamp.com), edX (https://www.edx.org), or CodeSchool (https://www.codeschool.com). Of course, owing to the nature of such courses, a successful acquisition of R skills is somewhat subjective, however, in recent years, they have grown in popularity enormously, and they have also gained rather positive reviews from employers and recruiters alike. Online courses may then be very suitable, especially for those who, for various reasons, cannot attend a traditional university degree with R components, or just prefer to learn R at their own leisure or around their working hours. Before we move on to the practical part, whichever strategy you are going to use to learn R, please do not be discouraged by the first difficulties. R, like any other programming language, or should I say, like any other language (including foreign languages), needs time, patience, long hours of practice, and a large number of varied exercises to let you explore many different dimensions and complexities of its syntax and rich libraries of functions. If you are still struggling with your R skills, however, I am sure the next section will get them off the ground. Revisiting R basics In the following section we will present a short revision of the most useful and frequently applied R functions and statements. We will start from a quick R and RStudio installation guide and then proceed to creating R data structures, data manipulation, and transformation techniques, and basic methods used in the Exploratory Data Analysis (EDA). Although the R codes listed in this book have been tested extensively, as always in such cases, please make sure that your equipment is not faulty and that you will be running all the following scripts at your own risk. Getting R and RStudio ready Depending on your operating system (whether Mac OS X, Windows, or Linux) you can download and install specific base R files directly from https://cran.r-project.org/. If you prefer to use RStudio IDE you still need to install R core available from CRAN website first and then download and run installers of the most recent version of RStudio IDE specific for your platform from https://www.rstudio.com/products/rstudio/download/. Personally I prefer to use RStudio, owing to its practical add-ons such as code highlighting and more user-friendly GUI, however, there is no particular reason why you can't use just the simple R core installation if you want to. Having said that, in this book we will be using RStudio in most of the examples. All code snippets have been executed and run on a MacBook Pro laptop with Mac OS X (Yosemite) operating system, 2.3 GHz Intel Core i5 processor, 1TB solid-state hard drive and 16GB of RAM memory, but you should also be fine with a much weaker configuration. In this article we won't be using any large data, and even in the remaining parts of this book the data sets used are limited to approximately 100MB to 130MB in size each. You are also provided with links and references to full Big Data whenever possible. If you would like to follow the practical parts of this book you are advised to download and unzip the R code and data for each article from the web page created for this book by Packt Publishing. If you use this book in PDF format it is not advisable to copy the code and paste it into the R console. When printed, some characters (like quotation marks " ") may be encoded differently than in R and the execution of such commands may result in errors being returned by the R console. Once you have downloaded both R core and RStudio installation files, follow the on-screen instructions for each installer. When you have finished installing them, open your RStudio software. Upon initialization of the RStudio you should see its GUI with a number of windows distributed on the screen. The largest one is the console in which you input and execute the code, line by line. You can also invoke the editor panel (it is recommended) by clicking on the white empty file icon in the top left corner of the RStudio software or alternatively by navigating to File | New File | R Script. If you have downloaded the R code from the book page of the Packt Publishing website, you may also just click on the Open an existing file (Ctrl + O) (a yellow open folder icon) and locate the downloaded R code on your computer's hard drive (or navigate to File | Open File…). Now your RStudio session is open and we can adjust some most essential settings. First, you need to set your working directory to the location on your hard drive where your data files are. If you know the specific location you can just type the setwd() command with a full and exact path to the location of your data as follows: > setwd("/Users/simonwalkowiak/Desktop/data") Of course your actual path will differ from mine, shown in the preceding code, however please mind that if you copy the path from the Windows Explorer address bar you will need to change the backslashes to forward slashes / (or to double backslashes \). Also, the path needs to be kept within the quotation marks "…". Alternatively you can set your working directory by navigating to Session | Set Working Directory | Choose Directory… to manually select the folder in which you store the data for this session. Apart from the ones we have already described, there are other ways to set your working directory correctly. In fact most of the operations, and even more complex data analysis and processing activities, can be achieved in R in numerous ways. For obvious reasons, we won't be presenting all of them, but we will just focus on the frequently used methods and some tips and hints applicable to special or difficult scenarios. You can check whether your working directory has been set correctly by invoking the following line: > getwd() [1] "/Users/simonwalkowiak/Desktop/data" From what you can see, the getwd() function returned the correct destination for my previously defined working directory. Setting the URLs to R repositories It is always good practice to check whether your R repositories are set correctly. R repositories are servers located at various institutes and organizations around the world, which store recent updates and new versions of third-party R packages. It is recommended that you set the URL of your default repository to the CRAN server and choose a mirror that is located relatively close to you. To set the repositories you may use the following code: > setRepositories(addURLs = c(CRAN = "https://cran.r-project.org/")) You can check your current, or default, repository URLs by invoking the following function: > getOption("repos") The output will confirm your URL selection:               CRAN "https://cran.r-project.org/" You will be able to choose specific mirrors when you install a new package for the first time during the session, or you may navigate to Tools | Global Options… | Packages. In the Package management section of the window you can alter the default CRAN mirror location—click on Change… button to adjust. Once your repository URLs and working directory are set, you can go on to create data structures that are typical for R programming language. R data structures The concept of data structures in various programming languages is extremely important and cannot be overlooked. Similarly in R, available data structures allow you to hold any type of data and use them for further processing and analysis. The kind of data structure which you use, puts certain constraints on how you can access and process data stored in this structure, and what manipulation techniques you can use. This section will briefly guide you through a number of basic data structures available in R language. Vectors Whenever I teach statistical computing courses, I always start by introducing R learners to vectors as the first data structure they should get familiar with. Vectors are one-dimensional structures that can hold any type of data that is numeric, character, or logical. In simple terms, a vector is a sequence of some sort of values (for example numeric, character, logical, and many more) of specified length. The most important thing that you need to remember is that an atomic vector may contain only one type of data. Let's then create a vector with 10 random deviates from a standard normal distribution, and store all its elements in an object which we will call vector1. In your RStudio console (or its editor) type the following: > vector1 <- rnorm(10) Let's now see the contents of our newly created vector1: > vector1 [1] -0.37758383 -2.30857701 2.97803059 -0.03848892 1.38250714 [6] 0.13337065 -0.51647388 -0.81756661 0.75457226 -0.01954176 As we drew random values, your vector most likely contains different elements to the ones shown in the preceding example. Let's then make sure that my new vector (vector2) is the same as yours. In order to do this we need to set a seed from which we will be drawing the values: > set.seed(123) > vector2 <- rnorm(10, mean=3, sd=2) > vector2 [1] 1.8790487 2.5396450 6.1174166 3.1410168 3.2585755 6.4301300 [7] 3.9218324 0.4698775 1.6262943 2.1086761 In the preceding code we've set the seed to an arbitrary number (123) in order to allow you to replicate the values of elements stored in vector2 and we've also used some optional parameters of the rnorm() function, which enabled us to specify two characteristics of our data, that is the arithmetic mean (set to 3) and standard deviation (set to 2). If you wish to inspect all available arguments of the rnorm() function, its default settings, and examples of how to use it in practice, type ?rnorm to view help and information on that specific function. However, probably the most common way in which you will be creating a vector of data is by using the c() function (c stands for concatenate) and then explicitly passing the values of each element of the vector: > vector3 <- c(6, 8, 7, 1, 2, 3, 9, 6, 7, 6) > vector3 [1] 6 8 7 1 2 3 9 6 7 6 In the preceding example we've created vector3 with 10 numeric elements. You can use the length() function of any data structure to inspect the number of elements: > length(vector3) [1] 10 The class() and mode() functions allow you to determine how to handle the elements of vector3 and how the data are stored in vector3 respectively. > class(vector3) [1] "numeric" > mode(vector3) [1] "numeric" The subtle difference between both functions becomes clearer if we create a vector that holds levels of categorical variable (known as a factor in R) with character values: > vector4 <- c("poor", "good", "good", "average", "average", "good", "poor", "good", "average", "good") > vector4 [1] "poor" "good" "good" "average" "average" "good" "poor" [8] "good" "average" "good" > class(vector4) [1] "character" > mode(vector4) [1] "character" > levels(vector4) NULL In the preceding example, both the class() and mode() outputs of our character vector are the same, as we still haven't set it to be treated as a categorical variable, and we haven't defined its levels (the contents of the levels() function is empty—NULL). In the following code we will explicitly set the vector to be recognized as categorical with three levels: > vector4 <- factor(vector4, levels = c("poor", "average", "good")) > vector4 [1] poor good good average average good poor good [8] average good Levels: poor average good The sequence of levels doesn't imply that our vector is ordered. We can order the levels of factors in R using the ordered() command. For example, you may want to arrange the levels of vector4 in reverse order, starting from "good": > vector4.ord <- ordered(vector4, levels = c("good", "average", "poor")) > vector4.ord [1] poor good good average average good poor good [8] average good Levels: good < average < poor You can see from the output that R has now properly recognized the order of our levels, which we had defined. We can now apply class() and mode() functions on the vector4.ord object: > class(vector4.ord) [1] "ordered" "factor" > mode(vector4.ord) [1] "numeric" You may very likely be wondering why the mode() function returned "numeric" type instead of "character". The answer is simple. By setting the levels of our factor, R has assigned values 1, 2, and 3 to "good", "average" and "poor" respectively, exactly in the same order as we had defined them in the ordered() function. You can check this using levels() and str() functions: > levels(vector4.ord) [1] "good" "average" "poor" > str(vector4.ord) Ord.factor w/ 3 levels "good"<"average"<..: 3 1 1 2 2 1 3 1 2 1 Just to finalize the subject of vectors, let's create a logical vector, which contains only TRUE and FALSE values: > vector5 <- c(TRUE, FALSE, TRUE, FALSE, FALSE, FALSE, TRUE, FALSE, FALSE, FALSE) > vector5 [1] TRUE FALSE TRUE FALSE FALSE FALSE TRUE FALSE FALSE FALSE Similarly, for all other vectors already presented, feel free to check their structure, class, mode, and length using appropriate functions shown in this section. What outputs did those commands return? Scalars The reason why I always start from vectors is that scalars just seem trivial when they follow vectors. To simplify things even more, think of scalars as one-element vectors which are traditionally used to hold some constant values for example: > a1 <- 5 > a1 [1] 5 Of course you may use scalars in computations and also assign any one-element outputs of mathematical or statistical operations to another, arbitrary named scalar for example: > a2 <- 4 > a3 <- a1 + a2 > a3 [1] 9 In order to complete this short subsection on scalars, create two separate scalars which will hold a character and a logical value. Matrices A matrix is a two-dimensional R data structure in which each of its elements must be of the same type; that is numeric, character, or logical. As matrices consist of rows and columns, their shape resembles tables. In fact, when creating a matrix, you can specify how you want to distribute values across its rows and columns for example: > y <- matrix(1:20, nrow=5, ncol=4) > y [,1] [,2] [,3] [,4] [1,] 1 6 11 16 [2,] 2 7 12 17 [3,] 3 8 13 18 [4,] 4 9 14 19 [5,] 5 10 15 20 In the preceding example we have allocated a sequence of 20 values (from 1 to 20) into five rows and four columns, and by default they have been distributed by column. We may now create another matrix in which we will distribute the values by rows and give names to rows and columns using the dimnames argument (dimnames stands for names of dimensions) in the matrix() function: > rows <- c("R1", "R2", "R3", "R4", "R5") > columns <- c("C1", "C2", "C3", "C4") > z <- matrix(1:20, nrow=5, ncol=4, byrow=TRUE, dimnames=list(rows, columns)) > z C1 C2 C3 C4 R1 1 2 3 4 R2 5 6 7 8 R3 9 10 11 12 R4 13 14 15 16 R5 17 18 19 20 As we are talking about matrices it's hard not to mention anything about how to extract specific elements stored in a matrix. This skill will actually turn out to be very useful when we get to subsetting real data sets. Looking at the matrix y, for which we didn't define any names of its rows and columns, notice how R denotes them. The row numbers come in the format [r, ], where r is a consecutive number of a row, whereas the column are identified by [ ,c], where c is a consecutive number of a column. If you then wished to extract a value stored in the fourth row of the second column of our matrix y, you could use the following code to do so: > y[4,2] [1] 9 In case you wanted to extract the whole column number three from our matrix y, you could type the following: > y[,3] [1] 11 12 13 14 15 As you can see, we don't even need to allow an empty space before the comma in order for this short script to work. Let's now imagine you would like to extract three values stored in the second, third and fifth rows of the first column in our vector z with named rows and columns. In this case, you may still want to use the previously shown notation, you do not need to refer explicitly to the names of dimensions of our matrix z. Additionally, notice that for several values to extract we have to specify their row locations as a vector—hence we will put their row coordinates inside the c() function which we had previously used to create vectors: > z[c(2, 3, 5), 1] R2 R3 R5 5 9 17 Similar rules of extracting data will apply to other data structures in R such as arrays, lists, and data frames, which we are going to present next. Arrays Arrays are very similar to matrices with only one exception: they contain more dimensions. However, just like matrices or vectors, they may only hold one type of data. In R language, arrays are created using the array() function: > array1 <- array(1:20, dim=c(2,2,5)) > array1 , , 1 [,1] [,2] [1,] 1 3 [2,] 2 4 , , 2 [,1] [,2] [1,] 5 7 [2,] 6 8 , , 3 [,1] [,2] [1,] 9 11 [2,] 10 12 , , 4 [,1] [,2] [1,] 13 15 [2,] 14 16 , , 5 [,1] [,2] [1,] 17 19 [2,] 18 20 The dim argument, which was used within the array() function, specifies how many dimensions you want to distribute your data across. As we had 20 values (from 1 to 20) we had to make sure that our array can hold all 20 elements, therefore we decided to assign them into two rows, two columns, and five dimensions (2 x 2 x 5 = 20). You can check dimensionality of your multi-dimensional R objects with dim() command: > dim(array1) [1] 2 2 5 As with matrices, you can use standard rules for extracting specific elements from your arrays. The only difference is that now you have additional dimensions to take care of. Let's assume you would want to extract a specific value located in the second row of the first column in the fourth dimension of our array1: > array1[2, 1, 4] [1] 14 Also, if you need to find a location of a specific value, for example 11, within the array, you can simply type the following line: > which(array1==11, arr.ind=TRUE) dim1 dim2 dim3 [1,] 1 2 3 Here, the which() function returns indices of the array (arr.ind=TRUE), where the sought value equals 11 (hence ==). As we had only one instance of value 11 in our array, there is only one row specifying its location in the output. If we had more instances of 11, additional rows would be returned indicating indices for each element equal to 11. Data frames The following two short subsections concern two of probably the most widely used R data structures. Data frames are very similar to matrices, but they may contain different types of data. Here you might have suddenly thought of a typical rectangular data set with rows and columns or observations and variables. In fact you are correct. Most of the data sets are indeed imported into R as data frames. You can also create a simple data frame manually with the data.frame() function, but as each column in the data frame may be of a different type, we must first create vectors which will hold data for specific columns: > subjectID <- c(1:10) > age <- c(37,23,42,25,22,25,48,19,22,38) > gender <- c("male", "male", "male", "male", "male", "female", "female", "female", "female", "female") > lifesat <- c(9,7,8,10,4,10,8,7,8,9) > health <- c("good", "average", "average", "good", "poor", "average", "good", "poor", "average", "good") > paid <- c(T, F, F, T, T, T, F, F, F, T) > dataset <- data.frame(subjectID, age, gender, lifesat, health, paid) > dataset subjectID age gender lifesat health paid 1 1 37 male 9 good TRUE 2 2 23 male 7 average FALSE 3 3 42 male 8 average FALSE 4 4 25 male 10 good TRUE 5 5 22 male 4 poor TRUE 6 6 25 female 10 average TRUE 7 7 48 female 8 good FALSE 8 8 19 female 7 poor FALSE 9 9 22 female 8 average FALSE 10 10 38 female 9 good TRUE The preceding example presents a simple data frame which contains some dummy imaginary data, possibly a sample from a basic psychological experiment, which measured subjects' life satisfaction (lifesat) and their health status (health) and also collected other socio-demographic information such as age and gender, and whether the participant was a paid subject or a volunteer. As we deal with various types of data, the elements for each column had to be amalgamated into a single structure of a data frame using the data.frame() command, and specifying the names of objects (vectors) in which we stored all values. You can inspect the structure of this data frame with the previously mentioned str() function: > str(dataset) 'data.frame': 10 obs. of 6 variables: $ subjectID: int 1 2 3 4 5 6 7 8 9 10 $ age : num 37 23 42 25 22 25 48 19 22 38 $ gender : Factor w/ 2 levels "female","male": 2 2 2 2 2 1 1 1 1 1 $ lifesat : num 9 7 8 10 4 10 8 7 8 9 $ health : Factor w/ 3 levels "average","good",..: 2 1 1 2 3 1 2 3 1 2 $ paid : logi TRUE FALSE FALSE TRUE TRUE TRUE ... The output of str() gives you some basic insights into the shape and format of your data in the dataset object, for example, number of observations and variables, names of variables, types of data they hold, and examples of values for each variable. While discussing data frames, it may also be useful to introduce you to another way of creating subsets. As presented earlier, you may apply standard extraction rules to subset data of your interest. For example, suppose you want to print only those columns which contain age, gender, and life satisfaction information from our dataset data frame. You may use the following two alternatives (the output not shown to save space, but feel free to run it): > dataset[,2:4] #or > dataset[, c("age", "gender", "lifesat")] Both lines of code will produce exactly the same results. The subset() function however gives you additional capabilities of defining conditional statements which will filter the data, based on the output of logical operators. You can replicate the preceding output using subset() in the following way: > subset(dataset[c("age", "gender", "lifesat")]) Assume now that you want to create a subset with all subjects who are over 30 years old, and with a score of greater than or equal to eight on the life satisfaction scale (lifesat). The subset() function comes very handy: > subset(dataset, age > 30 & lifesat >= 8) subjectID age gender lifesat health paid 1 1 37 male 9 good TRUE 3 3 42 male 8 average FALSE 7 7 48 female 8 good FALSE 10 10 38 female 9 good TRUE Or you want to produce an output with two socio-demographic variables of age and gender, of only these subjects who were paid to participate in this experiment: > subset(dataset, paid==TRUE, select=c("age", "gender")) age gender 1 37 male 4 25 male 5 22 male 6 25 female 10 38 female We will perform much more thorough and complex data transformations on real data frames in the second part of this article. Lists A list in R is a data structure, which is a collection of other objects. For example, in the list you can store vectors, scalars, matrices, arrays, data frames, and even other lists. In fact, lists in R are vectors, but they differ from atomic vectors, which we introduced earlier in this section as lists that can hold many different types of data. In the following example, we will construct a simple list (using list() function) which will include a variety of other data structures: > simple.vector1 <- c(1, 29, 21, 3, 4, 55) > simple.matrix <- matrix(1:24, nrow=4, ncol=6, byrow=TRUE) > simple.scalar1 <- 5 > simple.scalar2 <- "The List" > simple.vector2 <- c("easy", "moderate", "difficult") > simple.list <- list(name=simple.scalar2, matrix=simple.matrix, vector=simple.vector1, scalar=simple.scalar1, difficulty=simple.vector2) >simple.list $name [1] "The List" $matrix [,1] [,2] [,3] [,4] [,5] [,6] [1,] 1 2 3 4 5 6 [2,] 7 8 9 10 11 12 [3,] 13 14 15 16 17 18 [4,] 19 20 21 22 23 24 $vector [1] 1 29 21 3 4 55 $scalar [1] 5 $difficulty [1] "easy" "moderate" "difficult" > str(simple.list) List of 5 $ name : chr "The List" $ matrix : int [1:4, 1:6] 1 7 13 19 2 8 14 20 3 9 ... $ vector : num [1:6] 1 29 21 3 4 55 $ scalar : num 5 $ difficulty: chr [1:3] "easy" "moderate" "difficult" Looking at the preceding output, you can see that we have assigned names to each component in our list and the str() function prints them as if they were variables of a standard rectangular data set. In order to extract specific elements from a list, you first need to use a double square bracket notation [[x]] to identify a component x within the list. For example, assuming you want to print an element stored in its first row and the third column of the second component you may use the following line in R: > simple.list[[2]][1,3] [1] 3 Owing to their flexibility, lists are commonly used as preferred data structures in the outputs of statistical functions. It is then important for you to know how you can deal with lists and what sort of methods you can apply to extract and process data stored in them. Once you are familiar with the basic features of data structures available in R, you may wish to visit Hadley Wickham's online book at http://adv-r.had.co.nz/ in which he explains various more advanced concepts related to each native data structure in R language, and different techniques of subsetting data, depending on the way they are stored. Exporting R data objects In the previous section we created numerous objects, which you can inspect in the Environment tab window in RStudio. Alternatively, you may use the ls() function to list all objects stored in your global environment: > ls() If you've followed the article along, and run the script for this book line-by-line, the output of the ls() function should hopefully return 27 objects: [1] "a1" "a2" "a3" [4] "age" "array1" "columns" [7] "dataset" "gender" "health" [10] "lifesat" "paid" "rows" [13] "simple.list" "simple.matrix" "simple.scalar1" [16] "simple.scalar2" "simple.vector1" "simple.vector2" [19] "subjectID" "vector1" "vector2" [22] "vector3" "vector4" "vector4.ord" [25] "vector5" "y" "z" In this section we will present various methods of saving the created objects to your local drive and exporting their contents to a number of the most commonly used file formats. Sometimes, for various reasons, it may happen that you need to leave your project and exit RStudio or shut your PC down. If you do not save your created objects, you will lose all of them, the moment you close RStudio. Remember that R stores created data objects in the RAM of your machine, and whenever these objects are not in use any longer, R frees them from the memory, which simply means that they get deleted. Of course this might turn out to be quite costly, especially if you had not saved your original R script, which would have enabled you to replicate all the steps of your data processing activities when you start a new session in R. In order to prevent the objects from being deleted, you can save all or selected ones as .RData files on your hard drive. In the first case, you may use the save.image() function which saves your whole current workspace with all objects to your current working directory: > save.image(file = "workspace.RData") If you are dealing with large objects, first make sure you have enough storage space available on your drive (this is normally not a problem any longer), or alternatively you can reduce the size of the saved objects using one of the compression methods available. For example, the above workspace.RData file was 3,751 bytes in size without compression, but when xz compression was applied the size of the resulting file decreased to 3,568 bytes. > save.image(file = "workspace2.RData", compress = "xz") Of course, the difference in sizes in the presented example is minuscule, as we are dealing with very small objects, however it gets much more significant for bigger data structures. The trade-off of applying one of the compression methods is the time it takes for R to save and load .RData files. If you prefer to save only chosen objects (for example dataset data frame and simple.list list) you can achieve this with the save() function: > save(dataset, simple.list, file = "two_objects.RData") You may now test whether the above solutions worked by cleaning your global environment of all objects, and then loading one of the created files, for example: > rm(list=ls()) > load("workspace2.RData") As an additional exercise, feel free to explore other functions which allow you to write text representations of R objects, for example dump() or dput(). More specifically, run the following commands and compare the returned outputs: > dump(ls(), file = "dump.R", append = FALSE) > dput(dataset, file = "dput.txt") The save.image() and save() functions only create images of your workspace or selected objects on the hard drive. It is an entirely different story if you want to export some of the objects to data files of specified formats, for example, comma-separated, tab-delimited, or proprietary formats like Microsoft Excel, SPSS, or Stata. The easiest way to export R objects to generic file formats like CSV, TXT, or TAB is through the cat() function, but it only works on atomic vectors: > cat(age, file="age.txt", sep=",", fill=TRUE, labels=NULL, append=TRUE) > cat(age, file="age.csv", sep=",", fill=TRUE, labels=NULL, append=TRUE) The preceding code creates two files, one as a text file and another one as a comma-separated format, both of which contain values from the age vector that we had previously created for the dataset data frame. The sep argument is a character vector of strings to append after each element, the fill option is a logical argument which controls whether the output is automatically broken into lines (if set to TRUE), the labels parameter allows you to add a character vector of labels for each printed line of data in the file, and the append logical argument enables you to append the output of the call to the already existing file with the same name. In order to export vectors and matrices to TXT, CSV, or TAB formats you can use the write() function, which writes out a matrix or a vector in a specified number of columns for example: > write(age, file="agedata.csv", ncolumns=2, append=TRUE, sep=",") > write(y, file="matrix_y.tab", ncolumns=2, append=FALSE, sep="t") Another method of exporting matrices provides the MASS package (make sure to install it with the install.packages("MASS") function) through the write.matrix() command: > library(MASS) > write.matrix(y, file="ymatrix.txt", sep=",") For large matrices, the write.matrix() function allows users to specify the size of blocks in which the data are written through the blocksize argument. Probably the most common R data structure that you are going to export to different file formats will be a data frame. The generic write.table() function gives you an option to save your processed data frame objects to standard data formats for example TAB, TXT, or CSV: > write.table(dataset, file="dataset1.txt", append=TRUE, sep=",", na="NA", col.names=TRUE, row.names=FALSE, dec=".") The append and sep arguments should already be clear to you as they were explained earlier. In the na option you may specify an arbitrary string to use for missing values in the data. The logical parameter col.names allows users to append the names of columns to the output file, and the dec parameter sets the string used for decimal points and must be a single character. In the example, we used row.names set to FALSE, as the names of the rows in the data are the same as the values of the subjectID column. However, it is very likely that in other data sets the ID variable may differ from the names (or numbers) of rows, so you may want to control it depending on the characteristics of your data. Two similar functions write.csv() and write.csv2() are just convenience wrappers for saving CSV files, and they only differ from the generic write.table() function by default settings of some of their parameters, for example sep and dec. Feel free to explore these subtle differences at your leisure. To complete this section of the article we need to present how to export your R data frames to third-party formats. Amongst several frequently used methods, at least four of them are worth mentioning here. First, if you wish to write a data frame to a proprietary Microsoft Excel format, such as XLS or XLSX, you should probably use the WriteXLS package (please use install.packages("WriteXLS") if you have not done it yet) and its WriteXLS() function: > library(WriteXLS) > WriteXLS("dataset", "dataset1.xlsx", SheetNames=NULL, row.names=FALSE, col.names=TRUE, AdjWidth=TRUE, envir=parent.frame()) The WriteXLS() command offers users a number of interesting options, for instance you can set the names of the worksheets (SheetNames argument), adjust the widths of columns depending on the number of characters of the longest value (AdjWidth), or even freeze rows and columns just as you do it in Excel (FreezeRow and FreezeCol parameters). Please note that in order for the WriteXLS package to work, you need to have Perl installed on your machine. The package creates Excel files using Perl scripts called WriteXLS.pl for Excel 2003 (XLS) files, and WriteXLSX.pl for Excel 2007 and later version (XLSX) files. If Perl is not present on your system, please make sure to download and install it from https://www.perl.org/get.html. After the Perl installation, you may have to restart your R session and load the WriteXLS package again to apply the changes. For solutions to common Perl issues please visit the following websites: https://www.perl.org/docs.html, http://www.ahinea.com/en/tech/perl-unicode-struggle.html, and http://www.perl.com/pub/2012/04/perlunicook-standard-preamble.html or search StackOverflow and similar websites for R and Perl related specific problems. Another very useful way of writing R objects to the XLSX format is provided by the openxlsx package through the write.xlsx() function, which, apart from data frames, also allows lists to be easily written to Excel spreadsheets. Please note that Windows users may need to install the Rtools package in order to use openxlsx functionalities. The write.xlsx() function gives you a large choice of possible options to set, including a custom style to apply to column names (through headerStyle argument), the color of cell borders (borderColour), or even its line style (borderStyle). The following example utilizes only the most common and minimal arguments required to write a list to the XLSX file, but be encouraged to explore other options offered by this very flexible function: > write.xlsx(simple.list, file = "simple_list.xlsx") A third-party package called foreign makes it possible to write data frames to other formats used by well-known statistical tools such as SPSS, Stata, or SAS. When creating files, the write.foreign() function requires users to specify the names of both the data and code files. Data files hold raw data, whereas code files contain scripts with the data structure and metadata (value and variable labels, variable formats, and so on) written in the proprietary syntax. In the following example, the code writes the dataset data frame to the SPSS format: > library(foreign) > write.foreign(dataset, "datafile.txt", "codefile.txt", package="SPSS") Finally, another package called rio contains only three functions, allowing users to quickly import(), export() and convert() data between a large array of file formats, (for example TSV, CSV, RDS, RData, JSON, DTA, SAV, and many more). The package, in fact, is dependent on a number of other R libraries, some of which, for example foreign and openxlsx, have already been presented in this article. The rio package does not introduce any new functionalities apart from the default arguments characteristic for underlying export functions, so you still need to be familiar with the original functions and their parameters if you require more advanced exporting capabilities. But, if you are only looking for a no-fuss general export function, the rio package is definitely a good shortcut to take: > export(dataset, format = "stata") > export(dataset, "dataset1.csv", col.names = TRUE, na = "NA") Summary In this article, we have provided you with quite a bit of theory, and hopefully a lot of practical examples of data structures available to R users. You've created several objects of different types, and you've become familiar with a variety of data and file formats to offer. We then showed you how to save R objects held in your R workspace to external files on your hard drive, or to export them to various standard and proprietary file formats. Resources for Article: Further resources on this subject: Fast Data Manipulation with R [article] The Data Science Venn Diagram [article] Deployment and DevOps [article]

Have you tried using React and now you want to increase your abilities? Are you ready to scale up your small React app? Have you wondered how to offload all of your state into a single place and keep your components more modular? Using Redux with React allows you to have a single source of truth for all of your app's state. The two of them together allow you to never have to set state on a component and allows your components to be completely reusable. For some added sugar, you'll also learn how to leverage Firebase and use Redux actions to subscribe to data that is updated in real time. In this two-part post, you'll walk through creating a new chat app called Yak using React's new CLI, integrating Redux into your app, updating your state and connecting it all to Firebase. Let's get started. Setting up This post is written with the assumption that you have Node.js and NPM already installed. Also, it assumes some knowledge of JavaScript and React.js. If you don't already have Node.js and NPM installed, head over to the Node.js install instructions. At the time of writing this post, the Node.js version is 6.6.0 and NPM version is 3.10.8. Once you have Node.js installed, open up your favorite terminal app and install the NPM package Create React App; the current version at the time of writing this post is 0.6.0, so make sure to specify that version. [~]$ npm install -g create-react-app@0.6.0 Now we'll want to set up our app and install our dependencies. First we'll navigate to where we want our project to live. I like to keep my projects at ~/code, so I'll navigate there. You may need to create the directory using mkdir if you don't have it, or you might want to store it elsewhere. It doesn't matter which you choose; just head to where you want to store your project. [~]$ cd ~/code Once there, use Create React App to create the app: [~/code]$ create-react-app yak This command is going to create a directory called yak and create all the necessary files you need in order to start a baseline React.js app. Once the command has completed you should see some commands that you can run in your new app. Create React App has created the boilerplate files for you. Take a moment to familiarize yourself with these files. .gitignore All the files and directories you want ignored by git. README.md Documentation on what has been created. This is a good resource to lean on as you're learning React.js and using your app. node_modules All the packages that are required to run and build the application up to this point. package.json Instructs NPM how scripts run on your app, which packages your app depending on and other meta things such as version and app name. public All the static files that aren't used within the app. Mainly for index.html and favicon.ico. src All the app files; the app is run by Webpack and is set up to watch all the files inside of this directory. This is where you will spend the majority of your time. There are two files that cannot be moved while working on the app; they are public/index.html and src/index.js. The app relies on these two files in order to run. You can change them, but don't move them. Now to get started, navigate into the app folder and start the app. [~/code]$ cd yak [~/code/yak]$ npm start The app should start and automatically open http://localhost:3000/ in your default browser. You should see a black banner with the React.js logo spinning and some instructions on how to get started. To stop the app, press ctrl-c in the terminal window that is running the app. Getting started with Redux Next install Redux and React-Redux: [~/code/yak]$ npm install --save redux react-redux Redux will allow the app to have a single source of truth for state throughout the app. The idea is to keep all the React components ignorant of the state, and to pass that state to them via props. Containers will be used to select data from the state and pass the data to the ignorant components via props. React-Redux is a utility that assists in integrating React with Redux. Redux's state is read-only and you can only change the state by emitting an action that a reducer function uses to take the previous state and return a new state. Make sure as you are writing your reducers to never mutate the state (more on that later). Now you will add Redux to your app, in src/index.js. Just below importing ReactDOM, add: import { createStore, compose } from 'redux'; import { Provider } from 'react-redux'; You now have the necessary functions and components to set up your Redux store and pass it to your React app. Go ahead and get your store initialized. After the last import statement and before ReactDOM.render() is where you will create your store. const store = createStore(); Yikes! If you run the app and open your inspector, you should see the following console error: Uncaught Error: Expected the reducer to be a function. That error is thrown because the createStore function requires a reducer as the first parameter. The second parameter is an optional initial state and the last parameter is for middleware that you may want for your store. Go ahead and create a reducer for your store, and ignore the other two parameters for now. [~/code/yak]$ touch src/reducer.js Now open reducer.js and add the following code to the reducer: const initialState = { messages: [] }; export function yakApp(state = initialState, action) { return state; } Here you have created an initial state for the current reducer, and a function that is either accepting a new state or using ES6 default arguments to set an undefined state to the initial state. The function is simply returning the state and not making any changes for now. This is a perfectly valid reducer and will work to solve the console error and get the app running again. Now it's time to add it to the store. Back in src/index.js, import the reducer in and then set the yakApp function to your store. import { yakApp } from './reducer'; const store = createStore(yakApp); Restart the app and you'll see that it is now working again! One last thing to get things set up in your bootstrapping file src/index.js. You have your store and have imported Provider; now it’s time to connect the two and allow the app to have access to the store. Update the ReactDOM.render method to look like the following. ReactDOM.render( <Provider store={store}> <App /> </Provider>, document.getElementById('root') ); Now you can jump into App.js and connect your store. In App.js, add the following import statement: import { connect } from 'react-redux'; At the bottom of the file, just before the export statement, add: function mapStateToProps(state) { return { messages: state.messages } } And change the export statement to be: export default connect(mapStateToProps)(App); That's it! Your App component is now connected to the redux store. The messages array is being mapped to this.props. Go ahead and try it; add a console log to the render() method just before the return statement. console.log(this.props.messages); The console should log an empty array. This is great! Conclusion In this post, you've learned to create a new React app without having to worry about tooling. You've integrated Redux into the app and created a simple reducer. You've also connected the reducer to your React component. But how do you add data to the array as messages are sent? How do you persist the array of messages after you leave the app? How do you connect all of this to your UI? How do you allow your users to create glorious data for you? In the next post, you'll learn to do all those things. Stay tuned! About the author AJ Webb is a team lead and frontend engineer for @tannerlabs, and the co-creator of Payba.cc.

In this article by Martin Wood, the author of the book Mastering ServiceNow Second Edition, we will see how data is stored, manipulated, processed, and displayed. With these tools, you can create almost any forms-based application. But building from the foundations up each time would be time consuming and repetitive. To help with this, the ServiceNow platform provides baseline functionality that allows you to concentrate on the parts that matter. (For more resources related to this topic, see here.) If Business Rules, tables, Client Scripts, and fields are the foundations of ServiceNow, the Task table, approvals, and the service catalog are the readymade lintels, elevator shafts, and staircases—the essential, tried and tested components that make up the bulk of the building. This article looks at the standard components behind many applications: The Task table is probably the most frequently used and important table in a ServiceNow instance. The functionality it provides is explored in this article, and several gotchas are outlined. How do you control these tasks? Using business rules is one way, but Graphical Workflow provides a drag-and-drop option to control your application. While you can bake in rules, you often need personal judgment. Approval workflows lets you decide whom to ask and lets them respond easily. The Service Catalog application is the go-to place to work with the applications that are hosted in ServiceNow. It provides the main interface for end users to interact with your applications. We will also briefly explore request fulfillment, which enables users to respond to queries quickly and effectively. Service Level Management lets you monitor the effectiveness of your services by setting timers and controlling breaches. Introducing tasks ServiceNow is a forms-based workflow platform. The majority of applications running on ServiceNow can be reduced to a single, simple concept: the management of tasks. A task in ServiceNow is work that is assigned to someone. You may ask your colleague to make you a cup of tea, or you may need to fix a leaking tap in a hotel guest's bedroom. Both of these are tasks. There are several parts to each of these tasks: A requester is someone who specifies the work. This could be a guest or even yourself. A fulfiller is someone who completes the work. This may, less frequently, be the same person as the requester. The fulfiller is often part of a group of people. Perhaps someone among them could work on the task. Information about the task itself is included—perhaps a description or a priority, indicating how important the task is. The status of the task—is it complete? Or is the fulfiller still working on it? There is a place to store notes to record what has happened. An identifier is a unique number to represent the task. The sys_id parameter is an identifier that is very specific and unique, but not very friendly! Links, references, and relationships to other records are present. Is this task a subdivision of another task, or is it connected to others? Perhaps you are moving house—that's a big job! But this could be broken down into separate individual tasks. Sometimes, a task may be as a simple as the equivalent of a Post-it note. Many of us have had something similar to "Frank called, could you ring him back?" attached to our desk. But you often need something that's more permanent, reportable, automated, and doesn't fall to the floor when someone walks by. Looking at the Task table The Task table in ServiceNow is designed to store, manage, and process tasks. It contains fields to capture all the details and a way to access them consistently and reliably. In addition, there is a whole host of functionality described in this article for automating and processing tasks more efficiently. The Product Documentation has an introductory article to the Task table: https://docs.servicenow.com/bundle/helsinki-servicenow-platform/page/administer/task-table/concept/c_TaskTable.html. It also covers some of the lesser-used elements, such as the task interceptor. To begin our journey, let's inspect the record representing the Task table. Navigate to System Definition > Tables, and then find the entry labelled Task.In the Helsinki version of ServiceNow, there are 65 fields in the Task table. There are also many other associated scripts, UI Actions, and linked tables. What do they all do? The Task table is designed to be extended, so you can of course add your own fields to capture the information you want. We’ll do just that in a later example. The important fields It is often instructive to view the fields that have been placed in the form by default. Click on the Show Form related link to take a look. Number: A unique identifier, it's a seven-digit number prefixed by TASK. This is constructed using a script specified in the dictionary entry. The script uses details in the Numbers [sys_number] table, accessible via System Definition > Number Maintenance. Assigned to: This field represents the fulfiller—the person who is working on the task. It is a reference field that points to the User table. The Assigned to field is also dependent on the Assignment group field. This means that if the Assignment group field is populated, you can only select users that belong to that particular group. Assignment group: This field is not in the Task form by default. You would typically want to add it using the Form Designer. Groups and users are discussed further in the article, but in short, it shows the team of people responsible for the task.Assignment group has been made a tree picker field. The Group table has a parent field, which allows groups to be nested in a hierarchical structure. If you click on the Reference Lookup icon (the magnifying glass), it will present a different interface to the usual list. This is controlled via an attribute in the dictionary. The following screenshot shows how an hierarchical group structure would be displayed using the tree picker: Active: This field represents whether a task is "operational". Closed tickets are not active, nor are tasks that are due to start. Tasks that are being worked on are active. There is a direct correlation between the state of a task and whether it is active. If you change the choices available for the State field, you may be tempted to write business rules to control the Active flag. Don't. There is a script called TaskStateUtil that does just this. Try not to cause a fight between the business rules! Refer to the wiki for more information on this: https://docs.servicenow.com/bundle/helsinki-servicenow-platform/page/app-store/dev_portal/API_reference/TaskStateUtil/concept/c_TaskActiveStateMgmtBusRule.html Priority: This is a choice field designed to give the person working on the task some idea as to which task they should complete first. It has a default value of 4. State: This is probably the most complex field in the table—so much so that it has its own section later in this article! It provides more details than the Active flag as to how the task is currently being processed. Parent: This is a reference field to another task record. A parent-to-child relationship is a one-to-many relationship. A child is generally taken to be a subdivision of a parent; a child task breaks down the parent task. A parent task may also represent a master task if there are several related tasks that need to be grouped together. Breadcrumbs can be added to a form to represent the parent relationship more visually. You can read up more about this here: https://docs.servicenow.com/bundle/helsinki-servicenow-platform/page/administer/form-administration/task/t_TaskParentBreadcrumbsFormatter.html Short description: Provide a quick summary of the task here. It is free text, but it should be kept short since it has an attribute in the dictionary that prevents it from being truncated in a list view. It is often used for reporting and e-mail notifications. Short description is a suggestion field. It will attempt to autocomplete when you begin typing: type in "issue" as an example. While you are free to add your own suggestions, it is not usually done. Check out the product documentation for more details: https://docs.servicenow.com/bundle/helsinki-servicenow-platform/page/administer/field-administration/concept/c_SuggestionFields.html Description: Here, you provide a longer description of the task. It is a simple large text field. Work notes: This field is one of the most well-used fields in the form. It is a journal_input field that always presents an empty text box. When the form is saved, the contents of a journal field are saved in a separate table called sys_journal_field. The journal_output fields such as Comments and Work notes are used to display them. Work notes and Comments are heavily integrated into Connect. When you follow and subsequently chat about a task, the appropriate field is automatically updated. We'll cover more about this later. Populating fields automatically The majority of the fields in the Task table aren't directly used. Instead, many fields are auto-populated, through logic actions such as business rules and as default values. Others are optional, available to be filled in if appropriate. All the data is then available for reporting or to drive processes. Some of the more notable fields are explained here, but this list is not exhaustive! The Approval field is discussed in more detail later in the article. There are several automatic fields, such as Approval set, that represent when a decision was made. A business rule populates Duration when the task becomes inactive. It records how long the task took in "calendar" time. There is also a Business Duration field to perform the same calculation in working hours, but it uses calendars, which are deprecated. The more modern equivalent is Service Levels, discussed at the end of this article. When a task is first created, a business rule records the logged-in user who performed the action and populates Opened by. When the task is set to inactive, it populates the Closed by field. Opened at and Closed at are date/time fields that also get populated. Company and Location are reference fields that provide extra detail about who the task is for and where it is. Location is dependent upon Company: if you populate the Company field, it will only show locations for that company. Location is also a tree picker field, like Assignment group. Due date is a date/time field to represent until when a task should be completed. Time worked is a specialized duration field that records how long the form has been on the screen for. If it is added to the form, a timer is shown. On saving, a business rule then populates the Time Worked [task_time_worked] table with the current user, how long it took, and any added comments. A formatter is an element that is added to a form (such as a field), but it uses a custom interface to present information. The activity formatter uses the Audit tables to present the changes made to fields on the task: who changed something, what they changed, and when. Recording room maintenance tasks At Gardiner Hotels, we have the highest standards of quality. Rooms must be clean and tidy, with all the light bulbs working, and no dripping taps! Let's create a table that will contain jobs for our helpful staff to complete. The process for dealing with a maintenance issue at Gardiner Hotels is straightforward; the need gets recorded in ServiceNow, and it gets assigned to the right team, who then work on it till it is resolved. Sometimes, a more complex issue will require the assistance of Cornell Hotel Services, a service company that will come equipped with the necessary tools. But to ensure that the team isn't used unnecessarily, management needs to approve any of their assignments. Have a look at the following figure, which represents what the process is: These requirements suggest the use of the Task table. In order to take advantage of its functionality, a new Maintenance table should be extended from it. Any time that a record is to be assigned to a person or a team, consider using the Task table as a base. There are other indicators too, such as the need for approvals. In general, you should always extend the Task table when supporting a new process. Table extension gives natural separation for different types of records, with the ability to create specific logic yet with inheritance. Now, let's create the Maintenance table by performing the following steps: Navigate to System Definition > Tables. Click New. Fill out the form with the following details, and Save when done: Label: Maintenance Extends table: Task Auto-number: <ticked>  (In the Controls tab) Then, using the Columns related list, create a new field using this data. Save when done. Column label: Room Type: Reference Reference: Room Click on Design Form in Related Links, and do the following: Add the Assignment Group, Approval, and Room fields and Activities (Filtered) Formatter from the selection of fields on the left. Remove the Configuration Item, Parent, and Active fields Click the Save button, then close the Form Design window once done. We want scripts to control the Approval field. So let's make that read-only. You should be in the Maintenance Table record. Find the Approval field in the Columns related list, and click on it. Once in the Dictionary record, you will notice the form is read-only and there is a message at the top of the screen saying you can’t edit this record since it is in a different scope. We do not want to take ownership of this field. Changing the settings of a field in the Task table will change it for all tables extended from Task. This is often not what you want! Contrarily, a Dictionary Override will only affect the selected table. Read more about this here: https://docs.servicenow.com/bundle/helsinki-servicenow-platform/page/administer/data-dictionary-tables/concept/c_DictionaryOverrides.html Instead, find the Dictionary Override tab, and click on New. Fill out the form with the following details, and Save. Table: Maintenance [x_hotel_maintenance] Override read only: <ticked> Read only: <ticked> If you navigate to Hotel > Maintenances and click New, the Maintenance form should now look something like this: Working with tasks You may be familiar with a work queue. It provides a list of items that you should complete, perhaps representing your work for the day. Often, this is achieved by assigning the item to a particular group. The members of that group then have access to the ticket and should do the necessary work to close it. In ServiceNow, this concept is represented in the Service Desk application menu. Open up the Service Desk Application Menu. You will find these options: My Work and My Groups Work. The former is a simple list view of the Task table with the following filters: Active is true. This filters out closed tasks. Assigned to is the current user, so if you are logged in as the user called System Administrator, you see tasks where the Assigned to field is set to System Administrator. State is not Pending, which filters out tasks that should not be worked on right now. The My Groups Work entry is very similar, but it shows tasks that haven't been given to a fulfiller and are still something that your group should deal with. It does this by showing tasks where the Assigned to field is empty and the Assignment group field is one of your groups. This means that when the My Work list is empty, you probably should get more work from My Groups Work. The My Work list shows all records that are derived from the Task table. This means you will see a mixture of records from many tables in this list. It is incredibly useful to have a "single pane of glass", where all your activities are in a single place, with consistent fields and data values. They can be manipulated easily and effectively: assign all your tickets to your colleague when you go on leave with a couple of clicks! Working without a queue Some platforms make the use of a work queue mandatory; the only way to look at a task is through your work queue. It is important to realize that ServiceNow does not have this restriction. The My Work list is a filtered list like any other. You do not have to be "assigned" the work before you can update or comment on it. There are many ways to find tasks to work on. This usually involves creating filters on lists. This may include tasks that have been marked as high priority or those that have been open for more than two weeks. In many IT organizations, a central service desk team is the single point of contact. They have the responsibility of ensuring tasks are completed quickly and effectively, regardless of who they are currently assigned to. ServiceNow makes this easy by ensuring tasks can be accessed in a variety of ways and not just through a work queue. Working socially Social media concepts are infiltrating many aspects of the IT industry, and ServiceNow is not immune. Some useful ideas have been pulled into the platform in an attempt to make working on tasks a more collaborative experience, ensuring the right people are involved. Chatting with Connect Connect Chat focuses on bringing fulfillers together. The UI16 Connect sidebar is easy to activate and use, letting you swap text, pictures, videos, and links easily and efficiently. The real benefit that ServiceNow brings with Connect is the ability to create a record conversation especially around tasks. This allows you to have a chat session that is connected to a particular record, allowing your conversation to be recorded and embedded. In Gardiner Hotels, the experienced staff probably already know how to deal with common maintenance tasks, and so by giving a newer team member easy access to them, our guests get better service. The Follow button is already available on every table extended from Task. But what’s special about Connect with the Task table is that the messages are added either as comments or work notes. While this is very useful for monitoring the progress of multiple tasks at the same time, record conversations are far less private: many users will have access to the activity log that shows the chat conversation. It probably isn’t a good idea to share a meme in the work notes of a high-priority task. Additional comments and work notes are discussed later in the article Communicating some more In addition to Connect Chat, ServiceNow provides several other options to share information. Connect Support allows requesters to ask for help via chat. Generally, the requester initiates the session through a self-service portal, by clicking on a button and entering a queue. A Service Desk Agent can then work with multiple fulfillers in the Connect window. Older versions of ServiceNow had chat functionality, but it was limited in capability—it did not support in-browser notifications, for instance. Help Desk Chat was also limited by having to use a custom page rather than having it integrated into the main interface. Both Chat and Connect use the same tables to store the message; they should not be used at the same time. More information is available in the Product Documentation https://docs.servicenow.com/bundle/helsinki-servicenow-platform/page/use/using-social-it/concept/c_HelpDeskChat.html Live Feed gives a Facebook-wall type interaction, where all types of users can read and add messages. This can be used as a self-service system, since the messages are searchable and referenceable by copying a link. It is a communication mechanism that allows users to be as involved as they'd like. Unlike e-mail, Live Feed is pull-style communication, where users must go to the right place to receive information. To ensure it gets checked regularly and is therefore most beneficial, the right culture must be cultivated in a company. Navigate to Collaborate > Live Feed to use it. Table Notification creates Chat and Live Feed messages automatically based on conditions and configurations. For example, during a service outage, the service desk may want an automatic communication to be sent out, alerting people proactively. Check out Collaborate > Feed Administration > Table Notifications. Summary In this article, we discussed looking at the Task table in which we covered the important fields, populating fields automatically and recording room maintenance tasks. Also covered working with Tasks in which we discussed working without a queue and working socially. Resources for Article: Further resources on this subject: VM, It Is Not What You Think! [article] Getting Started with Force.com [article] Overview of Certificate Management [article]

In this article by Narayan Prusty, the author of the book Modern JavaScript Applications, we will see the architecture of server side application development for complex and large applications (applications with huge number of users and large volume of data) shouldn't just involve faster response and providing web services for wide variety of platforms. It should be easy to scale, upgrade, update, test, and deploy. It should also be highly available, allowing the developers write components of the server side application in different programming languages and use different databases. Therefore, this leads the developers who build large and complex applications to switch from the common monolithic architecture to microservices architecture that allows us to do all this easily. As microservices architecture is being widely used in enterprises that build large and complex applications, it's really important to learn how to design and create server side applications using this architecture. In this chapter, we will discuss how to create applications based on microservices architecture with Node.js using the Seneca toolkit. (For more resources related to this topic, see here.) What is monolithic architecture? To understand microservices architecture, it's important to first understand monolithic architecture, which is its opposite. In monolithic architecture, different functional components of the server side application, such as payment processing, account management, push notifications, and other components, all blend together in a single unit. For example, applications are usually divided into three parts. The parts are HTML pages or native UI that run on the user's machine, server side application that runs on the server, and database that also runs on the server. The server side application is responsible for handling HTTP requests, retrieving and storing data in a database, executing algorithms, and so on. If the server side application is a single executable (that is running is a single process) that does all these task, than we say that the server side application is monolithic. This is a common way of building server side applications. Almost every major CMS, web servers, server side frameworks, and so on are built using monolithic architecture. This architecture may seem successful, but problems are likely to arise when your application is large and complex. Demerits of monolithic architecture The following are some of the issues caused by server side applications built using the monolithic architecture. Scaling monolithic architecture As traffic to your server side application increases, you will need to scale your server side application to handle the traffic. In case of monolithic architecture, you can scale the server side application by running the same executable on multiple servers and place the servers behind a load balancer or you can use round robin DNS to distribute the traffic among the servers: In the preceding diagram, all the servers will be running the same server side application. Although scaling is easy, scaling monolithic server side application ends up with scaling all the components rather than the components that require greater resource. Thus, causing unbalanced utilization of resources sometimes, depending on the quantity and types of resources the components need. Let's consider some examples to understand the issues caused while scaling monolithic server side applications: Suppose there is a component of server side application that requires a more powerful or special kind of hardware, we cannot simply scale this particular component as all the components are packed together, therefore everything needs to be scaled together. So, to make sure that the component gets enough resources, you need to run the server side application on some more servers with powerful or special hardware, leading to consumption of more resources than actually required. Suppose we have a component that requires to be executed on a specific server operating system that is not free of charge, we cannot simply run this particular component in a non-free operating system as all the components are packed together and therefore, just to execute this specific component, we need to install the non-free operating system in all servers, increasing the cost largely. These are just some examples. There are many more issues that you are likely to come across while scaling a monolithic server side application. So, when we scale monolithic server side applications, the components that don't need more powerful or special kind of resource starts receiving them, therefore deceasing resources for the component that needs them. We can say that scaling monolithic server side application involves scaling all components that are forcing to duplicate everything in the new servers. Writing monolithic server side applications Monolithic server side applications are written in a particular programming language using a particular framework. Enterprises usually have developers who are experts in different programming languages and frameworks to build server side applications; therefore, if they are asked to build a monolithic server side application, then it will be difficult for them to work together. The components of a monolithic server side application can be reused only in the same framework using, which it's built. So, you cannot reuse them for some other kind of project that's built using different technologies. Other issues of monolithic architecture Here are some other issues that developers might face. Depending on the technology that is used to build the monolithic server side application: It may need to be completely rebuild and redeployed for every small change made to it. This is a time-consuming task and makes your application inaccessible for a long time. It may completely fail if any one of the component fails. It's difficult to build a monolithic application to handle failure of specific components and degrade application features accordingly. It may be difficult to find how much resources are each components consuming. It may be difficult to test and debug individual components separately. Microservices architecture to the rescue We saw the problems caused by monolithic architecture. These problems lead developers to switch from monolithic architecture to microservices architecture. In microservices architecture, the server side application is divided into services. A service (or microservice) is a small and independent process that constitutes a particular functionality of the complete server side application. For example, you can have a service for payment processing, another service for account management, and so on; the services need to communicate with each other via network. What do you mean by "small" service? You must be wondering how small a service needs to be and how to tell whether a service is small or not? Well, it actually depends on many factors such as the type of application, team management, availability of resources, size of application, and how small you think is small? However, a small service doesn't have to be the one that is written is less lines of code or provides a very basic functionality. A small service can be the one on which a team of developers can work independently, which can be scaled independently to other services, scaling it doesn't cause unbalanced utilization of recourses, and overall they are highly decoupled (independent and unaware) of other services. You don't have to run each service in a different server, that is, you can run multiple services in a single computer. The ratio of server to services depends on different factors. A common factor is the amount and type of resources and technologies required. For example, if a service needs a lot of RAM and CPU time, then it would be better to run it individually on a server. If there are some services that don't need much resources, then you can run them all in a single server together. The following diagram shows an example of the microservices architecture: Here, you can think of Service 1 as the web server with which a browser communicates and other services providing APIs for various functionalities. The web services communicate with other services to get data. Merits of microservices architecture Due to the fact that services are small and independent and communicate via network, it solves many problems that monolithic architecture had. Here are some of the benefits of microservices architecture: As the services communicate via network, they can be written in different programming languages using different frameworks Making a change to a service only requires that particular service to be redeployed instead of all the services, which is a faster procedure It becomes easier to measure how much resources are consumed by each service as each service runs in a different process It becomes easier to test and debug, as you can analyze each service separately Services can be reused by other applications as they interact via network calls Scaling services Apart from the preceding benefits, one of the major benefits of microservices architecture is that you can scale individual services that require scaling instead of all the services, therefore preventing duplication of resources and unbalanced utilization of resources. Suppose we want to scale Service 1 in the preceding diagram. Here is a diagram that shows how it can be scaled: Here, we are running two instances of Service 1 on two different servers kept behind a load balancer, which distributes the traffic between them. All other services run the same way as scaling them wasn't required. If you wanted to scale Service 3, then you can run multiple instances of Service 3 on multiple servers and place them behind a load balancer. Demerits of microservices architecture Although there are a lot of merits of using microservices architecture compared to monolithic architecture, there are some demerits of microservices architecture as well: As the server side application is divided into services, deploying, and optionally, configuring each service separately is cumbersome and a time-consuming task. Note that developers often use some sort automation technology (such as AWS, Docker, and so on) to make deployment somewhat easier; however, to use it, you still need a good level of experience and expertise of that technology. Communication between services is likely to lag as it's done via network. This sort of server side applications is more prone to network security vulnerabilities as services communicate via network. Writing code for communicating with other services can be harder, that is, you need to make network calls and then parse the data to read it. This also requires more processing. Note that although there are frameworks to build server side applications using microservices that make fetching and parsing of data easier, it still doesn't deduct the processing and network wait time. You will surely need some sort of monitoring tool to monitor services as they may go down due to network, hardware, or software failure. Although you may use the monitoring tool only when your application suddenly stops, to build the monitoring software or use some sort of service, monitoring software needs some level of extra experience and expertise. Microservices-based server side applications are slower than monolithic-based server side applications as communication via networks is slower compared to memory. When to use microservices architecture? It may seem like its difficult to choose between monolithic and microservices architecture, but it's actually not so hard to decide between them. If you are building a server side application using monolithic architecture and you feel that you are unlikely to face any monolithic issues that we discussed earlier, then you can stick to monolithic architecture. In future, if you are facing issues that can be solved using microservices architecture, then you should switch to microservices architecture. If you are switching from a monolithic architecture to microservices architecture, then you don't have to rewrite the complete application, instead you can only convert the components that are causing issues to services by doing some code refactoring. This sort of server side applications where the main application logic is monolithic but some specific functionality is exposed via services is called microservices architecture with monolithic core. As issues increase further, you can start converting more components of the monolithic core to services. If you are building a server side application using monolithic architecture and you feel that you are likely to face any of the monolithic issues that we discussed earlier, then you should immediately switch to microservices architecture or microservices architecture with monolithic core, depending on what suits you the best. Data management In microservices architecture, each service can have its own database to store data and can also use a centralized database to store. Some developers don't use a centralized database at all, instead all services have their own database to store the data. To synchronize the data between the services, the services omit events when their data is changed and other services subscribe to the event and update the data. The problem with this mechanism is that if a service is down, then it may miss some events. There is also going to be a lot of duplicate data, and finally, it is difficult to code this kind of system. Therefore, it's a good idea to have a centralized database and also let each service to maintain their own database if they want to store something that they don't want to share with others. Services should not connect to the centralized database directly, instead there should be another service called database service that provides APIs to work with the centralized database. This extra layer has many advantages, such as the underlying schema can be changed without updating and redeploying all the services that are dependent on the schema, we can add a caching layer without making changes to the services, you can change the type of database without making any changes to the services and there are many other benefits. We can also have multiple database services if there are multiple schemas, or if there are different types of databases, or due to some other reason that benefits the overall architecture and decouples the services. Implementing microservices using Seneca Seneca is a Node.js framework for creating server side applications using microservices architecture with monolithic core. Earlier, we discussed that in microservices architecture, we create a separate service for every component, so you must be wondering what's the point of using a framework for creating services that can be done by simply writing some code to listen to a port and reply to requests. Well, writing code to make requests, send responses, and parse data requires a lot of time and work, but a framework like Seneca make all this easy. Also converting components of monolithic core to services is also a cumbersome task as it requires a lot of code refactoring, but Seneca makes it easy by introducing a concept of actions and plugins. Finally, services written in any other programming language or framework will be able to communicate with Seneca services. In Seneca, an action represents a particular operation. An action is a function that's identified by an object literal or JSON string called as the action's pattern. In Seneca, these operations of a component of monolithic core are written using actions, which we may later want to move from monolithic core to a service and expose it to other services and monolithic core via network. Why actions? You might be wondering what is the benefit of using actions instead of functions to write operations and how actions make it easy to convert components of monolithic core to services? Suppose you want to move an operation of monolithic core that is written using a function to a separate service and expose the function via network then you cannot simply copy and paste the function to the new service, instead you need to define a route (if you are using Express). To call the function inside the monolithic core, you will need to write code to make an HTTP request to the service. To call this operation inside the service, you can simply call a function so that there are two different code snippets depending from where you are executing the operation. Therefore, moving operations requires a lot of code refactoring. However, if you would have written the preceding operation using the Seneca action, then it would have been really easy to move the operation to a separate service. In case the operation is written using action, and you want to move the operation to a separate service and expose the operation via network, then you can simply copy and paste the action to the new service. That's it. Obviously, we also need to tell the service to expose the action via network and tell the monolithic core where to find the action, but all these require just couple of lines of code. A Seneca service exposes actions to other services and monolithic core. While making request to a service, we need to provide a pattern matching an action's pattern to be called in the service. Why patterns? Patterns make it easy to map a URL to action, patterns can overwrite other patterns for specific conditions, therefore it prevents editing of the existing code, as editing of the existing code in a production site is not safe and have many other disadvantages. Seneca also has a concept of plugins. A seneca plugin is actually a set of actions that can be easily distributed and plugged in to a service or monolithic core. As our monolithic core becomes larger and complex, we can convert components to services. That is, move actions of certain components to services. Summary In this chapter, we saw the difference between monolithic and microservices architecture. Then we discussed what microservices architecture with monolithic core means and its benefits. Finally, we jumped into the Seneca framework for implementing microservices architecture with monolithic core and discussed how to create a basic login and registration functionality to demonstrate various features of the Seneca framework and how to use it. In the next chapter, we will create a fully functional e-commerce website using Seneca and Express frameworks Resources for Article: Further resources on this subject: Microservices – Brave New World [article] Patterns for Data Processing [article] Domain-Driven Design [article]

In this article by Agus Kurniawan, authors of Smart Internet of Things Projects, we will explore how to make your IoT board speak something. Various sound and speech modules will be explored as project journey. (For more resources related to this topic, see here.) We explore the following topics Introduce a speech technology Introduce sound sensor and actuator Introduce pattern recognition for speech technology Review speech and sound modules Build your own voice commands for IoT projects Make your IoT board speak Make Raspberry Pi speak Introduce a speech technology Speech is the primary means of communication among people. A speech technology is a technology which is built by speech recognition research. A machine such as a computer can understand what human said even the machine can recognize each speech model so the machine can differentiate each human's speech. A speech technology covers speech-to-text and text-to-speech topics. Some researchers already define several speech model for some languages, for instance, English, German, China, French. A general of speech research topics can be seen in the following figure: To convert speech to text, we should understand about speech recognition. Otherwise, if we want to generate speech sounds from text, we should learn about speech synthesis. This article doesn't cover about speech recognition and speech synthesis in heavy mathematics and statistics approach. I recommend you read textbook related to those topics. In this article, we will learn how to work sound and speech processing on IoT platform environment. Introduce sound sensors and actuators Sound sources can come from human, animal, car, and etc. To process sound data, we should capture the sound source from physical to digital form. This happens if we use sensor devices which capture the physical sound source. A simple sound sensor is microphone. This sensor can record any source via microphone. We use a microphone module which is connected to your IoT board, for instance, Arduino and Raspberry Pi. One of them is Electret Microphone Breakout, https://www.sparkfun.com/products/12758. This is a breakout module which exposes three pin outs: AUD, GND, and VCC. You can see it in the following figure. Furthermore, we can generate sound using an actuator. A simple sound actuator is passive buzzer. This component can generate simple sounds with limited frequency. You can generate sound by sending on signal pin through analog output or PWM pin. Some manufacturers also provide a breakout module for buzzer. Buzzer actuator form is shown in the following figure. Buzzer usually is passive actuator. If you want to work with active sound actuator, you can use a speaker. This component is easy to find on your local or online store. I also found it on Sparkfun, https://www.sparkfun.com/products/11089 which you can see it in the following figure. To get experiences how to work sound sensor/actuator, we build a demo to capture sound source by getting sound intensity. In this demo, I show how to detect a sound intensity level using sound sensor, an Electret microphone. The sound source can come from sounds of voice, claps, door knocks or any sounds loud enough to be picked up by a sensor device. The output of sensor device is analog value so MCU should convert it via a microcontroller's analog-to-digital converter. The following is a list of peripheral for our demo. Arduino board. Resistor 330 Ohm. Electret Microphone Breakout, https://www.sparkfun.com/products/12758. 10 Segment LED Bar Graph - Red, https://www.sparkfun.com/products/9935. You can use any color for LED bar. You can also use Adafruit Electret Microphone Breakout to be attached into Arduino board. You can review it on https://www.adafruit.com/product/1063. To build our demo, you wire those components as follows Connect Electret Microphone AUD pin to Arduino A0 pin Connect Electret Microphone GND pin to Arduino GND pin Connect Electret Microphone VCC pin to Arduino 3.3V pin Connect 10 Segment LED Bar Graph pins to Arduino digital pins: 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 which already connected to resistor 330 Ohm You can see the final wiring of our demo in the following figure: 10 segment led bar graph module is used to represent of sound intensity level. In Arduino we can use analogRead() to read analog input from external sensor. Output of analogRead() returns value 0 - 1023. Total output in voltage is 3.3V because we connect Electret Microphone Breakout with 3.3V on VCC. From this situation, we can set 3.3/10 = 0.33 voltage for each segment led bar. The first segment led bar is connected to Arduino digital pin 3. Now we can implement to build our sketch program to read sound intensity and then convert measurement value into 10 segment led bar graph. To obtain a sound intensity, we try to read sound input from analog input pin. We read it during a certain time, called sample window time, for instance, 250 ms. During that time, we should get the peak value or maximum value of analog input. The peak value will be set as sound intensity value. Let's start to implement our program. Open Arduino IDE and write the following sketch program. // Sample window width in mS (250 mS = 4Hz) const int sampleWindow = 250; unsigned int sound; int led = 13; void setup() { Serial.begin(9600); pinMode(led, OUTPUT); pinMode(3, OUTPUT); pinMode(4, OUTPUT); pinMode(5, OUTPUT); pinMode(6, OUTPUT); pinMode(7, OUTPUT); pinMode(8, OUTPUT); pinMode(9, OUTPUT); pinMode(10, OUTPUT); pinMode(11, OUTPUT); pinMode(12, OUTPUT); } void loop() { unsigned long start= millis(); unsigned int peakToPeak = 0; unsigned int signalMax = 0; unsigned int signalMin = 1024; // collect data for 250 milliseconds while (millis() - start < sampleWindow) { sound = analogRead(0); if (sound < 1024) { if (sound > signalMax) { signalMax = sound; } else if (sound < signalMin) { signalMin = sound; } } } peakToPeak = signalMax - signalMin; double volts = (peakToPeak * 3.3) / 1024; Serial.println(volts); display_bar_led(volts); } void display_bar_led(double volts) { display_bar_led_off(); int index = round(volts/0.33); switch(index){ case 1: digitalWrite(3, HIGH); break; case 2: digitalWrite(3, HIGH); digitalWrite(3, HIGH); break; case 3: digitalWrite(3, HIGH); digitalWrite(4, HIGH); digitalWrite(5, HIGH); break; case 4: digitalWrite(3, HIGH); digitalWrite(4, HIGH); digitalWrite(5, HIGH); digitalWrite(6, HIGH); break; case 5: digitalWrite(3, HIGH); digitalWrite(4, HIGH); digitalWrite(5, HIGH); digitalWrite(6, HIGH); digitalWrite(7, HIGH); break; case 6: digitalWrite(3, HIGH); digitalWrite(4, HIGH); digitalWrite(5, HIGH); digitalWrite(6, HIGH); digitalWrite(7, HIGH); digitalWrite(8, HIGH); break; case 7: digitalWrite(3, HIGH); digitalWrite(4, HIGH); digitalWrite(5, HIGH); digitalWrite(6, HIGH); digitalWrite(7, HIGH); digitalWrite(8, HIGH); digitalWrite(9, HIGH); break; case 8: digitalWrite(3, HIGH); digitalWrite(4, HIGH); digitalWrite(5, HIGH); digitalWrite(6, HIGH); digitalWrite(7, HIGH); digitalWrite(8, HIGH); digitalWrite(9, HIGH); digitalWrite(10, HIGH); break; case 9: digitalWrite(3, HIGH); digitalWrite(4, HIGH); digitalWrite(5, HIGH); digitalWrite(6, HIGH); digitalWrite(7, HIGH); digitalWrite(8, HIGH); digitalWrite(9, HIGH); digitalWrite(10, HIGH); digitalWrite(11, HIGH); break; case 10: digitalWrite(3, HIGH); digitalWrite(4, HIGH); digitalWrite(5, HIGH); digitalWrite(6, HIGH); digitalWrite(7, HIGH); digitalWrite(8, HIGH); digitalWrite(9, HIGH); digitalWrite(10, HIGH); digitalWrite(11, HIGH); digitalWrite(12, HIGH); break; } } void display_bar_led_off() { digitalWrite(3, LOW); digitalWrite(4, LOW); digitalWrite(5, LOW); digitalWrite(6, LOW); digitalWrite(7, LOW); digitalWrite(8, LOW); digitalWrite(9, LOW); digitalWrite(10, LOW); digitalWrite(11, LOW); digitalWrite(12, LOW); } Save this sketch program as ch05_01. Compile and deploy this program into Arduino board. After deployed the program, you can open Serial Plotter tool. You can find this tool from Arduino menu Tools -| Serial Plotter. Set the baud rate as 9600 baud on the Serial Plotter tool. Try to make noise on a sound sensor device. You can see changing values on graphs from Serial Plotter tool. A sample of Serial Plotter can be seen in the following figure: How to work? The idea to obtain a sound intensity is easy. We get a value among sound signal peaks. Firstly, we define a sample width, for instance, 250 ms for 4Hz. // Sample window width in mS (250 mS = 4Hz) const int sampleWindow = 250; unsigned int sound; int led = 13; On the setup() function, we initialize serial port and our 10 segment led bar graph. void setup() { Serial.begin(9600); pinMode(led, OUTPUT); pinMode(3, OUTPUT); pinMode(4, OUTPUT); pinMode(5, OUTPUT); pinMode(6, OUTPUT); pinMode(7, OUTPUT); pinMode(8, OUTPUT); pinMode(9, OUTPUT); pinMode(10, OUTPUT); pinMode(11, OUTPUT); pinMode(12, OUTPUT); } On the loop() function, we perform to calculate a sound intensity related to a sample width. After obtained a peak-to-peak value, we convert it into voltage form. unsigned long start= millis(); unsigned int peakToPeak = 0; unsigned int signalMax = 0; unsigned int signalMin = 1024; // collect data for 250 milliseconds while (millis() - start < sampleWindow) { sound = analogRead(0); if (sound < 1024) { if (sound > signalMax) { signalMax = sound; } else if (sound < signalMin) { signalMin = sound; } } } peakToPeak = signalMax - signalMin; double volts = (peakToPeak * 3.3) / 1024; Then, we show a sound intensity in volt form in serial port and 10 segment led by calling display_bar_led(). Serial.println(volts); display_bar_led(volts); Inside the display_bar_led() function, we turn off all LEDs on 10 segment led bar graph by calling display_bar_led_off() which sends LOW on all LEDs using digitalWrite(). After that, we calculate a range value from volts. This value will be converted as total showing LEDs. display_bar_led_off(); int index = round(volts/0.33); Introduce pattern recognition for speech technology Pattern recognition is one of topic in machine learning and as baseline for speech recognition. In general, we can construct speech recognition system in the following figure: From human speech, we should convert it into digital form, called discrete data. Some signal processing methods are applied to handle pre-processing such as removing noise from data. Now in pattern recognition we do perform speech recognition method. Researchers did some approaches such as computing using Hidden Markov Model (HMM) to identity sound related to word. Performing feature extraction in speech digital data is a part of pattern recognition activities. The output will be used as input in pattern recognition input. The output of pattern recognition can be applied as Speech-to-Text and Speech command on our IoT projects. Reviewing speech and sound modules for IoT devices In this section, we review various speech and sound modules which can be integrated into our MCU board. There are a lot of modules related to speech and sound processing. Each module has unique features which fits with your works. One of speech and sound modules is EasyVR 3 & EasyVR Shield 3 from VeeaR. You can review this module on http://www.veear.eu/introducing-easyvr-3-easyvr-shield-3/. Several languages already have been supported such as English (US), Italian, German, French, Spanish, and Japanese. You can see EasyVR 3 module in the following figure: EasyVR 3 board also is available as a shield for Arduino. If you buy an EasyVR Shield 3, you will obtain EasyVR board and its Arduino shield. You can see the form of EasyVR Shield 3 on the following figure: The second module is Emic 2. It was designed by Parallax in conjunction with Grand Idea Studio, http:// www.grandideastudio.com/, to make voice synthesis a total no-brainer. You can send texts to the module to generate human speech through serial protocol. This module is useful if you want to make boards speak. Further information about this module, you can visit and buy this module on https://www.parallax.com/product/30016. The following is a form of Emic-2 module: Summary We have learned some basic sound and voice processing. We also explore several sound and speech modules to integrate into your IoT project. We built a program to read sound intensity level at the first. Resources for Article: Further resources on this subject: Introducing IoT with Particle's Photon and Electron [article] Web Typography [article] WebRTC in FreeSWITCH [article]

In this article by Wolf Halton and Bo Weaver, the authors of the book Kali Linux 2: Windows Penetration Testing, we try to debunk the myth that all Windows systems are easy to exploit. This is not entirely true. Almost any Windows system can be hardened to the point that it takes too long to exploit its vulnerabilities. In this article, you will learn the following: How to footprint your Windows network and discover the vulnerabilities before the bad guys do Ways to investigate and map your Windows network to find the Windows systems that are susceptible to exploits (For more resources related to this topic, see here.) In some cases, this will be adding to your knowledge of the top 10 security tools, and in others, we will show you entirely new tools to handle this category of investigation. Footprinting the network You can't find your way without a good map. In this article, we are going to learn how to gather network information and assess the vulnerabilities on the network. In the Hacker world this is called Footprinting. This is the first step to any righteous hack. This is where you will save yourself time and massive headaches. Without Footprinting your targets, you are just shooting in the dark. The biggest tool in any good pen tester's toolbox is Mindset. You have to have the mind of a sniper. You learn your targets habits and its actions. You learn the traffic flows on the network where your target lives. You find the weaknesses in your target and then attack those weaknesses. Search and destroy! In order to do good Footprinting, you have to use several tools that come with Kali. Each tool has it strong points and looks at the target from a different angle. The more views you have of your target, the better plan of attack you have. Footprinting will differ depending on whether your targets are external on the public network, or internal and on a LAN. We will be covering both aspects. Please read the paragraph above again, and remember you do not have our permission to attack these machines. Don't do the crime if you can't do the time. Exploring the network with Nmap You can't talk about networking without talking about Nmap. Nmap is the Swiss Army knife for network administrators. It is not only a great Footprinting tool, but also the best and cheapest network analysis tool any sysadmin can get. It's a great tool for checking a single server to make sure the ports are operating properly. It can heartbeat and ping an entire network segment. It can even discover machines when ICMP (ping) has been turned off. It can be used to pressure-test services. If the machine freezes under the load, it needs repairs. Nmap was created in 1997 by Gordon Lyon, who goes by the handle Fyodor on the Internet. Fyodor still maintains Nmap and it can be downloaded from http://insecure.org. You can also order his book Nmap Network Scanning on that website. It is a great book, well worth the price! Fyodor and the Nmap hackers have collected a great deal of information and security e-mail lists on their site. Since you have Kali Linux, you have a full copy of Nmap already installed! Here is an example of Nmap running against a Kali Linux instance. Open the terminal from the icon on the top bar or by clicking on the menu link Application | Accessories | Terminal. You could also choose the Root Terminal if you want, but since you are already logged in as Root, you will not see any differences in how the terminal emulator behaves. Type nmap -A 10.0.0.4 at the command prompt (you need to put in the IP of the machine you are testing). The output shows the open ports among 1000 commonly used ports. Kali Linux, by default, has no running network services, and so in this run you will see a readout showing no open ports. To make it a little more interesting, start the built-in webserver by typing /etc/init.d/apache2 start. With the web server started, run the Nmap command again: nmap -A 10.0.0.4 As you can see, Nmap is attempting to discover the operating system (OS) and to tell which version of the web server is running: Here is an example of running Nmap from the Git Bash application, which lets you run Linux commands on your Windows desktop. This view shows a neat feature of Nmap. If you get bored or anxious and think the system is taking too much time to scan, you can hit the down arrow key and it will print out a status line to tell you what percentage of the scan is complete. This is not the same as telling you how much time is left on the scan, but it does give you an idea what has been done: Zenmap Nmap comes with a GUI frontend called Zenmap. Zenmap is a friendly graphic interface for the Nmap application. You will find Zenmap under Applications | Information Gathering | Zenmap. Like many Windows engineers, you may like Zenmap more than Nmap: Here we see a list of the most common scans in a drop-down box. One of the cool features of Zenmap is when you set up a scan using the buttons, the application also writes out the command-line version of the command, which will help you learn the command-line flags used when using Nmap in command-line mode. Hacker tip Most hackers are very comfortable with the Linux Command Line Interface (CLI). You want to learn the Nmap commands on the command line because you can use Nmap inside automated Bash scripts and make up cron jobs to make routine scans much simpler. You can set a cron job to run the test in non-peak hours, when the network is quieter, and your tests will have less impact on the network's legitimate users. The choice of intense scan produces a command line of nmap -T4 -A -v. This produces a fast scan. The T stands for Timing (from 1 to 5), and the default timing is -T3. The faster the timing, the rougher the test, and the more likely you are to be detected if the network is running an Intrusion Detection System (IDS). The -A stands for All, so this single option gets you a deep port scan, including OS identification, and attempts to find the applications listening on the ports, and the versions of those applications.  Finally, the -v stands for verbose. -vv means very verbose: Summary In this article, we learned about penetration testing in a Windows environment. Contrary to popular belief, Windows is not riddled with wide-open security holes ready for attackers to find. We learned how to use nmap to obtain detailed statistics about the network, making it an indispensible tool in our pen testing kit. Then, we looked at Zenmap, which is a GUI frontend for nmap and makes it easy for us to view the network. Think of nmap as flight control using audio transmissions and Zenmap as a big green radar screen—that's how much easier it makes our work. Resources for Article: Further resources on this subject: Bringing DevOps to Network Operations [article] Installing Magento [article] Zabbix Configuration [article]

In this article by Ivan Morgillo and Stefano Viola, the authors of the book Learning Embedded Android N Programming, you will learn about the kernel customization to the boot sequence. You will learn how to retrieve the proper source code for Google devices, how to set up the build environment, how to build your first custom (For more resources related to this topic, see here.) version of the Linux kernel, and deploy it to your device. You will learn about: Toolchain overview How to configure the host system to compile your own Linux kernel How to configure the Linux kernel Linux kernel overview Android boot sequence The Init process An overview of the Linux kernel We learned how Android has been designed and built around the Linux kernel. One of the reasons to choose the Linux kernel was its unquestioned flexibility and the infinite possibilities to adjust it to any specific scenario and requirement. These are the features that have made Linux the most popular kernel in the embedded industry. Linux kernel comes with a GPL license. This particular license allowed Google to contribute to the project since the early stages of Android. Google provided bug fixing and new features, helping Linux to overcome a few obstacles and limitations of the 2.6 version. In the beginning, Linux 2.6.32 was the most popular version for the most part of the Android device market. Nowadays, we see more and more devices shipping with the new 3.x versions. The following screenshot shows the current build for the official Google Motorola Nexus 6, with kernel 3.10.40: The Android version we created in the previouly was equipped with a binary version of the Linux kernel. Using an already compiled version of the kernel is the standard practice: as we have seen, AOSP provides exactly this kind of experience. As advanced users, we can take it a step further and build a custom kernel for our custom Android system. The Nexus family offers an easy entry into this world as we can easily obtain the kernel source code we need to build a custom version. We can also equip our custom Android system with our custom Linux kernel and we will have a full-customized ROM, tailored for our specific needs. In this book, we are using Nexus devices on purpose—Google is one of the few companies that formally make available the kernel source code. Even if every company producing and selling Android devices is forced by law to release the kernel source code, very few of them actually do it, despite all the GPL license rules. Obtaining the kernel Google provides the kernel source code and binary version for every single version of Android for every single device of the Nexus family. The following table shows where the binary version and the source code are located, ordered by device code name: Device Binary location Source location Build configuration shamu device/moto/shamu-kernel kernel/msm shamu_defconfig fugu device/asus/fugu-kernel kernel/x86_64 fugu_defconfig volantis device/htc/flounder-kernel kernel/tegra flounder_defconfig hammerhead device/lge/ hammerhead-kernel kernel/msm hammerhead_defconfig flo device/asus/flo-kernel/kernel kernel/msm flo_defconfig deb device/asus/flo-kernel/kernel kernel/msm flo_defconfig manta device/samsung/manta/kernel kernel/exynos manta_defconfig mako device/lge/mako-kernel/kernel kernel/msm mako_defconfig grouper device/asus/grouper/kernel kernel/tegra tegra3_android_defconfig tilapia device/asus/grouper/kernel kernel/tegra tegra3_android_defconfig maguro device/samsung/tuna/kernel kernel/omap tuna_defconfig toro device/samsung/tuna/kernel kernel/omap tuna_defconfig panda device/ti/panda/kernel kernel/omap panda_defconfig stingray device/moto/wingray/kernel kernel/tegra stingray_defconfig wingray device/moto/wingray/kernel kernel/tegra stingray_defconfig crespo device/samsung/crespo/kernel kernel/samsung herring_defconfig crespo4g device/samsung/crespo/kernel kernel/samsung herring_defconfig We are going to work with the Motorola Nexus 6, code name Shamu. Both the kernel binary version and the kernel source code are stored in a git repository. All we need to do is compose the proper URL and clone the corresponding repository. Retrieving the kernel's binary version In this section, we are going to obtain the kernel as a binary, prebuilt file. All we need is the previous table that shows every device model, with its codename and its binary location that we can use to compose the download of the URL. We are targeting Google Nexus 6, codename shamu with binary location: device/moto/shamu-kernel So, to retrieve the binary version of the Motorola Nexus 6 kernel, we need the following command: $ git clone https://android.googlesource.com/device/moto/shamu-kernel The previous command will clone the repo and place it in the shamu-kernel folder. This folder contains a file named zImage-dtb—this file is the actual kernel image that can be integrated in our ROM and flashed into our device. Having the kernel image, we can obtain the kernel version with the following command: $ $ dd if=kernel bs=1 skip=$(LC_ALL=C grep -a -b -o $'x1fx8bx08x00x00x00x00x00' kernel | cut -d ':' -f 1) | zgrep -a 'Linux version' Output: The previous screenshot shows the command output: our kernel image version is 3.10.40 and it has been compiled with GCC version 4.8 on October the the twenty-second at 22:49. Obtaining the kernel source code As for the binary version, the previous table is critical also to download the kernel source code. Targeting the Google Nexus 6, we create the download URL using the source location string for the device codename shamu: kernel/msm.git Once we have the exact URL, we can clone the GIT repository with the following command: $ git clone https://android.googlesource.com/kernel/msm.git Git will create an msm folder. The folder will be strangely empty—that's because the folder is tracking the master branch by default. To obtain the kernel for our Nexus 6, we need to switch to the proper branch. There are a lot of available branches and we can check out the list with the following command: $ git branch -a The list will show every single branch, targeting a specific Android version for a specific Nexus device. The following screenshot shows a subset of these repositories: Now that you have the branch name, for your device and your Android version, you just need to checkout the proper branch: $ git checkout android-msm-shamu-3.10-lollipop-release The following screenshot shows the expected command output: Setting up the toolchain The toolchain is the set of all the tools needed to effectively compile a specific software to a binary version, enabling the user to run it. In our specific domain, the toolchain allows us to create a system image ready to be flashed to our Android device. The interesting part is that the toolchain allows us to create a system image for an architecture that is different from our current one: odds are that we are using an x86 system and we want to create a system image targeting an ARM (Advanced RISC Machine) device. Compiling software targeting an architecture different from the one on our host system is called cross-compilation. The Internet offers a couple of handy solutions for this task—we can use the standard toolchain, available with the AOSP (Android Open Source Project) or we can use an alternative, very popular toolchain, the Linaro toolchain. Both toolchains will do the job—compile every single C/C++ file for the ARM architecture. As usual, even the toolchain is available as precompiled binary or as source code, ready to be compiled. For our journey, we are going to use the official toolchain, provided by Google, but when you need to explore this world even more, you could try out the binary version of Linaro toolchain, downloadable from www.linaro.org/download. Linaro toolchain is known to be the most optimized and performing toolchain in the market, but our goal is not to compare toolchains or stubbornly use the best or most popular one. Our goal is to create the smoothest possible experience, removing unnecessary variables from the whole building a custom Android system equation. Getting the toolchain We are going to use the official toolchain, provided by Google. We can obtain it with Android source code or downloading it separately. Having your trusted Android source code folder at hand, you can find the toolchain in the following folder: AOSP/prebuilts/gcc/linux-x86/arm/arm-eabi-4.8/ This folder contains everything we need to build a custom kernel—the compiler, the linker, and few more tools such as a debugger. If, for some unfortunate reason, you are missing the Android source code folder, you can download the toolchain using the following git command: $ git clone https://android.googlesource.com/platform/prebuilts/gcc/linux-x86/arm/arm-eabi-4.8 Preparing the host system To successfully compile our custom kernel, we need a properly configured host system. The requirements are similar to those we satisfied to build the whole Android system: Ubuntu Linux kernel source code Toolchain Fastboot Ubuntu needs a bit of love to accomplish this task: we need to install the ncurses-dev package: $ sudo apt-get install ncurses-dev Once we have all the required tools installed, we can start configuring the environment variables we need. These variables are used during the cross-compilation and can be set via the console. Fire up your trusted Terminal and launch the following commands: $ export PATH=<toolchain-path>/arm-eabi-4.8/bin:$PATH $ export ARCH=arm $ export SUBARCH=arm $ export CROSS_COMPILE=arm-eabi- Configuring the kernel Before being able to compile the kernel, we need to properly configure it. Every device in the Android repository has a specific branch with a specific kernel with a specific configuration to be applied. The table on page 2 has a column with the exact information we need—Build configuration. This information represents the parameter we need to properly configure the kernel build system. Let's configure everything for our Google Nexus 6. In your terminal, launch the following command: $ make shamu_defconfig This command will create a kernel configuration specific for your device. The following screenshot shows the command running and the final success message: Once the .config file is in place, you could already build the kernel, using the default configuration. As advanced users, we want more and that's why we will take full control of the system, digging into the kernel configuration. Editing the configuration could enable missing features or disable unneeded hardware support, to create the perfect custom kernel, and fit your needs. Luckily, to alter the kernel configuration, we don't need to manually edit the .config file. The Linux kernel provides a graphical tool that will allow you to navigate the whole configuration file structure, get documentation about the single configurable item, and prepare a custom configuration file with zero effort. To access the configuration menu, open your terminal, navigate to the kernel folder and launch the following command: $ make menuconfig The following screenshot shows the official Linux kernel configuration tool—no frills, but very effective: In the upper half of the screenshot, you can see the version of the kernel we are going to customize and a quick doc about how you can navigate all those menu items: you navigate using the arrow keys, you enter a subsection with the Enter key, you select or deselect an item using Y/N or Spacebar to toggle. With great power comes great responsibility, so be careful enabling and disabling features—check the documentation in menuconfig, check the Internet, and, most of all, be confident. A wrong configuration could cause a freeze during the boot sequence and this would force you to learn, to create a different configuration and try again. As a real-world example, we are going to enable the FTDI support. Future Technology Devices International or FTDI is a worldwide known semiconductor company, popular for its RS-232/TTL to USB devices. These devices come in very handy to communicate to embedded devices using a standard USB connection. To enable the FTDI support, you need to navigate to the right menu by following these steps: Device Drivers|USB support|USB Serial Converter support Once you reach this section, you need to enable the following item: USB FTDI Single Port Serial Driver The following screenshot shows the correctly selected item and gives you an idea of how many devices we could possibly support (this screen only shows the USB Serial Converter support): Once you have everything in place, just select Exit and save the configuration, as shown in the following screenshot: With the exact same approach, you can add every new feature you want. One important note, we added the FTDI package merging it into the kernel image. Linux kernel gives you the opportunity to make a feature available also as a module. A module is an external file, with .ko extension, that can be injected and loaded in the kernel at runtime. The kernel modules are a great and handy feature when you are working on a pure Linux system, but they are very impractical on Android. With the hope of having a modular kernel, you should code yourself the whole module loading system, adding unnecessary complexity to the system. The choice we made of having the FTDI feature inside the kernel image penalizes the image from a size point of view, but relieves us from the manual management of the module itself. That's why the common strategy is to include every new feature we want right into the kernel core. Compiling the kernel Once you have a properly configured environment and a brand new configuration file, you just need one single command to start the building process. On your terminal emulator, in the kernel source folder, launch: $ make The make command will wrap up the necessary configuration and will launch the compiling and assembling process. The duration of the process heavily depends on the performance of your system: it could be one minute or one hour. As a reference, an i5 2.40 GHz CPU with 8 GB of RAM takes 5-10 minutes to complete a clean build. This is incredibly quicker than compiling the whole AOSP image, as you can see, due to the different complexity and size of the code base. Working with non-Google devices So far, we have worked with Google devices, enjoying the Google open-source mindset. As advanced users, we frequently deal with devices that are not from Google or that are not even a smartphone. As a real-world example, we are going to use again a UDOO board: a single-board computer that supports Ubuntu or Android. For the time being, the most popular version of UDOO is the UDOO Quad and that's the version we are targeting. As for every other device, the standard approach is to trust the manufacturer's website to obtain kernel source code and any useful documentation for the process: most of all, how to properly flash the new kernel to the system. When working with a custom kernel, the procedure is quite consolidated. You need the source code, the toolchain, a few configuration steps, and, maybe, some specific software package to be installed on to your host system. When it comes to flashing the kernel, every device can have a different procedure. This depends on how the system has been designed and which tools the manufacturing team provides. Google provides fastboot to flash our images to our devices. Other manufactures usually provide tools that are similar or that can do similar things with little effort. The UDOO development team worked hard to make the UDOO board fully compatible with fastboot—instead of forcing you to adjust to their tools, they adjusted their device to work with the tools you already know. They tuned up the board's bootloader and you can now flash the boot.img using fastboot, like you were flashing a standard Google Android device. To obtain the kernel, we just need to clone a git repository. With your trusted terminal, launch the following command: $ git clone http://github.com/UDOOBoard/Kernel_Unico kernel Once we have the kernel, we need to install a couple of software packages in our Ubuntu system to be able to work with it. With the following command, everything will be installed and put in place: $ sudo apt-get install build-essential ncurses-dev u-boot-tools Time to pick a toolchain! UDOO gives you a few possibilities—you can use the same toolchain you used for the Nexus 6 or you can use the one provided by the UDOO team itself. If you decide to use the UDOO official toolchain, you can download it with a couple of terminal commands. Be sure you have already installed curl. If not, just install it with the following command: $ sudo apt-get install curl Once you have curl, you can use the following command to download the toolchain: $ curl http://download.udoo.org/files/crosscompiler/arm-fsl-linux-gnueabi.tar.gz | tar -xzf Now, you have everything in place to launch the build process: $ cd kernel $ make ARCH=arm UDOO_defconfig The following is the output: /sites/default/files/Article-Images/B04293_05_09.png The previous screenshot shows the output of the configuration process. When the default .config file is ready, you can launch the build process with the following command: $ make –j4 CROSS_COMPILE ../arm-fsl-linux-gnueabi/bin/arm-fsl-linux-gnueabi- ARCH=arm uImage modules When the build process is over, you can find the kernel image in the arch folder: $ arch/arm/boot/uImage As for the Nexus 6, we can customize the UDOO kernel using menuconfig. From the kernel source folder, launch the following command: $ make ARCH=arm menuconfig The following screenshot shows the UDOO kernel configuration menu. It's very similar to the Nexus 6 configuration menu. We have the same combination of keys to navigate, select and deselect features, and so on: Working with UDOO, the same warnings we had with the Nexus 6 apply here too—be careful while removing components from the kernel. Some of them are just meant to be there to support specific hardware, some of them, instead, are vital for the system to boot. As always, feel free to experiment, but be careful about gambling! This kind of development device makes debugging the kernel a bit easier compared to a smartphone. UDOO, as with a lot of other embedded development boards, provides a serial connection that enables you to monitor the whole boot sequence. This comes in handy if you are going to develop a driver for some hardware and you want to integrate it into your kernel or even if you are simply playing around with some custom kernel configuration. Every kernel and boot-related message will be printed to the serial console, ready to be captured and analyzed. The next screenshot shows the boot sequence for our UDOO Quad board: As you can see, there is plenty of debugging information, from the board power-on to the Android system prompt. Driver management Since version 2.6.x, Linux gives the developer the opportunity to compile parts of the kernel as separated modules that can be injected into the core, to add more features at runtime. This approach gives flexibility and freedom: there is no need to reboot the system to enjoy new features and there is no need to rebuild the whole kernel if you only need to update a specific module. This approach is widely use in the PC world, by embedded devices such as routers, smart TVs, and even by our familiar UDOO board. To code a new kernel module is no easy task and it's far from the purpose of this book: there are plenty of books on the topic and most of the skill set comes from experience. In these pages, you are going to learn about the big picture, the key points, and the possibilities. Unfortunately, Android doesn't use this modular approach: every required feature is built in a single binary kernel file, for practical and simplicity reasons. In the last few years there has been a trend to integrate into the kernel even the logic needed for Wi-Fi functionality, that was before it was loaded from a separated module during the boot sequence. As we saw with the FTDI example in the previous pages, the most practical way to add a new driver to our Android kernel is using menuconfig and building the feature as a core part of the kernel. Altering the CPU frequency Overclocking a CPU is one of the most loved topics among advanced users. The idea of getting the maximum amount of powerfrom your device is exciting. Forums and blogs are filled with discussions about overclocking and in this section we are going to have an overview and clarify a few tricky aspects that you could deal with on your journey. Every CPU is designed to work with a specific clock frequency or within a specific frequency range. Any modern CPU has the possibility to scale its clock frequency to maximize performance when needed and power consumption when performance is not needed, saving precious battery in case of our beloved mobile devices. Overclocking, then, denotes the possibility to alter this working clock frequency via software, increasing it to achieve performance higher than the one the CPU was designed for. Contrary to what we often read on unscrupulous forum threads or blogs, overclocking a CPU can be a very dangerous operation: we are forcing the CPU to work with a clock frequency that formally hasn't been tested. This could backfire on us with a device rebooting autonomously, for its own protection, or we could even damage the CPU, in the worst-case scenario. Another interesting aspect of managing the CPU clock frequency is the so-called underclock. Leveraging the CPU clock frequency scaling feature, we can design and implement scaling policies to maximize the efficiency, according to CPU load and other aspects. We could, for instance, reduce the frequency when the device is idle or in sleep mode and push the clock to the maximum when the device is under heavy load, to enjoy the maximum effectiveness in every scenario. Pushing the CPU management even further, lots of smartphone CPUs come with a multicore architecture: you can completely deactivate a core if the current scenario doesn't need it. The key concept of underclocking a CPU is adding a new frequency below the lowest frequency provided by the manufacturer. Via software, we would be able to force the device to this frequency and save battery. This process is not riskless. We could create scenarios in which the device has a CPU frequency so low that it will result in an unresponsive device or even a frozen device. As for overclocking, these are unexplored territories and only caution, experience and luck will get you to a satisfying result. An overview of the governors Linux kernel manages CPU scaling using specific policies called governors. There are a few pre-build governors in the Linux kernel, already available via menuconfig, but you can also add custom-made governors, for your specific needs. The following screenshot shows the menuconfig section of Google Nexus 6 for CPU scaling configuration: As you can see, there are six prebuild governors. Naming conventions are quite useful and make names self-explanatory: for instance, the performance governor aims to keep the CPU always at maximum frequency, to achieve the highest performance at every time, sacrificing battery life. The most popular governors on Android are definitely the ondemand and interactive governors: these are quite common in many Android-based device kernels. Our reference device, Google Nexus 6, uses interactive as the default governor. As you would expect, Google disallows direct CPU frequency management, for security reasons. There is no quick way to select a specific frequency or a specific governor on Android. However, advanced users can satisfy their curiosity or their needs with a little effort. Customizing the boot image So far, you learned how to obtain the kernel source code, how to set up the system, how to configure the kernel, and how to create your first custom kernel image. The next step is about equipping your device with your new kernel. To achieve this, we are going to analyze the internal structure of the boot.img file used by every Android device. Creating the boot image A custom ROM comes with four .img files, necessary to create a working Android system. Two of them (system.img and data.img) are compressed images of a Linux compatible filesystem. The remaining two files (boot.img and recovery.img) don't contain a standard filesystem. Instead, they are custom image files, specific to Android. These images contain a 2KB header sector, the kernel core, compressed with gzip, a ramdisk, and an optional second stated loader. Android provides further info about the internal structure of the image file in the boot.img.h file contained in the mkbootimg package in the AOSP source folder. The following screenshot shows a snippet of the content of this file: As you can see, the image contains a graphical representation of the boot.img structure. This ASCII art comes with a deeper explanation of sizes and pages. To create a valid boot.img file, you need the kernel image you have just built and a ramdisk. A ramdisk is a tiny filesystem that is mounted into the system RAM during the boot time. A ramdisk provides a set of critically important files, needed for a successful boot sequence. For instance, it contains the init file that is in charge of launching all the services needed during the boot sequence. There are two main ways to generate a boot image: We could use the mkbootimg tool We could use the Android build system Using mkbootimg gives you a lot of freedom, but comes with a lot of complexity. You would need a serious amount of command-line arguments to properly configure the generating system and create a working image. On the other hand, the Android build system comes with the whole set of configuration parameters already set and ready to go, with zero effort for us to create a working image. Just to give you a rough idea of the complexity of mkbootimg, the following screenshot shows an overview of the required parameters: Playing with something so powerful is tempting, but, as you can see, the amount of possible wrong parameters passed to mkbootimg is large. As pragmatic developers, dealing with mkbootimg is not worth the risk at the moment. We want the job done, so we are going to use the Android build system to generate a valid boot image with no effort. All that you need to do is export a new environment variable, pointing to the kernel image you have created just a few pages ago. With your trusted terminal emulator, launch: $ export TARGET_PREBUILT_KERNEL=<kernel_src>/arch/arm/boot/zImage-dtb Once you have set and exported the TARGET_PREBUILT_KERNEL environment variable, you can launch: $ make bootimage A brand new, fully customized, boot image will be created by the Android build system and will be placed in the following folder: $ target/product/<device-name>/boot.img With just a couple of commands, we have a brand new boot.img file, ready to be flashed. Using the Android build system to generate the boot image is the preferred way for all the Nexus devices and for all those devices, such as the UDOO, that are designed to be as close as possible to an official Google device. For all those devices on the market that are compliant to this philosophy, things start to get tricky, but not impossible. Some manufactures take advantage of the Apache v2 license and don't provide the whole Android source code. You could find yourself in a scenario where you only have the kernel source code and you won't be able to leverage the Android build system to create your boot image or even understand how boot.img is actually structured. In these scenarios, one possible approach could be to pull the boot.img from a working device, extract the content, replace the default kernel with your custom version, and recreate boot.img using mkbootimg: easier said than done. Right now, we want to focus on the main scenario, dealing with a system that is not fighting us. Upgrading the new boot image Once you have your brand new, customized boot image, containing your customized kernel image, you only need to flash it to your device. We are working with Google devices or, at least, Google-compatible devices, so you will be able to use fastboot to flash your boot.img file to your device. To be able to flash the image to the device, you need to put the device in fastboot mode, also known as bootloader mode. Once your device is in fastboot mode, you can connect it via USB to your host computer. Fire up a terminal emulator and launch the command to upgrade the boot partition: $ sudo fastboot flash boot boot.img In a few seconds, fastboot will replace the content of the device boot partition with the content of your boot.img file. When the flashing process is successfully over, you can reboot your device with: $ sudo fastboot reboot The device will reboot using your new kernel and, thanks to the new USB TTL support that you added a few pages ago, you will be able to monitor the whole boot sequence with your terminal emulator. Android boot sequence To fully understand all Android internals, we are going to learn how the whole boot sequence works: from the power-on to the actual Android system boot. The Android boot sequence is similar to any other embedded system based on Linux: in a very abstract way, after the power-on, the system initializes the hardware, loads the kernel, and finally the Android framework. Any Linux-based system undergoes a similar process during its boot sequence: your Ubuntu computer or even your home DSL router. In the next sections, we are going to dive deeper in to these steps to fully comprehend the operating system we love so much. Internal ROM – bios When you press the power button on your device, the system loads a tiny amount of code, stored inside a ROM memory. You can think about this as an equivalent of the BIOS software you have in your PC. This software is in charge of setting up all the parameters for CPU clock and running the RAM memory check. After this, the system loads the bootloader into memory and launches it. An overview of bootloader So far, the bootloader has been loaded into the RAM memory and started. The bootloader is in charge of loading the system kernel into the RAM memory and launching it, to continue the boot sequence. The most popular bootloader software for Android devices is U-Boot, the Universal Bootloader. U-Boot is widely used in all kinds of embedded systems: DSL routers, smart TVs, infotainment systems, for example. U-boot is open source software and its flexibility to be customized for any device is definitely one of the reasons for its popularity. U-boot's main task is to read the kernel image from the boot partition, load it into the RAM memory, and run it. From this moment on, the kernel is in charge of finishing the boot sequence. You could think about U-boot on Android like GRUB on your Ubuntu system: it reads the kernel image, decompresses it, loads it into the RAM memory, and executes it. The following diagram gives you a graphical representation of the whole boot sequence as on an embedded Linux system, an Android system, and a Linux PC: The kernel After the bootloader loads the kernel, the kernel's first task is to initialize the hardware. With all the necessary hardware properly set up, the kernel mounts the ramdisk from boot.img and launches init. The Init process In a standard Linux system, the init process takes care of starting all the core services needed to boot the system. The final goal is to complete the boot sequence and start the graphical interface or the command line to make the system available to the user. This whole process is based on a specific sequence of system scripts, executed in a rigorous order to assure system integrity and proper configuration. Android follows the same philosophy, but it acts in a different way. In a standard Android system, the ramdisk, contained in the boot.img, provides the init script and all the scripts necessary for the boot. The Android init process consists of two main files: init.rc init.${ro.hardware}.rc The init.rc file is the first initialization script of the system. It takes care of initializing those aspects that are common to all Android systems. The second file is very hardware specific. As you can guess, ${ro.hardware} is a placeholder for the reference of a particular hardware where the boot sequence is happening. For instance, ${ro.hardware} is replaced with goldfinsh in the emulator boot configuration. In a standard Linux system, the init sequence executes a set of bash scripts. These bash scripts start a set of system services. Bash scripting is a common solution for a lot of Linux systems, because it is very standardized and quite popular. Android systems use a different language to deal with the initialization sequence: Android Init Language. The Android init language The Android team chose to not use Bash for Android init scripts, but to create its own language to perform configurations and services launches. The Android Init Language is based on five classes of statements: Actions Commands Services Options Imports Every statement is line-oriented and is based on specific tokens, separated by white spaces. Comment lines start with a # symbol. Actions An Action is a sequence of commands bound to a specific trigger that's used to execute the particular action at a specific moment. When the desired event happens, the Action is placed in an execution queue, ready to be performed. This snippet shows an example of an Action statement: on <trigger> [&& <trigger>]* <command> <command> <command> Actions have unique names. If a second Action is created with the same name in the same file, its set of commands is added to the first Action commands, set and executed as a single action. Services Services are programs that the init sequence will execute during the boot. These services can also be monitored and restarted if it's mandatory they stay up. The following snippet shows an example of a service statement: service <name> <pathname> [ <argument> ]* <option> <option> ... Services have unique names. If in the same file, a service with a nonunique name exists, only the first one is evaluated as valid; the second one is ignored and the developer is notified with an error message. Options Options statements are coupled with services. They are meant to influence how and when init manages a specific service. Android provides quite an amount of possible options statements: critical: This specifies a device-critical service. The service will be constantly monitored and if it dies more than four times in four minutes, the device will be rebooted in Recovery Mode. disabled: This service will be in a default stopped state. init won't launch it. A disabled service can only be launched manually, specifying it by name. setenv <name> <value>: This sets an environment variable using name and value. socket <name> <type> <perm> [ <user> [ <group> [ <seclabel> ] ] ]: This command creates a Unix socket, with a specified name, (/dev/socket/<name>) and provides its file descriptor the specified service. <type> specifies the type of socket: dgram, stream, or seqpacket. Default <user> and <group> are 0. <seclabel> specifies the SELinx security context for the created socket. user <username>: This changes the username before the service is executed. The default username is root. group <groupname> [ <groupname> ]*: This changes the group name before the service is executed. seclabel <seclabel>: This changes the SELinux level before launching the service. oneshot: This disables the service monitoring and the service won't be restarted when it terminates. class <name>: This specifies a service class. Classes of services can be launched or stopped at the same time. A service with an unspecified class value will be associated to the default class. onrestart: This executes a command when the service is restarted. writepid <file...>: When a services forks, this option will write the process ID (PID) in a specified file. Triggers Triggers specify a condition that has to be satisfied to execute a particular action. They can be event triggersor property triggers. Event triggers can be fired by the trigger command or by the QueueEventTrigger() function. The example event triggers are boot and late-init. Property triggers can be fired when an observed property changes value. Every Action can have multiple Property triggers, but only one Event trigger; refer to the following code for instance: on boot && property_a=b This Action will be executed when the boot event is triggered and the property a is equal to b. Commands The Command statement specifies a command that can be executed during the boot sequence, placing it in the init.rc file. Most of these commands are common Linux system commands. The list is quite extensive. Let's look at them in detail: bootchart_init: This starts bootchart if it is properly configured. Bootchart is a performance monitor and can provide insights about the boot performance of a device. chmod <octal-mode-permissions> <filename>: This changes file permissions. chown <owner> <group> <filename>: This changes the owner and the group for the specified file. class_start <serviceclass>: This starts a service specified by its class name. class_stop <serviceclass>: This stops and disables a service specified by its class name. class_reset <serviceclass>: This stops a service specified by its class name. It doesn't disable the service. copy <src> <dst>: This copies a source file to a new destination file. domainname <name>: This sets the domain name. enable <servicename>: This starts a service by its name. If the service is already queued to be started, then it starts the service immediately. exec [<seclabel>[<user>[<group> ]* ]] -- <command> [ <argument> ]*: This forks and executes the specified command. The execution is blocking: no other command can be executed in the meantime. export <name> <value>: This sets and exports an environment variable. hostname <name>: This sets the hostname. ifup <interface>: This enables the specified network interface. insmod <path>: This loads the specified kernel module. load_all_props: This loads all the system properties. load_persist_props: This loads the persistent properties, after the successful decryption of the /data partition. loglevel <level>: This sets the kernel log level. mkdir <path> [mode] [owner] [group]: This creates a folder with the specified name, permissions, owner, and group. The defaults are 755 as permissions, and root as owner and group. mount_all <fstab>: This mounts all the partitions in the fstab file. mount <type> <device> <dir> [ <flag> ]* [<options>]: This mounts a specific device in a specific folder. A few mount flags are available: rw, ro, remount, noatime, and all the common Linux mount flags. powerctl: This is used to react to changes of the sys.powerctl system parameter, critically important for the implementation of the reboot routing. restart <service>: This restarts the specified service. rm <filename>: This deletes the specified file. rmdir <foldername>: This deletes the specified folder. setpropr <name> <value>: This sets the system property with the specified name with the specified value. start <service>: This starts a service. stop <service>: This stops a service. swapon_all <fstab>: This enables the swap partitions specified in the fstab file. symlink <target> <path>: This creates a symbolic link from the target file to the destination path. sysclktz <mins_west_of_gtm>: This sets the system clock. trigger <event>: This programmatically triggers the specified event. wait <filename > [ <timeout> ]: This monitors a path for a file to appear. A timeout can be specified. If not, the default timeout value is 5 seconds. write <filename> <content>: This writes the specified content to the specified file. If the file doesn't exist, it creates the file. If the file already exists, it won't append the content, but it will override the whole file. Imports Imports specify all the external files that are needed in the current file and imports them: import <path> The previous snippet is an example of how the current init script can be extended, importing an external init script. path can be a single file or even a folder. In case path is a folder, all the files that exists in the first level of the specified folder will be imported. The command doesn't act recursively on folders: nested folders must be imported programmatically one by one. Summary In this article, you learned how to obtain the Linux kernel for your device, how to set up your host PC to properly build your custom kernel, how to add new features to the kernel, build it, package it, and flash it to your device. You learned how the Android boot sequence works and how to manipulate the init scripts to customize the boot sequence. Resources for Article: Further resources on this subject: Virtualization [article] Building Android (Must know) [article] Network Based Ubuntu Installations [article]

In this article by Dimitri Aivaliotis, author Mastering NGINX - Second Edition, The NGINX configuration file follows a very logical format. Learning this format and how to use each section is one of the building blocks that will help you to create a configuration file by hand. Constructing a configuration involves specifying global parameters as well as directives for each individual section. These directives and how they fit into the overall configuration file is the main subject of this article. The goal is to understand how to create the right configuration file to meet your needs. (For more resources related to this topic, see here.) The basic configuration format The basic NGINX configuration file is set up in a number of sections. Each section is delineated as shown: <section> { <directive> <parameters>; } It is important to note that each directive line ends with a semicolon (;). This marks the end of line. The curly braces ({}) actually denote a new configuration context, but we will read these as sections for the most part. The NGINX global configuration parameters The global section is used to configure the parameters that affect the entire server and is an exception to the format shown in the preceding section. The global section may include configuration directives, such as user and worker_processes, as well as sections, such as events. There are no open and closing braces ({}) surrounding the global section. The most important configuration directives in the global context are shown in the following table. These configuration directives will be the ones that you will be dealing with for the most part. Global configuration directives Explanation user The user and group under which the worker processes run is configured using this parameter. If the group is omitted, a group name equal to that of the user is used. worker_processes This directive shows the number of worker processes that will be started. These processes will handle all the connections made by the clients. Choosing the right number depends on the server environment, the disk subsystem, and the network infrastructure. A good rule of thumb is to set this equal to the number of processor cores for CPU-bound loads and to multiply this number by 1.5 to 2 for the I/O bound loads. error_log This directive is where all the errors are written. If no other error_log is given in a separate context, this log file will be used for all errors, globally. A second parameter to this directive indicates the level at which (debug, info, notice, warn, error, crit, alert, and emerg) errors are written in the log. Note that the debug-level errors are only available if the --with-debug configuration switch is given at compilation time. pid This directive is the file where the process ID of the main process is written, overwriting the compiled-in default. use This directive indicates the connection processing method that should be used. This will overwrite the compiled-in default and must be contained in an events context, if used. It will not normally need to be overridden, except when the compiled-in default is found to produce errors over time. worker_connections This directive configures the maximum number of simultaneous connections that a worker process may have opened. This includes, but is not limited to, client connections and connections to upstream servers. This is especially important on reverse proxy servers—some additional tuning may be required at the operating system level in order to reach this number of simultaneous connections. Here is a small example using each of these directives: # we want nginx to run as user 'www' user www; # the load is CPU-bound and we have 12 cores worker_processes 12; # explicitly specifying the path to the mandatory error log error_log /var/log/nginx/error.log; # also explicitly specifying the path to the pid file pid /var/run/nginx.pid; # sets up a new configuration context for the 'events' module events { # we're on a Solaris-based system and have determined that nginx # will stop responding to new requests over time with the default # connection-processing mechanism, so we switch to the second-best use /dev/poll; # the product of this number and the number of worker_processes # indicates how many simultaneous connections per IP:port pair are # accepted worker_connections 2048; } This section will be placed at the top of the nginx.conf configuration file. Using the include files The include files can be used anywhere in your configuration file, to help it be more readable and to enable you to reuse parts of your configuration. To use them, make sure that the files themselves contain the syntactically correct NGINX configuration directives and blocks; then specify a path to those files: include /opt/local/etc/nginx/mime.types; A wildcard may appear in the path to match multiple files: include /opt/local/etc/nginx/vhost/*.conf; If the full path is not given, NGINX will search relative to its main configuration file. A configuration file can easily be tested by calling NGINX as follows: nginx -t -c <path-to-nginx.conf> This command will test the configuration, including all files separated out into the include files, for syntax errors. Sample configuration The following code is an example of an HTTP configuration section: http { include /opt/local/etc/nginx/mime.types; default_type application/octet-stream; sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; server_names_hash_max_size 1024; } This context block would go after any global configuration directives in the nginx.conf file. The virtual server section Any context beginning with the keyword server is considered as a virtual server section. It describes a logical separation of a set of resources that will be delivered under a different server_name directive. These virtual servers respond to the HTTP requests, and are contained within the http section. A virtual server is defined by a combination of the listen and server_name directives. The listen directive defines an IP address/port combination or path to a UNIX-domain socket: listen address[:port]; listen port; listen unix:path; The listen directive uniquely identifies a socket binding under NGINX. There are a number of optional parameters that listen can take: The listen parameters Explanation Comments default_server This parameter defines this address/port combination as being the default value for the requests bound here.   setfib This parameter sets the corresponding FIB for the listening socket. This parameter is only supported on FreeBSD and not for UNIX-domain sockets. backlog This parameter sets the backlog parameter in the listen() call. This parameter defaults to -1 on FreeBSD and 511 on all other platforms. rcvbuf This parameter sets the SO_RCVBUF parameter on the listening socket.   sndbuf This parameter sets the SO_SNDBUF parameter on the listening socket.   accept_filter This parameter sets the name of the accept filter to either dataready or httpready. This parameter is only supported on FreeBSD. deferred This parameter sets the TCP_DEFER_ACCEPT option to use a deferred accept() call. This parameter is only supported on Linux. bind This parameter makes a separate bind() call for this address/port pair. A separate bind() call will be made implicitly if any of the other socket-specific parameters are used. ipv6only This parameter sets the value of the IPV6_ONLY parameter. This parameter can only be set on a fresh start and not for UNIX-domain sockets. ssl This parameter indicates that only the HTTPS connections will be made on this port. This parameter allows for a more compact configuration. so_keepalive This parameter configures the TCP keepalive connection for the listening socket.   The server_name directive is fairly straightforward and it can be used to solve a number of configuration problems. Its default value is "", which means that a server section without a server_name directive will match a request that has no Host header field set. This can be used, for example, to drop requests that lack this header: server { listen 80; return 444; } The nonstandard HTTP code, 444, used in this example will cause NGINX to immediately close the connection. Besides a normal string, NGINX will accept a wildcard as a parameter to the server_name directive: The wildcard can replace the subdomain part: *.example.com The wildcard can replace the top-level domain part: www.example.* A special form will match the subdomain or the domain itself: .example.com (matches *.example.com as well as example.com) A regular expression can also be used as a parameter to server_name by prepending the name with a tilde (~): server_name ~^www.example.com$; server_name ~^www(d+).example.(com)$; The latter form is an example using captures, which can later be referenced (as $1, $2, and so on) in further configuration directives. NGINX uses the following logic when determining which virtual server should serve a specific request: Match the IP address and port to the listen directive. Match the Host header field against the server_name directive as a string. Match the Host header field against the server_name directive with a wildcard at the beginning of the string. Match the Host header field against the server_name directive with a wildcard at the end of the string. Match the Host header field against the server_name directive as a regular expression. If all the Host headers match fail, direct to the listen directive marked as default_server. If all the Host headers match fail and there is no default_server, direct to the first server with a listen directive that satisfies step 1. This logic is expressed in the following flowchart: The default_server parameter can be used to handle requests that would otherwise go unhandled. It is therefore recommended to always set default_server explicitly so that these unhandled requests will be handled in a defined manner. Besides this usage, default_server may also be helpful in configuring a number of virtual servers with the same listen directive. Any directives set here will be the same for all matching server blocks. Locations – where, when, and how The location directive may be used within a virtual server section and indicates a URI that comes either from the client or from an internal redirect. Locations may be nested with a few exceptions. They are used for processing requests with as specific configuration as possible. A location is defined as follows: location [modifier] uri {...} Or it can be defined for a named location: location @name {…} A named location is only reachable from an internal redirect. It preserves the URI as it was before entering the location block. It may only be defined at the server context level. The modifiers affect the processing of a location in the following way: Location modifiers Handling = This modifier uses exact match and terminate search. ~ This modifier uses case-sensitive regular expression matching. ~* This modifier uses case-insensitive regular expression matching. ^~ This modifier stops processing before regular expressions are checked for a match of this location's string, if it's the most specific match. Note that this is not a regular expression match—its purpose is to preempt regular expression matching. When a request comes in, the URI is checked against the most specific location as follows: Locations without a regular expression are searched for the most-specific match, independent of the order in which they are defined. Regular expressions are matched in the order in which they are found in the configuration file. The regular expression search is terminated on the first match. The most-specific location match is then used for request processing. The comparison match described here is against decoded URIs; for example, a %20 instance in a URI will match against a "" (space) specified in a location. A named location may only be used by internally redirected requests. The following directives are found only within a location: Location-only directives Explanation alias This directive defines another name for the location, as found on the filesystem. If the location is specified with a regular expression, alias should reference captures defined in that regular expression. The alias directive replaces the part of the URI matched by the location such that the rest of the URI not matched will be searched for in that filesystem location. Using the alias directive is fragile when moving bits of the configuration around, so using the root directive is preferred, unless the URI needs to be modified in order to find the file. internal This directive specifies a location that can only be used for internal requests (redirects defined in other directives, rewrite requests, error pages, and so on.) limit_except This directive limits a location to the specified HTTP verb(s) (GET also includes HEAD). Additionally, a number of directives found in the http section may also be specified in a location. Refer to Appendix A, Directive Reference, for a complete list. The try_files directive deserves special mention here. It may also be used in a server context, but will most often be found in a location. The try_files directive will do just that—try files in the order given as parameters; the first match wins. It is often used to match potential files from a variable and then pass processing to a named location, as shown in the following example: location / { try_files $uri $uri/ @mongrel; } location @mongrel { proxy_pass http://appserver; } Here, an implicit directory index is tried if the given URI is not found as a file and then processing is passed on to appserver via a proxy. We will explore how best to use location, try_files, and proxy_pass to solve specific problems throughout the rest of the article. Locations may be nested except in the following situations: When the prefix is = When the location is a named location Best practice dictates that regular expression locations be nested inside the string-based locations. An example of this is as follows: # first, we enter through the root location / { # then we find a most-specific substring # note that this is not a regular expression location ^~ /css { # here is the regular expression that then gets matched location ~* /css/.*.css$ { } } } Summary In this article, we saw how the NGINX configuration file is built. Its modular nature is a reflection, in part, of the modularity of NGINX itself. A global configuration block is responsible for all aspects that affect the running of NGINX as a whole. There is a separate configuration section for each protocol that NGINX is responsible for handling. We may further define how each request is to be handled by specifying servers within those protocol configuration contexts (either http or mail) so that requests are routed to a specific IP address/port. Within the http context, locations are then used to match the URI of the request. These locations may be nested, or otherwise ordered to ensure that requests get routed to the right areas of the filesystem or application server. Resources for Article: Further resources on this subject: Zabbix Configuration [article] Configuring the essential networking services provided by pfSense [article] Configuring a MySQL linked server on SQL Server 2008 [article]

In this article by Nilesh Nimkar, the author of the book Practical OneOps, we will look at a few steps that will help you manage both kinds of installations. (For more resources related to this topic, see here.) Upgrading OneOps with minimal downtime As mentioned preceding you might be running a stand-alone instance of OneOps or an enterprise instance. For both types, you will have to use different strategies to update the OneOps code. In general, it is easier and straightforward to update a standalone instance rather than a enterprise instance. Your strategies to update and the branch or tag of code that you will use will also differ based on kind of system that you have. Updating a standalone OneOps install If you have a standalone installation, it's possible you created it in one of the several ways. You either installed it using Vagrant or you used the Amazon Machine Images (AMI). It is also possible that you built your own installation on another cloud like Google, Azure or Rackspace. Irrespective of the way your instance of OneOps was created, the steps to upgrade it remains the same and are very simple. When you setup OneOps two scripts are run by the setup process, oo-preqs.sh and oo_setup.sh. Once an instance is setup, both these scripts are also copied to /home/oneops directory on the server. Of these two scripts, oo_setup.sh can be used to update an OneOps standalone install at any time. You need an active internet connection to upgrade OneOps. You can see the list of releases in the OneOps git repository for any of the OneOps components. For example releases for sensor can be seen at https://github.com/oneops/sensor/releases Release candidates have a RC1 at the end and stable releases have — STABLE at the end. If you want to install a particular release, like 16.09.29-RC1 then invoke the script and pass the release number as the argument. Passing master will build the master branch and will build and install the latest and greatest code. This is great to get all the latest features and bug fixes but this will also make your installation susceptible to new bugs. ./oo_setup.sh master Ensure that the script is invoked as root. Instead of running as sudo, it helps if you are logged in as root with: sudo su - After the script is invoked, it will do a bunch of things to upgrade your OneOps. First it sets three variables: OO_HOME which is set to /home/oneops BUILD_BASE which is set to /home/oneops/build GITHUB_URL which is set to https://github.com/oneops All the builds will take place under BUILD_BASE. Under the BUILD_BASE, the script then checks if dev-tools exists. If it does, it updates it to the latest code by doing a git pull on it. If it does not, then it does a git clone and gets a latest copy from GitHub. The dev-tools repository has a set of tools for core OneOps developers. The most important of which are under the setupscripts sub directory. The script then copies all the scripts from under the setupscripts sub directory to OO_HOME directory. Once done it invokes the oneops_build.sh script. If you passed any build tag to oo_setup.sh script, that tag is passed on to oneops_build.sh script as is. The oneops_build.sh script is a control script so the speak. What it means is in turn it will invoke a bunch of other scripts which will shutdown services, pull and build the OneOps code, install the built code and then restart the services once done. Most of the scripts that run henceforth set and export a few variables again. Namely OO_HOME, BUILD_BASE and GITHUB_URL. Another variable that is set is SEARCH_SITE whose value is always set to localhost. The first thing the script does is to shutdown Cassandra on the server to conserve memory on the server and reduce the load during the build since the build itself is very memory and CPU intensive. It also marks the start time of the script. Next it runs the install_build_srvr.sh script by passing the build tag that was passed to the original script. This is a very innovative script which does a quick installation of Jenkins, installs various plugins to Jenkins, runs various jobs to do builds, monitors the jobs for either success or failure and then shuts down Jenkins all in an automated fashion. If you have your own Jenkins installation, I highly recommend you read through this script as this will give you great ideas for your own automation of installation, monitoring and controlling of Jenkins. As mentioned in preceding, the install_build_srvr.sh script sets a bunch of variables first. It then clones the git repository called build-wf from the BUILD_BASE if it does not already exist. If it does exist, it does a git pull to update the code. Outside of a Docker container, the build-wf is a most compact Jenkins installation you will find. You can check it out at the URL following: https://github.com/oneops/build-wf It consists of a Rakefile to download and install Jenkins and its associated plugins, a config.xml that configures it, a plugins.txt that provides a list of plugins and a jobs directory with all the associated jobs in it. If the script detects a Jenkins server that is already present and a build is already in progress, it cleanly attempts to shut down the existing Jenkins server. It then attempts to install the latest Jenkins jar using the following command: rake install Once the installation is done a dist directory is created to store the resulting build packages. After setting the path to local maven, the server is brought up using the command following: rake server If you did not specify what revision to build, the last stable build is used. The actual release revision itself is hardcoded in this script. Every time a stable release is made, this file is manually changed and the release version is updated and the file is checked in. After the server comes up, it is available at port 3001 if you are running on any cloud. If you are running a Vagrant setup, it will be mapped to port 3003. If you connect to one of these ports on your machines via your browser, you should be able to see your Jenkins in action. The script calls the job oo-all-oss via curl using Jenkins REST API. The oo-all-oss is a master job that in turn builds all of OneOps components, including the database components. Even the installation of Jenkins plugins is done via a Jenkins job called Jenkins-plugin. The script then goes into an infinite loop and keeps checking the job status till the jobs are done. Once all jobs are finished or if an error is encountered, the server is shutdown using rake stop Once the build completes, the Cassandra sever is started again. Once it starts the Cassandra service the script start deploying all the built artifacts. The first artifact to be deployed is the database artifact. For that it runs the init_db.sh script. This script first creates the three main schemas, namely kloopzapp, kloopzdb and activitidb. Since you are upgrading an existing installation, this script may very well give an error. Next the script will run a bunch of database scripts which will create tables, partitions, functions and other ddl statements. Again since you are upgrading any errors here can be safely ignored. Next to be installed, is the display. The script backs up the current display from /opt/oneops/app to /opt/oneops/~app in case a rollback is needed. It then copies and untars the newly built package. Using rake, it detects if the Rails database is setup. If the database version is returned as 0 then rake db:setup command is run to setup a brand new database. Or else rake db:migrate command is run to migrate and upgrade the database. The next component to get installed is amq. This is done by calling the script deploy_amq.sh. The amq gets installed in the directory /opt/activemq. Before installation the activemq service is stopped. The script the copies over the amq-config and amqplugin-fat jar. It also takes a backup of the old configuration and overwrites it with new configuration. After that the service is started again. After AMQ, the script installs all the webapps under Tomcat. Tomcat itself is installed under /usr/local/tomcat7 and all the webapps get installed under /usr/local/tomcat7/webapps. Before copying over all the war files, the tomcat service is stopped. The script also creates directories that the controller, publisher and transmitter rely on for successful operation. Once the wars are copied Tomcat service is started again. Tomcat, at this point will automatically deploy the services. After the web services are deployed the script deploys the search service. Before deployment, the search-consumer service is stopped. The search.jar and the startup script is then copied to /opt/oneops-search directory and the search-consumer service is started again. As a final step in deployment, the OneOps Admin gem is deployed. The OneOps Admin gem contains two commands that help administer OneOps from the backend. These commands are inductor and circuit. The script then either updates the circuit repository if it exists or clones it if it does not from https://github.com/oneops/circuit-oneops-1 and installs it. After successfully installing the circuit an inductor is created using the shared queue using the command following. This command is also a great reference for you should you wish to create your own inductors for testing. inductor add --mqhost localhost --dns on --debug on --daq_enabled true --collector_domain localhost --tunnel_metrics on --perf_collector_cert /etc/pki/tls/logstash/certs/logstash-forwarder.crt --ip_attribute public_ip --queue shared --mgmt_url http://localhost:9090 --logstash_cert_location /etc/pki/tls/logstash/certs/logstash-forwarder.crt --logstash_hosts vagrant.oo.com:5000 --max_consumers 10 --local_max_consumers 10 --authkey superuser:amqpass --amq_truststore_location /opt/oneops/inductor/lib/client.ts --additional_java_args "" --env_vars "" After installing the inductor, the display service is started and the standalone OneOps upgrade is complete. Updating a Enterprise OneOps Install Updating an enterprise OneOps install takes a different approach for a few different reasons. First of all in an enterprise install all the services get installed on their own instances. Secondly, since an enterprise install caters to an entire enterprise, stability, availability and scalability are always a issue. So here are a few things that you should remember before you upgrade your enterprise install. Ensure you have your own Jenkins build server and it uploads the artifacts to your own Nexus repository. Ensure this Nexus repository is configured in the OneOps that manages your enterprise OneOps installation. Ensure you use a stable build and not a Release Candidate or the master build. This way you will have a well tested build for your enterprise. Make sure your backup server is configured and OneOps is being regularly backed up. Although the downtime should be minimal to none, make sure you do the upgrade during the least busy time to avoid any unforeseen events. If you have more than one OneOps installations, it is prudent to direct traffic to the second installation while one is being updated. With these things in mind the sequence for updating the various components is pretty much the same as updating a standalone OneOps install. However the steps involved are a bit different. The first thing you need to do, as mentioned preceding, is to choose an appropriate stable release that you want to deploy. Once you choose that, go to OneOps that manages your enterprise installation and click on the OneOps assembly. Select Design from the left hand side menu and then select Variables from the center screen. From the numerous variables you see, the one that you want to modify is called Version. Click on it and then click Edit in the upper right hand corner. Click Save. Once the changes are saved, you can go ahead and commit your changes. You will notice that all the components derive their local version variable from the global version variable. At this point if you click on Transition, and attempt a deployment, OneOps will generate a deployment plan which will have the latest revision of all the components that need the upgrade. Go ahead and click Deploy. OneOps should do the rest. Configuring database backups As seen so far, OneOps has a complex architecture and relies on many databases to provide optimum functionality as we have seen before. Again as with deployment, for database backup the steps needed to backup a single machine install and an enterprise install are different. Backup a standalone OneOps install For a standalone install the three main postgres databases you need to backup are activitidb, kloopzapp, and kloopzdb. You can access these databases directly by logging in to your OneOps server and then doing a sudo as the postgres user. # sudo su – postgres -bash-4.2$ psql Postgres=# l Once you issue these commands you can see these databases listed along with the default postgres database. Now you can design chef recipes to take backups or install puppet or ansible and automate the backup process. However in accordance with the KISS principle the simplest way you can setup backups are to use the built in postgres command The pg_dump command for single database backup or pg_dumpall for a all database backups. You can add a cron job to run these commands nightly and another cron job to scp the dumped files and delete the local copies. The KISS is an acronym coined by the US Navy in 1960 for a design principle that states that systems work best if the design is kept simple and unnecessary complexity is avoided. Please look it up online. Search for KISS Principle. As time goes by your database size will also increase. To tackle that you can pipe your backup commands directly to a compression program. pg_dumpall | gzip filename.gz Similarly you can restore the database with using the exact reverse of that command. gunzip filename.gz | pg_restore Backup an enterprise OneOps Install Again an enterprise OneOps install, as opposed to a standalone OneOps install comes with backups built in. To make the backups work, you have to setup a few things correctly to begin with. Firstly you have to setup the BACKUP-HOST global variable to point to a host that has plenty of storage attached to it. Once the variable is set, the value trickles down to the database components as local variables derived from the global variable. All backups taken are then copied to this host. For example, following is the screenshot for this variable for CMSDB: Once this is done, OneOps sets up automated jobs for database backups. These jobs are actually shell scripts which are wrappers over chef recipes for the database snapshot backup. Summary In this article we saw a few steps that will help you manage both kinds of installations. However as DevOps you will have to manage not only assemblies but the OneOps system itself. Depending on the size of the organization and the complexity of the deployments you handle, you may opt to choose either a single server installation or an enterprise install. Resources for Article: Further resources on this subject: Managing Application Configuration [article] Let's start with Extending Docker [article] Directory Services [article]

The frontend space is indeed crowded, but none of the more popular solutions are really convincing to me. I feel Angular is bloated and the double binding is not for me. I also do not like React and its syntax. Riot is, as stated by their creators, "A React-like user interface micro-library" with simpler syntax that is five times smaller than React. What we are going to learn We are going to build a simple Riot application backed by Express, using Jade as our template language. The backend will expose a simple REST API, which we will consume from the UI. We are not going to use any other dependency like JQuery, so this is also a good chance to try XMLHttpRequest2. I deliberately ommited the inclusion of a client package manager like webpack or jspm because I want to focus on the Expressjs + Riotjs. For the same reason, the application data is persisted in memory. Requirements You just need to have any recent version of node.js(+4), text editor of your choice, and some JS, Express and website development knowledge. Project layout Under my project directory we are going to have 3 directories: * public For assets like the riot.js library itself. * views Common in most Express setup, this is where we put the markup. * client This directory will host the Riot tags (we will see more of that later) We will also have the package.json, our project manifesto, and an app.js file, containing the Express application. Our Express server exposes a REST API; its code can be found in api.js. Here is how the layout of the final project looks: ├── api.js ├── app.js ├── client │ ├── tag-todo.jade │ └── tag-todo.js ├── package.json ├── node_modules ├── public │ └── js │ ├── client.js │ └── riot.js └── views └── index.jade Project setup Create your project directory and from there run the following to install the node.js dependencies: $ npm init -y $ npm install --save body-parser express jade And the application directories: $ mkdir -p views public/js client Start! Lets start by creating the Express application file, app.js: 'use strict'; const express = require('express'), app = express(), bodyParser = require('body-parser'); // Set the views directory and template engine app.set('views', __dirname + '/views'); app.set('view engine', 'jade'); // Set our static directory for public assets like client scripts app.use(express.static('public')); // Parses the body on incoming requests app.use(bodyParser.json()); // Pretty prints HTML output app.locals.pretty = true; // Define our main route, HTTP "GET /" which will print "hello" app.get('/', function (req, res) { res.send('hello'); }); // Start listening for connections app.listen(3000, function (err) { if (err) { console.error('Cannot listen at port 3000', err); } console.log('Todo app listening at port 3000'); }); The #app object we just created, is just a plain Express application. After setting up the application we call the listen function, which will create an HTTP server listening at port 3000. To test our application setup, we open another terminal, cd to our project directory and run $ node app.js. Open a web browser and load http://localhost:3000; can you read "hello"? Node.js will not reload the site if you change the files, so I recommend you to install nodemon. Nodemon monitors your code and reloads the site on every change you perform on the JS source code. The command,$ npm install -g nodemon, installs the program on your computer globally, so you can run it from any directory. Okay, kill our previously created server and start a new one with $ nodemon app.js. Our first Riot tag Riot allows you to encapsulate your UI logic in "custom tags". Tag syntax is pretty straightforward. Judge for yourself. <employee> <span>{ name }</span> </employee> Custom tags can contain code and can be nested as showed in the next code snippet: <employeeList> <employee each="{ items }" onclick={ gotoEmployee } /> <script> gotoEmployee (e) { var item = e.item; // do something } </script> </employeeList> This mechanism enables you to build complex functionality from simple units. Of course you can find more information at their documentation. On the next steps we will create our first tag: ./client/tag-todo.jade. Oh, we have not yet downloaded Riot! Here is the non minified Riot + compiler download. Download it to ./public/js/riot.js. Next step is to create our index view and tell our app to serve it. Locate / router handler, remove the res.send('hello) ', and update to: // Define our main route, HTTP "GET /" which will print "hello" app.get('/', function (req, res) { res.render('index'); }); Now, create the ./views/index.jade file: doctype html html head script(src="/js/riot.js") body h1 ToDo App todo Go to your browser and reload the page. You can read the big "ToDo App" but nothing else. There is a <todo></todo> tag there, but since the browser does not understand, this tag is not rendered. Let's tell Riot to mount the tag. Mount means Riot will use <todo></todo> as a placeholder for our —not yet there— todo tag. doctype html html head script(src="/js/riot.js") body h1 ToDo App script(type="riot/tag" src="/tags/todo.tag") todo script. riot.mount('todo'); Open your browser's dev console and reload the page. riot.mount failed because there was no todo.tag. Tags can be served in many ways, but I choose to serve them as regular Express templates. Of course, you can serve it as static assets or bundled. Just below the / route handler, add the /tags/:name.tag handler. // "/" route handler app.get('/', function (req, res) { res.render('index'); }); // tag route handler app.get('/tags/:name.tag', function (req, res) { var name = 'tag-' + req.params.name; res.render('../client/' + name); }); Now create the tag in ./client/tag-todo.jade: todo form(onsubmit="{ add }") input(type="text", placeholder="Needs to be done", name="todo") And reload the browser again. Errors gone and a new form in your browser. onsubmit="{ add }" is part of Riot's syntax and means on submit call the add function. You can add mix implementation with the markup, but I rather prefer to split markup from code. In Jade (and any other template language),it is trivial to include other files, which is exactly what we are going to do. Update the file as: todo form(onsubmit="{ add }") input(type="text", placeholder="Needs to be done", name="todo") script include tag-todo.js And create ./client/tag-todo.js with this snippet: 'use strict'; var self = this; var api = self.opts; When the tag gets mounted by Riot, it gets a context. That is the reason for var self = this;. That context can include the opts object. opts object can be anything of your choice, defined at the time you mount the tag. Let’s say we have an API object and we pass it to riot.mount as the second option at the time we mount the tag, that isriot.mount('todo', api). Then, at the time the tag is rendered this.opts will point to the api object. This is the mechanism we are going to use to expose our client api with the todo tag. Our form is still waiting for the add function, so edit the tag-todo.js again and append the following: self.add = function (e) { var title = self.todo.value; console.log('New ToDo', title); }; Reload the page, type something at the text field, and hit enter. The expected message should appear in your browser's dev console. Implementing our REST API We are ready to implement our REST API on the Express side. Create ./api.js file and add: 'use strict'; const express = require('express'); var app = module.exports = express(); // simple in memory DB var db = []; // handle ToDo creation app.post('/', function (req, res) { db.push({ title: req.body.title, done: false }); let todoID = db.length - 1; // mountpath = /api/todos/ res.location(app.mountpath + todoID); res.status(201).end(); }); // handle ToDo updates app.put('/', function (req, res) { db[req.body.id] = req.body; res.location('/' + req.body.id); res.status(204).end(); }); Our API supports ToDo creation/update, and it is architected as an Express sub application. To mount it, we just need to update app.js for the last time. Update the require block at app.js to: const express = require('express'), api = require('./api'), app = express(), bodyParser = require('body-parser'); ... And mount the api sub application just before the app.listen... // Mount the api sub application app.use('/api/todos/', api); We said we will implement a client for our API. It should expose two functions –create and update –located at ./public/client.js. Here is its source: 'use strict'; (function (api) { var url = '/api/todos/'; function extractIDFromResponse(xhr) { var location = xhr.getResponseHeader('location'); var result = +location.slice(url.length); return result; } api.create = function createToDo(title, callback) { var xhr = new XMLHttpRequest(); var todo = { title: title, done: false }; xhr.open('POST', url); xhr.setRequestHeader('Content-Type', 'application/json'); xhr.onload = function () { if (xhr.status === 201) { todo.id = extractIDFromResponse(xhr); } return callback(null, xhr, todo); }; xhr.send(JSON.stringify(todo)); }; api.update = function createToDo(todo, callback) { var xhr = new XMLHttpRequest(); xhr.open('PUT', url); xhr.setRequestHeader('Content-Type', 'application/json'); xhr.onload = function () { if (xhr.status === 200) { console.log('200'); } return callback(null, xhr, todo); }; xhr.send(JSON.stringify(todo)); }; })(this.todoAPI = {}); Okay, time to load the API client into the UI and share it with our tag. Modify the index view including it as a dependency: doctype html html head script(src="/js/riot.js") body h1 ToDo App script(type="riot/tag" src="/tags/todo.tag") script(src="/js/client.js") todo script. riot.mount('todo', todoAPI); We are now loading the API client and passing it as a reference to the todo tag. Our last change today is to update the add function to consume the API. Reload the browser again, type something into the textbox, and hit enter. Nothing new happens because our add function is not yet using the API. We need to update ./client/tag-todo.js as: 'use strict'; var self = this; var api = self.opts; self.items = []; self.add = function (e) { var title = self.todo.value; api.create(title, function (err, xhr, todo) { if (xhr.status === 201) { self.todo.value = ''; self.items.push(todo); self.update(); } }); }; We have augmented self with an array of items. Everytime we create a new ToDo task (after we get the 201 code from the server) we push that new ToDo object into the array because we are going to print that list of items. In Riot, we can loop the items adding each attribute to any tag. Last, update to ./client/tag-todo.jade todo form(onsubmit="{ add }") input(type="text", placeholder="Needs to be done", name="todo") ul li(each="{items}") span {title} script include tag-todo.js Finally! Reload the page and create a ToDo! Next steps You can find the complete source code for this article here. The final version of the code also implements a done/undone button, which you can try to implement by yourself. About the author Pedro NarcisoGarcíaRevington is a Senior Full Stack Developer with 10+ years experience in high scalability and availability, microservices, automated deployments, data processing, CI, (T,B,D)DD and polyglot persistence.

In this article by Jaroslaw Krochmalski, author of Developing with Docker, at the beginning, Docker was created as an internal tool by a Platform as a Service company, called dotCloud. Later on, in March 2013, it was released as open source. Apart from the Docker Inc. team, which is the main sponsor, there are some other big names contributing to the tool –Red Hat, IBM, Microsoft, Google, and Cisco Systems, just to name a few. Software development today needs to be agile and react quickly to the changes. We use methodologies such as Scrum, estimate our work in story points and attend the daily stand-ups. But what about preparing our software for shipment and the deployment? Let's see how Docker fits into that scenario and can help us being agile. (For more resources related to this topic, see here.) In this article, we will cover the following topics: The basic idea behind Docker A difference between virtualization and containerization Benefits of using Docker Components available to install We will begin with a basic idea behind this wonderful tool. The basic idea The basic idea behind Docker is to pack an application with all of its dependencies (let it be binaries, libraries, configuration files, scripts, jars, and so on) into a single, standardized unit for software development and deployment. Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, and system libraries—anything you can install on a server. This guarantees that it will always run in the same way, no matter what environment it will be deployed in. With Docker, you can build some Node.js or Java project (but you are of course not limited to those two) without having to install Node.js or Java on your host machine. Once you're done with it, you can just destroy the Docker image and it's as though nothing ever happened. It's not a programming language or a framework, rather think of it as about a tool that helps solving the common problems such as installing, distributing, and managing the software. It allows programmers and DevOps to build, ship, and run their code anywhere. You can think that maybe Docker is a virtualization engine, but it's far from it as we will explain in a while. Containerization versus virtualization To fully understand what Docker really is, first we need to understand the difference between traditional virtualization and containerization. Let's compare those two technologies now. Traditional virtualization A traditional virtual machine, which represents the hardware-level virtualization, is basically a complete operating system running on top of the host operating system. There are two types of virtualization hypervisors: Type 1 and Type 2. Type 1 hypervisors provide server virtualization on bare metal hardware—there is no traditional, end user's operating system. Type 2 hypervisors, on the other hand, are commonly used as a desktop virtualization—you run the virtualization engine on top of your own operating system. There can be a lot of use cases that would make an advantage from using virtualization—the biggest asset is that you can run many virtual machines with totally different operating systems on a single host. Virtual machines are fully isolated, hence very secure. But nothing comes without a price. There are many drawbacks—they contain all the features that an operating system needs to have: device drivers, core system libraries, and so on. They are heavyweight, usually resource-hungry and not so easy to set up—virtual machines require full installation. They require more computing resources to execute. To successfully run an application on a virtual machine, the hypervisor needs to first import the virtual machine and then power it up, and this takes time. Furthermore, their performance gets substantially degraded. As a result, only a few virtual machines can be provisioned and made available to work on a single machine. Containerization The Docker software runs in an isolated environment called a Docker container. A Docker container is not a virtual machine in the popular sense. It represents operating system virtualization. While each virtual machine image runs on an independent guest OS, the Docker images run within the same operating system kernel. A container has its own filesystem and environment variables. It's self-sufficient. Because of the containers run within the same kernel, they utilize fewer system resources. The base container can be, and usually is, very lightweight. Worth knowing is that Docker containers are isolated not only from the underlying OS, but from each other as well. There is no overhead related to a classic virtualization hypervisor and a guest operating system. This allows achieving almost bare metal, near native performance. The boot time of a dockerized application is usually very fast due to the low overhead of containers. It is also possible to speed up the roll-out of hundreds of application containers in seconds and to reduce the time taken for provisioning your software. As you can see, Docker is quite different from the traditional virtualization engines. Be aware that containers cannot substitute virtual machines for all use cases. A thoughtful evaluation is still required to determine what is best for your application. Both solutions have their advantages. On one hand we have the fully isolated, secure virtual machine with average performance and on the other hand, we have the containers that are missing some of the key features (such as total isolation), but are equipped with high performance that can be provisioned swiftly. Let's see what other benefits you will get when using Docker containerization. Benefits of using Docker When comparing the Docker containers with traditional virtual machines, we have mentioned some of its advantages. Let's summarize them now in more detail and add some more. Speed and size As we have said before, the first visible benefit of using Docker will be very satisfactory performance and short provisioning time. You can create or destroy containers quickly and easily. Docker shares only the Kernel, nothing less, nothing more. However, it reuses the image layers on which the specific image is built upon. Because of that, multiple versions of an application running in containers will be very lightweight. The result is faster deployment, easier migration, and nimble boot times. Reproducible and portable builds Using Docker enables you to deploy ready-to-run software, which is portable and extremely easy to distribute. Your containerized application simply runs within its container, there's no need for traditional installation. The key advantage of a Docker image is, it is bundled with all the dependency the containerized application needs to run. The lack of installation of dependencies process has a huge advantage. This eliminates problems such as software and library conflicts or even driver compatibility issues. Because of Docker's reproducible build environment, it's particularly well suited for testing, especially in your continuous integration flow. You can quickly boot up identical environments to run the tests. And because the container images are all identical each time, you can distribute the workload and run tests in parallel without a problem. Developers can run the same image on their machine that will be run in production later, which again has a huge advantage in testing. The use of Docker containers speeds up continuous integration. There are no more endless build-test-deploy cycles. Docker containers ensure that applications run identically in development, test, and production environments. One of Docker's greatest features is the portability. Docker containers are portable – they can be run from anywhere: your local machine, a nearby or distant server, and private or public cloud. When speaking about the cloud, all major cloud computing providers, such as Amazon Web Services, Google's Compute Platform have perceived Docker's availability and now support it. Docker containers can be run inside an Amazon EC2 instance, Google Compute Engine instance provided that the host operating system supports Docker. A container running on an Amazon EC2 instance can easily be transferred to some other environment, achieving the same consistency and functionality. Docker works very well with various other IaaS (Infrastructure-as-a-Service) providers such as Microsoft's Azure, IBM SoftLayer, or OpenStack. This additional level of abstraction from your infrastructure layer is an indispensable feature. You can just develop your software without worrying about the platform it will be run later on. It's truly a write once run everywhere solution. Immutable and agile infrastructure Maintaining a truly idempotent configuration management code base can be tricky and a time-consuming process. The code grows over time and becomes more and more troublesome. That's why the idea of an immutable infrastructure becomes more and more popular nowadays. Containerization comes to the rescue. By using containers during the process of development and deployment of your applications, you can simplify the process. Having a lightweight Docker server that needs almost no configuration management, you manage your applications simply by deploying and redeploying containers to the server. And again, because the containers are very lightweight, it takes only seconds of your time. As a starting point, you can download a prebuilt Docker image from the Docker Hub, which is like a repository of ready-to-use images. There are many choices of web servers, runtime platforms, databases, messaging servers and so on. It's like a real gold mine of software you can use for free to get a base foundation for your own project. The effect of the immutability of Docker's images is the result of a way they are being created. Docker makes use of a special file called a Dockerfile. This file contains all the setup instructions on how to create an image, like must-have components, libraries, exposed shared directories, network configuration, and so on. An image can be very basic, containing nothing but the operating system foundations, or—something that is more common—containing a whole prebuilt technology stack which is ready to launch. You can create images by hand, but it can be an automated process also. Docker creates images in a layered fashion: every feature you include will be added as another layer in the base image. This is another serious speed boost compared to the traditional virtualization techniques. Tools and APIs Of course, Docker is not just a Dockerfile processor and the runtime engine. It's a complete package with wide selection of tools and APIs that are helpful during the developer's and DevOp's daily work. First of all, there's The Docker Toolbox, which is an installer to quickly and easily install and setup a Docker environment on your own machine. The Kinematic is desktop developer environment for using Docker on Windows and Mac OS X. Docker distribution also contains a whole bunch of command-line tools. Let's look at them now. Tools overview On Windows, depending on the Windows version you use, there are two choices. It can be either Docker for Windows if you are on Windows 10 or later, or Docker Toolbox for all earlier versions of Windows. The same applies to MacOS. The newest offering is Docker for Mac, which runs as a native Mac application and uses xhyve to virtualize the Docker Engine environment and Linux kernel. For earlier version of Mac that doesn't meet the Docker for Mac requirements you should pick the Docker Toolbox for Mac. The idea behind Docker Toolbox and Docker native applications are the same—to virtualize Linux kernel and Docker engine on top of your operating system, we will be using Docker Toolbox, as it is more universal; it will run in all Windows and MacOS versions. The installation package for Windows and Mac OS is wrapped in an executable called the Docker Toolbox. The package contains all the tools you need to begin working with Docker. Of course there are tons of additional third-party utilities compatible with Docker, some of them very useful. But for now, let's focus on the default toolset. Before we start the installation, let's look at the tools that the installer package contains to better understand what changes will be made to your system. Docker Engine and Docker Engine client Docker is a client-server application. It consists of the daemon that does the important job: builds and downloads images, starts and stops containers and so on. It exposes a REST API that specifies interfaces for interacting with the daemon and is being used for remote management. Docker Engine accepts Docker commands from the command line, such as docker to run the image, docker ps to list running containers, docker images to list images, and so on. The Docker client is a command-line program that is being used to manage Docker hosts running Linux containers. It communicates with the Docker server using the REST API wrapper. You will interact with Docker by using the client to send commands to the server. Docker Engine works only on Linux. If you want run Docker on Windows or Mac OS, or want to provision multiple Docker hosts on a network or in the Cloud, you will then need the Docker Machine. Docker Machine Docker Machine is a fairly new command-line tool created by Docker team to manage Docker servers you can deploy containers to. It deprecated the old way of installing Docker with the Boot2Docker utility. The Docker Machine eliminates the need to create virtual machines manually and install Docker before starting Docker containers on them. It handles the provisioning and installation process for you behind the scenes. In other words, it's a quick way to get a new virtual machine provisioned and ready to run Docker containers. This is an indispensable tool when developing PaaS (Platform as a Service) architecture. Docker Machine not only creates a new VM with the Docker Engine installed in it, but sets up the certificate files for authentication and then configures the Docker client to talk to it. For flexibility purposes, the Docker Machine introduces the concept of drivers. Using drivers, Docker is able to communicate with various virtualization software and cloud providers. In fact, when you install Docker for Windows or Mac OS, the default VirtualBox driver will be used. The following command will be executed behind the scenes: docker-machine create --driver=virtualbox default Another available driver is amazonec2 for Amazon Web Services. It can be used to install Docker on the Amazon's cloud—we will do it later in this article. There are a lot of drivers ready to be used, and more are coming all the time. The list of existing official drivers with their documentation is always available at the Docker Drivers website: https://docs.docker.com/machine/drivers. The list contains the following drivers at the moment: Amazon Web Services Microsoft Azure Digital Ocean Exoscale Google Compute Engine Generic Microsoft Hyper-V OpenStack Rackspace IBM Softlayer Oracle VirtualBox VMware vCloud Air VMware Fusion VMware vSphere Apart from these, there is also a lot of third-party driver plugins available freely on the Internet sites such as GitHub. You can find additional drivers for different cloud providers and virtualization platforms, such as OVH Cloud or Parallels for Mac OS, for example, you are not limited to Amazon's AWS or Oracle's VirtualBox. As you can see, the choice is very broad. If you cannot find a specific driver for your Cloud provider, try looking for it on the GitHub. When installing the Docker Toolbox on Windows or Mac OS, Docker Machine will be selected by default. It's mandatory and currently the only way to run Docker on these operating systems. Installing the Docker Machine is not obligatory for Linux—there is no need to virtualize the Linux kernel there. However, if you want to deal with the Cloud providers or just want to have common runtime environment portable between Mac OS, Windows, and Linux, you can install Docker Machine for Linux as well. We will describe the process later in this article. Machine will be also used behind the scenes when using the graphical tool Kitematic, which we will present in a while. After the installation process, Docker Machine will be available as a command-line tool: docker-machine. Kitematic Kitematic is the software tool you can use to run containers through a plain, yet robust graphical user interface. In 2015, Docker acquired the Kitematic team, expecting to attract many more developers and hoping to open up the containerization solution to more developers and a wider, more general public. Kitematic is now included by default when installing Docker Toolbox on Mac OS and MS Windows. You can use it to comfortably search and fetch images you need from the Docker Hub. The tool can also be used to run your own app containers. Using the GUI, you can edit environment variables, map ports, configure volumes, study logs, and have command-line access to the containers. It is worth mentioning that you can seamlessly switch between Kitematic GUI and command-line interface to run and manage application containers. Kitematic is very convenient, however, if you want to have more control when dealing with the containers or just want to use scripting - the command line will be a better solution. In fact, Kitematic allows you to switch back and forth between the Docker CLI and the graphical interface. Any changes you make on the command-line interface will be directly reflected in Kitematic. The tool is simple to use, as you will see at the end of this article, when we are going to test our setup on Mac or Windows PC, we will be using the command-line interface for working with Docker. Docker compose Compose is a tool, executed from the command line as docker-compose. It replaces the old fig utility. It's used to define and run multicontainer Docker applications. Although it's very easy to imagine a multi-container application (like a web server in one container and a database in the other), it's not mandatory. So if you decide that your application will fit in a single Docker container, there will be no use for docker-compose. In real life, it's very likely that your application will span into multiple containers. With docker-compose, you use a compose file to configure your application's services, so they can be run together in an isolated environment. Then, using a single command, you create and start all the services from your configuration. When it comes to multicontainer applications, docker-compose is great for development and testing, as well as continuous integration workflows. Oracle VirtualBox Oracle VM VirtualBox is a free and open source hypervisor for x86 computers from Oracle. It will be installed by default when installing the Docker Toolbox. It supports the creation and management of virtual machines running Windows, Linux, BSD, OS/2, Solaris, and so on. In our case, the Docker Machine using VirtualBox driver, will use VirtualBox to create and boot a bitsy Linux distribution capable of the running docker-engine. It's worth mentioning that you can also run the teensy–weensy virtualized Linux straight from the VirtualBox itself. Every Docker Machine you have created using the docker-machine or Kitematic, will be visible and available to boot in the VirtualBox, when you run it directly, as shown in the following screenshot: You can start, stop, reset, change settings, and read logs in the same way as for other virtualized operating systems. You can use VirtualBox in Windows or Mac for other purposes than Docker. Git Git is a distributed version control system that is widely used for software development and other version control tasks. It has emphasis on speed, data integrity, and support for distributed, non-linear workflows. Docker Machine and Docker client follows the pull/push model of Git for fetching the needed dependencies from the network. For example, if you decide to run the Docker image which is not present on your local machine, Docker will fetch this image from the Docker Hub. Docker doesn't internally use Git for any kind of resource versioning. It does, however, rely on hashing to uniquely identify the filesystem layers which is very similar to what Git does. Docker also takes initial inspiration in the notion of commits, pushes, and pulls. Git is also included in the Docker Toolbox installation package. From a developer's perspective, there are tools especially useful in a programmer's daily job, be it IntelliJ IDEA Docker Integration Plugin for Java fans or Visual Studio 2015 Tools for Docker for those who prefer C#. They let you download and build Docker images, create and start containers, and carry out other related tasks straight from your favorite IDE. Apart from the tools included in the Docker's distribution package (it will be Docker Toolbox for older versions of Windows or Docker for Windows and Docker for Mac), there are hundreds of third-party tools, such as Kubernetes and Helios (for Docker orchestration), Prometheus (for monitoring of statistics) or Swarm and Shipyard for managing clusters. As Docker captures higher attention, more and more Docker-related tools pop-up almost every week. But these are not the only tools available for you. Additionally, Docker provides a set of APIs that can be very handy. One of them is the Remote API for the management of the images and containers. Using this API, you will be able to distribute your images to the runtime Docker engine. The container can be shifted to a different machine that runs Docker, and executed there without compatibility concerns. This may be especially useful when creating PaaS (Platform-as-a-Service) architectures. There's also the Stats API that will expose live resource usage information (such as CPU, memory, network I/O and block I/O) for your containers. This API endpoint can be used to create tools that show how your containers behave, for example, on a production system. Summary By now we understand the difference between the virtualization and containerization and also, I hope, we can see the advantages of using the latter. We also know what components are available for us to install and use. Let's begin our journey to the world of containers and go straight to the action by installing the software. Resources for Article: Further resources on this subject: Let's start with Extending Docker [article] Docker Hosts [article] Understanding Docker [article]

This article is by Pieter van der Westhuizen, author of the book Bootstrap for ASP.NET MVC - Second edition. As developers, we can find it difficult to create great-looking user interfaces from scratch when using HTML and CSS. This is especially hard when developers have extensive experience developing Windows Forms applications. Microsoft introduced Web Forms to remove the complexities of building websites and to ease the switch from Windows Forms to the Web. This in turn makes it very hard for Web Forms developers, and even harder for Windows Forms developers to switch to ASP.NET MVC. Bootstrap is a set of stylized components, plugins, and a layout grid that takes care of the heavy lifting. Microsoft included Bootstrap in all ASP.NET MVC project templates since 2013. In this article, we will cover the following topics: Files included in the Bootstrap distribution How to create an empty ASP.NET site and enable MVC and static files Adding the Bootstrap files using Bower Automatically compile the Bootstrap Sass files using Gulp Installing additional icon sets How to create a layout file that references the Bootstrap files (For more resources related to this topic, see here.) Files included in the Bootstrap distribution In order to get acquainted with the files inside the Bootstrap distribution, you need to download its source files. At the time of writing, Bootstrap 4 was still in Alpha, and its source files can be downloaded from http://v4-alpha.getbootstrap.com. Bootstrap style sheets Do not be alarmed by the amount of files inside the css folder. This folder contains four .css files and two .map files. We only need to include the bootstrap.css file in our project for the Bootstrap styles to be applied to our pages. The bootstrap.min.css file is simply a minified version of the aforementioned file. The .map files can be ignored for the project we'll be creating. These files are used as a type of debug symbol (similar to the .pdb files in Visual Studio), which allows developers to live edit their preprocessor source files. Bootstrap JavaScript files The js folder contains two files. All the Bootstrap plugins are contained in the bootstrap.js file. The bootstrap.min.js file is simply a minified version of the aforementioned file. Before including the file in your project, make sure that you have a reference to the jQuery library because all Bootstrap plugins require jQuery. Bootstrap fonts/icons Bootstrap 3 uses Glyphicons to display various icons and glyphs in Bootstrap sites. Bootstrap 4 will no longer ship with glyphicons included, but you still have the option to include it manually or to include your own icons. The following two icon sets are good alternatives to Glyphicons: Font Awesome, available from http://fontawesome.io/ GitHub's Octicons, available from https://octicons.github.com/ Bootstrap source files Before you can get started with Bootstrap, you first need to download the Bootstrap source files. At the time of writing, Bootstrap 4 was at version 4 Alpha 3. You have a few choices when adding Bootstrap to you project. You can download the compiled CSS and JavaScript files or you can use a number of package managers to install the Bootstrap Sass source to your project. In this article, you'll be using Bower to add the Bootstrap 4 source files to your project. For a complete list of Bootstrap 4 Alpha installation sources, visit http://v4-alpha.getbootstrap.com/getting-started/download/ CSS pre-processors CSS pre-processors process code written in a pre-processed language, such as LESS or Sass, and convert it into standard CSS, which in turn can be interpreted by any standard web browser. CSS pre-processors extend CSS by adding features that allow variables, mixins, and functions. The benefits of using CSS pre-processors are that they are not bound by any limitations of CSS. CSS pre-processors can give you more functionality and control over your style sheets and allows you to write more maintainable, flexible, and extendable CSS. CSS pre-processors can also help to reduce the amount of CSS and assist with the management of large and complex style sheets that can become harder to maintain as the size and complexity increases. In essence, CSS pre-processors such as Less and Sass enables programmatic control over your style sheets. Bootstrap moved their source files from Less to Sass with version 4.  Less and Sass are very alike in that they share a similar syntax as well as features such as variables, mixins, partials, and nesting, to name but a few. Less was influenced by Sass, and later on, Sass was influenced by Less when it adopted CSS-like block formatting, which worked very well for Less. Creating an empty ASP.NET MVC site and adding Bootstrap manually The default ASP.NET 5 project template in Visual Studio 2015 Update 3 currently adds Bootstrap 3 to the project. In order to use Bootstrap 4 in your ASP.NET project, you'll need to create an empty ASP.NET project and add the Bootstrap 4 files manually. To create a project that uses Bootstrap 4, complete the following process: In Visual Studio 2015, select New | Project from the File menu or use the keyboard shortcut Ctrl + Shift + N. From the New Project dialog window, select ASP.NET Core Web Application (.NET Core), which you'll find under Templates | Visual C# | Web. Select the Empty project template from the New ASP.NET Core Web Application (.NET Core) Project dialog window and click on OK. Enabling MVC and static files The previous steps will create a blank ASP.NET Core project. Running the project as-is will only show a simple Hello World output in your browser. In order for it to serve static files and enable MVC, we'll need to complete the following steps: Double-click on the project.json file inside the Solution Explorer in Visual Studio. Add the following two lines to the dependencies section, and save the project.json file: "Microsoft.AspNetCore.Mvc": "1.0.0", "Microsoft.AspNetCore.StaticFiles": "1.0.0" You should see a yellow colored notification inside the Visual Studio Solution Explorer with a message stating that it is busy restoring packages. Open the Startup.cs file. To enable MVC for the project, change the ConfigureServices method to the following: public void ConfigureServices(IServiceCollection services) {     services.AddMvc(); } Finally, update the Configure method to the following code: public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory) {     loggerFactory.AddConsole();       if (env.IsDevelopment())     {         app.UseDeveloperExceptionPage();     }       app.UseStaticFiles();       app.UseMvc(routes =>     {         routes.MapRoute(             name: "default",             template: "{controller=Home}/{action=Index}/{id?}");     }); } The preceding code will enable logging and the serving of static files such as images, style sheets, and JavaScript files. It will also set the default MVC route. Creating the default route controller and view When creating an empty ASP.NET Core project, no default controller or views will be created by default. In the previous steps, we've created a default route to the Index action of the Home controller. In order for this to work, we first need to complete the following steps: In the Visual Studio Solution Explorer, right-click on the project name and select Add | New Folder from the context menu. Name the new folder Controllers. Add another folder called Views. Right-click on the Controllers folder and select Add | New Item… from the context menu. Select MVC Controller Class from the Add New Item dialog, located under .NET Core | ASP.NET, and click on Add. The default name when adding a new controller will be HomeController.cs: Next, we'll need to add a subfolder for the HomeController in the Views folder. Right-click on the Views folder and select Add | New Folder from the context menu. Name the new folder Home. Right-click on the newly created Home folder and select Add | New Item… from the context menu. Select the MVC View Page item, located under .NET Core | ASP.NET; from the list, make sure the filename is Index.cshtml and click on the Add button: Your project layout should resemble the following in the Visual Studio Solution Explorer: Adding the Bootstrap 4 files using Bower With ASP.NET 5 and Visual Studio 2015, Microsoft provided the ability to use Bower as a client-side package manager. Bower is a package manager for web frameworks and libraries that is already very popular in the web development community. You can read more about Bower and search the packages it provides by visiting http://bower.io/ Microsoft's decision to allow the use of Bower and package managers other than NuGet for client-side dependencies is because it already has such a rich ecosystem. Do not fear! NuGet is not going away. You can still use NuGet to install libraries and components, including Bootstrap 4! To add the Bootstrap 4 source files to your project, you need to follow these steps: Right-click on the project name inside Visual Studio's Solution Explorer and select Add | New Item…. Under .NET Core | Client-side, select the Bower Configuration File item, make sure the filename is bower.json and click on Add, as shown here: If not already open, double-click on the bower.json file to open it and add Bootstrap 4 to the dependencies array. The code for the file should look similar to the following: {    "name": "asp.net",    "private": true,   "dependencies": {     "bootstrap": "v4.0.0-alpha.3"   } } Save the bower.json file. Once you've saved the bower.json file, Visual Studio will automatically download the dependencies into the wwwroot/lib folder of your project. In the case of Bootstrap 4 it also depends on jQuery and Tether, you'll notice that jQuery and Tether has also been downloaded as part of the Bootstrap dependency. After you've added Bootstrap to your project, your project layout should look similar to the following screenshot: Compiling the Bootstrap Sass files using Gulp When adding Bootstrap 4, you'll notice that the bootstrap folder contains a subfolder called dist. Inside the dist folder, there are ready-to-use Bootstrap CSS and JavaScript files that you can use as-is if you do not want to change any of the default Bootstrap colours or properties. However, because the source Sass files were also added, this gives you extra flexibility in customizing the look and feel of your web application. For instance, the default colour of the base Bootstrap distribution is gray; if you want to change all the default colours to shades of blue, it would be tedious work to find and replace all references to the colours in the CSS file. For example, if you open the _variables.scss file, located in wwwroot/lib/bootstrap/scss, you'll notice the following code: $gray-dark:                 #373a3c !default; $gray:                      #55595c !default; $gray-light:                #818a91 !default; $gray-lighter:              #eceeef !default; $gray-lightest:             #f7f7f9 !default; We're not going to go into too much detail regarding Sass in this article, but the $ in front of the names in the code above indicates that these are variables used to compile the final CSS file. In essence, changing the values of these variables will change the colors to the new values we've specified, when the Sass file is compiled. To learn more about Sass, head over to http://sass-lang.com/ Adding Gulp npm packages We'll need to add the gulp and gulp-sass Node packages to our solution in order to be able to perform actions using Gulp. To accomplish this, you will need to use npm. npm is the default package manager for the Node.js runtime environment. You can read more about it at https://www.npmjs.com/ To add the gulp and gulp-sass npm packages to your ASP.NET project, complete the following steps: Right-click on your project name inside the Visual Studio Solution Explorer and select Add | New Item… from the project context menu. Find the npm Configuration File item, located under .NET Core | Client-side. Keep its name as package.json and click on Add. If not already open, double-click on the newly added package.json file and add the following two dependencies to the devDependencies array inside the file: "devDependencies": {   "gulp": "3.9.1",   "gulp-sass": "2.3.2" } This will add version 3.9.1 of the gulp package and version 2.3.2 of the gulp-sass package to your project. At the time of writing, these were the latest versions. Your version numbers might differ. Enabling Gulp-Sass compilation Visual Studio does not compile Sass files to CSS by default without installing extensions, but we can enable it using Gulp. Gulp is a JavaScript toolkit used to stream client-side code through a series of processes when an event is triggered during build. Gulp can be used to automate and simplify development and repetitive tasks, such as the following: Minify CSS, JavaScript and image files Rename files Combine CSS files Learn more about Gulp at http://gulpjs.com/ Before you can use Gulp to compile your Sass files to CSS, you need to complete the following tasks: Add a new Gulp Configuration File to your project by right-cing Add | New Item… from the context menu. The location of the item is .NET Core | Client-side. Keep the filename as gulpfile.js and click on the Add button. Change the code inside the gulpfile.js file to the following: var gulp = require('gulp'); var gulpSass = require('gulp-sass'); gulp.task('compile-sass', function () {     gulp.src('./wwwroot/lib/bootstrap/scss/bootstrap.scss')         .pipe(gulpSass())         .pipe(gulp.dest('./wwwroot/css')); }); The code in the preceding step first declares that we require the gulp and gulp-sass packages, and then creates a new task called compile-sass that will compile the Sass source file located at /wwwroot/lib/bootstrap/scss/bootstrap.scss and output the result to the /wwwroot/css folder. Running Gulp tasks With the gulpfile.js properly configured, you are now ready to run your first Gulp task to compile the Bootstrap Sass to CSS. Accomplish this by completing the following steps: Right-click on gulpfile.js in the Visual Studio Solution Explorer and choose Task Runner Explorer from the context menu. You should see all tasks declared in the gulpfile.js listed underneath the Tasks node. If you do not see tasks listed, click on the Refresh button, located on the left-hand side of the Task Runner Explorer window. To run the compile-sass task, right-click on it and select Run from the context menu. Gulp will compile the Bootstrap 4 Sass files and output the CSS to the specified folder. Binding Gulp tasks to Visual Studio events Right-clicking on every task in the Task Runner Explorer in order to execute each, could involve a lot of manual steps. Luckily, Visual Studio allows us to bind tasks to the following events inside Visual Studio: Before Build After Build Clean Project Open If, for example, we would like to compile the Bootstrap 4 Sass files before building our project, simply select Before Build from the Bindings context menu of the Visual Studio Task Runner Explorer: Visual Studio will add the following line of code to the top of gulpfile.js to tell the compiler to run the task before building the project: /// <binding BeforeBuild='compile-sass' /> Installing Font Awesome Bootstrap 4 no longer comes bundled with the Glyphicons icon set. However, there are a number of free alternatives available for use with your Bootstrap and other projects. Font Awesome is a very good alternative to Glyphicons that provides you with 650 icons to use and is free for commercial use. Learn more about Font Awesome by visiting https://fortawesome.github.io/Font-Awesome/ You can add a reference to Font Awesome manually, but since we already have everything set up in our project, the quickest option is to simply install Font Awesome using Bower and compile it to the Bootstrap style sheet using Gulp. To accomplish this, follow these steps: Open the bower.json file, which is located in your project route. If you do not see the file inside the Visual Studio Solution Explorer, click on the Show All Files button on the Solution Explorer toolbar. Add font-awesome as a dependency to the file. The complete listing of the bower.json file is as follows: {   "name": "asp.net",   "private": true,   "dependencies": {     "bootstrap": "v4.0.0-alpha.3",     "font-awesome": "4.6.3"   } } Visual Studio will download the Font Awesome source files and add a font-awesome subfolder to the wwwroot/lib/ folder inside your project. Copy the fonts folder located under wwwroot/font-awesome to the wwwroot folder. Next, open the bootstrap.scss file located in the wwwroot/lib/bootstrap/scss folder and add the following line at the end of the file: $fa-font-path: "/fonts"; @import "../../font-awesome/scss/font-awesome.scss"; Run the compile-sass task via the Task Runner Explorer to recompile the Bootstrap Sass. The preceding steps will include Font Awesome in your Bootstrap CSS file, which in turn will enable you to use it inside your project by including the mark-up demonstrated here: <i class="fa fa-pied-piper-alt"></i> Creating a MVC Layout page The final step for using Bootstrap 4 in your ASP.NET MVC project is to create a Layout page that will contain all the necessary CSS and JavaScript files in order to include Bootstrap components in your pages. To create a Layout page, follow these steps: Add a new sub folder called Shared to the Views folder. Add a new MVC View Layout Page to the Shared folder. The item can be found in the .NET Core | Server-side category of the Add New Item dialog. Name the file _Layout.cshtml and click on the Add button: With the current project layout, add the following HTML to the _Layout.cshtml file: <!DOCTYPE html> <html lang="en"> <head>     <meta charset="utf-8">     <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">     <meta http-equiv="x-ua-compatible" content="ie=edge">     <title>@ViewBag.Title</title>     <link rel="stylesheet" href="~/css/bootstrap.css" /> </head> <body>     @RenderBody()       <script src="~/lib/jquery/dist/jquery.js"></script>     <script src="~/lib/bootstrap/dist/js/bootstrap.js"></script> </body> </html> Finally, add a new MVC View Start Page to the Views folder called _ViewStart.cshtml. The _ViewStart.cshtml file is used to specify common code shared by all views. Add the following Razor mark-up to the _ViewStart.cshtml file: @{     Layout = "_Layout"; } In the preceding mark-up, a reference to the Bootstrap CSS file that was generated using the Sass source files and Gulp is added to the <head> element of the file. In the <body> tag, the @RenderBody method is invoked using Razor syntax. Finally, at the bottom of the file, just before the closing </body> tag, a reference to the jQuery library and the Bootstrap JavaScript file is added. Note that jQuery must always be referenced before the Bootstrap JavaScript file. Content Delivery Networks You could also reference the jQuery and Bootstrap library from a Content Delivery Network (CDN). This is a good approach to use when adding references to the most widely used JavaScript libraries. This should allow your site to load faster if the user has already visited a site that uses the same library from the same CDN, because the library will be cached in their browser. In order to reference the Bootstrap and jQuery libraries from a CDN, change the <script> tags to the following: <script src="https://code.jquery.com/jquery-3.1.0.js"></script> <script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0-alpha.2/js/bootstrap.min.js"></script> There are a number of CDNs available on the Internet; listed here are some of the more popular options: MaxCDN: https://www.maxcdn.com/ Google Hosted Libraries: https://developers.google.com/speed/libraries/ CloudFlare: https://www.cloudflare.com/ Amazon CloudFront: https://aws.amazon.com/cloudfront/ Learn more about Bootstrap Frontend development with Bootstrap 4

In this article by Stephen Haney, the author of the book Swift 3 Game Development - Second Edition, we will see that apple's newest version of its flagship programming language, Swift 3, is the perfect choice for game developers. As it matures, Swift is realizing its opportunity to be something special, a revolutionary tool for app creators. Swift is the gateway for developers to create the next big game in the Apple ecosystem. We have only started to explore the wonderful potential of mobile gaming, and Swift is the modernization we need for our toolset. Swift is fast, safe, current, and attractive to developers coming from other languages. Whether you are new to the Apple world, or a seasoned veteran of Objective-C, I think you will enjoy making games with Swift. (For more resources related to this topic, see here.) Apple's website states the following: "Swift is a successor to the C and Objective-C languages." My goal is to guide you step-by-step through the creation of a 2D game for iPhones and iPads. We will start with installing the necessary software, work through each layer of game development, and ultimately publish our new game to the App Store. We will also have some fun along the way! We aim to create an endless flyer game featuring a magnificent flying penguin named Pierre. What is an endless flyer? Picture hit games like iCopter, Flappy Bird, Whale Trail, Jetpack Joyride, and many more—the list is quite long. Endless flyer games are popular on the App Store, and the genre necessitates that we cover many reusable components of 2D game design. I will show you how to modify our mechanics to create many different game styles. My hope is that our demo project will serve as a template for your own creative works. Before you know it, you will be publishing your own game ideas using the techniques we explore together. In this article, we will learn the following topics: Why you will love Swift What you will learn in this article New in Swift 3 Setting up your development environment Creating your first Swift game Why you will love Swift Swift, as a modern programming language, benefits from the collective experience of the programming community; it combines the best parts of other languages and avoids poor design decisions. Here are a few of my favorite Swift features: Beautiful syntax: Swift's syntax is modern and approachable, regardless of your existing programming experience. Apple balanced syntax with structure to make Swift concise and readable. Interoperability: Swift can plug directly into your existing projects and run side-by-side with your Objective-C code. Strong typing: Swift is a strongly typed language. This means the compiler will catch more bugs at compile time, instead of when your users are playing your game! The compiler will expect your variables to be of a certain type (int, string, and so on) and will throw a compile-time error if you try to assign a value of a different type. While this may seem rigid if you are coming from a weakly typed language, the added structure results in safer, more reliable code. Smart type inference: To make things easier, type inference will automatically detect the types of your variables and constants based upon their initial value. You do not need to explicitly declare a type for your variables. Swift is smart enough to infer variable types in most expressions. Automatic memory management: As the Apple Swift developer guide states, "memory management just works in Swift". Swift uses a method called Automatic Reference Counting (ARC) to manage your game's memory usage. Besides a few edge cases, you can rely on Swift to safely clean up and turn off the lights. An even playing field: One of my favorite things about Swift is how quickly the language is gaining mainstream adoption. We are all learning and growing together, and there is a tremendous opportunity to break new ground. Open source: From version 2.2 onwards, Apple made Swift open source, curetting it through the website www.swift.org, and launched a package manager with Swift 3. This is a welcome change as it fosters greater community involvement and a larger ecosystem of third party tools and add-ons. Eventually, we should see Swift migrate to new platforms. Prerequisites I will try to make this text easy to understand for all skill levels: I will assume you are brand new to Swift as a language Requires no prior game development experience, though it will help I will assume you have a fundamental understanding of common programming concepts What you will learn in this article You will be capable of creating and publishing your own iOS games. You will know how to combine the techniques we learned to create your own style of game, and you will be well prepared to dive into more advanced topics with a solid foundation in 2D game design. Embracing SpriteKit SpriteKit is Apple's 2D game development framework and your main tool for iOS game design. SpriteKit will handle the mechanics of our graphics rendering, physics, and sound playback. As far as game development frameworks go, SpriteKit is a terrific choice. It is built and supported by Apple and thus integrates perfectly with Xcode and iOS. You will learn to be highly proficient with SpriteKit as we will be using it exclusively in our demo game. We will learn to use SpriteKit to power the mechanics of our game in the following ways: Animate our player, enemies, and power-ups Paint and move side-scrolling environments Play sounds and music Apply physics-like gravity and impulses for movement Handle collisions between game objects Reacting to player input The control schemes in mobile games must be inventive. Mobile hardware forces us to simulate traditional controller inputs, such as directional pads and multiple buttons, on the screen. This takes up valuable visible area, and provides less precision and feedback than with physical devices. Many games operate with only a single input method: A single tap anywhere on the screen. We will learn how to make the best of mobile input, and explore new forms of control by sensing device motion and tilt. Structuring your game code It is important to write well-structured code that is easy to re-use and modify as your game design inevitably changes. You will often find mechanical improvements as you develop and test your games, and you will thank yourself for a clean working environment. Though there are many ways to approach this topic, we will explore some best practices to build an organized system with classes, protocols, inheritance, and composition. Building UI/menus/levels We will learn to switch between scenes in our game with a menu screen. We will cover the basics of user experience design and menu layout as we build our demo game. Integrating with Game Center Game Center is Apple's built-in social gaming network. Your game can tie into Game Center to store and share high scores and achievements. We will learn how to register for Game Center, tie it into our code, and create a fun achievement system. Maximizing fun If you are like me, you will have dozens of ideas for games floating around your head. Ideas come easily, but designing fun game play is difficult! It is common to find that your ideas need game play enhancements once you see your design in action. We will look at how to avoid dead-ends and see your project through to the finish line. Plus, I will share my tips and tricks to ensure your game will bring joy to your players. Crossing the finish line Creating a game is an experience you will treasure. Sharing your hard work will only sweeten the satisfaction. Once our game is polished and ready for public consumption, we will navigate the App Store submission process together. You will finish feeling confident in your ability to create games with Swift and bring them to market in the App Store. Monetizing your work Game development is a fun and rewarding process, even without compensation, but the potential exists to start a career, or side-job, selling games on the App Store. Successfully promoting and marketing your game is an important task. I will outline your options and start you down the path to monetization. New in Swift 3 The largest feature in Swift 3 is syntax compatibility and stability. Apple is trying to refine its young, shifting language into its final foundational shape. Each successive update of Swift has introduced breaking syntax changes that made older code incompatible with the newest version of Swift; this is very inconvenient for developers. Going forward, Swift 3 aims to reach maturity and maintain source compatibility with future releases of the language. Swift 3 also features the following:  A package manager that will help grow the ecosystem A more consistent, readable API that often results in less code for the same result Improved tooling and bug fixes in the IDE, Xcode Many small syntax improvements in consistency and clarity Swift has already made tremendous steps forward as a powerful, young language. Now Apple is working on polishing Swift into a mature, production-ready tool. The overall developer experience improves with Swift 3. Setting up your development environment Learning a new development environment can be a roadblock. Luckily, Apple provides some excellent tools for iOS developers. We will start our journey by installing Xcode. Introducing and installing Xcode Xcode is Apple's Integrated Development Environment (IDE). You will need Xcode to create your game projects, write and debug your code, and build your project for the App Store. Xcode also comes bundled with an iOS simulator to test your game on virtualized iPhones and iPads on your computer. Apple praises Xcode as "an incredibly productive environment for building amazing apps for Mac, iPhone, and iPad".   To install Xcode, search for Xcode in the AppStore, or visit http://developer.apple.com and select Developer and then Xcode. Swift is continually evolving, and each new Xcode release brings changes to Swift. If you run into errors because Swift has changed, you can always use Xcode's built-in syntax update tool. Simply use Xcode's Edit | Convert to Latest Syntax option to update your code. Xcode performs common IDE features to help you write better, faster code. If you have used IDEs in the past, then you are probably familiar with auto completion, live error highlighting, running and debugging a project, and using a project manager pane to create and organize your files. However, any new program can seem overwhelming at first. We will walk through some common interface functions over the next few pages. I have also found tutorial videos on YouTube to be particularly helpful if you are stuck. Most common search queries result in helpful videos. Creating our first Swift game Do you have Xcode installed? Let us see some game code in action in the simulator! We will start by creating a new project in Xcode. For our demo game, we will create a side-scrolling endless flyer featuring an astonishing flying penguin named Pierre. I am going to name this project Pierre Penguin Escapes the Antarctic, but feel free to name your project whatever you like. Follow these steps to create a new project in Xcode: Launch Xcode and navigate to File | New | Project. You will see a screen asking you to select a template for your new project. Select iOS | Application in the left pane, and Game in the right pane. It should look like this: Once you select Game, click Next. The following screen asks us to enter some basic information about our project. Don’t worry; we are almost at the fun bit. Fill in the Product Name field with the name of your game. Let us fill in the Team field. Do you have an active Apple developer account? If not, you can skip over the Team field for now. If you do, your Team is your developer account. Click Add Team and Xcode will open the accounts screen where you can log in. Enter your developer credentials as shown in the following screenshot: Once you're authenticated, you can close the accounts screen. Your developer account should appear in the Team dropdown. You will want to pick a meaningful Organization Name and Organization Identifier when you create your own games for publication. Your Organization Name is the name of your game development studio. For me, that's Joyful Games. By convention, your Organization Identifier should follow a reverse domain name style. I will use io.JoyfulGames since my website is JoyfulGames.io. After you fill out the name fields, be sure to select Swift for the Language, SpriteKit for Game Technology, and Universal for Devices. For now, uncheck Integrate GameplayKit, uncheck Include Unit Tests, uncheck Include UI Tests. We will not use these features in our demo game. Here are my final project settings: Click Next and you will see the final dialog box. Save your new project. Pick a location on your computer and click Next. And we are in! Xcode has pre-populated our project with a basic SpriteKit template. Navigating our project Now that we have created our project, you will see the project navigator on the left-hand side of Xcode. You will use the project navigator to add, remove, and rename files and generally organize your project. You might notice that Xcode has created quite a few files in our new project. We will take it slow; don’t feel that you have to know what each file does yet, but feel free to explore them if you are curious: Exploring the SpriteKit Demo Use the project navigator to open up the file named GameScene.swift. Xcode created GameScene.swift to store the default scene of our new game. What is a scene? SpriteKit uses the concept of scenes to encapsulate each unique area of a game. Think of the scenes in a movie; we will create a scene for the main menu, a scene for the Game Over screen, a scene for each level in our game, and so on. If you are on the main menu of a game and you tap Play, you move from the menu scene to the Level 1 scene. SpriteKit prepends its class names with the letters "SK"; consequently, the scene class is SKScene. You will see there is already some code in this scene. The SpriteKit project template comes with a very small demo. Let's take a quick look at this demo code and use it to test the iOS simulator. Please do not be concerned with understanding the demo code at this point. Your focus should be on learning the development environment. Look for the run toolbar at the top of the Xcode window. It should look something like the following: Select the iOS device of your choice to simulate using the dropdown on the far right. Which iOS device should you simulate? You are free to use the device of your choice. I will be using an iPhone 6 for the screenshots, so choose iPhone 6 if you want your results to match my images perfectly. Unfortunately, expect your game to play poorly in the simulator. SpriteKit suffers from poor FPS in the iOS simulator. Once our game becomes relatively complex, we will see our FPS drop, even on high-end computers. The simulator will get you through, but it is best if you can plug in a physical device to test. It is time for our first glimpse of SpriteKit in action! Press the gray play arrow in the toolbar (handy keyboard shortcut: command + r). Xcode will build the project and launch the simulator. The simulator starts in a new window, so make sure you bring it to the front. You should see a gray background with white text: Hello, World. Click around on the gray background. You will see colorful, spinning boxes spawning wherever you click: If you have made it this far, congratulations! You have successfully installed and configured everything you need to make your first Swift game. Once you have finished playing with the spinning squares, you can close the simulator down and return to Xcode. Note: You can use the keyboard command command + q to exit the simulator or press the stop button inside Xcode. If you use the stop button, the simulator will remain open and launch your next build faster. Examining the demo code Let's quickly explore the demo code. Do not worry about understanding everything just yet; we will cover each element in depth later. At this point, I am hoping you will acclimatize to the development environment and pick up a few things along the way. If you are stuck, keep going! Make sure you have GameScene.swift open in Xcode. The demo GameScene class implements some functions you will use in your games. Let’s examine these functions. Feel free to read the code inside each function, but I do not expect you to understand the specific code just yet. The game invokes the didMove function whenever it switches to the GameScene. You can think of it a bit like an initialize, or main, function for the scene. The SpriteKit demo uses it to draw the Hello, World text to the screen and set up the spinning square shape that shows up when we tap. There are seven functions involving touch which handle the user's touch input to the iOS device screen. The SpriteKit demo uses these functions to spawn the spinning square wherever we touch the screen. Do not worry about understanding these functions at this time. The update function runs once for every frame drawn to the screen. The SpriteKit demo does not use this function, but we may have reason to implement it later. Cleaning up I hope that you have absorbed some Swift syntax and gained an overview of Swift and SpriteKit. It is time to make room for our own game; let us clear all of that demo code out! We want to keep a little bit of the boilerplate, but we can delete most of what is inside the functions. To be clear, I do not expect you to understand this code yet. This is simply a necessary step towards the start of our journey. Please remove lines from your GameScene.swift file until it looks like the following code: import SpriteKit class GameScene: SKScene { override funcdidMove(to view: SKView) { } } Summary You have already accomplished a lot. You have had your first experience with Swift, installed and configured your development environment, launched code successfully into the iOS simulator. Great work! Resources for Article: Further resources on this subject: Swift for Open Source Developers [Article] Swift Power and Performance [Article] Introducing the Swift Programming Language [Article]