How-To Tutorials

In this article written by Suresh Kumar Gorakala, author of the book Building Recommendation Engines we will learn how to build a basic recommender system using R. In this article we will learn about various types of recommender systems in detail. This article explains neighborhood-similarity-based recommendations, personalized recommendation engines, model-based recommender systems and hybrid recommendation engines. Following are the different subtypes of recommender systems covered in this article: Neighborhood-based recommendation engines User-based collaborative filtering Item-based collaborative filtering Personalized recommendation engines Content-based recommendation engines Context-aware recommendation engines (For more resources related to this topic, see here.) Neighborhood-based recommendation engines As the name suggests, neighborhood-based recommender systems considers the preferences or likes of other users in the neighborhood before making suggestions or recommendations to the active user. While considering the preferences or tastes of neighbors, we first calculate how similar the other users are to the active user and then new items from more similar users are recommended to the user. Here the active user is the person to whom the system is serving recommendations. Since similarity calculations are involved these recommender systems are also called similarity-based recommender systems. Also since preferences or tastes are considered collaboratively from a pool of users these recommender systems are also called as collaborative filtering recommender systems. In this type of systems the main actors are the users, products and users preference information such as rating/ranking/liking towards the products. Preceding image is an example from Amazon showing collaborative filtering The collaborative filtering systems come in two flavors, They are: User-based collaborative filtering Item-based collaborative filtering Collaborative filtering When we have only the users interaction data of the products such as ratings, like/unlike, view/not viewed and we have to recommend new products then we have to choose Collaborative filtering approach. User-based Collaborative Filtering The basic intuition behind user-based collaborative filtering systems is, people with similar tastes in the past like similar items in future as well. For example, if user A and user B have very similar purchase history and if user A buys a new book which user B has not yet seen then we can suggest this book to User B as they have similar tastes. Item-based Collaborative filtering In this type of recommender systems unlike user-based collaborative filtering, we use similarity between items instead of similarity between users. Basic intuition for item-based recommender systems is if a User likes item A in past he might like item B which is similar to item A. In this approach instead of calculating similarity between users we calculate similarity between items or products. The most common similarity measure used for this approach is cosine similarity. Like user-based collaborative approach, we project the data into vector space and similarity between items is calculated using cosine angle between items. Similar to user-based collaborative filtering approach there are two steps for item-based collaborative approach. They are: Calculating the similarity between items. Predicting the ratings for the non rated item for a active user by making use of previous ratings given to other similar items Advantages of user-based collaborative filtering Easy to implement Neither the content information of the products nor users profile information is required for building recommendations New items are recommended to users giving Surprise factor to the users Disadvantages of user-based collaborative filtering This approach is computationally expensive as all the user information, product, rating information is loaded into the memory for similarity calculations. This approach fails for new users where we do not have any information about the users. This problem is called cold-start problem. This approach performs very poor if we have little data. Since we do not have content information about users or products. We cannot generate recommendations accurately only based on rating information only. Content-based recommender systems The recommendations are generated by considering only the rating or interaction information of the products by the users, that is suggesting new items for the active user are based on the ratings given to those new items by similar users to the active user. Assume the case of a person has given 4 star rating to a movie. In a collaborative filtering approach we only consider this rating information for generating recommendations. In real, a person rates a movie based on the features or content of the movie such as its genre, actor, director, story, and screenplay. Also the person watches a movie based on his personal choices. When we are building a recommendation engine to target users at personal level, the recommendations should not be based on the tastes of other similar people but should be based on the individual users’ tastes and the contents of the products. A recommendation which is targeted at personalized level and which considers individual preferences and contents of the products for generating recommendations are called content-based recommender systems. Another motivation for building content-based recommendation engines are they solve the cold start problem which new users face in collaborative filtering approach. When a new user comes based on the preferences of the person we can suggest new items which are similar to his tastes. Building content-based recommender systems involves three main steps as follows: Generating content information for products. Generating a user profile, preferences with respect to the features of the products. Generating recommendations predicting list of items which the user might like. Let us discuss each step in detail: Content extraction: In this step, we extract the features that represent the product. Most commonly the content of the products is represented in the vector space model with products name as rows and features as columns. User Profile generation: In this step, we build the user profile or preference matrix or vector space model matching the products content. Generating Recommendations: Now that, we have the generated the product content and user profile, the next step will be to generate the recommendations. Recommender systems using machine learning or any other mathematical, statistical models to generate recommendations are called as model-based systems Cosine similarity In this approach we first represent the user profiles and product content in vector forms and then we take cosine angle between each vector. The product which forms less angle with the user profile is considered as the most preferable item for the user. This approach is a standard approach while using neighborhood approach for Content based recommendations. Empirical studies shown that this approach gives more accurate results compared to other similarity measures. Classification-based approach Classification-based approaches fall under model-based recommender systems. In this approach, first we build a machine learning model by using the historical information, with user profile similar to the product content as input and the like/dislike of the product as output response classes. Supervised classification tasks such as logistic regression, KNN-classification methods, probabilistic methods and so on can be used. Advantages Content-based recommender systems are targeting at individual level Recommendations are generated using the user preferences alone unlike from user community as in collaborative filtering This approaches can be employed at real time as recommendation model doesn’t need to load the entire data for processing or generating recommendations Accuracy is high compared to collaborative approaches as they deal with the content of the products instead of rating information alone Cold start problem can be easily handled Disadvantages As the system is more personalized and the generated recommendations will become narrowed down to only user preferences with more and more user information comes into the system As a result, no new products that are not related to the user preferences will be shown to the user The user will not be able to look at what is happening around or what’s trending around Context-aware recommender Systems Over the years there has been evolution in recommender systems from neighborhood approaches to personalized recommender systems which are targeted to the individual users. These personalized recommender systems have become a huge success as this is useful at end user level and for organizations these systems become catalysts to increase their business. The personalized recommender systems, also called as content-based recommender systems are also getting evolved into Context aware recommender systems. Though the personalized recommender systems are targeted at individual user level and caters recommendations based on the personal preferences of the users, still there was scope to improve or refine the systems. Same person at different places might have different requirements. Likewise same person has different requirements at different times. Our intelligent recommender systems should be evolved enough to cater to the needs of the users for different places, at different times. Recommender System should be robust enough to suggest cotton shirts to a person during summer and suggesting Leather Jacket in winter. Similarly based on the time of the day suggesting Good restaurants serving a person’s personal choice breakfast and dinner would be very helpful. These kinds of recommender systems which considers location, time, mood, and so on that defines the context of user and suggests personalized recommendations are called context aware recommender systems. At broad level, context aware recommender systems are content-based recommenders with the inclusion of new dimension called context. In context aware systems, recommendations are generated in two steps: Generating list of recommendations of products for each user based on users’ preferences, that is content-based recommendations. Filtering out the recommendations that are specific to a current context. For example, based on past transaction history, interaction information, browsing patterns, ratings information on e-wallet mobile app, assume that User A is a movie lover, Sports lover, fitness freak. Using this information the content-based recommender systems generate recommendations of products such as Movie Tickets, 4G data offer for watching Football matches, Discount offers at GYM. Now based on the GPS co-ordinates of the mobile if the User A found to be at a 10K RUN marathon, then my Context aware recommendation engine will take this location information as the context and filters out the offers that are relevant to the current context and recommends Discount Offers at GYM to the user A. Most common approaches for building Context Aware Recommender systems are: Post filtering Approaches Pre-filtering approaches Pre-filtering approaches In pre-filtering approach, context information is applied to the User profile and product content. This step will filter out all the non relevant features and final personalized recommendations are generated on remaining feature set. Since filtering of features are made before generating personalized recommendations, these are called pre-filtering approaches. Post filtering approaches In post-filtering, firstly personalized recommendations are generated based on the user profile and product catalogue then the context information is applied for filtering out the relevant products to the user for the current context. Advantages Context aware systems are much advanced than the personalized content-based recommenders as these systems will be constantly in sync with user movements and generate recommendations as per current context. These systems are more real-time nature. Disadvantages Serendipity or surprise factor as in other personalized recommenders will be missing in this type of recommendations as well. Summary In this article, we have learned about popular recommendation engine techniques such as, collaborative filtering, content-based recommendations, context aware systems, hybrid recommendations, model-based recommendation systems with their advantages and disadvantages. Different similarity methods such as cosine similarity, Euclidean distance, Pearson-coefficient. Sub categories within each of the recommendations are also explained. Resources for Article: Further resources on this subject: Building a Recommendation Engine with Spark [article] Machine Learning Tasks [article] Machine Learning with R [article]

This article by Aleksandar Prokopec, author of the book Learning Concurrent Programming in Scala - Second Edition, explains the concepts of asynchronous programming in Scala. Asynchronous programming helps you eliminate blocking; instead of suspending the thread whenever a resource is not available, a separate computation is scheduled to proceed once the resource becomes available. In a way, many of the concurrency patterns seen so far support asynchronous programming; thread creation and scheduling execution context tasks can be used to start executing a computation concurrent to the main program flow. Still, it is not straightforward to use these facilities directly when avoiding blocking or composing asynchronous computations. In this article, we will focus on two abstractions in Scala that are specifically tailored for this task—futures and promises. More specifically, we will study the following topics: Starting asynchronous computations and using Future objects Using Promise objects to interface Blocking threads inside asynchronous computations Alternative future frameworks (For more resources related to this topic, see here.) Futures The parallel executions in a concurrent program proceed on entities called threads. At any point, the execution of a thread can be temporarily suspended until a specific condition is fulfilled. When this happens, we say that the thread is blocked. Why do we block threads in the first place in concurrent programming? One of the reasons is that we have a finite amount of resources; so, multiple computations that share these resources sometimes need to wait. In other situations, a computation needs specific data to proceed, and if that data is not yet available, threads responsible for producing the data could be slow or the source of the data could be external to the program. A classic example is waiting for the data to arrive over the network. Let's assume that we have a getWebpage method that returns that webpage's contents when given a url string with the location of the webpage: def getWebpage(url: String): String The return type of the getWebpage method is String; the method must return a string with the webpage's contents. Upon sending an HTTP request, though, the webpage's contents are not available immediately. It takes some time for the request to travel over the network to the server and back before the program can access the document. The only way for the method to return the contents of the webpage as a string value is to wait for the HTTP response to arrive. However, this can take a relatively long period of time from the program's point of view even with a high-speed Internet connection, the getWebpage method needs to wait. Since the thread that called the getWebpage method cannot proceed without the contents of the webpage, it needs to pause its execution; therefore, the only way to correctly implement the getWebpage method is for blocking. We already know that blocking can have negative side effects, so can we change the return value of the getWebpage method to some special value that can be returned immediately? The answer is yes. In Scala, this special value is called a future. The future is a placeholder, that is, a memory location for the value. This placeholder does not need to contain a value when the future is created; the value can be placed into the future eventually by getWebpage. We can change the signature of the getWebpage method to return a future as follows: def getWebpage(url: String): Future[String] Here, the Future[String] type means that the future object can eventually contain a String value. We can now implement getWebpage without blocking—we can start the HTTP request asynchronously and place the webpage's contents into the future when they become available. When this happens, we can say that the getWebpage method completes the future. Importantly, after the future is completed with some value, that value can no longer change. The Future[T] type encodes latency in the program—use it to encode values that will become available later during execution. This removes blocking from the getWebpage method, but it is not clear how the calling thread can extract the content of the future. Polling is one non-blocking way of extracting the content. In the polling approach, the calling thread calls a special method for blocking until the value becomes available. While this approach does not eliminate blocking, it transfers the responsibility of blocking from the getWebpage method to the caller thread. Java defines its own Future type to encode values that will become available later. However, as a Scala developer, you should use Scala's futures instead; they allow additional ways of handling future values and avoid blocking, as we will soon see. When programming with futures in Scala, we need to distinguish between future values and future computations. A future value of the Future[T] type denotes some value of the T type in the program that might not be currently available, but could become available later. Usually, when we say a future, we really mean a future value. In the scala.concurrent package, futures are represented with the Future[T] trait: trait Future[T] By contrast, a future computation is an asynchronous computation that produces a future value. A future computation can be started by calling the apply method on the Future companion object. This method has the following signature in the scala.concurrent package: def apply[T](b: =>T)(implicit e: ExecutionContext): Future[T] This method takes a by-name parameter of the T type. This is the body of the asynchronous computation that results in some value of type T. It also takes an implicit ExecutionContext parameter, which abstracts over where and when the thread gets executed. Recall that Scala's implicit parameters can either be specified when calling a method, in the same way as normal parameters, or they can be left out; in this case, the Scala compiler searches for a value of the ExecutionContext type in the surrounding scope. Most Future methods take an implicit execution context. Finally, the Future.apply method returns a future of the T type. This future is completed with the value resulting from the asynchronous computation, b. Starting future computations Let's see how to start a future computation in an example. We first import the contents of the scala.concurrent package. We then import the global execution context from the Implicits object. This makes sure that the future computations execute on the global context—the default execution context you can use in most cases: import scala.concurrent._ import ExecutionContext.Implicits.global object FuturesCreate extends App { Future { log("the future is here") } log("the future is coming") Thread.sleep(1000) } The order in which the log method calls (in the future computation and the main thread) execute is nondeterministic. The Future singleton object followed by a block is syntactic sugar for calling the Future.apply method. In the following example, we can use the scala.io.Source object to read the contents of our build.sbt file in a future computation. The main thread calls the isCompleted method on the future value, buildFile, returned from the future computation. Chances are that the build file was not read so fast, so isCompleted returns false. After 250 milliseconds, the main thread calls isCompleted again, and this time, isCompleted returns true. Finally, the main thread calls the value method, which returns the contents of the build file: import scala.io.Source object FuturesDataType extends App { val buildFile: Future[String] = Future { val f = Source.fromFile("build.sbt") try f.getLines.mkString("n") finally f.close() } log(s"started reading the build file asynchronously") log(s"status: ${buildFile.isCompleted}") Thread.sleep(250) log(s"status: ${buildFile.isCompleted}") log(s"value: ${buildFile.value}") } In this example, we used polling to obtain the value of the future. The Future singleton object's polling methods are non-blocking, but they are also nondeterministic; the isCompleted method will repeatedly return false until the future is completed. Importantly, completion of the future is in a happens-before relationship with the polling calls. If the future completes before the invocation of the polling method, then its effects are visible to the thread after the polling completes. Shown graphically, polling looks as shown in the following figure: Polling diagram Polling is like calling your potential employer every five minutes to ask if you're hired. What you really want to do is hand in a job application and then apply for other jobs instead of busy-waiting for the employer's response. Once your employer decides to hire you, he will give you a call on the phone number you left him. We want futures to do the same; when they are completed, they should call a specific function that we left for them. Promises Promises are objects that can be assigned a value or an exception only once. This is why promises are sometimes also called single-assignment variables. A promise is represented with the Promise[T] type in Scala. To create a promise instance, we use the Promise.apply method on the Promise companion object: def apply[T](): Promise[T] This method returns a new promise instance. Like the Future.apply method, the Promise.apply method returns immediately; it is non-blocking. However, the Promise.apply method does not start an asynchronous computation, it just creates a fresh Promise object. When the Promise object is created, it does not contain a value or an exception. To assign a value or an exception to a promise, we use the success or failure method, respectively. Perhaps you have noticed that promises are very similar to futures. Both futures and promises are initially empty and can be completed with either a value or an exception. This is intentional—every promise object corresponds to exactly one future object. To obtain the future associated with a promise, we can call the future method on the promise. Calling this method multiple times always returns the same future object. A promise and a future represent two aspects of a single-assignment variable. The promise allows you to assign a value to the future object, whereas the future allows you to read that value. In the following code snippet, we create two promises, p and q, that can hold string values. We then install a foreach callback on the future associated with the p promise and wait for 1 second. The callback is not invoked until the p promise is completed by calling the success method. We then fail the q promise in the same way and install a failed.foreach callback: object PromisesCreate extends App { val p = Promise[String] val q = Promise[String] p.future foreach { case x => log(s"p succeeded with '$x'") } Thread.sleep(1000) p success "assigned" q failure new Exception("not kept") q.future.failed foreach { case t => log(s"q failed with $t") } Thread.sleep(1000) } Alternatively, we can use the complete method and specify a Try[T] object to complete the promise. Depending on whether the Try[T] object is a success or a failure, the promise is successfully completed or failed. Importantly, after a promise is either successfully completed or failed, it cannot be assigned an exception or a value again in any way. Trying to do so results in an exception. Note that this is true even when there are multiple threads simultaneously calling success or complete. Only one thread completes the promise, and the rest throw an exception. Assigning a value or an exception to an already completed promise is not allowed and throws an exception. We can also use the trySuccess, tryFailure, and tryComplete methods that correspond to success, failure, and complete states, respectively, but return a Boolean value to indicate whether the assignment was successful. Recall that using the Future.apply method and callback methods with referentially transparent functions results in deterministic concurrent programs. As long as we do not use the trySuccess, tryFailure, and tryComplete methods, and none of the success, failure, and complete methods ever throws an exception, we can use promises and retain determinism in our programs. We now have everything we need to implement our custom Future.apply method. We call it the myFuture method in the following example. The myFuture method takes a b by-name parameter that is the asynchronous computation. First, it creates a p promise. Then, it starts an asynchronous computation on the global execution context. This computation tries to evaluate b and complete the promise. However, if the b body throws a nonfatal exception, the asynchronous computation fails the promise with that exception. In the meanwhile, the myFuture method returns the future immediately after starting the asynchronous computation: import scala.util.control.NonFatal object PromisesCustomAsync extends App { def myFuture[T](b: =>T): Future[T] = { val p = Promise[T] global.execute(new Runnable { def run() = try { p.success(b) } catch { case NonFatal(e) => p.failure(e) } }) p.future } val f = myFuture { "naa" + "na" * 8 + " Katamari Damacy!" } f foreach { case text => log(text) } } This is a common pattern when producing futures. We create a promise, let some other computation complete that promise, and return the corresponding future. However, promises were not invented just for our custom future's myFuture computation method. In the following sections, we will study use cases in which promises are useful. Futures and blocking Futures and asynchronous computations mainly exist to avoid blocking, but in some cases, we cannot live without it. It is, therefore, valid to ask how blocking interacts with futures. There are two ways to block with futures. The first way is to wait until a future is completed. The second way is by blocking from within an asynchronous computation. We will study both the topics in this section. Awaiting futures In rare situations, we cannot use callbacks or future combinators to avoid blocking. For example, the main thread that starts multiple asynchronous computations has to wait for these computations to finish. If an execution context uses daemon threads, as is the case with the global execution context, the main thread needs to block to prevent the JVM process from terminating. In these exceptional circumstances, we use the ready and result methods on the Await object from the scala.concurrent package. The ready method blocks the caller thread until the specified future is completed. The result method also blocks the caller thread, but returns the value of the future if it was completed successfully or throws the exception in the future if the future is failed. Both the methods require specifying a timeout parameter; the longest duration that the caller should wait for the completion of the future before a TimeoutException method is thrown. To specify a timeout, we import the scala.concurrent.duration package. This allows us to write expressions such as 10.seconds: import scala.concurrent.duration._ object BlockingAwait extends App { val urlSpecSizeFuture = Future { val specUrl = "http://www.w3.org/Addressing/URL/url-spec.txt" Source.fromURL(specUrl).size } val urlSpecSize = Await.result(urlSpecSizeFuture, 10.seconds) log(s"url spec contains $urlSpecSize characters") } In this example, the main thread starts a computation that retrieves the URL specification and then awaits. By this time, the World Wide Web Consortium (W3C) is worried that a DOS attack is under way, so this is the last time we download the URL specification. Blocking in asynchronous computations Waiting for the completion of a future is not the only way to block. Some legacy APIs do not use callbacks to asynchronously return results. Instead, such APIs expose the blocking methods. After we call a blocking method, we lose control over the thread; it is up to the blocking method to unblock the thread and return the control back. Execution contexts are often implemented using thread pools. By starting future computations that block, it is possible to reduce parallelism and even cause deadlocks. This is illustrated in the following example, in which 16 separate future computations call the sleep method, and the main thread waits until they complete for an unbounded amount of time: val startTime = System.nanoTime val futures = for (_ <- 0 until 16) yield Future { Thread.sleep(1000) } for (f <- futures) Await.ready(f, Duration.Inf) val endTime = System.nanoTime log(s"Total time = ${(endTime - startTime) / 1000000} ms") log(s"Total CPUs = ${Runtime.getRuntime.availableProcessors}") Assume that you have eight cores in your processor. This program does not end in one second. Instead, a first batch of eight futures started by Future.apply will block all the worker threads for one second, and then another batch of eight futures will block for another second. As a result, none of our eight processor cores can do any useful work for one second. Avoid blocking in asynchronous computations, as it can cause thread starvation. If you absolutely must block, then the part of the code that blocks should be enclosed within the blocking call. This signals to the execution context that the worker thread is blocked and allows it to temporarily spawn additional worker threads if necessary:   val futures = for (_ <- 0 until 16) yield Future { blocking { Thread.sleep(1000) } } With the blocking call around the sleep call, the global execution context spawns additional threads when it detects that there is more work than the worker threads. All 16 future computations can execute concurrently and the program terminates after one second. The Await.ready and Await.result statements block the caller thread until the future is completed and are in most cases used outside asynchronous computations. They are blocking operations. The blocking statement is used inside asynchronous code to designate that the enclosed block of code contains a blocking call. It is not a blocking operation by itself. Alternative future frameworks Scala futures and promises API resulted from an attempt to consolidate several different APIs for asynchronous programming, among them, legacy Scala futures, Akka futures, Scalaz futures, and Twitter's Finagle futures. Legacy Scala futures and Akka futures have already converged to the futures and promises API that you've learned about so far in this article. Finagle's com.twitter.util.Future type is planned to eventually implement the same interface as scala.concurrent.Future package, while the Scalaz scalaz.concurrent.Future type implements a slightly different interface. In this section, we give a brief of Scalaz futures. To use Scalaz, we add the following dependency to the build.sbt file: libraryDependencies += "org.scalaz" %% "scalaz-concurrent" % "7.0.6" We now encode an asynchronous tombola program using Scalaz. The Future type in Scalaz does not have the foreach method. Instead, we use its runAsync method, which asynchronously runs the future computation to obtain its value and then calls the specified callback: import scalaz.concurrent._ object Scalaz extends App { val tombola = Future { scala.util.Random.shuffle((0 until 10000).toVector) } tombola.runAsync { numbers => log(s"And the winner is: ${numbers.head}") } tombola.runAsync { numbers => log(s"... ahem, winner is: ${numbers.head}") } } Unless you are terribly lucky and draw the same permutation twice, running this program reveals that the two runAsync calls print different numbers. Each runAsync call separately computes the permutation of the random numbers. This is not surprising, as Scalaz futures have the pull semantics, in which the value is computed each time some callback requests it, in contrast to Finagle and Scala futures' push semantics, in which the callback is stored, and applied if and when the asynchronously computed value becomes available. To achieve the same semantics, as we would have with Scala futures, we need to use the start combinator that runs the asynchronous computation once, and caches its result: val tombola = Future { scala.util.Random.shuffle((0 until 10000).toVector) } start With this change, the two runAsync calls use the same permutation of random numbers in the tombola variable and print the same values. We will not dive further into the internals of alternate frameworks. The fundamentals about futures and promises that you learned about in this article should be sufficient to easily familiarize yourself with other asynchronous programming libraries, should the need arise. Summary This article presented some powerful abstractions for asynchronous programming. We saw how to encode latency with the Future type. You learned that futures and promises are closely tied together and that promises allow interfacing with legacy callback-based systems. In cases, where blocking was unavoidable, you learned how to use the Await object and the blocking statement. Finally, you learned that the Scala Async library is a powerful alternative for expressing future computations more concisely. Resources for Article: Further resources on this subject: Introduction to Scala [article] Concurrency in Practice [article] Integrating Scala, Groovy, and Flex Development with Apache Maven [article]

In this article by, Tejaswini Mandar Jog, author of the book, Learning Spring 5.0, we will cover the following topics: Introduction to Spring framework Problems address by Spring in enterprise application development Spring architecture What's new in Spring 5.0 Container Spring the fresh new start after the winter of traditional J2EE, is what Spring framework is in actual. A complete solution to the most of the problems occurred in handling the development of numerous complex modules collaborating with each other in a Java enterprise application. Spring is not a replacement to the traditional Java Development but it is a reliable solution to the companies to withstand in today's competitive and faster growing market without forcing the developers to be tightly coupled on Spring APIs. Problems addressed by Spring Java Platform is long term, complex, scalable, aggressive, and rapidly developing platform. The application development takes place on a particular version. The applications need to keep on upgrading to the latest version in order to maintain recent standards and cope up with them. These applications have numerous classes which interact with each other, reuse the APIs to take their fullest advantage so as to make the application is running smoothly. But this leads to some very common problems of as. Scalability The growth and development of each of the technologies in market is pretty fast both in hardware as well as software. The application developed, couple of years back may get outdated because of this growth in these areas. The market is so demanding that the developers need to keep on changing the application on frequent basis. That means whatever application we develop today should be capable of handling the upcoming demands and growth without affecting the working application. The scalability of an application is handling or supporting the handling of the increased load of the work to adapt to the growing environment instead of replacing them. The application when supports handling of increased traffic of website due to increase in numbers of users is a very simple example to call the application is scalable. As the code is tightly coupled, making it scalable becomes a problem. Plumbing code Let's take an example of configuring the DataSource in the Tomcat environment. Now the developers want to use this configured DataSource in the application. What will we do? Yes, we will do the JNDI lookup to get the DataSource. In order to handle JDBC we will acquire and then release the resources in try catch. The code like try catch as we discuss here, inter computer communication, collections too necessary but are not application specific are the plumbing codes. The plumbing code increases the length of the code and makes debugging complex. Boilerplate code How do we get the connection while doing JDBC? We need to register Driver class and invoke the getConnection() method on DriverManager to obtain the connection object. Is there any alternative to these steps? Actually NO! Whenever, wherever we have to do JDBC these same steps have to repeat every time. This kind of repetitive code, block of code which developer write at many places with little or no modification to achieve some task is called as boilerplate code. The boilerplate code makes the Java development unnecessarily lengthier and complex. Unavoidable non-functional code Whenever application development happens, the developer concentrate on the business logic, look and feel and persistency to be achieved. But along with these things the developers also give a rigorous thought on how to manage the transactions, how to handle increasing load on site, how to make the application secure and many more. If we give a close look, these things are not core concerns of the application but still these are unavoidable. Such kind of code which is not handling the business logic (functional) requirement but important for maintenance, trouble shooting, managing security of an application is called as non-functional code. In most of the Java application along with core concerns the developers have to write down non-functional code quite frequently. This leads to provide biased concentration on business logic development. Unit testing of the application Let's take an example. We want to test a code which is saving the data to the table in database. Here testing the database is not our motive, we just want to be sure whether the code which we have written is working fine or not. Enterprise Java application consists of many classes, which are interdependent. As there is dependency exists in the objects it becomes difficult to carry out the testing. POJO based development The class is a very basic structure of application development. If the class is getting extended or implementing an interface of the framework, reusing it becomes difficult as they are tightly coupled with API. The Plain Old Java Object (POJO) is very famous and regularly used terminology in Java application development. Unlike Struts and EJB Spring doesn't force developers to write the code which is importing or extending Spring APIs. The best thing about Spring is that developers can write the code which generally doesn't has any dependencies on framework and for this, POJOs are the favorite choice. POJOs support loosely coupled modules which are reusable and easy to test. The Spring framework is called to be non-invasive as it doesn't force the developer to use API classes or interfaces and allows to develop loosely coupled application. Loose coupling through DI Coupling, is the degree of knowledge in class has about the other. When a class is less dependent on the design of any other class, the class will be called as loosely coupled. Loose coupling can be best achieved by interface programming. In the Spring framework, we can keep the dependencies of the class separated from the code in a separate configuration file. Using interfaces and dependency injection techniques provided by Spring, developers can write loosely coupled code (Don't worry, very soon we will discuss about Dependency Injection and how to achieve it). With the help of loose coupling one can write a code which needs a frequent change, due to the change in the dependency it has. It makes the application more flexible and maintainable. Declarative programming In declarative programming, the code states what is it going to perform but not how it will be performed. This is totally opposite of imperative programming where we need to state stepwise what we will execute. The declarative programming can be achieved using XML and annotations. Spring framework keeps all configurations in XML from where it can be used by the framework to maintain the lifecycle of a bean. As the development happened in Spring framework, the 2.0 onward version gave an alternative to XML configuration with a wide range of annotations. Boilerplate code reduction using aspects and templates We just have discussed couple of pages back that repetitive code is boilerplate code. The boiler plate code is essential and without which providing transactions, security, logging, and so on, will become difficult. The framework gives solution of writing aspect which will deal with such cross cutting concerns and no need to write them along with business logic code. The use of aspect helps in reduction of boilerplate code but the developers still can achieve the same end effect. One more thing the framework provides, is the templates for different requirements. The JDBCTemplate and HibernateTemplate are couple of more useful concepts given by Spring which does reduction of boilerplate code. But as a matter of fact, you need to wait to understand and discover the actual potential. Layered architecture Unlike Struts and Hibernate which provides web persistency solutions respectively, Spring has a wide range of modules for numerous enterprise development problems. This layered architecture helps the developer to choose any one or more of the modules to write solution for his application in a coherent way. E.g. one can choose Web MVC module to handle web request efficiently without even knowing that there are many other modules available in the framework. Spring architecture Spring provides more than 20 different modules which can be broadly summaries under 7 main modules which are as follows: Spring modules What more Spring supports underneath? The following sections covers the additional features of Spring. Security module Now a days the applications alone with basic functionalities also need to provide sound ways to handle security at different levels. Spring5 support declarative security mechanism using Spring AOP. Batch module The Java Enterprise Applications needs to perform bulk processing, handling of large amount of data in many business solutions without user interactions. To handle such things in batches is the best solution available. Spring provides integration of batch processing to develop robust application. Spring integration In the development of enterprise application, the application may need interaction with them. Spring integration is extension of the core spring framework to provide integration of other enterprise applications with the help of declarative adapters. The messaging is one of such integration which is extensively supported by Spring. Mobile module The extensive use of mobiles opens the new doors in development. This module is an extension of Spring MVC which helps in developing mobile web applications known as Spring Android Project. It also provide detection of the type of device which is making the request and accordingly renders the views. LDAP module The basic aim of Spring was to simplify the development and to reduce the boilerplate code. The Spring LDAP module supports easy LDAP integration using template based development. .NEW module The new module has been introduced to support .NET platform. The modules like ADO.NET, NHibernate, ASP.NET has been in the .NET module includes to simplify the .NET development taking the advantages of features as DI, AOP, loose coupling. Container – the heart of Spring POJO development is the backbone of Spring framework. The POJO configured in the and whose object instantiation, object assembly, object management is done by Spring IoC container is called as bean or Spring bean. We use Spring IoC as it on the pattern of Inversion of Control. Inversion of Control (IoC) In every Java application, the first important thing which each developer does is, to get an object which he can use in the application. The state of an object can be obtained at runtime or it may be at compile time. But developers creates object where he use boiler plate code at a number of times. When the same developer uses Spring instead of creating object by himself he will be dependent on the framework to obtain object from. The term inversion of control comes as Spring container inverts the responsibility of object creation from developers. Spring IoC container is just a terminology, the Spring framework provides two containers: The BeanFactory The ApplicationContext Summary So in this article, we discussed about the general problems faced in Java enterprise application development and how they have been address by Spring framework. We have seen the overall major changes happened in each version of Spring from its first introduction in market. Enabling Spring Faces support Design with Spring AOP Getting Started with Spring Security

This article by Gilles Crettenand, author of the book Functional PHP, covers some of the concepts explained in book in a concise manner. We will look at the following: Declarative programming Functions Recursion Composing functions Benefits of functional programming (For more resources related to this topic, see here.) Functional programming has gained a lot of traction in the last few years. Various big tech companies started using functional languages, for example: Twitter on Scala (http://www.artima.com/scalazine/articles/twitter_on_scala.html) WhatsApp being written in Erlang (http://www.fastcompany.com/3026758/inside-erlang-the-rare-programming-language-behind-whatsapps-success) Facebook using Haskell (https://code.facebook.com/posts/302060973291128/open-sourcing-haxl-a-library-for-haskell/1) There is some really wonderful and successful work done on functional languages that compile to JavaScript—the Elm and PureScript languages to name a few. There are efforts to create new languages that either extend or compile to some more traditional languages, such as Hy and Coconut for Python. Even Apple's new language for iOS development, Swift, has multiple concepts from functional programming integrated into its core. However, this article is not about using a new language or learning a whole new technology, it is about benefiting from functional techniques without having to change our whole stack. By just applying some principles to our everyday PHP, we can greatly improve the quality of our life and code. Declarative programming Functional programming is also sometimes called declarative programming in contrast to imperative programming. This languages are called programming paradigms. Object-oriented programming is also a paradigm, but it is the one that is strongly tied to the imperative programming. Instead of explaining the difference at length, let's demonstrate with an example. First an imperative programming using PHP: <?php function getPrices(array $products) { // let's assume the $products parameter is an array of products. $prices = []; foreach($products as $p) { if($p->stock > 0) { $prices[] = $p->price; } } return $prices; } Now let's see how you can do the same with SQL which, among other things, is a declarative language: SELECT price FROM products WHERE stock > 0; Notice the difference? In the first example, you tell the computer what to do step by step, taking care of storing intermediary results yourself. In the second example, you only describe what you want and it will then be the role of the database engine to return the results. In a way, functional programming looks a lot more like SQL than the PHP code we just saw. Functions Functional programming, as it names suggests, revolves around functions. In order to apply functional techniques effectively, a language must support functions as a first-class citizen or first functions. This means that functions are considered like any other value. They can be created and passed around as parameters to other functions and they can be used as return values. Luckily, PHP is such a language, you can create functions at will, pass them around as parameters, and even return them. Another fundamental concept is the idea of a pure function or, in other words, functions that only use their input to produce a result. This means that you cannot use any kind of external or internal state to perform your computation. Another way to look at this is from the angle of dependencies. All of the dependencies of your functions need to be clearly declared in the signature. This helps a lot when someone tries to understand how and what your function is doing. Higher-order functions PHP functions can take functions as parameters and return functions as return values. A function that does either of those is called a higher-order function. It is as simple as that. There are a few of those that are commonly used in any functional code base. Map The map, or array_map, method in PHP is a higher order function that applies a given callback to all the elements of a collection. The return value is a collection in the same order. A simple example is as follows: <?php function square(int $x): int { return $x * $x; } $squared = array_map('square', [1, 2, 3, 4]); /* $squared contains [1, 4, 9, 16] */ Filter The filter, or array_filter, method in PHP is a higher order function that keeps only certain elements of a collection based on a Boolean predicate. The return value is a collection that will only contain elements returning true for the predicate function. A simple example is as follows: <?php function odd(int $a): bool { return $a % 2 === 1; } $filtered = array_filter([1, 2, 3, 4, 5, 6], 'odd'); /* $filtered contains [1, 3, 5] */ Fold or reduce Folding refers to a process where you reduce a collection to a return value using a combining function. Depending on the language, this operation can have multiple names—fold, reduce, accumulate, aggregate, or compress. As with other functions related to arrays, the PHP version is the array_reduce function. You may be familiar with the array_sum function, which calculates the sum of all the values in an array. This is in fact a fold and can be easily written using the array_reduce function: <?php function sum(int $carry, int $i): int { return $carry + $i; } $summed = array_reduce([1, 2, 3, 4], 'sum', 0); /* $summed contains 10 */ You don't necessarily need to use the elements to produce a value. You could, for example, implement a naive replacement for the in_array method using fold: <?php function in_array2(string $needle, array $haystack): bool { $search = function(bool $contains, string $i) use ($needle) : bool { return $needle == $i ? true : $contains; }; return array_reduce($haystack, $search, false); } var_dump(in_array2('two', ['one', 'two', 'three'])); // bool(true) Recursion In the academic sense, recursion is the idea of dividing a problem into smaller instances of the same problem. For example, if you need to scan a directory recursively, you first scan the starting directory and then scan its children and grandchildren. Most programming languages support recursion by allowing a function to call itself. This idea is often what is described as being recursion. Let's see how we can scan a directory using recursion: <?php function searchDirectory($dir, $accumulator = []) { foreach (scandir($dir) as $path) { // Ignore hidden files, current directory and parent directory if(strpos($path, '.') === 0) { continue; } $fullPath = $dir.DIRECTORY_SEPARATOR.$path; if(is_dir($fullPath)) { $accumulator = searchDirectory($path, $accumulator); } else { $accumulator[] = $fullPath; } } return $accumulator; } We start by using the scandir method to obtain all files and directories. Then, if we encounter a child directory, we call the function on it again. Otherwise, we simply add the file to the accumulator. This function is recursive because it calls itself. You can write this using control structures, but as you don't know in advance what the depth of your folder hierarchy is, the code will probably be a lot messier and harder to understand. Trampolines Each time you call a function, information gets added to the memory. This can be an issue when doing recursion as you only have a limited amount of memory available. Until the last recursive call, memory usage will continue growing and a stack overflow can happen. The only way we can avoid stack growth is to return a value instead of calling a new function. This value can hold the information that is needed to perform a new function call, which will continue the computation. This also means that we need some cooperation from the caller of the function. This helpful caller is called a trampoline and here is how it works: The trampoline calls our f function Instead of making a recursive call, the f function returns the next call encapsulated inside a data structure with all the arguments The trampoline extracts the information and performs a new call to the f function Repeat the two last steps until the f function returns a real value The trampoline receives a value and returns those to the real caller If you want to use trampolines in your own project, I invite you to install the following library, which offers some helpers as compared to our crude implementation: composer require functional-php/trampoline Here is an example taken from the documentation: <?php use FunctionalPHPTrampoline as t; function factorial($n, $acc = 1) { return $n <= 1 ? $acc : tbounce('factorial', $n - 1, $n * $acc); }; Composing functions Previously, we discussed the idea of building blocks and small pure functions. But, so far, we haven't even hinted at how those can be used to build something bigger. What good is a building block if you cannot use it? The answer partly lies in function's composition. As it is often the case in functional programming, the concept is borrowed from mathematics. If you have two functions f and g, you can create a third function by composing them. The usual notation in mathematics is (f   g)(x), which is equivalent to calling them one after the other as f(g(x)). You can compose any two given functions really easily with PHP using a wrapper function. Say, you want to display a title in all caps and only safe HTML characters: <?php function safe_title2(string $s) { return strtoupper(htmlspecialchars($s)); } Functional libraries for PHP often come with a helper that can create new functions out of multiple subparts easily. For example, using Lars Strojny's Functional PHP library, you can write the following: <?php $titles4 = array_map(compose('htmlspecialchars', 'strtoupper', 'trim'), $titles); Partial application You might want to set some parameters of a function but leave some of them unassigned for later. For example, we might want to create a function that returns an excerpt of a blog post. The dedicated term for setting such a value is "to bind a parameter" or "bind an argument". The process itself is called partial application and the new function is set to be partially applied. The Functional PHP library also comes with helpers to partially apply a function: <?php use function Functionalpartial_right; use function Functionalpartial_left; use function Functionalpartial_any; use const Functional…; $excerpt = partial_right('substr', 0, 5); echo $excerpt('Lorem ipsum dolor si amet.'); // Lorem $fixed_string = partial_left('substr', 'Lorem ipsum dolor si amet.'); echo $fixed_string(6, 5); // ipsum $start_placeholder = partial_any('substr', 'Lorem ipsum dolor si amet.', …(), 5); echo $start_placeholder(12); // dolor Currying Currying is often used as a synonym to partial application. Although both concepts allows us to bind some parameters of a function, the core ideas are a bit different. The idea behind currying is to transform a function that takes multiple arguments into a sequence of functions that take one argument. As this might be a bit hard to grasp, let's try to curry the substr method. The result is called a curryied function. Again, a helper to create such functions is available in the Functional PHP library: <?php use function Functionalcurry; function add($a, $b, $c, $d) { return $a + $b + $c + $d; } $curryedAdd = curry('add'); $add10 = $curryedAdd(10); $add15 = $add10(5); $add42 = $add15(27); $add42(10); // -> 52 Benefits of functional programing As we just saw, the functional world is moving, adoption by the enterprise world is growing, and even new imperative languages are taking inspiration from functional languages. But why it is so? Reduce the cognitive burden on developers You've probably often read or heard that a programmer should not be interrupted because even a small interruption can lead to literally tens of minutes being lost. This is partly due to the cognitive burden or, in other words, the amount of information you have to keep in memory in order to understand the problem or function at hand. By forcing you to clearly state the dependencies of your functions and avoiding using any kind of external data, functional programming helps a lot in writing self-contained code that can be readily understood and thus reduces cognitive burden a lot. Software with fewer bugs We just saw that functional programming reduces the cognitive burden and makes your code easier to reason about. This is already a huge win when it comes to bugs because it will allow you to spot issues quickly as you will spend less time understanding how the code works to focus on what it should do. But all the benefits we've just seen have another advantage. They make testing a lot easier too! If you have a pure function and you test it with a given set of values, you have the absolute certitude that it will always return exactly the same thing in production. Easier refactoring Refactoring is never easy. However, since the only inputs of a pure function are its parameters and its sole output is the returned value, things are simpler. If you're refactored function continues to return the same output for a given input, you can have the guarantee that your software will continue to work. You cannot forget to set a few state somewhere in an object because your function are side-effect free. Enforcing good practices This article and the related book are the proof that functional programming is more about the way we do things instead of a particular language. You can use functional techniques in nearly any language that has functions. Your language still needs to have certain properties, but not that much. I like to talk about having a functional mindset. If it is so, why do companies move to functional languages? Because those languages enforce the best practice that we will learn in this book. In PHP, you will have to always remember to use functional techniques. In Haskell, you cannot do anything else, the language forces you to write pure functions. Summary This small article is by no mean a complete introduction to functional programming, this is what the Functional PHP book is for. I however hope I convinced you it is a set of techniques worth learning. We only brushed the surface here, all topics are covered more in depth in the various chapters. You will also learn about more advanced topics like the following: Functors, applicatives, and Monads Type systems Pattern matching Functional reactive programming Property-based testing Parallel execution of functional code There is also a whole chapter about using functional programming in conjunction with various frameworks like Symfony, Drupal, Laraval, and Wordpress. Resources for Article: Further resources on this subject: Understanding PHP basics [article] Developing Middleware [article] Continuous Integration [article]

In this article by Kishore Gaddam, author of the book Building Bots with Microsoft Bot Framework, we introduced what is Microsoft Bot Framework and how it helps in the development of bots. (For more resources related to this topic, see here.) Since past several decades, the corporate, government, and business world has experienced several waves of IT architecture foundations, moving from mainframes, to minicomputers, to distributed PCs, to the Internet, to social/mobile and now the Cloud/Internet of Thuings (IoT) Stack. We call this the Sixth wave of Corporate IT, and like its predecessors, Cloud and IoT technologies are causing significant disruption and displacement, even while it drives new levels of productivity. Each architecture focused on key business processes and supported killer technology applications to drive new levels of value. Very soon we will be looking at an enormous networked interconnection of everyday machines to one another, as well as to humans. Machine-to-machine-to-human connectivity will have a profound impact on the consumer and corporate IT experience. As these machines become social and talkto us, we have enormous opportunity to greatly enhance their value proposition through improved product quality, customer experience, and lowered cost of operations. A heightened consumer expectation for more personal and real-time interactions is driving business to holistically embrace the next wave of technology innovation like Cloud, IoT, and Bots to boost business performance. In this age of billions of connected devices, there is a need for such a technology where our apps could talk back, like bots? Bots that have specific purposes and talk to any device or any app or to anyone, Bots that live in cloud, Bots that we can talk to you via any communication channel such as email, text, voice, chat, and others. Bots can go where no apps have gone before when it comes to machine-to-machine-to-human connectivity. And to make this happen we will need a whole new platform. A Platform for Conversations. Conservation as a Service (CaaS) Messaging apps in general are becoming a second home screen for many people, acting as their entry point into the internet. And where the youngins are, the brands will follow. Companies are coming to messaging apps as bots and apps, to offer everything from customer service to online shopping and banking. Conversations are shaping up be the next major human-computer interface. Thanks to advances in natural language processing and machine learning, the tech is finally getting fast and accurate enough to be viable. Imagine a platform where language is the new UI layer. When we talk about conversation as a platform, there are 3 parts: There are people talking to people – Skype translator as an ex where people can communicate across cross languages Then there is the presence or being able to enhance a conversation by the ability to be present and interact remotely Then there is personal assistance and the bots Think of Bots as the new mechanism that you can converse with. Instead of looking through multiple mobile apps or pages and pages of websites, you can call on any application as a bot within the conversational canvas. Bots are the new apps and digital assistants are the meta apps. This way intelligence is infused into all our interactions. This leads us to Microsoft Bot Framework, which is a comprehensive offering from Microsoft to build and deploy high quality bots for your users to interact using Conversation as a Platform (CaaP). This is a framework that lets you build and connect intelligent bots. The idea is that they interact naturally wherever your users are talking, like Skype, Slack, Facebook Messenger, Text/SMS, and others. Basically any kind of channel that you use today as a human being to talk to other people, well, you will be able to use them to talk to bots all using natural language. Microsoft Bot Framework is a Microsoft operated CaaP service and an open source SDK. Bot Framework is one of the many tools Microsoft is offering to for building a complete Bot. Other tools include Language Understanding Intelligent Service (LUIS), Speech APIs, Microsoft Azure, Cortana Intelligence Suit and many more. Your Bot The Microsoft Bot Builder SDK is one of three main components of the Microsoft Bot Framework. First you have to build your bot. Your bot lives in the cloud and you host it yourself. You write it just like a web service component using Node.js or C#, like a ASP.NET WebAPI component. Microsoft Bot builder SDK is open source and so you will have more languages and web stack get supported over time. Your bot will have its own logic, but you also need a conversation logic using dialogs to model a conversation. The Bot builder SDK gives you facilities for this and there are many types of dialogs that are included from simple Yes/No questions to full natural language understanding or LUIS, which is one of the API's provided in Microsoft Cognitive Services: Bot Connector Bot Connector is hosted and operated by Microsoft. Think of it as a central router between your bots and many channels to communicate with your bots. Apart from routing messages, it would be managing state within the conversation. The Bot Connector is an easy way to create a single back-end and then publish to a bunch of different platforms called channels. Bot Directory Bot Directory is where user will be able to find bots. It's like app store for mobile apps. The Bot Directory is a public directory of all reviewed bots registered through the developer portal. Users will be able to discover, try, and add bots to their favorite conversation experiences from the Bot Directory. Anyone can access it and anyone can submit Bots to the directory. As you begin your development with Microsoft Bot Framework, you might be wondering how to best get started. Bots can be built in C#, however, Microsoft's Bot Framework can also be used to build bots using Node.js. For developing any bots, we need to first setup the development environment and have the right tools installed for successfully developing and deploying a bot. Let's see how we can setup a development environment using Visual Studio. Setting up development environment Let's first look at the Prerequisites required to set up the development environment: Prerequisites To use the Microsoft Bot Framework Connector, you must have: A Microsoft Account (Hotmail, Live, or Outlook) to log into the Bot Framework developer portal, which you will use to register your Bot. An Azure subscription (Free trial: https://azure.microsoft.com/en-us/). This Azure subscription is essential for having an Azure-accessible REST endpoint exposing a callback for the Connector service. Developer accounts on one or more communication services (such as Skype, Slack, Facebook) where your Bot will communicate. In addition, you may wish to have an Azure App Insights account so you can capture telemetry from your Bot. There are additionally different ways to go about building a Bot; from scratch, coded directly to the Bot Connector REST API, the Bot Builder SDK's for Node.js and .NET, and the Bot Connector .NET template which is what this quick start guide demonstrates. Setting up Bot Framework Connector SDK .NET This is a step-by-step guide to setting up dev environment to develop a Bot in C# using the Bot Framework Connector SDK .NET template: Install prerequisite software Visual Studio 2015 (latest update) - you can download the community version here for free: www.visualstudio.com Important: Please update all Visual Studio extensions to their latest versions to do so navigate to Tools | Extensions and Updates | Updates Download and install the Bot Application template: Download the file from the direct download link at http://aka.ms/bf-bc-vstemplate Save the zip file to your Visual Studio 2015 templates directory which is traditionally in %USERPROFILE%DocumentsVisual Studio 2015TemplatesProjectTemplatesVisual C# Open Visual Studio. Create a new C# project using the new Bot Application template. The template is a fully functional Echo Bot that takes the user's text utterance as input and returns it as output. In order to run however: The bot has to be registered with Bot Connector The AppId and AppPassword from the Bot Framework registration page have to be recorded in the project's web.config The project needs to be published to the web Emulator Use the Bot Framework Emulator to test your Bot application. The Bot Framework provides a channel emulator that lets you test calls to your Bot as if it were being called by the Bot Framework cloud service. To install the Bot Framework Emulator, download it from https://download.botframework.com/bf-v3/tools/emulator/publish.html. One installed, you're ready to test. First, start your Bot in Visual Studio using a browser as the application host. The following screenshot uses Microsoft Edge: Summary In this article, we introduced what is Microsoft Bot Framework and how it helps in the development of bots. Also, we have seen how to setup development environment, Emulator and the tools needed for programming. This article is based on the thought that programming knowledge and experience grow best when they grow together.  Resources for Article: Further resources on this subject: Talking to Bot using Browser [article] Webhooks in Slack [article] Creating our first bot, WebBot [article]

In this article by Mike Topalovich, author of the book Salesforce Lightning Application Development Essentials, we will discuss about Salesforce. As Salesforce developers, we know since Dreamforce '15, Salesforce has been all Lightning, all the time. The flagship CRM products – Sales Cloud and Service Cloud – have been rebranded to Sales Cloud Lightning and Service Cloud Lightning. In fact, many Salesforce products have undergone the Lightning treatment, the word Lightning being added to their product names overnight with few noticeable changes to the products themselves. This has led many in the Salesforce ecosystem to step back and ask, What is Lightning? (For more resources related to this topic, see here.) Lightning changes everything Lightning is not just the new Salesforce UI—it is a complete re-imagining of the entire Salesforce application and platform. Lightning represents a grand vision of unifying the Salesforce experience for everyone who interacts with Salesforce products or the Salesforce platform, on any device. In no uncertain terms, Lightning is the most important product update in the history of Salesforce as a company. Lightning represents a completely new vision for both the flagship CRM products and the platform. Salesforce is betting the company on it. Lightning changes not only how we interact with Salesforce, but how we design and develop solutions for the platform. Developers and ISV partners now have the ability to create rich, responsive applications using the same framework and design tools that Salesforce uses internally, ensuring that the user experience maintains a consistent look and feel across all applications. While the initial overuse of the term Lightning may be a source of confusion for many in the Salesforce ecosystem, don't let the noise drown out the vision. Lightning changes everything we know about Salesforce, but to understand how, we need to focus on the three key pillars of Lightning: Lightning Experience Salesforce Lightning Design System Lightning Component framework If we think about Lightning as the unified Salesforce experience across all our devices, it makes our mission much clearer. If we think about designing and developing responsive, reusable components for this new unified experience, Lightning makes a lot more sense. A brief history of Lightning The unified Lightning vision as we know it today has been rolled out in fits and starts. To understand how we arrived at the current vision for Lightning, we can look back on prior Dreamforce events as milestones in the history of Lightning. Dreamforce 2013 With Dreamforce '13, Salesforce recognized that the world was moving to a mobile-first mindset and rebranded their mobile application as Salesforce1. With Salesforce1, they also tried to sell the vision of a unified customer experience for the first time. According to a press release for Dreamforce '13, "Salesforce1 is a new social, mobile and cloud customer platform built to transform sales, service and marketing apps for the Internet of Customers." The vision was too ambitious at the time, the messaging was too confusing, and the platform was nowhere close to being ready to support any type of unified experience. Dreamforce 2014 Lightning emerged as a platform with Dreamforce '14. Branded as Salesforce1 Lightning, Salesforce opened up the underlying Aura JavaScript UI framework to developers for the first time, enabling the development of custom components for the Salesforce1 mobile application using the same technology that Salesforce had used to develop the Salesforce1 mobile application. The press release for Dreamforce '14 hinted at what was in store for Lightning, "Now customers, developers and Salesforce partners can take advantage of the new Lightning Framework to quickly build engaging apps across every device. The same framework Salesforce's internal development team uses to build apps can now be used to build custom Lightning components and create any user experience." At this point, Salesforce was still using the Salesforce1 branding for the unified end-to-end experience across Salesforce products and platforms, but we now officially had the Lightning Framework to work with. Dreamforce 2015 Dreamforce '15 may have been the official coming out party for Lightning, but in an unprecedented move for Salesforce, the company held a special pre-Dreamforce Meet the New Salesforce event on August 25, 2015, to announce the new Lightning Experience user interface as well as a complete rebranding of the end-to-end experience of Lightning. The Dreamforce event focused on strengthening the branding and educating developers, admins, and end users on what this unified experience meant for the future of Salesforce. Since then, Salesforce has been all Lightning, all the time. Dreamforce 2016 With Dreamforce '16 and the Winter '17 release of Salesforce, Lightning had finally arrived as a stable, optimized, enterprise-ready platform. Dreamforce '16 was less about hype and more about driving Lightning adoption. Sessions focused on design patterns and best practices rather than selling the platform to developers. New tooling was introduced to make the Lightning development experience something that Salesforce developers could get excited about. With Winter '17, Lightning Experience felt like a true unified end-to-end experience instead of a patchwork of functionality. The Winter '17 release notes were packed with enhancements that would get many organizations off the fence about Lightning and shift the adoption bell curve from toward an early majority from the early adopter state that it had been lingering in while Salesforce filled in the gaps in the platform. This is the Lightning we had been waiting for. Lightning Experience If someone were to ask you what the Lightning Experience was all about, your first reaction might be, "It's the new Salesforce UI." While that is technically correct, as Lightning Experience is the brand name given to the user interface that replaces what we now refer to as Salesforce Classic, the implementation of this user interface is part of the larger vision of a unified Salesforce experience across all devices. Lightning Experience takes the old way of doing things in Salesforce Classic – long, scrolling pages – and blows it up into tiny pieces. You need to reassemble those pieces in a way that makes the most sense for your business users. You can even create your own custom pieces, called components, and include those alongside components that Salesforce gives you out of the box, or components built by third parties that you download from the AppExchange. It is all completely seamless. Focusing on getting things done While the initial release of Lightning Experience focused on making salespeople more productive, the interface is rapidly evolving to improve on all areas of CRM. The ability to seamlessly transition work between desktop and mobile devices will enable every Salesforce user to find new ways to connect with customers and increase the effectiveness of business processes across the enterprise. While the initial release of Lightning Experience wasn't quite complete, it did include over 25 new features and a total redesign of many pages. Some of the notable improvements include: Component-based record pages that focused on getting work done in the context of the Salesforce object record, rather than having to scroll to find pertinent related information Completely redesigned reports and dashboards that enable visualizing customer data in new ways, with flexible layouts and additional filtering options An opportunity workspace designed to help salespeople get to closed won faster by focusing on actions instead of raw data Kanban boards for visualizing opportunities in their various deal stages and enabling salespeople to drag and drop multiple opportunities to new stages rather than having to edit and save individual records An enhanced Notes feature that enables users to create notes with rich text and attach them to multiple records at the same time Unified Salesforce Experience Following the mobile-first mindset adopted with the launch of the Salesforce1 platform, the same component framework that is used to power Salesforce1 is what underlies Lightning Experience. The incorporation of design principles from the Salesforce Lightning Design System ensures that users get the same responsive experience whether they access Salesforce from desktop browsers, mobile devices, tablets, wearables, or anything else that comes along in the near future. Developers and ISV partners can now build custom components and applications that plug right into Lightning Experience, rather than having to build custom pages and standalone applications. Blurring the lines between clicks and code Experienced Salesforce developers know that a key consideration when designing Salesforce solutions is to find the right balance between using declarative, out-of-the-box functionality and the programmatic capabilities of the platform. In the world of Salesforce development, we lovingly refer to this dichotomy as clicks versus code. With Lightning Experience and the introduction of Lightning App Builder, the discussion shifts from clicks or code to clicks and code, as developers can now build custom components and expose them to the Lightning App Builder interface, allowing admins to drag and drop these reusable components onto a canvas and assemble new Lightning pages. While this sentiment may strike fear, uncertainty, and doubt into the hearts of developers, any time we can move from code to clicks, or enable admins to maintain Salesforce customizations, it is a good thing. Lightning Experience enables a closer relationship between admins and developers. Developers can focus on building reusable components, admins can focus on maintaining the user experience. Salesforce Lightning Design System A design system is a collection of design principles, style guides, and elements that enable developers to focus on how components and applications work, while designers can focus on how the application looks and feels. The Salesforce Lightning Design System (SLDS) is a trusted, living, platform-agnostic design system that was built from the ground up to provide developers with everything needed to implement the look and feel of Lightning Experience and the Salesforce1 mobile application. SLDS ensures consistency across all components and applications, whether they are written by Salesforce developers, ISV partners, or even Salesforce itself when designing and implementing product features. Salesforce developed SLDS with four key design principles in mind: Clarity Efficiency Consistency Beauty These principles are applied to colors, typography, layout, icons, and more, throughout the accompanying CSS framework. Developers can implement the design system by including SLDS Cascading Style Sheets (CSS) and icon libraries in components and applications, and applying the appropriate CSS classes to component markup. SLDS also includes CSS style sheets for applying the design system to Visualforce components, Heroku, and native iOS applications. When SLDS was first introduced, adding the style sheets or icon libraries to a Salesforce org required installing an unmanaged package or uploading files as static resources and manually upgrading to new versions of the design system as they were released. As of the Winter '17 release of Salesforce, SLDS is included out of the box with all Salesforce orgs and no longer requires an explicit reference to the static resources from Lightning components or applications. You simply reference the appropriate SLDS class names in your component markup and it will be applied automatically, or you can use Lightning Base Components, which apply SLDS implicitly without additional markup. Lightning Component framework Traditionally, Salesforce UI design came down to two questions: Do I want to recreate the look and feel of Salesforce with custom Visualforce pages, or do I want to install a third-party framework to create rich application interfaces? The prevailing design pattern since the fall of Adobe Flash has been to use Visualforce to simply render the output from JavaScript frameworks such as Backbone, Angular, Ember, Sencha, and others. Developers could continue to follow MVC patterns by using Apex controllers to retrieve data from the Salesforce database. While this may have enabled a rich, responsive experience for certain applications developed for Salesforce, users still had to deal with a mixed experience across all of Salesforce, especially if multiple JavaScript frameworks were in use. Lighting Experience and the Lightning Component framework solve a problem that has long been a barrier to a truly unified experience for all Salesforce users: Providing a single, integrated framework for developers that enabled the creation of rich, responsive applications that could be seamlessly plugged in anywhere in the UI rather than having to stand alone in separate pages or standalone applications. Because the Lightning Component framework is what underlies Lightning Experience and the Salesforce1 mobile applications, we no longer have to choose a JavaScript framework developed and maintained outside of Salesforce. We can now create components and applications using a rich JavaScript framework, which is provided and maintained by Salesforce. What is the Lightning Component framework? The Lightning Component framework is a client-side UI framework that was built by Salesforce using the open source Aura UI framework. The framework uses JavaScript for client-side operations and exposes an API to access Salesforce data using Apex controllers. The Lightning Component framework was initially created to support development for the Salesforce1 mobile application, but is now the standard for responsive, client-side single-page applications for the end-to-end Salesforce user and developer experience across all browsers and devices. The framework provides a number of reusable out-of-the-box components for you to get started building your own Lightning components, and the platform is fully maintained and supported by Salesforce. The problem Salesforce solved with the Lightning Component framework was to give Salesforce developers a single standardized and supported JavaScript framework to move beyond the limitations of Visualforce and build rich applications with a common design system without having to select, install, and maintain a third-party framework. Eliminating the JavaScript framework sprawl within the Salesforce development ecosystem enables developers and admins to deliver customized business solutions with a standardized look and feel without having to learn and maintain yet another framework from another vendor that wasn't built specifically for the Salesforce platform. What is JavaScript? Along with HTML and CSS, JavaScript is one of the three core languages of web development. If you do not have a background in JavaScript, don't worry. Even though it will be the most difficult thing you will have to learn when coming up to speed on Lightning component-development, JavaScript has been around for decades and there are countless resources available for learning the language. JavaScript was introduced in the mid-1990s as a scripting language for the Netscape Navigator browser, with Microsoft subsequently releasing a version for Internet Explorer. By the late 1990s, JavaScript was standardized across browsers with the introduction of what is called ECMAScript. Today, all modern browsers include JavaScript engines. JavaScript will not be completely foreign to Apex developers, or anyone with an object-oriented programming (OOP) background, as it follows object-oriented principles. What will throw you off if you have an Apex background is the fact that JavaScript is loosely typed, whereas Apex is a strongly typed language. This will take some getting used to, conceptually. At its core, JavaScript is a language that is used to read and manipulate what we call the Document Object Model (DOM). The DOM is a programmatic interface that gets built when a browser reads an HTML document and converts each HTML element into what are called node objects. These HTML nodes form a tree-like structure in the DOM, and we can use JavaScript to find nodes, add nodes, remove nodes, and modify nodes to make our web applications dynamic. The other core function that JavaScript performs is listening for and handling events that occur in the browser. HTML itself is a static markup language and was not built to be dynamic in its rendering, which is why JavaScript was created to handle events such as clicking a mouse, changing a picklist value, or putting a form element into focus. You can write JavaScript functions to handle DOM events, and your JavaScript functions can in turn manipulate the DOM by adding, removing, or modifying DOM elements. Many of us have only had to learn JavaScript at a cursory level because JavaScript libraries such as jQuery and JavaScript frameworks such as Sencha ExtJS, Angular, Node, and Backbone take care of a lot of the heavy lifting for us when it comes to actual JavaScript programming. Lightning requires more direct JavaScript programming than many frameworks do, which gives you greater control over the functions in your Lightning components and applications, but unfortunately, you're going to have to bone up on your JavaScript knowledge before you can take advantage of that level of control. What are JavaScript frameworks? JavaScript frameworks handle much of the behind-the-scenes complexity of JavaScript coding and DOM manipulation, and give developers a simplified template-based approach to building web applications. While JavaScript itself does not follow the Model-View-Controller (MVC) design pattern, many JavaScript frameworks implement MVC to provide developers with a familiar architecture for separating the data model and view of an application with a logic, or controller, layer. Each component of the MVC architecture can be maintained separately and be brought together in a cohesive application. Some frameworks, such as Sencha ExtJS, may implement MVC but are more focused on enabling rich user interfaces by giving developers pre-built UI widgets that can be configured declaratively. Other frameworks are designed for touch-driven responsive mobile applications. There are dozens of JavaScript frameworks out there, the most common examples being Backbone.js, AngularJS, Ember.js, Knockout.js, React.js, and ExtJS, among others. What is Aura? Aura is an open-source JavaScript UI framework that is maintained by Salesforce. Aura is component-based, and uses JavaScript on the client-side frontend, with Java on the server-side backend. Aura is the framework that underpins the Lightning Component framework. While the Lightning Component framework and Aura have many similarities on the surface, do not try to use Aura components or functions that are not explicitly supported in the Lightning Component framework documentation. Many developers have already found that these undocumented features may work at first, but unless they are explicitly supported by Salesforce, they can be taken away at any time. Why should I use the Lightning Component framework? Why should Salesforce developers consider moving to Lightning Component development? For starters, Lightning is the future of Salesforce development. There is no way around it: Salesforce is betting the company on Lightning. If you ignore that and simply focus on the value that Lightning can provide, you will find that there are many compelling reasons for making the jump to Lightning. Responsive design From a single code base, you can create reusable components that will give your users a unified experience across all form factors. You no longer have to maintain separate desktop applications, tablet applications and mobile applications. Reusable components You can create components that can be consumed by other Lightning components and applications, allowing your component to be reused many times in many different places. Admins can use your components in the Lightning App Builder, allowing them to declaratively create new Lightning Pages with your components. This is where the line between declarative and programmatic development starts to blur! Better performance Because Lightning components render on the client and do not require expensive round trips to a server-side controller to update data and context, you can build components that are lightning fast. Have you ever used the AJAX components in Visualforce to do something as simple as create a type-ahead function for an input field? There was always a lag between the time an event was handled and the target component was re-rendered. With Lightning components, any change to an attribute value will automatically re-render a component, giving you the ability to create high-performing client-side Salesforce applications. Rendering on the client side will reduce the need for mobile applications to make expensive calls to a server API, significantly improving performance in low bandwidth situations. JavaScript + HTML5 + CSS If you have at least a cursory knowledge of web development, Lightning follows open standards and practices that you are already familiar with. While there may be a learning curve for Visualforce developers who do not have experience with JavaScript, HTML5, or CSS, the use of web standards rather than proprietary framework means there is a wealth of information and resources available to quickly get up to speed on the basics and transition to Lightning Component development. For new developers coming onto the Salesforce platform, Lightning provides an opportunity to quickly apply existing web-application development skills to hit the ground running, without having to first learn Apex or Visualforce. Event-driven architecture With Visualforce development, we are constrained to a fairly rigid architecture, which executes server-side controller actions when user-driven events occur, such as clicking a link or a button. Granted, you can use specialized Visualforce tags or component-specific event handlers to handle supported events, but this requires a significant amount of hard-coding to server-side controller methods. With Lightning, you can listen for and handle just about any DOM event or custom component event and determine what action you want to take when that event occurs. For example, you can handle the onchange event when a picklist value is selected and immediately call a function in your client-side controller to take action based on that value changing. You even have the ability to define and raise custom events, and determine how those events should be handled within your component hierarchy. Component encapsulation Encapsulation simply means that we can wall off the inner workings of a Lightning component and not expose the secret sauce behind it to another component or application that references it. This allows us to update the code within a Lightning Component without breaking any upstream components or applications. Encapsulation enables us to write reusable and efficient code that is easily maintainable. Summary In this article we have learned about Salesforce, the history about Lightning and the need to use the Lightning Component framework. Resources for Article: Further resources on this subject: Introducing Salesforce Chatter [article] Subscribing to a report [article] Learning How to Manage Records in Visualforce [article]

In this article by Javier Fernández González, author of the book Java 9 Concurrency Cookbook - Second Edition we will cover how to run tasks asynchronously. When you execute ForkJoinTask in ForkJoinPool, you can do it in a synchronous or asynchronous way. When you do it in a synchronous way, the method that sends the task to the pool doesn't return until the task sent finishes its execution. When you do it in an asynchronous way, the method that sends the task to the executor returns immediately, so the task can continue with its execution. (For more resources related to this topic, see here.) You should be aware of a big difference between the two methods. When you use the synchronized methods, the task that calls one of these methods (for example, the invokeAll() method) is suspended until the tasks it sent to the pool finish their execution. This allows the ForkJoinPool class to use the work-stealing algorithm to assign a new task to the worker thread that executed the sleeping task. On the contrary, when you use the asynchronous methods (for example, the fork() method), the task continues with its execution, so the ForkJoinPool class can't use the work-stealing algorithm to increase the performance of the application. In this case, only when you call the join() or get() methods to wait for the finalization of a task, the ForkJoinPool class can use that algorithm. In addition to RecursiveAction and RecursiveTask classes, Java 8 introduced a new ForkJoinTask with the CountedCompleter class. With this kind of tasks you can include a completion action that will be executed when is launched and there is no child pending tasks. This mechanism is based in a method included in the class (the onCompletion() method) and a counter of pending tasks. This counter is initialized to zero by default and you can increment it when you need in an atomic way. Normally, you will increment this counter one by one when you launch a child task. Finally, when a task has finished is execution, you can try to complete the execution of the task and consequently, executes the onCompletion() method. If the pending count is bigger than zero, it is decremented by one. If it's zero, the onCompletion() method is executed and then the parent task is tried to complete. In this article, you will learn how to use the asynchronous methods provided by the ForkJoinPool and CountedCompleter classes for the management of tasks. You are going to implement a program that will search for files with a determined extension inside a folder and its subfolders. The CountedCompleter class you're going to implement will process the content of a folder. For each subfolder inside that folder, it will send a new task to the ForkJoinPool class in an asynchronous way. For each file inside that folder, the task will check the extension of the file and add it to the result list if it proceeds. When a task is completed, it will insert the result lists of all its child tasks in its result task. How to do it... Follow these steps to implement the example: Create a class named FolderProcessor and specify that it extends the CountedCompleter class parameterized with the List<String> type. public class FolderProcessor extends CountedCompleter<List<String>> { Declare the serial version UID of the class. This element is necessary because the parent class of the RecursiveTask class, the ForkJoinTask class, implements the Serializable interface. private static final long serialVersionUID = -1826436670135695513L; Declare a private String attribute named path. This attribute will store the full path of the folder this task is going to process. private String path; Declare a private String attribute named extension. This attribute will store the name of the extension of the files this task is going to look for. private String extension; Declare two List private attributes named tasks and resultList. We will use the first one to store all the child tasks launched from this task and the other one to store the list of results of this task. private List<FolderProcessor> tasks; private List<String> resultList; Implement one constructor for the class to initialize its attributes and its parent class. We declared this constructor as protected as it will only be used internally protected FolderProcessor (CountedCompleter<?> completer, String path, String extension) { super(completer); this.path=path; this.extension=extension; } We implement other public constructor to be used externally. As the task created by this constructor won't have parent task, we don't include this object as parameter. public FolderProcessor (String path, String extension) { this.path=path; this.extension=extension; } Implement the compute() method. As the base class of our task is the CountedCompleter class, the return type of this method is void. @Override protected void compute() { First, initialize the two list attributes. resultList=new ArrayList<>(); tasks=new ArrayList<>(); Get the content of the folder. File file=new File(path); File content[] = file.listFiles(); For each element in the folder, if there is a subfolder, create a new FolderProcessor object and execute it asynchronously using the fork() method. We use the first constructor of the class and pass the current task as the completer task of the new one. We also increment the counter of pending tasks using the addToPendingCount() method. if (content != null) { for (int i = 0; i < content.length; i++) { if (content[i].isDirectory()) { FolderProcessor task=new FolderProcessor(this, content[i].getAbsolutePath(), extension); task.fork(); addToPendingCount(1); tasks.add(task); Otherwise, compare the extension of the file with the extension you are looking for using the checkFile() method and, if they are equal, store the full path of the file in the list of strings declared earlier. } else { if (checkFile(content[i].getName())){ list.add(content[i].getAbsolutePath()); } } } If the list of the FolderProcessor subtasks has more than 50 elements, write a message to the console to indicate this circumstance. if (tasks.size()>50) { System.out.printf("%s: %d tasks ran.n",file.getAbsolutePath(),tasks.size()); } } Finally, try to complete the current task using the tryComplete() method: tryComplete(); } Implement the onCompletion() method. This method will be executed when all the child tasks (all the tasks that have been forked from the current task) have finished their execution. We add the result list of all the child tasks to the result list of the current task. @Override public void onCompletion(CountedCompleter<?> completer) { for (FolderProcessor childTask : tasks) { resultList.addAll(childTask.getResultList()); } } Implement the checkFile() method. This method compares if the name of a file passed as a parameter ends with the extension you are looking for. If so, the method returns the true value, otherwise it returns the false value. private boolean checkFile(String name) { return name.endsWith(extension); } Finally, implement the getResultList() method to return the result list of a task. The code of this method is very simple so it won't be included. Implement the main class of the example by creating a class named Main with a main() method. public class Main { public static void main(String[] args) { Create ForkJoinPool using the default constructor. ForkJoinPool pool=new ForkJoinPool(); Create three FolderProcessor tasks. Initialize each one with a different folder path. FolderProcessor system=new FolderProcessor("C:\Windows", "log"); FolderProcessor apps=new FolderProcessor("C:\Program Files","log"); FolderProcessor documents=new FolderProcessor("C:\Documents And Settings","log"); Execute the three tasks in the pool using the execute() method. pool.execute(system); pool.execute(apps); pool.execute(documents); Write to the console information about the status of the pool every second until the three tasks have finished their execution. do { System.out.printf("******************************************n"); System.out.printf("Main: Parallelism: %dn",pool.getParallelism()); System.out.printf("Main: Active Threads: %dn",pool.getActiveThreadCount()); System.out.printf("Main: Task Count: %dn",pool.getQueuedTaskCount()); System.out.printf("Main: Steal Count: %dn",pool.getStealCount()); System.out.printf("******************************************n"); try { TimeUnit.SECONDS.sleep(1); } catch (InterruptedException e) { e.printStackTrace(); } } while ((!system.isDone())||(!apps.isDone())||(!documents.isDone())); Shut down ForkJoinPool using the shutdown() method. pool.shutdown(); Write the number of results generated by each task to the console. List<String> results; results=system.join(); System.out.printf("System: %d files found.n",results.size()); results=apps.join(); System.out.printf("Apps: %d files found.n",results.size()); results=documents.join(); System.out.printf("Documents: %d files found.n",results.size()); How it works... The following screenshot shows part of an execution of this example: The key of this example is in the FolderProcessor class. Each task processes the content of a folder. As you know, this content has the following two kinds of elements: Files Other folders If the task finds a folder, it creates another FolderProcessor object to process that folder and sends it to the pool using the fork() method. This method sends the task to the pool that will execute it if it has a free worker-thread or it can create a new one. The method returns immediately, so the task can continue processing the content of the folder. For every file, a task compares its extension with the one it's looking for and, if they are equal, adds the name of the file to the list of results. Once the task has processed all the content of the assigned folder, we try to complete the current task. As we explained in the introduction of this article, when we try to complete a task, the code of the CountedCompleter looks for the value of the pending task counter. If this value is bigger than 0, it decrease of that counter. On the contrary, if the value is 0, the task executes the onCompletion() method and then try to completes its parent task. In our case, when a task is processing a folder and it finds a subfolder, it creates a new child task, launch that task using the fork() method and increment the counter of pending tasks. So, when a task has processed all its content, the counter of pending tasks of the task will be equal to the number of child tasks we have launched. When we call the tryComplete() method, if the folder of the current task has subfolders, this call will decrease the number of pending tasks. Only when all its child tasks have been completed, its onCompletion() method is executed. If the folder of the current task hasn't got any subfolders, the counter of pending tasks will be zero and the onComplete() method will be called immediately and then it will try to complete its parent task. By this way, we create a tree of tasks from top to bottom that are completed from bottom to top. In the onComplete() method, we process all the result lists of the child tasks and add their elements in the result list of the current task. The ForkJoinPool class also allows the execution of tasks in an asynchronous way. You have used the execute() method to send the three initial tasks to the pool. In the Main class, you also finished the pool using the shutdown() method and wrote information about the status and the evolution of the tasks that are running in it. The ForkJoinPool class includes more methods that can be useful for this purpose. There's more... In this example we have used the addToPendingCount() method to increment the counter of pending tasks, but we have other methods we can use to change the value of this counter. setPendingCount(): This method establish the value of the counter of pending tasks. compareAndSetPendingCount(): This method receives two parameters. The first one is the expected value and the second one is the new value. If the value of the counter of pending tasks is equal to the expected value, establish its value to the new one. decrementPendingCountUnlessZero(): This method decrements the value of the counter of pending tasks unless it's equal to zero. The CountedCompleter class also includes other methods to manage the completion of the tasks. These are the most significant ones: complete(): This method executes the onCompletion() method independently of the value of the counter of pending tasks try to complete its completer (parent) task. onExceptionalCompletion(): This method is executed when the completeExceptionally() method has been called or the compute() method has thrown an Exception. Override this method to include your code to process those exceptions. In this example, you have used the join() method to wait for the finalization of tasks and get their results. You can also use one of the two versions of the get() method with this purpose: get(long timeout, TimeUnit unit): This version of the get() method, if the result of the task isn't available, waits the specified time for it. If the specified period of time passes and the result isn't yet available, the method returns a null value. The TimeUnit class is an enumeration with the following constants: DAYS, HOURS, MICROSECONDS, MILLISECONDS, MINUTES, NANOSECONDS, and SECONDS. The join() method can't be interrupted. If you interrupt the thread that called the join() method, the method throws an InterruptedException exception. Summary In this article we learned how to use the asynchronous methods provided by the ForkJoinPool and CountedCompleter classes for the management of tasks. Resources for Article: Further resources on this subject: Why Bother? – Basic [article] Getting Started with Sorting Algorithms in Java [article] Saying Hello to Java EE [article]

In this article by Mark Brummel, David Studebaker, Christopher D Studebaker, authors of the book Programming Microsoft Dynamics NAV - Fifth Edition, we will discuss how Microsoft Dynamics NAV has one of the largest installed user bases of any enterprise resource planning (ERP) system serving over 100,000 companies and one million plus individual users. The community of supporting organizations, consultants, implementers, and developers continues to grow and prosper. The capabilities of the off-the-shelf product increase with every release. The selection of the add-on products and services expands both in variety and depth. (For more resources related to this topic, see here.) The release of Microsoft Dynamics NAV 2017 continues its 20+ year history of continuous product improvement. It provides more user options for access and output formatting. For new installations, NAV 2017 includes tools for rapid implementation. For all installations, it provides enhanced business functionality and more support for ERP computing in the cloud, including integration with Office 365. NAV 2017 is also the base foundation for Microsoft Dynamics 365 for Financials, which is a cloud-based subset of the product that you can customize using extensions. Our goal in this article is to gain a big picture understanding of NAV 2017. In this article, we will take a look at NAV 2017, including the following: A general overview of NAV 2017 NAV 2017 – An ERP system NAV 2017 is an integrated set of business applications designed to service a wide variety business operations. Microsoft Dynamics NAV 2017 is an ERP system. An ERP system integrates internal and external data across a variety of functional areas including manufacturing, accounting, supply chain management, customer relationships, service operations, and human resources management, as well as the management of other valued resources and activities. By having many related applications well integrated, a full featured ERP system provides an enter data once, use many ways information processing toolset. NAV 2017 ERP addresses the following functional areas (and more): Basic accounting functions (for example, general ledger, accounts payable, and accounts receivable) Order processing and inventory (for example, sales orders, purchase orders, shipping, inventory, and receiving) Relationship management (for example, vendors, customers, prospects, employees, and contractors) Planning (for example, MRP, sales forecasting, and production forecasting) Other critical business areas (for example, manufacturing, warehouse management, marketing, cash management, and fixed assets) A good ERP system such as NAV 2017 is modular in design, which simplifies implementation, upgrading, modification, integration with third-party products, and expansion for different types of clients. All the modules in the system share a common database and, where appropriate, common data. The groupings of individual NAV 2017 functions following is based on the Departments menu structure, supplemented by information from Microsoft marketing materials. The important thing is to understand the overall components that make up the NAV 2017 ERP system: NAV 2017 has two quite different styles of user interface (UI). One UI, the development environment, targets developers. The other UI style, the RoleTailored client, targets end users. In NAV 2017, there are four instances of the RoleTailored client – for Windows, for web interaction, for tablet use, and a phone client, which was introduced in NAV 2016. The example images in the following module descriptions are from the RoleTailored client's Departments menu in the Windows client. Financial management Financial management is the foundation of any ERP system. No matter what the business is, the money must be kept flowing, and the flow of money must be tracked. The tools that help to manage the capital resources of the business are part of NAV 2017's Financial Management module. These include all or part of the following application functions: General ledger—managing overall finances of the firm Cash management and banking—managing inventory of money Accounts receivable—tracking incoming revenue Accounts payable—tracking outgoing funds Analytical accounting—analyzing various flows of funds Inventory and fixed assets—managing inventories of goods and equipment Multi-currency and multi-language—supporting international business activities The Financial Management section of the Departments menu is as follows: Manufacturing NAV 2017 manufacturing is general-purpose enough to be appropriate for Make to Stock (MTS), Make to Order (MTO), and Assemble to Order (ATO), as well as various subsets and combinations of those. Although off-the-shelf NAV is not particularly suitable for most process manufacturing and some of the very high volume assembly line operations, there are third-party add-on and add-in enhancements available for these applications. As with most of the NAV application functions, manufacturing can be implemented to be used in a basic mode or as a full featured system. NAV manufacturing includes the following functions: Product design (BOMs and routings)—managing the structure of product components and the flow of manufacturing processes Capacity and supply requirements planning—tracking the intangible and tangible manufacturing resources Production scheduling (infinite and finite), execution, and tracking quantities and costs, plus tracking the planned use manufacturing resources, both on an unconstrained and constrained basis The Manufacturing section of the Departments menu is as follows: Supply chain management Obviously, some of the functions categorized as part of NAV 2017 supply chain management (SCM), for example, sales and purchasing) are actively used in almost every NAV implementation. The supply chain applications in NAV include all or parts of the following applications: Sales order processing and pricing—supporting the heart of every business Purchasing (including Requisitions)—planning, entering, pricing, and processing purchase orders Inventory management—managing inventories of goods and materials Warehouse management including receiving and shipping—managing the receipt, storage, retrieval, and shipment of material and goods in warehouses Even though we might consider Assembly part of Manufacturing, the standard NAV 2017 Departments menu includes it in the Warehouse section: As a whole, these functions constitute the base components of a system appropriate for distribution operations, including those that operate on an ATO basis. Business Intelligence and reporting Although Microsoft marketing materials identify Business Intelligence (BI) and reporting as though it were a separate module within NAV, it's difficult to physically identify it as such. Most of the components used for BI and reporting purposes are (appropriately) scattered throughout various application areas. In the words of one Microsoft document, Business Intelligence is a strategy, not a product. Functions within NAV that support a BI strategy include the following: Standard reports—distributed ready-to-use by end users Account schedules and analysis reports—a specialized report writer for General Ledger data Query, XML port and Report designers—developer tools to support the creation of a wide variety of report formats, charts, and XML and CSV files Analysis by dimensions—a capability embedded in many of the other tools Interfaces into Microsoft Office and Microsoft Office 365 including Excel—communications of data either into NAV or out of NAV RDLC report viewer—provides the ability to present NAV data in a variety of textual and graphic formats, including user interactive capabilities Interface capabilities such as DotNet interoperability and web services—technologies to support interfaces between NAV 2017 and external software products NAV 2017 has standard packages for Power BI, both integrated on the role center as well as dashboards Artificial Intelligence A new feature of NAV 2017 is integration with Cortana intelligence, which is used for forecasting. The algorithms used in Artificial Intelligence (AI) can be used to give users extra information to make business decisions. Relationship Management NAV's Relationship Management (RM) functionality is definitely the little sister (or, if you prefer, little brother) of the fully featured standalone Microsoft CRM system. The big advantage of NAV RM is its tight integration with NAV customer and sales data. With NAV 2016, Microsoft introduced a new way of integrating with CRM using OData. Also falling under the heading of the customer relationship module is the NAV Service Management (SM) functionality. While the RM component shows up in the menu as part of sales and marketing, the SM component is identified as an independent function in the menu structure: Relationship management: Marketing campaigns—plan and manage promotions Customer activity tracking—analyze customer orders To do lists—manage what is to be done and track what has been done Service management: Service contracts—support service operations Labor and part consumption tracking—track the resources consumed by the service business Planning and dispatching—managing service calls Human resource management NAV Human Resources is a very small module, but it relates to a critical component of the business, employees. Basic employee data can be stored and reported via the master table (in fact, one can use human resources (HR) to manage data about individual contractors in addition to employees). A wide variety of individual employee attributes can be tracked by the use of dimensions fields: Employee tracking—maintain basic employee description data Skills inventory—inventory of the capabilities of employees Absence tracking—maintain basic attendance information Employee statistics—tracking government required employee attribute data such as age, gender, length of service Project management The NAV project management module consists of the jobs functionality supported by the resources functionality. Projects can be short or long term. They can be external (in other words - billable) or internal. This module is often used by third parties as the base for vertical market add-ons (such as construction or job oriented manufacturing). This application area includes parts or all of the following functions: Budgeting and cost tracking—managing project finances Scheduling—planning project activities Resource requirements and usage tracking—managing people and equipment Project accounting—tracking the results Summary In this article, we covered some basic definitions of terms related to NAV. We had a quick overview highlighting NAV 2017 components and how NAV 2017 is designed to service a wide variety of business operations. Resources for Article:  Further resources on this subject: The Sales and Purchase Process [article] Manage Security in Excel [article] Code Analysis and Debugging Tools in Microsoft Dynamics NAV 2009 [article]

In this article by Anil Mahtani, Luis Sánchez, Enrique Fernández, and Aaron Martinez, authors of the book Effective Robotics Programming with ROS, Third Edition, you will learn the structure of ROS and the parts it is made up of. Furthermore, you will start to create nodes and packages and use ROS with examples using Turtlesim. The ROS architecture has been designed and divided into three sections or levels of concepts: The Filesystem level The Computation Graph level The Community level (For more resources related to this topic, see here.) The first level is the Filesystem level. In this level, a group of concepts are used to explain how ROS is internally formed, the folder structure, and the minimum number of files that it needs to work. The second level is the Computation Graph level where communication between processes and systems happens. In this section, we will see all the concepts and mechanisms that ROS has to set up systems, handle all the processes, and communicate with more than a single computer, and so on. The third level is the Community level, which comprises a set of tools and concepts to share knowledge, algorithms, and code between developers. This level is of great importance; as with most open source software projects, having a strong community not only improves the ability of newcomers to understand the intricacies of the software as well as solve the most common issues, it is also the main force driving its growth. Understanding the ROS Filesystem level The ROS Filesystem is one of the strangest concepts to grasp when starting to develop projects in ROS, but with time and patience, the reader will easily become familiar with it and realize its value for managing projects and its dependencies. The main goal of the ROS Filesystem is to centralize the build process of a project while at the same time provide enough flexibility and tooling to decentralize its dependencies. Similar to an operating system, an ROS program is divided into folders, and these folders have files that describe their functionalities: Packages: Packages form the atomic level of ROS. A package has the minimum structure and content to create a program within ROS. It may have ROS runtime processes (nodes), configuration files, and so on. Package manifests: Package manifests provide information about a package, licenses, dependencies, compilation flags, and so on. A package manifest is managed with a file called package.xml. Metapackages: When you want to aggregate several packages in a group, you will use metapackages. In ROS Fuerte, this form for ordering packages was called Stacks. To maintain the simplicity of ROS, the stacks were removed, and now, metapackages make up this function. In ROS, there exists a lot of these metapackages; for example, the navigation stack. Metapackage manifests: Metapackage manifests (package.xml) are similar to a normal package, but with an export tag in XML. It also has certain restrictions in its structure. Message (msg) types: A message is the information that a process sends to other processes. ROS has a lot of standard types of messages. Message descriptions are stored in my_package/msg/MyMessageType.msg. Service (srv) types: Service descriptions, stored in my_package/srv/MyServiceType.srv, define the request and response data structures for services provided by each process in ROS. In the following screenshot, you can see the content of the turtlesim package. What you see is a series of files and folders with code, images, launch files, services, and messages. Keep in mind that the screenshot was edited to show a short list of files; the real package has more: The workspace In general terms, the workspace is a folder which contains packages, those packages contain our source files and the environment or workspace provides us with a way to compile those packages. It is useful when you want to compile various packages at the same time and it is a good way to centralize all of our developments. A typical workspace is shown in the following screenshot. Each folder is a different space with a different role: The source space: In the source space (the src folder), you put your packages, projects, clone packages, and so on. One of the most important files in this space is CMakeLists.txt. The src folder has this file because it is invoked by cmake when you configure the packages in the workspace. This file is created with the catkin_init_workspace command. The build space: In the build folder, cmake and catkin keep the cache information, configuration, and other intermediate files for our packages and projects. Development (devel) space: The devel folder is used to keep the compiled programs. This is used to test the programs without the installation step. Once the programs are tested, you can install or export the package to share with other developers. You have two options with regard to building packages with catkin. The first one is to use the standard CMake workflow. With this, you can compile one package at a time, as shown in the following commands: $ cmakepackageToBuild/ $ make If you want to compile all your packages, you can use the catkin_make command line, as shown in the following commands: $ cd workspace $ catkin_make Both commands build the executable in the build space directory configured in ROS. Another interesting feature of ROS is its overlays. When you are working with a package of ROS, for example, turtlesim, you can do it with the installed version, or you can download the source file and compile it to use your modified version. ROS permits you to use your version of this package instead of the installed version. This is very useful information if you are working on an upgrade of an installed package. Packages Usually, when we talk about packages, we refer to a typical structure of files and folders. This structure looks as follows: include/package_name/: This directory includes the headers of the libraries that you would need. msg/: If you develop nonstandard messages, put them here. scripts/: These are executable scripts that can be in Bash, Python, or any other scripting language. src/: This is where the source files of your programs are present. You can create a folder for nodes and nodelets or organize it as you want. srv/: This represents the service (srv) types. CMakeLists.txt: This is the CMake build file. package.xml: This is the package manifest. To create, modify, or work with packages, ROS gives us tools for assistance, some of which are as follows: rospack: This command is used to get information or find packages in the system. catkin_create_pkg: This command is used when you want to create a new package. catkin_make: This command is used to compile a workspace. rosdep: This command installs the system dependencies of a package. rqt_dep: This command is used to see the package dependencies as a graph. If you want to see the package dependencies as a graph, you will find a plugin called package graph in rqt. Select a package and see the dependencies. To move between packages and their folders and files, ROS gives us a very useful package called rosbash, which provides commands that are very similar to Linux commands. The following are a few examples: roscd: This command helps us change the directory. This is similar to the cd command in Linux. rosed: This command is used to edit a file. roscp: This command is used to copy a file from a package. rosd: This command lists the directories of a package. rosls: This command lists the files from a package. This is similar to the ls command in Linux. Every package must contain a package.xml file, as it is used to specify information about the package. If you find this file inside a folder, it is very likely that this folder is a package or a metapackage. If you open the package.xml file, you will see information about the name of the package, dependencies, and so on. All of this is to make the installation and the distribution of these packages easy. Two typical tags that are used in the package.xml file are <build_depend> and <run _depend>. The <build_depend> tag shows which packages must be installed before installing the current package. This is because the new package might use functionality contained in another package. The <run_depend> tag shows the packages that are necessary to run the code of the package. The following screenshot is an example of the package.xml file: Metapackages As we have shown earlier, metapackages are special packages with only one file inside; this file is package.xml. This package does not have other files, such as code, includes, and so on. Metapackages are used to refer to others packages that are normally grouped following a feature-like functionality, for example, navigation stack, ros_tutorials, and so on. You can convert your stacks and packages from ROS Fuerte to Kinetic and catkin using certain rules for migration. These rules can be found at http://wiki.ros.org/catkin/migrating_from_rosbuild. In the following screenshot, you can see the content from the package.xml file in the ros_tutorialsmetapackage. You can see the <export> tag and the <run_depend> tag. These are necessary in the package manifest, which is also shown in the following screenshot: If you want to locate the ros_tutorialsmetapackage, you can use the following command: $ rosstack find ros_tutorials The output will be a path, such as /opt/ros/kinetic/share/ros_tutorials. To see the code inside, you can use the following command line: $ vim /opt/ros/kinetic/ros_tutorials/package.xml Remember that Kinetic uses metapackages, not stacks, but the rosstack find command-line tool is also capable of finding metapackages. Messages ROS uses a simplified message description language to describe the data values that ROS nodes publish. With this description, ROS can generate the right source code for these types of messages in several programming languages. ROS has a lot of messages predefined, but if you develop a new message, it will be in the msg/ folder of your package. Inside that folder, certain files with the .msg extension define the messages. A message must have two main parts: fields and constants. Fields define the type of data to be transmitted in the message, for example, int32, float32, and string, or new types that you have created earlier, such as type1 and type2. Constants define the name of the fields. An example of an msg file is as follows: int32 id float32vel string name In ROS, you can find a lot of standard types to use in messages, as shown in the following table list: Primitive type Serialization C++ Python bool (1) unsigned 8-bit int uint8_t(2) bool int8 signed 8-bit int int8_t int uint8 unsigned 8-bit int uint8_t int(3) int16 signed 16-bit int int16_t int uint16 unsigned 16-bit int uint16_t int int32 signed 32-bit int int32_t int uint32 unsigned 32-bit int uint32_t int int64 signed 64-bit int int64_t long uint64 unsigned 64-bit int uint64_t long float32 32-bit IEEE float float float float64 64-bit IEEE float double float string ascii string (4) std::string string time secs/nsecs signed 32-bit ints ros::Time rospy.Time duration secs/nsecs signed 32-bit ints ros::Duration rospy.Duration A special type in ROS is the header type. This is used to add the time, frame, and sequence number. This permits you to have the messages numbered, to see who is sending the message, and to have more functions that are transparent for the user and that ROS is handling. The header type contains the following fields: uint32seq time stamp string frame_id You can see the structure using the following command: $ rosmsg show std_msgs/Header Thanks to the header type, it is possible to record the timestamp and frame of what is happening with the robot. ROS provides certain tools to work with messages. The rosmsg tool prints out the message definition information and can find the source files that use a message type. In upcoming sections, we will see how to create messages with the right tools. Services ROS uses a simplified service description language to describe ROS service types. This builds directly upon the ROS msg format to enable request/response communication between nodes. Service descriptions are stored in .srv files in the srv/ subdirectory of a package. To call a service, you need to use the package name, along with the service name; for example, you will refer to the sample_package1/srv/sample1.srv file as sample_package1/sample1. Several tools exist to perform operations on services. The rossrv tool prints out the service descriptions and packages that contain the .srv files, and finds source files that use a service type. If you want to create a service, ROS can help you with the service generator. These tools generate code from an initial specification of the service. You only need to add the gensrv() line to your CMakeLists.txt file. In upcoming sections, you will learn how to create your own services. Understanding the ROS Computation Graph level ROS creates a network where all the processes are connected. Any node in the system can access this network, interact with other nodes, see the information that they are sending, and transmit data to the network: The basic concepts in this level are nodes, the master, Parameter Server, messages, services, topics, and bags, all of which provide data to the graph in different ways and are explained in the following list: Nodes: Nodes are processes where computation is done. If you want to have a process that can interact with other nodes, you need to create a node with this process to connect it to the ROS network. Usually, a system will have many nodes to control different functions. You will see that it is better to have many nodes that provide only a single functionality, rather than have a large node that makes everything in the system. Nodes are written with an ROS client library, for example, roscpp or rospy. The master: The master provides the registration of names and the lookup service to the rest of the nodes. It also sets up connections between the nodes. If you don't have it in your system, you can't communicate with nodes, services, messages, and others. In a distributed system, you will have the master in one computer, and you can execute nodes in this or other computers. Parameter Server: Parameter Server gives us the possibility of using keys to store data in a central location. With this parameter, it is possible to configure nodes while it's running or to change the working parameters of a node. Messages: Nodes communicate with each other through messages. A message contains data that provides information to other nodes. ROS has many types of messages, and you can also develop your own type of message using standard message types. Topics: Each message must have a name to be routed by the ROS network. When a node is sending data, we say that the node is publishing a topic. Nodes can receive topics from other nodes by simply subscribing to the topic. A node can subscribe to a topic even if there aren't any other nodes publishing to this specific topic. This allows us to decouple the production from the consumption. It's important that topic names are unique to avoid problems and confusion between topics with the same name. Services: When you publish topics, you are sending data in a many-to-many fashion, but when you need a request or an answer from a node, you can't do it with topics. Services give us the possibility of interacting with nodes. Also, services must have a unique name. When a node has a service, all the nodes can communicate with it, thanks to ROS client libraries. Bags: Bags are a format to save and play back the ROS message data. Bags are an important mechanism to store data, such as sensor data, that can be difficult to collect but is necessary to develop and test algorithms. You will use bags a lot while working with complex robots. In the following diagram, you can see the graphic representation of this level. It represents a real robot working in real conditions. In the graph, you can see the nodes, the topics, which node is subscribed to a topic, and so on. This graph does not represent messages, bags, Parameter Server, and services. It is necessary for other tools to see a graphic representation of them. The tool used to create the graph is rqt_graph. These concepts are implemented in the ros_comm repository. Nodes and nodelets Nodes are executable that can communicate with other processes using topics, services, or the Parameter Server. Using nodes in ROS provides us with fault tolerance and separates the code and functionalities, making the system simpler. ROS has another type of node called nodelets. These special nodes are designed to run multiple nodes in a single process, with each nodelet being a thread (light process). This way, we avoid using the ROS network among them, but permit communication with other nodes. With that, nodes can communicate more efficiently, without overloading the network. Nodelets are especially useful for camera systems and 3D sensors, where the volume of data transferred is very high. A node must have a unique name in the system. This name is used to permit the node to communicate with another node using its name without ambiguity. A node can be written using different libraries, such as roscpp and rospy; roscpp is for C++ and rospy is for Python. Throughout we will use roscpp. ROS has tools to handle nodes and give us information about it, such as rosnode. The rosnode tool is a command-line tool used to display information about nodes, such as listing the currently running nodes. The supported commands are as follows: rosnodeinfo NODE: This prints information about a node rosnodekill NODE: This kills a running node or sends a given signal rosnodelist: This lists the active nodes rosnode machine hostname: This lists the nodes running on a particular machine or lists machines rosnode ping NODE: This tests the connectivity to the node rosnode cleanup: This purges the registration information from unreachable nodes A powerful feature of ROS nodes is the possibility of changing parameters while you start the node. This feature gives us the power to change the node name, topic names, and parameter names. We use this to reconfigure the node without recompiling the code so that we can use the node in different scenes. An example of changing a topic name is as follows: $ rosrun book_tutorials tutorialX topic1:=/level1/topic1 This command will change the topic name topic1 to /level1/topic1. To change parameters in the node, you can do something similar to changing the topic name. For this, you only need to add an underscore (_) to the parameter name; for example: $ rosrun book_tutorials tutorialX _param:=9.0 The preceding command will set param to the float number 9.0. Bear in mind that you cannot use names that are reserved by the system. They are as follows: __name: This is a special, reserved keyword for the name of the node __log: This is a reserved keyword that designates the location where the node's log file should be written __ip and __hostname: These are substitutes for ROS_IP and ROS_HOSTNAME __master: This is a substitute for ROS_MASTER_URI __ns: This is a substitute for ROS_NAMESPACE Topics Topics are buses used by nodes to transmit data. Topics can be transmitted without a direct connection between nodes, which means that the production and consumption of data is decoupled. A topic can have various subscribers and can also have various publishers, but you should be careful when publishing the same topic with different nodes as it can create conflicts. Each topic is strongly typed by the ROS message type used to publish it, and nodes can only receive messages from a matching type. A node can subscribe to a topic only if it has the same message type. The topics in ROS can be transmitted using TCP/IP and UDP. The TCP/IP-based transport is known as TCPROS and uses the persistent TCP/IP connection. This is the default transport used in ROS. The UDP-based transport is known as UDPROS and is a low-latency, lossy transport. So, it is best suited to tasks such as teleoperation. ROS has a tool to work with topics called rostopic. It is a command-line tool that gives us information about the topic or publishes data directly on the network. This tool has the following parameters: rostopicbw /topic: This displays the bandwidth used by the topic. rostopic echo /topic: This prints messages to the screen. rostopic find message_type: This finds topics by their type. rostopichz /topic: This displays the publishing rate of the topic. rostopic info /topic: This prints information about the topic, such as its message type, publishers, and subscribers. rostopic list: This prints information about active topics. rostopic pub /topic type args: This publishes data to the topic. It allows us to create and publish data in whatever topic we want, directly from the command line. rostopic type /topic: This prints the topic type, that is, the type of message it publishes. We will learn to use this command-line tool in upcoming sections. Services When you need to communicate with nodes and receive a reply, in an RPC fashion, you cannot do it with topics; you need to do it with services. Services are developed by the user, and standard services don't exist for nodes. The files with the source code of the services are stored in the srv folder. Similar to topics, services have an associated service type that is the package resource name of the .srv file. As with other ROS filesystem-based types, the service type is the package name and the name of the .srv file. ROS has two command-line tools to work with services: rossrv and rosservice. With rossrv, we can see information about the services' data structure, and it has exactly the same usage as rosmsg. With rosservice, we can list and query services. The supported commands are as follows: rosservice call /service args: This calls the service with the arguments provided rosservice find msg-type: This finds services by service type rosservice info /service: This prints information about the service rosservice list: This lists the active services rosservice type /service: This prints the service type rosserviceuri /service: This prints the ROSRPC URI service Messages A node publishes information using messages which are linked to topics. The message has a simple structure that uses standard types or types developed by the user. Message types use the following standard ROS naming convention; the name of the package, then /, and then the name of the .msg file. For example, std_msgs/ msg/String.msg has the std_msgs/String message type. ROS has the rosmsg command-line tool to get information about messages. The accepted parameters are as follows: rosmsg show: This displays the fields of a message rosmsg list: This lists all messages rosmsg package: This lists all of the messages in a package rosmsg packages: This lists all of the packages that have the message rosmsg users: This searches for code files that use the message type rosmsgmd5: This displays the MD5 sum of a message Bags A bag is a file created by ROS with the .bag format to save all of the information of the messages, topics, services, and others. You can use this data later to visualize what has happened; you can play, stop, rewind, and perform other operations with it. The bag file can be reproduced in ROS just as a real session can, sending the topics at the same time with the same data. Normally, we use this functionality to debug our algorithms. To use bag files, we have the following tools in ROS: rosbag: This is used to record, play, and perform other operations rqt_bag: This is used to visualize data in a graphic environment rostopic: This helps us see the topics sent to the nodes The ROS master The ROS master provides naming and registration services to the rest of the nodes in the ROS system. It tracks publishers and subscribers to topics as well as services. The role of the master is to enable individual ROS nodes to locate one another. Once these nodes have located each other, they communicate with each other in a peer-to-peer fashion. You can see in a graphic example the steps performed in ROS to advertise a topic, subscribe to a topic, and publish a message, in the following diagram: The master also provides Parameter Server. The master is most commonly run using the roscore command, which loads the ROS master, along with other essential components. Parameter Server Parameter Server is a shared, multivariable dictionary that is accessible via a network. Nodes use this server to store and retrieve parameters at runtime. Parameter Server is implemented using XMLRPC and runs inside the ROS master, which means that its API is accessible via normal XMLRPC libraries. XMLRPC is a Remote Procedure Call (RPC) protocol that uses XML to encode its calls and HTTP as a transport mechanism. Parameter Server uses XMLRPC data types for parameter values, which include the following: 32-bit integers Booleans Strings Doubles ISO8601 dates Lists Base64-encoded binary data ROS has the rosparam tool to work with Parameter Server. The supported parameters are as follows: rosparam list: This lists all the parameters in the server rosparam get parameter: This gets the value of a parameter rosparam set parameter value: This sets the value of a parameter rosparam delete parameter: This deletes a parameter rosparam dump file: This saves Parameter Server to a file rosparam load file: This loads a file (with parameters) on Parameter Server Understanding the ROS Community level The ROS Community level concepts are the ROS resources that enable separate communities to exchange software and knowledge. These resources include the following: Distributions: ROS distributions are collections of versioned metapackages that you can install. ROS distributions play a similar role to Linux distributions. They make it easier to install a collection of software, and they also maintain consistent versions across a set of software. Repositories: ROS relies on a federated network of code repositories, where different institutions can develop and release their own robot software components. The ROS Wiki: The ROS Wiki is the main forum for documenting information about ROS. Anyone can sign up for an account, contribute their own documentation, provide corrections or updates, write tutorials, and more. Bug ticket system: If you find a problem or want to propose a new feature, ROS has this resource to do it. Mailing lists: The ROS user-mailing list is the primary communication channel about new updates to ROS as well as a forum to ask questions about the ROS software. ROS Answers: Users can ask questions on forums using this resource. Blog: You can find regular updates, photos, and news at http://www.ros.org/news. Summary This article provided you with general information about the ROS architecture and how it works. You saw certain concepts and tools of how to interact with nodes, topics, and services. Remember that if you have queries about something, you can use the official resources of ROS from http://www.ros.org. Additionally, you can ask the ROS Community questions at http://answers.ros.org. Resources for Article: Further resources on this subject: The ROS Filesystem levels [article] Using ROS with UAVs [article] Face Detection and Tracking Using ROS, Open-CV and Dynamixel Servos [article]

In this article by Nic Jackson, author of the book Building Microservices with Go, we will examine the encoding/json package to see just how easy Go makes it for us to use JSON objects for our requests and responses. (For more resources related to this topic, see here.) Reading and writing JSON Thanks to the encoding/json package, which is built into the standard library, encoding and decoding JSON to and from Go types is both fast and easy. It implements the simplistic Marshal and Unmarshal functions; however, if we need them, the package also provides Encoder and Decoder types, which allow us greater control when reading and writing streams of JSON data. In this section, we are going to examine both of these approaches, but first let's take a look at how simple it is to convert a standard Go struct into its corresponding JSON string. Marshalling Go structs to JSON To encode JSON data, the encoding/json package provides the Marshal function, which has the following signature: func Marshal(v interface{}) ([]byte, error) This function takes one parameter, which is of the interface type, so that's pretty much any object you can think of, since interface represents any type in Go. It returns a tuple of ([]byte, error). You will see this return style quite frequently in Go. Some languages implement a try...catch approach, which encourages an error to be thrown when an operation cannot be performed. Go suggests the (return type, error) pattern, where the error is nil when an operation succeeds. In Go, unhanded errors are a bad thing, and while the language does implement the panic and recover functions, which resemble exception handling in other languages, the situations in which you should use them are quite different (The Go Programming Language, Donovan and Kernighan). In Go, panic causes normal execution to stop, and all deferred function calls in the Go routine are executed; the program will then crash with a log message. It is generally used for unexpected errors that indicate a bug in the code, and good, robust Go code will attempt to handle these runtime exceptions and return a detailed error object back to the calling function. This pattern is exactly what is implemented with the Marshal function. In case Marshal cannot create a JSON-encoded byte array from the given object, which could be due to a runtime panic, then this is captured and an error object detailing the problem is returned to the caller. Let's try this out, expanding on our existing example. Instead of simply printing a string from our handler, let's create a simple struct for the response and return that: 10 type helloWorldResponse struct { 11 Message string 12 } In our handler, we will create an instance of this object, set the message, and then use the Marshal function to encode it to a string before returning. Let's see what that will look like: 23 func helloWorldHandler(w http.ResponseWriter, r *http.Request) { 24 response := helloWorldResponse{Message: "HelloWorld"} 25 data, err := json.Marshal(response) 26 if err != nil { 27 panic("Ooops") 28 } 29 30 fmt.Fprint(w, string(data)) 31 } Now when we rerun our program and refresh our browser, we'll see the following output rendered in valid JSON: {"Message":"Hello World"} This is awesome, but the default behavior of Marshal is to take the literal name of the field and use that as the field in the JSON output. What if I prefer to use camel case and would rather see message—could we just rename the field in our struct message? Unfortunately, we can't because in Go, lowercase properties are not exported. Marshal will ignore these and will not include them in the output. All is not lost: the encoding/json package implements struct field attributes, which allow us to change the output for the property to anything we choose. The example code is as follows: 10 type helloWorldResponse struct { 11 Message string `json:"message"` 12 } Using the struct field's tags, we can have greater control over how the output will look. In the preceding example, when we marshal this struct, the output from our server would be the following: {"message":"Hello World"} This is exactly what we want, but we can use field tags to control the output even further. We can convert object types and even ignore a field altogether if we need to: struct helloWorldResponse { // change the output field to be "message" Message string `json:"message"` // do not output this field Author string `json:"-"` // do not output the field if the value is empty Date string `json:",omitempty"` // convert output to a string and rename "id" Id int `json:"id, string"` } The channel, complex types, and functions cannot be encoded in JSON. Attempting to encode these types will result in an UnsupportedTypeError being returned by the Marshal function. It also can't represent cyclic data structures, so if your stuct contains a circular reference, then Marshal will result in an infinite recursion, which is never a good thing for a web request. If we want to export our JSON pretty formatted with indentation, we can use the MarshallIndent function, which allows you to pass an additional string parameter to specify what you would like the indent to be—two spaces, not a tab, right? func MarshalIndent(v interface{}, prefix, indent string) ([]byte, error) The astute reader might have noticed that we are decoding our struct into a byte array and then writing that to the response stream. This does not seem to be particularly efficient, and in fact, it is not. Go provides encoders and decoders, which can write directly to a stream. Since we already have a stream with the ResponseWriter interface, let's do just that. Before we do so, I think we need to look at the ResponseWriter interface a little to see what is going on there. ResponseWriter is an interface that defines three methods: // Returns the map of headers which will be sent by the // WriteHeader method. Header() // Writes the data to the connection. If WriteHeader has not // already been called then Write will call // WriteHeader(http.StatusOK). Write([]byte) (int, error) // Sends an HTTP response header with the status code. WriteHeader(int)   If we have a ResponseWriter, how can we use this with fmt.Fprint(w io.Writer, a ...interface{})? This method requires a Writer interface as a parameter, and we have a ResponseWriter. If we look at the signature for Writer, we can see that it is the following: Write(p []byte) (n int, err error) Because the ResponseWriter interface implements this method, it also satisfies the Writer interface; therefore, any object that implements ResponseWriter can be passed to any function that expects Writer. Amazing! Go rocks—but we don't have an answer to our question: is there any better way to send our data to the output stream without marshalling to a temporary string before we return it? The encoding/json package has a function called NewEncoder. This returns an Encoder object, which can be used to write JSON straight to an open writer, and guess what—we have one of those: func NewEncoder(w io.Writer) *Encoder So instead of storing the output of Marshal into a byte array, we can write it straight to the HTTP response, as shown in the following code: func helloWorldHandler(w http.ResponseWriter, r *http.Request) { response := HelloWorldResponse{Message: "HelloWorld"} encoder := json.NewEncoder(w) encoder.Encode(&response) } We will look at benchmarking in a later chapter, but to see why this is important, here's a simple benchmark to check the two methods against each other; have a look at the output: go test -v -run="none" -bench=. -benchtime="5s" -benchmem testing: warning: no tests to run PASS BenchmarkHelloHandlerVariable 10000000 1211 ns/op 248 B/op 5 allocs/op BenchmarkHelloHandlerEncoder 10000000 662 ns/op 8 B/op 1 allocs/op ok github.com/nicholasjackson/building-microservices-in-go/chapter1/bench 20.650s Using the Encoder rather than marshalling to a byte array is nearly 50% faster. We are dealing with nanoseconds here, so that time may seem irrelevant, but it isn't; this was two lines of code. If you have that level of inefficiency throughout the rest of your code, your application will run slower, you will need more hardware to satisfy the load, and that will cost you money. There is nothing clever in the differences between the two methods—all we have done is understood how the standard packages work and chosen the correct option for our requirements. That is not performance tuning, that is understanding the framework. Unmarshalling JSON to Go structs Now that we have learned how we can send JSON back to the client, what if we need to read input before returning the output? We could use URL parameters, and we will see what that is all about in the next chapter, but usually, you will need more complex data structures, which include the service to accept JSON as part of an HTTP POST request. If we apply techniques similar to those we learned in the previous section (to write JSON), reading JSON is just as easy. To decode JSON into a stuct, the encoding/json package provides us with the Unmarshal function: func Unmarshal(data []byte, v interface{}) error The Unmarshal function works in the opposite way to Marshal: it allocates maps, slices, and pointers as required. Incoming object keys are matched using either the struct field name or its tag and will work with a case-insensitive match; however, an exact match is preferred. Like Marshal, Unmarshal will only set exported struct fields: those that start with an upper case letter. We start by adding a new struct to represent the request, while Unmarshal can decode the JSON into an interface{} array, which would be of one of the following types: map[string]interface{} // for JSON objects []interface{} // for JSON arrays Which type it is depends on whether our JSON is an object or an array. In my opinion, it is much clearer to the readers of our code if we explicitly state what we are expecting as a request. We can also save ourselves work by not having to manually cast the data when we come to use it. Remember two things: You do not write code for the compiler; you write code for humans to understand You will spend more time reading code than you do writing it We are going to do ourselves a favor by taking into account these two points and creating a simple struct to represent our request, which will look like this: 14 type helloWorldRequest struct { 15 Name string `json:"name"` 16 } Again, we are going to use struct field tags because while we could let Unmarshal do case-insensitive matching so that {"name": "World} would correctly unmarshal into the struct the same as {"Name": "World"}, when we specify a tag, we are being explicit about the request form, and that is a good thing. In terms of speed and performance, it is also about 10% faster, and remember: performance matters. To access the JSON sent with the request, we need to take a look at the http.Request object passed to our handler. The following listing does not show all the methods in the request, just the ones we are going to be immediately dealing with. For the full documentation, I recommend checking out the docs at https://godoc.org/net/http#Request. type Requests struct { … // Method specifies the HTTP method (GET, POST, PUT, etc.). Method string // Header contains the request header fields received by the server. The type Header is a link to map[string] []string. Header Header // Body is the request's body. Body io.ReadCloser … } The JSON that has been sent with the request is accessible in the Body field. The Body field implements the io.ReadCloser interface as a stream and does not return []byte or string data. If we need the data contained in the body, we can simply read it into a byte array, like the following example: 30 body, err := ioutil.ReadAll(r.Body) 31 if err != nil { 32 http.Error(w, "Bad request", http.StatusBadRequest) 33 return 34 } Here is something we'll need to remember: we are not calling Body.Close(); if we were making a call with a client, we would need to do this as it is not automatically closed; however, when used in a ServeHTTP handler, the server automatically closes the request stream. To see how this all works inside our handler, we can look at the following handler: 28 func helloWorldHandler(w http.ResponseWriter, r *http.Request) { 29 30 body, err := ioutil.ReadAll(r.Body) 31 if err != nil { 32 http.Error(w, "Bad request", http.StatusBadRequest) 33 return 34 } 35 36 var request HelloWorldRequest 37 err = json.Unmarshal(body, &request) 38 if err != nil { 39 http.Error(w, "Bad request", http.StatusBadRequest) 40 return 41 } 42 43 response := HelloWorldResponse{Message: "Hello " + request.Name} 44 45 encoder := json.NewEncoder(w) 46 encoder.Encode(response) 47 } Let's run this example and see how it works; to test it, we can simply use curl to send a request to the running server. If you feel more comfortable using a GUI tool, then Postman, which is available for the Google Chrome browser, will work just fine. Otherwise, feel free to use your preferred tool. $ curl localhost:8080/helloworld -d '{"name":"Nic"}' You should see the following response: {"message":"Hello Nic"} What do you think will happen if you do not include a body with your request? $ curl localhost:8080/helloworld If you guessed correctly that you would get an "HTTP status 400 Bad Request" error, then you win a prize. The following error replies to the request with the given message and status code: func Error(w ResponseWriter, error string, code int) Once we have sent this, we need to return, stopping further execution of the function as this does not close the ResponseWriter and return flow to the calling function automatically. You might think you are done, but have a go and see whether you can improve the performance of the handler. Think about the things we were talking about when marshaling JSON. Got it? Well, if not, here is the answer: again, all we are doing is using Decoder, which is the opposite of the Encoder function we used when writing JSON, as shown in the following code example. This nets an instant 33% performance increase, and with less code, too. 27 func helloWorldHandler(w http.ResponseWriter, r *http.Request) { 28 29 var request HelloWorldRequest 30 decoder := json.NewDecoder(r.Body) 31 32 err := decoder.Decode(&request) 33 if err != nil { 34 http.Error(w, "Bad request", http.StatusBadRequest) 35 return 36 } 37 38 response := HelloWorldResponse{Message: "Hello " + request.Name} 39 40 encoder := json.NewEncoder(w) 41 encoder.Encode(response) 42 } Now that you can see just how easy it is to encode and decode JSON with Go, I would recommend taking 5 minutes to spend some time digging through the documentation for the encoding/json package as there is a whole lot more than you can do with it: https://golang.org/pkg/encoding/json/ Summary In this article, we looked at encoding and decoding data using the encoding/json package. Resources for Article: Further resources on this subject: Microservices – Brave New World [article] A capability model for microservices [article] Breaking into Microservices Architecture [article]

In this article by Steven F. Daniel, author of the book Mastering Xamarin UI Development, you will learn how to build stunning, maintainable, cross-platform mobile application user interfaces with the power of Xamarin. In this article, we will cover the following topics: Understanding the MVVM pattern architecture Implement the MVVM ViewModels within the app (For more resources related to this topic, see here.) Understanding the MVVM pattern architecture In this section we will be taking a look at the MVVM pattern architecture and the communication between the components that make up the architecture. The MVVM design pattern is designed to control the separation between the user interfaces (Views), the ViewModels that contain the actual binding to the Model, and the models that contain the actual structure of the entities representing information stored on a database or from a web service. The following screenshot shows the communication between each of the components contained within the MVVM design pattern architecture: The MVVM design pattern is divided into three main areas, as you can see from the preceding screenshot and these are explained in the following table: MVVM type Description Model The Model is basically a representation of business related entities used by an application, and is responsible for fetching data from either a database, or web service, and then de-serialized to the entities contained within the Model. View The View component of the MVVM model basically represents the actual screens that make up the application, along with any custom control components, and control elements, such as buttons, labels, and text fields. The Views contained within the MVVM pattern are platform-specific and are dependent on the platform APIs that are used to render the information that is contained within the application's user interface. ViewModel The ViewModel essentially controls, and manipulates the Views by acting as their main data context. The ViewModel contains a series of properties that are bound to the information contained within each Model, and those properties are then bound to each of the Views to represent this information within the user interface. ViewModels can also contain command objects that provide action-based events that can trigger the execution of event methods that occur within the View. For example, when the user taps on a toolbar item, or a button. ViewModels generally implement the INotifyPropertyChanged interface. Such a class fires a PropertyChanged event whenever one of their properties change. The data binding mechanism in Xamarin.Forms attaches a handler to this PropertyChanged event so it can be notified when a property changes and keep the target updated with the new value. Now that you have a good understanding of the components that are contained within MVVM design pattern architecture, we can begin to create our entity models and update our user interface files. In Xamarin.Forms, the term View is used to describe form controls, such as buttons and labels, and uses the term Page to describe the user interface or screen. Whereas, in MVVM, Views are used to describe the user interface, or screen. Implementing the MVVM ViewModels within your app In this section, we will begin by setting up the basic structure for our TrackMyWalks solution to include the folder that will be used to represent our ViewModels. Let's take a look at how we can achieve this, by following the steps: Launch the Xamarin Studio application and ensure that the TrackMyWalks solution is loaded within the Xamarin Studio IDE. Next, create a new folder within the TrackMyWalks PCL project, called ViewModels as shown in the following screenshot: Creating the WalkBaseViewModel for the TrackMyWalks app In this section, we will begin by creating a base MVVM ViewModel that will be used by each of our ViewModels when we create these, and then the Views (pages) will implement those ViewModels and use them as their BindingContext. Let's take a look at how we can achieve this, by following the steps: Create an empty class within the ViewModels folder, shown in the following screenshot: Next, choose the Empty Class option located within the General section, and enter in WalkBaseViewModel for the name of the new class file to create, as shown in the following screenshot: Next, click on the New button to allow the wizard to proceed and create the new empty class file, as shown in the preceding screenshot. Up until this point, all we have done is create our WalkBaseViewModel class file. This abstract class will act as the base ViewModel class that will contain the basic functionality that each of our ViewModels will inherit from. As we start to build the base class, you will see that it contains a couple of members and it will implement the INotifyPropertyChangedInterface,. As we progress through this article, we will build to this class, which will be used by the TrackMyWalks application. To proceed with creating the base ViewModel class, perform the following step as shown: Ensure that the WalkBaseViewModel.cs file is displayed within the code editor, and enter in the following code snippet: // // WalkBaseViewModel.cs // TrackMyWalks Base ViewModel // // Created by Steven F. Daniel on 22/08/2016. // Copyright © 2016 GENIESOFT STUDIOS. All rights reserved. // using System.ComponentModel; using System.Runtime.CompilerServices; namespace TrackMyWalks.ViewModels { public abstract class WalkBaseViewModel : INotifyPropertyChanged { protected WalkBaseViewModel() { } public event PropertyChangedEventHandler PropertyChanged; protected virtual void OnPropertyChanged([CallerMemberName] string propertyName = null) { var handler = PropertyChanged; if (handler != null) { handler(this, new PropertyChangedEventArgs(propertyName)); } } } } In the preceding code snippet, we begin by creating a new abstract class for our WalkBaseViewModel that implements from the INotifyPropertyChanged interface class, that allows the View or page to be notified whenever properties contained within the ViewModel have changed. Next, we declare a variable PropertyChanged that inherits from the PropertyChangedEventHandler that will be used to indicate whenever properties on the object have changed. Finally, within the OnPropertyChanged method, this will be called when it has determined that a change has occurred on a property within the ViewModel from a child class. The INotifyPropertyChanged interface is used to notify clients, typically binding clients, when the value of a property has changed. Implementing the WalksPageViewModel In the previous section, we built our base class ViewModel for our TrackMyWalks application, and this will act as the main class that will allow our View or pages to be notified whenever changes to properties within the ViewModel have been changed. In this section, we will need to begin building the ViewModel for our WalksPage,. This model will be used to store the WalkEntries, which will later be used and displayed within the ListView on the WalksPage content page. Let's take a look at how we can achieve this, by following the steps: First, create a new class file within the ViewModels folder called WalksPageViewModel, as you did in the previous section, entitled Creating the WalkBaseViewModel located within this article. Next, ensure that the WalksPageViewModel.cs file is displayed within the code editor, and enter in the following code snippet: // // WalksPageViewModel.cs // TrackMyWalks ViewModels // // Created by Steven F. Daniel on 22/08/2016. // Copyright © 2016 GENIESOFT STUDIOS. All rights reserved. // using System.Collections.ObjectModel; using TrackMyWalks.Models; namespace TrackMyWalks.ViewModels { public class WalksPageViewModel : WalkBaseViewModel { ObservableCollection<WalkEntries> _walkEntries; public ObservableCollection<WalkEntries> walkEntries { get { return _walkEntries; } set { _walkEntries = value; OnPropertyChanged(); } } In the above code snippet, we begin by ensuring that our ViewModel inherits from the WalkBaseViewModel class. Next, we create an ObservableCollection variable _walkEntries which is very useful when you want to know when the collection has changed, and an event is triggered that will tell the user what entries have been added or removed from the WalkEntries model. In our next step, we create the ObservableCollection constructor WalkEntries, that is defined within the System.Collections.ObjectModel class, and accepts a List parameter containing our WalkEntries model. The WalkEntries property will be used to bind to the ItemSource property of the ListView within the WalksMainPage. Finally, we define the getter (get) and setter (set) methods that will return and set the content of our _walkEntries when it has been determined when a property has been modified or not. Next, locate the WalksPageViewModel class constructor, and enter the following highlighted code sections:         public WalksPageViewModel() { walkEntries = new ObservableCollection<WalkEntries>() { new WalkEntries { Title = "10 Mile Brook Trail, Margaret River", Notes = "The 10 Mile Brook Trail starts in the Rotary Park near Old Kate, a preserved steam " + "engine at the northern edge of Margaret River. ", Latitude = -33.9727604, Longitude = 115.0861599, Kilometers = 7.5, Distance = 0, Difficulty = "Medium", ImageUrl = "http://trailswa.com.au/media/cache/media/images/trails/_mid/" + "FullSizeRender1_600_480_c1.jpg" }, new WalkEntries { Title = "Ancient Empire Walk, Valley of the Giants", Notes = "The Ancient Empire is a 450 metre walk trail that takes you around and through some of " + "the giant tingle trees including the most popular of the gnarled veterans, known as " + "Grandma Tingle.", Latitude = -34.9749188, Longitude = 117.3560796, Kilometers = 450, Distance = 0, Difficulty = "Hard", ImageUrl = "http://trailswa.com.au/media/cache/media/images/trails/_mid/" + "Ancient_Empire_534_480_c1.jpg" }, }; } } } In the preceding code snippet, we began by creating a new ObservableCollection for our walkEntries method and then added each of the walk list items that we would like to store within our model. As each item is added, the ObservableCollection, constructor is called, and the setter (set) method is invoked to add the item, and then the INotifyPropertyChanged event will be triggered to notify that a change has occurred. Summary In this article, you learned about the MVVM pattern architecture; we also implemented the MVVM ViewModels within the app. Additionally, we created and implemented the WalkBaseViewModel for the TrackMyWalks application. Resources for Article: Further resources on this subject: A cross-platform solution with Xamarin.Forms and MVVM architecture [article] Building a Gallery Application [article] Heads up to MvvmCross [article]

In this article by Jeeva Chelladhurai, Pethuru Raj Chelliah, and Vinod Singh, the authors of the book Learning Docker, Second Edition, we will learn how the Docker images are built by using Dockerfile, which is the standard way for bringing forth highly usable Docker images. Leveraging Dockerfile is the most competent way for building powerful images for the software development community. (For more resources related to this topic, see here.) Docker's integrated image building system The Docker images are the fundamental building blocks of containers. These images could be very basic operating environments such, as busybox or Ubuntu. Or the images could craft advanced application stacks for the enterprise and cloud IT environments. We could craft an image manually by launching a container from a base image, install all the required applications, make the necessary configuration file changes, and then commit the container as an image. As a better alternative, we could resort to the automated approach of crafting the images by using Dockerfile. Dockerfile is a text-based build script that contains special instructions in a sequence for building the right and the relevant images from the base images. The sequential instructions inside the Dockerfile can include the base image selection, installing the required application, adding the configuration and the data files, and automatically running the services as well as exposing those services to the external world. Thus, Docker file-based automated build system has remarkably simplified the image-building process. It also offers a great deal of flexibility in the way in which the build instructions are organized and in the way in which they visualize the complete build process. The Docker Engine tightly integrates this build process with the help of the docker build subcommand. In the client-server paradigm of Docker, the Docker server (or daemon) is responsible for the complete build process and the Docker command line interface is responsible for transferring the build context, including transferring Dockerfile to the daemon. In order to have a sneak peak into the Dockerfile integrated build system in this section, we introduce you to a basic Dockerfile. Then, we explain the steps for converting that Dockerfile into an image, and then launching a container from that image. Our Dockerfile is made up of two instructions, as shown here: $cat Dockerfile FROM busybox:latest CMD echo Hello World!! In the following, we cover/discuss the two instructions mentioned earlier: The first instruction is for choosing the base image selection. In this example, we select the busybox: latest image The second instruction is for carrying out the command CMD, that instructs the container to echo Hello World!! Now, let's proceed towards generating a Docker image by using the preceding Dockerfile by calling docker build along with the path of Dockerfile. In our example, we will invoke the docker build subcommand from the directory where we have stored Dockerfile, and the path will be specified by the following command: $sudo docker build . After issuing the preceding command, the build process will begin by sending build context to the daemon and then display the text shown here: Sending build context to Docker daemon 2.048 kB Step 1 : FROM busybox:latest The build process would continue and after completing itself, it will display the following: Successfully built 0a2abe57c325 In the preceding example, the image was built with the IMAGE ID0a2abe57c325. Let's use this image to launch a container by using the docker run subcommand as follows: $sudo docker run 0a2abe57c325 Hello World!! Cool, isn't it? With very little effort, we have been able to craft an image with busybox as the base image, and we have been able to extend that image to produce Hello World!!. This is a simple application, but the enterprise-scale images can also be realized by using the same technology. Now, let's look at the image details by using the docker images subcommand, as shown here: $ sudo docker images REPOSITORYTAG IMAGE IDCREATED VIRTUAL SIZE <none><none>0a2abe57c3252 hours ago2.433 MB Here, you may be surprised to see that the IMAGE(REPOSITORY) and TAG name have been listed as <none>.This is because we did not specify any image or any TAG name when we built this image. You could specify an IMAGE name and optionally a TAG name by using the docker tag subcommand, as shown here: $ sudo docker tag 0a2abe57c325 busyboxplus The alternative approach is to build the image with an image name during the build time by using the-t option for the docker build subcommand, as shown here: $sudo docker build -t busyboxplus . Since there is no change in the instructions in Dockerfile, the Docker Engine will efficiently reuse the old image that has ID0a2abe57c325 and update the image name to busyboxplus. By default, the build system would apply latest as the TAG name. This behavior can be modified by specifying the TAG name after the IMAGE name by having a : separator placed in between them. That is, <image name>:<tag name> is the correct syntax for modifying behaviors, wherein <image name> is the name of the image and <tag name> is the name of the tag. Once again, let's look at the image details by using the docker images subcommand, and you will notice that the image (Repository) name is busyboxplus and the tag name is latest: $ sudo docker images REPOSITORYTAG IMAGE IDCREATED VIRTUAL SIZE busyboxplus latest0a2abe57c3252 hours ago2.433 MB Building images with an image name is always recommended as the best practice. A quick overview of the Dockerfile's syntax In this section, we explain the syntax or the format of Dockerfile. A Dockerfile is made up of instructions, comments, parser directives, and empty lines, as shown here: # Comment INSTRUCTION arguments The instruction line of Dockerfile is made up of two components, where the instruction line begins with the instruction itself, which is followed by the arguments for the instruction. The instruction could be written in any case, in other words, it is case-insensitive. However, the standard practice or the convention is to use uppercase in order to differentiate it from the arguments. Let's take a relook at the content of Dockerfile in our previous example: FROM busybox:latest CMD echo Hello World!! Here, FROM is an instruction which has taken busybox:latest as an argument, and CMD is an instruction which has taken echo Hello World!! as an argument. The commentline The comment line in Dockerfile must begin with the# symbol. The # symbol after an instruction is considered as an argument. If the # symbol is preceded by a whitespace, then the docker build system would consider that as an unknown instruction and skip the line. Now, let's understand the preceding cases with the help of an example to get a better understanding of the comment line: A valid Dockerfile comment line always begins with a# symbol as the first character of the line: # This is my first Dockerfile comment The # symbol can be a part of an argument CMD echo ### Welcome to Docker ### If the # symbol is preceded by a whitespace, then it is considered as an unknown instruction by the build system # this is an invalid comment line The docker build system ignores any empty line in the Dockerfile and hence, the author of Dockerfile is encouraged to add comments and empty lines to substantially improve the readability of Dockerfile. The parser directives As the name implies, the parser directives instruct the Dockerfile parser to handle the content of the Dockerfile as specified in the directives. The parser directives are optional and they must be at the very top of a Dockerfile. Currently escape is the only supported directive. We use escape character to escape characters in a line or to extend a single line to multiple lines. On UNIX like platform is the escape character whereas on windows is a directory path separator and ` is the escape character. By default, Dockerfile parser considers as the escape character and you could override this on windows using escape parser directive as shown here: # escape=` The Dockerfile build instructions So far, we have looked at the integrated build system, the Dockerfile syntax and a sample lifecycle, wherein how a sample Dockerfile is leveraged for generating an image and how a container gets spun off from that image was discussed. In this section, we will introduce the Dockerfile instructions, their syntax, and a few befitting examples. The FROM instruction The FROM instruction is the most important one and it is the first valid instruction of a Dockerfile. It sets the base image for the build process. The subsequent instructions would use this base image and build on top of it. The Docker build system lets you flexibly use the images built by anyone. You can also extend them by adding more precise and practical features to them. By default, the Docker build system looks in the Docker host for the images. However, if the image is not found in the Docker host, then the Docker build system will pull the image from the publicly available Docker Hub Registry. The Docker build system will return an error if it could not find the specified image in the Docker host and the Docker Hub Registry. The FROM instruction has the following syntax: FROM <image>[:<tag>|@<digest>] In the preceding code statement, note the following: <image>: This is the name of the image which will be used as the base image <tag> or<digest>:Both tag and digest are optional attributes and you could qualify a particular Docker image version using either a tag or a digest. Tag latest is assumed by default if both tag and digest are not present. Here is an example of the FROM instruction with the image name centos: FROM centos The MAINTAINER instruction The MAINTAINER instruction is an informational instruction of a Dockerfile. This instruction capability enables the authors' to set the details in an image. Docker does not place any restrictions on placing the MAINTAINER instruction in Dockerfile. However, it is strongly recommended that you should place it after the FROM instruction. The following is the syntax of the MAINTAINER instruction, where <author's detail> can be in any text. However, it is strongly recommended that you should use the image, author's name and the e-mail address as shown in this code syntax: MAINTAINER <author's detail> Here is an example of the MAINTAINER instruction with the author name and the e-mail address: MAINTAINER Dr. Peter <peterindia@gmail.com> The COPY instruction The COPY instruction enables you to copy the files from the Docker host to the file system of the new image. The following is the syntax of the COPY instruction: COPY <src> ... <dst> The preceding code terms bear the explanations shown here: <src>: This is the source directory, the file in the build context, or the directory from where the docker build subcommand was invoked. ...: This indicates that multiple source files can either be specified directly or be specified by wildcards. <dst>:This is the destination path for the new image into which the source file or directory will get copied. If multiple files have been specified, then the destination path must be a directory and it must end with a slash /. Using an absolute path for the destination directory or a file has been recommended. In the absence of an absolute path, the COPY instruction will assume that the destination path will start from root /.The COPY instruction is powerful enough for creating a new directory and for overwriting the file system in the newly created image. The ADD instruction The ADD instruction is similar to the COPY instruction. However, in addition to the functionality supported by the COPY instruction, the ADD instruction can handle the TAR files and the remote URLs. We can annotate the ADD instruction as COPY on steroids. The following is the syntax of the ADD instruction: ADD <src> ... <dst> The arguments of the ADD instruction are very similar to those of the COPY instruction, as shown here: <src>: This is either the source directory or the file that is in the build context or in the directory from where the docker build subcommand will be invoked. However, the noteworthy difference is that the source can either be a tar file stored in the build context or be a remote URL. ...: This indicates that the multiple source files can either be specified directly or be specified by using wildcards. <dst>: This is the destination path for the new image into which the source file or directory will be copied. The ENV instruction The ENV instruction sets an environment variable in the new image. An environment variable is a key-value pair, which can be accessed by any script or application. The Linux applications use the environment variables a lot for a starting configuration. The following line forms the syntax of the ENV instruction: ENV <key><value> Here, the code terms indicate the following: <key>: This is the environment variable <value>: This is the value that is to be set for the environment variable The following lines give two examples for the ENV instruction, where, in the first line, DEBUG_LVL has been set to 3 and in the second line, APACHE_LOG_DIR has been set to /var/log/apache: ENV DEBUG_LVL 3 ENV APACHE_LOG_DIR /var/log/apache The ARG instruction The ARG instruction lets you define variables that can be passed during the Docker image build time. The Docker build subcommand supports --build-arg flag to pass value to the variables defined using ARG instruction. If you specify a build argument that was not defined in your Dockerfile, the build would fail. In other words, the build argument variables must be defined in the Dockerfile to be passed during the Docker image build time. The syntax of the ARG instruction is as follows: ARG<variable>[=<default value>] Wherein, the code terms mean the following: <variable>: This is the build argument variable <default value>: This is the default value you could optionally specify to the build argument variable The environmentvariables The environment variables declared using ENV or ARG instruction can be used in ADD, COPY, ENV, EXPOSE, LABEL, USER, WORKDIR, VOLUME, STOPSIGNAL and ONBUILD instruction. Here is an example of environment variable usage: ARGBUILD_VERSION LABEL com.example.app.build_version=${ BUILD_VERSION} The USER instruction The USER instruction sets the startup user ID or username in the new image. By default, the containers will be launched with root as the user ID or UID. Essentially, the USER instruction will modify the default user ID from root to the one specified in this instruction. The syntax of the USER instruction is as follows: USER <UID>|<UName> The USER instructions accept either <UID> or <UName> as its argument. <UID>: This is a numerical user ID <UName>: This is a valid user Name Following is an example for setting the default user ID at the time of startup to 73. Here 73 is the numerical ID of the user: USER 73 Though, it is recommended that you have a valid user ID to match with the /etc/passwd file, the user ID can contain any random numerical value. However, the username must match with a valid username in the /etc/passwd file, otherwise the docker run subcommand will fail and it will display the following error message: finalize namespace setup user get supplementary groups Unable to find user The WORKDIR instruction The WORKDIR instruction changes the current working directory from / to the path specified by this instruction. The ensuing instructions, such as RUN, CMD, and ENTRYPOINT will also work on the directory set by the WORKDIR instruction. The following line gives the appropriate syntax for the WORKDIR instruction: WORKDIR <dirpath> Here, <dirpath> is the path for the working directory to set in. The path can be either absolute or relative. In case of a relative path, it will be relative to the previous path set by the WORKDIR instruction. If the specified directory is not found in the target image file system, then the director will be created. The following line is a clear example of the WORKDIR instruction in a Dockerfile: WORKDIR /var/log The VOLUME instruction The VOLUME instruction creates a directory in the image file system, which can later be used for mounting volumes from the Docker host or the other containers. The VOLUME instruction has two types of syntax, as shown here: The first type is either exec or JSON array (all values must be within double-quotes (")). VOLUME ["<mountpoint>"] The second type is shell, as shown here: VOLUME <mountpoint> In the preceding lines, <mountpoint> is the mount point that has to be created in the new image. The EXPOSE instruction The EXPOSE instruction opens up a container network port for communicating between the container and the external world. The syntax of the EXPOSE instruction is as follows: EXPOSE <port>[/<proto>] [<port>[/<proto>]...] Here, the code terms mean the following: <port>:This is the network port that has to be exposed to the outside world. <proto>: This is an optional field provided for a specific transport protocol, such as TCP and UDP. If no transport protocol has been specified, then TCP is assumed to be the transport protocol. The EXPOSE instruction allows you to specify multiple ports in a single line. The LABEL instruction The LABEL instruction enables you to add key-value pairs as metadata to your Docker images. These metadata can be further leveraged to provide meaningful Docker image management and orchestration. The syntax of the LABEL instruction is as follows: LABEL<key-1>=<val-1><key-2>=<val-2> ... <key-n>=<val-n> The LABEL instruction can have one or more key-value pair. Though a Dockerfile can have more than one LABEL instruction, it is recommended to use single LABEL instruction with multiple key-value pairs. Here is an example for the LABEL instruction: LABEL version=”2.0” release-date=”2016-08-05” The preceding label keys are very simple and this could result in naming conflicts. Hence Docker recommends using namespaces to label keys using reverse domain notation. There is a community project called Label Schema that provides shared namespace. The shared namespace acts as a glue between the image creators and tool builders to provide standardized Docker image management and orchestration. Here is an example of LABEL instruction using Label Schema: LABELorg.label-schema.schema-version=”1.0” org.label-schema.version=”2.0” org.label-schema.description=”Learning Docker Example” The RUN instruction The RUN instruction is the real workhorse during the build time, and it can run any command. The general recommendation is to execute the multiple commands by using one RUN instruction. This reduces the layers in the resulting Docker image because the Docker system inherently creates a layer for each time an instruction is called in Dockerfile. The RUN instruction has two types of syntax. The first is the shell type, as shown here: RUN <command> Here, the <command> is the shell command that has to be executed during the build time. If this type of syntax is to be used, then the command is always executed by using /bin/sh -c. And, the second syntax type is either exec or the JSON array, as shown here: RUN ["<exec>", "<arg-1>", ..., "<arg-n>"] Wherein, the code terms mean the following: <exec>: This is the executable to run during the build time. <arg-1>, ..., <arg-n>: These are the variables (zero or more) number of the arguments for the executable. Unlike the first type of syntax, this type does not invoke /bin/sh -c. Hence, the types of shell processing, such as the variable substitution ($USER) and the wild card substitution (*,?), does not happen in this type. If shell processing is critical for you, then you are encouraged to use the shell type. However, if you still prefer the exec (JSON array type) type, then use your preferred shell as the executable and supply the command as an argument. For example, RUN ["bash", "-c", "rm", "-rf", "/tmp/abc"]. The CMD instruction The CMD instruction can run any command (or application), which is similar to the RUN instruction. However, the major difference between those two is the time of execution. The command supplied through the RUN instruction is executed during the build time, whereas the command specified through the CMD instruction is executed when the container is launched from the newly created image. Thus, the CMD instruction provides a default execution for this container. However, it can be overridden by the docker run subcommand arguments. When the application terminates, the container will also terminate along with the application and vice versa. The CMD instruction has three types of syntax, as shown here: The first syntax type is the shell type, as shown here: CMD <command> Wherein, the <command> is the shell command, which has to be executed during the launch of the container. If this type of syntax is used, then the command is always executed by using /bin/sh -c. The second type of syntax is exec or the JSON array, as shown here: CMD ["<exec>", "<arg-1>", ..., "<arg-n>"] Wherein, the code terms mean the following: <exec>: This is the executable, which is to be run during the container launch time <arg-1>, ..., <arg-n>: These are the variable (zero or more) number of the arguments for the executable The third type of syntax is also exec or the JSON array, which is similar to the previous type. However, this type is used for setting the default parameters to the ENTRYPOINT instruction, as shown here: CMD ["<arg-1>", ..., "<arg-n>"] Wherein, the code terms mean the following: <arg-1>, ..., <arg-n>: These are the variables (zero or more) number of the arguments for the ENTRYPOINT instruction Now, let's build a Docker image by using the docker build subcommand and cmd-demo as the image name. The docker build system will read the instruction from the Dockerfile that is stored in the current directory (.), and craft the image accordingly as shown here: $sudo docker build -t cmd-demo . Having built the image, we can launch the container by using the docker run subcommand, as shown here: $sudo docker run cmd-demo Dockerfile CMD demo Cool, isn't it? We have given a default execution for our container and our container has faithfully echoed Dockerfile CMD demo. However, this default execution can be easily overridden by passing another command as an argument to the docker run subcommand, as shown in the following example: $sudo docker run cmd-demo echo Override CMD demo Override CMD demo The ENTRYPOINT instruction The ENTRYPOINT instruction will help in crafting an image for running an application (entry point) during the complete lifecycle of the container, which would have been spun out of the image. When the entry point application is terminated, the container would also be terminated along with the application and vice versa. Thus, the ENTRYPOINT instruction would make the container function like an executable. Functionally, ENTRYPOINT is akin to the CMD instruction, but the major difference between the two is that the entry point application is launched by using the ENTRYPOINT instruction, which cannot be overridden by using the docker run subcommand arguments. However, these docker run subcommand arguments will be passed as additional arguments to the entry point application. Having said this, Docker provides a mechanism for overriding the entry point application through the--entrypoint option in the docker run subcommand. The --entrypoint option can accept only word as its argument and hence, it has limited functionality. Syntactically, the ENTRYPOINT instruction is very similar to the RUN, and the CMD instructions, and it has two types of syntax, as shown here: The first type of syntax is the shell type, as shown here: ENTRYPOINT <command> Here, <command> is the shell command, which is executed during the launch of the container. If this type of syntax is used, then the command is always executed by using /bin/sh -c. The second type of syntax is exec or the JSON array, as shown here: ENTRYPOINT ["<exec>", "<arg-1>", ..., "<arg-n>"] Wherein, the code terms mean the following: <exec>: This is the executable, which has to be run during the container launch time <arg-1>, ..., <arg-n>: These are the variable (zero or more) number of arguments for the executable Now, let's build a Docker image by using the docker build as the subcommand and entrypoint-demo as the image name. The docker build system would read the instruction from Dockerfile stored in the current directory (.) and craft the image, as shown here: $sudo docker build -t entrypoint-demo . Having built the image, we can launch the container by using the docker run subcommand: $sudo docker run entrypoint-demo Dockerfile ENTRYPOINT demo Here, the container will run like an executable by echoing the Dockerfile ENTRYPOINT demo string and then it will exit immediately. If we pass any additional arguments to the docker run subcommand, then the additional argument would be passed to the entry point command. Following is the demonstration of launching the same image with the additional arguments given to the docker run subcommand: $sudo docker run entrypoint-demo with additional arguments Dockerfile ENTRYPOINT demo with additional arguments Now, let's see an example where we override the build time entry point application with the--entrypoint option and then launch a shell (/bin/sh) in the docker run subcommand, as shown here: $sudo docker run --entrypoint="/bin/sh" entrypoint-demo / # The HEALTHCHECK instruction Any Docker container is designed to run just one process / application / service as a best practice and also to be uniquely compatible for the fast-evolving Micro services Architecture (MSA) The container dies if the process running inside the container dies. There is a possibility that the application running inside the container might be in an unhealthy state and such state must be externalized for effective container management. Here the HEALTHCHECK instruction comes handy to monitor the health of the containerized application by running a health monitoring command (or tool) on a prescribed interval. The syntax of the HEALTHCHECK instruction is as follows: HEALTHCHECK[<options>] CMD <command> Wherein, the code terms mean the following: <command>: The health check command is to be executed on a prescribed interval. If the command exit status is 0, the container is considered to be in health state. If the command exit status is 1, the container is considered to be in unhealthy state. <options>: By default the health check command is invoked every 30 seconds, the command timeout 30 seconds and the command is retried 3 times before the container is declared unhealthy. Optionally, you can modify the default interval, timeout and retries values using the following options: --interval=<DURATION> [default: 30s] --timeout=<DURATION> [default: 30s] --retries=<N> [default: 3] Here is an example for the HEALTHCHECK instruction: HEALTHCHECK--interval=5m --timeout=3s CMD curl -f http://localhost/ || exit 1 The ONBUILD instruction The ONBUILD instruction registers a build instruction to an image and this gets triggered when another image is built by using this image as its base image. Any build instruction can be registered as a trigger and those instructions will be triggered immediately after the FROM instruction in the downstream Dockerfile. Thus, the ONBUILD instruction can be used for deferring the execution of the build instruction from the base image to the target image. The syntax of the ONBUILD instruction is as follows: ONBUILD <INSTRUCTION> Wherein, <INSTRUCTION> is another Dockerfile build instruction, which will be triggered later. The ONBUILD instruction does not allow the chaining of another ONBUILD instruction. In addition, it does not allow the FROM, and MAINTAINER instruction as an ONBUILD trigger. Here is an example for the ONBUILD instruction: ONBUILD ADD config /etc/appconfig The STOPSIGNAL instruction The STOPSIGNAL instruction enables you to configure an exit signal for your container. The STOPSIGNAL instruction has the following syntax: STOPSIGNAL<signal> Wherein, <signal>is either a valid signal name like SIGKILL or a valid unsigned signal number The SHELL instruction The SHELL instruction allows us to override the default shell, that is,sh on Linux and cmd on Windows. The syntax of the SHELL instruction is as follows: SHELL ["<shell>", "<arg-1>", ..., "<arg-n>"] Wherein, the code terms mean the following: <shell>: The shell to be used during container runtime <arg-1>, ..., <arg-n>: These are the variables (zero or more) number of the arguments for the shell Summary Building the Docker images is a critical aspect of the Docker technology for streamlining the grueling journey of containerization. As indicated before, the Docker initiative has turned out to be disruptive and transformative for the containerization paradigm which has been present for a while now. Dockerfile is the most prominent one for producing the competent Docker images, which can be meticulously used across. We have illustrated all the commands, their syntax, and their usage techniques in order to empower you with all the easy-to-grasp details and this will simplify the image-building process for you. We have supplied a bevy of examples in order to substantiate the inner meaning of each command. Resources for Article: Further resources on this subject: Benefits and Components of Docker [article] Let's start with Extending Docker [article] Container Linking and Docker DNS [article]

In this article by Sheridan Yuen, author of the book Mastering Windows Presentation Foundation, we would learn that, when Windows Presentation Foundation (WPF) was first released as part of the .NET Framework version 3.0 in 2006, it was billed as the future of desktop application Graphical User Interface (GUI) languages and supporters claimed that it would put an end to the previous GUI technology, Windows Forms. However, as time passed, it has fallen far short of this claim. (For more resources related to this topic, see here.) There are three main reasons that WPF has not taken off as widely as previously expected. The first reason has nothing to do with WPF and stems from the recent push to host everything in the cloud and having web interfaces rather than desktop applications. The second reason relates to the very steep learning curve and the very different way of working that is required to master WPF. The last reason is that it is not a very efficient language and if a WPF application has lots of 'bells and whistles' in, then either the client computers will need to have additional RAM and/or graphics cards installed, or they could face a slow and stuttering user experience. This explains why many companies that make use of WPF today are in the finance industry, where they can afford to upgrade all users' computers to be able to run their applications optimally. This article will aim to make WPF more accessible to the rest of us by providing practical tips and tricks to help build our real-world applications more easily and more efficiently. One of the simplest changes with the biggest workflow improvements that we can make to improve the way we work with WPF is to follow the MVVM software architectural pattern. It describes how we can organize our classes to make our applications more maintainable, testable, and generally simpler to understand. In this article, we will take a closer look at this pattern and discover how it can help us to improve our applications. After discovering what MVVM is and what its benefits are, we'll learn several new ways to communicate between the various components in this new environment. We'll then focus on the physical structure of the code base in a typical MVVM application and investigate a variety of alternative arrangements. What is MVVM and how does it help? Model-View-View Model (MVVM) is a software architectural pattern that was famously introduced by John Gossman on his blog back in 2005 and is now commonly used when developing WPF applications. Its main purpose is to provide a Separation of Concerns between the business model, the User Interface (UI), and the business logic. It does this by dividing them into three distinct types of core components: Models, Views, and View Models. Let's take a look at how they are arranged and what each of these components represent. As we can see here, the View Models component sits between the Models and the Views and provides two-way access to each of them. It should be noted at this point that there should be no direct relationship between the Views and Models components and only loose connections between the other components. Let's now take a closer look at what each of these components represent. Models Unlike the other MVVM components, the Model constituent comprises of a number of elements. It encompasses the business data model along with its related validation logic and also the Data Access Layer (DAL), or data repositories, that provide the application with data access and persistence. The data model represents the classes that hold the data in the application. They typically mirror the columns in the database more or less, although it is common that they are hierarchical in form, and so may require joins to be performed in the data source in order to fully populate them. One alternative would be to design the data model classes to fit the requirements in the UI, but either way, the business logic or validation rules will typically reside in the same project as the data model. The code that is used to interface with whatever data persistence technology is used in our application is also included within the Models component of the pattern. Care should be taken when it comes to organizing this component in the code base, as there are a number of issues to take into consideration. We'll investigate this further in a while, but for now, let's continue to find out more about the components in this pattern. View Models The View Models can be explained easily; each View Model provides its associated View with all of the data and functionality that it requires. In some ways, they can be considered to be similar to the old Windows Forms code behind files, except that they have no direct relationship with the View that they are serving. A better analogy, if you're familiar with MVC, would be that they are similar to the Controllers in the Model-View-Controller (MVC) software architectural pattern. In fact, in his blog, John describes the MVVM pattern as being a variation of the MVC pattern. They have two-way connections with the Model component in order to access and update the data that the Views require, and often, they transform that data in some way to make it easier to display and interact with in the UI. They also have two-way connections with the Views through data binding and property change notification. In short, View Models form the bridge between the Model and the View, which otherwise have no connection to each other. However, it should be noted that the View Models are only loosely connected to the Views and Model components through their data binding and interfaces. The beauty of this pattern enables each element to be able to function independently from each other. To maintain the separation between the View Models and the View, we avoid declaring any properties of UI-related types in the View Model. We don't want any references to UI-related DLLs in our View Models project, and so we make use of custom IValueConverter implementations to convert them to primitive types. For example, we might convert Visibility objects from the UI to plain bool values or convert the selection of some colored Brush objects to an Enum instance that is safe to use in the View Model. Views The Views define the appearance and layout of the UI. They typically connect with a View Model through the use of their DataContext property and display the data that it supplies. They expose the functionality provided by the View Model by connecting its commands to the UI controls that the users interact with. In general, the basic rule of thumb is that each View has one associated View Model. This does not mean that a View cannot data bind to more than one data source or that we cannot reuse View Models. It simply means that, in general, if we have a class called SecurityView, it is more than likely that we'll also have an instance of a class named SecurityViewModel that will be set as the value of that View's DataContext property. Data binding One often overlooked aspect of the MVVM pattern is its requirement for data binding. We could not have the full Separation of Concerns without it, as there would be no easy way of communicating between the Views and View Models. The XAML markup, data binding classes, and ICommand and INotifyPropertyChanged interfaces are the main tools in WPF that provide this functionality. The ICommand interface is how commanding is implemented in the .NET Framework. It provides behavior that implements and even extends the ever useful Command pattern, in which an object encapsulates everything needed to perform an action. Most of the UI controls in WPF have Command properties that we can use to connect them to the functionality that the commands provide. The INotifyPropertyChanged interface is used to notify binding clients that property values have been changed. For example, if we had a User object and it had a Name property, then our User class would be responsible for raising the PropertyChanged event of the INotifyPropertyChanged interface, specifying the name of the property each time its value was changed. We'll look much deeper into all of this later, but now let's see how the arrangement of these components help us. So how does MVVM help? One major benefit of adopting MVVM is that it provides the crucial Separation of Concerns between the business model, the UI, and the business logic. This enables us to do several things. It frees the View Models from the Models, both the business model and the data persistence technology. This in turn enables us to reuse the business model in other applications and swap out the DAL and replace it with a mock data layer so that we can effectively test the functionality in our view models without requiring any kind of real data connection. It also disconnects the Views from the View logic that they require, as this is provided by the View Models. This allows us to run each component independently, which has the advantage of enabling one team to work on designing the Views, while another team works on the View Models. Having parallel work streams enables companies to benefit from vastly reduced production times. Furthermore, this separation also makes it easier for us to swap the Views for a different technology without needing to change our Model code. We may well need to change some aspects of the View Models, for example, the new technology used for the Views may not support the ICommand interface, but in principal, the amount of code that we would need to change would be fairly minimal. The simplicity of the MVVM pattern also makes WPF easier to comprehend. Knowing that each View has a View Model that provides it with all the data and functionality that it requires means that we always know where to look when we want to find where our data bound properties have been declared. Is there a downside? There are, however, a few drawbacks to using MVVM, and it will not help us in every situation. The main downside to implementing MVVM is that it adds a certain level of complexity to our applications. First, there's the data binding, which can take some time to master. Also, depending on your version of Visual Studio, data binding errors may only appear at runtime and can be very tricky to track down. Then, there are the different ways to communicate between the Views and View Models that we need to understand. Commanding and handling events in an unusual way takes a while to get used to. Having to discover the optimal arrangement of all the required components in the code base also takes time. So, there is a steep learning curve to climb before we can become competent at implementing MVVM for sure. However, even when we are well practiced at the pattern, there are still occasional situations when it wouldn't make sense to implement MVVM. One example would be if our application was going to be very small, it would be unlikely that we would want to have unit tests for it or swap out any of its components. It would, therefore, be impractical to go through the added complexity of implementing the pattern when the benefits of the Separation of Concerns that it provides were not required. Debunking the myth about code behind One of the great misconceptions about MVVM is that we should avoid putting any code into the code behind files of our Views. While there is some truth to this, it is certainly not true in all situations. If we think logically for a moment, we already know that the main reason to use MVVM is to take advantage of the Separation of Concerns that its architecture provides. Part of this separates the business functionality in the View Model from the user interface-related code in the Views. Therefore, the rule should really be we should avoid putting any business logic into the code behind files of our Views. Keeping this in mind, let's look at what code we might want to put into the code behind file of a View. The most likely suspects would be some UI-related code, maybe handling a particular event, or launching a child window of some kind. In these cases, using the code behind file would be absolutely fine. We have no business-related code here, and so we have no need to separate it from the other UI-related code. On the other hand, if we had written some business-related code in a View's code behind file, then how could we test it? In this case, we would have no way to separate this from the View, no longer have our Separation of Concerns and, therefore, would have broken our implementation of MVVM. So in cases like this, the myth is no longer a myth… it is good advice. However, even in cases like this where we want to call some business-related code from a View, it is possible to achieve without breaking any rules. As long as our business code resides in a View Model, it can be tested through that View Model, so it's not so important where it is called from during runtime. Understanding that we can always access the View Model that is data bound to a View's DataContext property, let's look at this simple example: private void Button_Click(object sender, RoutedEventArgs e) { UserViewModel viewModel = (UserViewModel)DataContext; viewModel.PerformSomeAction(); } Now, there are some who would balk at this code example, as they correctly believe that Views should not know anything about their related View Models. This code effectively ties this View Model to this View. If we wanted to change the UI layer in our application at some point or have designers work on the View, then this code would cause us a problem. However, we need to be realistic… what is the likelihood that we will really need do that? If it is likely, then we really shouldn't put this code into the code behind file and instead handle the event by wrapping it in an Attached Property, however, if it is not at all likely, then there is really no problem with putting it there. Let's follow rules when they make sense for us to follow them rather than blindly sticking to them because somebody in a different scenario said they were a good idea. One other situation when we can ignore this 'No code behind' rule is when writing self-contained controls based on the UserControl class. In these cases, the code behind files are often used for defining Dependency Properties and/or handling UI events and for implementing general UI functionality. Remember though, if these controls are implementing some business-related functionality, we should write that into a View Model and call it from the control so that it can still be tested. There is definitely perfect sense in the general idea of avoiding writing business-related code in the code behind files of our Views and we should always try to do so. However, we now hopefully understand the reasoning behind this idea and can use our logic to determine whether it is ok to do it or not in each particular case that may arise.  Summary In this article, we have discovered what the MVVM architectural pattern is and the benefits of using it when developing WPF applications. We also learned about the various components of the MVVM architectural pattern and their functionalities. We're now in a better position to decide which applications to use it with and which not to. Resources for Article: Further resources on this subject: Features of Dynamics GP [article] Auditing and E-discovery [article] Installing Microsoft Forefront UAG [article]

In this article by Randall Smith, the author of the book Docker Orchestration, we will see how to use Docker Swarm to orchestrate a Docker cluster. Docker Swarm is the native orchestration tool for Docker. It was rolled into the core docker suite with version 1.12. At its simplest, using Docker Swarm is just like using Docker. All of the tools that have been covered still work. Swarm adds a couple of features that make deploying and updating services very nice. (For more resources related to this topic, see here.) Setting up a Swarm The first step into running a Swarm is to have a number of hosts ready with Docker installed. It does not matter if you use the install script from get.docker.com or if you use Docker Machine. You also need to be sure that a few ports are open between the servers, as given here: 2377 tcp: cluster management 7946 tcp and udp: node communication 4789 tcp and udp: overlay network Take a moment to get or assign a static IP address to the hosts that will be the swarm managers. Each manager must have a static IP address so that workers know how to connect to them. Worker addresses can be dynamic but the IP of the manager must be static. As you plan your swarm, take a moment to decide how many managers you are going to need. The minimum number to run and still maintain fault tolerance is three. For larger clusters, you many need as many as five or seven. Very rarely will you need more than that. In any case, the number of managers should be odd. Docker Swarm can maintain a quorum as long as 50% + 1 managers are running. Having two or four managers provides no additional fault tolerance than one or three. If possible, you should spread your managers out so that a single failure will not take down your swarm. For example, make sure that they are not all running on the same VM host or connected to the same switch. The whole point is to have multiple managers to keep the swarm running in the event of a manager failure. Do not undermine your efforts by allowing a single point of failure somewhere else taking down your swarm. Initializing a Swarm If you are installing from your desktop, it may be better to use Docker Machine to connect to the host as this will set up the necessary TLS keys. Run the docker swarm init command to initialize the swarm. The --advertize-addr flag is optional. By default, it will guess an address on the host. This may not be correct. To be sure, set it to the static IP address you have for the manager host:     $ docker swarm init --advertise-addr 172.31.26.152 Swarm initialized: current node (6d6e6kxlyjuo9vb9w1uug95zh) is now a manager. To add a worker to this swarm, run the following command: docker swarm join --token SWMTKN-1-0061dt2qjsdr4gabxryrksqs0b8fnhwg6bjhs8cxzen7tmarbi-89mmok3p f6dsa5n33fb60tx0m 172.31.26.152:2377 To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions. Included in the output of the init command is the command to run on each worker host. It is also possible to run it from your desktop if you have set your environment to use the worker host. You can save the command somewhere, but you can always get it again by running docker swarm join-token manager, where manager is the name of a swarm manager. As part of the join process, Docker Swarm will create TLS keys, which will be used to encrypt communication between the managers and the hosts. It will not, however, encrypt network traffic between containers. The certificates are updated every three months. The time period can be updated by running docker swarm update --cert-expiry duration. The duration is specified as number of hours and minutes. For example, setting the duration to 1000h will tell Docker Swarm to reset the certificate every 1,000 hours. Managing a Swarm One of the easiest pieces to overlook when it comes to orchestrating Docker is how to manage the cluster itself. In this section, you will see how to manage your swarm including adding and removing nodes, changing their availability, and backup and recovery. Adding a node New worker nodes can be added at any time. Install Docker, then run docker swarm join-token worker to get the docker swarm join command to join the host to the swarm. Once added, the worker will be available to run tasks. Take note that the command to join the cluster is consistent. This makes it easy to add to a host configuration script and join the swarm on boot. You can get a list of all of the nodes in the swarm by running docker node ls: $ docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 1i3wtacdjz5p509bu3l3qfbei worker1 Ready Active Reachable 3f2t4wwwthgcahs9089b8przv worker2 Ready Active Reachable 6d6e6kxlyjuo9vb9w1uug95zh * manager Ready Active Leader The list will not only show the machines but it will also show if they are active in the swarm and if they are a manager. The node marked as Leader is the master of the swarm. It coordinates the managers. Promoting and demoting nodes In Docker Swarm, just like in real life, managers are also workers. This means that managers can run tasks. It also means that workers can be promoted to become managers. This can be useful to quickly replace a failed manager or to seamlessly increase the number of managers. The following command promotes the node named worker1 to be a manager: $ docker node promote worker1 Node worker1 promoted to a manager in the swarm. Managers can also be demoted to become plain workers. This should be done before a manager node is going to be decommissioned. The following command demotes worker1 back to a plain worker: $ docker node demote worker1 Manager worker1 demoted in the swarm. Whatever your reasons for promoting or demoting node be, make sure that when you are done, there are an odd number of managers. Changing node availability Docker Swarm nodes have a concept of availability. The availability of a node determines whether or not tasks can be scheduled on that node. Use the docker node update --availability <state><node-id>command to set the availability state. There are three availability states that can be set—pause, drain, and active. Pausing a node Setting a node's availability to pause will prevent the scheduler from assigning new tasks to the node. Existing tasks will continue to run. This can be useful for troubleshooting load issues on a node or for preventing new tasks from being assigned to an already overloaded node. The following command pauses worker2: $ docker node update --availability pause worker2 worker2 $ docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 1i3wtacdjz5p509bu3l3qfbei worker1 Ready Active Reachable 3f2t4wwwthgcahs9089b8przv worker2 Ready Pause Reachable 6d6e6kxlyjuo9vb9w1uug95zh * manager Ready Active Leader Do not rely on using pause to deal with overload issues. It is better to place reasonable resource limits on your services and let the scheduler figure things out for you. You can use pause to help determine what the resource limits should be. For example, you can start a task on a node, then pause the node to prevent new tasks from running while you monitor resource usage. Draining a node Like pause, setting a node's availability to drain will stop the scheduler from assigning new tasks to the node. In addition, drain will stop any running tasks and reschedule them to run elsewhere in the swarm. The drain mode has two common purposes. First, it is useful for preparing a node for an upgrade. Containers will be stopped and rescheduled in an orderly fashion. Updates can then be applied and the node rebooted, if necessary, without further disruption. The node can be set to active again once the updates are complete. Remember that, when draining a node, running containers have to be stopped and restarted elsewhere. This can cause disruption if your applications are not built to handle failure. The great thing about containers is that they start quickly, but some services, such as MySQL, take a few seconds to initialize. The second use of drain is to prevent services from being scheduled on manager nodes. Manager processing is very reliant on messages being passed in a timely manner. An unconstrained task running on a manager node can cause a denial of service outage for the node causing problems for your cluster. It is not uncommon to leave managers nodes in a drain state permanently. The following command will drain the node named manager: $ docker node update --availability drain manager manager $ docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 1i3wtacdjz5p509bu3l3qfbei worker1 Ready Active Reachable 3f2t4wwwthgcahs9089b8przv worker2 Ready Active Reachable 6d6e6kxlyjuo9vb9w1uug95zh * manager Ready Drain Leader Activating a node When a node is ready to accept tasks again, set the state to active. Do not be concerned if the node does not immediately fill up with containers. Tasks are only assigned when a new scheduling event happens, such as starting a service. The following command will reactivate the worker2 node: $ docker node update --availability active worker2 worker2 $ docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 1i3wtacdjz5p509bu3l3qfbei worker1 Ready Active Reachable 3f2t4wwwthgcahs9089b8przv worker2 Ready Active Reachable 6d6e6kxlyjuo9vb9w1uug95zh * manager Ready Active Leader Removing nodes Nodes may need to be removed for a variety of reasons including upgrades, failures, or simply eliminating capacity that is no longer needed. For example, it may be easier to upgrade nodes by building new ones rather than performing updates on the old nodes. The new node will be added and the old one removed. Step one for a healthy node is to set the availability to drain. This will ensure that all scheduled tasks have been stopped and moved to other nodes in the swarm. Step two is to run docker swarm leave from the node that will be leaving the swarm. This will assign the node a Down status. In the following example, worker2 has left the swarm: $ docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 1i3wtacdjz5p509bu3l3qfbei worker1 Ready Active Reachable 3f2t4wwwthgcahs9089b8przv worker2 Down Active 6d6e6kxlyjuo9vb9w1uug95zh * manager Ready Active Leader If the node that is being removed is a manager, it must first be demoted to a worker as described earlier before running docker swarm leave. When removing managers, take care that you do not lose quorum or your cluster will stop working. Once the node has been marked Down, it can be removed from the swarm. From a manager node, use docker node rm to remove the node from the swarm: $ docker node rm worker2 In some cases, the node that you want to remove was unreachable so that it was not possible to run docker swarm leave. When that happens, use the --force option. $ docker node rm --force worker2 Nodes that have been removed can be re-added with docker swarm join. Backup and recovery Just because Docker Swarm is resilient to failure does not mean that you should ignore backups. Good backups may be the difference between restoring services and a resume altering event. There are two major components to backup—the Swarm state data and your application data. Backing up the Swarm Each manager keeps the cluster information in /var/lib/docker/swarm/raft. If, for some reason, you need to completely rebuild your cluster, you will need this data. Make sure you have at least one good backup of the raft data. It does not matter which of the managers the backups are pulled from. It might be wise to pull backups from a couple of managers, just in case one is corrupted. Recovering a Swarm In most cases, losing a failed manager is an easy fix. Restart the failed manager and everything should be good. It may be necessary to build a new manager or promote an existing worker to a manager. In most circumstances, this will bring your cluster back into a healthy state. If you lose enough managers to lose a quorum, recovery gets more complex. The first step is to start enough managers to restore quorum. The data should be synced to the new managers, and once quorum is recovered, so is the cluster. If that does not work, you will have to rebuild the swarm. This is where your backups come in. If the manager node you choose to rebuild on has an otherwise healthy raft database, you can start there. If not, or if you are rebuilding on a brand new node, stop Docker and copy the raft data back to /var/lib/docker/swarm/raft. After the raft data is in place, ensure that Docker is running and run the following command: $ docker swarm init --force-new-cluster --advertise-addr manager The address set in with --advertise-addr has the same meaning as what was used to create the swarm initially. The magic here is the --force-new-cluster option. This option will ignore the swarm membership data that is the raft database, but will remember things such as the worker node list, running services, and tasks. Backing up services Service information is backed up as part of the raft database, but you should have a plan to rebuild them in case the database becomes corrupted. Backing up the output of docker swarm ls is a start. Application information, including networks, and volumes may be sourced for Docker Compose files, which should be backed up. The containers themselves and your application should be in version control and backed up. Most importantly, do not forget your data. If you have your Docker files and the application code, the applications can be rebuilt even if the registry is lost. In some cases, it is a valid choice to not backup the registry since the images can, potentially, be rebuilt. The data, however, usually cannot be. Have a strategy in place for backing up the data that works for your environment. I suggest creating a container for each application that is deployed that can properly backup data for the application. Docker Swarm does not have a native scheduled task option, but you can configure cron on a worker node that runs the various backup containers on a schedule. The pause availability option can be helpful here. Configure a worker node that will be your designated host to pull backups. Set the availability to pause so that other containers are not started on the node and resources are available to perform the backups. Using pause means that containers can be started in the background and will continue to run after the node is paused, allowing them to finish normally. Then cron can run a script that looks something like the following one. The contents of run-backup-containers is left as an exercise for the reader: #!/bin/bash docker node update --availability active backupworker run-backup-containers docker node update --availability pause You can also label to designate multiple nodes for backups and schedule services to ignore those nodes, or in the case of backup containers, run on them. Managing services Now that the swarm is up and running, it is time to look at services. A service is a collection of one or more tasks that do something. Each task is a container running somewhere in the swarm. Since services are potentially composed of multiple tasks, there are different tools to manage them. In most cases, these commands will be a subcommand of docker service. Running services Running tasks with Docker Swarm is a little bit different than running them under plain Docker. Instead of using docker run, the command is docker service create: $ docker service create --name web nginx When a service starts, a swarm manager schedules the tasks to run on active workers in the swarm. By default, Swarm will spread running containers across all of the active hosts in the swarm. Offering services to the Internet requires publishing ports with the -p flag. Multiple ports can be opened by specifying -p multiple times. When using the swarm overlay network, you also get ingress mesh routing. The mesh will route connections from any host in the cluster to the service no matter where it is running. Port publishing will also load balance across multiple containers: $ docker service create --name web -p 80 -p 443 nginx Creating replicas A service can be started with multiple containers using the --replicas option. The value is the number of desired replicas. It may take a moment to start all the desired replicas: $ docker service create --replicas 2 --name web -p 80 -p 443 nginx This example starts two copies of the nginx container under the service name web. The great news is that you can change the number of replicas at any time: $ docker service update --replicas 3 web Even better, service can be scaled up or down. This example scales the service up to three containers. It is possible to scale the number down later once the replicas are no longer needed. Use docker service ls to see a summary of running services and the number of replicas for each: $ docker service ls ID NAME REPLICAS IMAGE COMMAND 4i3jsbsohkxj web 3/3 nginx If you need to see the details of a service, including where tasks are running, use the docker service ps command. It takes the name of the service as an argument. This example shows three nginx tasks that are part of the service web: $ docker service ps web ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR d993z8o6ex6wz00xtbrv647uq web.1 nginx worker2 Running Running 31 minutes ago eueui4hw33eonsin9hfqgcvd7 web.2 nginx worker1 Running Preparing 4 seconds ago djg5542upa1vq4z0ycz8blgfo web.3 nginx worker2 Running Running 2 seconds ago   Take note of the name of the tasks. If you were to connect to worker1 and try to use docker exec to access the web.2 container, you will get an error: $ docker exec -it web.2 bash Error response from daemon: No such container: web.2 Tasks started with the docker service are named with the name of the service, a number, and the ID of the task separated by dots. Using docker ps on worker1, you can see the actual name of the web.2 container: $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b71ad831eb09 nginx:latest "nginx -g 'daemon off" 2 days ago Up 2 days 80/tcp, 443/tcp web.2.eueui4hw33eonsin9hfqgcvd7 Running global services There may be times when you want to run a task on every active node in the swarm. This is useful for monitoring tools: $ docker service create --mode global --name monitor nginx $ docker service ps monitor ID NAME IMAGE NODE DESIRED STATE CURRENT S TATE ERROR daxkqywp0y8bhip0f4ocpl5v1 monitor nginx worker2 Running Running 6 seconds ago a45opnrj3dcvz4skgwd8vamx8 _ monitor nginx worker1 Running Running 4 seconds ago It is important to reiterate that global services only run on active nodes. The task will not start on nodes that have the availability set to pause or drain. If a paused or drained node is set to active, the global service will be started on that node immediately. $ docker node update --availability active manager manager $ docker service ps monitor ID NAME IMAGE NODE DESIRED STATE CURRENT S TATE ERROR 0mpe2zb0mn3z6fa2ioybjhqr3 monitor nginx manager Running Preparing 3 seconds ago daxkqywp0y8bhip0f4ocpl5v1 _ monitor nginx worker2 Running Running 3 minutes ago a45opnrj3dcvz4skgwd8vamx8 _ monitor nginx worker1 Running Running 3 minutes ago Setting constraints It is often useful to limit which nodes a service can run on. For example, a service that might be dependent on fast disk might be limited to nodes that have SSDs. Constraints are added to the docker service create command with the --constraint flag. Multiple constraints can be added. The result will be the intersection of all of the constraints. For this example, assume that there exists a swarm with three nodes—manager, worker1, and worker2. The worker1 and worker2nodes have the env=prod label while manager has the env=dev label. If a service is started with the constraint that env is dev, it will only run service tasks on the manager node. $ docker service create --constraint "node.labels.env == dev" --name web-dev --replicas 2 nginx 913jm3v2ytrpxejpvtdkzrfjz $ docker service ps web-dev ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR 5e93fl10k9x6kq013ffotd1wf web-dev.1 nginx manager Running Running 6 seconds ago 5skcigjackl6b8snpgtcjbu12 web-dev.2 nginx manager Running Running 5 seconds ago Even though there are two other nodes in the swarm, the service is only running on the manager because it is the only node with the env=dev label. If another service was started with the constraint that env is prod, the tasks will start on the worker nodes: $ docker service create --constraint "node.labels.env == prod" --name web-prod --replicas 2 nginx 88kfmfbwksklkhg92f4fkcpwx $ docker service ps web-prod ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR 5f4s2536g0bmm99j7wc02s963 web-prod.1 nginx worker2 Running Running 3 seconds ago 5ogcsmv2bquwpbu1ndn4i9q65 web-prod.2 nginx worker1 Running Running 3 seconds ago The constraints will be honored if the services are scaled. No matter how many replicas are requested, the containers will only be run on nodes that match the constraints: $ docker service update --replicas 3 web-prod web-prod $ docker service ps web-prod ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR 5f4s2536g0bmm99j7wc02s963 web-prod.1 nginx worker2 Running Running about a minute ago 5ogcsmv2bquwpbu1ndn4i9q65 web-prod.2 nginx worker1 Running Running about a minute ago en15vh1d7819hag4xp1qkerae web-prod.3 nginx worker1 Running Running 2 seconds ago As you can see from the example, the containers are all running on worker1 and worker2. This leads to an important point. If the constraints cannot be satisfied, the service will be started but no containers will actually be running: $ docker service create --constraint "node.labels.env == testing" --name web-test --replicas 2 nginx 6tfeocf8g4rwk8p5erno8nyia $ docker service ls ID NAME REPLICAS IMAGE COMMAND 6tfeocf8g4rw web-test 0/2 nginx 88kfmfbwkskl web-prod 3/3 nginx 913jm3v2ytrp web-dev 2/2 nginx Notice that the number of replicas requested is two but the number of containers running is zero. Swarm cannot find a suitable node so it does not start the containers. If a node with the env=testing label were to be added or if that label were to be added to an existing node, swarm would immediately schedule the tasks: $ docker node update --label-add env=testing worker1 worker1 $ docker service ps web-test ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR 7hsjs0q0pqlb6x68qos19o1b0 web-test.1 nginx worker1 Running Running 18 seconds ago dqajwyqrah6zv83dsfqene3qa web-test.2 nginx worker1 Running Running 18 seconds ago In this example, the env label was changed to testing from prod on worker1. Since a node is now available that meets the constraints for the web-test service, swarm started the containers on worker1. However, the constraints are only checked when tasks are scheduled. Even though worker1 no longer has the env label set to prod, the existing containers for the web-prod service are still running: $ docker node ps worker1 ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR 7hsjs0q0pqlb6x68qos19o1b0 web-test.1 nginx worker1 Running Running 5 minutes ago dqajwyqrah6zv83dsfqene3qa web-test.2 nginx worker1 Running Running 5 minutes ago 5ogcsmv2bquwpbu1ndn4i9q65 web-prod.2 nginx worker1 Running Running 21 minutes ago en15vh1d7819hag4xp1qkerae web-prod.3 nginx worker1 Running Running 20 minutes ago Stopping services All good things come to an end and this includes services running in a swarm. When a service is no longer needed, it can be removed with docker service rm. When a service is removed, all tasks associated with that service are stopped and the containers removed from the nodes they were running on. The following example removes a service named web: $ docker service rm web Docker makes the assumption that the only time services are stopped is when they are no longer needed. Because of this, there is no docker service analog to the docker stop command. This might not be an issue since services are so easily recreated. That said, I have run into situations where I have needed to stop a service for a short time for testing and did not have the command at my fingertips to recreate it. The solution is very easy but not necessarily obvious. Rather than stopping the service and recreating it, set the number of replicas to zero. This will stop all running tasks and they will be ready to start up again when needed. $ docker service update --replicas 0 web web $ docker service ls ID NAME REPLICAS IMAGE COMMAND 5gdgmb7afupd web 0/0 nginx The containers for the tasks are stopped, but remain on the nodes. The docker service ps command will show that the tasks for the service are all in the Shutdown state. If needed, one can inspect the containers on the nodes: $ docker service ps web ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR 68v6trav6cf2qj8gp8wgcbmhu web.1 nginx worker1 Shutdown Shutdown 3 seconds ago 060wax426mtqdx79g0ulwu25r web.2 nginx manager Shutdown Shutdown 2 seconds ago 79uidx4t5rz4o7an9wtya1456 web.3 nginx worker2 Shutdown Shutdown 3 seconds ago When it is time to bring the service back, use docker swarm update to set the number of replicas back to what is needed. Swarm will start the containers across in the swarm just as if you had used docker swarm create. Upgrading a service with rolling updates It is likely that services running in a swarm will need to be upgraded. Traditionally, upgrades involved stopping a service, performing the upgrade, then restarting the service. If everything goes well, the service starts and works as expected and the downtime is minimized. If not, there can be an extended outage as the administrator and developers debug what went wrong and restore the service. Docker makes it easy to test new images before they are deployed and one can be confident that the service will work in production just as it did during testing. The question is, how does one deploy the upgraded service without a noticeable downtime for the users? For busy services, even a few seconds of downtime can be problematic. Docker Swarm provides a way to update services in the background with zero downtime. There are three options which are passed to docker service create that control how rolling updates are applied. These options can also be changed after a service has been created with docker service update. The options are as follows: --update-delay: This option sets the delay between each container upgrade. The delay is defined by a number of hours, minutes, and seconds indicated by a number followed by h, m, or s, respectively. For example, a 30 second delay will be written as 30s. A delay of 1 hour, 30 minutes, and 12 seconds will be written as 1h30m12s. --update-failure-action: This tells swarm how to handle an upgrade failure. By default, Docker Swarm will pause the upgrade if a container fails to upgrade. You can configure swarm to continue even if a task fails to upgrade. The allowed values are pause and continue. --update-parallelism: This tells swarm how many tasks to upgrade at one time. By default, Docker Swarm will only upgrade one task at a time. If this is set to 0 (zero), all running containers will be upgraded at once. For this example, a service named web is started with six replicas using the nginx:1.10 image. The service is configured to update two tasks at a time and wait 30 seconds between updates. The list of tasks from docker service ps shows that all six tasks are running and that they are all running the nginx:1.10 image: $ docker service create --name web --update-delay 30s --update-parallelism 2 --replicas 6 nginx:1.10 $ docker service ps web ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR 83p4vi4ryw9x6kbevmplgrjp4 web.1 nginx:1.10 worker2 Running Running 20 seconds ago 2yb1tchas244tmnrpfyzik5jw web.2 nginx:1.10 worker1 Running Running 20 seconds ago f4g3nayyx5y6k65x8n31x8klk web.3 nginx:1.10 worker2 Running Running 20 seconds ago 6axpogx5rqlg96bqt9qn822rx web.4 nginx:1.10 worker1 Running Running 20 seconds ago 2d7n5nhja0efka7qy2boke8l3 web.5 nginx:1.10 manager Running Running 16 seconds ago 5sprz723zv3z779o3zcyj28p1 web.6 nginx:1.10 manager Running Running 16 seconds ago Updates have started using the docker service update command and specifying a new image. In this case, the service will be upgraded from nginx:1.10 to nginx:1.11. Rolling updates work by stopping a number of tasks defined by --update-parallelism and starting new tasks based on the new image. Swarm will then wait until the delay set by --update-delay elapses before upgrading the next tasks: $ docker service update --image nginx:1.11 web When the updates begin, two tasks will be updated at a time. If the image is not found on the node, it will be pulled from the registry, slowing down the update. You can speed up the process by writing a script to pull the new image on each node before you run the update. The update process can be monitored by running docker service inspect or docker service ps: $ docker service inspect --pretty web ID: 4a60v04ux70qdf0fzyf3s93er Name: web Mode: Replicated Replicas: 6 Update status: State: updating Started: about a minute ago Message: update in progress Placement: UpdateConfig: Parallelism: 2 Delay: 30s On failure: pause ContainerSpec: Image: nginx:1.11 Resources: $ docker service ps web ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR 83p4vi4ryw9x6kbevmplgrjp4 web.1 nginx:1.10 worker2 Running Running 35 seconds ago 29qqk95xdrb0whcdy7abvji2p web.2 nginx:1.11 manager Running Preparing 2 seconds ago 2yb1tchas244tmnrpfyzik5jw _ web.2 nginx:1.10 worker1 Shutdown Shutdown 2 seconds ago f4g3nayyx5y6k65x8n31x8klk web.3 nginx:1.10 worker2 Running Running 35 seconds ago 6axpogx5rqlg96bqt9qn822rx web.4 nginx:1.10 worker1 Running Running 35 seconds ago 3z6ees2748tqsoacy114ol183 web.5 nginx:1.11 worker1 Running Running less than a second ago 2d7n5nhja0efka7qy2boke8l3 _ web.5 nginx:1.10 manager Shutdown Shutdown 1 seconds ago 5sprz723zv3z779o3zcyj28p1 web.6 nginx:1.10 manager Running Running 30 seconds ago As the upgrade starts, the web.2 and web.3tasks have been updated and are now running nginx:1.11. The others are still running the old version. Every 30 seconds, two more tasks will be upgraded until the entire service is running nginx:1.11. Docker Swarm does not care about the version that is used on the image tag. This means that you can just as easily downgrade the service. In this case, if the docker service update --image nginx:1.10 web command were to be run after the upgrade, Swarm will go through the same update process until all tasks are running nginx:1.10. This can be very helpful if an upgrade does work as it was supposed to. It also important to note that there is nothing that says that the new image has to be the same base as the old one. You can decide to run Apache instead of Nginx by running docker service update --image httpd web. Swarm will happily replace all the web tasks which were running an nginx image with one running the httpd image. Rolling updates require that your service is able to run multiple containers in parallel. This may not work for some services, such as SQL databases. Some updates may require a schema change that is incompatible with the older version. In either case, rolling updates may not work for the service. In those cases, you can set --update-parallelism 0 to force all tasks to update at once or manually recreate the service. If you are running a lot of replicas, you should pre-pull your image to ease the load on your registry. Summary In this article, you have seen how to use Docker Swarm to orchestrate a Docker cluster. The same tools that are used with a single node can be used with a swarm. Additional tools available through swarm allow for easy scale out of services and rolling updates with little to no downtime. Resources for Article: Further resources on this subject: Introduction to Docker [article] Docker in Production [article] Docker Hosts [article]

In this article by Michael Solberg and Ben Silverman, the author of the book Openstack for Architects, we will be walking through how to architect your cloud to avoid hardware and software failures. The OpenStack control plane is comprised of web services, application services, database services, and a message bus. Each of these tiers require different approaches to make them highly available and some organizations will already have defined architectures for each of the services. We've seen that customers either reuse those existing patterns or adopt new ones which are specific to the OpenStack platform. Both of these approaches make sense, depending on the scale of the deployment. Many successful deployments actually implement a blend of these. For example, if your organization already has a supported pattern for highly available MySQL databases, you might chose that pattern instead of the one outlined in this article. If your organization doesn't have a pattern for highly available MongoDB, you might have to architect a new one. (For more resources related to this topic, see here.) Building a highly available control plane Back in the Folsom and Grizzly days, coming up with an high availability (H/A) design for the OpenStack control plane was something of a black art. Many of the technologies recommended in the first iterations of the OpenStack High Availability Guide were specific to the Ubuntu distribution of Linux and were unavailable on the Red Hat Enterprise Linux-derived distributions. The now-standard cluster resource manager (Pacemaker) was unsupported by Red Hat at that time. As such, architects using Ubuntu might use one set of software, those using CentOS or RHEL might use another set of software, and those using a Rackspace or Mirantis distribution might use yet another set of software. However, these days, the technology stack has converged and the H/A pattern is largely consistent regardless of the distribution used. About failure and success When we design a highly available OpenStack control plane, we're looking to mitigate two different scenarios: The first is failure. When a physical piece of hardware dies, we want to make sure that we recover without human interaction and continue to provide service to our users The second and perhaps more important scenario is success Software systems always work as designed and tested until humans start using them. While our automated test suites will try to launch a reasonable number of virtual objects, humans are guaranteed to attempt to launch an unreasonable number. Also, many of the OpenStack projects we've worked on have grown far past their expected size and need to be expanded on the fly. There are a few different types of success scenarios that we need to plan for when architecting an OpenStack cloud. First, we need to plan for a growth in the number of instances. This is relatively straightforward. Each additional instance grows the size of the database, it grows the amount of metering data in Ceilometer, and, most importantly, it will grow the number of compute nodes. Adding compute nodes and reporting puts strain on the message bus, which is typically the limiting factor in the size of OpenStack regions or cells. We'll talk more about this when we talk about dividing up OpenStack clouds into regions, cells, and Availability Zones. The second type of growth we need to plan for is an increase in the number of API calls. Deployments which support Continuous Integration(CI) development environments might have (relatively) small compute requirements, but CI typically brings up and tears down environments rapidly. This will generate a large amount of API traffic, which in turn generates a large amount of database and message traffic. In hosting environments, end users might also manually generate a lot of API traffic as they bring up and down instances, or manually check the status of deployments they've already launched. While a service catalog might check the status of instances it has launched on a regular basis, humans tend to hit refresh on their browsers in an erratic fashion. Automated testing of the platform has a tendency to grossly underestimate this kind of behavior. With that in mind, any pattern that we adopt will need to provide for the following requirements: API services must continue to be available during a hardware failure in the control plane The systems which provide API services must be horizontally scalable (and ideally elastic) to respond to unanticipated demands The database services must be vertically or horizontally scalable to respond to unanticipated growth of the platform The message bus can either be vertically or horizontally scaled depending on the technology chosen Finally, every system has its limits. These limits should be defined in the architecture documentation so that capacity planning can account for them. At some point, the control plane has scaled as far as it can and a second control plane should be deployed to provide additional capacity. Although OpenStack is designed to be massively scalable, it isn't designed to be infinitely scalable. High availability patterns for the control plane There are three approaches commonly used in OpenStack deployments these days for achieving high availability of the control plane. The first is the simplest. Take the single-node cloud controller virtualize it, and then make the virtual machine highly available using either VMware clustering or Linux clustering. While this option is simple and it provides for failure scenarios, it scales vertically (not horizontally) and doesn't provide for success scenarios. As such, it should only be used in regions with a limited number of compute nodes and a limited number of API calls. In practice, this method isn't used frequently and we won't spend any more time on it here. The second pattern provides for H/A, but not horizontal scalability. This is the "Active/Passive" scenario described in the OpenStack High Availability Guide. At Red Hat, we used this a lot with our Folsom and Grizzly deployments, but moved away from it starting with Havana. It's similar to the virtualization solution described earlier but instead of relying on VMware clustering or Linux clustering to restart a failed virtual machine, it relies on Linux clustering to restart failed services on a second cloud controller node, also running the same subset of services. This pattern doesn't provide for success scenarios in the Web tier, but can still be used in the database and messaging tiers. Some networking services may still need to be provided as Active/Passive as well. The third H/A pattern available to OpenStack architectures is the Active/Active pattern. In this pattern, services are horizontally scaled out behind a load balancing service or appliance, which is Active/Passive. As a general rule, most OpenStack services should be enabled as Active/Active where possible to allow for success scenarios while mitigating failure scenarios. Ideally, Active/Active services can be scaled out elastically without service disruption by simply adding additional control plane nodes. Both of the Active/Passive and Active/Active designs require clustering software to determine the health of services and the hosts on which they run. In this article, we'll be using Pacemaker as the cluster manager. Some architects may choose to use Keepalived instead of Pacemaker. Active/Passive service configuration In the Active/Passive service configuration, the service is configured and deployed to two or more physical systems. The service is associated with a Virtual IP(VIP)address. A cluster resource manager (normally Pacemaker) is used to ensure that the service and its VIP are enabled on only one of the two systems at any point in time. The resource manager may be configured to favor one of the machines over the other. When the machine that the service is running on fails, the resource manager first ensures that the failed machine is no longer running and then it starts the service on the second machine. Ensuring that the failed machine is no longer running is accomplished through a process known as fencing. Fencing usually entails powering off the machine using the management interface on the BIOS. The fence agent may also talk to a power supply connected to the failed server to ensure that the system is down. Some services (such as the Glance image registry) require shared storage to operate. If the storage is network-based, such as NFS, the storage may be mounted on both the active and the passive nodes simultaneously. If the storage is block-based, such as iSCSI, the storage will only be mounted on the active node and the resource manager will ensure that the storage migrates with the service and the VIP. Active/Active service configuration Most of the OpenStack API services are designed to be run on more than one system simultaneously. This configuration, the Active/Active configuration, requires a load balancer to spread traffic across each of the active services. The load balancer manages the VIP for the service and ensures that the backend systems are listening before forwarding traffic to them. The cluster manager ensures that the VIP is only active on one node at a time. The backend services may or may not be managed by the cluster manager in the Active/Active configuration. Service or system failure is detected by the load balancer and failed services are brought out of rotation. There are a few different advantages to the Active/Active service configuration, which are as follows: The first advantage is that it allows for horizontal scalability. If additional capacity is needed for a given service, a new system can be brought up which is running the service and it can be added into rotation behind the load balancer without any downtime. The control plane may also be scaled down without downtime in the event that it was over-provisioned. The second advantage is that Active/Active services have a much shorter mean time to recovery. Fencing operations often take up to 2 minutes and fencing is required before the cluster resource manager will move a service from a failed system to a healthy one. Load balancers can immediately detect system failure and stop sending requests to unresponsive nodes while the cluster manager fences them in the background. Whenever possible, architects should employ the Active/Active pattern for the control plane services. OpenStack service specifics In this section, we'll walk through each of the OpenStack services and outline the H/A strategy for them. While most of the services can be configured as Active/Active behind a load balancer, some of them must be configured as Active/Passive and others may be configured as Active/Passive. Some of the configuration is dependent on a particular version of OpenStack as well, especially, Ceilometer, Heat, and Neutron. The following details are current as of the Liberty release of OpenStack. The OpenStack web services As a general rule, all of the web services and the Horizon dashboard may be run Active/Active. These include the API services for Keystone, Glance, Nova, Cinder, Neutron, Heat, and Ceilometer. The scheduling services for Nova, Cinder, Neutron, Heat, and Ceilometer may also be deployed Active/Active. These services do not require a load balancer, as they respond to requests on the message bus. The only web service which must be run Active/Passive is the Ceilometer Central agent. This service can be configured to split its workload among multiple instances, however, to support scaling horizontally. The database services All state for the OpenStack web services is stored in a central database—usually a MySQL database. MySQL is usually deployed in an Active/Passive configuration, but can be made Active/Active with the Galera replication extension. Galera is clustering software for MySQL (MariaDB in OpenStack) and this uses synchronous replication to achieve H/A. However, even with Galera, we still recommend directing writes to only one of the replicas—some queries used by the OpenStack services may deadlock when writing to more than one master. With Galera, a load balancer is typically deployed in front of the cluster and is configured to deliver traffic to only one replica at a time. This configuration reduces the mean time to recovery of the service while ensuring that the data is consistent. In practice, many organizations will defer to the database architects for their preference regarding highly available MySQL deployments. After all, it is typically the database administration team who is responsible for responding to failures of that component. Deployments which use the Ceilometer service also require a MongoDB database to store telemetry data. MongoDB is horizontally scalable by design and is typically deployed Active/Active with at least three replicas. The message bus All OpenStack services communicate through the message bus. Most OpenStack deployments these days use the RabbitMQ service as the message bus. RabbitMQ can be configured to be Active/Active through a facility known as "mirrored queues". The RabbitMQ service is not load balanced, each service is given a list of potential nodes and the client is responsible for determining which nodes are active and which ones have failed. Other messaging services used with OpenStack such as ZeroMQ, ActiveMQ, or Qpid may have different strategies and configurations for achieving H/A and horizontal scalability. For these services, refer to the documentation to determine the optimal architecture. Compute, storage, and network agents The compute, storage, and network components in OpenStack has a set of services which perform the work which is scheduled by the API services. These services register themselves with the schedulers on start up over the message bus. The schedulers are responsible for determining the health of the services and scheduling work to active services. The compute and storage services are all designed to be run Active/Active but the network services need some extra consideration. Each hypervisor in an OpenStack deployment runs the nova-compute service. When this service starts up, it registers itself with the nova-scheduler service. A list of currently available nova services is available via the nova service-list command. If a compute node is unavailable, its state is listed as down and the scheduler skips it when performing instance actions. When the node becomes available, the scheduler includes it in the list of available hosts. For KVM or Xen-based deployments, the nova-compute service runs once per hypervisor and is not made highly available. For VMware-based deployments though, a single nova-compute service is run for every vSphere cluster. As such, this service should be made highly available in an Active/Passive configuration. This is typically done by virtualizing the service within a vSphere cluster and configuring the virtual machine to be highly available. Cinder includes a service known as the volume service or cinder-volume. The volume service registers itself with the Cinder scheduler on startup and is responsible for creating, modifying, or deleting LUNs on block storage devices. For backends which support multiple writers, multiple copies of this service may be run in Active/Active configuration. The LVM backend (which is the reference backend) is not highly available, though, and may only have one cinder-volume service for each block device. This is because the LVM backend is responsible for providing iSCSI access to a locally attached storage device. For this reason, highly available deployments of OpenStack should avoid the LVM Cinder backend and instead use a backend that supports multiple cinder-volume services. Finally, the Neutron component of OpenStack has a number of agents, which all require some special consideration for highly available deployments. The DHCP agent can be configured as highly available, and the number of agents which will respond to DHCP requests for each subnet is governed by a parameter in the neutron.conf file, dhcp_agents_per_network. This is typically set to 2, regardless of the number of DHCP agents which are configured to run in a control plane. For most of the history of OpenStack, the L3 routing agent in Neutron has been a single point of failure. It could be made highly available in Active/Passive configuration, but its failover meant the interruption of network connections in the tenant space. Many of the third-party Neutron plugins have addressed this in different ways and the reference Open vSwitch plugin has a highly available L3 agent as of the Juno release. For details on implementing a solution to the single routing point of failure using OpenStack's Distributed Virtual Routers (DVR), refer to the OpenStack Foundation's Neutron documentation at http://docs.openstack.org/liberty/networking-guide/scenario-dvr-ovs.html. Regions, cells, and availability Zones As we mentioned before, OpenStack is designed to be scalable, but not infinitely scalable. There are three different techniques architects can use to segregate an OpenStack cloud—regions, cells, and Availability Zones. In this section, we'll walk through how each of these concepts maps to hypervisor topologies. Regions From an end user's perspective, OpenStack regions are equivalent to regions in Amazon Web Services. Regions live in separate data centers and are often named after their geographical location. If your organization has a data center in Phoenix and one in Raleigh (like ours does) you'll have at least a PHX and a RDU region. Users who want to geographically disperse their workloads will place some of them in PHX and some of them in RDU. Regions have separate API endpoints, and although the Horizon UI has some support for multiple regions, they essentially entirely separate deployments. From an architectural standpoint, there are two main design choices for implementing regions, which are as follows: The first is around authorization. Users will want to have the same credentials for accessing each of the OpenStack regions. There are a few ways to accomplish this. The simplest way is to use a common backing store (usually LDAP) for the Keystone service in each region. In this scenario, the user has to authenticate separately to each region to get a token, but the credentials are the same. In Juno and later, Keystone also supports federation across regions. In this scenario, a Keystone token granted by one region can be presented to another region to authenticate a user. While this currently isn't widely used, it is a major focus area for the OpenStack Foundation and will probably see broader adoption in the future. The second major consideration for regional architectures is whether or not to present a single set of Glance images to each region. While work is currently being done to replicate Glance images across federated clouds, most organizations are manually ensuring that the shared images are consistent. This typically involves building a workflow around image publishing and deprecation which is mindful of the regional layout. Another option for ensuring consistent images across regions is to implement a central image repository using Swift. This also requires shared Keystone and Glance services which span multiple data centers. Details on how to design multiple regions with shared services are in the OpenStack Architecture Design Guide. Cells The Nova compute service has a concept of cells, which can be used to segregate large pools of hypervisors within a single region. This technique is primarily used to mitigate the scalability limits of the OpenStack message bus. The deployment at CERN makes wide use of cells to achieve massive scalability within single regions. Support for cells varies from service to service and as such cells are infrequently used outside a few very large cloud deployments. The CERN deployment is well-documented and should be used as a reference for these types of deployments. In our experience, it's much simpler to deploy multiple regions within a single data center than to implement cells to achieve large scale. The added inconvenience of presenting your users with multiple API endpoints within a geographic location is typically outweighed by the benefits of having a more robust platform. If multiple control planes are available in a geographic region, the failure of a single control plane becomes less dramatic. The cells architecture has its own set of challenges with regard to networking and scheduling of instance placement. Some very large companies that support the OpenStack effort have been working for years to overcome these hurdles. However, many different OpenStack distributions are currently working on a new control plane design. These new designs would begin to split the OpenStack control plane into containers running the OpenStack services in a microservice type architecture. This way the services themselves can be placed anywhere and be scaled horizontally based on the load. One architecture that has garnered a lot of attention lately is the Kolla project that promotes Docker containers and Ansible playbooks to provide production-ready containers and deployment tools for operating OpenStack clouds. To see more, go to https://wiki.openstack.org/wiki/Kolla. Availability Zones Availability Zones are used to group hypervisors within a single OpenStack region. Availability Zones are exposed to the end user and should be used to provide the user with an indication of the underlying topology of the cloud. The most common use case for Availability Zones is to expose failure zones to the user. To ensure the H/A of a service deployed on OpenStack, a user will typically want to deploy the various components of their service onto hypervisors within different racks. This way, the failure of a top of rack switch or a PDU will only bring down a portion of the instances which provide the service. Racks form a natural boundary for Availability Zones for this reason. There are a few other interesting uses of Availability Zones apart from exposing failure zones to the end user. One financial services customer we work with had a requirement for the instances of each line of business to run on dedicated hardware. A combination of Availability Zones and the AggregateMultiTenancyIsolation Nova Scheduler filter were used to ensure that each tenant had access to dedicated compute nodes. Availability Zones can also be used to expose hardware classes to end users. For example, hosts with faster processors might be placed in one Availability Zone and hosts with slower processors might be placed in different Availability Zones. This allows end users to decide where to place their workloads based upon compute requirements. Updating the design document In this article, we walked through the different approaches and considerations for achieving H/A and scalability in OpenStack deployments. As Cloud Architects, we need to decide on the correct approach for our deployment and then document it thoroughly so that it can be evaluated by the larger team in our organization. Each of the major OpenStack vendors has a reference architecture for highly available deployments and those should be used as a starting point for the design. The design should then be integrated with existing Enterprise Architecture and modified to ensure that best practices established by the various stakeholders within an organization are followed. The system administrators within an organization may be more comfortable supporting Pacemaker than Keepalived. The design document presents the choices made for each of these key technologies and gives the stakeholders an opportunity to comment on them before the deployment. Planning the physical architecture The simplest way to achieve H/A is to add additional cloud controllers to the deployment and cluster them. Other deployments may choose to segregate services into different host classes, which can then be clustered. This may include separating the database services into database nodes, separating the messaging services into messaging nodes, and separating the memcached service into memcache nodes. Load balancing services might live on their own nodes as well. The primary considerations for mapping scalable services to physical (or virtual) hosts are the following: Does the service scale horizontally or vertically? Will vertically scaling the service impede the performance of other co-located services? Does the service have particular hardware or network requirements that other services don't have? For example, some OpenStack deployments which use the HAProxy load balancing service chose to separate out the load balancing nodes on a separate hardware. The VIPs which the load balancing nodes host must live on a public, routed network, while the internal IPs of services that they route to don't have that requirement. Putting the HAProxy service on separate hosts allows the rest of the control plane to only have private addressing. Grouping all of the API services on dedicated hosts may ease horizontal scalability. These services don't need to be managed by a cluster resource manager and can be scaled by adding additional nodes to the load balancers without having to update cluster definitions. Database services have high I/O requirements. Segregating these services onto machines which have access to high performance fiber channel may make sense. Finally, you should consider whether or not to virtualize the control plane. If the control plane will be virtualized, creating additional host groups to host dedicated services becomes very attractive. Having eight or nine virtual machines dedicated to the control plane is a very different proposition than having eight or nine physical machines dedicated to the control plane. Most highly available control planes require at least three nodes to ensure that quorum is easily determined by the cluster resource manager. While dedicating three physical nodes to the control function of a hundred node OpenStack deployment makes a lot of sense, dedicating nine physical nodes may not. Many of the organizations that we've worked with will already have a VMware-based cluster available for hosting management appliances and the control plane can be deployed within that existing footprint. Organizations which are deploying a KVM-only cloud may not want to incur the additional operational complexity of managing the additional virtual machines outside OpenStack. Updating the physical architecture design Once the mapping of services to physical (or virtual) machines has been determined, the design document should be updated to include definition of the host groups and their associated functions. A simple example is provided as follows: Load balancer: These systems provide the load balancing services in an Active/Passive configuration Cloud controller: These systems provide the API services, the scheduling services, and the Horizon dashboard services in an Active/Active configuration Database node: These systems provide the MySQL database services in an Active/Passive configuration Messaging node: These systems provide the RabbitMQ messaging services in an Active/Active configuration Compute node: These systems act as KVM hypervisors and run the nova-compute and openvswitch-agent services Deployments which will be using only the cloud controller host group might use the following definitions: Cloud controller: These systems provide the load balancing services in an Active/Passive configuration and the API services, MySQL database services, and RabbitMQ messaging services in an Active/Active configuration Compute node: These systems act as KVM hypervisors and run the nova-compute and openvswitch-agent services After defining the host groups, the physical architecture diagram should be updated to reflect the mapping of host groups to physical machines in the deployment. This should also include considerations for network connectivity. The following is an example architecture diagram for inclusion in the design document: Summary A complete guide to implementing H/A of the OpenStack services is probably worth a book to itself. In this article we started out by covering the main strategies for making OpenStack services highly available and which strategies apply well to each service. Then we covered how OpenStack deployments are typically segmented across physical regions. Finally, we updated our documentation and implemented a few of the technologies we discussed in the lab. While walking through the main considerations for highly available deployments in this article, we've tried to emphasize a few key points: Scalability is at least as important as H/A in cluster design. Ensure that your design is flexible in case of unexpected growth. OpenStack doesn't scale forever. Plan for multiple regions. Also, it's important to make sure that the strategy and architecture that you adopt for H/A is supportable by your organization. Consider reusing existing architectures for H/A in the message bus and database layers. Resources for Article:  Further resources on this subject: Neutron API Basics [article] The OpenFlow Controllers [article] OpenStack Networking in a Nutshell [article]