How-To Tutorials

In this article by Brett Lantz, author of the book Machine Learning with R, Second Edition, we will get a basic understanding about decision trees and rule learners, including the C5.0 decision tree algorithm. This algorithm will cover mechanisms such as choosing the best split and pruning the decision tree. While deciding between several job offers with various levels of pay and benefits, many people begin by making lists of pros and cons, and eliminate options based on simple rules. For instance, ''if I have to commute for more than an hour, I will be unhappy.'' Or, ''if I make less than $50k, I won't be able to support my family.'' In this way, the complex and difficult decision of predicting one's future happiness can be reduced to a series of simple decisions. This article covers decision trees and rule learners—two machine learning methods that also make complex decisions from sets of simple choices. These methods then present their knowledge in the form of logical structures that can be understood with no statistical knowledge. This aspect makes these models particularly useful for business strategy and process improvement. By the end of this article, you will learn: How trees and rules "greedily" partition data into interesting segments The most common decision tree and classification rule learners, including the C5.0, 1R, and RIPPER algorithms We will begin by examining decision trees, followed by a look at classification rules. (For more resources related to this topic, see here.) Understanding decision trees Decision tree learners are powerful classifiers, which utilize a tree structure to model the relationships among the features and the potential outcomes. As illustrated in the following figure, this structure earned its name due to the fact that it mirrors how a literal tree begins at a wide trunk, which if followed upward, splits into narrower and narrower branches. In much the same way, a decision tree classifier uses a structure of branching decisions, which channel examples into a final predicted class value. To better understand how this works in practice, let's consider the following tree, which predicts whether a job offer should be accepted. A job offer to be considered begins at the root node, where it is then passed through decision nodes that require choices to be made based on the attributes of the job. These choices split the data across branches that indicate potential outcomes of a decision, depicted here as yes or no outcomes, though in some cases there may be more than two possibilities. In the case a final decision can be made, the tree is terminated by leaf nodes (also known as terminal nodes) that denote the action to be taken as the result of the series of decisions. In the case of a predictive model, the leaf nodes provide the expected result given the series of events in the tree. A great benefit of decision tree algorithms is that the flowchart-like tree structure is not necessarily exclusively for the learner's internal use. After the model is created, many decision tree algorithms output the resulting structure in a human-readable format. This provides tremendous insight into how and why the model works or doesn't work well for a particular task. This also makes decision trees particularly appropriate for applications in which the classification mechanism needs to be transparent for legal reasons, or in case the results need to be shared with others in order to inform future business practices. With this in mind, some potential uses include: Credit scoring models in which the criteria that causes an applicant to be rejected need to be clearly documented and free from bias Marketing studies of customer behavior such as satisfaction or churn, which will be shared with management or advertising agencies Diagnosis of medical conditions based on laboratory measurements, symptoms, or the rate of disease progression Although the previous applications illustrate the value of trees in informing decision processes, this is not to suggest that their utility ends here. In fact, decision trees are perhaps the single most widely used machine learning technique, and can be applied to model almost any type of data—often with excellent out-of-the-box applications. This said, in spite of their wide applicability, it is worth noting some scenarios where trees may not be an ideal fit. One such case might be a task where the data has a large number of nominal features with many levels or it has a large number of numeric features. These cases may result in a very large number of decisions and an overly complex tree. They may also contribute to the tendency of decision trees to overfit data, though as we will soon see, even this weakness can be overcome by adjusting some simple parameters. Divide and conquer Decision trees are built using a heuristic called recursive partitioning. This approach is also commonly known as divide and conquer because it splits the data into subsets, which are then split repeatedly into even smaller subsets, and so on and so forth until the process stops when the algorithm determines the data within the subsets are sufficiently homogenous, or another stopping criterion has been met. To see how splitting a dataset can create a decision tree, imagine a bare root node that will grow into a mature tree. At first, the root node represents the entire dataset, since no splitting has transpired. Next, the decision tree algorithm must choose a feature to split upon; ideally, it chooses the feature most predictive of the target class. The examples are then partitioned into groups according to the distinct values of this feature, and the first set of tree branches are formed. Working down each branch, the algorithm continues to divide and conquer the data, choosing the best candidate feature each time to create another decision node, until a stopping criterion is reached. Divide and conquer might stop at a node in a case that: All (or nearly all) of the examples at the node have the same class There are no remaining features to distinguish among the examples The tree has grown to a predefined size limit To illustrate the tree building process, let's consider a simple example. Imagine that you work for a Hollywood studio, where your role is to decide whether the studio should move forward with producing the screenplays pitched by promising new authors. After returning from a vacation, your desk is piled high with proposals. Without the time to read each proposal cover-to-cover, you decide to develop a decision tree algorithm to predict whether a potential movie would fall into one of three categories: Critical Success, Mainstream Hit, or Box Office Bust. To build the decision tree, you turn to the studio archives to examine the factors leading to the success and failure of the company's 30 most recent releases. You quickly notice a relationship between the film's estimated shooting budget, the number of A-list celebrities lined up for starring roles, and the level of success. Excited about this finding, you produce a scatterplot to illustrate the pattern: Using the divide and conquer strategy, we can build a simple decision tree from this data. First, to create the tree's root node, we split the feature indicating the number of celebrities, partitioning the movies into groups with and without a significant number of A-list stars: Next, among the group of movies with a larger number of celebrities, we can make another split between movies with and without a high budget: At this point, we have partitioned the data into three groups. The group at the top-left corner of the diagram is composed entirely of critically acclaimed films. This group is distinguished by a high number of celebrities and a relatively low budget. At the top-right corner, majority of movies are box office hits with high budgets and a large number of celebrities. The final group, which has little star power but budgets ranging from small to large, contains the flops. If we wanted, we could continue to divide and conquer the data by splitting it based on the increasingly specific ranges of budget and celebrity count, until each of the currently misclassified values resides in its own tiny partition, and is correctly classified. However, it is not advisable to overfit a decision tree in this way. Though there is nothing to stop us from splitting the data indefinitely, overly specific decisions do not always generalize more broadly. We'll avoid the problem of overfitting by stopping the algorithm here, since more than 80 percent of the examples in each group are from a single class. This forms the basis of our stopping criterion. You might have noticed that diagonal lines might have split the data even more cleanly. This is one limitation of the decision tree's knowledge representation, which uses axis-parallel splits. The fact that each split considers one feature at a time prevents the decision tree from forming more complex decision boundaries. For example, a diagonal line could be created by a decision that asks, "is the number of celebrities is greater than the estimated budget?" If so, then "it will be a critical success." Our model for predicting the future success of movies can be represented in a simple tree, as shown in the following diagram. To evaluate a script, follow the branches through each decision until the script's success or failure has been predicted. In no time, you will be able to identify the most promising options among the backlog of scripts and get back to more important work, such as writing an Academy Awards acceptance speech. Since real-world data contains more than two features, decision trees quickly become far more complex than this, with many more nodes, branches, and leaves. In the next section, you will learn about a popular algorithm to build decision tree models automatically. The C5.0 decision tree algorithm There are numerous implementations of decision trees, but one of the most well-known implementations is the C5.0 algorithm. This algorithm was developed by computer scientist J. Ross Quinlan as an improved version of his prior algorithm, C4.5, which itself is an improvement over his Iterative Dichotomiser 3 (ID3) algorithm. Although Quinlan markets C5.0 to commercial clients (see http://www.rulequest.com/ for details), the source code for a single-threaded version of the algorithm was made publically available, and it has therefore been incorporated into programs such as R. To further confuse matters, a popular Java-based open source alternative to C4.5, titled J48, is included in R's RWeka package. Because the differences among C5.0, C4.5, and J48 are minor, the principles in this article will apply to any of these three methods, and the algorithms should be considered synonymous. The C5.0 algorithm has become the industry standard to produce decision trees, because it does well for most types of problems directly out of the box. Compared to other advanced machine learning models, the decision trees built by C5.0 generally perform nearly as well, but are much easier to understand and deploy. Additionally, as shown in the following table, the algorithm's weaknesses are relatively minor and can be largely avoided: Strengths Weaknesses An all-purpose classifier that does well on most problems Highly automatic learning process, which can handle numeric or nominal features, as well as missing data Excludes unimportant features Can be used on both small and large datasets Results in a model that can be interpreted without a mathematical background (for relatively small trees) More efficient than other complex models Decision tree models are often biased toward splits on features having a large number of levels It is easy to overfit or underfit the model Can have trouble modeling some relationships due to reliance on axis-parallel splits Small changes in the training data can result in large changes to decision logic Large trees can be difficult to interpret and the decisions they make may seem counterintuitive To keep things simple, our earlier decision tree example ignored the mathematics involved in how a machine would employ a divide and conquer strategy. Let's explore this in more detail to examine how this heuristic works in practice. Choosing the best split The first challenge that a decision tree will face is to identify which feature to split upon. In the previous example, we looked for a way to split the data such that the resulting partitions contained examples primarily of a single class. The degree to which a subset of examples contains only a single class is known as purity, and any subset composed of only a single class is called pure. There are various measurements of purity that can be used to identify the best decision tree splitting candidate. C5.0 uses entropy, a concept borrowed from information theory that quantifies the randomness, or disorder, within a set of class values. Sets with high entropy are very diverse and provide little information about other items that may also belong in the set, as there is no apparent commonality. The decision tree hopes to find splits that reduce entropy, ultimately increasing homogeneity within the groups. Typically, entropy is measured in bits. If there are only two possible classes, entropy values can range from 0 to 1. For n classes, entropy ranges from 0 to log2(n). In each case, the minimum value indicates that the sample is completely homogenous, while the maximum value indicates that the data are as diverse as possible, and no group has even a small plurality. In the mathematical notion, entropy is specified as follows: In this formula, for a given segment of data (S), the term c refers to the number of class levels and pi refers to the proportion of values falling into class level i. For example, suppose we have a partition of data with two classes: red (60 percent) and white (40 percent). We can calculate the entropy as follows: > -0.60 * log2(0.60) - 0.40 * log2(0.40) [1] 0.9709506 We can examine the entropy for all the possible two-class arrangements. If we know that the proportion of examples in one class is x, then the proportion in the other class is (1 – x). Using the curve() function, we can then plot the entropy for all the possible values of x: > curve(-x * log2(x) - (1 - x) * log2(1 - x),        col = "red", xlab = "x", ylab = "Entropy", lwd = 4) This results in the following figure: As illustrated by the peak in entropy at x = 0.50, a 50-50 split results in maximum entropy. As one class increasingly dominates the other, the entropy reduces to zero. To use entropy to determine the optimal feature to split upon, the algorithm calculates the change in homogeneity that would result from a split on each possible feature, which is a measure known as information gain. The information gain for a feature F is calculated as the difference between the entropy in the segment before the split (S1) and the partitions resulting from the split (S2): One complication is that after a split, the data is divided into more than one partition. Therefore, the function to calculate Entropy(S2) needs to consider the total entropy across all of the partitions. It does this by weighing each partition's entropy by the proportion of records falling into the partition. This can be stated in a formula as: In simple terms, the total entropy resulting from a split is the sum of the entropy of each of the n partitions weighted by the proportion of examples falling in the partition (wi). The higher the information gain, the better a feature is at creating homogeneous groups after a split on this feature. If the information gain is zero, there is no reduction in entropy for splitting on this feature. On the other hand, the maximum information gain is equal to the entropy prior to the split. This would imply that the entropy after the split is zero, which means that the split results in completely homogeneous groups. The previous formulae assume nominal features, but decision trees use information gain for splitting on numeric features as well. To do so, a common practice is to test various splits that divide the values into groups greater than or less than a numeric threshold. This reduces the numeric feature into a two-level categorical feature that allows information gain to be calculated as usual. The numeric cut point yielding the largest information gain is chosen for the split. Though it is used by C5.0, information gain is not the only splitting criterion that can be used to build decision trees. Other commonly used criteria are Gini index, Chi-Squared statistic, and gain ratio. For a review of these (and many more) criteria, refer to Mingers J. An Empirical Comparison of Selection Measures for Decision-Tree Induction. Machine Learning. 1989; 3:319-342. Pruning the decision tree A decision tree can continue to grow indefinitely, choosing splitting features and dividing the data into smaller and smaller partitions until each example is perfectly classified or the algorithm runs out of features to split on. However, if the tree grows overly large, many of the decisions it makes will be overly specific and the model will be overfitted to the training data. The process of pruning a decision tree involves reducing its size such that it generalizes better to unseen data. One solution to this problem is to stop the tree from growing once it reaches a certain number of decisions or when the decision nodes contain only a small number of examples. This is called early stopping or pre-pruning the decision tree. As the tree avoids doing needless work, this is an appealing strategy. However, one downside to this approach is that there is no way to know whether the tree will miss subtle, but important patterns that it would have learned had it grown to a larger size. An alternative, called post-pruning, involves growing a tree that is intentionally too large and pruning leaf nodes to reduce the size of the tree to a more appropriate level. This is often a more effective approach than pre-pruning, because it is quite difficult to determine the optimal depth of a decision tree without growing it first. Pruning the tree later on allows the algorithm to be certain that all the important data structures were discovered. The implementation details of pruning operations are very technical and beyond the scope of this article. For a comparison of some of the available methods, see Esposito F, Malerba D, Semeraro G. A Comparative Analysis of Methods for Pruning Decision Trees. IEEE Transactions on Pattern Analysis and Machine Intelligence. 1997;19: 476-491. One of the benefits of the C5.0 algorithm is that it is opinionated about pruning—it takes care of many decisions automatically using fairly reasonable defaults. Its overall strategy is to post-prune the tree. It first grows a large tree that overfits the training data. Later, the nodes and branches that have little effect on the classification errors are removed. In some cases, entire branches are moved further up the tree or replaced by simpler decisions. These processes of grafting branches are known as subtree raising and subtree replacement, respectively. Balancing overfitting and underfitting a decision tree is a bit of an art, but if model accuracy is vital, it may be worth investing some time with various pruning options to see if it improves the performance on test data. As you will soon see, one of the strengths of the C5.0 algorithm is that it is very easy to adjust the training options. Summary This article covered two classification methods that use so-called "greedy" algorithms to partition the data according to feature values. Decision trees use a divide and conquer strategy to create flowchart-like structures, while rule learners separate and conquer data to identify logical if-else rules. Both methods produce models that can be interpreted without a statistical background. One popular and highly configurable decision tree algorithm is C5.0. We used the C5.0 algorithm to create a tree to predict whether a loan applicant will default. This article merely scratched the surface of how trees and rules can be used. Resources for Article: Further resources on this subject: Introduction to S4 Classes [article] First steps with R [article] Supervised learning [article]

In this article by Phillip Fehre, author of the book JavaScript Domain-Driven Design, we will explore a practical approach to developing software with advanced business logic. There are many strategies to keep development flowing and the code and thoughts organized, there are frameworks building on conventions, there are different software paradigms such as object orientation and functional programming, or methodologies such as test-driven development. All these pieces solve problems, and are like tools in a toolbox to help manage growing complexity in software, but they also mean that today when starting something new, there are loads of decisions to make even before we get started at all. Do we want to develop a single-page application, do we want to develop following the standards of a framework closely or do we want to set our own? These kinds of decisions are important, but they also largely depend on the context of the application, and in most cases the best answer to the questions is: it depends. (For more resources related to this topic, see here.) So, how do we really start? Do we really even know what our problem is, and, if we understand it, does this understanding match that of others? Developers are very seldom the domain experts on a given topic. Therefore, the development process needs input from outside through experts of the business domain when it comes to specifying the behavior a system should have. Of course, this is not only true for a completely new project developed from the ground up, but also can be applied to any new feature added during development of to an application or product. So, even if your project is well on its way already, there will come a time when a new feature just seems to bog the whole thing down and, at this stage, you may want to think about alternative ways to go about approaching this new piece of functionality. Domain-driven design gives us another useful piece to play with, especially to solve the need to interact with other developers, business experts, and product owners. As in the modern era, JavaScript becomes a more and more persuasive choice to build projects in and, in many cases like browser-based web applications, it actually is the only viable choice. Today, the need to design software with JavaScript is more pressing than ever. In the past, the issues of a more involved software design were focused on either backend or client application development, with the rise of JavaScript as a language to develop complete systems in, this has changed. The development of a JavaScript client in the browser is a complex part of developing the application as a whole, and so is the development of server-side JavaScript applications with the rise of Node.js. In modern development, JavaScript plays a major role and therefore needs to receive the same amount of attention in development practices and processes as other languages and frameworks have in the past. A browser based client-side application often holds the same amount, or even more logic, than the backend. With this change, a lot of new problems and solutions have arisen, the first being the movement toward better encapsulation and modularization of JavaScript projects. New frameworks have arisen and established themselves as the bases for many projects. Last but not least, JavaScript made the jump from being the language in the browser to move more and more to the server side, by means of Node.js or as the query language of choice in some NoSQL databases. Let me take you on a tour of developing a piece of software, taking you through the stages of creating an application from start to finish using the concepts domain-driven design introduced and how they can be interpreted and applied. In this article, you will cover: The core idea of domain-driven design Our business scenario—managing an orc dungeon Tracking the business logic Understanding the core problem and selecting the right solution Learning what domain-driven design is The core idea of domain-driven design There are many software development methodologies around, all with pros and cons but all also have a core idea, which is to be applied and understood to get the methodology right. For a domain-driven design, the core lies in the realization that since we are not the experts in the domain the software is placed in, we need to gather input from other people who are experts. This realization means that we need to optimize our development process to gather and incorporate this input. So, what does this mean for JavaScript? When thinking about a browser application to expose a certain functionality to a consumer, we need to think about many things, for example: How does the user expect the application to behave in the browser? How does the business workflow work? What does the user know about the workflow? These three questions already involve three different types of experts: a person skilled in user experience can help with the first query, a business domain expert can address the second query, and a third person can research the target audience and provide input on the last query. Bringing all of this together is the goal we are trying to achieve. While the different types of people matter, the core idea is that the process of getting them involved is always the same. We provide a common way to talk about the process and establish a quick feedback loop for them to review. In JavaScript, this can be easier than in most other languages due to the nature of it being run in a browser, readily available to be modified and prototyped with; an advantage Java Enterprise Applications can only dream of. We can work closely with the user experience designer adjusting the expected interface and at the same time change the workflow dynamically to suit our business needs, first on the frontend in the browser and later moving the knowledge out of the prototype to the backend, if necessary. Managing an orc dungeon When talking about domain-driven design, it is often stated in the context of having complex business logic to deal with. In fact, most software development practices are not really useful when dealing with a very small, cut-out problem. Like with every tool, you need to be clear when it is the right time to use it. So, what does really fall in to the realm of complex business logic? It means that the software has to describe a real-world scenario, which normally involves human thinking and interaction. Writing software that deals with decisions, which 90 per cent of the time go a certain way and ten per cent of the time it's some other way, is notoriously hard, especially when explaining it to people not familiar with software. These kind of decisions are the core of many business problems, but even though this is an interesting problem to solve, following how the next accounting software is developed does not make an interesting read. With this in mind, I would like to introduce you to the problem we are trying to solve, that is, managing a dungeon. An orc Inside the dungeon Running an orc dungeon seems pretty simple from the outside, but managing it without getting killed is actually rather complicated. For this reason, we are contacted by an orc master who struggles with keeping his dungeon running smoothly. When we arrive at the dungeon, he explains to us how it actually works and what factors come into play. Even greenfield projects often have some status quo that work. This is important to keep in mind since it means that we don't have to come up with the feature set, but match the feature set of the current reality. Many outside factors play a role and the dungeon is not as independent at it would like to be. After all, it is part of the orc kingdom, and the king demands that his dungeons make him money. However, money is just part of the deal. How does it actually make money? The prisoners need to mine gold and to do that there needs to be a certain amount of prisoners in the dungeon that need to be kept. The way an orc kingdom is run also results in the constant arrival of new prisoners, new captures from war, those who couldn't afford their taxes, and so on. There always needs to be room for new prisoners. The good thing is that every dungeon is interconnected, and to achieve its goals it can rely on others by requesting a prisoner transfer to either fill up free cells or get rid of overflowing prisoners in its cells. These options allow the dungeon masters to keep a close watch on prisoners being kept and the amount of cell space available. Sending off prisoners into other dungeons as needed and requesting new ones from other dungeons, in case there is too much free cell space available, keeps the mining workforce at an optimal level for maximizing the profit, while at the same time being ready to accommodate the eventual arrival of a high value inmate sent directly to the dungeon. So far, the explanation is sound, but let's dig a little deeper and see what is going on. Managing incoming prisoners Prisoners can arrive for a couple of reasons, such as if a dungeon is overflowing and decides to transfer some of its inmates to a dungeon with free cells and, unless they flee on the way, they will eventually arrive at our dungeon sooner or later. Another source of prisoners is the ever expanding orc kingdom itself. The orcs will constantly enslave new folk and telling our king, "Sorry we don't have room", is not a valid option, it might actually result in us being one of the new prisoners. Looking at this, our dungeon will fill up eventually, but we need to make sure this doesn't happen. The way to handle this is by transferring inmates early enough to make room. This is obviously going to be the most complicated thing; we need to weigh several factors to decide when and how many prisoners to transfer. The reason we can't simply solve this via thresholds is that looking at the dungeon structure, this is not the only way we can lose inmates. After all, people are not always happy with being gold mining slaves and may decide the risk of dying in a prison is as high as dying while fleeing. Therefore, they decide to do so. The same is true while prisoners are on the move between different dungeons as well, and not unlikely. So even though we have a hard limit of physical cells, we need to deal with the soft number of incoming and outgoing prisoners. This is a classical problem in business software. Matching these numbers against each other and optimizing for a certain outcome is basically what computer data analysis is all about. The current state of the art With all this in mind, it becomes clear that the orc master's current system of keeping track via a badly written note on a napkin is not perfect. In fact, it almost got him killed multiple times already. To give you an example of what can happen, he tells the story of how one time the king captured four clan leaders and wanted to make them miners just to humiliate them. However, when arriving at the dungeon, he realized that there was no room and had to travel to the next dungeon to drop them off, all while having them laugh at him because he obviously didn't know how to run a kingdom. This was due to our orc master having forgotten about the arrival of eight transfers just the day before. Another time, the orc master was not able to deliver any gold when the king's sheriff arrived because he didn't know he only had one-third of his required prisoners to actually mine anything. This time it was due to having multiple people count the inmates, and instead of recoding them cell-by-cell, they actually tried to do it in their head. While being orc, this is a setup for failure. All this comes down to bad organization, and having your system to manage dungeon inmates drawn on the back of a napkin certainly qualifies as such. Digital dungeon management Guided by the recent failures, the orc master has finally realized it is time to move to modern times, and he wants to revolutionize the way to manage his dungeon by making everything digital. He strives to have a system that basically takes the busywork out of managing by automatically calculating the necessary transfers according to the current amount of cells filled. He would like to just sit back, relax and let the computer do all the work for him. A common pattern when talking with a business expert about software is that they are not aware of what can be done. Always remember that we, as developers, are the software experts and therefore are the only ones who are able to manage these expectations. It is time now for us to think about what we need to know about the details and how to deal with the different scenarios. The orc master is not really familiar with the concepts of software development, so we need to make sure we talk in a language he can follow and understand, while making sure we get all the answers we need. We are hired for our expertise in software development, so we need to make sure to manage the expectations as well as the feature set and development flow. The development itself is of course going to be an iterative process, since we can't expect to get a list of everything needed right in one go. It also means that we will need to keep possible changes in mind. This is an essential part of structuring complex business software. Developing software containing more complex business logic is prone to changing rapidly as the business is adapting itself and the users leverage the functionality the software provides. Therefore, it is essential to keep a common language between the people who understand the business and the developers who understand the software. Incorporate the business terms wherever possible, it will ease communication between the business domain experts and you as a developer and therefore prevent misunderstandings early on. Specification To create a good understanding of what a piece of software needs to do, at least to be useful in the best way, is to get an understanding of what the future users were doing before your software existed. Therefore, we sit down with the orc master as he is managing his incoming and outgoing prisoners, and let him walk us through what he is doing on a day-to-day basis. The dungeon is comprised of 100 cells that are either occupied by a prisoner or empty at the moment. When managing these cells, we can identify distinct tasks by watching the orc do his job. Drawing out what we see, we can roughly sketch it like this: There are a couple of organizational important events and states to be tracked, they are: Currently available or empty cells Outgoing transfer states Incoming transfer states Each transfer can be in multiple states that the master has to know about to make further decisions on what to do next. Keeping a view of the world like this is not easy especially accounting for the amount of concurrent updates happening. Tracking the state of everything results in further tasks for our master to do: Update the tracking Start outgoing transfers when too many cells are occupied Respond to incoming transfers by starting to track them Ask for incoming transfers if the occupied cells are to low So, what does each of them involve? Tracking available cells The current state of the dungeon is reflected by the state of its cells, so the first task is to get this knowledge. In its basic form, this is easily achievable by simply counting every occupied and every empty cell, writing down what the values are. Right now, our orc master tours the dungeon in the morning, noting each free cell assuming that the other one must be occupied. To make sure he does not get into trouble, he no longer trusts his subordinates to do that! The problem being that there only is one central sheet to keep track of everything, so his keepers may overwrite each other's information accidently if there is more than one person counting and writing down cells. Also, this is a good start and is sufficient as it is right now, although it misses some information that would be interesting to have, for example, the amount of inmates fleeing the dungeon and an understanding of the expected free cells based on this rate. For us, this means that we need to be able track this information inside the application, since ultimately we want to project the expected amount of free cells so that we can effectively create recommendations or warnings based on the dungeon state. Starting outgoing transfers The second part is to actually handle getting rid of prisoners in case the dungeon fills up. In this concrete case, this means that if the number of free cells drops beneath 10, it is time to move prisoners out, since there may be new prisoners coming at any time. This strategy works pretty reliably since, from experience, it has been established that there are hardly any larger transports, so the recommendation is to stick with it in the beginning. However, we can already see some optimizations which currently are too complex. Drawing from the experience of the business is important, as it is possible to encode such knowledge and reduces mistakes, but be mindful since encoding detailed experience is probably one of the most complex things to do. In the future, we want to optimize this based on the rate of inmates fleeing the dungeon, new prisoners arriving due to being captured, as well as the projection of new arrivals from transfers. All this is impossible right now, since it will just overwhelm the current tracking system, but it actually comes down to capturing as much data as possible and analyzing it, which is something modern computer systems are good at. After all, it could save the orc master's head! Tracking the state of incoming transfers On some days, a raven will arrive bringing news that some prisoners have been sent on their way to be transferred to our dungeon. There really is nothing we can do about it, but the protocol is to send the raven out five days prior to the prisoners actually arriving to give the dungeon a chance to prepare. Should prisoners flee along the way, another raven will be sent informing the dungeon of this embarrassing situation. These messages have to be sifted through every day, to make sure there actually is room available for those arriving. This is a big part of projecting the amount of filled cells, and also the most variable part, we get told. It is important to note that every message should only be processed once, but it can arrive at any time during the day. Right now, they are all dealt with by one orc, who throws them out immediately after noting what the content results in. One problem with the current system is that since other dungeons are managed the same way ours is currently, they react with quick and large transfers when they get in trouble, which makes this quite unpredictable. Initiating incoming transfers Besides keeping the prisoners where they belong, mining gold is the second major goal of the dungeon. To do this, there needs to be a certain amount of prisoners available to man the machines, otherwise production will essentially halt. This means that whenever too many cells become abandoned it is time to fill them, so the orc master sends a raven to request new prisoners in. This again takes five days and, unless they flee along the way, works reliably. In the past, it still has been a major problem for the dungeon due to the long delay. If the filled cells drop below 50, the dungeon will no longer produce any gold and not making money is a reason to replace the current dungeon master. If all the orc master does is react to the situation, it means that there will probably be about five days in which no gold will be mined. This is one of the major pain points in the current system because projecting the amount of filled cells five days out seems rather impossible, so all the orcs can do right now is react. All in all, this gives us a rough idea what the dungeon master is looking for and which tasks need to be accomplished to replace the current system. Of course, this does not have to happen in one go, but can be done gradually so everybody adjusts. Right now, it is time for us to identify where to start. From greenfield to application We are JavaScript developers, so it seems obvious for us to build a web application to implement this. As the problem is described, it is clear that starting out simply and growing the application as we further analyze the situation is clearly the way to go. Right now, we don't really have a clear understanding how some parts should be handled since the business process has not evolved to this level, yet. Also, it is possible that new features will arise or things start being handled differently as our software begins to get used. The steps described leave room for optimization based on collected data, so we first need the data to see how predictions can work. This means that we need to start by tracking as many events as possible in the dungeon. Running down the list, the first step is always to get a view of which state we are in, this means tracking the available cells and providing an interface for this. To start out, this can be done via a counter, but this can't be our final solution. So, we then need to grow toward tracking events and summing those to be able to make predictions for the future. The first route and model Of course there are many other ways to get started, but what it boils down to in most cases is that it is time now to choose the base to build on. By this I mean deciding on a framework or set of libraries to build upon. This happens alongside the decision on what database is used to back our application and many other small decisions, which are influenced by influenced by those decisions around framework and libraries. A clear understanding on how the frontend should be built is important as well, since building a single-page application, which implements a large amount of logic in the frontend and is backed by an API layer that differs a lot from an application, which implements most logic on the server side. Don't worry if you are unfamiliar with express or any other technology used in the following. You don't need to understand every single detail, but you will get the idea of how developing an application with a framework is achieved. Since we don't have a clear understanding, yet, which way the application will ultimately take, we try to push as many decisions as possible out, but decide on the stuff we immediately need. As we are developing in JavaScript, the application is going to be developed in Node.js and express is going to be our framework of choice. To make our life easier, we first decide that we are going to implement the frontend in plain HTML using EJS embedded JavaScript templates, since it will keep the logic in one place. This seems sensible since spreading the logic of a complex application across multiple layers will complicate things even further. Also, getting rid of the eventual errors during transport will ease our way toward a solid application in the beginning. We can push the decision about the database out and work with simple objects stored in RAM for our first prototype; this is, of course, no long-term solution, but we can at least validate some structure before we need to decide on another major piece of software, which brings along a lot of expectations as well. With all this in mind, we setup the application. In the following section and throughout the book, we are using Node.js to build a small backend. At the time of the writing, the currently active version was Node.js 0.10.33. Node.js can be obtained from http://nodejs.org/ and is available for Windows, Mac OS X, and Linux. The foundation for our web application is provided by express, available via the Node Package Manager (NPM) at the time of writing in version 3.0.3: $ npm install –g express$ express --ejs inmatr For the sake of brevity, the glue code in the following is omitted, but like all other code presented in the book, the code is available on the GitHub repository https://github.com/sideshowcoder/ddd-js-sample-code. Creating the model The most basic parts of the application are set up now. We can move on to creating our dungeon model in models/dungeon.js and add the following code to it to keep a model and its loading and saving logic: var Dungeon = function(cells) {this.cells = cellsthis.bookedCells = 0} Keeping in mind that this will eventually be stored in a database, we also need to be able to find a dungeon in some way, so the find method seems reasonable. This method should already adhere to the Node.js callback style to make our lives easier when switching to a real database. Even though we pushed this decision out, the assumption is clear since, even if we decide against a database, the dungeon reference will be stored and requested from outside the process in the future. The following shows an example with the find method: var dungeons = {}Dungeon.find = function(id, callback) {if(!dungeons[id]) {   dungeons[id] = new Dungeon(100)}callback(null, dungeons[id])} The first route and loading the dungeon Now that we have this in place, we can move on to actually react to requests. In express defining, the needed routes do this. Since we need to make sure we have our current dungeon available, we also use middleware to load it when a request comes in. Using the methods we just created, we can add a middleware to the express stack to load the dungeon whenever a request comes in. A middleware is a piece of code, which gets executed whenever a request reaches its level of the stack, for example, the router used to dispatch requests to defined functions is implemented as a middleware, as is logging and so on. This is a common pattern for many other kinds of interactions as well, such as user login. Our dungeon loading middleware looks like this, assuming for now we only manage one dungeon we can create it by adding a file in middleware/load_context.js with the following code: function(req, res, next) {req.context = req.context || {}Dungeon.find('main', function(err, dungeon) {   req.context.dungeon = dungeon   next()})} Displaying the page With this, we are now able to simply display information about the dungeon and track any changes made to it inside the request. Creating a view to render the state, as well as a form to modify it, are the essential parts of our GUI. Since we decided to implement the logic server-side, they are rather barebones. Creating a view under views/index.ejs allows us to render everything to the browser via express later. The following example is the HTML code for the frontend: <h1>Inmatr</h1> <p>You currently have <%= dungeon.free %> of <%= dungeon.cells %> cells available.</p>   <form action="/cells/book" method="post"> <select name="cells">    <% for(var i = 1; i < 11; i++) { %>    <option value="<%= i %>"><%= i %></option> <% } %> </select> <button type="submit" name="book" value="book"> Book cells</button> <button type="submit" name="free" value="free"> Free cells</button> </form> Gluing the application together via express Now that we are almost done, we have a display for the state, a model to track what is changing, and a middleware to load this model as needed. Now, to glue it all together we will use express to register our routes and call the necessary functions. We mainly need two routes: one to display the page and one to accept and process the form input. Displaying the page is done when a user hits the index page, so we need to bind to the root path. Accepting the form input is already declared in the form itself as /cells/book. We can just create a route for it. In express, we define routes in relation to the main app object and according to the HTTP verbs as follows: app.get('/', routes.index) app.post('/cells/book', routes.cells.book) Adding this to the main app.js file allows express to wire things up, the routes itself are implemented as follows in the routes/index.js file: var routes = { index: function(req, res){    res.render('index', req.context) },   cells: { book: function(req, res){    var dungeon = req.context.dungeon    var cells = parseInt(req.body.cells)    if (req.body.book) {    dungeon.book(cells) } else {    dungeon.unbook(cells) }        res.redirect('/')    } } } With this done, we have a working application to track free and used cells. The following shows the frontend output for the tracking system: Moving the application forward This is only the first step toward the application that will hopefully automate what is currently done by hand. With the first start in place, it is now time to make sure we can move the application along. We have to think about what this application is supposed to do and identify the next steps. After presenting the current state back to the business the next request is most likely to be to integrate some kind of login, since it will not be possible to modify the state of the dungeon unless you are authorized to do it. Since this is a web application, most people are familiar with them having a login. This moves us into a complicated space in which we need to start specifying the roles in the application along with their access patterns; so it is not clear if this is the way to go. Another route to take is starting to move the application towards tracking events instead of pure numbers of the free cells. From a developer's point of view, this is probably the most interesting route but the immediate business value might be hard to justify, since without the login it seems unusable. We need to create an endpoint to record events such as fleeing prisoner, and then modify the state of the dungeon according to those tracked events. This is based on the assumption that the highest value for the application will lie in the prediction of the prisoner movement. When we want to track free cells in such a way, we will need to modify the way our first version of the application works. The logic on what events need to be created will have to move somewhere, most logically the frontend, and the dungeon will no longer be the single source of truth for the dungeon state. Rather, it will be an aggregator for the state, which is modified by the generation of events. Thinking about the application in such a way makes some things clear. We are not completely sure what the value proposition of the application ultimately will be. This leads us down a dangerous path since the design decisions that we make now will impact how we build new features inside the application. This is also a problem in case our assumption about the main value proposition turns out to be wrong. In this case, we may have built quite a complex event tracking system which does not really solve the problem but complicates things. Every state modification needs to be transformed into a series of events where a simple state update on an object may have been enough. Not only does this design not solve the real problem, explaining it to the orc master is also tough. There are certain abstractions missing, and the communication is not following a pattern established as the business language. We need an alternative approach to keep the business more involved. Also, we need to keep development simple using abstraction on the business logic and not on the technologies, which are provided by the frameworks that are used. Summary In this article you were introduced to a typical business application and how it is developed. It showed how domain-driven design can help steer clear of common issues during the development to create a more problem-tailored application. Resources for Article: Further resources on this subject: An Introduction to Mastering JavaScript Promises and Its Implementation in Angular.js [article] Developing a JavaFX Application for iOS [article] Object-Oriented JavaScript with Backbone Classes [article]

In this article written by Adam Boduch, author of the book JavaScript at Scale, goes on to say how we don't scale our software systems just because we can. While it's common to tout scalability, these claims need to be put into practice. In order to do so, there has to be a reason for scalable software. If there's no need to scale, then it's much easier, not to mention cost-effective, to simply build a system that doesn't scale. Putting something that was built to handle a wide variety of scaling issues into a context where scale isn't warranted just feels clunky. Especially to the end user. So we, as JavaScript developers and architects, need to acknowledge and understand the influences that necessitate scalability. While it's true that not all JavaScript applications need to scale, it may not always be the case. For example, it's difficult to say that we know this system isn't going to need to scale in any meaningful way, so let's not invest the time and effort to make it scalable. Unless we're developing a throw-away system, there's always going to be expectations of growth and success. At the opposite end of the spectrum, JavaScript applications aren't born as mature scalable systems. They grow up, accumulating scalable properties along the way. Scaling influencers are an effective tool for those of us working on JavaScript projects. We don't want to over-engineer something straight from inception, and we don't want to build something that's tied-down by early decisions, limiting its ability to scale. (For more resources related to this topic, see here.) The need for scale Scaling software is a reactive event. Thinking about scaling influencers helps us proactively prepare for these scaling events. In other systems, such as web application backends, these scaling events may be brief spikes, and are generally handled automatically. For example, there's an increased load due to more users issuing more requests. The load balancer kicks in and distributes the load evenly across backend servers. In the extreme case, the system may automatically provision new backend resources when needed, and destroy them when they're no longer of use. Scaling events in the frontend aren't like that. Rather, the scaling events that take place generally happen over longer periods of time, and are more complex. The unique aspect of JavaScript applications is that the only hardware resources available to them are those available to the browser in which they run. They get their data from the backend, and this may scale up perfectly fine, but that's not what we're concerned with. As our software grows, a necessary side-effect of doing something successfully, is that we need to pay attention to the influencers of scale. The preceding figure shows us a top-down flow chart of scaling influencers, starting with users, who require that our software implements features. Depending on various aspects of the features, such as their size and how they relate to other features, this influences the team of developers working on features. As we move down through the scaling influencers, this grows. Growing user base We're not building an application for just one user. If we were, there would be no need to scale our efforts. While what we build might be based on the requirements of one user representative, our software serves the needs of many users. We need to anticipate a growing user base as our application evolves. There's no exact target user count, although, depending on the nature of our application, we may set goals for the number of active users, possibly by benchmarking similar applications using a tool such as http://www.alexa.com/. For example, if our application is exposed on the public internet, we want lots of registered users. On the other hand, we might target private installations, and there, the number of users joining the system is a little slower. But even in the latter case, we still want the number of deployments to go up, increasing the total number of people using our software. The number of users interacting with our frontend is the largest influencer of scale. With each user added, along with the various architectural perspectives, growth happens exponentially. If you look at it from a top-down point of view, users call the shots. At the end of the day, our application exists to serve them. The better we're able to scale our JavaScript code, the more users we'll please. Building new features Perhaps the most obvious side-effect of successful software with a strong user base is the features necessary to keep those users happy. The feature set grows along with the users of the system. This is often overlooked by projects, despite the obviousness of new features. We know they're coming, yet, little thought goes into how the endless stream of features going into our code impedes our ability to scale up our efforts. This is especially tricky when the software is in its infancy. The organization developing the software will bend over backwards to reel in new users. And there's little consequence of doing so in the beginning because the side-effects are limited. There's not a lot of mature features, there's not a huge development team, and there's less chance of annoying existing users by breaking something that they've come to rely on. When these factors aren't there, it's easier for us to nimbly crank out the features and dazzle existing/prospective users. But how do we force ourselves to be mindful of these early design decisions? How do we make sure that we don't unnecessarily limit our ability to scale the software up, in terms of supporting more features? New feature development, as well as enhancing existing features, is an ongoing issue with scalable JavaScript architecture. It's not just the number of features listed in the marketing literature of our software that we need to be concerned about . There's also the complexity of a given feature, how common our features are with one another, and how many moving parts each of these features has. If the user is the first level when looking at JavaScript architecture from a top-down perspective, each feature is the next level, and from there, it expands out into enormous complexity. It's not just the individual users who make a given feature complex. Instead, it's a group of users that all need the same feature in order to use our software effectively. And from there, we have to start thinking about personas, or roles, and which features are available for which roles. The need for this type of organizational structure isn't made apparent till much later on in the game; after we've made decisions that make it difficult to introduce role-based feature delivery. And depending on how our software is deployed, we may have to support a variety of unique use cases. For example, if we have several large organizations as our customers, each with their own deployments, they'll likely have their own unique constraints on how users are structured. This is challenging, and our architecture needs to support the disparate needs of many organizations, if we're going to scale. Hiring more developers Making these features a reality requires solid JavaScript developers who know what they're doing, and if we're lucky, we'll be able to hire a team of them. The team part doesn't happen automatically. There's a level of trust and respect that needs to be established before the team members begin to actively rely on one another to crank out some awesome code. Once that starts happening, we're in good shape. Turning once again to the top-down perspective of our scaling influencers, the features we deliver can directly impact the health of our team. There's a balance that's essentially impossible to maintain, but we can at least get close. Too many features and not enough developers lead to a sense of perpetual inadequacy among team members. When there's no chance of delivering what's expected, there's not much sense in trying. On the other hand, if you have too many developers, and there's too much communication overhead due to a limited number of features, it's tough to define responsibilities. When there's no shared understanding of responsibilities, things start to break down. It's actually easier to deal with not enough developers for the features we're trying to develop, than having too many developers. When there's a large burden of feature development, it's a good opportunity to step back and think—"what would we do differently if we had more developers?" This question usually gets skipped. We go hire more developers, and when they arrive, it's to everyone's surprise that there's no immediate improvement in feature throughput. This is why it's best to have an open development culture where there are no stupid questions, and where responsibilities are defined. There's no one correct team structure or development methodology. The development team needs to apply itself to the issues faced by the software we're trying to deliver. The biggest hurdle is for sure the number, size, and complexity of features. So that's something we need to consider when forming our team initially, as well as when growing the team. This latter point is especially true because the team structure we used way back when the software was new isn't going to fit what we face when the features scale up. Architectural perspectives The preceding section was a sampling of the factors that influence scale in JavaScript applications. Starting from the top, each of these influencers affects the influencer below it. The number and nature of our users is the first and foremost influencer, and this has a direct impact on the number and nature of the features we develop. Further more, the size of the development team, and the structure of that team, are influenced by these features. Our job is to take these influencers of scale, and translate them into factors to consider from an architectural perspective: Scaling influences the perspectives of our architecture. Our architecture, in turn, determines responses to scaling influencers. The process is iterative and never-ending throughout the lifetime of our software. The browser is a unique environment Scaling up in the traditional sense doesn't really work in a browser environment. When backend services are overwhelmed by demand, it's common to "throw more hardware" at the problem. Easier said than done of course, but it's a lot easier to scale up our data services these days, compared to 20 years ago. Today's software systems are designed with scalability in mind. It's helpful to our frontend application if the backend services are always available and always responsive, but that's just a small portion of the issues we face. We can't throw more hardware at the web browsers running our code; given that; the time and space complexities of our algorithms are important. Desktop applications generally have a set of system requirements for running the software, such as OS version, minimum memory, minimum CPU, and so on. If we were to advertise requirements such as these in our JavaScript applications, our user base would shrink dramatically, and possibly generate some hate mail. The expectation that browser-based web applications be lean and fast is an emergent phenomenon. Perhaps, that's due in part to the competition we face. There are a lot of bloated applications out there, and whether they're used in the browser or natively on the desktop, users know what bloat feels like, and generally run the other way: JavaScript applications require many resources, all of different types; these are all fetched by the browser, on the application's behalf. Adding to our trouble is the fact that we're using a platform that was designed as a means to download and display hypertext, to click on a link, and repeat. Now we're doing the same thing, except with full-sized applications. Multi-page applications are slowly being set aside in favor of single-page applications. That being said, the application is still treated as though it were a web page. Despite all that, we're in the midst of big changes. The browser is a fully viable web platform, the JavaScript language is maturing, and there are numerous W3C specifications in progress; they assist with treating our JavaScript more like an application and less like a document. Take a look at the following diagram: A sampling of the technologies found in the growing web platform We use architectural perspectives to assess any architectural design we come up with. It's a powerful technique to examine our design through a different lens. JavaScript architecture is no different, especially for those that scale. The difference between JavaScript architecture and architecture for other environments is that ours have unique perspectives. The browser environment requires that we think differently about how we design, build, and deploy applications. Anything that runs in the browser is transient by nature, and this changes software design practices that we've taken for granted over the years. Additionally, we spend more time coding our architectures than diagramming them. By the time we sketch anything out, it's been superseded by another specification or another tool. Component design At an architectural level, components are the main building blocks we work with. These may be very high-level components with several levels of abstraction. Or, they could be something exposed by a framework we're using, as many of these tools provide their own idea of "components". When we first set out to build a JavaScript application with scale in mind, the composition of our components began to take shape. How our components are composed is a huge limiting factor in how we scale, because they set the standard. Components implement patterns for the sake of consistency, and it's important to get those patterns right: Components have an internal structure. The complexity of this composition depends on the type of component under consideration As we'll see, the design of our various components is closely-tied to the trade-offs we make in other perspectives. And that's a good thing, because it means that if we're paying attention to the scalable qualities we're after, we can go back and adjust the design of our components in order to meet those qualities. Component communication Components don't sit in the browser on their own. Components communicate with one another all the time. There's a wide variety of communication techniques at our disposal here. Component communication could be as simple as method invocation, or as complex as an asynchronous publish-subscribe event system. The approach we take with our architecture depends on our more specific goals. The challenge with components is that we often don't know what the ideal communication mechanism will be, till after we've started implementing our application. We have to make sure that we can adjust the chosen communication path: The component communication mechanism decouples components, enabling scalable structures Seldom will we implement our own communication mechanism for our components. Not when so many tools exist, that solve at least part of the problem for us. Most likely, we'll end up with a concoction of an existing tool for communication and our own implementation specifics. What's important is that the component communication mechanism is its own perspective, which can be designed independently of the components themselves. Load time JavaScript applications are always loading something. The biggest challenge is the application itself, loading all the static resources it needs to run, before the user is allowed to do anything. Then there's the application data. This needs to be loaded at some point, often on demand, and contributes to the overall latency experienced by the user. Load time is an important perspective, because it hugely contributes to the overall perception of our product quality. The initial load is the user's first impression and this is where most components are initialized; it's tough to get the initial load to be fast without sacrificing performance in other areas There's lots we can do here to offset the negative user experience of waiting for things to load. This includes utilizing web specifications that allow us to treat applications and the services they use as installable components in the web browser platform. Of course, these are all nascent ideas, but worth considering as they mature alongside our application. Responsiveness The second part of the performance perspective of our architecture is concerned with responsiveness. That is, after everything has loaded, how long does it take for us to respond to user input? Although this is a separate problem from that of loading resources from the backend, they're still closely-related. Often, user actions trigger API requests, and the techniques we employ to handle these workflows impact user-perceived responsiveness. User-perceived responsiveness is affected by the time taken by our components to respond to DOM events; a lot can happen in between the initial DOM event and when we finally notify the user by updating the DOM. Because of this necessary API interaction, user-perceived responsiveness is important. While we can't make the API go any faster, we can take steps to ensure that the user always has feedback from the UI and that feedback is immediate. Then, there's the responsiveness of simply navigating around the UI, using cached data that's already been loaded, for example. Every other architectural perspective is closely-tied to the performance of our JavaScript code, and ultimately, to the user-perceived responsiveness. This perspective is a subtle sanity-check for the design of our components and their chosen communication paths. Addressability Just because we're building a single-page application doesn't mean we no longer care about addressable URIs. This is perhaps the crowning achievement of the web— unique identifiers that point to the resource we want. We paste them in to our browser address bar and watch the magic happen. Our application most certainly has addressable resources, we just point to them differently. Instead of a URI that's parsed by the backend web server, where the page is constructed and sent back to the browser, it's our local JavaScript code that understands the URI: Components listen to routers for route events and respond accordingly. A changing browser URI triggers these events. Typically, these URIs will map to an API resource. When the user hits one of these URIs in our application, we'll translate the URI into another URI that's used to request backend data. The component we use to manage these application URIs is called a router, and there's lots of frameworks and libraries with a base implementation of a router. We'll likely use one of these. The addressability perspective plays a major role in our architecture, because ensuring that the various aspects of our application have an addressable URI complicates our design. However, it can also make things easier if we're clever about it. We can have our components utilize the URIs in the same way a user utilizes links. Configurability Rarely does software do what you need it to straight out of the box. Highly-configurable software systems are touted as being good software systems. Configuration in the frontend is a challenge because there's several dimensions of configuration, not to mention the issue of where we store these configuration options. Default values for configurable components are problematic too—where do they come from? For example, is there a default language setting that's set until the user changes it? As is often the case, different deployments of our frontend will require different default values for these settings: Component configuration values can come from the backend server, or from the web browser. Defaults must reside somewhere Every configurable aspect of our software complicates its design. Not to mention the performance overhead and potential bugs. So, configurability is a large issue, and it's worth the time spent up-front discussing with various stakeholders what they value in terms of configurability. Depending on the nature of our deployment, users may value portability with their configuration. This means that their values need to be stored in the backend, under their account settings. Obviously decisions like these have backend design implications, and sometimes it's better to get away with approaches that don't require a modified backend service. Making architectural trade-offs There's a lot to consider from the various perspectives of our architecture, if we're going to build something that scales. We'll never get everything that we need out of every perspective simultaneously. This is why we make architectural trade-offs—we trade one aspect of our design for another more desirable aspect. Defining your constants Before we start making trade-offs, it's important to state explicitly what cannot be traded. What aspects of our design are so crucial to achieving scale that they must remain constant? For instance, a constant might be the number of entities rendered on a given page, or a maximum level of function call indirection. There shouldn't be a ton of these architectural constants, but they do exist. It's best if we keep them narrow in scope and limited in number. If we have too many strict design principles that cannot be violated or otherwise changed to fit our needs, we won't be able to easily adapt to changing influencers of scale. Does it make sense to have constant design principles that never change, given the unpredictability of scaling influencers? It does, but only once they emerge and are obvious. So this may not be an up-front principle, though we'll often have at least one or two up-front principles to follow. The discovery of these principles may result from the early refactoring of code or the later success of our software. In any case, the constants we use going forward must be made explicit and be agreed upon by all those involved. Performance for ease of development Performance bottlenecks need to be fixed, or avoided in the first place where possible. Some performance bottlenecks are obvious and have an observable impact on the user experience. These need to be fixed immediately, because it means our code isn't scaling for some reason, and might even point to a larger design issue. Other performance issues are relatively small. These are generally noticed by developers running benchmarks against code, trying by all means necessary to improve the performance. This doesn't scale well, because these smaller performance bottlenecks that aren't observable by the end user are time-consuming to fix. If our application is of a reasonable size, with more than a few developers working on it, we're not going to be able to keep up with feature development if everyone's fixing minor performance problems. These micro-optimizations introduce specialized solutions into our code, and they're not exactly easy reading for other developers. On the other hand, if we let these minor inefficiencies go, we will manage to keep our code cleaner and thus easier to work with. Where possible, trade off optimized performance for better code quality. This improves our ability to scale from a number of perspectives. Configurability for performance It's nice to have generic components where nearly every aspect is configurable. However, this approach to component design comes at a performance cost. It's not noticeable at first, when there are few components, but as our software scales in feature count, the number of components grows, and so does the number of configuration options. Depending on the size of each component (its complexity, number of configuration options, and so forth) the potential for performance degradation increases exponentially. Take a look at the following diagram: The component on the left has twice as many configuration options as the component on the right. It's also twice as difficult to use and maintain. We can keep our configuration options around as long as there're no performance issues affecting our users. Just keep in mind that we may have to remove certain options in an effort to remove performance bottlenecks. It's unlikely that configurability is going to be our main source of performance issues. It's also easy to get carried away as we scale and add features. We'll find, retrospectively, that we created configuration options at design time that we thought would be helpful, but turned out to be nothing but overhead. Trade off configurability for performance when there's no tangible benefit to having the configuration option. Performance for substitutability A related problem to that of configurability is substitutability. Our user interface performs well, but as our user base grows and more features are added, we discover that certain components cannot be easily substituted with another. This can be a developmental problem, where we want to design a new component to replace something pre-existing. Or perhaps we need to substitute components at runtime. Our ability to substitute components lies mostly with the component communication model. If the new component is able to send/receive messages/events the same as the existing component, then it's a fairly straightforward substitution. However, not all aspects of our software are substitutable. In the interest of performance, there may not even be a component to replace. As we scale, we may need to re-factor larger components into smaller components that are replaceable. By doing so, we're introducing a new level of indirection, and a performance hit. Trade off minor performance penalties to gain substitutability that aids in other aspects of scaling our architecture. Ease of development for addressability Assigning addressable URIs to resources in our application certainly makes implementing features more difficult. Do we actually need URIs for every resource exposed by our application? Probably not. For the sake of consistency though, it would make sense to have URIs for almost every resource. If we don't have a router and URI generation scheme that's consistent and easy to follow, we're more likely to skip implementing URIs for certain resources. It's almost always better to have the added burden of assigning URIs to every resource in our application than to skip out on URIs. Or worse still, not supporting addressable resources at all. URIs make our application behave like the rest of the Web; the training ground for all our users. For example, perhaps URI generation and routes are a constant for anything in our application—a trade-off that cannot happen. Trade off ease of development for addressability in almost every case. The ease of development problem with regard to URIs can be tackled in more depth as the software matures. Maintainability for performance The ease with which features are developed in our software boils down to the development team and it's scaling influencers. For example, we could face pressure to hire entry-level developers for budgetary reasons. How well this approach scales depends on our code. When we're concerned with performance, we're likely to introduce all kinds of intimidating code that relatively inexperienced developers will have trouble swallowing. Obviously, this impedes the ease of developing new features, and if it's difficult, it takes longer. This obviously does not scale with respect to customer demand. Developers don't always have to struggle with understanding the unorthodox approaches we've taken to tackle performance bottlenecks in specific areas of the code. We can certainly help the situation by writing quality code that's understandable. Maybe even documentation. But we won't get all of this for free; if we're to support the team as a whole as it scales, we need to pay the productivity penalty in the short term for having to coach and mentor. Trade off ease of development for performance in critical code paths that are heavily utilized and not modified often. We can't always escape the ugliness required for performance purposes, but if it's well-hidden, we'll gain the benefit of the more common code being comprehensible and self-explanatory. For example, low-level JavaScript libraries perform well and have a cohesive API that's easy to use. But if you look at some of the underlying code, it isn't pretty. That's our gain—having someone else maintain code that's ugly for performance reasons. Our components on the left follow coding styles that are consistent and easy to read; they all utilize the high-performance library on the right, giving our application performance while isolating optimized code that's difficult to read and understand. Less features for maintainability When all else fails, we need to take a step back and look holistically at the featureset of our application. Can our architecture support them all? Is there a better alternative? Scrapping an architecture that we've sunk many hours into almost never makes sense—but it does happen. The majority of the time, however, we'll be asked to introduce a challenging set of features that violate one or more of our architectural constants. When that happens, we're disrupting stable features that already exist, or we're introducing something of poor quality into the application. Neither case is good, and it's worth the time, the headache, and the cursing to work with the stakeholders to figure out what has to go. If we've taken the time to figure out our architecture by making trade-offs, we should have a sound argument for why our software can't support hundreds of features. When an architecture is full, we can't continue to scale. The key is understanding where that breaking threshold lies, so we can better understand and communicate it to stakeholders. Leveraging frameworks Frameworks exist to help us implement our architecture using a cohesive set of patterns. There's a lot of variety out there, and choosing which framework is a combination of personal taste, and fitness based on our design. For example, one JavaScript application framework will do a lot for us out-of-the-box, while another has even more features, but a lot of them we don't need. JavaScript application frameworks vary in size and sophistication. Some come with batteries included, and some tend toward mechanism over policy. None of these frameworks were specifically designed for our application. Any purported ability of a framework needs to be taken with a grain of salt. The features advertised by frameworks are applied to a general case, and a simple one at that. Applied in the context of our architecture is something else entirely. That being said, we can certainly use a given framework of our liking as input to the design process. If we really like the tool, and our team has experience using it, we can let it influence our design decisions. Just as long as we understand that the framework does not automatically respond to scaling influencers—that part is up to us. It's worth the time investigating the framework to use for our project because choosing the wrong framework is a costly mistake. The realization that we should have gone with something else usually comes after we've implemented lots of functionality. The end result is lots of re-writing, re-planning, re-training, and re-documenting. Not to mention the time lost on the first implementation. Choose your frameworks wisely, and be cautious about being framework-coupling. Summary Scaling a JavaScript application isn't the same as scaling other types of applications. Although we can use JavaScript to create large-scale backend services, our concern is with scaling the applications our users interact with in the browser. And there're a number of influencers that guide our decision making process on producing an architecture that scales. We reviewed some of these influencers, and how they flow in a top-down fashion, creating challenges unique to frontend JavaScript development. We examined the effect of more users, more features, and more developers; we can see that there's a lot to think about. While the browser is becoming a powerful platform, onto which we're delivering our applications, it still has constraints not found on other platforms. Designing and implementing a scalable JavaScript application requires having an architecture. What the software must ultimately do is just one input to that design. The scaling influencers are key as well. From there, we address different perspectives of the architecture under consideration. Things such as component composition and responsiveness come into play when we talk about scale. These are observable aspects of our architecture that are impacted by influencers of scale. As these scaling factors change over time, we use architectural perspectives as tools to modify our design, or the product to align with scaling challenges. Resources for Article: Further resources on this subject: Developing a JavaFX Application for iOS [article] Deploying a Play application on CoreOS and Docker [article] Developing Location-based Services with Neo4j [article]

In this article by, Suhas Chatekar, author of the book Learning NHibernate 4, we would dig deeper into that statement and try to understand what those downsides are and what can be done about them. In our attempt to address the downsides of repository, we would present two data access patterns, namely specification pattern and query object pattern. Specification pattern is a pattern adopted into data access layer from a general purpose pattern used for effectively filtering in-memory data. Before we begin, let me reiterate – repository pattern is not bad or wrong choice in every situation. If you are building a small and simple application involving a handful of entities then repository pattern can serve you well. But if you are building complex domain logic with intricate database interaction then repository may not do justice to your code. The patterns presented can be used in both simple and complex applications, and if you feel that repository is doing the job perfectly then there is no need to move away from it. (For more resources related to this topic, see here.) Problems with repository pattern A lot has been written all over the Internet about what is wrong with repository pattern. A simple Google search would give you lot of interesting articles to read and ponder about. We would spend some time trying to understand problems introduced by repository pattern. Generalization FindAll takes name of the employee as input along with some other parameters required for performing the search. When we started putting together a repository, we said that Repository<T> is a common repository class that can be used for any entity. But now FindAll takes a parameter that is only available on Employee, thus locking the implementation of FindAll to the Employee entity only. In order to keep the repository still reusable by other entities, we would need to part ways from the common Repository<T> class and implement a more specific EmployeeRepository class with Employee specific querying methods. This fixes the immediate problem but introduces another one. The new EmployeeRepository breaks the contract offered by IRepository<T> as the FindAll method cannot be pushed on the IRepository<T> interface. We would need to add a new interface IEmployeeRepository. Do you notice where this is going? You would end up implementing lot of repository classes with complex inheritance relationships between them. While this may seem to work, I have experienced that there are better ways of solving this problem. Unclear and confusing contract What happens if there is a need to query employees by a different criteria for a different business requirement? Say, we now need to fetch a single Employee instance by its employee number. Even if we ignore the above issue and be ready to add a repository class per entity, we would need to add a method that is specific to fetching the Employee instance matching the employee number. This adds another dimension to the code maintenance problem. Imagine how many such methods we would end up adding for a complex domain every time someone needs to query an entity using a new criteria. With several methods on repository contract that query same entity using different criteria makes the contract less clear and confusing for new developers. Such a pattern also makes it difficult to reuse code even if two methods are only slightly different from each other. Leaky abstraction In order to make methods on repositories reusable in different situations, lot of developers tend to add a single method on repository that does not take any input and return an IQueryable<T> by calling ISession.Query<T> inside it, as shown next: public IQueryable<T> FindAll() {    return session.Query<T>(); } IQueryable<T> returned by this method can then be used to construct any query that you want outside of repository. This is a classic case of leaky abstraction. Repository is supposed to abstract away any concerns around querying the database, but now what we are doing here is returning an IQueryable<T> to the consuming code and asking it to build the queries, thus leaking the abstraction that is supposed to be hidden into repository. IQueryable<T> returned by the preceding method holds an instance of ISession that would be used to ultimately interact with database. Since repository has no control over how and when this IQueryable would invoke database interaction, you might get in trouble. If you are using "session per request" kind of pattern then you are safeguarded against it but if you are not using that pattern for any reason then you need to watch out for errors due to closed or disposed session objects. God object anti-pattern A god object is an object that does too many things. Sometimes, there is a single class in an application that does everything. Such an implementation is almost always bad as it majorly breaks the famous single responsibility principle (SRP) and reduces testability and maintainability of code. A lot can be written about SRP and god object anti-pattern but since it is not the primary topic, I would leave the topic with underscoring the importance of staying away from god object anti-pattern. Avid readers can Google on the topic if they are interested. Repositories by nature tend to become single point of database interaction. Any new database interaction goes through repository. Over time, repositories grow organically with large number of methods doing too many things. You may spot the anti-pattern and decide to break the repository into multiple small repositories but the original single repository would be tightly integrated with your code in so many places that splitting it would be a difficult job. For a contained and trivial domain model, repository pattern can be a good choice. So do not abandon repositories entirely. It is around complex and changing domain that repositories start exhibiting the problems just discussed. You might still argue that repository is an unneeded abstraction and we can very well use NHibernate directly for a trivial domain model. But I would caution against any design that uses NHibernate directly from domain or domain services layer. No matter what design I use for data access, I would always adhere to "explicitly declare capabilities required" principle. The abstraction that offers required capability can be a repository interface or some other abstractions that we would learn. Specification pattern Specification pattern is a reusable and object-oriented way of applying business rules on domain entities. The primary use of specification pattern is to select subset of entities from a larger collection of entities based on some rules. An important characteristic of specification pattern is combining multiple rules by chaining them together. Specification pattern was in existence before ORMs and other data access patterns had set their feet in the development community. The original form of specification pattern dealt with in-memory collections of entities. The pattern was then adopted to work with ORMs such as NHibernate as people started seeing the benefits that specification pattern could bring about. We would first discuss specification pattern in its original form. That would give us a good understanding of the pattern. We would then modify the implementation to make it fit with NHibernate. Specification pattern in its original form Let's look into an example of specification pattern in its original form. A specification defines a rule that must be satisfied by domain objects. This can be generalized using an interface definition, as follows: public interface ISpecification<T> { bool IsSatisfiedBy(T entity); } ISpecification<T> defines a single method IsSatisifedBy. This method takes the entity instance of type T as input and returns a Boolean value depending on whether the entity passed satisfies the rule or not. If we were to write a rule for employees living in London then we can implement a specification as follows: public class EmployeesLivingIn : ISpecification<Employee> { public bool IsSatisfiedBy(Employee entity) {    return entity.ResidentialAddress.City == "London"; } } The EmployeesLivingIn class implements ISpecification<Employee> telling us that this is a specification for the Employee entity. This specification compares the city from the employee's ResidentialAddress property with literal string "London" and returns true if it matches. You may be wondering why I have named this class as EmployeesLivingIn. Well, I had some refactoring in mind and I wanted to make my final code read nicely. Let's see what I mean. We have hardcoded literal string "London" in the preceding specification. This effectively stops this class from being reusable. What if we need a specification for all employees living in Paris? Ideal thing to do would be to accept "London" as a parameter during instantiation of this class and then use that parameter value in the implementation of the IsSatisfiedBy method. Following code listing shows the modified code: public class EmployeesLivingIn : ISpecification<Employee> { private readonly string city;   public EmployeesLivingIn(string city) {    this.city = city; }   public bool IsSatisfiedBy(Employee entity) {    return entity.ResidentialAddress.City == city; } } This looks good without any hardcoded string literals. Now if I wanted my original specification for employees living in London then following is how I could build it: var specification = new EmployeesLivingIn("London"); Did you notice how the preceding code reads in plain English because of the way class is named? Now, let's see how to use this specification class. Usual scenario where specifications are used is when you have got a list of entities that you are working with and you want to run a rule and find out which of the entities in the list satisfy that rule. Following code listing shows a very simple use of the specification we just implemented: List<Employee> employees = //Loaded from somewhere List<Employee> employeesLivingInLondon = new List<Employee>(); var specification = new EmployeesLivingIn("London");   foreach(var employee in employees) { if(specification.IsSatisfiedBy(employee)) {    employeesLivingInLondon.Add(employee); } } We have a list of employees loaded from somewhere and we want to filter this list and get another list comprising of employees living in London. Till this point, the only benefit we have had from specification pattern is that we have managed to encapsulate the rule into a specification class which can be reused anywhere now. For complex rules, this can be very useful. But for simple rules, specification pattern may look like lot of plumbing code unless we overlook the composability of specifications. Most power of specification pattern comes from ability to chain multiple rules together to form a complex rule. Let's write another specification for employees who have opted for any benefit: public class EmployeesHavingOptedForBenefits : ISpecification<Employee> { public bool IsSatisfiedBy(Employee entity) {    return entity.Benefits.Count > 0; } } In this rule, there is no need to supply any literal value from outside so the implementation is quite simple. We just check if the Benefits collection on the passed employee instance has count greater than zero. You can use this specification in exactly the same way as earlier specification was used. Now if there is a need to apply both of these specifications to an employee collection, then very little modification to our code is needed. Let's start with adding an And method to the ISpecification<T> interface, as shown next: public interface ISpecification<T> { bool IsSatisfiedBy(T entity); ISpecification<T> And(ISpecification<T> specification); } The And method accepts an instance of ISpecification<T> and returns another instance of the same type. As you would have guessed, the specification that is returned from the And method would effectively perform a logical AND operation between the specification on which the And method is invoked and specification that is passed into the And method. The actual implementation of the And method comes down to calling the IsSatisfiedBy method on both the specification objects and logically ANDing their results. Since this logic does not change from specification to specification, we can introduce a base class that implements this logic. All specification implementations can then derive from this new base class. Following is the code for the base class: public abstract class Specification<T> : ISpecification<T> { public abstract bool IsSatisfiedBy(T entity);   public ISpecification<T> And(ISpecification<T> specification) {    return new AndSpecification<T>(this, specification); } } We have marked Specification<T> as abstract as this class does not represent any meaningful business specification and hence we do not want anyone to inadvertently use this class directly. Accordingly, the IsSatisfiedBy method is marked abstract as well. In the implementation of the And method, we are instantiating a new class AndSepcification. This class takes two specification objects as inputs. We pass the current instance and one that is passed to the And method. The definition of AndSpecification is very simple. public class AndSpecification<T> : Specification<T> { private readonly Specification<T> specification1; private readonly ISpecification<T> specification2;   public AndSpecification(Specification<T> specification1, ISpecification<T> specification2) {    this.specification1 = specification1;    this.specification2 = specification2; }   public override bool IsSatisfiedBy(T entity) {    return specification1.IsSatisfiedBy(entity) &&    specification2.IsSatisfiedBy(entity); } } AndSpecification<T> inherits from abstract class Specification<T> which is obvious. IsSatisfiedBy is simply performing a logical AND operation on the outputs of the ISatisfiedBy method on each of the specification objects passed into AndSpecification<T>. After we change our previous two business specification implementations to inherit from abstract class Specification<T> instead of interface ISpecification<T>, following is how we can chain two specifications using the And method that we just introduced: List<Employee> employees = null; //= Load from somewhere List<Employee> employeesLivingInLondon = new List<Employee>(); var specification = new EmployeesLivingIn("London")                                     .And(new EmployeesHavingOptedForBenefits());   foreach (var employee in employees) { if (specification.IsSatisfiedBy(employee)) {    employeesLivingInLondon.Add(employee); } } There is literally nothing changed in how the specification is used in business logic. The only thing that is changed is construction and chaining together of two specifications as depicted in bold previously. We can go on and implement other chaining methods but point to take home here is composability that the specification pattern offers. Now let's look into how specification pattern sits beside NHibernate and helps in fixing some of pain points of repository pattern. Specification pattern for NHibernate Fundamental difference between original specification pattern and the pattern applied to NHibernate is that we had an in-memory list of objects to work with in the former case. In case of NHibernate we do not have the list of objects in the memory. We have got the list in the database and we want to be able to specify rules that can be used to generate appropriate SQL to fetch the records from database that satisfy the rule. Owing to this difference, we cannot use the original specification pattern as is when we are working with NHibernate. Let me show you what this means when it comes to writing code that makes use of specification pattern. A query, in its most basic form, to retrieve all employees living in London would look something as follows: var employees = session.Query<Employee>()                .Where(e => e.ResidentialAddress.City == "London"); The lambda expression passed to the Where method is our rule. We want all the Employee instances from database that satisfy this rule. We want to be able to push this rule behind some kind of abstraction such as ISpecification<T> so that this rule can be reused. We would need a method on ISpecification<T> that does not take any input (there are no entities in-memory to pass) and returns a lambda expression that can be passed into the Where method. Following is how that method could look: public interface ISpecification<T> where T : EntityBase<T> { Expression<Func<T, bool>> IsSatisfied(); } Note the differences from the previous version. We have changed the method name from IsSatisfiedBy to IsSatisfied as there is no entity being passed into this method that would warrant use of word By in the end. This method returns an Expression<Fund<T, bool>>. If you have dealt with situations where you pass lambda expressions around then you know what this type means. If you are new to expression trees, let me give you a brief explanation. Func<T, bool> is a usual function pointer. This pointer specifically points to a function that takes an instance of type T as input and returns a Boolean output. Expression<Func<T, bool>> takes this function pointer and converts it into a lambda expression. An implementation of this new interface would make things more clear. Next code listing shows the specification for employees living in London written against the new contract: public class EmployeesLivingIn : ISpecification<Employee> { private readonly string city;   public EmployeesLivingIn(string city) {    this.city = city; }   public override Expression<Func<Employee, bool>> IsSatisfied() {    return e => e.ResidentialAddress.City == city; } } There is not much changed here compared to the previous implementation. Definition of IsSatisfied now returns a lambda expression instead of a bool. This lambda is exactly same as the one we used in the ISession example. If I had to rewrite that example using the preceding specification then following is how that would look: var specification = new EmployeeLivingIn("London"); var employees = session.Query<Employee>()                .Where(specification.IsSatisfied()); We now have a specification wrapped in a reusable object that we can send straight to NHibernate's ISession interface. Now let's think about how we can use this from within domain services where we used repositories before. We do not want to reference ISession or any other NHibernate type from domain services as that would break onion architecture. We have two options. We can declare a new capability that can take a specification and execute it against the ISession interface. We can then make domain service classes take a dependency on this new capability. Or we can use the existing IRepository capability and add a method on it which takes the specification and executes it. We started this article with a statement that repositories have a downside, specifically when it comes to querying entities using different criteria. But now we are considering an option to enrich the repositories with specifications. Is that contradictory? Remember that one of the problems with repository was that every time there is a new criterion to query an entity, we needed a new method on repository. Specification pattern fixes that problem. Specification pattern has taken the criterion out of the repository and moved it into its own class so we only ever need a single method on repository that takes in ISpecification<T> and execute it. So using repository is not as bad as it sounds. Following is how the new method on repository interface would look: public interface IRepository<T> where T : EntityBase<T> { void Save(T entity); void Update(int id, Employee employee); T GetById(int id); IEnumerable<T> Apply(ISpecification<T> specification); } The Apply method in bold is the new method that works with specification now. Note that we have removed all other methods that ran various different queries and replaced them with this new method. Methods to save and update the entities are still there. Even the method GetById is there as the mechanism used to get entity by ID is not same as the one used by specifications. So we retain that method. One thing I have experimented with in some projects is to split read operations from write operations. The IRepository interface represents something that is capable of both reading from the database and writing to database. Sometimes, we only need a capability to read from database, in which case, IRepository looks like an unnecessarily heavy object with capabilities we do not need. In such a situation, declaring a new capability to execute specification makes more sense. I would leave the actual code for this as a self-exercise for our readers. Specification chaining In the original implementation of specification pattern, chaining was simply a matter of carrying out logical AND between the outputs of the IsSatisfiedBy method on the specification objects involved in chaining. In case of NHibernate adopted version of specification pattern, the end result boils down to the same but actual implementation is slightly more complex than just ANDing the results. Similar to original specification pattern, we would need an abstract base class Specification<T> and a specialized AndSepcificatin<T> class. I would just skip these details. Let's go straight into the implementation of the IsSatisifed method on AndSpecification where actual logical ANDing happens. public override Expression<Func<T, bool>> IsSatisfied() { var p = Expression.Parameter(typeof(T), "arg1"); return Expression.Lambda<Func<T, bool>>(Expression.AndAlso(          Expression.Invoke(specification1.IsSatisfied(), p),          Expression.Invoke(specification2.IsSatisfied(), p)), p); } Logical ANDing of two lambda expression is not a straightforward operation. We need to make use of static methods available on helper class System.Linq.Expressions.Expression. Let's try to go from inside out. That way it is easier to understand what is happening here. Following is the reproduction of innermost call to the Expression class: Expression.Invoke(specification1.IsSatisfied(), parameterName) In the preceding code, we are calling the Invoke method on the Expression class by passing the output of the IsSatisfied method on the first specification. Second parameter passed to this method is a temporary parameter of type T that we created to satisfy the method signature of Invoke. The Invoke method returns an InvocationExpression which represents the invocation of the lambda expression that was used to construct it. Note that actual lambda expression is not invoked yet. We do the same with second specification in question. Outputs of both these operations are then passed into another method on the Expression class as follows: Expression.AndAlso( Expression.Invoke(specification1.IsSatisfied(), parameterName), Expression.Invoke(specification2.IsSatisfied(), parameterName) ) Expression.AndAlso takes the output from both specification objects in the form of InvocationExpression type and builds a special type called BinaryExpression which represents a logical AND between the two expressions that were passed to it. Next we convert this BinaryExpression into an Expression<Func<T, bool>> by passing it to the Expression.Lambda<Func<T, bool>> method. This explanation is not very easy to follow and if you have never used, built, or modified lambda expressions programmatically like this before, then you would find it very hard to follow. In that case, I would recommend not bothering yourself too much with this. Following code snippet shows how logical ORing of two specifications can be implemented. Note that the code snippet only shows the implementation of the IsSatisfied method. public override Expression<Func<T, bool>> IsSatisfied() { var parameterName = Expression.Parameter(typeof(T), "arg1"); return Expression.Lambda<Func<T, bool>>(Expression.OrElse( Expression.Invoke(specification1.IsSatisfied(), parameterName), Expression.Invoke(specification2.IsSatisfied(), parameterName)), parameterName); } Rest of the infrastructure around chaining is exactly same as the one presented during discussion of original specification pattern. I have avoided giving full class definitions here to save space but you can download the code to look at complete implementation. That brings us to end of specification pattern. Though specification pattern is a great leap forward from where repository left us, it does have some limitations of its own. Next, we would look into what these limitations are. Limitations Specification pattern is great and unlike repository pattern, I am not going to tell you that it has some downsides and you should try to avoid it. You should not. You should absolutely use it wherever it fits. I would only like to highlight two limitations of specification pattern. Specification pattern only works with lambda expressions. You cannot use LINQ syntax. There may be times when you would prefer LINQ syntax over lambda expressions. One such situation is when you want to go for theta joins which are not possible with lambda expressions. Another situation is when lambda expressions do not generate optimal SQL. I will show you a quick example to understand this better. Suppose we want to write a specification for employees who have opted for season ticket loan benefit. Following code listing shows how that specification could be written: public class EmployeeHavingTakenSeasonTicketLoanSepcification :Specification<Employee> { public override Expression<Func<Employee, bool>> IsSatisfied() {    return e => e.Benefits.Any(b => b is SeasonTicketLoan); } } It is a very simple specification. Note the use of Any to iterate over the Benefits collection to check if any of the Benefit in that collection is of type SeasonTicketLoan. Following SQL is generated when the preceding specification is run: SELECT employee0_.Id           AS Id0_,        employee0_.Firstname     AS Firstname0_,        employee0_.Lastname     AS Lastname0_,        employee0_.EmailAddress AS EmailAdd5_0_,        employee0_.DateOfBirth   AS DateOfBi6_0_,        employee0_.DateOfJoining AS DateOfJo7_0_,        employee0_.IsAdmin       AS IsAdmin0_,        employee0_.Password     AS Password0_ FROM   Employee employee0_ WHERE EXISTS (SELECT benefits1_.Id FROM   Benefit benefits1_ LEFT OUTER JOIN Leave benefits1_1_ ON benefits1_.Id = benefits1_1_.Id LEFT OUTER JOIN SkillsEnhancementAllowance benefits1_2_ ON benefits1_.Id = benefits1_2_.Id LEFT OUTER JOIN SeasonTicketLoan benefits1_3_ ON benefits1_.Id = benefits1_3_.Id WHERE employee0_.Id = benefits1_.Employee_Id AND CASE WHEN benefits1_1_.Id IS NOT NULL THEN 1      WHEN benefits1_2_.Id IS NOT NULL THEN 2      WHEN benefits1_3_.Id IS NOT NULL THEN 3       WHEN benefits1_.Id IS NOT NULL THEN 0      END = 3) Isn't that SQL too complex? It is not only complex on your eyes but this is not how I would have written the needed SQL in absence of NHibernate. I would have just inner-joined the Employee, Benefit, and SeasonTicketLoan tables to get the records I need. On large databases, the preceding query may be too slow. There are some other such situations where queries written using lambda expressions tend to generate complex or not so optimal SQL. If we use LINQ syntax instead of lambda expressions, then we can get NHibernate to generate just the SQL. Unfortunately, there is no way of fixing this with specification pattern. Summary Repository pattern has been around for long time but suffers through some issues. General nature of its implementation comes in the way of extending repository pattern to use it with complex domain models involving large number of entities. Repository contract can be limiting and confusing when there is a need to write complex and very specific queries. Trying to fix these issues with repositories may result in leaky abstraction which can bite us later. Moreover, repositories maintained with less care have a tendency to grow into god objects and maintaining them beyond that point becomes a challenge. Specification pattern and query object pattern solve these issues on the read side of the things. Different applications have different data access requirements. Some applications are write-heavy while others are read-heavy. But there are a minute number of applications that fit into former category. A large number of applications developed these days are read-heavy. I have worked on applications that involved more than 90 percent database operations that queried data and only less than 10 percent operations that actually inserted/updated data into database. Having this knowledge about the application you are developing can be very useful in determining how you are going to design your data access layer. That brings use to the end of our NHibernate journey. Not quite, but yes, in a way. Resources for Article: Further resources on this subject: NHibernate 3: Creating a Sample Application [article] NHibernate 3.0: Using LINQ Specifications in the data access layer [article] NHibernate 2: Mapping relationships and Fluent Mapping [article]

In this article by Victor Wu and Eagle Huang, authors of the book, Mastering VMware vSphere Storage, we will learn that, SAN storage is a key component of a VMware vSphere environment. We can choose different vendors and types of SAN storage to deploy on a VMware Sphere environment. The advanced settings of each storage can affect the performance of the virtual machine, for example, FC or iSCSI SAN storage. It has a different configuration in a VMware vSphere environment. Host connectivity of Fibre Channel storage is accessed by Host Bus Adapter (HBA). Host connectivity of iSCSI storage is accessed by the TCP/IP networking protocol. We first need to know the concept of storage. Then we can optimize the performance of storage in a VMware vSphere environment. In this article, you will learn these topics: What the vSphere storage APIs for Array Integration (VAAI) and Storage Awareness (VASA) are The virtual machine storage profile VMware vSphere Storage DRS and VMware vSphere Storage I/O Control (For more resources related to this topic, see here.) vSphere storage APIs for array integration and storage awareness VMware vMotion is a key feature in vSphere hosts. An ESXi host cannot provide the vMotion feature if it is without shared SAN storage. SAN storage is a key component in a VMware vSphere environment. In large-scale virtualization environments, there are many virtual machines stored in SAN storage. When a VMware administrator executes virtual machine cloning or migrates a virtual machine to another ESXi host by vMotion, this operation allocates the resource on that ESXi host and SAN storage. In vSphere 4.1 and later versions, it can support VAAI. The vSphere storage API is used by a storage vendor who provides hardware acceleration or offloads vSphere I/O between storage devices. These APIs can reduce the resource overhead on ESXi hosts and improve performance for ESXi host operations, for example, vMotion, virtual machine cloning, creating a virtual machine, and so on. VAAI has two APIs: the hardware acceleration API and the array thin provisioning API. The hardware acceleration API is used to integrate with VMware vSphere to offload storage operations to the array and reduce the CPU overload on the ESXi host. The following table lists the features of the hardware acceleration API for block and NAS: Array integration Features Description Block Fully copy This blocks clone or copy offloading. Block zeroing This is also called write same. When you provision an eagerzeroedthick VMDK, the SCSI command is issued to write zeroes to disks. Atomic Test & Set (ATS) This is a lock mechanism that prevents the other ESXi host from updating the same VMFS metadata. NAS Full file clone This is similar to Extended Copy (XCOPY) hardware acceleration. Extended statistics This feature is enabled in space usage in the NAS data store. Reserved space The allocated space of virtual disk in thick format. The array thin provisioning API is used to monitor the ESXi data store space on the storage arrays. It helps prevent the disk from running out of space and reclaims disk space. For example, if the storage is assigned as 1 x 3 TB LUN in the ESXi host, but the storage can only provide 2 TB of data storage space, it is considered to be 3 TB in the ESXi host. Streamline its monitoring LUN configuration space in order to avoid running out of physical space. When vSphere administrators delete or remove files from the data store that is provisioned LUN, the storage can reclaim free space in the block level. In vSphere 4.1 or later, it can support VAAI features. In vSphere 5.5, you can reclaim the space on thin provisioned LUN using esxcli. VMware VASA is a piece of software that allows the storage vendor to provide information about their storage array to VMware vCenter Server. The information includes storage capability, the state of physical storage devices, and so on. vCenter Server collects this information from the storage array using a software component called VASA provider, which is provided by the storage array vendor. A VMware administrator can view the information in VMware vSphere Client / VMware vSphere Web Client. The following diagram shows the architecture of VASA with vCenter Server. For example, the VMware administrator requests to create a 1 x data store in VMware ESXi Server. It has three main components: the storage array, the storage provider and VMware vCenter Server. The following is the procedure to add the storage provider to vCenter Server: Log in to vCenter by vSphere Client. Go to Home | Storage Providers. Click on the Add button. Input information about the storage vendor name, URL, and credentials. Virtual machine storage profile The storage provider can help the vSphere administrator know the state of the physical storage devices and the capabilities on which their virtual machines are located. It also helps choose the correct storage in terms of performance and space by using virtual machine storage policies. A virtual machine storage policy helps you ensure that a virtual machine guarantees a specified level of performance or capacity of storage, for example, the SSD/SAS/NL-SAS data store, spindle I/O, and redundancy. Before you define a storage policy, you need to specify the storage requirement for your application that runs on the virtual machine. It has two types of storage requirement, which is storage-vendor-specific storage capability and user-defined storage capability. Storage-vendor-specific storage capability comes from the storage array. The storage vendor provider informs vCenter Server that it can guarantee the use of storage features by using storage-vendor-specific storage capability. vCenter Server assigns vendor-specific storage capability to each ESXi data store. User-defined storage capability is the one that you can define and assign storage profile to each ESXi datastore. In vSphere 5.1/5.5, the name of the storage policy is VM storage profile. Virtual machine storage policies can include one or more storage capabilities and assign to one or more VM. The virtual machine can be checked for storage compliance if it is placed on compliant storage. When you migrate, create, or clone a virtual machine, you can select the storage policy and apply it to that machine. The following procedure shows how to create a storage policy and apply it to a virtual machine in vSphere 5.1 using user-defined storage capability: The vSphere ESXi host requires the license edition of Enterprise Plus to enable the VM storage profile feature. The following procedure is adding the storage profile into vCenter Server: Log in to vCenter Server using vSphere Client. Click on the Home button in the top bar, and choose the VM Storage Profiles button under Management. Click on the Manage Storage Capabilities button to create user-defined storage capability. Click on the Add button to create the name of the storage capacity, for example, SSD Storage, SAS Storage, or NL-SAS Storage. Then click on the Close button. Click on the Create VM Storage Profile button to create the storage policy. Input the name of the VM storage profile, as shown in the following screenshot, and then click on the Next button to select the user-defined storage capability, which is defined in step 4. Click on the Finish button. Assign the user-defined storage capability to your specified ESXi data store. Right-click on the data store that you plan to assign the user-defined storage capability to. This capability is defined in step 4. After creating the VM storage profile, click on the Enable VM Storage Profiles button. Then click on the Enable button to enable the profiles. The following screenshot shows Enable VM Storage Profiles: After enabling the VM storage profile, you can see VM Storage Profile Status as Enabled and Licensing Status as Licensed, as shown in this screenshot: We have successfully created the VM storage profile. Now we have to associate the VM storage profile with a virtual machine. Right-click on a virtual machine that you plan to apply to the VM storage profile, choose VM Storage Profile, and then choose Manage Profiles. From the drop-down menu of VM Storage Profile select your profile. Then you can click on the Propagate to disks button to associate all virtual disks or decide which virtual disks you want to associate with that profile by setting manually. Click on OK. Finally, you need to check the compliance of VM Storage Profile on this virtual machine. Click on the Home button in the top bar. Then choose the VM Storage Profiles button under Management. Go to Virtual Machines and click on the Check Compliance Now button. The Compliance Status will display Compliant after compliance checking, as follows: Pluggable Storage Architecture (PSA) exists in the SCSI middle layer of the VMkernel storage stack. PSA is used to allow thirty-party storage vendors to use their failover and load balancing techniques for their specific storage array. A VMware ESXi host uses its multipathing plugin to control the ownership of the device path and LUN. The VMware default Multipathing Plugin (MPP) is called VMware Native Multipathing Plugin (NMP), which includes two subplugins as components: Storage Array Type Plugin (SATP) and Path Selection Plugin (PSP). SATP is used to handle path failover for a storage array, and PSP is used to issue an I/O request to a storage array. The following diagram shows the architecture of PSA: This table lists the operation tasks of PSA and NMP in the ESXi host:   PSA NMP Operation tasks Discovers the physical paths Manages the physical path Handles I/O requests to the physical HBA adapter and logical devices Creates, registers, and deregisters logical devices Uses predefined claim rules to control storage devices Selects an optimal physical path for the request The following is an example of operation of PSA in a VMkernel storage stack: The virtual machine sends out an I/O request to a logical device that is managed by the VMware NMP. The NMP requests the PSP to assign to this logical device. The PSP selects a suitable physical path to send the I/O request. When the I/O operation is completed successfully, the NMP reports that the I/O operation is complete. If the I/O operation reports an error, the NMP calls the SATP. The SATP fails over to the new active path. The PSP selects a new active path from all available paths and continues the I/O operation. The following diagram shows the operation of PSA: VMware vSphere provides three options for the path selection policy. These are Most Recently Used (MRU), Fixed, and Round Robin (RR). The following table lists the advantages and disadvantages of each path: Path selection Description Advantage Disadvantage MRU The ESXi host selects the first preferred path at system boot time. If this path becomes unavailable, the ESXi host changes to the other active path. You can select your preferred path manually in the ESXi host. The ESXi host does not revert to the original path when that l path becomes available again. Fixed You can select the preferred path manually. The ESXi host can revert to the original path when the preferred path becomes available again. If the ESXi host cannot select the preferred path, it selects an available preferred path randomly. RR The ESXi host uses automatic path selection. The storage I/O across all available paths and enable load balancing across all paths. The storage is required to support ALUA mode. You cannot know which path is preferred because the storage I/O across all available paths. The following is the procedure of changing the path selection policy in an ESXi host: Log in to vCenter Server using vSphere Client. Go to the configuration of your selected ESXi host, choose the data store that you want to configure, and click on the Properties… button. Click on the Manage Paths… button. Select the drop-down menu and click on the Change button. If you plan to deploy a third-party MPP on your ESXi host, you need to follow up the storage vendor's instructions for the installation, for example, EMC PowerPath/VE for VMware that it is a piece of path management software for VMware's vSphere server and Microsoft's Hyper-V server. It also can provide load balancing and path failover features. VMware vSphere Storage DRS VMware vSphere Storage DRS (SDRS) is the placement of virtual machines in an ESX's data store cluster. According to storage capacity and I/O latency, it is used by VMware storage vMotion to migrate the virtual machine to keep the ESX's data store in a balanced status that is used to aggregate storage resources, and enable the placement of the virtual disk (VMDK) of virtual machine and load balancing of existing workloads. What is a data store cluster? It is a collection of ESXi's data stores grouped together. The data store cluster is enabled for vSphere SDRS. SDRS can work in two modes: manual mode and fully automated mode. If you enable SDRS in your environment, when the vSphere administrator creates or migrates a virtual machine, SDRS places all the files (VMDK) of this virtual machine in the same data store or different a data store in the cluster, according to the SDRS affinity rules or anti-affinity rules. The VMware ESXi host cluster has two key features: VMware vSphere High Availability (HA) and VMware vSphere Distributed Resource Scheduler (DRS). SDRS is different from the host cluster DRS. The latter is used to balance the virtual machine across the ESXi host based on the memory and CPU usage. SDRS is used to balance the virtual machine across the SAN storage (ESX's data store) based on the storage capacity and IOPS. The following table lists the difference between SDRS affinity rules and anti-affinity rules: Name of SDRS rules Description VMDK affinity rules This is the default SDRS rule for all virtual machines. It keeps each virtual machine's VMDKs together on the same ESXi data store. VMDK anti-affinity rules Keep each virtual machine's VMDKs on different ESXi data stores. You can apply this rule into all virtual machine's VMDKs or to dedicated virtual machine's VMDKs. VM anti-affinity rules Keep the virtual machine on different ESXi data stores. This rule is similar to the ESX DRS anti-affinity rules. The following is the procedure to create a storage DRS in vSphere 5: Log in to vCenter Server using vSphere Client. Go to home and click on the Datastores and Datastore Clusters button. Right-click on the data center and choose New Datastore Cluster. Input the name of the SDRS and then click on the Next button. Choose Storage DRS mode, Manual Mode and Fully Automated Mode. Manual Mode: According to the placement and migration recommendation, the placement and migration of the virtual machine are executed manually by the user.Fully Automated Mode: Based on the runtime rules, the placement of the virtual machine is executed automatically. Set up SDRS Runtime Rules. Then click on the Next button. Enable I/O metric for SDRS recommendations is used to enable I/O load balancing. Utilized Space is the percentage of consumed space allowed before the storage DRS executes an action. I/O Latency is the percentage of consumed latency allowed before the storage DRS executes an action. This setting can execute only if the Enable I/O metric for SDRS recommendations checkbox is selected. No recommendations until utilization difference between source and destination is is used to configure the space utilization difference threshold. I/O imbalance threshold is used to define the aggressive of IOPs load balancing. This setting can execute only if the Enable I/O metric for SDRS recommendations checkbox is selected. Select the ESXi host that is required to create SDRS. Then click on the Next button. Select the data store that is required to join the data store cluster, and click on the Next button to complete. After creating SDRS, go to the vSphere Storage DRS panel on the Summary tab of the data store cluster. You can see that Storage DRS is Enabled. On the Storage DRS tab on the data store cluster, it displays the recommendation, placement, and reasons. Click on the Apply Recommendations button if you want to apply the recommendations. Click on the Run Storage DRS button if you want to refresh the recommendations. VMware vSphere Storage I/O Control What is VMware vSphere Storage I/O Control? It is used to control in order to share and limit the storage of I/O resources, for example, the IOPS. You can control the number of storage IOPs allocated to the virtual machine. If a certain virtual machine is required to get more storage I/O resources, vSphere Storage I/O Control can ensure that that virtual machine can get more storage I/O than other virtual machines. The following table shows example of the difference between vSphere Storage I/O Control enabled and without vSphere Storage I/O Control: In this diagram, the VMware ESXi Host Cluster does not have vSphere Storage I/O Control. VM 2 and VM 5 need to get more IOPs, but they can allocate only a small amount of I/O resources. On the contrary, VM 1 and VM 3 can allocate a large amount of I/O resources. Actually, both VMs are required to allocate a small amount of IOPs. In this case, it wastes and overprovisions the storage resources. In the diagram to the left, vSphere Storage I/O Control is enabled in the ESXi Host Cluster. VM 2 and VM 5 are required to get more IOPs. They can allocate a large amount of I/O resources after storage I/O control is enabled. VM 1, VM 3, and VM 4 are required to get a small amount of I/O resources, and now these three VMs allocate a small amount of IOPs. After enabling storage I/O control, it helps reduce waste and overprovisioning of the storage resources. When you enable VMware vSphere Storage DRS, vSphere Storage I/O Control is automatically enabled on the data stores in the data store cluster. The following is the procedure to be carried out to enable vSphere Storage I/O control on an ESXi data store, and set up storage I/O shares and limits using vSphere Client 5: Log in to vCenter Server using vSphere Client. Go to the Configuration tab of the ESXi host, select the data store, and then click on the Properties… button. Select Enabled under Storage I/O Control, and click on the Close button. After Storage I/O Control is enabled, you can set up the storage I/O shares and limits on the virtual machine. Right-click on the virtual machine and select Edit Settings. Click on the Resources tab in the virtual machine properties box, and select Disk. You can individually set each virtual disk's Shares and Limit field. By default, all virtual machine shares are set to Normal and with Unlimited IOPs. Summary In this article, you learned what VAAI and VASA are. In a vSphere environment, the vSphere administrator learned how to configure the storage profile in vCenter Server and assign to the ESXi data store. We covered the benefits of vSphere Storage I/O Control and vSphere Storage DRS. When you found that it has a storage performance problem in the vSphere host, we saw how to troubleshoot the performance problem, and found out the root cause. Resources for Article: Further resources on this subject: Essentials of VMware vSphere [Article] Introduction to vSphere Distributed switches [Article] Network Virtualization and vSphere [Article]

In this article, by Aurobindo Sarkar and Amit Shah, author of the book Learning AWS, we will introduce some key design principles and approaches to achieving high availability in your applications deployed on the AWS cloud. As a good practice, you want to ensure that your mission-critical applications are always available to serve your customers. The approaches in this article will address availability across the layers of your application architecture including availability aspects of key infrastructural components, ensuring there are no single points of failure. In order to address availability requirements, we will use the AWS infrastructure (Availability Zones and Regions), AWS Foundation Services (EC2 instances, Storage, Security and Access Control, Networking), and the AWS PaaS services (DynamoDB, RDS, CloudFormation, and so on). (For more resources related to this topic, see here.) Defining availability objectives Achieving high availability can be costly. Therefore, it is important to ensure that you align your application availability requirements with your business objectives. There are several options to achieve the level of availability that is right for your application. Hence, it is essential to start with a clearly defined set of availability objectives and then make the most prudent design choices to achieve those objectives at a reasonable cost. Typically, all systems and services do not need to achieve the highest levels of availability possible; at the same time ensure you do not introduce a single point of failure in your architecture through dependencies between your components. For example, a mobile taxi ordering service needs its ordering-related service to be highly available; however, a specific customer's travel history need not be addressed at the same level of availability. The best way to approach high availability design is to assume that anything can fail, at any time, and then consciously design against it. "Everything fails, all the time." - Werner Vogels, CTO, Amazon.com In other words, think in terms of availability for each and every component in your application and its environment because any given component can turn into a single point of failure for your entire application. Availability is something you should consider early on in your application design process, as it can be hard to retrofit it later. Key among these would be your database and application architecture (for example, RESTful architecture). In addition, it is important to understand that availability objectives can influence and/or impact your design, development, test, and running your system on the cloud. Finally, ensure you proactively test all your design assumptions and reduce uncertainty by injecting or forcing failures instead of waiting for random failures to occur. The nature of failures There are many types of failures that can happen at any time. These could be a result of disk failures, power outages, natural disasters, software errors, and human errors. In addition, there are several points of failure in any given cloud application. These would include DNS or domain services, load balancers, web and application servers, database servers, application services-related failures, and data center-related failures. You will need to ensure you have a mitigation strategy for each of these types and points of failure. It is highly recommended that you automate and implement detailed audit trails for your recovery strategy, and thoroughly test as many of these processes as possible. In the next few sections, we will discuss various strategies to achieve high availability for your application. Specifically, we will discuss the use of AWS features and services such as: VPC Amazon Route 53 Elastic Load Balancing, auto-scaling Redundancy Multi-AZ and multi-region deployments Setting up VPC for high availability Before setting up your VPC, you will need to carefully select your primary site and a disaster recovery (DR) site. Leverage AWS's global presence to select the best regions and availability zones to match your business objectives. The choice of a primary site is usually the closest region to the location of a majority of your customers and the DR site could be in the next closest region or in a different country depending on your specific requirements. Next, we need to set up the network topology, which essentially includes setting up the VPC and the appropriate subnets. The public facing servers are configured in a public subnet; whereas the database servers and other application servers hosting services such as the directory services will usually reside in the private subnets. Ensure you chose different sets of IP addresses across the different regions for the multi-region deployment, for example 10.0.0.0/16 for the primary region and 192.168.0.0/16 for the secondary region to avoid any IP addressing conflicts when these regions are connected via a VPN tunnel. Appropriate routing tables and ACLs will also need to be defined to ensure traffic can traverse between them. Cross-VPC connectivity is required so that data transfer can happen between the VPCs (say, from the private subnets in one region over to the other region). The secure VPN tunnels are basically IPSec tunnels powered by VPN appliances—a primary and a secondary tunnel should be defined (in case the primary IPSec tunnel fails). It is imperative you consult with your network specialists through all of these tasks. An ELB is configured in the primary region to route traffic across multiple availability zones; however, you need not necessarily commission the ELB for your secondary site at this time. This will help you avoid costs for the ELB in your DR or secondary site. However, always weigh these costs against the total cost/time required for recovery. It might be worthwhile to just commission the extra ELB and keep it running. Gateway servers and NAT will need to be configured as they act as gatekeepers for all inbound and outbound Internet access. Gateway servers are defined in the public subnet with appropriate licenses and keys to access your servers in the private subnet for server administration purposes. NAT is required for servers located in the private subnet to access the Internet and is typically used for automatic patch updates. Again, consult your network specialists for these tasks. Elastic load balancing and Amazon Route 53 are critical infrastructure components for scalable and highly available applications; we discuss these services in the next section. Using ELB and Route 53 for high availability In this section, we describe different levels of availability and the role ELBs and Route 53 play from an availability perspective. Instance availability The simplest guideline here is to never run a single instance in a production environment. The simplest approach to improving greatly from a single server scenario is to spin up multiple EC2 instances and stick an ELB in front of them. The incoming request load is shared by all the instances behind the load balancer. ELB uses the least outstanding requests routing algorithm to spread HTTP/HTTPS requests across healthy instances. This algorithm favors instances with the fewest outstanding requests. Even though it is not recommended to have different instance sizes between or within the AZs, the ELB will adjust for the number of requests it sends to smaller or larger instances based on response times. In addition, ELBs use cross-zone load balancing to distribute traffic across all healthy instances regardless of AZs. Hence, ELBs help balance the request load even if there are unequal number of instances in different AZs at any given time (perhaps due to a failed instance in one of the AZs). There is no bandwidth charge for cross-zone traffic (if you are using an ELB). Instances that fail can be seamlessly replaced using auto scaling while other instances continue to operate. Though auto-replacement of instances works really well, storing application state or caching locally on your instances can be hard to detect problems. Instance failure is detected and the traffic is shifted to healthy instances, which then carries the additional load. Health checks are used to determine the health of the instances and the application. TCP and/or HTTP-based heartbeats can be created for this purpose. It is worthwhile implementing health checks iteratively to arrive at the right set that meets your goals. In addition, you can customize the frequency and the failure thresholds as well. Finally, if all your instances are down, then AWS will return a 503. Zonal availability or availability zone redundancy Availability zones are distinct geographical locations engineered to be insulated from failures in other zones. It is critically important to run your application stack in more than one zone to achieve high availability. However, be mindful of component level dependencies across zones and cross-zone service calls leading to substantial latencies in your application or application failures during availability zone failures. For sites with very high request loads, a 3-zone configuration might be the preferred configuration to handle zone-level failures. In this situation, if one zone goes down, then other two AZs can ensure continuing high availability and better customer experience. In the event of a zone failure, there are several challenges in a Multi-AZ configuration, resulting from the rapidly shifting traffic to the other AZs. In such situations, the load balancers need to expire connections quickly and lingering connections to caches must be addressed. In addition, careful configuration is required for smooth failover by ensuring all clusters are appropriately auto scaled, avoiding cross-zone calls in your services, and avoiding mismatched timeouts across your architecture. ELBs can be used to balance across multiple availability zones. Each load balancer will contain one or more DNS records. The DNS record will contain multiple IP addresses and DNS round-robin can be used to balance traffic between the availability zones. You can expect the DNS records to change over time. Using multiple AZs can result in traffic imbalances between AZs due to clients caching DNS records. However, ELBs can help reduce the impact of this caching. Regional availability or regional redundancy ELB and Amazon Route 53 have been integrated to support a single application across multiple regions. Route 53 is AWS's highly available and scalable DNS and health checking service. Route 53 supports high availability architectures by health checking load balancer nodes and rerouting traffic to avoid the failed nodes, and by supporting implementation of multi-region architectures. In addition, Route 53 uses Latency Based Routing (LBR) to route your customers to the endpoint that has the least latency. If multiple primary sites are implemented with appropriate health checks configured, then in cases of failure, traffic shifts away from that site to the next closest region. Region failures can present several challenges as a result of rapidly shifting traffic (similar to the case of zone failures). These can include auto scaling, time required for instance startup, and the cache fill time (as we might need to default to our data sources, initially). Another difficulty usually arises from the lack of information or clarity on what constitutes the minimal or critical stack required to keep the site functioning as normally as possible. For example, any or all services will need to be considered as critical in these circumstances. The health checks are essentially automated requests sent over the Internet to your application to verify that your application is reachable, available, and functional. This can include both your EC2 instances and your application. As answers are returned only for the resources that are healthy and reachable from the outside world, the end users can be routed away from a failed application. Amazon Route 53 health checks are conducted from within each AWS region to check whether your application is reachable from that location. The DNS failover is designed to be entirely automatic. After you have set up your DNS records and health checks, no manual intervention is required for failover. Ensure you create appropriate alerts to be notified when this happens. Typically, it takes about 2 to 3 minutes from the time of the failure to the point where traffic is routed to an alternate location. Compare this to the traditional process where an operator receives an alarm, manually configures the DNS update, and waits for the DNS changes to propagate. The failover happens entirely within the Amazon Route 53 data plane. Depending on your availability objectives, there is an additional strategy (using Route 53) that you might want to consider for your application. For example, you can create a backup static site to maintain a presence for your end customers while your primary dynamic site is down. In the normal course, Route 53 will point to your dynamic site and maintain health checks for it. Furthermore, you will need to configure Route 53 to point to the S3 storage, where your static site resides. If your primary site goes down, then traffic can be diverted to the static site (while you work to restore your primary site). You can also combine this static backup site strategy with a multiple region deployment. Setting up high availability for application and data layers In this section, we will discuss approaches for implementing high availability in the application and data layers of your application architecture. The auto healing feature of AWS OpsWorks provides a good recovery mechanism from instance failures. All OpsWorks instances have an agent installed. If an agent does not communicate with the service for a short duration, then OpsWorks considers the instance to have failed. If auto healing is enabled at the layer and an instance becomes unhealthy, then OpsWorks first terminates the instance and starts a new one as per the layer configuration. In the application layer, we can also do cold starts from preconfigured images or a warm start from scaled down instances for your web servers and application servers in a secondary region. By leveraging auto scaling, we can quickly ramp up these servers to handle full production loads. In this configuration, you would deploy the web servers and application servers across multiple AZs in your primary region while the standby servers need not be launched in your secondary region until you actually need them. However, keep the preconfigured AMIs for these servers ready to launch in your secondary region. The data layer can comprise of SQL databases, NoSQL databases, caches, and so on. These can be AWS managed services such as RDS, DynamoDB, and S3, or your own SQL and NoSQL databases such as Oracle, SQL Server, or MongoDB running on EC2 instances. AWS services come with HA built-in, while using database products running on EC2 instances offers a do-it-yourself option. It can be advantageous to use AWS services if you want to avoid taking on database administration responsibilities. For example, with the increasing sizes of your databases, you might choose to share your databases, which is easy to do. However, resharding your databases while taking in live traffic can be a very complex undertaking and present availability risks. Choosing to use the AWS DynamoDB service in such a situation offloads this work to AWS, thereby resulting in higher availability out of the box. AWS provides many different data replication options and we will discuss a few of those in the following several paragraphs. DynamoDB automatically replicates your data across several AZs to provide higher levels of data durability and availability. In addition, you can use data pipelines to copy your data from one region to another. DynamoDB streams functionality that can be leveraged to replicate to another DynamoDB in a different region. For very high volumes, low latency Kinesis services can also be used for this replication across multiple regions distributed all over the world. You can also enable the Multi-AZ setting for the AWS RDS service to ensure AWS replicates your data to a different AZ within the same region. In the case of Amazon S3, the S3 bucket contents can be copied to a different bucket and the failover can be managed on the client side. Depending on the volume of data, always think in terms of multiple machines, multiple threads and multiple parts to significantly reduce the time it takes to upload data to S3 buckets. While using your own database (running on EC2 instances), use your database-specific high availability features for within and cross-region database deployments. For example, if you are using SQL Server, you can leverage the SQL Server Always-on feature for synchronous and asynchronous replication across the nodes. If the volume of data is high, then you can also use the SQL Server log shipping to first upload your data to Amazon S3 and then restore into your SQL Server instance on AWS. A similar approach in case of Oracle databases uses OSB Cloud Module and RMAN. You can also replicate your non-RDS databases (on-premise or on AWS) to AWS RDS databases. You will typically define two nodes in the primary region with synchronous replication and a third node in the secondary region with asynchronous replication. NoSQL databases such as MongoDB and Cassandra have their own asynchronous replication features that can be leveraged for replication to a different region. In addition, you can create Read Replicas for your databases in other AZs and regions. In this case, if your master database fails followed by a failure of your secondary database, then one of the read replicas can be promoted to being the master. In hybrid architectures, where you need to replicate between on-premise and AWS data sources, you can do so through a VPN connection between your data center and AWS. In case of any connectivity issues, you can also temporarily store pending data updates in SQS, and process them when the connectivity is restored. Usually, data is actively replicated to the secondary region while all other servers like the web servers and application servers are maintained in a cold state to control costs. However, in cases of high availability for web scale or mission critical applications, you can also choose to deploy your servers in active configuration across multiple regions. Implementing high availability in the application In this section, we will discuss a few design principles to use in your application from a high availability perspective. We will briefly discuss using highly available AWS services to implement common features in mobile and Internet of Things (IoT) applications. Finally, we also cover running packaged applications on the AWS cloud. Designing your application services to be stateless and following a micro services-oriented architecture approach can help the overall availability of your application. In such architectures, if a service fails then that failure is contained or isolated to that particular service while the rest of your application services continue to serve your customers. This approach can lead to an acceptable degraded experience rather than outright failures or worse. You should also store user or session information in a central location such as the AWS ElastiCache and then spread information across multiple AZs for high availability. Another design principle is to rigorously implement exception handling in your application code, and in each of your services to ensure graceful exit in case of failures. Most mobile applications share common features including user authentication and authorization, data synchronization across devices; user behavior analytics; retention tracking, storing, sharing, and delivering media globally; sending push notifications; store shared data; stream real-time data; and so on. There are a host of highly available AWS services that can be used for implementing such mobile application functionality. For example, you can use Amazon Cognito to authenticate users, Amazon Mobile Analytics for analyzing user behavior and tracking retention, Amazon SNS for push notifications and Amazon Kinesis for streaming real-time data. In addition, other AWS services such as S3, DynamoDB, IAM, and so on can also be effectively used to complete most mobile application scenarios. For mobile applications, you need to be especially sensitive about latency issues; hence, it is important to leverage AWS regions to get as close to your customers as possible. Similar to mobile applications, for IoT applications you can use the same highly available AWS services to implement common functionality such as device analytics and device messaging/notifications. You can also leverage Amazon Kinesis to ingest data from hundreds of thousands of sensors that are continuously generating massive quantities of data. Aside from your own custom applications, you can also run packaged applications such as SAP on AWS. These would typically include replicated standby systems, Multi-AZ and multi-region deployments, hybrid architectures spanning your own data center, and AWS cloud (connected via VPN or AWS Direct Connect service), and so on. For more details, refer to the specific package guides for achieving high availability on the AWS cloud. Summary In this article, we reviewed some of the strategies you can follow for achieving high availability in your cloud application. We emphasized the importance of both designing your application architecture for availability and using the AWS infrastructural services to get the best results. Resources for Article: Further resources on this subject: Securing vCloud Using the vCloud Networking and Security App Firewall [article] Introduction to Microsoft Azure Cloud Services [article] AWS Global Infrastructure [article]

In this article by Piyush Verma, author of the book Wireshark Network Security, you will be introduced to using Wireshark to detect network reconnaissance activities performed by an insider. A dictionary definition of reconnaissance is "military observation of a region to locate an enemy or ascertain strategic features." A good analogy for reconnaissance will be a thief studying the neighborhood to observe which houses are empty and which are occupied, the number of family members who live at the occupied houses, their entrance points, the time during which these occupied houses are empty, and so on, before he/she can even think about stealing anything from that neighborhood. Network reconnaissance relates to the act of gathering information about the target’s network infrastructure, the devices that reside on the network, the platform used by such devices, and the ports opened on them, to ultimately come up with a brief network diagram of devices and then plan the attack accordingly. The tools required to perform network scanning are readily available and can be downloaded easily over the Internet. One such popular tool is Nmap, short for Network Mapper. It is written by Gordon “Fyodor” Lyon and is a popular tool of choice to perform network-based reconnaissance. Network scanning activities can be as follows: Scanning for live machines Port scans Detecting the presence of a firewall or additional IP protocols (For more resources related to this topic, see here.) Detect the scanning activity for live machines An attacker would want to map out the live machines on the network rather than performing any activity with an assumption that all the machines are live. The following are the two popular techniques, which can be used, and the ways to detect them using Wireshark. Ping sweep This technique makes use of a simple technique to ping an IP address in order to identify whether it is alive or not. Almost all modern networks block the ICMP protocol; hence, this technique is not very successful. However, in case your network supports ICMP-based traffic, you can detect this attack by looking for large number of ping requests going to a range of IP addresses on your network. A helpful filter in this case will be: icmp.type == 8 || icmp.type == 0 ICMP Type 8 = ECHO Request ICMP Type 0 = ECHO Reply ARP sweep ARP responses cannot be disabled on the network; hence, this technique works very well while trying to identify live machines on a local network. Using this technique, an attacker can discover hosts that may be hidden from other discovery methods, such as ping sweeps, by a firewall. To perform this, an attacker sends ARP broadcasts (destination MAC address: FF:FF:FF:FF:FF:FF) for all possible IP addresses on a given subnet, and the machines responding to these requests are noted as alive or active. To detect ARP sweep attempts, we need to look for a massive amount of ARP broadcasts from a client machine on the network. Another thing to note will be the duration in which these broadcasts are sent. These are highlighted in the following screenshot: Shows an ARP sweep in action A point to note is the source of these ARP requests to avoid false positives because such requests may also be made by legitimate services, such as SNMP. Identifying port scanning attempts Now, we will look at different port scanning techniques used by attackers and how to detect them using Wireshark. TCP Connect scan In a TCP Connect scan, a client/attacker sends a SYN packet to the server/victim on a range of port numbers. For the ports that respond to SYN/ACK, the client completes the 3-way handshake by sending an ACK and then terminates the connection by sending an RST to the server/victim, while the ports that are closed reply with RST/ACK packets to the SYN sent by the client/attacker. Hence, in order to identify this type of scan, we will need to look for a significantly large number of RST (Expert Info) or SYN/ACK packets. Generally, when a connection is established, some form of data is transferred; however, in scanning attempts, no data is sent across indicating a suspicious activity (Conversations | TCP). Another indication is the short period of time under which these packets are sent (Statistics | Flow Graph). Wireshark’s Flow Graph While observing the TCP flow in the Flow Graph, we noted a sequence of SYN, SYN/ACK, and ACKs along with SYN and RST/ACKs. Another indication is the fraction of seconds (displayed on left-hand side) under which these packets are sent. Shows a complete 3-way handshake with open-ports and how quickly the packets were sent under the ‘Time’ column Wireshark’s Expert Info Even the Expert Info window indicates a significant number of connection resets. Shows Warning Tab under Expert Info Wireshark’s Conversations We can look at the TCP Conversations, to observe which type of scan is underway and the number of Bytes associated with each conversation. Shows the number of packets and Bytes transferred for each Conversation The number 4 in the Packets column indicates a SYN, SYN/ACK, ACK, and RST packets, and the number 2 indicates the SYN sent by Nmap and RST/ACK received for a closed port. Stealth scan A stealth scan is different from the TCP Connect scan explained earlier and is never detected by the application layer as the complete TCP 3-way handshake is never established during this scan and hence also known as half-open scan. During this scan, a client/attacker sends a SYN packet to the server/victim on a range of port numbers. If Nmap receives a SYN/ACK to the SYN request, it means that the port is open, and Nmap then sends an RST to close the connection without ever completing the 3-way handshake, while the ports that are closed reply with RST/ACK packets to the SYN requests. The way to detect this attack is similar to the previous scan, where you will note a lot of RST (Expert Info) or SYN/ACK packets without data transfers (Conversations | TCP) on the network. Another indication is the short period of time under which these packets are sent (Statistics | Flow Graph). Now, we will look at the Flow Graph, Expert Info, and Conversations in Wireshark for Stealth scan. Wireshark’s Flow Graph While observing the TCP flow in the Flow Graph, we noted a sequence of SYN, SYN/ACK, and RSTs (indicating a half-open connection) along with SYN and RST/ACKs. Another indication is the fraction of seconds (displayed on the left-hand side) under which these packets are sent. Shows the half-open scan underway and how quickly the packets were sent under the "Time" column Wireshark’s Expert Info The huge number of connection resets is another indication of a scan underway. Shows Warning Tab under Expert Info Wireshark’s Conversations TCP Conversations also provide an insight to indicate that a half-open scan is underway and the number of Bytes associated with each attempt. Shows the number of packets and Bytes transferred for each Conversation The number 3 in the Packets column indicates a SYN, SYN/ACK, and RST packets, and the number 2 indicates the SYN sent by Nmap and RST/ACK received for a closed port. NULL scan During a null scan, unusual TCP packets are sent with no flags set. If the resultant of this is no response, it means that the port is either open or filtered; while the RST/ACK response means that the port is closed. A quick way to detect, whether such a scan is underway, is to filter on tcp.flags == 0x00. UDP scan The last three techniques were related to TCP-based scans. Many common protocols work over UDP as well (DNS, SNMP, TFTP, and so on), and scans are conducted to detect whether such ports are open or not. No response to a UDP port scan indicates that the port is either open or firewalled, and a response of ICMP Destination Unreachable/Port Unreachable means that the port is closed. Detect UDP scans by filtering on (icmp.type == 3) && (icmp.code == 3). ICMP Type 3 = Destination Unreachable ICMP Code 3 = Port Unreachable Other scanning attempts The following scanning techniques go beyond the traditional port scanning techniques and help the attacker in further enumeration of the network. ACK scan ACK Flag scan never locates an open port; rather, it only provides the result in form of "filtered" or "unfiltered" and is generally used to detect the presence of a firewall. No response means port is filtered, and RST response indicates that the port is unfiltered. Shows Flow Graph (TCP) of an ACK Flag Scan IP Protocol scan IP Protocol scan is conducted by attackers to determine the presence of additional IP protocols in use by the victim. For example, if a router is scanned using this technique, it might reveal the use of other protocols as EGP, IGP, EIGRP, and so on. No response indicates that a protocol is present or the response is filtered, while an ICMP Destination Unreachable/Protocol Unreachable indicates that the protocol is not supported by the device. To detect this scan using Wireshark, we can filter the traffic based on (icmp.type == 3) && (icmp.code == 2). ICMP Type 3 = Destination Unreachable ICMP Code 2 = Protocol Unreachable Summary In this article, we used Wireshark and the set of robust features it has to offer, to analyze the network scanning attempts performed by attackers. Resources for Article: Further resources on this subject: Wireshark [article] Wireshark: Working with Packet Streams [article] UDP/TCP Analysis [article]

In this article by Yohan Rohinton Wadia the author of Learning VMware vCloud Air, we are going to walk through setting up and accessing virtual machines. (For more resources related to this topic, see here.) What is a virtual machine? Most of you reading this article must be aware of what a virtual machine is, but for the sake of simplicity, let's have a quick look at what it really is. A virtual machine is basically an emulation of a real or physical computer which runs on an operating system and can host your favorite applications as well. Each virtual machine consists of a set of files that govern the way the virtual machine is configured and run. The most important of these files would be a virtual drive, that acts just as a physical drive storing all your data, applications and operating system; and a configuration file that basically tells the virtual machine how much resources are dedicated to it, which networks or storage adapters to use, and so on. The beauty of these files is that you can port them from one virtualization platform to another and manage them more effectively and securely as compared to a physical server. The following diagram shows an overview of how a virtual machine works over a host: Virtual machine creation in vCloud Air is a very simple and straight forward process. vCloud Air provides you with three mechanisms using which you can create your own virtual machines briefly summarized as follows: Wizard driven: vCloud Air provides a simple wizard using which you can deploy virtual machines from pre-configured templates. This option is provided via the vCloud Air web interface itself. Using vCloud Director: vCloud Air provides an advanced option as well for users who want to create their virtual machines from scratch. This is done via the vCloud Director interface and is a bit more complex as compared to the wizard driven option. Bring your own media: Because vCloud Air natively runs on VMware vSphere and vCloud Director platforms, its relatively easy for you to migrate your own media, templates and vApps into vCloud Air using a special tool called as VMware vCloud Connector. Create a virtual machine using template As we saw earlier, VMware vCloud Air provides us with a default template using which you can deploy virtual machines in your public cloud in a matter of seconds. The process is a wizard driven activity where you can select and configure the virtual machine's resources such as CPU, memory, hard disk space all with a few simple clicks. The following steps will you create a virtual machine using a template: Login to your vCloud Air (https://vchs.vmware.com/login) using the username and password that we set during the sign in process. From the Home page, select the VPC on Demand tab. Once there, from the drop-down menu above the tabs, select your region and the corresponding VDC where you would like to deploy your first virtual machine. In this case, I have selected the UK-Slough-6 as the region and MyFirstVDC as the default VDC where I will deploy my virtual machines:If you have selected more than one VDC, you will be prompted to select a specific virtual data center before you start the wizard as a virtual machine cannot span across regions or VDCs. From the Virtual Machines tab, select the Create your first virtual machine option. This will bring up the VM launch wizard as shown here: As you can see here, there are two tabs provided by default: a VMware Catalog and another section called as My Catalog. This is an empty catalog by default but this is the place where all your custom templates and vApps will be shown if you have added them from the vCloud Director portal or purchased them from the Solutions Exchange site as well. Select any particular template to get started with. You can choose your virtual machine to be either powered by a 32 bit or a 64 bit operating system. In my case, I have selected a CentOS 6.4 64 bit template for this exercise. Click Continue once done. Templates provided by vCloud Air are either free or paid. The paid ones generally have a $ sign marked next to the OS architecture, indicating that you will be charged once you start using the virtual machine. You can track all your purchases using the vCloud Air billing statement. The next step is to define the basic configuration for your virtual machine. Provide a suitable name for your virtual machine. You can add an optional description to it as well. Next, select the CPU, memory and storage for the virtual machine. The CPU and memory resources are linked with each other so changing the CPU will automatically set the default vRAM for the virtual machine as well; however you can always increase the vRAM as per your needs. In this case, the virtual machine has 2 CPUs and 4 GB vRAM allocated to it. Select the amount of storage you want to provide to your virtual machine. VMware can allocate a maximum of 2 TB of storage as a single drive to a virtual machine. However as a best practice; it is always good to add more storage by adding multiple drives rather than storing it all on one single drive. You can optionally select your disks to be either standard or SSD-accelerated; both features we will discuss shortly. Virtual machine configuration Click on Create Virtual Machine once you are satisfied with your changes. Your virtual machine will now be provisioned within few minutes. By default, the virtual machine is not powered on after it is created. You can power it on by selecting the virtual machine and clicking on the Power On icon in the tool bar above the virtual machine: Status of the virtual machine created There you have it. Your very first virtual machine is now ready for use! Once powered on, you can select the virtual machine name to view its details along with a default password that is auto-generated by vCloud Air. Accessing virtual machines using the VMRC Once your virtual machines are created and powered on, you can access and view them easily using the virtual machine remote console (VMRC). There are two ways to invoke the VMRC, one is by selecting your virtual machine from the vCloud Air dashboard, selecting the Actions tab and select the option Open in Console as shown: The other way to do so is by selecting the virtual machine name. This will display the Settings page for that particular virtual machine. To launch the console select the Open Virtual Machine option as shown: Make a note of the Guest OS Password from the Guest OS section. This is the default password that will be used to log in to your virtual machine. To log in to the virtual machine, use the following credentials: Username: root Password: <Guest_OS_Password> This is shown in the following screenshot: You will be prompted to change this password on your first login. Provide a strong new password that contains at least one special character and contains an alphanumeric pattern as well. Summary There you have it! Your very own Linux virtual machine on the cloud! Resources for Article: Further resources on this subject: vCloud Networks [Article] Creating your first VM using vCloud technology [Article] Securing vCloud Using the vCloud Networking and Security App Firewall [Article]

In this article by Christopher John Pecoraro, author of the book, Mastering Laravel, you learn the fundamentals of using the form builder. (For more resources related to this topic, see here.) Building web pages with Laravel Laravel's approach to building web content is flexible. As much or as little of Laravel can be used to create HTML. Laravel uses the filename.blade.php convention to state that the file should be parsed by the blade parser, which actually converts the file into plain PHP. The name blade was inspired by the .NET's razor templating engine, so this may be familiar to someone who has used it. Laravel 5 provides a working demonstration of a form in the /resources/views/ directory. This view is shown when the /home route is requested and the user is not currently logged in. This form is obviously not created using the Laravel form methods. The route is defined in the routes file as follows: Route::get('home', 'HomeController@index'); The master template This is the following app (or master) template: <!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1"> <title>Laravel</title> <link href="/css/app.css" rel="stylesheet"> <!-- Fonts --> <link href='//fonts.googleapis.com/css?family=Roboto:400,300' rel='stylesheet' type='text/css'> <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> <!-- WARNING: Respond.js doesn't work if you view the page via file:// --> <!--[if lt IE 9]> <script src="https://oss.maxcdn.com/html5shiv/3.7.2/ html5shiv.min.js"></script> <script src="https://oss.maxcdn.com/respond/1.4.2/ respond.min.js"></script> <![endif]--> </head> <body> <nav class="navbarnavbar-default"> <div class="container-fluid"> <div class="navbar-header"> <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" datatarget="# bs-example-navbar-collapse-1"> <span class="sr-only">Toggle Navigation</span> <span class="icon-bar"></span> <span class="icon-bar"></span> <span class="icon-bar"></span> </button> <a class="navbar-brand" href="#">Laravel</a> </div> <div class="collapse navbar-collapse" id=" bs-example-navbar-collapse-1"> <ul class="navnavbar-nav"> <li><a href="/">Home</a></li> </ul> <ul class="navnavbar-navnavbar-right"> @if (Auth::guest()) <li><a href="{{ route('auth.login') }}">Login</a></li> <li><a href="/auth/register"> Register</a></li> @else <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">{{ Auth::user()->name }} <span class="caret"></span></a> <ul class="dropdown-menu" role="menu"> <li><a href="/auth/ logout">Logout</a></li> </ul> </li> @endif </ul> </div> </div> </nav> @yield('content') <!-- Scripts --> <script src="//cdnjs.cloudflare.com/ajax/libs/jquery/ 2.1.3/jquery.min.js"></script> <script src="//cdnjs.cloudflare.com/ajax/libs/twitterbootstrap/ 3.3.1/js/bootstrap.min.js"></script> </body> </html> The Laravel 5 master template is a standard HTML5 template with the following features: If the browser is older than Internet Explorer 9: Uses the HTML5 Shim from the CDN Uses the Respond.js JavaScript code from the CDN to retrofit media queries and CSS3 features Using @if (Auth::guest()), if the user is not authenticated, the login form is displayed; otherwise, the logout option is displayed Twitter bootstrap 3.x is included in the CDN The jQuery2.x is included in the CDN Any template that extends this template can override the content section An example page The source code for the login page is as follows: @extends('app') @section('content') <div class="container-fluid"> <div class="row"> <div class="col-md-8 col-md-offset-2"> <div class="panel panel-default"> <div class="panel-heading">Login</div> <div class="panel-body"> @if (count($errors) > 0) <div class="alert alert-danger"> <strong>Whoops!</strong> There were some problems with your input.<br><br> <ul> @foreach ($errors->all() as $error) <li>{{ $error }}</li> @endforeach </ul> </div> @endif <form class="form-horizontal" role="form" method="POST" action="/auth/login"> <input type="hidden" name="_token" value="{{ csrf_token() }}"> <div class="form-group"> <label class="col-md-4 controllabel"> E-Mail Address</label> <div class="col-md-6"> <input type="email" class="formcontrol" name="email" value="{{ old('email') }}"> </div> </div> <div class="form-group"> <label class="col-md-4 controllabel"> Password</label> <div class="col-md-6"> <input type="password" class="form-control" name="password"> </div> </div> <div class="form-group"> <div class="col-md-6 col-md-offset-4"> <div class="checkbox"> <label> <input type="checkbox" name="remember"> Remember Me </label> </div> </div> </div> <div class="form-group"> <div class="col-md-6 col-md-offset-4"> <button type="submit" lass="btn btn-primary" style="margin-right: 15px;"> Login </button> <a href="/password/email">Forgot Your Password?</a> </div> </div> </form> </div> </div> </div> </div> </div> @endsection From static HTML to static methods This login page begins with the following: @extends('app') It obviously uses the object-oriented paradigm to state that the app.blade.php template will be rendered. The following line overrides the content: @section('content') For this exercise, the form builder will be used instead of the static HTML. The form tag We will convert a static form tag to a FormBuilder method. The HTML is as follows: <form class="form-horizontal" role="form" method="POST"   action="/auth/login"> The method facade that we will use is as follows: Form::open(); In the FormBuilder.php class, the $reserved attribute is defined as follows: protected $reserved = ['method', 'url', 'route',   'action', 'files']; The attributes that we need to pass to an array to the open() method are class, role, method, and action. Since method and action are reserved words, it is necessary to pass the parameters in the following manner: Laravel form facade method array element HTML Form tag attribute method method url action role role class class Thus, the method call looks like this: {!! Form::open(['class'=>'form-horizontal', 'role =>'form', 'method'=>'POST', 'url'=>'/auth/login']) !!} The {!! !!} tags are used to start and end parsing of the form builder methods. The form method, POST, is placed first in the list of attributes in the HTML form tag. The action attribute actually needs to be a url. If the action parameter is used, then it refers to the controller action. In this case, the url parameter produces the action attribute of the form tag. Other attributes will be passed to the array and added to the list of attributes. The resultant HTML will be produced as follows: <form method="POST" action="http://laravel.example/auth/login" accept-charset="UTF-8" class="form-horizontal" role="form"> <input name="_token" type="hidden" value="wUY2hFSEWCzKHFfhywHvFbq9TXymUDiRUFreJD4h"> The CRSF token is automatically added, as the form method is POST. The text input field To convert the input fields, a facade is used. The input field's HTML is as follows: <input type="email" class="form-control" name="email" value="{{ old('email') }}"> Converting the preceding input field using a façade looks like this: {!! Form::input('email','email',old('email'), ['class'=>'form-control' ]) !!} Similarly, the text field becomes: {!! Form::input('password','password',null, ['class'=>'form-control']) !!} The input fields have the same signature. Of course, this can be refactored as follows: <?php $inputAttributes = ['class'=>'form-control'] ?> {!! Form::input('email','email',old('email'), $inputAttributes ) !!} ... {!! Form::input('password','password',null,$inputAttributes ) !!} The label tag The label tags are as follows: <label class="col-md-4 control-label">E-Mail Address</label> <label class="col-md-4 control-label">Password</label> To convert the label tags (E-Mail Address and Password), we will first create an array to hold the attributes, and then pass this array to the labels, as follows: $labelAttributes = ['class'=>'col-md-4 control-label']; Here is the form label code: {!! Form::label('email', 'E-Mail Address', $labelAttributes) !!} {!! Form::label('password', 'Password', $labelAttributes) !!} Checkbox To convert the checkbox to a facade, we will convert this: <input type="checkbox" name="remember"> Remember Me The preceding code is converted to the following code: {!! Form::checkbox('remember','') !!} Remember Me Remember that the PHP parameters should be sent in single quotation marks if there are no variables or other special characters, such as line breaks, inside the string to parse, while the HTML produced will have double quotes. The submit button Lastly, the submit button will be converted as follows: <button type="submit" class="btn btn-primary" style="margin-right: 15px;"> Login </button> The preceding code after conversion is as follows:   {!! Form::submit('Login', ['class'=>'btn btn-primary', 'style'=>'margin-right: 15px;']) !!} Note that the array parameter provides an easy way to provide any desired attributes, even those that are not among the list of standard HTML form elements. The anchor tag with links To convert the links, a helper method is used. Consider the following line of code: <a href="/password/email">Forgot Your Password?</a> The preceding line of code after conversion becomes: {!! link_to('/password/email', $title = 'Forgot Your Password?', $attributes = array(), $secure = null) !!} The link_to_route() method may be used to link to a route. For similar helper functions, visit http://laravelcollective.com/docs/5.0/html. Closing the form To end the form, we'll convert the traditional HTML form tag </form> to a Laravel {!! Form::close() !!} form method. The resultant form By putting everything together, the page now looks like this: @extends('app') @section('content') <div class="container-fluid"> <div class="row"> <div class="col-md-8 col-md-offset-2"> <div class="panel panel-default"> <div class="panel-heading">Login</div> <div class="panel-body"> @if (count($errors) > 0) <div class="alert alert-danger"> <strong>Whoops!</strong> There were some problems with your input.<br><br> <ul> @foreach ($errors->all() as $error) <li>{{ $error }}</li> @endforeach </ul> </div> @endif <?php $inputAttributes = ['class'=>'form-control']; $labelAttributes = ['class'=>'col-md-4 control-label']; ?> {!! Form::open(['class'=>'form-horizontal','role'=> 'form','method'=>'POST','url'=>'/auth/login']) !!} <div class="form-group"> {!! Form::label('email', 'E-Mail Address',$labelAttributes) !!} <div class="col-md-6"> {!! Form::input('email','email',old('email'), $inputAttributes) !!} </div> </div> <div class="form-group"> {!! Form::label('password', 'Password',$labelAttributes) !!} <div class="col-md-6"> {!! Form::input('password', 'password',null,$inputAttributes) !!} </div> </div> <div class="form-group"> <div class="col-md-6 col-md-offset-4"> <div class="checkbox"> <label> {!! Form::checkbox('remember','') !!} Remember Me </label> </div> </div> </div> <div class="form-group"> <div class="col-md-6 col-md-offset-4"> {!! Form::submit('Login',['class'=> 'btn btn-primary', 'style'=> 'margin-right: 15px;']) !!} {!! link_to('/password/email', $title = 'Forgot Your Password?', $attributes = array(), $secure = null); !!} </div> </div> {!! Form::close() !!} </div> </div> </div> </div> </div> @endsection Our example If we want to create a form to reserve a room in our accommodation, we can easily call a route from our controller: /** * Show the form for creating a new resource. * * @return Response */ public function create() { return view('auth/reserve'); } Now we need to create a new view that is located at resources/views/auth/reserve.blade.php. In this view, we can create a form to reserve a room in an accommodation where the user can select the start date, which comprises of the start day of the month and year, and the end date, which also comprises of the start day of the month and year. The form would begin as before, with a POST to reserve-room. Then, the form label would be placed next to the select input fields. Finally, the day, the month, and the year select form elements would be created as follows: {!! Form::open(['class'=>'form-horizontal', 'role'=>'form', 'method'=>'POST', 'url'=>'reserve-room']) !!} {!! Form::label(null, 'Start Date',$labelAttributes) !!} {!! Form::selectMonth('month',date('m')) !!} {!! Form::selectRange('date',1,31,date('d')) !!} {!! Form::selectRange('year',date('Y'),date('Y')+3) !!} {!! Form::label(null, 'End Date',$labelAttributes) !!} {!! Form::selectMonth('month',date('m')) !!} {!! Form::selectRange('date',1,31,date('d')) !!} {!! Form::selectRange('year',date('Y'), date('Y')+3,date('Y')) !!} {!! Form::submit('Reserve', ['class'=>'btn btn-primary', 'style'=>'margin-right: 15px;']) !!} {!! Form::close() !!} Month select Firstly, in the selectMonth method, the first parameter is the name of the input attribute, while the second attribute is the default value. Here, the PHP date method is used to extract the numeric portion of the current month—March in this case: {!! Form::selectMonth('month',date('m')) !!} The output, shown here formatted, is as follows: <select name="month"> <option value="1">January</option> <option value="2">February</option> <option value="3" selected="selected">March</option> <option value="4">April</option> <option value="5">May</option> <option value="6">June</option> <option value="7">July</option> <option value="8">August</option> <option value="9">September</option> <option value="10">October</option> <option value="11">November</option> <option value="12">December</option> </select> Date select A similar technique is applied for the selection of the date, but using the selectRange method, the range of the days in the month are passed to the method. Similarly, the PHP date function is used to send the current date to the method as the fourth parameter: {!! Form::selectRange('date',1,31,date('d')) !!} Here is the formatted output: <select name="date"> <option value="1">1</option> <option value="2">2</option> <option value="3">3</option> <option value="4">4</option> ... <option value="28">28</option> <option value="29">29</option> <option value="30" selected="selected">30</option> <option value="31">31</option> </select> The date that should be selected is 30, since today is March 30, 2015. For the months that do not have 31 days, usually a JavaScript method would be used to modify the number of days based on the month and/or the year. Year select The same technique that is used for the date range is applied for the selection of the year; once again, using the selectRange method. The range of the years is passed to the method. The PHP date function is used to send the current year to the method as the fourth parameter: {!! Form::selectRange('year',date('Y'),date('Y')+3,date('Y')) !!} Here is the formatted output: <select name="year"> <option value="2015" selected="selected">2015</option> <option value="2016">2016</option> <option value="2017">2017</option> <option value="2018">2018</option> </select> Here, the current year that is selected is 2015. Form macros We have the same code that generates our month, date, and year selection form block two times: once for the start date and once for the end date. To refactor the code, we can apply the DRY (don't repeat yourself) principle and create a form macro. This will allow us to avoid calling the form element creation method twice, as follows: <?php Form::macro('monthDayYear',function($suffix='') { echo Form::selectMonth(($suffix!=='')?'month- '.$suffix:'month',date('m')); echo Form::selectRange(($suffix!=='')?'date- '.$suffix:'date',1,31,date('d')); echo Form::selectRange(($suffix!=='')?'year- '.$suffix:'year',date('Y'),date('Y')+3,date('Y')); }); ?> Here, the month, date, and year generation code is placed into a macro, which is inside the PHP tags, and it is necessary to add echo to print out the result. The monthDayYear name is given to this macro method. Calling our macro two times: once after each label; each time adding a different suffix via the $suffix variable. Now, our form code looks like this: <?php Form::macro('monthDayYear',function($suffix='') { echo Form::selectMonth(($suffix!=='')?'month- '.$suffix:'month',date('m')); echo Form::selectRange(($suffix!=='')?'date- '.$suffix:'date',1,31,date('d')); echo Form::selectRange(($suffix!=='')?'year- '.$suffix:'year',date('Y'),date('Y')+3,date('Y')); }); ?> {!! Form::open(['class'=>'form-horizontal', 'role'=>'form', 'method'=>'POST', 'url'=>'/reserve-room']) !!} {!! Form::label(null, 'Start Date',$labelAttributes) !!} {!! Form::monthDayYear('-start') !!} {!! Form::label(null, 'End Date',$labelAttributes) !!} {!! Form::monthDayYear('-end') !!} {!! Form::submit('Reserve',['class'=>'btn btn-primary', 'style'=>'margin-right: 15px;']) !!} {!! Form::close() !!} Conclusion The choice to include the HTML form generation package in Laravel 5 can ease the burden of having to create numerous HTML forms. This approach allows developers to use methods, create reusable macros, and use a familiar Laravel approach to build the frontend. Once the basic methods are learned, it is very easy to simply copy and paste the previously created form elements, and then change their element's name and/or the array that is sent to them. Depending on the size of the project, this approach may or may not be the right choice. For a very small application, the difference in the amount of code that needs to be written is not very evident, although, as is the case with the selectMonth and selectRange methods, the amount of code necessary is drastic. This technique, combined with the use of macros, makes it easy to reduce the occurrence of copy duplication. Also, one of the major problems with the frontend design is that the contents of the class of the various elements may need to change throughout the entire application. This would mean performing a large find and replace operation, where changes are required to be made to HTML, such as changing class attributes. By creating an array of attributes, including class, for similar elements, changes made to the entire form can be performed simply by modifying the array that those elements use. In a larger project, however, where parts of forms may be repeated throughout the application, the wise use of macros can easily reduce the amount of code necessary to be written. Not only this, but macros can isolate the code inside from changes that would require more than one block of code to be changed throughout multiple files. In the example, where the month, date, and year is to be selected, it is possible that this could be used up to 20 times in a large application. Any changes made to the desired block of HTML can be simply done to the macro and the result would be reflected in all of the elements that use it. Ultimately, the choice of whether or not to use this package will reside with the developer and the designer. Since a designer who wants to use an alternative frontend design tool may not prefer, nor be competent, to work with the methods in the package, he or she may want to not use it. Summary The construction of the master template was explained and then the form components, such as the various form input types, were shown through examples. Finally, the construction of a form for the room reservation, was explained, as well as a "do not repeat yourself" form macro creation technique. Resources for Article: Further resources on this subject: Eloquent… without Laravel! [Article] Laravel 4 - Creating a Simple CRUD Application in Hours [Article] Exploring and Interacting with Materials using Blueprints [Article]

In this article by Jonas Andersson and Mike Pfeiffer, authors of the book Microsoft Exchange Server PowerShell Cookbook - Third Edition, we will see how the mailboxes are managed in the Exchange Management Shell. (For more resources related to this topic, see here.) Adding, modifying, and removing mailboxes One of the most common tasks performed within the Exchange Management Shell is mailbox management. In this recipe, we'll take a look at the command syntax required to create, update, and remove mailboxes from your Exchange organization. The concepts outlined in this recipe can be used to perform basic day-to-day tasks and will be useful for more advanced scenarios, such as creating mailboxes in bulk. How to do it... Let's see how to add, modify, and delete mailboxes using the following steps: Let's start off by creating a mailbox-enabled Active Directory user account. To do this, we can use the New-Mailbox cmdlet, as shown in the following example: $password = ConvertTo-SecureString -AsPlainText P@ssw0rd `-ForceNew-Mailbox -UserPrincipalName dave@contoso.com `-Alias dave `-Database DAGDB1 `-Name 'Dave Jones' `-OrganizationalUnit Sales `-Password $password `-FirstName Dave `-LastName Jones `-DisplayName 'Dave Jones' Once the mailbox has been created, we can modify it using the Set-Mailbox cmdlet: Set-Mailbox -Identity dave `-UseDatabaseQuotaDefaults $false `-ProhibitSendReceiveQuota 5GB `-IssueWarningQuota 4GB To remove the Exchange attributes from the Active Directory user account and mark the mailbox in the database for removal, use the Disable-Mailbox cmdlet: Disable-Mailbox -Identity dave -Confirm:$false How it works... When running the New-Mailbox cmdlet, the -Password parameter is required, and you need to provide a value for it, using a secure string object. As you can see from the code, we used the ConvertTo-SecureString cmdlet to create a $password variable that stores a specified value as an encrypted string. This $password variable is then assigned to the -Password parameter when running the cmdlet. It's not required to first store this object in a variable; we could have done it inline, as shown next: New-Mailbox -UserPrincipalName dave@contoso.com `-Alias dave `-Database DAGDB1 `-Name 'Dave Jones' `-OrganizationalUnit Sales `-Password (ConvertTo-SecureString -AsPlainText P@ssw0rd -Force) `-FirstName Dave `-LastName Jones `-DisplayName 'Dave Jones' Keep in mind that the password used here needs to comply with your Active Directory password policies, which may enforce a minimum password length and have requirements for complexity. Only a few parameters are actually required when running New-Mailbox, but the cmdlet itself supports several useful parameters that can be used to set certain properties when creating the mailbox. You can run Get-Help New-Mailbox -Detailed to determine which additional parameters are supported. The New-Mailbox cmdlet creates a new Active Directory user and then mailbox-enables that account. We can also create mailboxes for existing users with the Enable-Mailbox cmdlet, using a syntax similar to the following: Enable-Mailbox steve -Database DAGDB1 The only requirement when running the Enable-Mailbox cmdlet is that you need to provide the identity of the Active Directory user that should be mailbox-enabled. In the previous example, we specified the database in which the mailbox should be created, but this is optional. The Enable-Mailbox cmdlet supports a number of other parameters that you can use to control the initial settings for the mailbox. You can use a simple one-liner to create mailboxes in bulk for existing Active Directory users: Get-User -RecipientTypeDetails User | Enable-Mailbox -Database DAGDB1 Notice that we run the Get-User cmdlet, specifying User as the value for the -RecipientTypeDetails parameter. This will retrieve only the accounts in Active Directory that have not been mailbox-enabled. We then pipe those objects down to the Enable-Mailbox cmdlet and mailboxes will be created for each of those users in one simple operation. Once the mailboxes have been created, they can be modified with the Set-Mailbox cmdlet. As you may recall from our original example, we used the Set-Mailbox cmdlet to configure the custom storage quota settings after creating a mailbox for Dave Jones. Keep in mind that the Set-Mailbox cmdlet supports over 100 parameters, so anything that can be done to modify a mailbox can be scripted. Bulk modifications to mailboxes can be done easily by taking advantage of the pipeline and the Set-Mailbox cmdlet. Instead of configuring storage quotas on a single mailbox, we can do it for multiple users at once: Get-Mailbox -OrganizationalUnit contoso.com/sales |Set-Mailbox -UseDatabaseQuotaDefaults $false `-ProhibitSendReceiveQuota 5GB `-IssueWarningQuota 4GB Here, we are simply retrieving every mailbox in the sales OU using the Get-Mailbox cmdlet. The objects returned from this command are piped down to Set-Mailbox, which then modifies the quota settings for each mailbox in one shot. The Disable-Mailbox cmdlet will strip the Exchange attributes from an Active Directory user and will disconnect the associated mailbox. By default, disconnected mailboxes are retained for 30 days. You can modify this setting on the database that holds the mailbox. In addition to this, you can also use the Remove-Mailbox cmdlet to delete both the Active Directory account and the mailbox at once: Remove-Mailbox -Identity dave -Confirm:$false After running this command, the mailbox will be purged once it exceeds the deleted mailbox retention setting on the database. One common mistake is when administrators use the Remove-Mailbox cmdlet when the Disable-Mailbox cmdlet should have been used. It's important to remember that the Remove-Mailbox cmdlet will delete the Active Directory user account and mailbox, while the Disable-Mailbox cmdlet only removes the mailbox, but the Active Directory user account still remains. There's more... When we ran the New-Mailbox cmdlet in the previous examples, we assigned a secure string object to the –Password parameter using the ConvertTo-SecureString cmdlet. This is a great technique to use when your scripts need complete automation, but you can also allow an operator to enter this information interactively. For example, you might build a script that prompts an operator for a password when creating one or more mailboxes. There are a couple of ways you can do this. First, you can use the Read-Host cmdlet to prompt the user running the script to enter a password: $pass = Read-Host "Enter Password" -AsSecureString Once a value has been entered in the shell, your script can assign the $pass variable to the -Password parameter of the New-Mailbox cmdlet. Alternatively, you can supply a value for the -Password parameter using the Get-Credential cmdlet: New-Mailbox -Name Dave -UserPrincipalName dave@contoso.com `-Password (Get-Credential).password You can see that the value we are assigning to the -Password parameter in this example is actually the password property of the object returned by the Get-Credential cmdlet. Executing this command will first launch a Windows authentication dialog box, where the caller can enter a username and password. Once the credential object has been created, the New-Mailbox cmdlet will run. Even though a username and password must be entered in the authentication dialog box, only the password value will be used when the command executes. Setting the Active Directory attributes Some of the Active Directory attributes that you may want to set when creating a mailbox might not be available when using the New-Mailbox cmdlet. Good examples of this are a user's city, state, company, and department attributes. In order to set these attributes, you'll need to call the Set-User cmdlet after the mailbox has been created: Set-User –Identity dave –Office IT –City Seattle –State Washington You can run Get-Help Set-User -Detailed to view all of the available parameters supported by this cmdlet. Notice that the Set-User cmdlet is an Active Directory cmdlet and not an Exchange cmdlet. In the examples using the New-Mailbox cmdlet to create new mailboxes, it is not required to use all these parameters from the example. The only required parameters are UserPrincipalName, Name, and Password. Working with contacts Once you've started managing mailboxes using the Exchange Management Shell, you'll probably notice that the concepts and command syntax used to manage contacts are very similar. The difference, of course, is that we need to use a different set of cmdlets. In addition, we also have two types of contacts to deal with in Exchange. We'll take a look at how you can manage both of them in this recipe. How to do it... Let's see how to work with contacts using the following steps: To create a mail-enabled contact, use the New-MailContact cmdlet: New-MailContact -Alias rjones `-Name "Rob Jones" `-ExternalEmailAddress rob@fabrikam.com `-OrganizationalUnit sales Mail-enabled users can be created with the New-MailUser cmdlet: New-MailUser -Name 'John Davis' `-Alias jdavis `-UserPrincipalName jdavis@contoso.com `-FirstName John `-LastName Davis `-Password (ConvertTo-SecureString -AsPlainText P@ssw0rd `-Force) `-ResetPasswordOnNextLogon $false `-ExternalEmailAddress jdavis@fabrikam.com How it works... Mail contacts are useful when you have external e-mail recipients that need to show up in your global address list. When you use the New-MailContact cmdlet, an Active Directory contact object is created and mail-enabled with the external e-mail address assigned. You can mail-enable an existing Active Directory contact using the Enable-MailContact cmdlet. Mail users are similar to mail contacts in the way that they have an associated external e-mail address. The difference is that these objects are mail-enabled Active Directory users and this explains why we need to assign a password when creating the object. You might use a mail user for a contractor who works onsite in your organization and needs to be able to log on to your domain. When users in your organization need to e-mail this person, they can select them from the global address list and messages sent to these recipients will be delivered to the external address configured for the account. When dealing with mailboxes, there are a couple of things that should be taken into consideration when it comes to removing contacts and mail users. You can remove the Exchange attributes from a contact using the Disable-MailContact cmdlet. The Remove-MailContact cmdlet will remove the contact object from the Active Directory and Exchange. Similarly, the Disable-MailUser and Remove-MailUser cmdlets work in the same fashion. There's more... Like mailboxes, mail contacts and mail-enabled user accounts have several Active Directory attributes that can be set, such as the job title, company, department, and so on. To update these attributes, you can use the Set-* cmdlets that are available for each respective type. For example, to update our mail contact, we can use the Set-Contact cmdlet with the following syntax: Set-Contact -Identity rjones `-Title 'Sales Contractor' `-Company Fabrikam `-Department Sales To modify the same settings for a mail-enabled user, use the Set-User cmdlet: Set-User -Identity jdavis `-Title 'Sales Contractor' `-Company Fabrikam `-Department Sales Both cmdlets can be used to modify a number of different settings. Use the help system to view all of the available parameters. Managing distribution groups In many Exchange environments, distribution groups are heavily relied upon and require frequent changes. This recipe will cover the creation of distribution groups and how to add members to groups, which might be useful when performing these tasks interactively in the shell or through automated scripts. How to do it... Let's see how to create distribution groups using the following steps: To create a distribution group, use the New-DistributionGroup cmdlet: New-DistributionGroup -Name Sales Once the group has been created, adding multiple members can be done easily using a one-liner, as follows: Get-Mailbox -OrganizationalUnit Sales |Add-DistributionGroupMember -Identity Sales We can also create distribution groups whose memberships are set dynamically: New-DynamicDistributionGroup -Name Accounting `-Alias Accounting `-IncludedRecipients MailboxUsers,MailContacts `-OrganizationalUnit Accounting `-ConditionalDepartment accounting,finance `-RecipientContainer contoso.com How it works... There are two types of distribution groups that can be created with Exchange. Firstly, there are regular distribution groups, which contain a distinct list of users. Secondly, there are dynamic distribution groups, whose members are determined at the time a message is sent based on a number of conditions or filters that have been defined. Both types have a set of cmdlets that can be used to add, remove, update, enable, or disable these groups. By default, when creating a standard distribution group, the group scope will be set to Universal. You can create a mail-enabled security group using the New-DistributionGroup cmdlet by setting the -Type parameter to Security. If you do not provide a value for the -Type parameter, the group will be created using the Distribution group type. You can mail-enable an existing Active Directory universal distribution group using the Enable-DistributionGroup cmdlet. After creating the Sales distribution group in our previous example, we added all of the mailboxes in the Sales OU to the group using the Add-DistributionGroupMember cmdlet. You can do this in bulk, or for one user at a time, using the –Member parameter: Add-DistributionGroupMember -Identity Sales -Member administrator Dynamic distribution groups determine their membership based on a defined set of filters and conditions. When we created the Accounting distribution group, we used the -IncludedRecipients parameter to specify that only the MailboxUsers and MailContacts object types would be included in the group. This eliminates resource mailboxes, groups, or mail users from being included as members. The group will be created in the Accounting OU based on the value used with the -OrganizationalUnit parameter. Using the –ConditionalDepartment parameter, the group will only include users that have a department setting of either Accounting or Finance. Finally, since the -RecipientContainer parameter is set to the FQDN of the domain, any user located in the Active Directory can potentially be included in the group. You can create more complex filters for dynamic distribution groups using a recipient filter. You can modify both group types using the Set-DistributionGroup and Set-DynamicDistributionGroup cmdlets. There's more... When dealing with other recipient types, there are a couple of things that should be taken into consideration when it comes to removing distribution groups. You can remove the Exchange attributes from a group using the Disable-DistributionGroup cmdlet. The Remove-DistributionGroup cmdlet will remove the group object from the Active Directory and Exchange. Managing resource mailboxes In addition to mailboxes, groups, and external contacts, recipients can also include specific rooms or pieces of equipment. Locations such as a conference room or a classroom can be given a mailbox so that they can be reserved for meetings. Equipment mailboxes can be assigned to physical, non-location specific resources, such as laptops or projectors, and can then be checked out to individual users or groups by booking a time with the mailbox. In this recipe, we'll take a look at how you can manage resource mailboxes using the Exchange Management Shell. How to do it... When creating a resource mailbox from within the shell, the syntax is similar to creating a mailbox for a regular user. For example, you still use the New-Mailbox cmdlet when creating a resource mailbox: New-Mailbox -Name "CR23" -DisplayName "Conference Room 23" `-UserPrincipalName CR23@contoso.com -Room How it works... There are two main differences when it comes to creating a resource mailbox, as opposed to a standard user mailbox. First, you need to use either the -Room switch parameter or the -Equipment switch parameter to define the type of resource mailbox that will be created. Second, you do not need to provide a password value for the user account. When using either of these resource mailbox switch parameters to create a mailbox, the New-Mailbox cmdlet will create a disabled Active Directory user account that will be associated with the mailbox. The entire concept of room and equipment mailboxes revolves around the calendars used by these resources. If you want to reserve a room or a piece of equipment, you book a time through Outlook or OWA with these resources for the duration that you'll need them. The requests sent to these resources need to be accepted, either by a delegate or automatically using the Resource Booking Attendant. To configure the room mailbox created in the previous example, to automatically accept new meeting requests, we can use the Set-CalendarProcessing cmdlet to set the Resource Booking Attendant for that mailbox to AutoAccept: Set-CalendarProcessing CR23 -AutomateProcessing AutoAccept When the Resource Booking Attendant is set to AutoAccept, the request will be immediately accepted as long as there is no conflict with another meeting. If there is a conflict, an e-mail message will be returned to the requestor, which explains that the request was declined due to scheduling conflicts. You can allow conflicts by adding the –AllowConflicts switch parameter to the previous command. When working with resource mailboxes with AutomateProcessing set to AutoAccept, you'll get an automated e-mail response from the resource after booking a time. This e-mail message will explain whether the request was accepted or declined, depending on your settings. You can add an additional text to the response message that the meeting organizer will receive using the following syntax: Set-CalendarProcessing -Identity CR23 `-AddAdditionalResponse $true `-AdditionalResponse 'For Assistance Contact Support at Ext. #3376' This example uses the Set-CalendarProcessing cmdlet to customize the response messages sent from the CR23 room mailbox. You can see here that we added a message that tells the user the help desk number to call if assistance is required. Keep in mind that you can only add an additional response text, when the AutomateProcessing property is set to AutoAccept. If you do not want to automate the calendar processing for a resource mailbox, then you'll need to add delegates that can accept or deny meetings for that resource. Again, we can turn to the Set-CalendarProcessing cmdlet to accomplish this: Set-CalendarProcessing -Identity CR23 `-ResourceDelegates "joe@contoso.com","steve@contoso.com" `-AutomateProcessing None In this example, we added two delegates to the resource mailbox and have turned off automated processing. When a request comes into the CR23 mailbox, both Steve and Joe will be notified and can accept or deny the request on behalf of the resource mailbox. There's more... When it comes to working with resource mailboxes, another useful feature is the ability to assign custom resource properties to rooms and equipment resources. For example, you may have a total of 5, 10, or 15 conference rooms, but maybe only four of these have whiteboards. It might be useful for your users to know this information when booking a resource for a meeting, where they will be conducting a training session. Using the shell, we can add custom resource properties to the Exchange organization by modifying the resource schema. Once these custom resource properties have been added, they can then be assigned to specific resource mailboxes. You can use the following code to add a whiteboard resource property to the Exchange organization's resource schema: Set-ResourceConfig -ResourcePropertySchema 'Room/Whiteboard' Now that the whiteboard resource property is available within the Exchange organization, we can add this to our Conference Room 23 mailbox using the following command: Set-Mailbox -Identity CR23 -ResourceCustom Whiteboard When users access the Select Rooms or Add Rooms dialog box in Outlook 2007, 2010, or 2013, they will see that Conference Room 23 has a whiteboard available. Converting mailboxes If you've moved from an old version, you may have a number of mailboxes that were used as resource mailboxes. Once these mailboxes have been moved they will be identified as Shared mailboxes. You can convert them to different types using the Set-Mailbox cmdlet so that they'll have all of the properties of a resource mailbox: Get-Mailbox conf* | Set-Mailbox -Type Room You can run the Set-Mailbox cmdlet against each mailbox one at a time and convert them to Room mailboxes, using the -Type parameter. Or, if you use a common naming convention, you may be able to do them in bulk by retrieving a list of mailboxes using a wildcard and piping them to Set-Mailbox, as shown previously. Summary In this article we have discussed how the various types of mailboxes are managed, how to set up the Active Directory attributes, how to work with contacts and distribution groups. Resources for Article: Further resources on this subject: Unleashing Your Development Skills with PowerShell [Article] Inventorying Servers with PowerShell [Article] Exchange Server 2010 Windows PowerShell: Working with Address Lists [Article]

In this article, by Daniel Lélis Baggio, author of the book OpenCV 3.0 Computer Vision with Java, you will learn to perform basic operations required in computer vision, such as dealing with matrices, pixels, and opening files for prototype applications. In this article, the following topics will be covered: Basic matrix manipulation Pixel manipulation How to load and display images from files (For more resources related to this topic, see here.) Basic matrix manipulation From a computer vision background, we can see an image as a matrix of numerical values, which represents its pixels. For a gray-level image, we usually assign values ranging from 0 (black) to 255 (white) and the numbers in between show a mixture of both. These are generally 8-bit images. So, each element of the matrix refers to each pixel on the gray-level image, the number of columns refers to the image width, as well as the number of rows refers to the image's height. In order to represent a color image, we usually adopt each pixel as a combination of three basic colors: red, green, and blue. So, each pixel in the matrix is represented by a triplet of colors. It is important to observe that with 8 bits, we get 2 to the power of eight (28), which is 256. So, we can represent the range from 0 to 255, which includes, respectively the values used for black and white levels in 8-bit grayscale images. Besides this, we can also represent these levels as floating points and use 0.0 for black and 1.0 for white. OpenCV has a variety of ways to represent images, so you are able to customize the intensity level through the number of bits considering whether one wants signed, unsigned, or floating point data types, as well as the number of channels. OpenCV's convention is seen through the following expression: CV_<bit_depth>{U|S|F}C(<number_of_channels>) Here, U stands for unsigned, S for signed, and F stands for floating point. For instance, if an 8-bit unsigned single-channel image is required, the data type representation would be CV_8UC1, while a colored image represented by 32-bit floating point numbers would have the data type defined as CV_32FC3. If the number of channels is omitted, it evaluates to 1. We can see the ranges according to each bit depth and data type in the following list: CV_8U: These are the 8-bit unsigned integers that range from 0 to 255 CV_8S: These are the 8-bit signed integers that range from -128 to 127 CV_16U: These are the 16-bit unsigned integers that range from 0 to 65,535 CV_16S: These are the 16-bit signed integers that range from -32,768 to 32,767 CV_32S: These are the 32-bit signed integers that range from -2,147,483,648 to 2,147,483,647 CV_32F: These are the 32-bit floating-point numbers that range from -FLT_MAX to FLT_MAX and include INF and NAN values CV_64F: These are the 64-bit floating-point numbers that range from -DBL_MAX to DBL_MAX and include INF and NAN values You will generally start the project from loading an image, but it is important to know how to deal with these values. Make sure you import org.opencv.core.CvType and org.opencv.core.Mat. Several constructors are available for matrices as well, for instance: Mat image2 = new Mat(480,640,CvType.CV_8UC3); Mat image3 = new Mat(new Size(640,480), CvType.CV_8UC3); Both of the preceding constructors will construct a matrix suitable to fit an image with 640 pixels of width and 480 pixels of height. Note that width is to columns as height is to rows. Also pay attention to the constructor with the Size parameter, which expects the width and height order. In case you want to check some of the matrix properties, the methods rows(), cols(), and elemSize() are available: System.out.println(image2 + "rows " + image2.rows() + " cols " + image2.cols() + " elementsize " + image2.elemSize()); The output of the preceding line is: Mat [ 480*640*CV_8UC3, isCont=true, isSubmat=false, nativeObj=0xceeec70, dataAddr=0xeb50090 ]rows 480 cols 640 elementsize 3 The isCont property tells us whether this matrix uses extra padding when representing the image, so that it can be hardware-accelerated in some platforms; however, we won't cover it in detail right now. The isSubmat property refers to fact whether this matrix was created from another matrix and also whether it refers to the data from another matrix. The nativeObj object refers to the native object address, which is a Java Native Interface (JNI) detail, while dataAddr points to an internal data address. The element size is measured in the number of bytes. Another matrix constructor is the one that passes a scalar to be filled as one of its elements. The syntax for this looks like the following: Mat image = new Mat(new Size(3,3), CvType.CV_8UC3, new Scalar(new double[]{128,3,4})); This constructor will initialize each element of the matrix with the triple {128, 3, 4}. A very useful way to print a matrix's contents is using the auxiliary method dump() from Mat. Its output will look similar to the following: [128, 3, 4, 128, 3, 4, 128, 3, 4; 128, 3, 4, 128, 3, 4, 128, 3, 4; 128, 3, 4, 128, 3, 4, 128, 3, 4] It is important to note that while creating the matrix with a specified size and type, it will also immediately allocate memory for its contents. Pixel manipulation Pixel manipulation is often required for one to access pixels in an image. There are several ways to do this and each one has its advantages and disadvantages. A straightforward method to do this is the put(row, col, value) method. For instance, in order to fill our preceding matrix with values {1, 2, 3}, we will use the following code: for(int i=0;i<image.rows();i++){ for(int j=0;j<image.cols();j++){    image.put(i, j, new byte[]{1,2,3}); } } Note that in the array of bytes {1, 2, 3}, for our matrix, 1 stands for the blue channel, 2 for the green, and 3 for the red channel, as OpenCV stores its matrix internally in the BGR (blue, green, and red) format. It is okay to access pixels this way for small matrices. The only problem is the overhead of JNI calls for big images. Remember that even a small 640 x 480 pixel image has 307,200 pixels and if we think about a colored image, it has 921,600 values in a matrix. Imagine that it might take around 50 ms to make an overloaded call for each of the 307,200 pixels. On the other hand, if we manipulate the whole matrix on the Java side and then copy it to the native side in a single call, it will take around 13 ms. If you want to manipulate the pixels on the Java side, perform the following steps: Allocate memory with the same size as the matrix in a byte array. Put the image contents into that array (optional). Manipulate the byte array contents. Make a single put call, copying the whole byte array to the matrix. A simple example that will iterate all image pixels and set the blue channel to zero, which means that we will set to zero every element whose modulo is 3 equals zero, that is {0, 3, 6, 9, …}, as shown in the following piece of code: public void filter(Mat image){ int totalBytes = (int)(image.total() * image.elemSize()); byte buffer[] = new byte[totalBytes]; image.get(0, 0,buffer); for(int i=0;i<totalBytes;i++){    if(i%3==0) buffer[i]=0; } image.put(0, 0, buffer); } First, we find out the number of bytes in the image by multiplying the total number of pixels (image.total) with the element size in bytes (image.elemenSize). Then, we build a byte array with that size. We use the get(row, col, byte[]) method to copy the matrix contents in our recently created byte array. Then, we iterate all bytes and check the condition that refers to the blue channel (i%3==0). Remember that OpenCV stores colors internally as {Blue, Green, Red}. We finally make another JNI call to image.put, which copies the whole byte array to OpenCV's native storage. An example of this filter can be seen in the following screenshot, which was uploaded by Mromanchenko, licensed under CC BY-SA 3.0: Be aware that Java does not have any unsigned byte data type, so be careful when working with it. The safe procedure is to cast it to an integer and use the And operator (&) with 0xff. A simple example of this would be int unsignedValue = myUnsignedByte & 0xff;. Now, unsignedValue can be checked in the range of 0 to 255. Loading and displaying images from files Most computer vision applications need to retrieve images from some where. In case you need to get them from files, OpenCV comes with several image file loaders. Unfortunately, some loaders depend on codecs that sometimes aren't shipped with the operating system, which might cause them not to load. From the documentation, we see that the following files are supported with some caveats: Windows bitmaps: *.bmp, *.dib JPEG files: *.jpeg, *.jpg, *.jpe JPEG 2000 files: *.jp2 Portable Network Graphics: *.png Portable image format: *.pbm, *.pgm, *.ppm Sun rasters: *.sr, *.ras TIFF files: *.tiff, *.tif Note that Windows bitmaps, the portable image format, and sun raster formats are supported by all platforms, but the other formats depend on a few details. In Microsoft Windows and Mac OS X, OpenCV can always read the jpeg, png, and tiff formats. In Linux, OpenCV will look for codecs supplied with the OS, as stated by the documentation, so remember to install the relevant packages (do not forget the development files, for example, "libjpeg-dev" in Debian* and Ubuntu*) to get the codec support or turn on the OPENCV_BUILD_3RDPARTY_LIBS flag in CMake, as pointed out in Imread's official documentation. The imread method is supplied to get access to images through files. Use Imgcodecs.imread (name of the file) and check whether dataAddr() from the read image is different from zero to make sure the image has been loaded correctly, that is, the filename has been typed correctly and its format is supported. A simple method to open a file could look like the one shown in the following code. Make sure you import org.opencv.imgcodecs.Imgcodecs and org.opencv.core.Mat: public Mat openFile(String fileName) throws Exception{ Mat newImage = Imgcodecs.imread(fileName);    if(newImage.dataAddr()==0){      throw new Exception ("Couldn't open file "+fileName);    } return newImage; } Displaying an image with Swing OpenCV developers are used to a simple cross-platform GUI by OpenCV, which was called as HighGUI, and a handy method called imshow. It constructs a window easily and displays an image within it, which is nice to create quick prototypes. As Java comes with a popular GUI API called Swing, we had better use it. Besides, no imshow method was available for Java until its 2.4.7.0 version was released. On the other hand, it is pretty simple to create such functionality. Let's break down the work in to two classes: App and ImageViewer. The App class will be responsible for loading the file, while ImageViewer will display it. The application's work is simple and will only need to use Imgcodecs's imread method, which is shown as follows: package org.javaopencvbook;   import java.io.File; … import org.opencv.imgcodecs.Imgcodecs;   public class App { static{ System.loadLibrary(Core.NATIVE_LIBRARY_NAME); }   public static void main(String[] args) throws Exception { String filePath = "src/main/resources/images/cathedral.jpg"; Mat newImage = Imgcodecs.imread(filePath); if(newImage.dataAddr()==0){    System.out.println("Couldn't open file " + filePath); } else{    ImageViewer imageViewer = new ImageViewer();    imageViewer.show(newImage, "Loaded image"); } } } Note that the App class will only read an example image file in the Mat object and it will call the ImageViewer method to display it. Now, let's see how the ImageViewer class's show method works: package org.javaopencvbook.util;   import java.awt.BorderLayout; import java.awt.Dimension; import java.awt.Image; import java.awt.image.BufferedImage;   import javax.swing.ImageIcon; import javax.swing.JFrame; import javax.swing.JLabel; import javax.swing.JScrollPane; import javax.swing.UIManager; import javax.swing.UnsupportedLookAndFeelException; import javax.swing.WindowConstants;   import org.opencv.core.Mat; import org.opencv.imgproc.Imgproc;   public class ImageViewer { private JLabel imageView; public void show(Mat image){    show(image, ""); }   public void show(Mat image,String windowName){    setSystemLookAndFeel();       JFrame frame = createJFrame(windowName);               Image loadedImage = toBufferedImage(image);        imageView.setIcon(new ImageIcon(loadedImage));               frame.pack();        frame.setLocationRelativeTo(null);        frame.setVisible(true);    }   private JFrame createJFrame(String windowName) {    JFrame frame = new JFrame(windowName);    imageView = new JLabel();    final JScrollPane imageScrollPane = new JScrollPane(imageView);        imageScrollPane.setPreferredSize(new Dimension(640, 480));        frame.add(imageScrollPane, BorderLayout.CENTER);        frame.setDefaultCloseOperation(WindowConstants.EXIT_ON_CLOSE);    return frame; }   private void setSystemLookAndFeel() {    try {      UIManager.setLookAndFeel (UIManager.getSystemLookAndFeelClassName());    } catch (ClassNotFoundException e) {      e.printStackTrace();    } catch (InstantiationException e) {      e.printStackTrace();    } catch (IllegalAccessException e) {      e.printStackTrace();    } catch (UnsupportedLookAndFeelException e) {      e.printStackTrace();    } }   public Image toBufferedImage(Mat matrix){    int type = BufferedImage.TYPE_BYTE_GRAY;    if ( matrix.channels() > 1 ) {      type = BufferedImage.TYPE_3BYTE_BGR;    }    int bufferSize = matrix.channels()*matrix.cols()*matrix.rows();    byte [] buffer = new byte[bufferSize];    matrix.get(0,0,buffer); // get all the pixels    BufferedImage image = new BufferedImage(matrix.cols(),matrix.rows(), type);    final byte[] targetPixels = ((DataBufferByte) image.getRaster().getDataBuffer()).getData();    System.arraycopy(buffer, 0, targetPixels, 0, buffer.length);    return image; }   } Pay attention to the show and toBufferedImage methods. Show will try to set Swing's look and feel to the default native look, which is cosmetic. Then, it will create JFrame with JScrollPane and JLabel inside it. It will then call toBufferedImage, which will convert an OpenCV Mat object to a BufferedImage AWT. This conversion is made through the creation of a byte array that will store matrix contents. The appropriate size is allocated through the multiplication of the number of channels by the number of columns and rows. The matrix.get method puts all the elements into the byte array. Finally, the image's raster data buffer is accessed through the getDataBuffer() and getData() methods. It is then filled with a fast system call to the System.arraycopy method. The resulting image is then assigned to JLabel and then it is easily displayed. Note that this method expects a matrix that is either stored as one channel's unsigned 8-bit or three channel's unsigned 8-bit. In case your image is stored as a floating point, you should convert it using the following code before calling this method, supposing that the image you need to convert is a Mat object called originalImage: Mat byteImage = new Mat(); originalImage.convertTo(byteImage, CvType.CV_8UC3); This way, you can call toBufferedImage from your converted byteImage property. The image viewer can be easily installed in any Java OpenCV project and it will help you to show your images for debugging purposes. The output of this program can be seen in the next screenshot: Summary In this article, we learned dealing with matrices, pixels, and opening files for GUI prototype applications. Resources for Article: Further resources on this subject: Wrapping OpenCV [article] Making subtle color shifts with curves [article] Linking OpenCV to an iOS project [article]

In this article by Francesco Marchioni, author of the book MongoDB for Java Developers, you will be able to perform all the create/read/update/delete (CRUD) operations that we have so far accomplished using the mongo shell. (For more resources related to this topic, see here.) Querying data We will now see how to use the Java API to query for your documents. Querying for documents with MongoDB resembles JDBC queries; the main difference is that the returned object is a com.mongodb.DBCursor class, which is an iterator over the database result. In the following example, we are iterating over the javastuff collection that should contain the documents inserted so far: package com.packtpub.mongo.chapter2;   import com.mongodb.DB; import com.mongodb.DBCollection; import com.mongodb.DBCursor; import com.mongodb.DBObject; import com.mongodb.MongoClient;   public class SampleQuery{   private final static String HOST = "localhost"; private final static int PORT = 27017;   public static void main( String args[] ){    try{      MongoClient mongoClient = new MongoClient( HOST,PORT );           DB db = mongoClient.getDB( "sampledb" );           DBCollection coll = db.getCollection("javastuff");      DBCursor cursor = coll.find();      try {        while(cursor.hasNext()) {          DBObject object = cursor.next();          System.out.println(object);        }      }      finally {        cursor.close();      }       }    catch(Exception e) {      System.err.println( e.getClass().getName() + ": " +       e.getMessage() );    } } } Depending on the documents you have inserted, the output could be something like this: { "_id" : { "$oid" : "5513f8836c0df1301685315b"} , "name" : "john" , "age" : 35 , "kids" : [ { "name" : "mike"} , { "name" : "faye"}] , "info" : { "email" : "john@mail.com" , "phone" : "876-134-667"}} . . . . Restricting the search to the first document The find operator executed without any parameter returns the full cursor of a collection; pretty much like the SELECT * query in relational DB terms. If you are interested in reading just the first document in the collection, you could use the findOne() operation to get the first document in the collection. This method returns a single document (instead of the DBCursor that the find() operation returns). As you can see, the findOne() operator directly returns a DBObject instead of a com.mongodb.DBCursor class: DBObject myDoc = coll.findOne(); System.out.println(myDoc); Querying the number of documents in a collection Another typical construct that you probably know from the SQL is the SELECT count(*) query that is useful to retrieve the number of records in a table. In MongoDB terms, you can get this value simply by invoking the getCount against a DBCollection class: DBCollection coll = db.getCollection("javastuff"); System.out.println(coll.getCount()); As an alternative, you could execute the count() method over the DBCursor object: DBCursor cursor = coll.find(); System.out.println(cursor.count()); Eager fetching of data using DBCursor When find is executed and a DBCursor is executed you have a pointer to a database document. This means that the documents are fetched in the memory as you call next() method on the DBCursor. On the other hand, you can eagerly load all the data into the memory by executing the toArray() method, which returns a java.util.List structure: List list = collection.find( query ).toArray(); The problem with this approach is that you could potentially fill up the memory with lots of documents, which are eagerly loaded. You are therefore advised to include some operators such as skip() and limit() to control the amount of data to be loaded into the memory: List list = collection.find( query ).skip( 100 ). limit( 10 ).toArray(); Just like you learned from the mongo shell, the skip operator can be used as an initial offset of your cursor whilst the limit construct can eventually load the first n occurrences in the cursor. Filtering through the records Typically, you will not need to fetch the whole set of documents in a collection. So, just like SQL uses WHERE conditions to filter records, in MongoDB you can restrict searches by creating a BasicDBObject and passing it to the find function as an argument. See the following example: DBCollection coll = db.getCollection("javastuff");   DBObject query = new BasicDBObject("name", "owen");   DBCursor cursor = coll.find(query);   try {    while(cursor.hasNext()) {        System.out.println(cursor.next());    } } finally {    cursor.close(); } In the preceding example, we retrieve the documents in the javastuff collection, whose name key equals to owen. That's the equivalent of an SQL query like this: SELECT * FROM javastuff WHERE name='owen' Building more complex searches As your collections keep growing, you will need to be more selective with your searches. For example, you could include multiple keys in your BasicDBObject that will eventually be passed to find. We can then apply the same functions in our queries. For example, here is how to find documents whose name does not equal ($ne) to Frank and whose age is greater than 10: DBCollection coll = db.getCollection("javastuff");   DBObject query = new      BasicDBObject("name", new BasicDBObject("$ne",        "frank")).append("age", new BasicDBObject("$gt", 10));   DBCursor cursor = coll.find(query); Updating documents Having learned about create and read, we are half way through our CRUD track. The next operation you will learn is update. The DBCollection class contains an update method that can be used for this purpose. Let's say we have the following document: > db.javastuff.find({"name":"frank"}).pretty() { "_id" : ObjectId("55142c27627b27560bd365b1"), "name" : "frank", "age" : 31, "info" : { "email" : "frank@mail.com", "phone" : "222-111-444" } } Now we want to change the age value for this document by setting it to 23: DBCollection coll = db.getCollection("javastuff");   DBObject newDocument = new BasicDBObject(); newDocument.put("age", 23);   DBObject searchQuery = new BasicDBObject().append("name", "owen");   coll.update(searchQuery, newDocument); You might think that would do the trick, but wait! Let's have a look at our document using the mongo shell: > db.javastuff.find({"age":23}).pretty()   { "_id" : ObjectId("55142c27627b27560bd365b1"), "age" : 23 } As you can see, the update statement has replaced the original document with another one, including only the keys and values we have passed to the update. In most cases, this is not what we want to achieve. If we want to update a particular value, we have to use the $set update modifier. DBCollection coll = db.getCollection("javastuff");   BasicDBObject newDocument = new BasicDBObject(); newDocument.append("$set", new BasicDBObject().append("age",   23));   BasicDBObject searchQuery = new BasicDBObject().append("name", "frank");   coll.update(searchQuery, newDocument); So, suppose we restored the initial document with all the fields, this is the outcome of the update using the $set update modifier: > db.javastuff.find({"age":23}).pretty() { "_id" : ObjectId("5514326e627b383428c2ccd8"), "name" : "frank", "age" : 23, "in,fo" : {    "email" : "frank@mail.com",    "phone" : "222-111-444" } } Please note that the DBCollection class overloads the method update with update (DBObject q, DBObject o, boolean upsert, boolean multi). The first parameter (upsert) determines whether the database should create the element if it does not exist. The second one (multi) causes the update to be applied to all matching objects. Deleting documents The operator to be used for deleting documents is obviously delete. As for other operators, it includes several variants. In its simplest form, when executed over a single document returned, it will remove it: MongoClient mongoClient = new MongoClient("localhost", 27017);   DB db = mongoClient.getDB("sampledb"); DBCollection coll = db.getCollection("javastuff"); DBObject doc = coll.findOne();   coll.remove(doc); Most of the time you will need to filter the documents to be deleted. Here is how to delete the document with the key frank: DBObject document = new BasicDBObject(); document.put("name", "frank");   coll.remove(document); Deleting a set of documents Bulk deletion of documents can be achieved by including the keys in a List and building an $in modifier expression that uses this list. Let's see, for example, how to delete all records whose age ranges from 0 to 49: BasicDBObject deleteQuery = new BasicDBObject(); List<Integer> list = new ArrayList<Integer>();   for (int i=0;i<50;i++) list.add(i);   deleteQuery.put("age", new BasicDBObject("$in", list)); Summary In this article, we covered how to perform the same operations that are available in the mongo shell. Resources for Article: Further resources on this subject: MongoDB data modeling [article] Apache Solr and Big Data – integration with MongoDB [article] About MongoDB [article]

XAML is also known as Extensible Application Markup Language. XAML is a generic language like XML and can be used for multiple purposes and in the WPF and Silverlight applications. XAML is majorly used to declaratively design user interfaces. In this article by Abhishek Shukla, author of the book Blend for Visual Studio 2012 by Example Beginner's Guide, we had a look at the various layout controls and at how to use them in our application. In this article, we will have a look at the following topics: The fundamentals of XAML The use of XAML for applications in Blend (For more resources related to this topic, see here.) An important point to note here is that almost everything that we can do using XAML can also be done using C#. The following is the XAML and C# code to accomplish the same task of adding a rectangle within a canvas. In the XAML code, a canvas is created and an instance of a rectangle is created, which is placed inside the canvas: XAML code: In the following code, we declare the Rectangle element within Canvas, and that makes the Rectangle element the child of Canvas. The hierarchy of the elements defines the parent-child relationship of the elements. When we declare an element in XAML, it is the same as initializing it with a default constructor, and when we set an attribute in XAML, it is equivalent to setting the same property or event handler in code. In the following code, we set the various properties of Rectangle, such as Height, Width, and so on: <Canvas>    <Rectangle Height="100" Width="250" Fill="AliceBlue"               StrokeThickness="5" Stroke="Black" /> </Canvas> C# code: We created Canvas and Rectangle. Then, we set a few properties of the Rectangle element and then placed Rectangle inside Canvas: Canvas layoutRoot = new Canvas(); Rectangle rectangle = new Rectangle();   rectangle.Height = 100; rectangle.Width = 250; rectangle.Fill = Brushes.AliceBlue; rectangle.StrokeThickness = 5; rectangle.Stroke = Brushes.Black;   layoutRoot.Children.Add(rectangle);   AddChild(layoutRoot); We can clearly see why XAML is the preferred choice to define UI elements—XAML code is shorter, more readable, and has all the advantages that a declarative language provides. Another major advantage of working with XAML rather than C# is instant design feedback. As we type XAML code, we see the changes on the art board instantly. Whereas in the case of C#, we have to run the application to see the changes. Thanks to XAML, the process of creating a UI is now more like visual design than code development. When we drag and drop an element or draw an element on the art board, Blend generates XAML in the background. This is helpful as we do not need to hand-code XAML as we would have to when working with text-editing tools. Generally, the order of attaching properties and event handlers is performed based on the order in which they are defined in the object element. However, this should not matter, ideally, because, as per the design guidelines, the classes should allow properties and event handlers to be specified in any order. Wherever we create a UI, we should use XAML, and whatever relates to data should be processed in code. XAML is great for UIs that may have logic, but XAML is not intended to process data, which should be prepared in code and posted to the UI for displaying purposes. It is data processing that XAML is not designed for. The basics of XAML Each element in XAML maps to an instance of a .NET class, and the name of the element is exactly the same as the class. For example, the <Button> element in XAML is an instruction to create an instance of the Button class. XAML specifications define the rules to map the namespaces, types, events, and properties of object-oriented languages to XML namespaces. Time for action – taking a look at XAML code Perform the following steps and take a look at the XAML namespaces after creating a WPF application: Let's create a new WPF project in Expression Blend and name it Chapter03. In Blend, open MainWindow.xaml, and click on the split-view option so that we can see both the design view as well as the XAML view. The following screenshot shows this: You will also notice that a grid is present under the window and there is no element above the window, which means that Window is the root element for the current document. We see this in XAML, and we will also see multiple attributes set on Window: <Window x_Class="Chapter03.MainWindow" Title="MainWindow" Height="350" Width="525">    We can see in the preceding code that the XAML file has a root element. For a XAML document to be valid, there should be one and only one root element. Generally, we have a window, page, or user control as the root element. Other root elements that are used are ResourceDictionary for dictionaries and Application for application definition.    The Window element also contains a few attributes, including a class name and two XML namespaces. The three properties (Title, Height, and Width) define the caption of the window and default size of the window. The class name, as you might have guessed, defines the class name of the window. The http://schemas.microsoft.com/winfx/2006/xaml/presentation is is the default namespace. The default namespace allows us to add elements to the page without specifying any prefix. So, all other namespaces defined must have a unique prefix for reference. For example, the namespace http://schemas.microsoft.com/winfx/2006/xaml is defined with the prefix :x. The prefix that is mapped to the schema allows us to reference this namespace just using the prefix instead of the full-schema namespace.    XML namespaces don't have a one-to-one mapping with .NET namespaces. WPF types are spread across multiple namespaces, and one-to-one mapping would be rather cumbersome. So, when we refer to the presentation, various namespaces, such as System.Windows and System.Windows.Controls, are included.    All the elements and their attributes defined in MainPage.xaml should be defined in at least one of the schemas mentioned in the root element of XAML; otherwise, the XAML document will be invalid, and there is no guarantee that the compiler will understand and continue. When we add Rectangle in XAML, we expect Rectangle and its attributes to be part of the default schema: <Rectangle x_Name="someRectangle" Fill="AliceBlue"/> What just happened? We had a look at the XAML namespaces that we use when we create a WPF application. Time for action – adding other namespaces in XAML In this section, we will add another namespace apart from the default namespace: We can use any other namespace in XAML as well. To do that, we need to declare the XML namespaces the schemas of which we want to adhere to. The syntax to do that is as follows: Prefix is the XML prefix we want to use in our XAML to represent that namespace. For example, the XAML namespace uses the :x prefix. Namespace is the fully qualified .NET namespace. AssemblyName is the assembly where the type is declared, and this assembly could be the current project assembly or a referenced assembly. Open the XAML view of MainWindow.xaml if it is not already open, and add the following line of code after the reference to To create an instance of an object we would have to use this namespace prefix as <system:Double></system:Double> We can access the types defined in the current assembly by referencing the namespace of the current project:   To create an instance of an object we would have to use this namespace prefix as   <local:MyObj></local:MyObj> What just happened? We saw how we can add more namespaces in XAML apart from the ones present by default and how we can use them to create objects. Naming elements In XAML, it is not mandatory to add a name to every element, but we might want to name some of the XAML elements that we want to access in the code or XAML. We can change properties or attach the event handler to, or detach it from, elements on the fly. We can set the name property by hand-coding in XAML or setting it in the properties window. This is shown in the following screenshot: The code-behind class The x:Class attribute in XAML references the code-behind class for XAML. You would notice the x:, which means that the Class attribute comes from the XAML namespace, as discussed earlier. The value of the attribute references the MainWindow class in the Chapter03 namespace. If we go to the MainWindow.xaml.cs file, we can see the partial class defined there. The code-behind class is where we can put C# (or VB) code for the implementation of event handlers and other application logic. As we discussed earlier, it is technically possible to create all of the XAML elements in code, but that bypasses the advantages of having XAML. So, as these two are partial classes, they are compiled into one class. So, as C# and XAML are equivalent and both these are partial classes, they are compiled into the same IL. Time for action – using a named element in a code-behind class Go to MainWindow.xaml.cs and change the background of the LayoutRoot grid to Green: public MainWindow() {    InitializeComponent();    LayoutRoot.Background = Brushes.Green; } Run the application; you will see that the background color of the grid is green. What just happened? We accessed the element defined in XAML by its name. Default properties The content of a XAML element is the value that we can simply put between the tags without any specific references. For example, we can set the content of a button as follows: <Button Content="Some text" /> or <Button > Some text </Button> The default properties are specified in the help file. So, all we need to do is press F1 on any of our controls to see what the value is. Expressing properties as attributes We can express the properties of an element as an XML attribute. Time for action – adding elements in XAML by hand-coding In this section, instead of dragging and dropping controls, we will add XAML code: Move MainWindow.xaml to XAML and add the code shown here to add TextBlock and three buttons in Grid: <Grid x_Name="LayoutRoot"> <TextBlock Text="00:00" Height="170" Margin="49,32,38,116" Width="429" FontSize="48"/>    <Button Content="Start" Height="50" Margin="49,220,342,49"     Width="125"/>    <Button Content="Stop" Height="50" Margin="203,220,188,49"     Width="125"/>    <Button Content="Reset" Height="50" Margin="353,220,38,49"    Width="125"/> </Grid> We set a few properties for each of the elements. The property types of the various properties are also mentioned. These are the .NET types to which the type converter would convert them:    Content: This is the content displayed onscreen. This property is of the Object type.    Height: This is the height of the button. This property is of the Double type.    Width: This is the width of the button. This property is of the Double type.    Margin: This is the amount of space outside the control, that is, between the edge of the control and its container. This property is of the Thickness type.    Text: This is the text displayed onscreen. This property is of the String type.    FontSize: This is the size of the font of the text. This property is of the Double type. What just happened? We added elements in XAML with properties as XML attributes. Non-attribute syntax We will define a gradient background for the grid, which is a complex property. Notice that the code in the next section sets the background property of the grid using a different type converter this time. Instead of a string-to-brush converter, the LinearGradientBrush to brush type converter would be used. Time for action – defining the gradient for the grid Add the following code inside the grid. We have specified two gradient stops. The first one is black and the second one is a shade of green. We have specified the starting point of LinearGradientBrush as the top-left corner and the ending point as the bottom-right corner, so we will see a diagonal gradient: <Grid x_Name="LayoutRoot"> <Grid.Background>    <LinearGradientBrush EndPoint="1,1" StartPoint="0,0">      <GradientStop Color="Black"/>      <GradientStop Color="#FF27EC07" Offset="1"/>    </LinearGradientBrush> </Grid.Background> The following is the output of the preceding code: Comments in XAML Using <!-- --> tags, we can add comments in XAML just as we add comments in XML. Comments are really useful when we have complex XAML with lots of elements and complex layouts: <!-- TextBlock to show the timer --> Styles in XAML We would like all the buttons in our application to look the same, and we can achieve this using styles. Here, we will just see how styles can be defined and used in XAML. Defining a style We will define a style as a static resource in the window. We can move all the properties we define for each button to that style. Time for action – defining style in XAML Add a new <Window.Resources> tag in XAML, and then add the code for the style, as shown here: <Window.Resources> <Style TargetType="Button" x_Key="MyButtonStyle">    <Setter Property="Height" Value="50" />    <Setter Property="Width" Value="125" />    <Setter Property="Margin" Value="0,10" />    <Setter Property="FontSize" Value="18"/>    <Setter Property="FontWeight" Value="Bold" />    <Setter Property="Background" Value="Black" />    <Setter Property="Foreground" Value="Green" /> </Style> </Window.Resources> We defined a few properties on style that are worth noting: TargetType: This property specifies the type of element to which we will apply this style. In our case, it is Button. x:Key: This is the unique key to reference the style within the window. Setter: The setter elements contain the property name and value. What just happened We defined a style for a button in the window. Using a style Let's use the style that we defined in the previous section. All UI controls have a style property (from the FrameworkElement base class). Time for action – defining style in XAML To use a style, we have to set the style property of the element as shown in the following code. Add the same style code to all the buttons:    We set the style property using curly braces ({}) because we are using another element.    StaticResource denotes that we are using another resource.    MyButtonStyle is the key to refer to the style. The following code encapsulates the style properties: <Button x_Name="BtnStart" Content="Start" Grid.Row="1" Grid.Column="0" Style="{StaticResource MyButtonStyle}"/> The following is the output of the preceding code: What just happened? We used a style defined for a button. Where to go from here This article gave a brief overview of XAML, which helps you to start designing your applications in Expression Blend. However, if you wish know more about XAML, you could visit the following MSDN links and go through the various XAML specifications and details: XAML in Silverlight: http://msdn.microsoft.com/en-us/library/cc189054(v=vs.95).aspx XAML in WPF: http://msdn.microsoft.com/en-us/library/ms747122(v=vs.110).aspx Pop quiz Q1. How can we find out the default property of a control? F1. F2. F3. F4. Q2. Is it possible to create a custom type converter? Yes. No. Summary We had a look at the basics of XAML, including namespaces, elements, and properties. With this introduction, you can now hand-edit XAML, where necessary, and this will allow you to tweak the output of Expression Blend or even hand-code an entire UI if you are more comfortable with that. Resources for Article: Further resources on this subject: Windows Phone 8 Applications [article] Building UI with XAML for Windows 8 Using C [article] Introduction to Modern OpenGL [article]

In this article by Omar Khedher, author of the book Mastering OpenStack, we will create a VPN setup between two sites using the Neutron VPN driver plugin in Neutron. The VPN as a Service is a new network functionality that is provided by Neutron, which is the network service that introduces the VPN feature set. It was completely integrated in OpenStack since the Havana release. We will set up two private networks in two different OpenStack sites and create some IPSec site-to-site connections. The instances that are located in each OpenStack private network should be able to connect to each other across the VPN tunnel. The article assumes that you have two OpenStack environments in different networks. General settings The following figure illustrates the VPN topology between two OpenStack sites to provide a secure connection between a database instance that runs in a data center in Amsterdam along with a web server that runs in a data center in Tunis. Each site has a private and a public network as well as subnets that are managed by a neutron router. Enabling the VPNaaS The VPN driver plugin must be configured in /etc/neutron/neutron.conf. To enable it, add the following driver: # nano /etc/neutron/neutron.conf service_plugins = neutron.services.vpn.plugin.VPNDriverPlugin Next, we'll add the VPNaaS module interface in /usr/share/openstack-dashboard/openstack_dashboard/local/local_settings.py, as follows: 'enable_VPNaaS': True, Finally, restart the neutron-server and server web using the following commands: # service httpd restart # /etc/init.d/neutron-server restart Site's configuration Let's assume that each OpenStack site has at least a router attached within an external gateway and can provide access to the private network attached to an instance, such as a database or a web server virtual machine. A simple network topology of a private OpenStack environment running in the Amsterdam site will look like this: The private OpenStack environment running in the Tunis site will look like the following image: VPN management in the CLI Neutron offers several commands that can be used to create and manage the VPN connections in OpenStack. The essential commands that are associated with router management include the following: vpn-service-create vpn-service-delete vpn-service-list vpn-ikepolicy-create vpn-ikepolicy-list vpn-ipsecpolicy-create vpn-ipsecpolicy-delete vpn-ipsecpolicy-list vpn-ipsecpolicy-show ipsec-site-connection-create ipsec-site-connection-delete ipsec-site-connection-list Creating an IKE policy In the first VPN phase, we can create an IKE policy in Horizon. The following figure shows a simple IKE setup of the OpenStack environment that is located in the Amsterdam site: The same settings can be applied via the Neutron CLI using the vpn-ikepolicy-create command, as follows: Syntax: neutron vpn-ikepolicy-create --description <description> --auth-algorithm <auth-algorithm> --encryption-algorithm <encryption-algorithm> --ike-version <ike-version> --lifetime <units=UNITS, value=VALUE> --pfs <pfs> --phase1-negotiation-mode <phase1-negotiation-mode> --name <NAME> Creating an IPSec policy An IPSec policy in an OpenStack environment that is located in the Amsterdam site can be created in Horizon in the following way: The same settings can be applied via the Neutron CLI using the vpn-ipsecpolicy-create command, as follows: Syntax: neutron vpn-ipsecpolicy-create --description <description> --auth-algorithm <auth-algorithm> --encapsulation-mode <encapsulation-mode> --encryption-algorithm <encryption-algorithm> --lifetime <units=UNITS,value=VALUE> --pfs <pfs> --transform-protocol <transform-algorithm> --name <NAME> Creating a VPN service To create a VPN service, you will need to specify the router facing the external and attach the web server instance to the private network in the Amsterdam site. The router will act as a VPN gateway. We can add a new VPN service from Horizon in the following way: The same settings can be applied via the Neutron CLI using the vpn-service-create command, as follows: Syntax: neutron vpn-service-create --tenant-id <tenant-id> --description <description> ROUTER SUBNET --name <name> Creating an IPSec connection The following step requires an identification of the peer gateway of the remote site that is located in Tunis. We will need to collect the IPv4 address of the external gateway interface of the remote site as well as the remote private subnet. In addition to this, a pre-shared key (PSK) has to be defined and exchanged between both the sites in order to bring the tunnel up after establishing a successful PSK negotiation during the second phase of the VPN setup. You can collect the router information either from Horizon or from the command line. To get subnet-related information on the OpenStack Cloud that is located in the Tunis site, you can run the following command in the network node: # neutron router-list The preceding command yields the following result: We can then list the ports of Router-TN and check the attached subnets of each port using the following command line: # neutron router-port-list Router-TN The preceding command gives the following result: Now, we can add a new IPSec site connection from the Amsterdam OpenStack Cloud in the following way: Similarly, we can perform the same configuration to create the VPN service using the neutron ipsec-site-connection-create command line, as follows: Syntax: neutron ipsec-site-connection-create --name <NAME> --description <description> ---vpnservice-id <VPNSERVICE> --ikepolicy-id <IKEPOLICY> --ipsecpolicy-id <IPSECPOLICY> --peer-address <PEER-ADDRESS> --peer-id <PEER-ID> --psk <PRESHAREDKEY> Remote site settings A complete VPN setup requires you to perform the same steps in the OpenStack Cloud that is located in the Tunis site. For a successful VPN phase 1 configuration, you must set the same IKE and IPSec policy attributes and change only the naming convention for each IKE and IPSec setup. Creating a VPN service on the second OpenStack Cloud will look like this: The last tricky part involves gathering the same router information in the Amsterdam site. Using the command line in the network node in the Amsterdam Cloud side, we can perform the following Neutron command: # neutron router-list The preceding command yields the following result: To get a list of networks that are attached to Router-AMS, we can execute the following command line: # neutron router-port-list Router-AMS The preceding command gives the following result: The external and private subnets that are associated with Router-AMS can be checked from Horizon as well. Now that we have the peer router gateway and remote private subnet, we will need to fill the same PSK that was configured previously. The next figure illustrates the new IPSec site connection on the OpenStack Cloud that is located in the Tunis site: At the Amsterdam site, we can check the creation of the new IPSec site connection by means of the neutron CLI, as follows: # neutron ipsec-site-connection-list The preceding command will give the following result: The same will be done at the Tunis site, as follows: # neutron ipsec-site-connection-list The preceding command gives the following result: Managing security groups For VPNaaS to work when connecting the Amsterdam and Tunis subnets, you will need to create a few additional rules in each project's default security group to enable not only the pings by adding a general ICMP rule, but also SSH on port 22. Additionally, we can create a new security group called Application_PP to restrict traffic on port 53 (DNS), 80 (HTTP), and 443 (HTTPS), as follows: # neutron security-group-create Application_PP --description "allow web traffic from the Internet" # neutron security-group-rule-create --direction ingress --protocol tcp --port_range_min 80 --port_range_max 80 Application_PP --remote-ip-prefix 0.0.0.0/0 # neutron security-group-rule-create --direction ingress --protocol tcp --port_range_min 53 --port_range_max 53 Application_PP --remote-ip-prefix 0.0.0.0/0 # neutron security-group-rule-create --direction ingress --protocol tcp --port_range_min 443 --port_range_max 443 Application_PP --remote-ip-prefix 0.0.0.0/0 From Horizon, we will see the following security group rules added: Summary In this article, we created a VPN setup between two sites using the Neutron VPN driver plugin in Neutron. This process included enabling the VPNaas, configuring the site, and creating an IKE policy, an IPSec policy, a VPN service, and an IPSec connection. To continue Mastering OpenStack, take a look to see what else you will learn in the book here.

In this article by Carlos A. Méndez, the author of the book Learning Ext JS - Fourth Edition, we will see some of the important features in Ext JS. When learning a new technology such as Ext JS, some developers face a hard time to begin with, so this article will cover up certain important points that have been included in the recent version of Ext JS. We will be referencing certain online documentations, blogs and forums looking for answers, trying to figure out how the library and all the components work together. Even though there are tutorials in the official learning center, it would be great to have a guide to learn the library from the basics to a more advanced level. Ext JS is a state-of-the-art framework to create Rich Internet Applications (RIAs). The framework allows us to create cross-browser applications with a powerful set of components and widgets. The idea behind the framework is to create user-friendly applications in rapid development cycles, facilitate teamwork (MVC or MVVM), and also have a long-term maintainability. Ext JS is not just a library of widgets anymore; the brand new version is a framework full of new exciting features for us to play with. Some of these features are the new class system, the loader, the new application package, which defines a standard way to code our applications, and much more awesome stuff. The company behind the Ext JS library is Sencha Inc. They work on great products that are based on web standards. Some of the most famous products that Sencha also have are Sencha Touch and Sencha Architect. In this article, we will cover some of the basic concepts of the framework of version 5 and take a look at some of the new features in Ext JS 5. (For more resources related to this topic, see here.) Considering Ext JS for your next project Ext JS is a great library to create RIAs that require a lot of interactivity with the user. If you need complex components to manage your information, then Ext is your best option because it contains a lot of widgets such as the grid, forms, trees, panels, and a great data package and class system. Ext JS is best suited for enterprise or intranet applications; it's a great tool to develop an entire CRM or ERP software solution. One of the more appealing examples is the Desktop sample (http://dev.sencha.com/ext/5.1.0/examples/desktop/index.html). It really looks and feels like a native application running in the browser. In some cases, this is an advantage because the users already know how to interact with the components and we can improve the user experience. Ext JS 5 came out with a great tool to create themes and templates in a very simple way. The framework for creating themes is built on top of Compass and Sass, so we can modify some variables and properties and in a few minutes we can have a custom template for our Ext JS applications. If we want something more complex or unique, we can modify the original template to suit our needs. This might be more time-consuming depending on our experience with Compass and Sass. Compass and Sass are extensions for CSS. We can use expressions, conditions, variables, mixins, and many more awesome things to generate well-formatted CSS. You can learn more about Compass on their website at http://compass-style.org/. The new class system allows us to define classes incredibly easily. We can develop our application using the object-oriented programming paradigm and take advantage of the single and multiple inheritances. This is a great advantage because we can implement any of the available patterns such as MVC, MVVM, Observable, or any other. This will allow us to have a good code structure, which leads us to have easy access for maintenance. Another thing to keep in mind is the growing community around the library; there are lot of people around the world that are working with Ext JS right now. You can even join the meeting groups that have local reunions frequently to share knowledge and experiences; I recommend you to look for a group in your city or create one. The new loader system is a great way to load our modules or classes on demand. We can load only the modules and applications that the user needs just in time. This functionality allows us to bootstrap our application faster by loading only the minimal code for our application to work. One more thing to keep in mind is the ability to prepare our code for deployment. We can compress and obfuscate our code for a production environment using the Sencha Command, a tool that we can run on our terminal to automatically analyze all the dependencies of our code and create packages. Documentation is very important and Ext JS has great documentation, which is very descriptive with a lot of examples, videos, and sample code so that we can see it in action right on the documentation pages, and we can also read the comments from the community. What's new in Ext JS 5 Ext JS 5 introduces a great number of new features, and we'll briefly cover a few of the significant additions in version 5 as follows: Tablet support and new themes: This has introduced the ability to create apps compatible with touch-screen devices (touch-screen laptops, PCs, and tablets). The Crisp theme is introduced and is based on the Neptune theme. Also, there are new themes for tablet support, which are Neptune touch and Crisp touch. New application architecture – MVVM: Adding a new alternative to MVC Sencha called MVVM (which stands for Model-View-ViewModel), this new architecture has data binding and two-way data binding, allowing us to decrease much of the extra code that some of us were doing in past versions. This new architecture introduces: Data binding View controllers View models Routing: Routing provides deep linking of application functionality and allows us to perform certain actions or methods in our application by translating the URL. This gives us the ability to control the application state, which means that we can go to a specific part or a direct link to our application. Also, it can handle multiple actions in the URL. Responsive configurations: Now we have the ability to set the responsiveConfig property (new property) to some components, which will be a configuration object that represents conditions and criteria on which the configurations set will be applied, if the rule meets these configurations. As an example: responsiveConfig: { 'width > 800': { region: 'west' }, 'width <= 800':{ region: 'north' } } Data package improvements: Some good changes came in version 5 relating to data handling and data manipulation. These changes allowed developers an easier journey in their projects, and some of the new things are: Common Data (the Ext JS Data class, Ext.Data, is now part of the core package) Many-to-many associations Chained stores Custom field types Event system: The event logic was changed, and is now a single listener attached at the very top of the DOM hierarchy. So this means when a DOM element fires an event, it bubbles to the top of the hierarchy before it's handled. So Ext JS intercepts this and checks the relevant listeners you added to the component or store. This reduces the number of interactions on the DOM and also gives us the ability to enable gestures. Sencha Charts: Charts can work on both Ext JS and Sencha Touch, and have enhanced performance on tablet devices. Legacy Ext JS 4 charts were converted into a separate package to minimize the conversion/upgrade. In version 5, charts have new features such as: Candlestick and OHLC series Pan, zoom, and crosshair interactions Floating axes Multiple axes SVG and HTML Canvas support Better performance Greater customization Chart themes Tab Panels: Tab panels have more options to control configurations such as icon alignment and text rotation. Thanks to new flexible Sass mixins, we can easily control presentation options. Grids: This component, which has been present since version 2x, is one of the most popular components, and we may call it one of the cornerstones of this framework. In version 5, it got some awesome new features: Components in Cells Buffered updates Cell updaters Grid filters (The popular "UX" (user extension) has been rewritten and integrated into the framework. Also filters can be saved in the component state.) Rendering optimizations Widgets: This is a lightweight component, which is a middle ground between Ext.Component and the Cell renderer. Breadcrumb bars: This new component displays the data of a store (a specific data store for the tree component) in a toolbar form. This new control can be a space saver on small screens or tablets. Form package improvements: Ext JS 5 introduces some new controls and significant changes on others: Tagfield: This is a new control to select multiple values. Segmented buttons: These are buttons with presentation such as multiple selection on mobile interfaces. Goodbye to TriggerField: In version 5, TriggerField is deprecated and now the way to create triggers is by using the Text field and implementing the triggers on the TextField configuration. (TriggerField in version 4 is a text field with a configured button(s) on the right side.)  Field and Form layouts: Layouts were refactored using HTML and CSS, so there is improvement as the performance is now better. New SASS Mixins (http://sass-lang.com/): Several components that were not able to be custom-themed now have the ability to be styled in multiple ways in a single theme or application. These components are: Ext.menu.Menu Ext.form.Labelable Ext.form.FieldSet Ext.form.CheckboxGroup Ext.form.field.Text Ext.form.field.Spinner Ext.form.field.Display Ext.form.field.Checkbox The Sencha Core package: The core package contains code shared between Ext JS and Sencha Touch and in the future, this core will be part of the next major release of Sencha Touch. The Core includes: Class system Data Events Element Utilities Feature/environment detection Preparing for deployment So far, we have seen a few features that helps to architect a JavaScript code; but we need to prepare our application for a production environment. So initially, when an application is in the development environment, we need to make sure that Ext JS classes (also our own classes) are dynamically loaded when the application requires to use them. In this environment, it's really helpful to load each class in a separate file. This will allow us to debug the code easily, and find and fix bugs. Now, before the application is compiled, we must know the three basic parts of an application, as marked here: app.json: This file contains specific details about our application. Also, Sencha CMD processes this file first. build.xml: This file contains a minimal initial Ant script, and imports a task file located at .sencha/app/build-impl.xml. .sencha: This folder contains many files related to, and are to be used for, the build process. The app.json file As we said before, the app.json file contains the information about the settings of the application. Open the file and take a look. We can make changes to this file, such as the theme that our application is going to use. For example, we can use the following line of code: "theme": "my-custom-theme-touch", Alternatively, we can use a normal theme: "theme": "my-custom-theme", We can also use the following for using charts: "requires": [ "sencha-charts" ], This was to specify that we are going to use the charts or draw classes in our application (the chart package for Ext JS 5). Now, at the end of the file, there is an ID for the application: "id": "7833ee81-4d14-47e6-8293-0cb8120281ab" After this ID, we can add other properties. As an example, suppose the application will be generated for Central and South America. Then we need to include the locale (ES or PT), so we can add the following: ,"locales":["es"] We can also add multiple languages: ,"locales":["es","pt","en"] This will cause the compilation process to include the corresponding locale files located at ext/packages/ext-locale/build. However, this article can't cover each property in the file, so it's recommended that you take a deep look into the Sencha CMD documentation at: http://docs-origin.sencha.com/cmd/5.x/microloader.html to learn more about the app.json file. The Sencha command To create our production build, we need to use the Sencha Command. This tool will help us in our purpose. If you are running Sencha CMD on Windows 7 or Windows 8, it's recommended that you run the tool with "administrator privileges". So let's type this in our console tool: [path of my app]\sencha app build In my case (Windows OS 7; 64-bit), I typed: K:\x_extjsdev\app_test\myapp>sencha app build After the command runs, you will see something like this in your console tool: So, let's check out the build folder inside our application folder. We may have the following list of files: Notice that the build process has created these: resources: This file will contain a copy of our resources folder, plus one or more CSS files starting with myApp-all app.js: This file contains all of the necessary JS (Ext JS core classes, components, and our custom application classes) app.json: This is a small manifest file compressed index.html: This file is similar to our index file in development mode, except for the line: <script id="microloader" type="text/javascript" src="bootstrap.js"></script> This was replaced by some compressed JavaScript code, which will act in a similar way to the micro loader. Notice that the serverside folder, where we use some JSON files (other cases can be PHP, ASP, and so on), does not exist in the production folder. Well, the reason is that that folder is not part of what Sencha CMD and build files consider. Normally, many developers will say, "Hey, let's copy the folder and let's move on." However, the good news is that we can include that folder with an Apache Ant task Customizing the build.xml file We can add custom code (Apache Ant style) to perform new tasks and things we need in order to make our application build even better. Let's open the build.xml file. You will see something like this: <?xml version="1.0" encoding="utf-8"?> <project name="myApp" default=".help"> <!-- comments... --> <import file="${basedir}/.sencha/app/build-impl.xml"/> <!-- comments... --> </project> So, let's place the following code before </project>: <target name="-after-build" depends="init"> <copy todir="${build.out.base.path}/serverside" overwrite="false"> <fileset dir="${app.dir}/serverside" includes="**/*"/> </copy> </target> </project> This new code inside the build.xml file establishes that after making the whole building process, if there is no error during the Init process then it will copy the (${app.dir}/ serverside) folder to the (${build.out.base.path}/serverside) output path. So now, let's type the command for building the application again: sencha app build –c In this case, we added -c to first clean the build/production folder and create a new set of files. After the process completes, take a look at the folder contents, and you will see this: Notice that now the serverside folder has been copied to the production build folder, thanks to the custom code we placed in build.xml file. Compressing the code After building our application, let's open the app.js file. We may see something like what is shown here: By default, the build process uses the YUI compressor to compact the JS code (http://yui.github.io/yuicompressor/). Inside the .sencha folder, there are many files, and depending on the type of build we are creating, there are some files such as the base file, where the properties are defined in defaults.properties. This file must not be changed whatsoever; for that, we have other files that can override the values defined in this file. As an example for the production build, we have the following files: production.defaults.properties: This file will contain some properties/variables that will be used for the production build. production.properties: This file has only comments. The idea behind this file is that developers place the variables they want in order to customize the production build. By default, in the production.defaults.properties file, you will see something like the following code: # Comments ...... # more comments...... build.options.logger=no build.options.debug=false # enable the full class system optimizer app.output.js.optimize=true build.optimize=${build.optimize.enable} enable.cache.manifest=true enable.resource.compression=true build.embedded.microloader.compressor=-closure Now, as an example of compression, let's make a change and place some variables inside the production.properties file. The code we will place here will override the properties set in defaults.properties and production.defaults.properties. So, let's write the following code after the comments: build.embedded.microloader.compressor=-closure build.compression.yui=0 build.compression.closure=1 build.compression=-closure With this code, we are setting up the build process to use closure as the JavaScript compressor and also for the micro loader. Now save the file and use the Sencha CMD tool once again: sencha app build Wait for the process to end and take a look at app.js. You can notice that the code is quite different. This is because the code compiler (closure) was the one that made the compression. Run the app and you will notice no change in the behavior and use of the application. As we have used the production.properties file in this example, notice that in the .sencha folder, we have some other files for different environments, such as: Environment File (or files) Testing testing.defaults.properties and testing.properties Development development.defaults.properties and development.properties Production production.defaults.properties and production.properties It's not recommended that you change the *.default.properties file. That's the reason of the *.properties file, so that you can set your own variables, and doing this will override the settings on default file. Packaging and deploying Finally, after we have built our application, we have our production build/package ready to be deployed. We will have the following structure in our folder: Now we have all the files required to make our application work on a public server. We don't need to upload anything from the Ext JS folder because we have all that we need in app.js (all of the Ext JS code and our code). Also, the resources file contains the images, CSS (the theme used in the app), and of course our serverside folder. So now, we need to upload all of the content to the server: And we are ready to test the production in a public server. Summary In this article, you learned the reasons that will make us to consider using Ext JS 5 for developing projects. We briefly mentioned some of the significant additional features in version 5 that are instrumental in developing applications. Later, we talked about compiling and preparing an application for a production environment. Using Sencha CMD and also configuring JSON or XML files to build a project can sometimes be an overwhelming situation, but don't panic! Check out the documentation of Sencha and Apache. Do remember that there's no reason to be afraid of testing and playing with the configurations. It's all part of learning and knowing how to use Sencha Ext JS. Resources for Article: Further resources on this subject: The Login Page using Ext JS [Article] So, what is Ext JS? [Article] AngularJS Performance [Article]