Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7014 Articles
article-image-creating-slack-progress-bar
Bradley Cicenas
13 Sep 2016
4 min read
Save for later

Creating a Slack Progress Bar

Bradley Cicenas
13 Sep 2016
4 min read
There are many ways you can customize your Slack program, but with a little bit of programming you can create things that can provide you with all sorts of productivity-enhancing options. In this post, you are going to learn how to use Python to create a custom progress bar that will run in your very own Slack channel. You can use this progress bar in so many different ways. We won’t get into the specifics beyond the progress bar, but one idea would be to use it to broadcast the progress of a to-do list in Trello. That is something that is easy to set up in Slack. Let’s get started. Real-time notifications are a great way to follow along with the events of the day. For ever-changing and long-running events though, a stream of update messages quickly becomes spammy and overwhelming. If you make a typo in a sent message, or want to append additional details, you know that you can go back and edit it without sending a new chat(and subsequently a new notification for those in the channel); so why can't your bot friends do the same? As you may have guessed by now, they can! This program will make use of Slack’s chat.update method to create a dynamic progress bar with just one single-line message. Save the below script to a file namedprogress.py, updating the SLACK_TOKEN and SLACK_CHANNEL variables at the top with your configured token and desired channel name. from time import sleep from slacker import Slacker from random importrandint SLACK_TOKEN = '<token>' SLACK_CHANNEL = '#general' classSlackSimpleProgress(object): def__init__(self, slack_token, slack_channel): self.slack = Slacker(slack_token) self.channel = slack_channel res = self.slack.chat.post_message(self.channel, self.make_bar(0)) self.msg_ts = res.body['ts'] self.channel_id = res.body['channel'] def update(self, percent_done): self.slack.chat.update(self.channel_id, self.msg_ts, self.make_bar(percent_done)) @staticmethod defmake_bar(percent_done): return'%s%s%%' % ((round(percent_done / 5) * chr(9608)), percent_done) if__name__ == '__main__': progress_bar = SlackSimpleProgress(SLACK_TOKEN, SLACK_CHANNEL) # initialize the progress bar percent_complete = 1 whilepercent_complete<100: progress.update(percent_complete) percent_complete += randint(1,10) sleep(1) progress_bar.update(100) Run the script with a simple python progress.py and, like magic, your progress bar will appear in the channel, updating at regular intervals until completion:   How It Works In the last six lines of the script: A progress bar is initially created with SlackSimpleProgress. The update method is used to refresh the current position of the bar once a second. percent_complete is slowly incremented by a random amount using Python’s random.randint. We set the progress bar to 100% when completed. Extending With the basic usage down, the same script can be updated to serve a real purpose. Say we're copying a large amount of files between directories for a backup—we'll replace the __main__ part of the script with: importos import sys importshutil src_path = sys.argv[0] dest_path = sys.argv[1] all_files = next(os.walk('/usr/lib'))[2] progress_bar = SlackSimpleProgress(SLACK_TOKEN, SLACK_CHANNEL) # initialize the progress bar files_copied = 0 forfile in all_files: src_file = '%s/%s' % (src_path, file) dest_file = '%s/%s' % (dest_path, file) print('copying %s to %s' % (src_file, dest_file)) # print the file being copied shutil.copy2(src_file, dest_file) files_copied += 1# increment files copied percent_done = files_copied / len(all_files) * 100# calculate percent done progress_bar.update(percent_done) The script can now be run with two arguments: python progress.py /path/to/files /path/to/destination. You'll see the name of each file copied in the output of the script, and your teammates will be able to follow along with the status of the copy as it progresses in your Slack channel! I hope you find many uses for ways to implement this progress bar in Slack! About the author Bradley Cicenas is a New York City-based infrastructure engineer with an affinity for microservices, systems design, data science, and stoops.
Read more
  • 0
  • 0
  • 9137

article-image-gearing-bootstrap-4
Packt
12 Sep 2016
28 min read
Save for later

Gearing Up for Bootstrap 4

Packt
12 Sep 2016
28 min read
In this article by Benjamin Jakobus and Jason Marah, the authors of the book Mastering Bootstrap 4, we will be discussing the key points about Bootstrap as a web development framework that helps developers build web interfaces. Originally conceived at Twitter in 2011 by Mark Otto and Jacob Thornton, the framework is now open source and has grown to be one of the most popular web development frameworks to date. Being freely available for private, educational, and commercial use meant that Bootstrap quickly grew in popularity. Today, thousands of organizations rely on Bootstrap, including NASA, Walmart, and Bloomberg. According to BuiltWith.com, over 10% of the world's top 1 million websites are built using Bootstrap (http://trends.builtwith.com/docinfo/Twitter-Bootstrap). As such, knowing how to use Bootstrap will be an important skill and serve as a powerful addition to any web developer’s tool belt. (For more resources related to this topic, see here.) The framework itself consists of a mixture of JavaScript and CSS, and provides developers with all the essential components required to develop a fully functioning web user interface. Over the course of the book, we will be introducing you to all of the most essential features that Bootstrap has to offer by teaching you how to use the framework to build a complete website from scratch. As CSS and HTML alone are already the subject of entire books in themselves, we assume that you, the reader, has at least a basic knowledge of HTML, CSS, and JavaScript. We begin this article by introducing you to our demo website—MyPhoto. This website will accompany us throughout the book, and serve as a practical point of reference. Therefore, all lessons learned will be taught within the context of MyPhoto. We will then discuss the Bootstrap framework, listing its features and contrasting the current release to the last major release (Bootstrap 3). Last but not least, this article will help you set up your development environment. To ensure equal footing, we will guide you towards installing the right build tools, and precisely detail the various ways in which you can integrate Bootstrap into a project. To summarize, this article will do the following: Introduce you to what exactly we will be doing Explain what is new in the latest version of Bootstrap, and how the latest version differs to the previous major release Show you how to include Bootstrap in our web project Introducing our demo project The book will teach you how to build a complete Bootstrap website from scratch. We will build and improve the website's various sections as we progress through the book. The concept behind our website is simple. To develop a landing page for photographers. Using this landing page, (hypothetical) users will be able to exhibit their wares and services. While building our website, we will be making use of the same third-party tools and libraries that you would if you were working as a professional software developer. We chose these tools and plugins specifically because of their widespread use. Learning how to use and integrate them will save you a lot of work when developing websites in the future. Specifically, the tools that we will use to assist us throughout the development of MyPhoto are Bower, node package manager (npm) and Grunt. From a development perspective, the construction of MyPhoto will teach you how to use and apply all of the essential user interface concepts and components required to build a fully functioning website. Among other things, you will learn how to do the following: Use the Bootstrap grid system to structure the information presented on your website. Create a fixed, branded, navigation bar with animated scroll effects. Use an image carousel for displaying different photographs, implemented using Bootstrap's carousel.js and jumbotron (jumbotron is a design principle for displaying important content). It should be noted that carousels are becoming an increasingly unpopular design choice, however, they are still heavily used and are an important feature of Bootstrap. As such, we do not argue for or against the use of carousels as their effectiveness depends very much on how they are used, rather than on whether they are used. Build custom tabs that allow users to navigate across different contents. Use and apply Bootstrap's modal dialogs. Apply a fixed page footer. Create forms for data entry using Bootstrap's input controls (text fields, text areas, and buttons) and apply Bootstrap's input validation styles. Make best use of Bootstrap's context classes. Create alert messages and learn how to customize them. Rapidly develop interactive data tables for displaying product information. How to use drop-down menus, custom fonts, and icons. In addition to learning how to use Bootstrap 4, the development of MyPhoto will introduce you to a range of third-party libraries such as Scrollspy (for scroll animations), SalvattoreJS (a library for complementing our Bootstrap grid), Animate.css (for beautiful CSS animations, such as fade-in effects at https://daneden.github.io/animate.css/) and Bootstrap DataTables (for rapidly displaying data in tabular form). The website itself will consist of different sections: A Welcome section An About section A Services section A Gallery section A Contact Us section The development of each section is intended to teach you how to use a distinct set of features found in third-party libraries. For example, by developing the Welcome section, you will learn how to use Bootstrap's jumbotron and alert dialogs along with different font and text styles, while the About section will show you how to use cards. The Services section of our project introduces you to Bootstrap's custom tabs. That is, you will learn how to use Bootstrap's tabs to display a range of different services offered by our website. Following on from the Services section, you will need to use rich imagery to really show off the website's sample services. You will achieve this by really mastering Bootstrap's responsive core along with Bootstrap's carousel and third-party jQuery plugins. Last but not least, the Contact Us section will demonstrate how to use Bootstrap's form elements and helper functions. That is, you will learn how to use Bootstrap to create stylish HTML forms, how to use form fields and input groups, and how to perform data validation. Finally, toward the end of the book, you will learn how to optimize your website, and integrate it with the popular JavaScript frameworks AngularJS (https://angularjs.org/) and React (http://facebook.github.io/react/). As entire books have been written on AngularJS alone, we will only cover the essentials required for the integration itself. Now that you have glimpsed a brief overview of MyPhoto, let’s examine Bootstrap 4 in more detail, and discuss what makes it so different to its predecessor. Take a look at the following screenshot: Figure 1.1: A taste of what is to come: the MyPhoto landing page. What Bootstrap 4 Alpha 4 has to offer Much has changed since Twitter’s Bootstrap was first released on August 19th, 2011. In essence, Bootstrap 1 was a collection of CSS rules offering developers the ability to lay out their website, create forms, buttons, and help with general appearance and site navigation. With respect to these core features, Bootstrap 4 Alpha 4 is still much the same as its predecessors. In other words, the framework's focus is still on allowing developers to create layouts, and helping to develop a consistent appearance by providing stylings for buttons, forms, and other user interface elements. How it helps developers achieve and use these features however, has changed entirely. Bootstrap 4 is a complete rewrite of the entire project, and, as such, ships with many fundamental differences to its predecessors. Along with Bootstrap's major features, we will be discussing the most striking differences between Bootstrap 3 and Bootstrap 4 in the sub sections below. Layout Possibly the most important and widely used feature is Bootstrap's ability to lay out and organize your page. Specifically, Bootstrap offers the following: Responsive containers. Responsive breakpoints for adjusting page layout in response to differing screen sizes. A 12 column grid layout for flexibly arranging various elements on your page. Media objects that act as building blocks and allow you to build your own structural components. Utility classes that allow you to manipulate elements in a responsive manner. For example, you can use the layout utility classes to hide elements, depending on screen size. Content styling Just like its predecessor, Bootstrap 4 overrides the default browser styles. This means that many elements, such as lists or headings, are padded and spaced differently. The majority of overridden styles only affect spacing and positioning, however, some elements may also have their border removed. The reason behind this is simple. To provide users with a clean slate upon which they can build their site. Building on this clean slate, Bootstrap 4 provides styles for almost every aspect of your webpage such as buttons (Figure 1.2), input fields, headings, paragraphs, special inline texts, such as keyboard input (Figure 1.3), figures, tables, and navigation controls. Aside from this, Bootstrap offers state styles for all input controls, for example, styles for disabled buttons or toggled buttons. Take a look at the following screenshot: Figure 1.2: The six button styles that come with Bootstrap 4 are btn-primary,btn-secondary, btn-success,btn-danger, btn-link,btn-info, and btn-warning. Take a look at the following screenshot: Figure 1.3: Bootstrap's content styles. In the preceding example, we see inline styling for denoting keyboard input. Components Aside from layout and content styling, Bootstrap offers a large variety of reusable components that allow you to quickly construct your website's most fundamental features. Bootstrap's UI components encompass all of the fundamental building blocks that you would expect a web development toolkit to offer: Modal dialogs, progress bars, navigation bars, tooltips, popovers, a carousel, alerts, drop-down menu, input groups, tabs, pagination, and components for emphasizing certain contents. Let's have a look at the following modal dialog screenshot: Figure 1.4: Various Bootstrap 4 components in action. In the screenshot above we see a sample modal dialog, containing an info alert, some sample text, and an animated progress bar. Mobile support Similar to its predecessor, Bootstrap 4 allows you to create mobile friendly websites without too much additional development work. By default, Bootstrap is designed to work across all resolutions and screen sizes, from mobile, to tablet, to desktop. In fact, Bootstrap's mobile first design philosophy implies that its components must display and function correctly at the smallest screen size possible. The reasoning behind this is simple. Think about developing a website without consideration for small mobile screens. In this case, you are likely to pack your website full of buttons, labels, and tables. You will probably only discover any usability issues when a user attempts to visit your website using a mobile device only to find a small webpage that is crowded with buttons and forms. At this stage, you will be required to rework the entire user interface to allow it to render on smaller screens. For precisely this reason, Bootstrap promotes a bottom-up approach, forcing developers to get the user interface to render correctly on the smallest possible screen size, before expanding upwards. Utility classes Aside from ready-to-go components, Bootstrap offers a large selection of utility classes that encapsulate the most commonly needed style rules. For example, rules for aligning text, hiding an element, or providing contextual colors for warning text. Cross-browser compatibility Bootstrap 4 supports the vast majority of modern browsers, including Chrome, Firefox, Opera, Safari, Internet Explorer (version 9 and onwards; Internet Explorer 8 and below are not supported), and Microsoft Edge. Sass instead of Less Both Less and Sass (Syntactically Awesome Stylesheets) are CSS extension languages. That is, they are languages that extend the CSS vocabulary with the objective of making the development of many, large, and complex style sheets easier. Although Less and Sass are fundamentally different languages, the general manner in which they extend CSS is the same—both rely on a preprocessor. As you produce your build, the preprocessor is run, parsing the Less/Sass script and turning your Less or Sass instructions into plain CSS. Less is the official Bootstrap 3 build, while Bootstrap 4 has been developed from scratch, and is written entirely in Sass. Both Less and Sass are compiled into CSS to produce a single file, bootstrap.css. It is this CSS file that we will be primarily referencing throughout this book (with the exception of Chapter 3, Building the Layout). Consequently, you will not be required to know Sass in order to follow this book. However, we do recommend that you take a 20 minute introductory course on Sass if you are completely new to the language. Rest assured, if you already know CSS, you will not need more time than this. The language's syntax is very close to normal CSS, and its elementary concepts are similar to those contained within any other programming language. From pixel to root em Unlike its predecessor, Bootstrap 4 no longer uses pixel (px) as its unit of typographic measurement. Instead, it primarily uses root em (rem). The reasoning behind choosing rem is based on a well known problem with px, that is websites using px may render incorrectly, or not as originally intended, as users change the size of the browser's base font. Using a unit of measurement that is relative to the page's root element helps address this problem, as the root element will be scaled relative to the browser's base font. In turn, a page will be scaled relative to this root element. Typographic units of measurement Simply put, typographic units of measurement determine the size of your font and elements. The most commonly used units of measurement are px and em. The former is an abbreviation for pixel, and uses a reference pixel to determine a font's exact size. This means that, for displays of 96 dots per inch (dpi), 1 px will equal an actual pixel on the screen. For higher resolution displays, the reference pixel will result in the px being scaled to match the display's resolution. For example, specifying a font size of 100 px will mean that the font is exactly 100 pixels in size (on a display with 96 dpi), irrespective of any other element on the page. Em is a unit of measurement that is relative to the parent of the element to which it is applied. So, for example, if we were to have two nested div elements, the outer element with a font size of 100 px and the inner element with a font size of 2 em, then the inner element's font size would translate to 200 px (as in this case 1 em = 100 px). The problem with using a unit of measurement that is relative to parent elements is that it increases your code's complexity, as the nesting of elements makes size calculations more difficult. The recently introduced rem measurement aims to address both em's and px's shortcomings by combining their two strengths—instead of being relative to a parent element, rem is relative to the page's root element. No more support for Internet Explorer 8 As was already implicit in the feature summary above, the latest version of Bootstrap no longer supports Internet Explorer 8. As such, the decision to only support newer versions of Internet Explorer was a reasonable one, as not even Microsoft itself provides technical support and updates for Internet Explorer 8 anymore (as of January 2016). Furthermore, Internet Explorer 8 does not support rem, meaning that Bootstrap 4 would have been required to provide a workaround. This in turn would most likely have implied a large amount of additional development work, with the potential for inconsistencies. Lastly, responsive website development for Internet Explorer 8 is difficult, as the browser does not support CSS media queries. Given these three factors, dropping support for this version of Internet Explorer was the most sensible path of action. A new grid tier Bootstrap's grid system consists of a series of CSS classes and media queries that help you lay out your page. Specifically, the grid system helps alleviate the pain points associated with horizontal and vertical positioning of a page's contents and the structure of the page across multiple displays. With Bootstrap 4, the grid system has been completely overhauled, and a new grid tier has been added with a breakpoint of 480 px and below. We will be talking about tiers, breakpoints, and Bootstrap's grid system extensively in this book. Bye-bye GLYPHICONS Bootstrap 3 shipped with a nice collection of over 250 font icons, free of use. In an effort to make the framework more lightweight (and because font icons are considered bad practice), the GLYPHICON set is no longer available in Bootstrap 4. Bigger text – No more panels, wells, and thumbnails The default font size in Bootstrap 4 is 2 px bigger than in its predecessor, increasing from 14 px to 16 px. Furthermore, Bootstrap 4 replaced panels, wells, and thumbnails with a new concept—cards. To readers unfamiliar with the concept of wells, a well is a UI component that allows developers to highlight text content by applying an inset shadow effect to the element to which it is applied. A panel too serves to highlight information, but by applying padding and rounded borders. Cards serve the same purpose as their predecessors, but are less restrictive as they are flexible enough to support different types of content, such as images, lists, or text. They can also be customized to use footers and headers. Take a look at the following screenshot: Figure 1.5: The Bootstrap 4 card component replaces existing wells, thumbnails, and panels. New and improved form input controls Bootstrap 4 introduces new form input controls—a color chooser, a date picker, and a time picker. In addition, new classes have been introduced, improving the existing form input controls. For example, Bootstrap 4 now allows for input control sizing, as well as classes for denoting block and inline level input controls. However, one of the most anticipated new additions is Bootstrap's input validation styles, which used to require third-party libraries or a manual implementation, but are now shipped with Bootstrap 4 (see Figure 1.6 below). Take a look at the following screenshot: Figure 1.6: The new Bootstrap 4 input validation styles, indicating the successful processing of input. Last but not least, Bootstrap 4 also offers custom forms in order to provide even more cross-browser UI consistency across input elements (Figure 1.7). As noted in the Bootstrap 4 Alpha 4 documentation, the input controls are: "built on top of semantic and accessible markup, so they're solid replacements for any default form control" – Source: http://v4-alpha.getbootstrap.com/components/forms/ Take a look at the following screenshot: Figure 1.7: Custom Bootstrap input controls that replace the browser defaults in order to ensure cross-browser UI consistency. Customization The developers behind Bootstrap 4 have put specific emphasis on customization throughout the development of Bootstrap 4. As such, many new variables have been introduced that allow for the easy customization of Bootstrap. Using the $enabled-*- Sass variables, one can now enable or disable specific global CSS preferences. Setting up our project Now that we know what Bootstrap has to offer, let us set up our project: Create a new project directory named MyPhoto. This will become our project root directory. Create a blank index.html file and insert the following HTML code: <!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <meta http-equiv="x-ua-compatible" content="ie=edge"> <title>MyPhoto</title> </head> <body> <div class="alert alert-success"> Hello World! </div> </body> </html> Note the three meta tags: The first tag tells the browser that the document in question is utf-8 encoded. Since Bootstrap optimizes its content for mobile devices, the subsequent meta tag is required to help with viewport scaling. The last meta tag forces the document to be rendered using the latest document rendering mode available if viewed in Internet Explorer. Open the index.html in your browser. You should see just a blank page with the words Hello World. Now it is time to include Bootstrap. At its core, Bootstrap is a glorified CSS style sheet. Within that style sheet, Bootstrap exposes very powerful features of CSS with an easy-to-use syntax. It being a style sheet, you include it in your project as you would with any other style sheet that you might develop yourself. That is, open the index.html and directly link to the style sheet. Viewport scaling The term viewport refers to the available display size to render the contents of a page. The viewport meta tag allows you to define this available size. Viewport scaling using meta tags was first introduced by Apple and, at the time of writing, is supported by all major browsers. Using the width parameter, we can define the exact width of the user's viewport. For example, <meta name="viewport" content="width=320px"> will instruct the browser to set the viewport's width to 320 px. The ability to control the viewport's width is useful when developing mobile-friendly websites; by default, mobile browsers will attempt to fit the entire page onto their viewports by zooming out as far as possible. This allows users to view and interact with websites that have not been designed to be viewed on mobile devices. However, as Bootstrap embraces a mobile-first design philosophy, a zoom out will, in fact, result in undesired side-effects. For example, breakpoints will no longer work as intended, as they now deal with the zoomed out equivalent of the page in question. This is why explicitly setting the viewport width is so important. By writing content="width=device-width, initial-scale=1, shrink-to-fit=no", we are telling the browser the following: To set the viewport's width equal to whatever the actual device's screen width is. We do not want any zoom, initially. We do not wish to shrink the content to fit the viewport. For now, we will use the Bootstrap builds hosted on Bootstrap's official Content Delivery Network (CDN). This is done by including the following HTML tag into the head of your HTML document (the head of your HTML document refers to the contents between the <head> opening tag and the </head> closing tag): <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0-alpha.4/css/bootstrap.min.css"> Bootstrap relies on jQuery, a JavaScript framework that provides a layer of abstraction in an effort to simplify the most common JavaScript operations (such as element selection and event handling). Therefore, before we include the Bootstrap JavaScript file, we must first include jQuery. Both inclusions should occur just before the </body> closing tag: <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.4/jquery.min.js"> </script> <script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0-alpha.4/js/bootstrap.min.js"></script> Note that, while these scripts could, of course, be loaded at the top of the page, loading scripts at the end of the document is considered best practice to speed up page loading times and to avoid JavaScript issues preventing the page from being rendered. The reason behind this is that browsers do not download all dependencies in parallel (although a certain number of requests are made asynchronously, depending on the browser and the domain). Consequently, forcing the browser to download dependencies early on will block page rendering until these assets have been downloaded. Furthermore, ensuring that your scripts are loaded last will ensure that once you invoke Document Object Model (DOM) operations in your scripts, you can be sure that your page's elements have already been rendered. As a result, you can avoid checks that ensure the existence of given elements. What is a Content Delivery Network? The objective behind any Content Delivery Network (CDN) is to provide users with content that is highly available. This means that a CDN aims to provide you with content, without this content ever (or rarely) becoming unavailable. To this end, the content is often hosted using a large, distributed set of servers. The BootstrapCDN basically allows you to link to the Bootstrap style sheet so that you do not have to host it yourself. Save your changes and reload the index.html in your browser. The Hello World string should now contain a green background: Figure 1.5: Our "Hello World" styled using Bootstrap 4. Now that the Bootstrap framework has been included in our project, open your browser's developer console (if using Chrome on Microsoft Windows, press Ctrl + Shift + I. On Mac OS X you can press cmd + alt + I). As Bootstrap requires another third-party library, Tether for displaying popovers and tooltips, the developer console will display an error (Figure 1.6). Take a look at the following screenshot: Figure 1.6: Chrome's Developer Tools can be opened by going to View, selecting Developer and then clicking on Developer Tools. At the bottom of the page, a new view will appear. Under the Console tab, an error will indicate an unmet dependency. Tether is available via the CloudFare CDN, and consists of both a CSS file and a JavaScript file. As before, we should include the JavaScript file at the bottom of our document while we reference Tether's style sheet from inside our document head: <!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <meta http-equiv="x-ua-compatible" content="ie=edge"> <title>MyPhoto</title> <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0-alpha.4/css/bootstrap.min.css"> <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/tether/1.3.1/css/tether.min.css"> </head> <body> <div class="alert alert-success"> Hello World! </div> <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.4/jquery.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/tether/1.3.1/js/tether.min.js"></script> <script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0-alpha.4/js/bootstrap.min.js"></script> </body> </html> While CDNs are an important resource, there are several reasons why, at times, using a third party CDN may not be desirable: CDNs introduce an additional point of failure, as you now rely on third-party servers. The privacy and security of users may be compromised, as there is no guarantee that the CDN provider does not inject malicious code into the libraries that are being hosted. Nor can one be certain that the CDN does not attempt to track its users. Certain CDNs may be blocked by the Internet Service Providers of users in different geographical locations. Offline development will not be possible when relying on a remote CDN. You will not be able to optimize the files hosted by your CDN. This loss of control may affect your website's performance (although typically you are more often than not offered an optimized version of the library through the CDN). Instead of relying on a CDN, we could manually download the jQuery, Tether, and Bootstrap project files. We could then copy these builds into our project root and link them to the distribution files. The disadvantage of this approach is the fact that maintaining a manual collection of dependencies can quickly become very cumbersome, and next to impossible as your website grows in size and complexity. As such, we will not manually download the Bootstrap build. Instead, we will let Bower do it for us. Bower is a package management system, that is, a tool that you can use to manage your website's dependencies. It automatically downloads, organizes, and (upon command) updates your website's dependencies. To install Bower, head over to http://bower.io/. How do I install Bower? Before you can install Bower, you will need two other tools: Node.js and Git. The latter is a version control tool—in essence, it allows you to manage different versions of your software. To install Git, head over to http://git-scm.com/and select the installer appropriate for your operating system. NodeJS is a JavaScript runtime environment needed for Bower to run. To install it, simply download the installer from the official NodeJS website: https://nodejs.org/ Once you have successfully installed Git and NodeJS, you are ready to install Bower. Simply type the following command into your terminal: npm install -g bower This will install Bower for you, using the JavaScript package manager npm, which happens to be used by, and is installed with, NodeJS. Once Bower has been installed, open up your terminal, navigate to the project root folder you created earlier, and fetch the bootstrap build: bower install bootstrap#v4.0.0-alpha.4 This will create a new folder structure in our project root: bower_components bootstrap Gruntfile.js LICENSE README.md bower.json dist fonts grunt js less package.js package.json We will explain all of these various files and directories later on in this book. For now, you can safely ignore everything except for the dist directory inside bower_components/bootstrap/. Go ahead and open the dist directory. You should see three sub directories: css fonts js The name dist stands for distribution. Typically, the distribution directory contains the production-ready code that users can deploy. As its name implies, the css directory inside dist includes the ready-for-use style sheets. Likewise, the js directory contains the JavaScript files that compose Bootstrap. Lastly, the fonts directory holds the font assets that come with Bootstrap. To reference the local Bootstrap CSS file in our index.html, modify the href attribute of the link tag that points to the bootstrap.min.css: <link rel="stylesheet" href="bower_components/bootstrap/dist/css/bootstrap.min.css"> Let's do the same for the Bootstrap JavaScript file: <script src="bower_components/bootstrap/dist/js/bootstrap.min.js"></script> Repeat this process for both jQuery and Tether. To install jQuery using Bower, use the following command: bower install jquery Just as before, a new directory will be created inside the bower_components directory: bower_components jquery AUTHORS.txt LICENSE.txt bower.json dist sizzle src Again, we are only interested in the contents of the dist directory, which, among other files, will contain the compressed jQuery build jquery.min.js. Reference this file by modifying the src attribute of the script tag that currently points to Google's jquery.min.js by replacing the URL with the path to our local copy of jQuery: <script src="bower_components/jquery/dist/jquery.min.js"></script> Last but not least, repeat the steps already outlined above for Tether: bower install tether Once the installation completes, a similar folder structure than the ones for Bootstrap and jQuery will have been created. Verify the contents of bower_components/tether/dist and replace the CDN Tether references in our document with their local equivalent. The final index.html should now look as follows: <!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <meta http-equiv="x-ua-compatible" content="ie=edge"> <title>MyPhoto</title> <link rel="stylesheet" href="bower_components/bootstrap/dist/css/bootstrap.min.css"> <link rel="stylesheet" href="bower_components/tether/dist/css/tether.min.css"> </head> <body> <div class="alert alert-success"> Hello World! </div> <script src="bower_components/jquery/dist/jquery.min.js"></script> <script src="bower_components/tether/dist/js/tether.min.js"></script> <script src="bower_components/bootstrap/dist/js/bootstrap.min.js"></script> </body> </html> Refresh the index.html in your browser to make sure that everything works. What IDE and browser should I be using when following the examples in this book? While we recommend a JetBrains IDE or Sublime Text along with Google Chrome, you are free to use whatever tools and browser you like. Our taste in IDE and browser is subjective on this matter. However, keep in mind that Bootstrap 4 does not support Internet Explorer 8 or below. As such, if you do happen to use Internet Explorer 8, you should upgrade it to the latest version. Summary Aside from introducing you to our sample project MyPhoto, this article was concerned with outlining Bootstrap 4, highlighting its features, and discussing how this new version of Bootstrap differs to the last major release (Bootstrap 3). The article provided an overview of how Bootstrap can assist developers in the layout, structuring, and styling of pages. Furthermore, we noted how Bootstrap provides access to the most important and widely used user interface controls through the form of components that can be integrated into a page with minimal effort. By providing an outline of Bootstrap, we hope that the framework's intrinsic value in assisting in the development of modern websites has become apparent to the reader. Furthermore, during the course of the wider discussion, we highlighted and explained some important concepts in web development, such as typographic units of measurement or the definition, purpose and justification of the use of Content Delivery Networks. Last but not least, we detailed how to include Bootstrap and its dependencies inside an HTML document.
Read more
  • 0
  • 0
  • 30382

article-image-intro-swift-repl-and-playgrounds
Dov Frankel
11 Sep 2016
6 min read
Save for later

Intro to the Swift REPL and Playgrounds

Dov Frankel
11 Sep 2016
6 min read
When Apple introduced Swift at WWDC (its annual Worldwide Developers Conference) in 2014, it had a few goals for the new language. Among them was being easy to learn, especially compared to other compiled languages. The following is quoted from Apple's The Swift Programming Language: Swift is friendly to new programmers. It is the first industrial-quality systems programming language that is as expressive and enjoyable as a scripting language. The REPL Swift's playgrounds embody that friendliness by modernizing the concept of a REPL (Read-Eval-Print Loop, pronounced "repple"). Introduced by the LISP language, and now a common feature of many modern scripting languages, REPLs allow you to quickly and interactively build up your code, one line at a time. A post on the Swift Blog gives an overview of the Swift REPL's features, but this is what using it looks like (to launch it, enter swift in Terminal, if you have Xcode already installed): Welcome to Apple Swift version 2.2 (swiftlang-703.0.18.1 clang-703.0.29). Type :help for assistance.   1> "Hello world" $R0: String = "Hello World"   2> let a = 1 a: Int = 1   3> let b = 2 b: Int = 2   4> a + b $R1: Int = 3   5> func aPlusB() {   6.     print("(a + b)")   7. }   8> aPlusB() 3 If you look at what's there, each line containing input you give has a small indentation. A line that starts a new command begins with the line number, followed by > (1, 2, 3, 4, 5, and 8), and each subsequent line for a given command begins with the line number, followed by  (6 and 7). These help to keep you oriented as you enter your code one line at a time. While it's certainly possible to work on more complicated code this way, it requires the programmer to keep more state about the code in his head, and it limits the output to data types that can be represented in text only. Playgrounds Playgrounds take the concept of a REPL to the next level by allowing you to see all of your code in a single editable code window, and giving you richer options to visualize your data. To get started with a Swift playground, launch Xcode, and select File > New > Playground… (⌥⇧⌘N) to create a new playground. In the following, you can see a new playground with the same code entered into the previous  REPL:   The results are shown in the gray area on the right-hand side of the window update as you type, which allows for rapid iteration. You can write standalone functions, classes, or whatever level of abstraction you wish to work in for the task at hand, removing barriers that prevent the expression of your ideas, or experimentation with the language and APIs. So, what types of goals can you accomplish? Experimentation with the language and APIs Playgrounds are an excellent place to learn about Swift, whether you're new to the language, or new to programming. You don't need to worry about main(), app bundles, simulators, or any of the other things that go into making a full-blown app. Or, if you hear about a new framework and would like to try it out, you can import it into your playground and get your hands dirty with minimal effort. Crucially, it also blows away the typical code-build-run-quit-repeat cycle that can often take up so much development time. Providing living documentation or code samples Playgrounds provide a rich experience to allow users to try out concepts they're learning, whether they're reading a technical article, or learning to use a framework. Aside from interactivity, playgrounds provide a whole other type of richness: built-in formatting via Markdown, which you can sprinkle in your playground as easily as writing comments. This allows some interesting options such as describing exercises for students to complete or providing sample code that runs without any action required of the user. Swift blog posts have included playgrounds to experiment with, as does the Swift Programming Language's A Swift Tour chapter. To author Markdown in your playground, start a comment block with /*:. Then, to see the comment rendered, click on Editor > Show Rendered Markup. There are some unique features available, such as marking page boundaries and adding fields that populate the built-in help Xcode shows. You can learn more at Apple's Markup Formatting Reference page. Designing code or UI You can also use playgrounds to interactively visualize how your code functions. There are a few ways to see what your code is doing: Individual results are shown in the gray side panel and can be Quick-Looked (including your own custom objects that implement debugQuickLookObject()). Individual or looped values can be pinned to show inline with your code. A line inside a loop will read "(10 times)," with a little circle you can toggle to pin it. For instance, you can show how a value changes with each iteration, or how a view looks: Using some custom APIs provided in the XCPlayground module, you can show live UIViews and captured values. Just import XCPlayground and set the XCPlayground.currentPage.liveView to a UIView instance, or call XCPlayground.currentPage.captureValue(someValue, withIdentifier: "My label") to fill the Assistant view. You also still have Xcode's console available to you for when debugging is best served by printing values and keeping scrolled to the bottom. As with any Swift code, you can write to the console with NSLog and print. Working with resources Playgrounds can also include resources for your code to use, such as images: A .playground file is an OS X package (a folder presented to the user as a file), which contains two subfolders: Sources and Resources. To view these folders in Xcode, show the Project Navigator in Xcode's left sidebar, the same as for a regular project. You can then drag in any resources to the Resources folder, and they'll be exposed to your playground code the same as resources are in an app bundle. You can refer to them like so: let spriteImage = UIImage(named:"sprite.png") Xcode versions starting with 7.1 even support image literals, meaning you can drag a Resources image into your source code and treat it as a UIImage instance. It's a neat idea, but makes for some strange-looking code. It's more useful for UIColors, which allow you to use a color-picker. The Swift blog post goes into more detail on how image, color, and file literals work in playgrounds: Wrap-up Hopefully this has opened your eyes to the opportunities afforded by playgrounds. They can be useful in different ways to developers of various skill and experience levels (in Swift, or in general) when learning new things or working on solving a given problem. They allow you to iterate more quickly, and visualize how your code operates more easily than other debugging methods. About the author Dov Frankel (pronounced as in "he dove swiftly into the shallow pool") works on Windows in-house software at a Connecticut hedge fund by day, and independent Mac and iOS apps by night. Find his digital comic reader, on the Mac App Store, and his iPhone app Afterglo is in the iOS App Store. He blogs when the mood strikes him, at dovfrankel.com; he's @DovFrankel on Twitter, and @abbeycode on GitHub.
Read more
  • 0
  • 0
  • 26696

article-image-simple-slack-websocket-integrations-10-lines-python
Bradley Cicenas
09 Sep 2016
3 min read
Save for later

Simple Slack Websocket Integrations in <10 lines of Python

Bradley Cicenas
09 Sep 2016
3 min read
If you use Slack, you've probably added a handful of integrations for your team from the ever-growing App Directory, and maybe even had an idea for your own Slack app. While the Slack API is featureful and robust, writing your own integration can be exceptionally easy. Through the Slack RTM (Real Time Messaging) API, you can write our own basic integrations in just a few lines of Python using the SlackSocket library. Want an accessible introduction to Python that's comprehensive enough to give you the confidence you need to dive deeper? This week, follow our Python Fundamentals course inside Mapt. It's completely free - so what have you got to lose? Structure Our integration will be structured with the following basic components: Listener Integration/bot logic Response The listener watches for one or more pre-defined "trigger" words, while the response posts the result of our intended task. Basic Integration We'll start by setting up SlackSocket with our API token: fromslacksocketimportSlackSocket slack=SlackSocket('<slack-token>', event_filter=['message']) By default, SlackSocket will listen for all Slack events. There are a lot of different events sent via RTM, but we're only concerned with 'message' events for our integration, so we've set an event_filter for only this type. Using the SlackSocketevents() generator, we'll read each 'message' event that comes in and can act on various conditions: for e inslack.events(): ife.event['text'] =='!hello': slack.send_msg('it works!', channel_name=e.event['channel']) If our message text matches the string '!hello', we'll respond to the source channel of the event with a given message('it works!'). At this point, we've created a complete integration that can connect to Slack as a bot user(or regular user), follow messages, and respond accordingly. Let's build something a bit more useful, like a password generator for throwaway accounts. Expanding Functionality For this integration command, we'll write a simple function to generate a random alphanumeric string 15 characters long: import random import string defrandomstr(): chars=string.ascii_letters+string.digits return''.join(random.choice(chars) for _ inrange(15)) Now we're ready to provide our random string generator to the rest of the team using the same chat logic as before, responding to the source channel with our generated password: for e inslack.events(): e.event['text'].startswith('!random'): slack.send_msg(randomstr(), channel_name=e.event['channel']) Altogether: import random import string fromslacksocketimportSlackSocket slack=SlackSocket('<slack-token>', event_filter=['message']) defrandomstr(): chars=string.ascii_letters+string.digits return''.join(random.choice(chars) for _ inrange(15)) for e inslack.events(): ife.event['text'].startswith('!random'): slack.send_msg(randomstr(), channel_name=e.event['channel']) And the results:  A complete integration in 10 lines of Python. Not bad! Beyond simplicity, SlackSocket provides a great deal of flexibility for writing apps, bots, or integrations. In the case of massive Slack groups with several thousand users, messages are buffered locally to ensure that none are missed. Dropped websocket connections are automatically re-connected as well, making it an ideal base for chat client. The code for SlackSocket is available on GitHub, and as always, we welcome any contributions or feature requests! About the author Bradley Cicenas is a New York City-based infrastructure engineer with an affinity for microservices, systems design, data science, and stoops.
Read more
  • 0
  • 0
  • 5703

article-image-using-web-api-extend-your-application
Packt
08 Sep 2016
14 min read
Save for later

Using Web API to Extend Your Application

Packt
08 Sep 2016
14 min read
In this article by Shahed Chowdhuri author of book ASP.Net Core Essentials, we will work through a working sample of a web API project. During this lesson, we will cover the following: Web API Web API configuration Web API routes Consuming Web API applications (For more resources related to this topic, see here.) Understanding a web API Building web applications can be a rewarding experience. The satisfaction of reaching a broad set of potential users can trump the frustrating nights spent fine-tuning an application and fixing bugs. But some mobile users demand a more streamlined experience that only a native mobile app can provide. Mobile browsers may experience performance issues in low-bandwidth situations, where HTML5 applications can only go so far with a heavy server-side back-end. Enter web API, with its RESTful endpoints, built with mobile-friendly server-side code. The case for web APIs In order to create a piece of software, years of wisdom tell us that we should build software with users in mind. Without use cases, its features are literally useless. By designing features around user stories, it makes sense to reveal public endpoints that relate directly to user actions. As a result, you will end up with a leaner web application that works for more users. If you need more convincing, here's a recap of features and benefits: It lets you build modern lightweight web services, which are a great choice for your application, as long as you don't need SOAP It's easier to work with than any past work you may have done with ASP.NET Windows Communication Foundation (WCF) services It supports RESTful endpoints It's great for a variety of clients, both mobile and web It's unified with ASP.NET MVC and can be included with/without your web application Creating a new web API project from scratch Let's build a sample web application named Patient Records. In this application, we will create a web API from scratch to allow the following tasks: Add a new patient Edit an existing patient Delete an existing patient View a specific patient or a list of patients These four actions make up the so-called CRUD operations of our system: to Create, Read, Update or Delete patient records. Following the steps below, we will create a new project in Visual Studio 2015: Create a new web API project. Add an API controller. Add methods for CRUD operations. The preceding steps have been expanded into detailed instructions with the following screenshots: In Visual Studio 2015, click File | New | Project. You can also press Ctrl+Shift+N on your keyboard. On the left panel, locate the Web node below Visual C#, then select ASP.NET Core Web Application (.NET Core), as shown in the following screenshot: With this project template selected, type in a name for your project, for examplePatientRecordsApi, and choose a location on your computer, as shown in the following screenshot: Optionally, you may select the checkboxes on the lower right to create a directory for your solution file and/or add your new project to source control. Click OK to proceed. In the dialog that follows, select Empty from the list of the ASP.NET Core Templates, then click OK, as shown in the following screenshot: Optionally, you can check the checkbox for Microsoft Azure to host your project in the cloud. Click OK to proceed. Building your web API project In the Solution Explorer, you may observe that your References are being restored. This occurs every time you create a new project or add new references to your project that have to be restored through NuGet,as shown in the following screenshot: Follow these steps, to fix your references, and build your Web API project: Rightclickon your project, and click Add | New Folder to add a new folder, as shown in the following screenshot: Perform the preceding step three times to create new folders for your Controllers, Models, and Views,as shown in the following screenshot: Rightclick on your Controllers folder, then click Add | New Item to create a new API controller for patient records on your system, as shown in the following screenshot: In the dialog box that appears, choose Web API Controller Class from the list of options under .NET Core, as shown in the following screenshot: Name your new API controller, for examplePatientController.cs, then click Add to proceed. In your new PatientController, you will most likely have several areas highlighted with red squiggly lines due to a lack of necessary dependencies, as shown in the following screenshot. As a result, you won't be able to build your project/solution at this time: In the next section, we will learn about how to configure your web API so that it has the proper references and dependencies in its configuration files. Configuring the web API in your web application How does the web server know what to send to the browser when a specific URL is requested? The answer lies in the configuration of your web API project. Setting up dependencies In this section, we will learn how to set up your dependencies automatically using the IDE, or manually by editing your project's configuration file. To pull in the necessary dependencies, you may right-click on the using statement for Microsoft.AspNet.Mvc and select Quick Actions and Refactorings…. This can also be triggered by pressing Ctrl +. (period) on your keyboard or simply by hovering over the underlined term, as shown in the following screenshot: Visual Studio should offer you several possible options, fromwhich you can select the one that adds the package Microsoft.AspNetCore.Mvc.Corefor the namespace Microsoft.AspNetCore.Mvc. For the Controller class, add a reference for the Microsoft.AspNetCore.Mvc.ViewFeaturespackage, as shown in the following screenshot: Fig12: Adding the Microsoft.AspNetCore.Mvc.Core 1.0.0 package If you select the latest version that's available, this should update your references and remove the red squiggly lines, as shown in the following screenshot: Fig13:Updating your references and removing the red squiggly lines The precedingstep should automatically update your project.json file with the correct dependencies for theMicrosoft.AspNetCore.Mvc.Core, and Microsoft.AspNetCore.Mvc.ViewFeatures, as shown in the following screenshot: The "frameworks" section of theproject.json file identifies the type and version of the .NET Framework that your web app is using, for examplenetcoreapp1.0 for the 1.0 version of .NET Core. You will see something similar in your project, as shown in the following screenshot: Click the Build Solution button from the top menu/toolbar. Depending on how you have your shortcuts set up, you may press Ctrl+Shift+B or press F6 on your keyboard to build the solution. You should now be able to build your project/solution without errors, as shown in the following screenshot: Before running the web API project, open the Startup.cs class file, and replace the app.Run() statement/block (along with its contents) with a call to app.UseMvc()in the Configure() method. To add the Mvc to the project, add a call to the services.AddMvcCore() in the ConfigureServices() method. To allow this code to compile, add a reference to Microsoft.AspNetCore.Mvc. Parts of a web API project Let's take a closer look at the PatientController class. The auto-generated class has the following methods: public IEnumerable<string> Get() public string Get(int id) public void Post([FromBody]string value) public void Put(int id, [FromBody]string value) public void Delete(int id) The Get() method simply returns a JSON object as an enumerable string of values, while the Get(int id) method is an overridden variant that gets a particular value for a specified ID. The Post() and Put() methods can be used for creating and updating entities. Note that the Put() method takes in an ID value as the first parameter so that it knows which entity to update. Finally, we have the Delete() method, which can be used to delete an entity using the specified ID. Running the web API project You may run the web API project in a web browser that can display JSON data. If you use Google Chrome, I would suggest using the JSONView Extension (or other similar extension) to properly display JSON data. The aforementioned extension is also available on GitHub at the following URL: https://github.com/gildas-lormeau/JSONView-for-Chrome If you use Microsoft Edge, you can view the raw JSON data directly in the browser.Once your browser is ready, you can select your browser of choice from the top toolbar of Visual Studio. Click on the tiny triangle icon next to the Debug button, then select a browser, as shown in the following screenshot: In the preceding screenshot, you can see that multiple installed browsers are available, including Firefox, Google Chrome, Internet Explorer,and Edge. To choose a different browser, simply click on Browse With…, in the menu to select a different one. Now, click the Debug button (that isthe green play button) to see the web API project in action in your web browser, as shown in the following screenshot. If you don't have a web application set up, you won't be able to browse the site from the root URL: Don’t worry if you see this error, you can update the URL to include a path to your API controller, for an example seehttp://localhost:12345/api/Patient. Note that your port number may vary. Now, you should be able to see a list of views that are being spat out by your API controller, as shown in the following screenshot: Adding routes to handle anticipated URL paths Back in the days of classic ASP, application URL paths typically reflected physical file paths. This continued with ASP.NET web forms, even though the concept of custom URL routing was introduced. With ASP.NET MVC, routes were designed to cater to functionality rather than physical paths. ASP.NET web API continues this newer tradition, with the ability to set up custom routes from within your code. You can create routes for your application using fluent configuration in your startup code or with declarative attributes surrounded by square brackets. Understanding routes To understand the purpose of having routes, let's focus on the features and benefits of routes in your application. This applies to both ASP.NET MVC and ASP.NET web API: By defining routes, you can introduce predictable patterns for URL access This gives you more control over how URLs are mapped to your controllers Human-readable route paths are also SEO-friendly, which is great for Search Engine Optimization It provides some level of obscurity when it comes to revealing the underlying web technology and physical file names in your system Setting up routes Let's start with this simple class-level attribute that specifies a route for your API controller, as follows: [Route("api/[controller]")] public class PatientController : Controller { // ... } Here, we can dissect the attribute (seen in square brackets, used to affect the class below it) and its parameter to understand what's going on: The Route attribute indicates that we are going to define a route for this controller. Within the parentheses that follow, the route path is defined in double quotes. The first part of this path is thestring literal api/, which declares that the path to an API method call will begin with the term api followed by a forward slash. The rest of the path is the word controller in square brackets, which refers to the controller name. By convention, the controller's name is part of the controller's class name that precedes the term Controller. For a class PatientController, the controller name is just the word Patient. This means that all API methods for this controller can be accessed using the following syntax, where MyApplicationServer should be replaced with your own server or domain name:http://MyApplicationServer/api/Patient For method calls, you can define a route with or without parameters. The following two examples illustrate both types of route definitions: [HttpGet] public IEnumerable<string> Get() {     return new string[] { "value1", "value2" }; } In this example, the Get() method performs an action related to the HTTP verb HttpGet, which is declared in the attribute directly above the method. This identifies the default method for accessing the controller through a browser without any parameters, which means that this API method can be accessed using the following syntax: http://MyApplicationServer/api/Patient To include parameters, we can use the following syntax: [HttpGet("{id}")] public string Get(int id) {     return "value"; } Here, the HttpGet attribute is coupled with an "{id}" parameter, enclosed in curly braces within double quotes. The overridden version of the Get() method also includes an integer value named id to correspond with the expected parameter. If no parameter is specified, the value of id is equal to default(int) which is zero. This can be called without any parameters with the following syntax: http://MyApplicationServer/api/Patient/Get In order to pass parameters, you can add any integer value right after the controller name, with the following syntax: http://MyApplicationServer/api/Patient/1 This will assign the number 1 to the integer variable id. Testing routes To test the aforementioned routes, simply run the application from Visual Studio and access the specified URLs without parameters. The preceding screenshot show the results of accessing the following path: http://MyApplicationServer/api/Patient/1 Consuming a web API from a client application If a web API exposes public endpoints, but there is no client application there to consume it, does it really exist? Without getting too philosophical, let's go over the possible ways you can consume a client application. You can do any of the following: Consume the Web API using external tools Consume the Web API with a mobile app Consume the Web API with a web client Testing with external tools If you don't have a client application set up, you can use an external tool such as Fiddler. Fiddler is a free tool that is now available from Telerik, available at http://www.telerik.com/download/fiddler, as shown in the following screenshot: You can use Fiddler to inspect URLs that are being retrieved and submitted on your machine. You can also use it to trigger any URL, and change the request type (Get, Post, and others). Consuming a web API from a mobile app Since this article is primarily about the ASP.NET core web API, we won't go into detail about mobile application development. However, it's important to note that a web API can provide a backend for your mobile app projects. Mobile apps may include Windows Mobile apps, iOS apps, Android apps, and any modern app that you can build for today's smartphones and tablets. You may consult the documentation for your particular platform of choice, to determine what is needed to call a RESTful API. Consuming a web API from a web client A web client, in this case, refers to any HTML/JavaScript application that has the ability to call a RESTful API. At the least, you can build a complete client-side solution with straight JavaScript to perform the necessary actions. For a better experience, you may use jQuery and also one of many popular JavaScript frameworks. A web client can also be a part of a larger ASP.NET MVC application or a Single-Page Application (SPA). As long as your application is spitting out JavaScript that is contained in HTML pages, you can build a frontend that works with your backend web API. Summary In this article, we've taken a look at the basic structure of an ASP.NET web API project, and observed the unification of web API with MVC in an ASP.NET core. We also learned how to use a web API as our backend to provide support for various frontend applications. Resources for Article:   Further resources on this subject: Introducing IoT with Particle's Photon and Electron [article] Schema Validation with Oracle JDeveloper - XDK 11g [article] Getting Started with Spring Security [article]
Read more
  • 0
  • 0
  • 13799

article-image-customizing-xtext-components
Packt
08 Sep 2016
30 min read
Save for later

Customizing Xtext Components

Packt
08 Sep 2016
30 min read
In this article written by Lorenzo Bettini, author of the book Implementing Domain Specific Languages Using Xtend and Xtext, Second Edition, the author describes the main mechanism for customizing Xtext components—Google Guice, a Dependency Injection framework. With Google Guice, we can easily and consistently inject custom implementations of specific components into Xtext. In the first section, we will briefly show some Java examples that use Google Guice. Then, we will show how Xtext uses this dependency injection framework. In particular, you will learn how to customize both the runtime and the UI aspects. This article will cover the following topics: An introduction to Google Guice dependency injection framework How Xtext uses Google Guice How to customize several aspects of an Xtext DSL (For more resources related to this topic, see here.) Dependency injection The Dependency Injection pattern (see the article Fowler, 2004) allows you to inject implementation objects into a class hierarchy in a consistent way. This is useful when classes delegate specific tasks to objects referenced in fields. These fields have abstract types (that is, interfaces or abstract classes) so that the dependency on actual implementation classes is removed. In this first section, we will briefly show some Java examples that use Google Guice. Of course, all the injection principles naturally apply to Xtend as well. If you want to try the following examples yourself, you need to create a new Plug-in Project, for example, org.example.guice and add com.google.inject and javax.inject as dependencies in the MANIFEST.MF. Let's consider a possible scenario: a Service class that abstracts from the actual implementation of a Processor class and a Logger class. The following is a possible implementation: public class Service {   private Logger logger;   private Processor processor;     public void execute(String command) { logger.log("executing " + command); processor.process(command); logger.log("executed " + command);   } }   public class Logger {   public void log(String message) { out.println("LOG: " + message);   } }   public interface Processor {   public void process(Object o); }   public classProcessorImplimplements Processor {   private Logger logger;     public void process(Object o) { logger.log("processing"); out.println("processing " + o + "...");   } } These classes correctly abstract from the implementation details, but the problem of initializing the fields correctly still persists. If we initialize the fields in the constructor, then the user still needs to hardcode the actual implementation classnames. Also, note that Logger is used in two independent classes; thus, if we have a custom logger, we must make sure that all the instances use the correct one. These issues can be dealt with using dependency injection. With dependency injection, hardcoded dependencies will be removed. Moreover, we will be able to easily and consistently switch the implementation classes throughout the code. Although the same goal can be achieved manually by implementing factory method or abstract factory patterns (see the book Gamma et al, 1995), with dependency injection framework it is easier to keep the desired consistency and the programmer needs to write less code. Xtext uses the dependency injection framework Google Guice, https://github.com/google/guice. We refer to the Google Guice documentation for all the features provided by this framework. In this section, we just briefly describe its main features. You annotate the fields you want Guice to inject with the @Inject annotation (com.google.inject.Inject): public class Service {   @Inject private Logger logger;   @Inject private Processor processor;     public void execute(String command) { logger.log("executing " + command); processor.process(command); logger.log("executed " + command);   } }   public class ProcessorImpl implements Processor {   @Inject private Logger logger;     public void process(Object o) { logger.log("processing"); out.println("processing " + o + "...");   } } The mapping from injection requests to instances is specified in a Guice Module, a class that is derived from com.google.inject.AbstractModule. The method configure is implemented to specify the bindings using a simple and intuitive API. You only need to specify the bindings for interfaces, abstract classes, and for custom classes. This means that you do not need to specify a binding for Logger since it is a concrete class. On the contrary, you need to specify a binding for the interface Processor. The following is an example of a Guice module for our scenario: public class StandardModule extends AbstractModule {   @Override   protected void configure() {     bind(Processor.class).to(ProcessorImpl.class);   } } You create an Injector using the static method Guice.createInjector by passing a module. You then use the injector to create instances: Injector injector = Guice.createInjector(newStandardModule()); Service service = injector.getInstance(Service.class); service.execute("First command"); The initialization of injected fields will be done automatically by Google Guice. It is worth noting that the framework is also able to initialize (inject) private fields, like in our example. Instances of classes that use dependency injection must be created only through an injector. Creating instances with new will not trigger injection, thus all the fields annotated with @Inject will be null. When implementing a DSL with Xtext you will never have to create a new injector manually. In fact, Xtext generates utility classes to easily obtain an injector, for example, when testing your DSL with JUnit. We also refer to the article Köhnlein, 2012 for more details. The example shown in this section only aims at presenting the main features of Google Guice. If we need a different configuration of the bindings, all we need to do is define another module. For example, let's assume that we defined additional derived implementations for logging and processing. Here is an example where Logger and Processor are bound to custom implementations: public class CustomModule extends AbstractModule {   @Override   protected void configure() {     bind(Logger.class).to(CustomLogger.class);     bind(Processor.class).to(AdvancedProcessor.class);   } } Creating instances with an injector obtained using this module will ensure that the right classes are used consistently. For example, the CustomLogger class will be used both by Service and Processor. You can create instances from different injectors in the same application, for example: executeService(Guice.createInjector(newStandardModule())); executeService(Guice.createInjector(newCustomModule()));   voidexecuteService(Injector injector) {   Service service = injector.getInstance(Service.class); service.execute("First command"); service.execute("Second command"); } It is possible to request injection in many different ways, such as injection of parameters to constructors, using named instances, specification of default implementation of an interface, setter methods, and much more. In this book, we will mainly use injected fields. Injected fields are instantiated only once when the class is instantiated. Each injection will create a new instance, unless the type to inject is marked as @Singleton(com.google.inject.Singleton). The annotation @Singleton indicates that only one instance per injector will be used. We will see an example of Singleton injection. If you want to decide when you need an element to be instantiated from within method bodies, you can use a provider. Instead of injecting an instance of the wanted type C, you inject a com.google.inject.Provider<C> instance, which has a get method that produces an instance of C. For example: public class Logger {   @Inject   private Provider<Utility>utilityProvider;     public void log(String message) { out.println("LOG: " + message + " - " + utilityProvider.get().m());    } } Each time we create a new instance of Utility using the injected Provider class. Even in this case, if the type of the created instance is annotated with @Singleton, then the same instance will always be returned for the same injector. The nice thing is that to inject a custom implementation of Utility, you do not need to provide a custom Provider: you just bind the Utility class in the Guice module and everything will work as expected: public classCustomModule extends AbstractModule {   @Override   protected void configure() {     bind(Logger.class).to(CustomLogger.class);     bind(Processor.class).to(AdvancedProcessor.class);     bind(Utility.class).to(CustomUtility.class);   } }   It is crucial to keep in mind that once classes rely on injection, their instances must be created only through an injector; otherwise, all the injected elements will be null. In general, once dependency injection is used in a framework, all classes of the framework must rely on injection. Google Guice in Xtext All Xtext components rely on Google Guice dependency injection, even the classes that Xtext generates for your DSL. This means that in your classes, if you need to use a class from Xtext, you just have to declare a field of such type with the @Inject annotation. The injection mechanism allows a DSL developer to customize basically every component of the Xtext framework. This boils down to another property of dependency injection, which, in fact, inverts dependencies. The Xtext runtime can use your classes without having a dependency to its implementer. Instead, the implementer has a dependency on the interface defined by the Xtext runtime. For this reason, dependency injection is said to implement inversion of control and the dependency inversion principle. When running the MWE2 workflow, Xtext generates both a fully configured module and an empty module that inherits from the generated one. This allows you to override generated or default bindings. Customizations are added to the empty stub module. The generated module should not be touched. Xtext generates one runtime module that defines the non-user interface-related parts of the configuration and one specific for usage in the Eclipse IDE. Guice provides a mechanism for composing modules that is used by Xtext—the module in the UI project uses the module in the runtime project and overrides some bindings. Let's consider the Entities DSL example. You can find in the src directory of the runtime project the Xtend class EntitiesRuntimeModule, which inherits from AbstractEntitiesRuntimeModule in the src-gen directory. Similarly, in the UI project, you can find in the src directory the Xtend class EntitiesUiModule, which inherits from AbstractEntitiesUiModule in the src-gen directory. The Guice modules in src-gen are already configured with the bindings for the stub classes generated during the MWE2 workflow. Thus, if you want to customize an aspect using a stub class, then you do not have to specify any specific binding. The generated stub classes concern typical aspects that the programmer usually wants to customize, for example, validation and generation in the runtime project, and labels, and outline in the UI project (as we will see in the next sections). If you need to customize an aspect which is not covered by any of the generated stub classes, then you will need to write a class yourself and then specify the binding for your class in the Guice module in the src folder. We will see an example of this scenario in the Other customizations section. Bindings in these Guice module classes can be specified as we saw in the previous section, by implementing the configure method. However, Xtext provides an enhanced API for defining bindings; Xtext reflectively searches for methods with a specific signature in order to find Guice bindings. Thus, assuming you want to bind a BaseClass class to your derived CustomClass, you can simply define a method in your module with a specific signature, as follows: def Class<? extendsBaseClass>bindBaseClass() {   returnCustomClass } Remember that in Xtend, you must explicitly specify that you are overriding a method of the base class; thus, in case the bind method is already defined in the  base class, you need to use override instead of def. These methods are invoked reflectively, thus their signature must follow the expected convention. We refer to the official Xtext documentation for the complete description of the module API. Typically, the binding methods that you will see in this book will have the preceding shape, in particular, the name of the method must start with bind followed by the name of the class or interface we want to provide a binding for. It is important to understand that these bind methods do not necessarily have to override a method in the module base class. You can also make your own classes, which are not related to Xtext framework classes at all, participants of this injection mechanism, as long as you follow the preceding convention on method signatures. In the rest of this article, we will show examples of customizations of both IDE and runtime concepts. For most of these customizations, we will modify the corresponding Xtend stub class that Xtext generated when running the MWE2 workflow. As hinted before, in these cases, we will not need to write a custom Guice binding. We will also show an example of a customization, which does not have an automatically generated stub class. Xtext uses injection to inject services and not to inject state (apart from EMF Singleton registries). Thus, the things that are injected are interfaces consisting of functions that take state as arguments (for example, the document, the resource, and so on.). This leads to a service-oriented architecture, which is different from an object-oriented architecture where state is encapsulated with operations. An advantage of this approach is that there are far less problems with synchronization of multiple threads. Customizations of IDE concepts In this section, we show typical concepts of the IDE for your DSL that you may want to customize. Xtext shows its usability in this context as well, since, as you will see, it reduces the customization effort. Labels Xtext UI classes make use of an ILabelProvider interface to obtain textual labels and icons through its methods getText and getImage, respectively. ILabelProvider is a standard component of Eclipse JFace-based viewers. You can see the label provider in action in the Outline view and in content assist proposal popups (as well as in various other places). Xtext provides a default implementation of a label provider for all DSLs, which does its best to produce a sensible representation of the EMF model objects using the name feature, if it is found in the corresponding object class, and a default image. You can see that in the Outline view when editing an entities file, refer to the following screenshot: However, you surely want to customize the representation of some elements of your DSL. The label provider Xtend stub class for your DSL can be found in the UI plug-in project in the subpackageui.labeling. This stub class extends the base class DefaultEObjectLabelProvider. In the Entities DSL, the class is called EntitiesLabelProvider. This class employs a Polymorphic Dispatcher mechanism, which is also used in many other places in Xtext. Thus, instead of implementing the getText and getImage methods, you can simply define several versions of methods text and image taking as parameter an EObject object of the type you want to provide a representation for. Xtext will then search for such methods according to the runtime type of the elements to represent. For example, for our Entities DSL, we can change the textual representation of attributes in order to show their names and a better representation of types (for example, name : type). We then define a method text taking Attribute as a parameter and returning a string: classEntitiesLabelProviderextends ... {     @Inject extensionTypeRepresentation   def text(Attribute a) { a.name +       if (a.type != null)          " : " + a.type.representation       else ""   } } To get a representation of the AttributeType element, we use an injected extension, TypeRepresentation, in particular its method representation: classTypeRepresentation { def representation(AttributeType t) { valelementType = t.elementType valelementTypeRepr =       switch (elementType) { BasicType : elementType.typeName EntityType : elementType?.entity.name       } elementTypeRepr + if (t.array) "[]"else""   } } Remember that the label provider is used, for example, for the Outline view, which is refreshed when the editor contents change, and its contents might contain errors. Thus, you must be ready to deal with an incomplete model, and some features might still be null. That is why you should always check that the features are not null before accessing them. Note that we inject an extension field of type TypeRepresentation instead of creating an instance with new in the field declaration. Although it is not necessary to use injection for this class, we decided to rely on that because in the future we might want to be able to provide a different implementation for that class. Another point for using injection instead of new is that the other class may rely on injection in the future. Using injection leaves the door open for future and unanticipated customizations. The Outline view now shows as in the following screenshot: We can further enrich the labels for entities and attributes using images for them. To do this, we create a directory in the org.example.entities.ui project where we place the image files of the icons we want to use. In order to benefit from Xtext's default handling of images, we call the directory icons, and we place two gif images there, Entity.gif and Attribute.gif (for entities and attributes, respectively). You fill find the icon files in the accompanying source code in the org.example.entities.ui/icons folder. We then define two image methods in EntitiesLabelProvider where we only need to return the name of the image files and Xtext will do the rest for us: class EntitiesLabelProvider extends DefaultEObjectLabelProvider {   ... as before def image(Entity e) { "Entity.gif" }   def image(Attribute a) { "Attribute.gif" } } You can see the result by relaunching Eclipse, as seen in the following screenshot: Now, the entities and attributes labels look nicer. If you plan to export the plugins for your DSL so that others can install them in their Eclipse, you must make sure that the icons directory is added to the build.properties file, otherwise that directory will not be exported. The bin.includes section of the build.properties file of your UI plugin should look like the following: bin.includes = META-INF/,                ., plugin.xml,                icons/ The Outline view The default Outline view comes with nice features. In particular, it provides toolbar buttons to keep the Outline view selection synchronized with the element currently selected in the editor. Moreover, it provides a button to sort the elements of the tree alphabetically. By default, the tree structure is built using the containment relations of the metamodel of the DSL. This strategy is not optimal in some cases. For example, an Attribute definition also contains the AttributeType element, which is a structured definition with children (for example, elementType, array, and length). This is reflected in the Outline view (refer to the previous screenshot) if you expand the Attribute elements. This shows unnecessary elements, such as BasicType names, which are now redundant since they are shown in the label of the attribute, and additional elements which are not representable with a name, such as the array feature. We can influence the structure of the Outline tree using the generated stub class EntitiesOutlineTreeProvider in the src folder org.example.entities.ui.outline. Also in this class, customizations are specified in a declarative way using the polymorphic dispatch mechanism. The official documentation, https://www.eclipse.org/Xtext/documentation/, details all the features that can be customized. In our example, we just want to make sure that the nodes for attributes are leaf nodes, that is, they cannot be further expanded and they have no children. In order to achieve this, we just need to define a method named _isLeaf (note the underscore) with a parameter of the type of the element, returning true. Thus, in our case we write the following code: classEntitiesOutlineTreeProviderextends DefaultOutlineTreeProvider { def _isLeaf(Attribute a) { true } } Let's relaunch Eclipse, and now see that the attribute nodes do not expose children anymore. Besides defining leaf nodes, you can also specify the children in the tree for a specific node by defining a _createChildren method taking as parameters the type of outline node and the type of the model element. This can be useful to define the actual root elements of the Outline tree. By default, the tree is rooted with a single node for the source file. In this example, it might be better to have a tree with many root nodes, each one representing an entity. The root of the Outline tree is always represented by a node of type DefaultRootNode. The root node is actually not visible, it is just the container of all nodes that will be displayed as roots in the tree. Thus, we define the following method (our Entities model is rooted by a Model element): public classEntitiesOutlineTreeProvider ... {   ... as before def void _createChildren(DocumentRootNodeoutlineNode,                            Model model) { model.entities.forEach[       entity | createNode(outlineNode, entity);    ]   } } This way, when the Outline tree is built, we create a root node for each entity instead of having a single root for the source file. The createNode method is part of the Xtext base class. The result can be seen in the following screenshot: Customizing other aspects We will show how to customize the content assistant. There is no need to do this for the simple Entities DSL since the default implementation already does a fine job. Custom formatting An editor for a DSL should provide a mechanism for rearranging the text of the program in order to improve its readability, without changing its semantics. For example, nested regions inside blocks should be indented, and the user should be able to achieve that with a menu. Besides that, implementing a custom formatter has also other benefits, since the formatter is automatically used by Xtext when you change the EMF model of the AST. If you tried to apply the quickfixes, you might have noticed that after the EMF model has changed, the editor immediately reflects this change. However, the resulting textual representation is not well formatted, especially for the quickfix that adds the missing referred entity. In fact, the EMF model representing the AST does not contain any information about the textual representation, that is, all white space characters are not part of the EMF model (after all, the AST is an abstraction of the actual program). Xtext keeps track of such information in another in-memory model called the nodemodel. The node model carries the syntactical information, that is, offset and length in the textual document. However, when we manually change the EMF model, we do not provide any formatting directives, and Xtext uses the default formatter to get a textual representation of the modified or added model parts. Xtext already generates the menu for formatting your DSL source programs in the Eclipse editor. As it is standard in Eclipse editors (for example, the JDT editor), you can access the Format menu from the context menu of the editor or using the Ctrl + Shift + F key combination. The default formatter is OneWhitespaceFormatter and you can test this in the Entities DSL editor; this formatter simply separates all tokens of your program with a space. Typically, you will want to change this default behavior. If you provide a custom formatter, this will be used not only when the Format menu is invoked, but also when Xtext needs to update the editor contents after a manual modification of the AST model, for example, a quickfix performing a semantic modification. The easiest way to customize the formatting is to have the Xtext generator create a stub class. To achieve this, you need to add the following formatter specification in the StandardLanguage block in the MWE2 workflow file, requesting to generate an Xtend stub class: language = StandardLanguage {     name = "org.example.entities.Entities" fileExtensions = "entities"     ...     formatter = { generateStub = true generateXtendStub = true     } } If you now run the workflow, you will find the formatter Xtend stub class in the main plugin project in the formatting2 package. For our Entities DSL, the class is org.example.entities.formatting2.EntitiesFormatter. This stub class extends the Xtext class AbstractFormatter2. Note that the name of the package ends with 2. That is because Xtext recently completely changed the customization of the formatter to enhance its mechanisms. The old formatter is still available, though deprecated, so the new formatter classes have the 2 in the package in order not to be mixed with the old formatter classes. In the generated stub class, you will get lots of warnings of the shape Discouraged access: the type AbstractFormatter2 is not accessible due to restriction on required project org.example.entities. That is because the new formatting API is still provisional, and it may change in future releases in a non-backward compatible way. Once you are aware of that, you can decide to ignore the warnings. In order to make the warnings disappear from the Eclipse project, you configure the specific project settings to ignore such warnings, as shown in the following screenshot: The Xtend stub class already implements a few dispatch methods, taking as parameters the AST element to format and an IFormattableDocument object. The latter is used to specify the formatting requests. A formatting request will result in a textual replacement in the program text. Since it is an extension parameter, you can use its methods as extension methods (for more details on extension methods. The IFormattableDocument interface provides a Java API for specifying formatting requests. Xtend features such as extension methods and lambdas will allow you to specify formatting request in an easy and readable way. The typical formatting requests are line wraps, indentations, space addition and removal, and so on. These will be applied on the textual regions of AST elements. As we will show in this section, the textual regions can be specified by the EObject of AST or by its keywords and features. For our Entities DSL, we decide to perform formatting as follows: Insert two newlines after each entity so that entities will be separated by an empty line; after the last entity, we want a single empty line. Indent attributes between entities curly brackets. Insert one line-wrap after each attribute declaration. Make sure that entity name, super entity, and the extends keyword are surrounded by a single space. Remove possible white spaces around the ; of an attribute declaration. To achieve the empty lines among entities, we modify the stub method for the Entities Model element: def dispatch void format(Model model,                                 extensionIFormattableDocument document) { vallastEntity = model.entities.last   for (entity : model.entities) { entity.format     if (entity === lastEntity) entity.append[setNewLines(1)]     else entity.append[setNewLines(2)]   } } We append two newlines after each entity. This way, each entity will be separated by an empty line, since each entity, except for the first one, will start on the second added newline. We append only one newline after the last entity. Now start a new Eclipse instance and manually test the formatter with some entities, by pressing Ctrl + Shift + F. We modify the format stub method for the Entity elements. In order to separate each attribute, we follow a logic similar to the previous format method. For the sake of the example, we use a different version of setNewLines, that is setNewLines(intminNewLines, intdefaultNewLines, intmaxNewLines), whose signature is self-explanatory: for (attribute : entity.attributes) { attribute.append[setNewLines(1, 1, 2)] } Up to now, we referred to a textual region of the AST by specifying the EObject. Now, we need to specify the textual regions of keywords and features of a given AST element. In order to specify that the "extends" keyword is surrounded by one single space we write the following: entity.regionFor.keyword("extends").surround[oneSpace] We also want to have no space around the terminating semicolon of attributes, so we write the following: attribute.regionFor.keyword(";").surround[noSpace] In order to specify that the the entity's name and the super entity are surrounded by one single space we write the following: entity.regionFor.feature(ENTITY__NAME).surround[oneSpace] entity.regionFor.feature(ENTITY__SUPER_TYPE).surround[oneSpace] After having imported statically all the EntitiesPackage.Literals members, as follows: import staticorg.example.entities.entities.EntitiesPackage.Literals.* Finally, we want to handle the indentation inside the curly brackets of an entity and to have a newline after the opening curly bracket. This is achieved with the following lines: val open = entity.regionFor.keyword("{") val close = entity.regionFor.keyword("}") open.append[newLine] interior(open, close)[indent] Summarizing, the format method for an Entity is the following one: def dispatch void format(Entity entity,                           extensionIFormattableDocument document) { entity.regionFor.keyword("extends").surround[oneSpace] entity.regionFor.feature(ENTITY__NAME).surround[oneSpace] entity.regionFor.feature(ENTITY__SUPER_TYPE).surround[oneSpace]   val open = entity.regionFor.keyword("{") val close = entity.regionFor.keyword("}") open.append[newLine]   interior(open, close)[indent]     for (attribute : entity.attributes) { attribute.regionFor.keyword(";").surround[noSpace] attribute.append[setNewLines(1, 1, 2)]   } } Now, start a new Eclipse instance and manually test the formatter with some attributes and entities, by pressing Ctrl + Shift + F. In the generated Xtend stub class, you also find an injected extension for accessing programmatically the elements of your grammar. In this DSL it is the following: @Inject extensionEntitiesGrammarAccess For example, to specify the left curly bracket of an entity, we could have written this alternative line: val open = entity.regionFor.keyword(entityAccess.leftCurlyBracketKeyword_3) Similarly, to specify the terminating semicolon of an attribute, we could have written this alternative line: attribute.regionFor.keyword(attributeAccess.semicolonKeyword_2)   .surround[noSpace] Eclipse content assist will help you in selecting the right method to use. Note that the method names are suffixed with numbers that relate to the position of the keyword in the grammar's rule. Changing a rule in the DSL's grammar with additional elements or by removing some parts will make such method invocations invalid since the method names will change. On the other hand, if you change a keyword in your grammar, for example, you use square brackets instead of curly brackets, then referring to keywords with string literals as we did in the original implementation of the format methods will issue no compilation errors, but the formatting will not work anymore as expected. Thus, you need to choose your preferred strategy according to the likeliness of your DSL's grammar evolution. You can also try and apply our quickfixes for missing entities and you will see that the added entity is nicely formatted, according to the logic we implemented. What is left to be done is to format the attribute type nicely, including the array specification. This is left as an exercise. The EntitiesFormatter you find in the accompanying sources of this example DSL contains also this formatting logic for attribute types. You should specify formatting requests avoiding conflicting requests on the same textual region. In case of conflicts, the formatter will throw an exception with the details of the conflict. Other customizations All the customizations you have seen so far were based on modification of a generated stub class with accompanying generated Guice bindings in the module under the src-gen directory. However, since Xtext relies on injection everywhere, it is possible to inject a custom implementation for any mechanism, even if no stub class has been generated. If you installed Xtext SDK in your Eclipse, the sources of Xtext are available for you to inspect. You should learn to inspect these sources by navigating to them and see what gets injected and how it is used. Then, you are ready to provide a custom implementation and inject it. You can use the Eclipse Navigate menu. In particular, to quickly open a Java file (even from a library if it comes with sources), use Ctrl + Shift + T (Open Type…). This works both for Java classes and Xtend classes. If you want to quickly open another source file (for example, an Xtext grammar file) use Ctrl + Shift + R (Open Resource…). Both dialogs have a text field where, if you start typing, the available elements soon show up. Eclipse supports CamelCase everywhere, so you can just type the capital letters of a compound name to quickly get to the desired element. For example, to open the EntitiesRuntimeModule Java class, use the Open Type… menu and just digit ERM to see the filtered results. As an example, we show how to customize the output directory where the generated files will be stored (the default is src-gen). Of course, this output directory can be modified by the user using the Properties dialog that Xtext generated for your DSL, but we want to customize the default output directory for Entities DSL so that it becomes entities-gen. The default output directory is retrieved internally by Xtext using an injected IOutputConfigurationProvider instance. If you take a look at this class (see the preceding tip), you will see the following: importcom.google.inject.ImplementedBy; @ImplementedBy(OutputConfigurationProvider.class) public interfaceIOutputConfigurationProvider {   Set<OutputConfiguration>getOutputConfigurations();   ...  The @ImplementedByGuice annotation tells the injection mechanism the default implementation of the interface. Thus, what we need to do is create a subclass of the default implementation (that is, OutputConfigurationProvider) and provide a custom binding for the IOutputConfigurationProvider interface. The method we need to override is getOutputConfigurations; if we take a look at its default implementation, we see the following: public Set<OutputConfiguration>getOutputConfigurations() { OutputConfigurationdefaultOutput = new OutputConfiguration(IFileSystemAccess.DEFAULT_OUTPUT); defaultOutput.setDescription("Output Folder"); defaultOutput.setOutputDirectory("./src-gen"); defaultOutput.setOverrideExistingResources(true); defaultOutput.setCreateOutputDirectory(true); defaultOutput.setCleanUpDerivedResources(true); defaultOutput.setSetDerivedProperty(true); defaultOutput.setKeepLocalHistory(true);   returnnewHashSet(defaultOutput); } Of course, the interesting part is the call to setOutputDirectory. We define an Xtend subclass as follows: classEntitiesOutputConfigurationProviderextends OutputConfigurationProvider {     public static valENTITIES_GEN = "./entities-gen"     overridegetOutputConfigurations() { super.getOutputConfigurations() => [ head.outputDirectory = ENTITIES_GEN     ]   } } Note that we use a public constant for the output directory since we might need it later in other classes. We use several Xtend features: the with operator, the implicit static extension method head, which returns the first element of a collection, and the syntactic sugar for setter method. We create this class in the main plug-in project, since this concept is not just an UI concept and it is used also in other parts of the framework. Since it deals with generation, we create it in the generatorsubpackage. Now, we must bind our implementation in the EntitiesRuntimeModule class: classEntitiesRuntimeModuleextends AbstractEntitiesRuntimeModule {   def Class<? extendsIOutputConfigurationProvider> bindIOutputConfigurationProvider() {     returnEntitiesOutputConfigurationProvider   } } If we now relaunch Eclipse, we can verify that the Java code is generated into entities-gen instead of src-gen. If you previously used the same project, the src-gen directory might still be there from previous generations; you need to manually remove it and set the new entities-gen as a source folder. Summary In this article, we introduced the Google Guice dependency injection framework on which Xtext relies. You should now be aware of how easy it is to inject custom implementations consistently throughout the framework. You also learned how to customize some basic runtime and IDE concepts for a DSL. Resources for Article: Further resources on this subject: Testing with Xtext and Xtend [article] Clojure for Domain-specific Languages - Design Concepts with Clojure [article] Java Development [article]
Read more
  • 0
  • 0
  • 4676
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-getting-ready-fight
Packt
08 Sep 2016
15 min read
Save for later

Getting Ready to Fight

Packt
08 Sep 2016
15 min read
In this article by Ashley Godbold author of book Mastering Unity 2D Game Development, Second Edition, we will start out by laying the main foundation for the battle system of our game. We will create the Heads Up Display (HUD) as well as design the overall logic of the battle system. The following topics will be covered in this article: Creating a state manager to handle the logic behind a turn-based battle system Working with Mecanim in the code Exploring RPG UI Creating the game's HUD (For more resources related to this topic, see here.) Setting up our battle statemanager The most unique and important part of a turn-based battle system is the turns. Controlling the turns is incredibly important, and we will need something to handle the logic behind the actual turns for us. We'll accomplish this by creating a battle state machine. The battle state manager Starting back in our BattleScene, we need to create a state machine using all of Mecanim's handy features. Although we will still only be using a fraction of the functionality with the RPG sample, I advise you to investigate and read more about its capabilities. Navigate to AssetsAnimationControllers and create a new Animator Controller called BattleStateMachine, and then we can begin putting together the battle state machine. The following screenshot shows you the states, transitions, and properties that we will need: As shown in the preceding screenshot, we have created eight states to control the flow of a battle with two Boolean parameters to control its transition. The transitions are defined as follows: From Begin_Battle to Intro BattleReadyset to true Has Exit Timeset to false (deselected) Transition Durationset to 0 From Intro to Player_Move Has Exit Timeset totrue Exit Timeset to0.9 Transition Durationset to2 From Player_Move to Player_Attack PlayerReadyset totrue Has Exit Timeset tofalse Transition Durationset to0 From Player_Attack to Change_Control PlayerReadyset tofalse Has Exit Timeset tofalse Transition Durationset to2 From Change_Control to Enemy_Attack Has Exit Timeset totrue Exit Timeset to0.9 Transition Durationset to2 From Enemy_Attack to Player_Move BattleReadyset totrue Has Exit Timeset tofalse Transition Durationset to2 From Enemy_Attack to Battle_Result BattleReadyset tofalse Has Exit Timeset tofalse Transition Timeset to2 From Battle_Result to Battle_End Has Exit Timeset totrue Exit Timeset to0.9 Transition Timeset to5 Summing up, what we have built is a steady flow of battle, which can be summarized as follows: The battle begins and we show a little introductory clip to tell the player about the battle. Once the player has control, we wait for them to finish their move. We then perform the player's attack and switch the control over to the enemy AI. If there are any enemies left, they get to attack the player (if they are not too scared and have not run away). If the battle continues, we switch back to the player, otherwise we show the battle result. We show the result for five seconds (or until the player hits a key), and then finish the battle and return the player to the world together with whatever loot and experience gained. This is just a simple flow, which can be extended as much as you want, and as we continue, you will see all the points where you could expand it. With our animator state machine created, we now just need to attach it to our battle manager so that it will be available when the battle runs; the following are the ensuing steps to do this: Open up BattleScene. Select the BattleManager game object in the project Hierarchy and add an Animator component to it. Now drag the BattleStateMachine animator controller we just created into the Controller property of the Animator component. The preceding steps attached our new battle state machine to our battle engine. Now, we just need to be able to reference the BattleStateMachine Mecanim state machine from theBattleManager script. To do so, open up the BattleManager script in AssetsScripts and add the following variable to the top of the class: private Animator battleStateManager; Then, to capture the configuredAnimator in our BattleManager script, we add the following to an Awake function place before the Start function: voidAwake(){ battleStateManager=GetComponent<Animator>(); if(battleStateManager==null){ Debug.LogError("NobattleStateMachineAnimatorfound."); }   } We have to assign it this way because all the functionality to integrate the Animator Controller is built into the Animator component. We cannot simply attach the controller directly to the BattleManager script and use it. Now that it's all wired up, let's start using it. Getting to the state manager in the code Now that we have our state manager running in Mecanim, we just need to be able to access it from the code. However, at first glance, there is a barrier to achieving this. The reason being that the Mecanim system uses hashes (integer ID keys for objects) not strings to identify states within its engine (still not clear why, but for performance reasons probably). To access the states in Mecanim, Unity provides a hashing algorithm to help you, which is fine for one-off checks but a bit of an overhead when you need per-frame access. You can check to see if a state's name is a specific string using the following: GetCurrentAnimatorStateInfo(0).IsName("Thing you're checking") But there is no way to store the names of the current state, to a variable. A simple solution to this is to generate and cache all the state hashes when we start and then use the cache to talk to the Mecanim engine. First, let's remove the placeholder code, for the old enum state machine.So, remove the following code from the top of the BattleManager script: enum BattlePhase {   PlayerAttack,   EnemyAttack } private BattlePhase phase; Also, remove the following line from the Start method: phase = BattlePhase.PlayerAttack; There is still a reference in the Update method for our buttons, but we will update that shortly; feel free to comment it out now if you wish, but don't delete it. Now, to begin working with our new state machine, we need a replica of the available states we have defined in our Mecanim state machine. For this, we just need an enumeration using the same names (you can create this either as a new C# script or simply place it in the BattleManager class) as follows: publicenumBattleState { Begin_Battle, Intro, Player_Move, Player_Attack, Change_Control, Enemy_Attack, Battle_Result, Battle_End } It may seem strange to have a duplicate of your states in the state machine and in the code; however, at the time of writing, it is necessary. Mecanim does not expose the names of the states outside of the engine other than through using hashes. You can either use this approach and make it dynamic, or extract the state hashes and store them in a dictionary for use. Mecanim makes the managing of state machines very simple under the hood and is extremely powerful, much better than trawling through code every time you want to update the state machine. Next, we need a location to cache the hashes the state machine needs and a property to keep the current state so that we don't constantly query the engine for a hash. So, add a new using statement to the beginning of the BattleManager class as follows: using System.Collections; using System.Collections.Generic; using UnityEngine; Then, add the following variables to the top of the BattleManager class: private Dictionary<int, BattleState> battleStateHash = new Dictionary<int, BattleState>(); private BattleState currentBattleState; Finally, we just need to integrate the animator state machine we have created. So, create a new GetAnimationStates method in the BattleManager class as follows: void GetAnimationStates() {   foreach (BattleState state in (BattleState[])System.Enum.     GetValues(typeof(BattleState)))   {     battleStateHash.Add(Animator.StringToHash       (state.ToString()), state);   } } This simply generates a hash for the corresponding animation state in Mecanim and stores the resultant hashes in a dictionary that we can use without having to calculate them at runtime when we need to talk to the state machine. Sadly, there is no way at runtime to gather the information from Mecanim as this information is only available in the editor. You could gather the hashes from the animator and store them in a file to avoid this, but it won't save you much. To complete this, we just need to call the new method in the Start function of the BattleManager script by adding the following: GetAnimationStates(); Now that we have our states, we can use them in our running game to control both the logic that is applied and the GUI elements that are drawn to the screen. Now add the Update function to the BattleManager class as follows: voidUpdate() {   currentBattleState = battleStateHash[battleStateManager.     GetCurrentAnimatorStateInfo(0).shortNameHash];     switch (currentBattleState)   {     case BattleState.Intro:       break;     case BattleState.Player_Move:       break;     case BattleState.Player_Attack:       break;     case BattleState.Change_Control:       break;     case BattleState.Enemy_Attack:       break;     case BattleState.Battle_Result:       break;     case BattleState.Battle_End:       break;     default:       break;   } } The preceding code gets the current state from the animator state machine once per frame and then sets up a choice (switch statement) for what can happen based on the current state. (Remember, it is the state machine that decides which state follows which in the Mecanim engine, not nasty nested if statements everywhere in code.) Now we are going to update the functionality that turns our GUI button on and off. Update the line of code in the Update method we wrote as follows: if(phase==BattlePhase.PlayerAttack){ so that it now reads: if(currentBattleState==BattleState.Player_Move){ This will make it so that the buttons are now only visible when it is time for the player to perform his/her move. With these in place, we are ready to start adding in some battle logic. Starting the battle As it stands, the state machine is waiting at the Begin_Battle state for us to kick things off. Obviously, we want to do this when we are ready and all the pieces on the board are in place. When the current Battle scene we added, starts, we load up the player and randomly spawn in a number of enemies into the fray using a co-routine function called SpawnEnemies. So, only when all the dragons are ready and waiting to be chopped down do we want to kick things off. To tell the state machine to start the battle, we simple add the following line just after the end of the forloop in the SpawnEnemies IEnumerator co-routine function: battleStateManager.SetBool("BattleReady", true); Now when everything is in place, the battle will finally begin. Introductory animation When the battle starts, we are going to display a little battle introductory image that states who the player is going to be fighting against. We'll have it slide into the scene and then slide out. You can do all sorts of interesting stuff with this introductory animation, like animating the individual images, but I'll leave that up to you to play with. Can't have all the fun now, can I? Start by creating a new Canvas and renaming it IntroCanvas so that we can distinguish it from the canvas that will hold our buttons. At this point, since we are adding a second canvas into the scene, we should probably rename ours to something that is easier for you to identify. It's a matter of preference, but I like to use different canvases for different UI elements. For example, one for the HUD, one for pause menus, one for animations, and so on. You can put them all on a single canvas and use Panels and CanvasGroup components to distinguish between them; it's really up to you. As a child of the new IntroCanvas, create a Panel with the properties shown in the following screenshot. Notice that the Imageoblect's Color property is set to black with the alpha set to about half: Now add as a child of the Panel two UI Images and a UI Text. Name the first image PlayerImage and set its properties as shown in the following screenshot. Be sure to set Preserve Aspect to true: Name the second image EnemyImage and set the properties as shown in the following screenshot: For the text, set the properties as shown in the following screenshot: Your Panel should now appear as mine did in the image at the beginning of this section. Now let's give this Panel its animation. With the Panel selected, select the Animation tab. Now hit the Create button. Save the animation as IntroSlideAnimation in the Assets/Animation/Clipsfolder. At the 0:00 frame, set the Panel's X position to 600, as shown in the following screenshot: Now, at the 0:45 frame, set the Panel's X position to 0. Place the playhead at the 1:20 frame and set the Panel's X position to 0, there as well, by selecting Add Key, as shown in the following screenshot: Create the last frame at 2:00 by setting the Panel's X position to -600. When the Panel slides in, it does this annoying bounce thing instead of staying put. We need to fix this by adjusting the animation curve. Select the Curves tab: When you select the Curves tab, you should see something like the following: The reason for the bounce is the wiggle that occurs between the two center keyframes. To fix this, right-click on the two center points on the curve represented by red dots and select Flat,as shown in the following screenshot: After you do so, the curve should be constant (flat) in the center, as shown in the following screenshot: The last thing we need to do to connect this to our BattleStateMananger isto adjust the properties of the Panel's Animator. With the Panel selected, select the Animator tab. You should see something like the following: Right now, the animation immediately plays when the scene is entered. However, since we want this to tie in with our BattleStateManager and only begin playing in the Intro state, we do not want this to be the default animation. Create an empty state within the Animator and set it as the default state. Name this state OutOfFrame. Now make a Trigger Parameter called Intro. Set the transition between the two states so that it has the following properties: The last things we want to do before we move on is make it so this animation does not loop, rename this new Animator, and place our Animator in the correct subfolder. In the project view, select IntroSlideAnimation from the Assets/Animation/Clips folder and deselect Loop Time. Rename the Panel Animator to VsAnimator and move it to the Assets/Animation/Controllersfolder. Currently, the Panel is appearing right in the middle of the screen at all times, so go ahead and set the Panel's X Position to600, to get it out of the way. Now we can access this in our BattleStateManager script. Currently, the state machine pauses at the Intro state for a few seconds; let's have our Panel animation pop in. Add the following variable declarations to our BattleStateManager script: public GameObjectintroPanel; Animator introPanelAnim; And add the following to the Awake function: introPanel Anim=introPanel.GetComponent<Animator>(); Now add the following to the case line of the Intro state in the Updatefunction: case BattleState.Intro: introPanelAnim.SetTrigger("Intro"); break; For this to work, we have to drag and drop the Panel into the Intro Panel slot in the BattleManager Inspector. As the battle is now in progress and the control is being passed to the player, we need some interaction from the user. Currently, the player can run away, but that's not at all interesting. We want our player to be able to fight! So, let's design a graphic user interface that will allow her to attack those adorable, but super mean, dragons. Summary Getting the battle right based on the style of your game is very important as it is where the player will spend the majority of their time. Keep the player engaged and try to make each battle different in some way, as receptiveness is a tricky problem to solve and you don't want to bore the player. Think about different attacks your player can perform that possibly strengthen as the player strengthens. In this article, you covered the following: Setting up the logic of our turn-based battle system Working with state machines in the code Different RPG UI overlays Setting up the HUD of our game so that our player can do more than just run away Resources for Article: Further resources on this subject: Customizing an Avatar in Flash Multiplayer Virtual Worlds [article] Looking Good – The Graphical Interface [article] The Vertex Functions [article]
Read more
  • 0
  • 0
  • 23415

article-image-introducing-iot-particles-photon-and-electron
Packt
07 Sep 2016
14 min read
Save for later

Introducing IoT with Particle's Photon and Electron

Packt
07 Sep 2016
14 min read
In this article by Rashid Khan, Kajari Ghoshdastidar, and Ajith Vasudevan, authors of the book Learning IoT with Particle Photon and Electron, we will have a brief walkthrough of the evolution of Internet of Things (IoT) followed by an overview of the basics of IoT-related software and hardware, which every IoT enthusiast should know. The discussion then moves on to introduce Particle, an IoT company (https://www.particle.io/), followed by a description of Particle's popular IoT products—Core, Photon and Electron. This article will cover following topics: Evolution of IoT Hardware and software in the IoT ecosystem Market survey of IoT development boards and cloud services What is Particle? Summary (For more resources related to this topic, see here.) Evolution of IoT It is not very clear exactly who coined the term IoT. Kevin Ashton (https://en.wikipedia.org/wiki/Kevin_Ashton) supposedly coined the phrase IoT while working for Procter & Gamble (P&G) in 1999. Kevin was then working on an RFID (https://en.wikipedia.org/wiki/Radio-frequency_identification) initiative by P&G, and proposed taking the system online to the Internet. In 2005, UN's International Telecommunications Union (ITU) - http://www.itu.int/, published its first report on IoT. In 2008, the global non-profit organization IPSO Alliance (http://www.ipso-alliance.org/) was launched to serve the various communities seeking to establish IoT by providing coordinated marketing efforts available to the general public. IPSO currently has more than 50 member companies including Google, Cisco, Intel, Texas Instruments, Bosch, Atmel. In 2012, IoT Consortium (IoTC) - http://iofthings.org/, was founded to educate technology firms, retailers, insurance companies, marketers, media companies, and the wider business community about the value of IoT. IoTC has more than 60 member companies in the area of hardware, software, and analytics, a few of them being Logitech, Node, and SigFox. A 2014 Forbes article by Gil Press mentions: "Gartner estimates that IoT product and service suppliers will generate incremental revenue exceeding $300 billion in 2020. IDC forecasts that the worldwide market for IoT solutions will grow from $1.9 trillion in 2013 to $7.1 trillion in 2020". Why IoT has become a household word now IoT has, in recent years, become quite popular and an everyday phenomenon primarily due to IoT-related hardware, software, accessories, sensors, and the Internet connection becoming very affordable and user friendly. An explosion in the availability of free Integrated Development Environments (IDEs) and Software Development Kits (SDKs) have made programming and deployment of IoT really simple and easy. Thus, IoT enthusiasts range from school kids, hobbyists, and non-programmers to embedded software engineers specialized in this area. Hardware and software in the IoT ecosystem Advancement in technology and affordability has made acquisition and usage of IoT devices very simple. However, in order to decide which IoT package (boards, accessories, sensors, software) to choose for a particular application, and actually building projects, it is essential to have knowledge of IoT terminology, hardware, and software. In this section, we will introduce the reader to the essential terminology used when dealing with IoT. This will also help the reader understand and appreciate the features of the Particle IoT products—Core, Photon, and Electron. Essential terminology Let's learn about a few terms that we're going to be hearing all throughout this article, and whenever we work with IoT hardware and software components: Term Definition IoT Development Board A development board is essentially a programmable circuit board which wraps an IoT device. The IoT device's processor/microcontroller, memory, communications ports, input-output pins, sensors, Wi-Fi module, and so on are exposed by the development board, in a convenient way, to the user. A board manufacturer usually provides an IDE with it to write and deploy code to the physical board. A development board with the IDE enables rapid prototyping of IoT projects. Microcontroller A microcontroller is a highly compact single Integrated Circuit (IC) with a processor and limited Random Access Memory (RAM) and Read Only Memory (ROM) embedded in it with programmable peripherals. Microcontrollers are "computer on a single chip". Because of its limited memory and architecture constraints, usually, only one specific program is deployable and runnable on a microcontroller at one time. Preprogrammed microcontrollers are used in electrical machinery such as washing machines, dish-washers, microwave, and so on. Microprocessor A microprocessor is a single integrated chip which in itself is a Central Processing Unit (CPU). The microprocessor has separate RAM and ROM modules, and digital inputs and outputs. The Microprocessor CPU is usually more powerful than that of a microcontroller, and there is provision to add larger amounts of memory externally. This makes microprocessors suitable for general-purpose programming, and are used in desktop computers, Laptops, and the like. Flash Memory Flash memory is an electronic non-volatile storage device, for example, USB pen-drives, memory cards, and so on. Data in Flash memory can be erased and rewritten. Unlike RAM, access speed is lower for flash memories, and also unlike RAM, the data stored in flash memory is not erased when power is switched off. Flash memories are generally used as reusable extra storage. RTOS RTOS as the name suggests, RTOS responds to events in real time. This means, as soon as an event occurs, a response is guaranteed within an acceptable and calculable amount of time. RTOS can be hard, firm, or soft depending on the amount of flexibility allowed in missing a task deadline. RTOS is essential in embedded systems, where real-time response is necessary. M2M Machine-to-Machine (M2M) communication encompasses communication between two or more machines (devices, computers, sensors, and so on) over a network (wireless/wired). Basically, a variant of IoT, where things are machines. Cloud Technology Cloud refers to computing resources available for use over a network (usually, the Internet). An end user can use such a resource on demand without having to install anything more than a lightweight client in the local machine. The major resources relevant to IoT include data storage, data analytics, data streaming, and communication with other devices. mBaaS Mobile Backend as a Service (mBaaS) is a infrastructure that provides cloud storage, data streaming, push notifications, and other related services for mobile application developers (web, native, IoT app development). The services are exposed via web-based APIs. BaaS is usually provided as a pay-per-use service. GPIO General Purpose Input Output (GPIO), these are electrical terminals or 'pins' exposed from ICs and IoT devices/boards that can be used to either send a signal to the device from the outside (input mode), or get a signal out from the inside of the device (output mode). Input or Output mode can be configured by the user at runtime. Module Unit of electronics, sometimes a single IC and at other times a group of components that may include ICs, providing a logical function to the device/board. For example, a Wi-Fi module provides Wi-Fi functionality to a board. Other examples are Bluetooth, Ethernet, USB, and so on, Port An electrical or Radio-Frequency-based interface available on a board through which external components can communicate with the board. For example, HDMI, USB, Ethernet, 3.5mm jack, UART (https://en.wikipedia.org/wiki/Universal_asynchronous_receiver/transmitter). Table 1: Terminology Network Protocols Connected smart devices need to communicate with each other, and exchange large volumes of messages between themselves and the cloud. To ensure near real-time response, smart bandwidth usage, and energy savings on the resource-constrained IoT devices, new protocols have been added to the traditional seven-layer network model (OSI model: https://en.wikipedia.org/wiki/OSI_model). The following table shows the major OSI network protocols and the IoT network protocols suitable for various smart, connected devices. Layer Examples of Traditional Network Protocols (OSI) Examples of IoT Network Protocols Application, Presentation, Session HTTP, FTP, SMTP, TLS, RPC, JSON, CSS, GIF, XML CoAP, MQTT, DDS, M2M service layer Transport TCP, UDP UDP, DTLS Network ICMP, IPsec, IPv4, IPv6 6LoWPAN, RPL (Zigbee) Data Link IEEE 802.2, L2TP, LLDP, MAC, PPP IEEE 802.15.4, BLE4.0, RFID, NFC, Cellular Physical DSL, Ethernet physical layer, RS-232, any physical transmission medium (for example, Cables) Wires, Sensor drivers to read from sensor devices Table 2: Layerwise Network Protocols – OSI vs IoT Market survey of IoT development boards and cloud services Here we list some of the most popular IoT boards and cloud services, available in the market at the time of writing this article, with some of their important specifications and features. These tables should help the reader get an idea as to where Particle products fit in on the IoT map. IoT development boards The next table lists the main specifications of popular IoT boards. These specs are the basic details one has to consider while selecting a board—its specifications in terms of processor and speed, memory, available communication modules and ports, and IO Pins. Also, while selecting a board, one has to analyze and match the project's requirements with the available boards, so that the right board is selected for the application in terms of fitment and performance. Board Name Microcontroller Microprocessor Memory Modules Ports IO Pins Raspberry Pi 1/2/3 Broadcom SoC BCM2835/6/7 Single/Quad-core ARM 11/Cortex-A7/A53 CPU, VideoCore IV GPU 256MB/512MB/1 GB RAM Ethernet, Wi-Fi, Serial UART, I2C HDMI, USB, Ethernet (RJ45), GPIO 26/40/40 Arduino Mini ATmega328 NA 32 KB Flash 2 KB SRAM NA NA 14 Arduino Yun ATmega32u4 Atheros AR9331 32 KB Flash 2.5 KB SRAM, 16 MB Flash, 64 MB RAM Wi-Fi, Ethernet USB, Ethernet (RJ45) 20 Intel Edison MCU at 100 MHz ( Intel Atom Soc) Dual-core CPU at 500 MHz (Intel Atom Soc) 4 GB Flash, 1 GB RAM Wi-Fi, Bluetooth 4.0 USB, UART, SPI, GPIO 28 Libelium Waspmote ATmega1281 NA 128 KB Flash, 8 KB SRAM Temp, Humidity, Light Sensors, (optional) GPS UART, I2C, SPI, USB 19 NodeMCU ESP8266 ESP 8266 SoC ESP-12 module 4 MB Flash Wi-Fi, Serial UART, ADC UART, GPIO, SPI 14 BeagleBone Black Sitara SoC AM3358/9 AM335x 1 GHz ARM Cortex-A8 512 MB RAM, 2/4 GB flash storage Ethernet, Serial UART, ADC, I2C Ethernet (RJ45), HDMI, USB, GPIO 24 CubieBoard ARM Cortex-A8 CPU AllWinner A10 SoC 512 MB/ 1 GB RAM, 4 GB flash memory Ethernet, Serial UART, ADC, I2C Ethernet (RJ45) , USB, SATA 96 Table 3: IoT development Boards Cloud services (PaaS, BaaS, M2M) It is important to know what kind of cloud service we will be dealing with, and whether our board has open standards and allows us to use our own personal service easily, or whether the board-provided service needs some manipulation to use in the current project. Cloud Service Name Salient Features Amazon Web Services (https://aws.amazon.com/) Microsoft Azure (https://azure.microsoft.com/) Cloud Foundry (https://www.cloudfoundry.org/) IBM Bluemix (http://www.ibm.com/cloud-computing/bluemix/) Platform as a Service (PaaS) Infrastructure (VM, Storage), Big Data Analytics, Application Services, Deployment and Management, Mobile and Device Services Parse (http://www.parse.com/) Kinvey (http://www.kinvey.com/) AnyPresence (http://www.anypresence.com/) Appcelerator (http://www.appcelerator.com/) mBaaS ThingWorx (https://www.thingworx.com/) M2M offering from PTC (http://www.ptc.com/) Table 4: Cloud services What is Particle? Particle (https://www.particle.io), formerly known as Spark, is a company started by Zach Supalla. It provides hardware and software for development of IoT projects. Journey of Particle The first company started by Zach Supalla in 2011 was known as Hex Goods, and it sold designer products online. In early 2012, Hex Goods was shut down, and Zach started a second company called Switch Devices, which dealt with connected lighting. Switch Devices was then renamed Spark Devices. The name Spark was used as it provided a double meaning to the founders. Spark stood for spark of light and also sparks of inspiration. In early 2013, Spark transformed to an IoT platform for engineers and developers. The name Spark also did not last long as the founders felt Spark created confusion for a lot of users. There exist 681 live trademarks that include the word Spark. Apart from the number of trademarks, there are some other great, unrelated software and hardware products employing the name Spark in them—some of them being Apache Spark, SparkFun, and Spark NZ. It has been reported that a lot of people logged on to Zach's #spark IRC channel and asked doubts about big data. The name Particle was finally chosen, as it gave plenty of room to grow in terms of products and offerings. Particle, in scientific terms, is a single discreet unit within a larger system. The name draws a parallel with the mission of Particle—the company which provides development kits and devices as single units used to build the greater whole of IoT. We'll cover Particle IoT products in depth, and see how and when they perform better than other IoT development boards. Why Particle? Today, the most recurring problem with all existing IoT prototyping boards is that of connectivity. In order to connect the existing boards to the Internet, additional components such as Wi-Fi or GSM modules have to be attached in the development environment as well as in production. Attaching external devices for communication is cumbersome, and adds another point of failure with frequent issues such as Internet unavailability, intermittent network connectivity, and so on. This leads to a bad experience for the developer. Developers have to frequently (re)write code, deploy it onto the device(s), test, debug, fix any bugs, rinse, and repeat. The problem with code deployment with existing boards is that the boards need to be connected to a computer, which means for even the smallest code update, the device/board needs to be connected to the developer's computer, either by moving the computer to the device (which may be located at a not-so-easily accessible location) or vice versa. This poses a problem when the device, after an update at the developer's site, has to be placed back in its original production environment for testing and debugging the new changes. This means large turnaround times to load new code into production. Particle provides products that have built-in Wi-Fi modules or GSM modules, which help in easy connection to a network or the internet, with support for OTA (Over-The-Air) code deployment. This removes the hassle of adding extra modules on the prototyping boards for connectivity, and it also allows for pushing code or testing/debugging on site. As previously mentioned, one of the important features which differentiates Particle products from other devices is the Particle device's ability of deployment of code over the air. New code can be deployed onto the device or burnt, as the process is called in embedded systems' parlance, via REST API calls, which makes it very convenient to provide updates. This feature of Particle products helps in faster code release cycle and testing/debugging. What does Particle offer? Particle offers a suite of hardware and software tools to help prototype, scale, and manage the IoT products. It also provides the ability to build cloud-connected IoT prototypes quickly. If you're satisfied with your prototype and want to productize your IoT design, no problem there. It helps us to go from a single prototype to millions of units with a cloud platform that can scale as the number of devices grow. The popular Particle hardware devices are listed as follows: Core: A tiny Wi-Fi development kit for prototyping and scaling your IoT product. Reprogrammable and connected to the cloud, this has now been superseded by the Photon. Photon: A tiny Wi-Fi development kit for prototyping and scaling your IoT product. Reprogrammable and connected to the cloud. Electron: A tiny development kit for creating 2G/3G cellular connected products. The Photon and the Core are bundled with Wi-Fi modules, which help them connect to a network or the Internet without adding any extra modules. The Electron has a 3G/2G GSM module, which can be used to send or receive messages directly or connect to the Internet. The firmware for the Photon, Electron, and Core can be written in a web-based IDE provided by Particle, and the deployment of the firmware code to the device is done over-the-air. Particle also offers SDKs for mobile and web to extend the IoT experience from the devices/sensors to the phone and web. Summary In this article, we learnt about IoT, and how it all began. We briefly touched upon major organizations involved in IoT, common terminology used, and we looked at different hardware products and cloud services we have available for building IoT projects.  Resources for Article: Further resources on this subject: Identity and Access-Management Solutions for the IoT [article] Internet of Things with BeagleBone [article] The Internet of Things [article]
Read more
  • 0
  • 0
  • 14637

article-image-introducing-penetration-testing
Packt
07 Sep 2016
28 min read
Save for later

Introducing Penetration Testing

Packt
07 Sep 2016
28 min read
In this article by Kevin Cardwell, the author of the book Building Virtual Pentesting Labs for Advanced Penetration Testing, we will discuss the role that pen testing plays in the professional security testing framework. We will discuss the following topics: Defining security testing An abstract security testing methodology Myths and misconceptions about pen testing (For more resources related to this topic, see here.) If you have been doing penetration testing for some time and are very familiar with the methodology and concept of professional security testing, you can skip this article or just skim it. But you might learn something new or at least a different approach to penetration testing. We will establish some fundamental concepts in this article. Security testing If you ask 10 consultants to define what security testing is today, you will more than likely get a variety of responses. Here is the Wikipedia definition: "Security testing is a process and methodology to determine that an information system protects and maintains functionality as intended." In my opinion, this is the most important aspect of penetration testing. Security is a process and not a product. I'd also like to add that it is a methodology and not a product. Another component to add to our discussion is the point that security testing takes into account the main areas of a security model. A sample of this is as follows: Authentication Authorization Confidentiality Integrity Availability Non-repudiation Each one of these components has to be considered when an organization is in the process of securing their environment. Each one of these areas in itself has many subareas that also have to be considered when it comes to building a secure architecture. The lesson is that when testing security, we have to address each of these areas. Authentication It is important to note that almost all systems and/or networks today have some form of authentication and, as such, it is usually the first area we secure. This could be something as simple as users selecting a complex password or us adding additional factors to authentication, such as a token, biometrics, or certificates. No single factor of authentication is considered to be secure by itself in today's networks. Authorization Authorization is often overlooked since it is assumed and not a component of some security models. That is one approach to take, but it's preferred to include it in most testing models, as the concept of authorization is essential since it is how we assign the rights and permissions to access a resource, and we would want to ensure it is secure. Authorization enables us to have different types of users with separate privilege levels coexist within a system. We do this when we have the concept of discretionary access, where a user can have administrator privileges on a machine or assume the role of an administrator to gain additional rights or permissions, whereas we might want to provide limited resource access to a contractor. Confidentiality Confidentiality is the assurance that something we want to be protected on the machine or network is safe and not at risk of being compromised. This is made harder by the fact that the protocol (TCP/IP) running the Internet today is a protocol that was developed in the early 1970s. At that time, the Internet consisted of just a few computers, and now, even though the Internet has grown to the size it is today, we are still running the same protocol from those early days. This makes it more difficult to preserve confidentiality. It is important to note that when the developers created the protocol and the network was very small, there was an inherent sense of trust on who you could potentially be communicating with. This sense of trust is what we continue to fight from a security standpoint today. The concept from that early creation was and still is that you can trust data received to be from a reliable source. We know now that the Internet is at this huge size and that is definitely not the case. Integrity Integrity is similar to confidentiality, in that we are concerned with the compromising of information. Here, we are concerned with the accuracy of data and the fact that it is not modified in transit or from its original form. A common way of doing this is to use a hashing algorithm to validate that the file is unaltered. Availability One of the most difficult things to secure is availability, that is, the right to have a service when required. The irony of availability is that a particular resource is available to one user, and it is later available to all. Everything seems perfect from the perspective of an honest/legitimate user. However, not all users are honest/legitimate, and due to the sheer fact that resources are finite, they can be flooded or exhausted; hence, is it is more difficult to protect this area. Non-repudiation Non-repudiation makes the claim that a sender cannot deny sending something after the fact. This is the one I usually have the most trouble with, because a computer system can be compromised and we cannot guarantee that, within the software application, the keys we are using for the validation are actually the ones being used. Furthermore, the art of spoofing is not a new concept. With these facts in our minds, the claim that we can guarantee the origin of a transmission by a particular person from a particular computer is not entirely accurate. Since we do not know the state of the machine with respect to its secureness, it would be very difficult to prove this concept in a court of law. All it takes is one compromised machine, and then the theory that you can guarantee the sender goes out the window. We won't cover each of the components of security testing in detail here, because that is beyond the scope of what we are trying to achieve. The point I want to get across in this article is that security testing is the concept of looking at each and every one of these and other components of security, addressing them by determining the amount of risk an organization has from them, and then mitigating that risk. An abstract testing methodology As mentioned previously, we concentrate on a process and apply that to our security components when we go about security testing. For this, I'll describe an abstract methodology here. We will define our testing methodology as consisting of the following steps: Planning Nonintrusive target search Intrusive target search Data analysis Reporting Planning Planning is a crucial step of professional testing. But, unfortunately, it is one of the steps that is rarely given the time that is essentially required. There are a number of reasons for this, but the most common one is the budget: clients do not want to provide consultants days and days to plan their testing. In fact, planning is usually given a very small portion of the time in the contract due to this reason. Another important point about planning is that a potential adversary is going to spend a lot of time on it. There are two things we should tell clients with respect to this step that as a professional tester we cannot do but an attacker could: 6 to 9 months of planning: The reality is that a hacker who targets someone is going to spend a lot of time before the actual attack. We cannot expect our clients to pay us for 6 to 9 months of work just to search around and read on the Internet. Break the law: We could break the law and go to jail, but it is not something that is appealing for most. Additionally, being a certified hacker and licensed penetration tester, you are bound to an oath of ethics, and you can be pretty sure that breaking the law while testing is a violation of this code of ethics. Nonintrusive target search There are many names that you will hear for nonintrusive target search. Some of these are open source intelligence, public information search, and cyber intelligence. Regardless of which name you use, they all come down to the same thing: using public resources to extract information about the target or company you are researching. There is a plethora of tools that are available for this. We will briefly discuss those tools to get an idea of the concept, and those who are not familiar with them can try them out on their own. Nslookup The nslookup tool can be found as a standard program in the majority of the operating systems we encounter. It is a method of querying DNS servers to determine information about a potential target. It is very simple to use and provides a great deal of information. Open a command prompt on your machine, and enter nslookup www.packtpub.com. This will result in output like the following screenshot:  As you can see, the response to our command is the IP address of the DNS server for the www.packtpub.com domain. If we were testing this site, we would have explored this further. Alternatively, we may also use another great DNS-lookup tool called dig. For now, we will leave it alone and move to the next resource. Central Ops The https://centralops.net/co/ website has a number of tools that we can use to gather information about a potential target. There are tools for IP, domains, name servers, e-mail, and so on. The landing page for the site is shown in the next screenshot: The first thing we will look at in the tool is the ability to extract information from a web server header page: click on TcpQuery, and in the window that opens, enter www.packtpub.com and click on Go. An example of the output from this is shown in the following screenshot: As the screenshot shows, the web server banner has been modified and says packt. If we do the query against the www.packtpub.com domain we have determined that the site is using the Apache web server, and the version that is running; however we have much more work to do in order to gather enough information to target this site. The next thing we will look at is the capability to review the domain server information. This is accomplished by using the domain dossier. Return to the main page, and in the Domain Dossier dialog box, enter yahoo.com and click on go. An example of the output from this is shown in the following screenshot:  There are many tools we could look at, but again, we just want to briefly acquaint ourselves with tools for each area of our security testing procedure. If you are using Windows and you open a command prompt window and enter tracert www.microsoft.com, you will observe that it fails, as indicated in this screenshot:  The majority of you reading this article probably know why this is blocked; for those of you who do not, it is because Microsoft has blocked the ICMP protocol, which is what the tracert command uses by default. It is simple to get past this because the server is running services; we can use those protocols to reach it, and in this case, that protocol is TCP. If you go to http://www.websitepulse.com/help/testtools.tcptraceroute-test.html and enter www.microsoft.com in the IP address/domain field with the default location and conduct the TCP Traceroute test, you will see it will now be successful, as shown in the following screenshot: As you can see, we now have additional information about the path to the potential target; moreover, we have additional machines to add to our target database as we conduct our test within the limits of the rules of engagement. The Wayback Machine The Wayback Machine is proof that nothing that has ever been on the Internet leaves! There have been many assessments in which a client informed the team that they were testing a web server that hadn't placed into production, and when they were shown the site had already been copied and stored, they were amazed that this actually does happen. I like to use the site to download some of my favorite presentations, tools, and so on, that have been removed from a site or, in some cases, whose site no longer exists. As an example, one of the tools used to show students the concept of steganography is the tool infostego. This tool was released by Antiy Labs, and it provided students an easy-to-use tool to understand the concepts. Well, if you go to their site at http://www.antiy.net/, you will find no mention of the tool—in fact, it will not be found on any of their pages. They now concentrate more on the antivirus market. A portion from their page is shown in the following screenshot: Now, let's try and use the power of the Wayback Machine to find our software. Open the browser of your choice and go to www.archive.org. The Wayback Machine is hosted there and can be seen in the following screenshot: As indicated, there are 491 billion pages archived at the time of writing this article. In the URL section, enter www.antiy.net and hit Enter. This will result in the site searching its archives for the entered URL. After a few moments, the results of the search will be displayed. An example of this is shown in the following screenshot: We know we don't want to access a page that has been recently archived, so to be safe, click on 2008. This will result in the calendar being displayed and showing all the dates in 2008 on which the site was archived. You can select any one that you want; an example of the archived site from December 18 is shown in the following screenshot: as you can see, the infostego tool is available, and you can even download it! Feel free to download and experiment with the tool if you like. Shodan The Shodan site is one of the most powerful cloud scanners available. You are required to register with the site to be able to perform the more advanced types of queries. To access the site, go to https://www.shodan.io/. It is highly recommended that you register, since the power of the scanner and the information you can discover is quite impressive, especially after registration. The page that is presented once you log in is shown in the following screenshot: The screenshot shows recently shared search queries as well as the most recent searches the logged-in user has conducted. This is another tool you should explore deeply if you do professional security testing. For now, we will look at one example and move on, since an entire article could be written just on this tool. If you are logged in as a registered user, you can enter iphone us into the search query window. This will return pages with iphone in the query and mostly in the United States, but as with any tool, there will be some hits on other sites as well. An example of the results of this search is shown in the following screenshot: Intrusive target search This is the step that starts the true hacker-type activity. This is when you probe and explore the target network; consequently, ensure that you have with you explicit written permission to carry out this activity. Never perform an intrusive target search without permission, as this written authorization is the only aspect which differentiates you and a malicious hacker. Without it, you are considered a criminal like them. Within this step, there are a number of components that further define the methodology. Find live systems No matter how good our skills are, we need to find systems that we can attack. This is accomplished by probing the network and looking for a response. One of the most popular tools to do this with is the excellent open source tool nmap, written by Fyodor. You can download nmap from https://nmap.org/, or you can use any number of toolkit distributions for the tool. We will use the exceptional penetration-testing framework Kali Linux. You can download the distribution from https://www.kali.org/. Regardless of which version of nmap you explore with, they all have similar, if not the same, command syntax. In a terminal window, or a command prompt window if you are running it on Windows, type nmap –sP <insert network IP address>. The network we are scanning is the 192.168.4.0/24 network; yours more than likely will be different. An example of this ping sweep command is shown in the following screenshot:  We now have live systems on the network that we can investigate further. For those of you who would like a GUI tool, you can use Zenmap. Discover open ports Now that we have live systems, we want to see what is open on these machines. A good analogy to a port is a door, and it's that if the door is open, I can approach it. There might be things that I have to do once I get to the door to gain access, but if it is open, then I know it is possible to get access, and if it is closed, then I know I cannot go through that door. Furthermore, we might need to know the type of lock that is on the door, because it might have weaknesses or additional protection that we need to know about. The same is with ports: if they are closed, then we cannot go into that machine using that port. We have a number of ways to check for open ports, and we will continue with the same theme and use nmap. We have machines that we have identified, so we do not have to scan the entire network as we did previously—we will only scan the machines that are up. Additionally, one of the machines found is our own machine; therefore, we will not scan ourselves—we could, but it's not the best plan. The targets that are live on our network are 1, 2, 16, and 18. We can scan these by entering nmap –sS 192.168.4.1,2,16,18. Those of you who want to learn more about the different types of scans can refer to http://nmap.org/book/man-port-scanning-techniques.html. Alternatively, you can use the nmap –h option to display a list of options. The first portion of the stealth scan (not completing the three-way handshake) result is shown in the following screenshot:   Discover services We now have live systems and openings that are on the machine. The next step is to determine what, if anything, is running on the ports we have discovered—it is imperative that we identify what is running on the machine so that we can use it as we progress deeper into our methodology. We once again turn to nmap. In most command and terminal windows, there is history available; hopefully, this is the case for you and you can browse through it with the up and down arrow keys on your keyboard. For our network, we will enter nmap –sV 192.168.4.1. From our previous scan, we've determined that the other machines have all scanned ports closed, so to save time, we won't scan them again. An example of this is shown in the following screenshot: From the results, you can now see that we have additional information about the ports that are open on the target. We could use this information to search the Internet using some of the tools we covered earlier, or we could let a tool do it for us. Enumeration Enumeration is the process of extracting more information about the potential target to include the OS, usernames, machine names, and other details that we can discover. The latest release of nmap has a scripting engine that will attempt to discover a number of details and in fact enumerate the system to some aspect. To process the enumeration with nmap, use the -A option. Enter nmap -A 192.168.4.1. Remember that you will have to enter your respective target address, which might be different from the one mentioned here. Also, this scan will take some time to complete and will generate a lot of traffic on the network. If you want an update, you can receive one at any time by pressing the spacebar. This command's output is quite extensive; so a truncated version is shown in the following screenshot:   As you can see, you have a great deal of information about the target, and you are quite ready to start the next phase of testing. Additionally, we have the OS correctly identified; until this step, we did not have that. Identify vulnerabilities After we have processed the steps up to this point, we have information about the services and versions of the software that are running on the machine. We could take each version and search the Internet for vulnerabilities, or we could use a tool—for our purposes, we will choose the latter. There are numerous vulnerability scanners out there in the market, and the one you select is largely a matter of personal preference. The commercial tools for the most part have a lot more information and details than the free and open source ones, so you will have to experiment and see which one you prefer. We will be using the Nexpose vulnerability scanner from Rapid7. There is a community version of their tool that will scan a limited number of targets, but it is worth looking into. You can download Nexpose from http://www.rapid7.com/. Once you have downloaded it, you will have to register, and you'll receive a key by e-mail to activate it. I will leave out the details of this and let you experience them on your own. Nexpose has a web interface, so once you have installed and started the tool, you have to access it. You can access it by entering https://localhost:3780. It seems to take an extraordinary amount of time to initialize, but eventually, it will present you with a login page, as shown in the following screenshot: The credentials required for login will have been created during the installation. It is quite an involved process to set up a scan, and since we are just detailing the process and there is an excellent quick start guide available, we will just move on to the results of the scan. We will have plenty of time to explore this area as the article progresses. The result of a typical scan is shown in the following screenshot: As you can see, the target machine is in bad shape. One nice thing about Nexpose is the fact that since they also own Metasploit, they will list the vulnerabilities that have a known exploit within Metasploit. Exploitation This is the step of the security testing that gets all the press, and it is, in simple terms, the process of validating a discovered vulnerability. It is important to note that it is not a 100-percent successful process—some vulnerabilities will not have exploits and some will have exploits for a certain patch level of the OS but not others. As I like to say, it is not an exact science and in reality is an infinitesimal part of professional security testing, but it is fun, so we will briefly look at the process. We also like to say in security testing that we have to validate and verify everything a tool reports to our client, and that is what we try to do with exploitation. The point is that you are executing a piece of code on a client's machine, and this code could cause damage. The most popular free tool for exploitation is the Rapid7-owned tool Metasploit. There are entire articles written on using the tool, so we will just look at the results of running it and exploiting a machine here. As a reminder, you have to have written permission to do this on any network other than your own; if in doubt, do not attempt it. Let's look at the options: There is quite a bit of information in the options. The one we will cover is the fact that we are using the exploit for the MS08-067 vulnerability, which is a vulnerability in the server service. It is one of the better ones to use as it almost always works and you can exploit it over and over again. If you want to know more about this vulnerability, you can check it out here: http://technet.microsoft.com/en-us/security/bulletin/ms08-067. Since the options are set, we are ready to attempt the exploit, and as indicated in the following screenshot, we are successful and have gained a shell on the target machine. The process for this we will cover as we progress through the article. For now, we will stop here. Here onward, it is only your imagination that can limit you. The shell you have opened is running at system privileges; therefore, it is the same as running a Command Prompt on any Windows machine with administrator rights, so whatever you can do in that shell, you can also do in this one. You can also do a number of other things, which you will learn as we progress through the article. Furthermore, with system access, we can plant code as malware: a backdoor or really anything we want. While we might not do that as a professional tester, a malicious hacker could do it, and this would require additional analysis to discover on the client's end. Data analysis Data analysis is often overlooked, and it can be a time-consuming process. This is the process that takes the most time to develop. Most testers can run tools and perform manual testing and exploitation, but the real challenge is taking all of the results and analyzing them. We will look at one example of this in the next screenshot. Take a moment and review the protocol analysis captured with the tool Wireshark—as an analyst, you need to know what the protocol analyzer is showing you. Do you know what exactly is happening? Do not worry, I will tell you after we have a look at the following screenshot: You can observe that the machine with the IP address 192.168.3.10 is replying with an ICMP packet that is type 3 code 13; in other words, the reason the packet is being rejected is because the communication is administratively filtered. Furthermore, this tells us that there is a router in place and it has an access control list (ACL) that is blocking the packet. Moreover, it tells us that the administrator is not following best practices— absorbing packets and not replying with any error messages that can assist an attacker. This is just a small example of the data analysis step; there are many things you will encounter and many more that you will have to analyze to determine what is taking place in the tested environment. Remember: the smarter the administrator, the more challenging pen testing can become—which is actually a good thing for security! Reporting Reporting is another one of the areas in testing that is often overlooked in training classes. This is unfortunate since it is one of the most important things you need to master. You have to be able to present a report of your findings to the client. These findings will assist them in improving their security practices, and if they like the report, it is what they will most often share with partners and other colleagues. This is your advertisement for what separates you from others. It is a showcase that not only do you know how to follow a systematic process and methodology of professional testing, you also know how to put it into an output form that can serve as a reference going forward for the clients. At the end of the day, as professional security testers, we want to help our clients improve their security scenario, and that is where reporting comes in. There are many references for reports, so the only thing we will cover here is the handling of findings. There are two components we use when it comes to findings, the first of which is a summary-of-findings table. This is so the client can reference the findings early on in the report. The second is the detailed findings section. This is where we put all of the information about the findings. We rate them according to severity and include the following: Description This is where we provide the description of the vulnerability, specifically, what it is and what is affected. Analysis and exposure For this article, you want to show the client that you have done your research and aren't just repeating what the scanning tool told you. It is very important that you research a number of resources and write a good analysis of what the vulnerability is, along with an explanation of the exposure it poses to the client site. Recommendations We want to provide the client a reference to the patches and measures to apply in order to mitigate the risk of discovered vulnerabilities. We never tell the client not to use the service and/or protocol! We do not know what their policy is, and it might be something they have to have in order to support their business. In these situations, it is our job as consultants to recommend and help the client determine the best way to either mitigate the risk or remove it. When a patch is not available, we should provide a reference to potential workarounds until one is available. References If there are references such as a Microsoft bulletin number or a Common Vulnerabilities and Exposures (CVE) number, this is where we would place them. Myths and misconceptions about pen testing After more than 20 years of performing professional security testing, it is amazing to me really how many are confused as to what a penetration test is. I have on many occasions gone to a meeting where the client is convinced they want a penetration test, and when I explain exactly what it is, they look at me in shock. So, what exactly is a penetration test? Remember our abstract methodology had a step for intrusive target searching and part of that step was another methodology for scanning? Well, the last item in the scanning methodology, exploitation, is the step that is indicative of a penetration test. That's right! That one step is the validation of vulnerabilities, and this is what defines penetration testing. Again, it is not what most clients think when they bring a team in. The majority of them in reality want a vulnerability assessment. When you start explaining to them that you are going to run exploit code and all these really cool things on their systems and/or networks, they usually are quite surprised. The majority of the times, the client will want you to stop at the validation step. On some occasions, they will ask you to prove what you have found, and then you might get to show validation. I once was in a meeting with the IT department of a foreign country's stock market, and when I explained what we were about to do for validating vulnerabilities, the IT director's reaction was, "Those are my stock broker records, and if we lose them, we lose a lot of money!" Hence, we did not perform the validation step in that test. Summary In this article, we defined security testing as it relates to this article, and we identified an abstract methodology that consists of the following steps: planning, nonintrusive target search, intrusive target search, data analysis, and reporting. More importantly, we expanded the abstract model when it came to intrusive target searching, and we defined within that a methodology for scanning. This consisted of identifying live systems, looking at open ports, discovering services, enumeration, identifying vulnerabilities, and finally, exploitation. Furthermore, we discussed what a penetration test is and that it is a validation of vulnerabilities and is associated with one step in our scanning methodology. Unfortunately, most clients do not understand that when you validate vulnerabilities, it requires you to run code that could potentially damage a machine or, even worse, damage their data. Because of this, once they discover this, most clients ask that it not be part of the test. We created a baseline for what penetration testing is in this article, and we will use this definition throughout the remainder of this article. In the next article, we will discuss the process of choosing your virtual environment.  Resources for Article:  Further resources on this subject: CISSP: Vulnerability and Penetration Testing for Access Control [article] Web app penetration testing in Kali [article] BackTrack 4: Security with Penetration Testing Methodology [article]
Read more
  • 0
  • 0
  • 27384

article-image-introducing-mobile-forensics
Packt
07 Sep 2016
21 min read
Save for later

Introducing Mobile Forensics

Packt
07 Sep 2016
21 min read
In this article by Oleg Afonin and Vladimir Katalov, the authors of the book Mobile Forensics – Advanced Investigative Strategies, we will see that today's smartphones are used less for calling and more for socializing, this has resulted in the smartphones holding a lot of sensitive data about their users. Mobile devices keep the user's contacts from a variety of sources (including the phone, social networks, instant messaging, and communication applications), information about phone calls, sent and received text messages, and e-mails and attachments. There are also browser logs and cached geolocation information; pictures and videos taken with the phone's camera; passwords to cloud services, forums, social networks, online portals, and shopping websites; stored payment data; and a lot of other information that can be vital for an investigation. (For more resources related to this topic, see here.) Needless to say, this information is very important for corporate and forensic investigations. In this book, we'll discuss not only how to gain access to all this data, but also what type of data may be available in each particular case. Tablets are no longer used solely as entertainment devices. Equipped with powerful processors and plenty of storage, even the smallest tablets are capable of running full Windows, complete with the Office suite. While not as popular as smartphones, tablets are still widely used to socialize, communicate, plan events, and book trips. Some smartphones are equipped with screens as large as 6.4 inches, while many tablets come with the ability to make voice calls over cellular network. All this makes it difficult to draw a line between a phone (or phablet) and a tablet. Every smartphone on the market has a camera that, unlike a bigger (and possibly better) camera, is always accessible. As a result, an average smartphone contains more photos and videos than a dedicated camera; sometimes, it's gigabytes worth of images and video clips. Smartphones are also storage devices. They can be used (and are used) to keep, carry, and exchange information. Smartphones connected to a corporate network may have access to files and documents not meant to be exposed. Uncontrolled access to corporate networks from employees' smartphones can (and does) cause leaks of highly-sensitive information. Employees come and go. With many companies allowing or even encouraging bring your own device policies, controlling the data that is accessible to those connecting to a corporate network is essential. What You Get Depends on What You Have Unlike personal computers that basically present a single source of information (the device itself consisting of hard drive(s) and volatile memory), mobile forensics deals with multiple data sources. Depending on the sources that are available, investigators may use one or the other tool to acquire information. The mobile device If you have access to the mobile device, you can attempt to perform physical or logical acquisition. Depending on the device itself (hardware) and the operating system it is running, this may or may not be possible. However, physical acquisition still counts as the most complete and up-to-date source of evidence among all available. Generally speaking, physical acquisition is available for most Android smartphones and tablets, older Apple hardware (iPhones up to iPhone 4, the original iPad, iPad mini, and so on), and recent Apple hardware with known passcode. As a rule, Apple devices can only be physically acquired if jailbroken. Since a jailbreak obtains superuser privileges by exploiting a vulnerability in iOS, and Apple actively fixes such vulnerabilities, physical acquisition of iOS devices remains iffy. A physical acquisition technique has been recently developed for some Windows phone devices using Cellebrite Universal Forensic Extraction Device (UFED). Physical acquisition is also available for 64-bit Apple hardware (iPhone 5S and newer, iPad mini 2, and so on). It is worth noting that physical acquisition of 64-bit devices is even more restrictive compared to the older 32-bit hardware, as it requires not only jailbreaking the device and unlocking it with a passcode, but also removing the said passcode from the security settings. Interestingly, according to Apple, even Apple itself cannot extract information from 64-bit iOS devices running iOS 8 and newer, even if they are served a court order. Physical acquisition is available on a limited number of BlackBerry smartphones running BlackBerry OS 7 and earlier. For BlackBerry smartphones, physical acquisition is available for unlocked BlackBerry 7 and lower devices, where supported, using Cellebrite UFED Touch/4PC through the bootloader method. For BlackBerry 10 devices where device encryption is not enabled, a chip-off can successfully acquire the device memory by parsing the physical dump using Cellebrite UFED. Personal computer Notably, the user's personal computer can help acquiring mobile evidence. The PC may contain the phone's offline data backups (such as those produced by Apple iTunes) that contain most of the information stored in the phone and available (or unavailable) during physical acquisition. Lockdown records are created when an iOS device is physically connected to the computer and authorized through iTunes. Lockdown records may be used to gain access to an iOS device without entering the passcode. In addition, the computer may contain binary authentication tokens that can be used to access respective cloud accounts linked to user's mobile devices. Access to cloud storage Many smartphones and tablets, especially those produced by Apple, offer the ability to back up information into an online cloud. Apple smartphones, for example, will automatically back up their content to Apple iCloud every time they are connected to a charger within the reach of a known Wi-Fi network. Windows phone devices exhibit similar behavior. Google, while not featuring full cloud backups like Apple or Microsoft, collects and retains even more information through Google Mobile Services (GMS). This information can also be pulled from the cloud. Since cloud backups are transparent, non-intrusive and require no user interaction, they are left enabled by default by many smartphone users, which makes it possible for an investigator to either acquire the content of the cloud storage or request it from the respective company with a court order. In order to successfully access the phone's cloud storage, one needs to know the user's authentication credentials (login and password). It may be possible to access iCloud by using binary authentication tokens extracted from the user's computer. With manufacturers quickly advancing in their security implementations, cloud forensics is quickly gaining importance and recognition among digital forensic specialists. Stages of mobile forensics This section will briefly discuss the general stages of mobile forensics and is not intended to provide a detailed explanation of each stage. There is more-than-sufficient documentation that can be easily accessed on the Internet that provides an intimate level of detail regarding the stages of mobile forensics. The most important concept for the reader to understand is this: have the least level of impact on the mobile device during all the stages. In other words, an examiner should first work on the continuum of the least-intrusive method to the most-intrusive method, which can be dictated by the type of data needing to be obtained from the mobile device and complexity of the hardware/software of the mobile device. Stage one – device seizure This stage pertains to the physical seizure of the device so that it comes under the control and custody of the investigator/examiner. Consideration must also be given to the legal authority or written consent to seize, extract, and search this data. The physical condition of the device at the time of seizure should be noted, ideally through digital photographic documentation and written notes, such as: Is the device damaged? If, yes, then document the type of damage. Is the device on or off? What is the device date and time if the device is on? If the device is on, what apps are running or observable on the device desktop? If the device is on, is the device desktop accessible to check for passcode and security settings? Several other aspects of device seizure are described in the following as they will affect post-seizure analysis: radio isolation, turning the device off if it is on, remote wipe, and anti-forensics. Seizing – what and how to seize? When it comes to properly acquiring a mobile device, one must be aware of the many differences in how computers and mobile devices operate. Seizing, handling, storing, and extracting mobile devices must follow a different route compared to desktop and even laptop computers. Unlike PCs that can be either online or offline (which includes energy-saving states of sleep and hibernation), smartphones and tablets use a different, always-connected modus of operandi. Tremendous amounts of activities are carried out in the background, even while the device is apparently sleeping. Activities can be scheduled or triggered by a large number of events, including push events from online services and events that are initiated remotely by the user. Another thing to consider when acquiring a mobile device is security. Mobile devices are carried around a lot, and they are designed to be inherently more secure than desktop PCs. Non-removable storage and soldered RAM chips, optional or enforced data encryption, remote kill switches, secure lock screens, and locked bootloaders are just a few security measures to be mentioned. The use of Faraday bags Faraday bags are commonly used to temporarily store seized devices without powering them down. A Faraday bag blocks wireless connectivity to cellular networks, Wi-Fi, Bluetooth, satellite navigation, and any other radios used in mobile devices. Faraday bags are normally designed to shield the range of radio frequencies used by local cellular carriers and satellite navigation (typically the 700-2,600 MHz), as well as the 2.4-5 GHz range used by Wi-Fi networks and Bluetooth. Many Faraday bags are made of specially-coated metallic shielding material that blocks a wide range of radio frequencies. Keeping the power on When dealing with a seized device, it is essential to prevent the device from powering off. Never switching off a working device is one thing, preventing it from powering down is another. Since mobile devices consume power even while the display is off, the standard practice is connecting the device to a charger and placing it into a wireless-blocking Faraday bag. This will prevent the mobile device from shutting down after reaching the low-power state. Why exactly do we need this procedure? The thing is, you may be able to extract more information from a device that was used (unlocked at least once) after the last boot cycle compared to a device that boots up in your lab and you don't know the passcode. To illustrate the potential outcome, let's say you seized an iPhone that is locked with an unknown passcode. The iPhone happens to be jailbroken, so you can attempt using Elcomsoft iOS Forensic Toolkit to extract information. If the device is locked and you don't know the passcode, you will have access to a very limited set of data: Recent geolocation information: Since the main location database remains encrypted, it is only possible to extract limited location data. This limited location data is only accessible if the device was unlocked at least once after the boot has completed. As a result, if you keep the device powered on, you may pull recent geolocation history from this device. If, however, the device shuts down and is only powered on in the lab, the geolocation data will remain inaccessible until the device is unlocked. Incoming calls (numbers only) and text messages: Incoming text messages are temporarily retained unencrypted before the first unlock after cold boot. Once the device is unlocked for the first time after cold boot, the messages will be transferred into the main encrypted database. This means that acquiring a device that was never unlocked after a cold start will only allow access to text messages received by the device during the time it remained locked after the boot. If the iPhone being acquired was unlocked at least once after it was booted (for example, if the device was seized in a turned-on state), you may be able to access significantly more information. The SMS database is decrypted on first unlock, allowing you pulling all text messages and not just those that were received while the device remained locked. App and system logs (installs and updates, net access logs, and so on). SQLite temp files, including write-ahead logs (WAL): These WAL may include messages received by applications such as Skype, Viber, Facebook Messenger, and so on. Once the device is unlocked, the data is merged with corresponding apps' main databases. When extracting a device after a cold boot (never unlocked), you will only have access to notifications received after the boot. If, however, you are extracting a device that was unlocked at least once after booting up, you may be able to extract the complete database with all messages (depending on the data protection class selected by the developer of a particular application). Dealing with the kill switch Mobile operating systems such as Apple iOS, recent versions of Google Android, all versions of BlackBerry OS, and Microsoft Windows phone 8/8.1 (Windows 10 mobile) have an important security feature designed to prevent unauthorized persons from accessing information stored in the device. The so-called kill switch enables the owner to lock or erase the device if the device is reported lost or stolen. While used by legitimate customers to safeguard their data, this feature is also used by suspects who may attempt to remotely destroy evidence if their mobile device is seized. In the recent Morristown man accused of remotely wiping nude photos of underage girlfriend on confiscated phone report (http://wate.com/2015/04/07/morristown-man-accused-of-remotely-wiping-nude-photos-of-underage-girlfriend-on-confiscated-phone/), the accused used the remote kill switch to wipe data stored on his iPhone. Using the Faraday bag is essential to prevent suspects from accessing the kill switch. However, even if the device in question has already been wiped remotely, it does not necessarily mean that all the data is completely lost. Apple iOS, Windows phone 8/8.1, Windows 10 mobile, and the latest version of Android (Android 6.0 Marshmallow) support cloud backups (albeit Android cloud backups contains limited amounts of data). When it comes to BlackBerry 10, the backups are strictly offline, yet the decryption key is tied to the user's BlackBerry ID and stored on BlackBerry servers. The ability to automatically upload backup copies of data into the cloud is a double-edged sword. While offering more convenience to the user, cloud backups make remote acquisition techniques possible. Depending on the platform, all or some information from the device can be retrieved from the cloud by either making use of a forensic tool (for example, Elcomsoft Phone Breaker, Oxygen Forensic Detective) or by serving a government request to the corresponding company (Apple, Google, Microsoft, or BlackBerry). Mobile device anti-forensics There are numerous anti-forensic methods that target evidence acquisition methods used by the law enforcement. It is common for the police to seize a device, connect it to a charger, and place into a Faraday bag. The anti-forensic method used by some technologically-advanced suspects on Android phones involves rooting the device and installing a tool that monitors wireless connectivity of the device. If the tool detects that the device has been idle, connected to a charger, and without wireless connectivity for a predefined period, it performs a factory reset. Since there is no practical way of determining whether such protection is active on the device prior to acquisition, simply following established guidelines presents a risk of evidence being destroyed. If there are reasonable grounds to suspect such a system may be in place, the device can be powered down (while realizing the risk of full-disk encryption preventing subsequent acquisition). While rooting or jailbreaking devices generally makes the device susceptible to advanced acquisition methods, we've seen users who unlocked their bootloader to install a custom recovery, protected access to this custom recovery with a password, and relocked the bootloader. Locked bootloader and password-protected access to custom recovery is an extremely tough combination to break. In several reports, we've become aware of the following anti-forensic technique used by a group of cyber criminals. The devices were configured to automatically wipe user data if certain predefined conditions were met. In this case, the predefined conditions triggering the wipe matched the typical acquisition scenario of placing the device inside a Faraday bag and connecting it to a charger. Once the device reported being charged without wireless connectivity (but not in airplane mode) for a certain amount of time, a special tool triggers a full factory reset of the device. Notably, this is only possible on rooted/jailbroken devices. So far, this anti-forensic technique did not receive a wide recognition. It's used by a small minority of smartphone users, mostly those into cybercrime. The low probability of a smartphone being configured that way is small enough to consider implementing changes to published guidelines. Stage two – data acquisition This stage refers to various methods of extracting data from the device. The methods of data extraction that can be employed are influenced by the following: Type of mobile device: The make, model, hardware, software, and vendor configuration. Availability of a diverse set of hardware and software extraction/analysis tools at the examiner's disposal: There is no tool that does it all, an examiner needs to have access to a number of tools that can assist with data extraction. Physical state of device: Has the device been exposed to damage, such as physical, water, biological fluids such as blood? Often the type of damage can dictate the types of data extraction measures that will be employed on the device. There are several different types of data extractions that determine how much data is obtained from the device: Physical: Binary image of the device has the most potential to recover deleted data and obtains the largest amount of data from the device. This can be the most challenging type of extraction to obtain. File system: This is a representation of the files and folders from the user area of the device, and can contain deleted data, specific to databases. This method will contain less data than a physical data extraction. Logical: This acquires the least amount of data from the device. Examples of this are call history, messages, contacts, pictures, movies, audio files, and so on. This is referred to as low-hanging fruit. No deleted data or source files are obtained. Often the resulting output will be a series of reports produced by the extraction tool. This is often the easiest and quickest type of extraction. Photographic documentation: This method is typically used when all other data extraction avenues are exhausted. In this procedure, the examiner uses a digital camera to photographically document the content being displayed by the device. This is a time-consuming method when there is an extensive amount of information to photograph. Specific data-extraction concepts are explained here: bootloader, jailbreak, rooting, adb, debug, and sim cloning. Root, jailbreak, and unlocked bootloader Rooting or jailbreaking the mobile devices in general makes them susceptible to a wide range of exploits. In the context of mobile forensics, rooted devices are easy to acquire since many forensic acquisition tools rely on root/jailbreak to perform physical acquisition. Devices with unlocked bootloaders allow booting unsigned code, effectively permitting full access to the device even if it's locked with a passcode. However, if the device is encrypted and the passcode is part of the encryption key, bypassing passcode protection may not automatically enable access to encrypted data. Rooting or jailbreaking enables unrestricted access to the filesystem, bypassing the operating system's security measures and allowing the acquisition tool to read information from protected areas. This is one of the reasons for banning rooted devices (as well as devices with unlocked bootloaders) from corporate premises. Installing a jailbreak on iOS devices always makes the phone less secure, enabling third-party code to be injected and run on a system level. This fact is well-known to forensic experts who make use of tools such as Cellebrite UFED or Elcomsoft iOS Forensic Toolkit to perform physical acquisition of jailbroken Apple smartphones. Some Android devices allow unlocking the bootloader, which enables easy and straightforward rooting of the device. While not all Android devices with unlocked bootloaders are rooted, installing root access during acquisition of a bootloader-unlocked device has a much higher chance of success compared to devices that are locked down. Tools such as Cellebrite UFED, Forensic Toolkit (FTK), Oxygen Forensic Suite, and many others can make use of the phone's root status in order to inject acquisition applets and image the device. Unlocked bootloaders can be exploited as well if you use UFED. A bootloader-level exploit exists and is used in UFED to perform acquisition of many Android and Windows phone devices based on the Qualcomm reference platform even if their bootloader is locked. Android ADB debugging Android has a hidden Developer Options menu. Accessing this menu requires a conscious effort of tapping on the OS build number multiple times. Some users enable Developer Options out of curiosity. Once enabled, the Developer Options menu may or may not be possible to hide. Among other things, the Developer Options menu lists an option called USB debugging or ADB debugging. If enabled, this option allows controlling the device via ADB command line, which in turn allows experts using Android debugging tools (adb.exe) to connect to the device from a PC even if it's locked with a passcode. Activated USB debugging exposes a lot of possibilities and can make acquisition possible even if the device is locked with a passcode. Memory card Most smartphone devices and tablets (except iOS devices) have the capability of increasing their storage capacity by using a microSD card. An examiner would remove the memory card from the mobile device/tablet and use either hardware or software write-protection methods, and create a bit stream forensic image of the memory card, which can then be analyzed using forensic software tools, such as X-Ways, Autopsy Sleuth Kit, Forensic Explorer (GetData), EnCase, or FTK (AccessData). Stage three – data analysis This stage of mobile device forensics entails analysis of the acquired data from the device and its components (SIM card and memory card if present). Most mobile forensic acquisition tools that acquire the data from the device memory can also parse the extracted data and provide the examiner functionality within the tool to perform analysis. This entails review of any non-deleted and deleted data. When reviewing non-deleted data, it would be prudent to also perform a manual review of the device to ensure that the extracted and parsed data matches what is displayed by the device. As mobile device storage capacities have increased, it is suggested that a limited subset of data records from the relevant areas be reviewed. So, for example, if a mobile device has over 200 call records, reviewing several call records from missed calls, incoming calls, and outgoing calls can be checked on the device in relation to the similar records in the extracted data. By doing this manual review, it is then possible to discover any discrepancies in the extracted data. Manual device review can only be completed when the device is still in the custody of the examiner. There are situations where, after the data extraction has been completed, the device is released back to the investigator or owner. In situations such as this, the examiner should document that very limited or no manual verification can be performed due to these circumstances. Finally, the reader should be keenly aware that more than one analysis tool can be used to analyze the acquired data. Multiple analysis tools should be considered, especially when a specific type of data cannot be parsed by one tool, but can be analyzed by another. Summary In this article, we've covered the basics of mobile forensics. We discussed the amount of evidence available in today's mobile devices and covered the general steps of mobile forensics. We also discussed how to seize, handle, and store mobile devices, and looked at how criminals can use technology to prevent forensic access. We provided a general overview of the acquisition and analysis steps. For more information on mobile forensics, you can refer to the following books by Packt: Practical Mobile Forensics - Second Edition: https://www.packtpub.com/networking-and-servers/practical-mobile-forensics-second-edition Mastering Mobile Forensics: https://www.packtpub.com/networking-and-servers/mastering-mobile-forensics Learning iOS Forensics: https://www.packtpub.com/networking-and-servers/learning-ios-forensics  Resources for Article: Further resources on this subject: Mobile Forensics [article] Mobile Forensics and Its Challanges [article] Introduction to Mobile Forensics [article]
Read more
  • 0
  • 0
  • 22477
article-image-mapping-requirements-modular-web-shop-app
Packt
07 Sep 2016
11 min read
Save for later

Mapping Requirements for a Modular Web Shop App

Packt
07 Sep 2016
11 min read
In this article by Branko Ajzele, author of the book Modular Programming with PHP 7, we will be building a software application from the ground up requires diverse skills, as it involves more than just writing down a code. Writing down functional requirements and sketching out a wireframe are often among the first steps in the process, especially if we are working on a client project. These steps are usually done by roles other than the developer, as they require certain insight into client business case, user behavior, and alike. Being part of a larger development team means that, we as developers, usually get requirements, designs, and wireframes then start coding against them. Delivering projects by oneself, makes it tempting to skip these steps and get our hands started with code alone. More often than not, this is an unproductive approach. Laying down functional requirements and a few wireframes is a skill worth knowing and following, even if one is just a developer. (For more resources related to this topic, see here.) Later in this article, we will go over a high-level application requirement, alongside a rough wireframe. In this article, we will be covering the following topics: Defining application requirements Wireframing Defining a technology stack Defining application requirements We need to build a simple, but responsive web shop application. In order to do so, we need to lay out some basic requirements. The types of requirements we are interested in at the moment are those that touch upon interactions between a user and a system. The two most common techniques to specify requirements in regards to user usage are use case and user story. The user stories are a less formal, yet descriptive enough way to outline these requirements. Using user stories, we encapsulate the customer and store manager actions as mentioned here. A customer should be able to do the following: Browse through static info pages (about us, customer service) Reach out to the store owner via a contact form Browse the shop categories See product details (price, description) See the product image with a large view (zoom) See items on sale See best sellers Add the product to the shopping cart Create a customer account Update customer account info Retrieve a lost password Check out See the total order cost Choose among several payment methods Choose among several shipment methods Get an email notification after an order has been placed Check order status Cancel an order See order history A store manager should be able to do the following: Create a product (with the minimum following attributes: title, price, sku, url-key, description, qty, category, and image) Upload a picture to the product Update and delete a product Create a category (with the minimum following attributes: title, url-key, description, and image) Upload a picture to a category Update and delete a category Be notified if a new sales order has been created Be notified if a new sales order has been canceled See existing sales orders by their statuses Update the status of the order Disable a customer account Delete a customer account User stories are a convenient high-level way of writing down application requirements. Especially useful as an agile mode of development. Wireframing With user stories laid out, let's shift our focus to actual wireframing. For reasons we will get into later on, our wireframing efforts will be focused around the customer perspective. There are numerous wireframing tools out there, both free and commercial. Some commercial tools like https://ninjamock.com, which we will use for our examples, still provide a free plan. This can be very handy for personal projects, as it saves us a lot of time. The starting point of every web application is its home page. The following wireframe illustrates our web shop app's homepage: Here we can see a few sections determining the page structure. The header is comprised of a logo, category menu, and user menu. The requirements don't say anything about category structure, and we are building a simple web shop app, so we are going to stick to a flat category structure, without any sub-categories. The user menu will initially show Register and Login links, until the user is actually logged in, in which case the menu will change as shown in following wireframes. The content area is filled with best sellers and on sale items, each of which have an image, title, price, and Add to Cart button defined. The footer area contains links to mostly static content pages, and a Contact Us page. The following wireframe illustrates our web shop app's category page: The header and footer areas remain conceptually the same across the entire site. The content area has now changed to list products within any given category. Individual product areas are rendered in the same manner as it is on the home page. Category names and images are rendered above the product list. The width of a category image gives some hints as to what type of images we should be preparing and uploading onto our categories. The following wireframe illustrates our web shop app's product page: The content area here now changes to list individual product information. We can see a large image placeholder, title, sku, stock status, price, quantity field, Add to Cart button, and product description being rendered. The IN STOCK message is to be displayed when an item is available for purchase and OUT OF STOCK when an item is no longer available. This is to be related to the product quantity attribute. We also need to keep in mind the "See the product image with a big view (zoom)" requirement, where clicking on an image would zoom into it. The following wireframe illustrates our web shop app's register page: The content area here now changes to render a registration form. There are many ways that we can implement the registration system. More often than not, the minimal amount of information is asked on a registration screen, as we want to get the user in as quickly as possible. However, let's proceed as if we are trying to get more complete user information right here on the registration screen. We ask not just for an e-mail and password, but for entire address information as well. The following wireframe illustrates our web shop app's login page: The content area here now changes to render a customer login and forgotten password form. We provide the user with Email and Password fields in case of login, or just an Email field in case of a password reset action. The following wireframe illustrates our web shop app's customer account page: The content area here now changes to render the customer account area, visible only to logged in customers. Here we see a screen with two main pieces of information. The customer information being one, and order history being the other. The customer can change their e-mail, password, and other address information from this screen. Furthermore, the customer can view, cancel, and print all of their previous orders. The My Orders table lists orders top to bottom, from newest to oldest. Though not specified by the user stories, the order cancelation should work only on pending orders. This is something that we will touch upon in more detail later on. This is also the first screen that shows the state of the user menu when the user is logged in. We can see a dropdown showing the user's full name, My Account, and Sign Out links. Right next to it, we have the Cart (%s) link, which is to list exact quantities in a cart. The following wireframe illustrates our web shop app's checkout cart page: The content area here now changes to render the cart in its current state. If the customer has added any products to the cart, they are to be listed here. Each item should list the product title, individual price, quantity added, and subtotal. The customer should be able to change quantities and press the Update Cart button to update the state of the cart. If 0 is provided as the quantity, clicking the Update Cart button will remove such an item from the cart. Cart quantities should at all time reflect the state of the header menu Cart (%s) link. The right-hand side of a screen shows a quick summary of current order total value, alongside a big, clear Go to Checkout button. The following wireframe illustrates our web shop app's checkout cart shipping page: The content area here now changes to render the first step of a checkout process, the shipping information collection. This screen should not be accessible for non-logged in customers. The customer can provide us with their address details here, alongside a shipping method selection. The shipping method area lists several shipping methods. On the right hand side, the collapsible order summary section is shown, listing current items in the cart. Below it, we have the cart subtotal value and a big clear Next button. The Next button should trigger only when all of the required information is provided, in which case it should take us to payment information on the checkout cart payment page. The following wireframe illustrates our web shop app's checkout cart payment page: The content area here now changes to render the second step of a checkout process, the payment information collection. This screen should not be accessible for non-logged in customers. The customer is presented with a list of available payment methods. For the simplicity of the application, we will focus only on flat/fixed payments, nothing robust such as PayPal or Stripe. On the right-hand side of the screen, we can see a collapsible Order summary section, listing current items in the cart. Below it, we have the order totals section, individually listing Cart Subtotal, Standard Delivery, Order Total, and a big clear Place Order button. The Place Order button should trigger only when all of the required information is provided, in which case it should take us to the checkout success page. The following wireframe illustrates our web shop app's checkout success page: The content area here now changes to output the checkout successful message. Clearly this page is only visible to logged in customers that just finished the checkout process. The order number is clickable and links to the My Account area, focusing on the exact order. By reaching this screen, both the customer and store manager should receive a notification email, as per the Get email notification after order has been placed and Be notified if the new sales order has been created requirements. With this, we conclude our customer facing wireframes. In regards to store manager user story requirements, we will simply define a landing administration interface for now, as shown in the following screenshot: Using the framework later on, we will get a complete auto-generated CRUD interface for the multiple Add New and List & Manage links. The access to this interface and its links will be controlled by the framework's security component, since this user will not be a customer or any user in the database as such. Defining a technology stack Once the requirements and wireframes are set, we can focus our attention to the selection of a technology stack. Choosing the right one in this case, is more of a matter of preference, as application requirements for the most part can be easily met by any one of those frameworks. Our choice however, falls onto Symfony. Aside from PHP frameworks, we still need a CSS framework to deliver some structure, styling, and responsiveness within the browser on the client side. Since the focus of this book is on PHP technologies, let's just say we choose the Foundation CSS framework for that task. Summary Creating web applications can be a tedious and time consuming task. Web shops probably being one of the most robust and intensive type of application out there, as they encompass a great deal of features. There are many components involved in delivering the final product; from database, server side (PHP) code to client side (HTML, CSS, and JavaScript) code. In this article, we started off by defining some basic user stories which in turn defined high-level application requirements for our small web shop. Adding wireframes to the mix helped us to visualize the customer facing interface, while the store manager interface is to be provided out of the box by the framework. We further glossed over two of the most popular frameworks that support modular application design. We turned our attention to Symfony as server side technology and Foundation as a client side responsive framework. Resources for Article: Further resources on this subject: Running Simpletest and PHPUnit [article] Understanding PHP basics [article] PHP Magic Features [article]
Read more
  • 0
  • 0
  • 10526

article-image-hello-small-world
Packt
07 Sep 2016
20 min read
Save for later

Hello, Small World!

Packt
07 Sep 2016
20 min read
In this article by Stefan Björnander, the author of the book C++ Windows Programming, we will see how to create Windows applications using C++. This article introduces Small Windows by presenting two small applications: The first application writes "Hello, Small Windows!" in a window The second application handles circles of different colors in a document window (For more resources related to this topic, see here.) Hello, Small Windows! In The C Programming Language by Brian Kernighan and Dennis Richie, the hello-world example was introduced. It was a small program that wrote hello, world on the screen. In this section, we shall write a similar program for Small Windows. In regular C++, the execution of the application starts with the main function. In Small Windows, however, main is hidden in the framework and has been replaced by MainWindow, which task is to define the application name and create the main window object. The argumentList parameter corresponds to argc and argv in main. The commandShow parameter forwards the system's request regarding the window's appearance. MainWindow.cpp #include "..\SmallWindows\SmallWindows.h" #include "HelloWindow.h" void MainWindow(vector<String> /* argumentList */, WindowShow windowShow) { Application::ApplicationName() = TEXT("Hello"); Application::MainWindowPtr() = new HelloWindow(windowShow); } In C++, there are to two character types: char and wchar_t, where char holds a regular character of one byte and wchar_t holds a wide character of larger size, usually two bytes. There is also the string class that holds a string of char values and the wstring class that holds a string of wchar_t values. However, in Windows there is also the generic character type TCHAR that is char or wchar_t, depending on system settings. There is also the String class holds a string of TCHAR values. Moreover, TEXT is a macro that translates a character value to TCHAR and a text value to an array of TCHAR values. To sum it up, following is a table with the character types and string classes: Regular character Wide character Generic character char wchar_t TCHAR string wstring String In the applications of this book, we always use the TCHAR type, the String class, and the TEXT macro. The only exception to that rule is the clipboard handling. Our version of the hello-world program writes Hello, Small Windows! in the center of the client area. The client area of the window is the part of the window where it is possible to draw graphical objects. In the following window, the client area is the white area. The HelloWindow class extends the Small Windows Window class. It holds a constructor and the Draw method. The constructor calls the Window constructor with suitable information regarding the appearance of the window. Draw is called every time the client area of the window needs to be redrawn. HelloWindow.h class HelloWindow : public Window { public: HelloWindow(WindowShow windowShow); void OnDraw(Graphics& graphics, DrawMode drawMode); }; The constructor of HelloWindow calls the constructor of Window with the following parameter: The first parameter of the HelloWindow constructor is the coordinate system. LogicalWithScroll indicates that each logical unit is one hundredth of a millimeter, regardless of the physical resolution of the screen. The current scroll bar settings are taken into consideration. The second parameter of the window constructor is the preferred size of the window. It indicates that a default size shall be used. The third parameter is a pointer to the parent window. It is null since the window has no parent window. The fourth and fifth parameters set the window's style, in this case overlapped windows. The last parameter is windowShow given by the surrounding system to MainWindow, which decide the window's initial appearance (minimized, normal, or maximized). Finally, the constructor sets the header of the window by calling the Window method SetHeader. HelloWindow.cpp #include "..\SmallWindows\SmallWindows.h" #include "HelloWindow.h" HelloWindow::HelloWindow(WindowShow windowShow) :Window(LogicalWithScroll, ZeroSize, nullptr, OverlappedWindow, NoStyle, windowShow) { SetHeader(TEXT("Hello Window")); } The OnDraw method is called every time the client area of the window needs to be redrawn. It obtains the size of the client area and draws the text in its center with black text on white background. The SystemFont parameter will make the text appear in the default system font. The Small Windows Color class holds the constants Black and White. Point holds a 2-dimensional point. Size holds a width and a height. The Rect class holds a rectangle. More specifically, it holds the four corners of a rectangle. void HelloWindow::OnDraw(Graphics& graphics, DrawMode /* drawMode */) { Size clientSize = GetClientSize(); Rect clientRect(Point(0, 0), clientSize); Font textFont("New Times Roman", 12, true); graphics.DrawText(clientRect, TEXT("Hello, Small Windows!"), textFont , Black, White); } The Circle application In this section, we look into a simple circle application. As the name implies, it provides the user the possibility to handle circles in a graphical application. The user can add a new circle by clicking the left mouse button. They can also move an existing circle by dragging it. Moreover, the user can change the color of a circle as well as save and open the document.   The main window As we will see thought out this book, MainWindow does always do the same thing: it sets the application name and creates the main window of the application. The name is used by the Save and Open standard dialogs, the About menu item, and the registry. The difference between the main window and other windows of the application is that when the user closes the main window, the application exits. Moreover, when the user selects the Exit menu item the main window is closed, and its destructor is called. MainWindow.cpp #include "..\SmallWindows\SmallWindows.h" #include "Circle.h" #include "CircleDocument.h" void MainWindow(vector<String> /* argumentList */, WindowShow windowShow) { Application::ApplicationName() = TEXT("Circle"); Application::MainWindowPtr() = new CircleDocument(windowShow); } The CircleDocument class The CircleDocumentclass extends the Small Windows class StandardDocument, which in turn extends Document and Window. In fact, StandardDocument constitutes of a framework; that is, a base class with a set of virtual methods with functionality we can override and further specify. The OnMouseDown and OnMouseUp methods are overridden from Window and are called when the user presses or releases one of the mouse buttons. OnMouseMove is called when the user moves the mouse. The OnDraw method is also overridden from Window and is called every time the window needs to be redrawn. The ClearDocument, ReadDocumentFromStream, and WriteDocumentToStream methods are overridden from Standard­Document and are called when the user creates a new file, opens a file, or saves a file. CircleDocument.h class CircleDocument : public StandardDocument { public: CircleDocument(WindowShow windowShow); ~CircleDocument(); void OnMouseDown(MouseButton mouseButtons, Point mousePoint, bool shiftPressed, bool controlPressed); void OnMouseUp(MouseButton mouseButtons, Point mousePoint, bool shiftPressed, bool controlPressed); void OnMouseMove(MouseButton mouseButtons, Point mousePoint, bool shiftPressed, bool controlPressed); void OnDraw(Graphics& graphics, DrawMode drawMode); bool ReadDocumentFromStream(String name, istream& inStream); bool WriteDocumentToStream(String name, ostream& outStream) const; void ClearDocument(); The DEFINE_BOOL_LISTENER and DEFINE_VOID_LISTENER macros define listeners: methods without parameters that are called when the user selects a menu item. The only difference between the macros is the return type of the defined methods: bool or void. In the applications of this book, we use the common standard that the listeners called in response to user actions are prefixed with On, for instance OnRed. The methods that decide whether the menu item shall be enabled are suffixed with Enable, and the methods that decide whether the menu item shall be marked with a check mark or a radio button are suffixed with Check or Radio. In this application, we define menu items for the red, green, and blue colors. We also define a menu item for the Color standard dialog.     DEFINE_VOID_LISTENER(CircleDocument,OnRed);     DEFINE_VOID_LISTENER(CircleDocument,OnGreen);     DEFINE_VOID_LISTENER(CircleDocument,OnBlue);     DEFINE_VOID_LISTENER(CircleDocument,OnColorDialog); When the user has chosen one of the color red, green, or blue, its corresponding menu item shall be checked with a radio button. RedRadio, GreenRadio, and BlueRadio are called before the menu items become visible and return a Boolean value indicating whether the menu item shall be marked with a radio button.     DEFINE_BOOL_LISTENER(CircleDocument, RedRadio);     DEFINE_BOOL_LISTENER(CircleDocument, GreenRadio);     DEFINE_BOOL_LISTENER(CircleDocument, BlueRadio); The circle radius is always 500 units, which correspond to 5 millimeters.     static const int CircleRadius = 500; The circleList field holds the circles, where the topmost circle is located at the beginning of the list. The nextColor field holds the color of the next circle to be added by the user. It is initialized to minus one to indicate that no circle is being moved at the beginning. The moveIndex and movePoint fields are used by OnMouseDown and OnMouseMove to keep track of the circle being moved by the user. private: vector<Circle> circleList; Color nextColor; int moveIndex = -1; Point movePoint; }; In the StandardDocument constructor call, the first two parameters are LogicalWithScroll and USLetterPortrait. They indicate that the logical size is hundredths of millimeters and that the client area holds the logical size of a US letter: 215.9 * 279.4 millimeters (8.5 * 11 inches). If the window is resized so that the client area becomes smaller than a US letter, scroll bars are added to the window. The third parameter sets the file information used by the standard Save and Open dialogs, the text description is set to Circle Files and the file suffix is set to cle. The null pointer parameter indicates that the window does not have a parent window. The OverlappedWindow constant parameter indicates that the window shall overlap other windows and the windowShow parameter is the window's initial appearance passed on from the surrounding system by MainWindow. CircleDocument.cpp #include "..\SmallWindows\SmallWindows.h" #include "Circle.h" #include "CircleDocument.h" CircleDocument::CircleDocument(WindowShow windowShow) :StandardDocument(LogicalWithScroll, USLetterPortrait, TEXT("Circle Files, cle"), nullptr, OverlappedWindow, windowShow) { The StandardDocument framework adds the standard File, Edit, and Help menus to the window menu bar. The File menu holds the New, Open, Save, Save As, Page Setup, Print Preview, and Exit items. The Page Setup and Print Preview items are optional. The seventh parameter of the StandardDocument constructor (default false) indicates their presence. The Edit menu holds the Cut, Copy, Paste, and Delete items. They are disabled by default; we will not use them in this application. The Help menu holds the About item, the application name set in MainWindow is used to display a message box with a standard message: Circle, version 1.0. We add the standard File and Edit menus to the menu bar. Then we add the Color menu, which is the application-specific menu of this application. Finally, we add the standard Help menu and set the menu bar of the document. The Color menu holds the menu items used to set the circle colors. The OnRed, OnGreen, and OnBlue methods are called when the user selects the menu item, and the RedRadio, GreenRadio, BlueRadio are called before the user selects the color menu in order to decide if the items shall be marked with a radio button. OnColorDialog opens a standard color dialog. In the text &RedtCtrl+R, the ampersand (&) indicates that the menu item has a mnemonic; that is, the letter R will be underlined and it is possible to select the menu item by pressing R after the menu has been opened. The tabulator character (t) indicates that the second part of the text defines an accelerator; that is, the text Ctrl+R will occur right-justified in the menu item and the item can be selected by pressing Ctrl+R. Menu menuBar(this); menuBar.AddMenu(StandardFileMenu(false)); The AddItem method in the Menu class also takes two more parameters for enabling the menu item and setting a check box. However, we do not use them in this application. Therefore, we send null pointers. Menu colorMenu(this, TEXT("&Color")); colorMenu.AddItem(TEXT("&RedtCtrl+R"), OnRed, nullptr, nullptr, RedRadio); colorMenu.AddItem(TEXT("&GreentCtrl+G"), OnGreen, nullptr, nullptr, GreenRadio); colorMenu.AddItem(TEXT("&BluetCtrl+B"), OnBlue, nullptr, nullptr, BlueRadio); colorMenu.AddSeparator(); colorMenu.AddItem(TEXT("&Dialog ..."), OnColorDialog); menuBar.AddMenu(colorMenu); menuBar.AddMenu(StandardHelpMenu()); SetMenuBar(menuBar); Finally, we read the current color (the color of the next circle to be added) from the registry; red is the default color in case there is no color stored in the registry. nextColor.ReadColorFromRegistry(TEXT("NextColor"), Red); } The destructor saves the current color in the registry. In this application, we do not need to perform the destructor's normal tasks, such as deallocate memory or closing files. CircleDocument::~CircleDocument() { nextColor.WriteColorToRegistry(TEXT("NextColor")); } The ClearDocument method is called when the user selects the New menu item. In this case, we just clear the circle list. Every other action, such as redrawing the window or changing its title, is taken care of by StandardDocument. void CircleDocument::ClearDocument() { circleList.clear(); } The WriteDocumentToStream method is called by StandardDocument when the user saves a file (by selecting Save or Save As). It writes the number of circles (the size of the circle list) to the output stream and calls WriteCircle for each circle in order to write their states to the stream. bool CircleDocument::WriteDocumentToStream(String name, ostream& outStream) const { int size = circleList.size(); outStream.write((char*) &size, sizeof size); for (Circle circle : circleList) { circle.WriteCircle(outStream); } return ((bool) outStream); } The ReadDocumentFromStream method is called by StandardDocument when the user opens a file by selecting the Open menu item. It reads the number of circles (the size of the circle list) and for each circle it creates a new object of the Circle class, calls ReadCircle in order to read the state of the circle, and adds the circle object to circleList. bool CircleDocument::ReadDocumentFromStream(String name, istream& inStream) { int size; inStream.read((char*) &size, sizeof size); for (int count = 0; count < size; ++count) { Circle circle; circle.ReadCircle(inStream); circleList.push_back(circle); } return ((bool) inStream); } The OnMouseDown method is called when the user presses one of the mouse buttons. First we need to check that they have pressed the left mouse button. If they have, we loop through the circle list and call IsClick for each circle in order to decide whether they have clicked at a circle. Note that the top-most circle is located at the beginning of the list; therefore, we loop from the beginning of the list. If we find a clicked circle, we break the loop. If the user has clicked at a circle, we store its index moveIndex and the current mouse position in movePoint. Both values are needed by OnMouseMove method that will be called when the user moves the mouse. void CircleDocument::OnMouseDown (MouseButton mouseButtons, Point mousePoint, bool shiftPressed /* = false */, bool controlPressed /* = false */) { if (mouseButtons == LeftButton) { moveIndex = -1; int size = circleList.size(); for (int index = 0; index < size; ++index) { if (circleList[index].IsClick(mousePoint)) { moveIndex = index; movePoint = mousePoint; break; } } However, if the user has not clicked at a circle, we add a new circle. A circle is defined by its center position (mousePoint), radius (CircleRadius), and color (nextColor). An invalidated area is a part of the client area that needs to be redrawn. Remember that in Windows we normally do not draw figures directly. Instead, we call Invalidate to tell the system that an area needs to be redrawn and forces the actually redrawing by calling UpdateWindow, which eventually results in a call to OnDraw. The invalidated area is always a rectangle. Invalidate has a second parameter (default true) indicating that the invalidated area shall be cleared. Technically, it is painted in the window's client color, which in this case is white. In this way, the previous location of the circle becomes cleared and the circle is drawn at its new location. The SetDirty method tells the framework that the document has been altered (the document has become dirty), which causes the Save menu item to be enabled and the user to be warned if they try to close the window without saving it. if (moveIndex == -1) { Circle newCircle(mousePoint, CircleRadius, nextColor); circleList.push_back(newCircle); Invalidate(newCircle.Area()); UpdateWindow(); SetDirty(true); } } } The OnMouseMove method is called every time the user moves the mouse with at least one mouse button pressed. We first need to check whether the user is pressing the left mouse button and is clicking at a circle (whether moveIndex does not equal minus one). If they have, we calculate the distance from the previous mouse event (OnMouseDown or OnMouseMove) by comparing the previous mouse position movePoint by the current mouse position mousePoint. We update the circle position, invalidate both the old and new area, forcing a redrawing of the invalidated areas with UpdateWindow, and set the dirty flag. void CircleDocument::OnMouseMove (MouseButton mouseButtons, Point mousePoint, bool shiftPressed /* = false */, bool controlPressed /* = false */) { if ((mouseButtons == LeftButton)&&(moveIndex != -1)) { Size distanceSize = mousePoint - movePoint; movePoint = mousePoint; Circle& movedCircle = circleList[moveIndex]; Invalidate(movedCircle.Area()); movedCircle.Center() += distanceSize; Invalidate(movedCircle.Area()); UpdateWindow(); SetDirty(true); } } Strictly speaking, OnMouseUp could be excluded since moveIndex is set to minus one in OnMouseDown, which is always called before OnMouseMove. However, it has been included for the sake of completeness. void CircleDocument::OnMouseUp (MouseButton mouseButtons, Point mousePoint, bool shiftPressed /* = false */, bool controlPressed /* = false */) { moveIndex = -1; } The OnDraw method is called every time the window needs to be (partly or completely) redrawn. The call can have been initialized by the system as a response to an event (for instance, the window has been resized) or by an earlier call to UpdateWindow. The Graphics reference parameter has been created by the framework and can be considered a toolbox for drawing lines, painting areas and writing text. However, in this application we do not write text. We iterate throw the circle list and, for each circle, call the Draw method. Note that we do not care about which circles are to be physically redrawn. We simple redraw all circles. However, only the circles located in an area that has been invalidated by a previous call to Invalidate will be physically redrawn. The Draw method has a second parameter indicating the draw mode, which can be Paint or Print. Paint indicates that OnDraw is called by OnPaint in Window and that the painting is performed in the windows' client area. The Print method indicates that OnDraw is called by OnPrint and that the painting is sent to a printer. However, in this application we do not use that parameter. void CircleDocument::OnDraw(Graphics& graphics, DrawMode /* drawMode */) { for (Circle circle : circleList) { circle.Draw(graphics); } } The RedRadio, GreenRadio, and BlueRadio methods are called before the menu items are shown, and the items will be marked with a radio button in case they return true. The Red, Green, and Blue constants are defined in the Color class. bool CircleDocument::RedRadio() const { return (nextColor == Red); } bool CircleDocument::GreenRadio() const { return (nextColor == Green); } bool CircleDocument::BlueRadio() const { return (nextColor == Blue); } The OnRed, OnGreen, and OnBlue methods are called when the user selects the corresponding menu item. They all set the nextColor field to an appropriate value. void CircleDocument::OnRed() { nextColor = Red; } void CircleDocument::OnGreen() { nextColor = Green; } void CircleDocument::OnBlue() { nextColor = Blue; } The OnColorDialog method is called when the user selects the Color dialog menu item and displays the standard Color dialog. If the user choses a new color, nextcolor will be given the chosen color value. void CircleDocument::OnColorDialog() { ColorDialog(this, nextColor); } The Circle class The Circle class is a class holding the information about a single circle. The default constructor is used when reading a circle from a file. The second constructor is used when creating a new circle. The IsClick method returns true if the given point is located inside the circle (to check whether the user has clicked in the circle), Area returns the circle's surrounding rectangle (for invalidating), and Draw is called to redraw the circle. Circle.h class Circle { public: Circle(); Circle(Point center, int radius, Color color); bool WriteCircle(ostream& outStream) const; bool ReadCircle(istream& inStream); bool IsClick(Point point) const; Rect Area() const; void Draw(Graphics& graphics) const; Point Center() const {return center;} Point& Center() {return center;} Color GetColor() {return color;} As mentioned in the previous section, a circle is defined by its center position (center), radius (radius), and color (color). private: Point center; int radius; Color color; }; The default constructor does not need to initialize the fields, since it is called when the user opens a file and the values are read from the file. The second constructor, however, initializes the center point, radius, and color of the circle. Circle.cpp #include "..\SmallWindows\SmallWindows.h" #include "Circle.h" Circle::Circle() { // Empty. } Circle::Circle(Point center, int radius, Color color) :color(color), center(center), radius(radius) { // Empty. } The WriteCircle method writes the color, center point, and radius to the stream. Since the radius is a regular integer, we simply use the C standard function write, while Color and Point have their own methods to write their values to a stream. In ReadCircle we read the color, center point, and radius from the stream in a similar manner. bool Circle::WriteCircle(ostream& outStream) const { color.WriteColorToStream(outStream); center.WritePointToStream(outStream); outStream.write((char*) &radius, sizeof radius); return ((bool) outStream); } bool Circle::ReadCircle(istream& inStream) { color.ReadColorFromStream(inStream); center.ReadPointFromStream(inStream); inStream.read((char*) &radius, sizeof radius); return ((bool) inStream); } The IsClick method uses the Pythagoras theorem to calculate the distance between the given point and the circle's center point, and return true if the point is located inside the circle (if the distance is less than or equal to the circle radius). Circle::IsClick(Point point) const { int width = point.X() - center.X(), height = point.Y() - center.Y(); int distance = (int) sqrt((width * width) + (height * height)); return (distance <= radius); } The top-left corner of the resulting rectangle is the center point minus the radius, and the bottom-right corner is the center point plus the radius. Rect Circle::Area() const { Point topLeft = center - radius, bottomRight = center + radius; return Rect(topLeft, bottomRight); } We use the FillEllipse method (there is no FillCircle method) of the Small Windows Graphics class to draw the circle. The circle's border is always black, while its interior color is given by the color field. void Circle::Draw(Graphics& graphics) const { Point topLeft = center - radius, bottomRight = center + radius; Rect circleRect(topLeft, bottomRight); graphics.FillEllipse(circleRect, Black, color); } Summary In this article, we have looked into two applications in Small Windows: a simple hello-world application and a slightly more advance circle application, which has introduced the framework. We have looked into menus, circle drawing, and mouse handling. Resources for Article: Further resources on this subject: C++, SFML, Visual Studio, and Starting the first game [article] Game Development Using C++ [article] Boost.Asio C++ Network Programming [article]
Read more
  • 0
  • 0
  • 12615

article-image-computer-vision-keras-part-1
Sasank Chilamkurthy
07 Sep 2016
6 min read
Save for later

Computer Vision with Keras, Part 1

Sasank Chilamkurthy
07 Sep 2016
6 min read
In this two-part post, we will get our hands dirty with deep learning by solving a real-world problem. The problem we are gonna tackle is The German Traffic Sign Recognition Benchmark (GTSRB). The problem is to to recognize the traffic sign from the images. Solving this problem is essential for self-driving cars to operate on roads. Here are representative images for each of the traffic sign classes in the GTSRB dataset: The dataset features 43 different signs under various sizes, lighting conditions, and occlusions and is very similar to real-life data. The training set includes about 39,000 images while the test set has around 12,000 images. Images are not guaranteed to be of fixed dimensions and the sign is not necessarily centered in each image. Each image contains about a 10% border around the actual traffic sign. Our approach to solving the problem will of course be very successful convolutional neural networks (CNNs). CNNs are multi-layered, feed-forward neural networks that are able to learn task-specific invariant features in a hierarchical manner. You can read more about them in very readable Neural Networks and Deep Learning book by Michael Nielsen. Chapter 6 is the essential reading. Just read the first section in this chapter if you are in a hurry. A note about the code: This tutorial is a recommended way to run the code in this post, and for experimenting with it is Jupyter notebook. A notebook with slightly improved code is available here. We will implement our CNNs in Keras. Keras is a deep learning library written in Python and allows us to do quick experimentation. Let's start by installing Keras and other libraries(Use the anaconda python distribution): $ sudo pip install kerasscikit-image pandas Then download 'Images and annotations' for training and the test set from GTSRB website and extract them into a folder. Also download the 'Extended annotations including class ids' file for the test set. Organize these files so that the directory structure looks like this: GTSRB ├── GT-final_test.csv ├── Final_Test │ └── Images └── Final_Training └── Images ├── 00000 ├── 00001 ├── ... ├── 00041 └── 00042 Preprocessing As you can see from the representative images above, images vary a lot in illumination. They also vary in size. So, let's write a function to do histogram equalization in HSV color space and resize the images to a standard size: Import numpy as np From skimage import color, exposure, transform NUM_CLASSES = 43 IMG_SIZE = 48 Def preprocess_img(img): # Histogram normalization in v channel hsv = color.rgb2hsv(img) hsv[:,:,2] = exposure.equalize_hist(hsv[:,:,2]) img = color.hsv2rgb(hsv) # central square crop min_side = min(img.shape[:-1]) centre = img.shape[0]//2, img.shape[1]//2 img = img[centre[0]-min_side//2:centre[0]+min_side//2, centre[1]-min_side//2:centre[1]+min_side//2, :] # rescale to standard size img = transform.resize(img, (IMG_SIZE, IMG_SIZE)) # roll color axis to axis 0 img = np.rollaxis(img,-1) return img Input image to preprocess_img (scaled 4x). Processed image (scaled 4x): Let's preprocess all the training images and store in numpy arrays. We'll also get labels of images from paths. We'll convert targets to one-hot form as is required by Keras: From skimage import io importos import glob def get_class(img_path): return int(img_path.split('/')[-2]) root_dir = 'GTSRB/Final_Training/Images/' imgs = [] labels = [] all_img_paths = glob.glob(os.path.join(root_dir, '*/*.ppm')) np.random.shuffle(all_img_paths) for img_path in all_img_paths: img = preprocess_img(io.imread(img_path)) label = get_class(img_path) imgs.append(img) labels.append(label) X = np.array(imgs, dtype='float32') # Make one hot targets Y = np.eye(NUM_CLASSES, dtype='uint8')[labels] Models Let's now define our models. We'll use a feed-forward network with 6 convolutional layers followed by a fully connected hidden layer. We'll also use dropout layers in between. Dropout regularizes the networks, i.e. it prevents the network from overfitting. All our layers have relu activations except the output layer. Output layer uses softmax activation as it has to output the probability for each of the classes. Sequential is a Keras container for linear stack of layers. Each of the layers in the model needs to know the input shape it should expect, but it is enough to specify input_shape for the first layer of the Sequential model. Rest of the layers do automatic shape inference. To attach a fully connected layer (that is, dense layer) to a convolutional layer, we will have to reshape/flatten the output of the conv layer. This is achieved with the Flatten layer. Go through the documentation of Keras (the relevant documentation is here and here) to understand what the parameters for each of the layers mean. from keras.models import Sequential from keras.layers import Dense, Dropout, Activation, Flatten from keras.layers import Convolution2D, MaxPooling2D from keras.optimizers import SGD defcnn_model(): model = Sequential() model.add(Convolution2D(32, 3, 3, border_mode='same', input_shape=(3, IMG_SIZE, IMG_SIZE), activation='relu')) model.add(Convolution2D(32, 3, 3, activation='relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Dropout(0.2)) model.add(Convolution2D(64, 3, 3, border_mode='same', activation='relu')) model.add(Convolution2D(64, 3, 3, activation='relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Dropout(0.2)) model.add(Convolution2D(128, 3, 3, border_mode='same', activation='relu')) model.add(Convolution2D(128, 3, 3, activation='relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Dropout(0.2)) model.add(Flatten()) model.add(Dense(512, activation='relu')) model.add(Dropout(0.5)) model.add(Dense(NUM_CLASSES, activation='softmax')) return model Before training the model, we need to configure the learning algorithm  model and compile it. We need to specify: loss :The loss function we want to optimize. We cannot use error percentage because it is not continuous and thus non-differentiable. We therefore use a proxy for it: categorical_crossentropy. optimizer : We use standard stochastic gradient descent with Nesterov momentum. metric : Since we are dealing with a classification problem, our metric is accuracy. from keras.optimizers import SGD model = cnn_model() # let's train the model using SGD + momentum lr = 0.01 sgd = SGD(lr=lr, decay=1e-6, momentum=0.9, nesterov=True) model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy']) The next step is for us to actually train our model, which is what we will do in part 2. We will also evaluate our model and then do some augmentation to improve it. Conclusion In this post, we have used the Keras deep learning framework to implement CNNs in Python. In part 2, you will see how we achieve performance close to human-level performance. You will also see how to improve the accuracy of the model using augmentation of the training data. References: Stallkamp, Johannes, et al. "Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition." Neural networks 32 (2012): 323-332. About the author Sasank Chilamkurthy works at Qure.ai. His work involves deep learning on medical images obtained from radiology and pathology. He completed his UG in Mumbai at the Indian Institute of Technology, Bombay. He can be found on Github at here.
Read more
  • 0
  • 0
  • 18930
article-image-pfsense-essentials
Packt
06 Sep 2016
60 min read
Save for later

pfSense Essentials

Packt
06 Sep 2016
60 min read
In this article by David Zientara, the author of the book Mastering pfSense, While high-speed Internet connectivity is becoming more and more common, many in the online world—especially those with residential connections or small office/home office (SOHO) setups—lack the hardware to fully take advantage of those speeds. Fiber optic technology brings with it the promise of a gigabit speed or greater, and the technology surrounding traditional copper networks is also yielding improvements. Yet many people are using consumer-grade routers that offer, at best, mediocre performance. (For more resources related to this topic, see here.) pfSense, an open source router/firewall solution is a far better alternative that is available to you. You have likely already downloaded, installed, and configured pfSense, possibly in a residential or SOHO environment. As an intermediate-level pfSense user, you do not need to be sold on the benefits of pfSense. Nevertheless, you may be looking to deploy pfSense in a different environment (for example, a corporate network), or you may just be looking to enhance your knowledge of pfSense. This chapter is designed to review the process of getting your pfSense system up and running. It will guide you through the process of choosing the right hardware for your deployment, but it will not provide a detailed treatment of installation and initial configuration. The emphasis will be on troubleshooting, as well as some of the newer configuration options. Finally, the article will provide a brief treatment of how to upgrade, back up, and restore pfSense. This article will cover the following topics: A brief overview of the pfSense project pfSense deployment scenarios Minimum specifications and hardware sizing guidelines An introduction to Virtual local area networks (VLANs) and Domain Name System (DNS) The best practices for installation and configuration Basic configuration from both the console and the pfSense web GUI Upgrading, backing up, and restoring pfSense pfSense project overview The origins of pfSense can be traced to the OpenBSD packet filter known as PF, which was incorporated into FreeBSD in 2001. As PF is limited to a command-line interface, several projects have been launched in order to provide a graphical interface for PF. m0n0wall, which was released in 2003, was the earliest attempt at such a project. pfSense began as a fork of the m0n0wall project. Version 1.0 of pfSense was released on October 4, 2006. Version 2.0 was released on September 17, 2011. Version 2.1 was released on September 15, 2013, and Version 2.2 was released on January 23, 2015. As of writing this, Version 2.2.6 (released on December 21, 2015) is the latest version. Version 2.3 is expected to be released soon, and will be a watershed release in many respects. The web GUI has had a major facelift, and support for some legacy technologies is being phased out. Support for Point-to-Point Tunnelling Protocol (PPTP) will be discontinued, as will support for Wireless Encryption Protocol (WEP). The current version of pfSense incorporates such functions as traffic shaping, the ability to act as a Virtual Private Network (VPN) client or server, IPv6 support, and through packages, intrusion detection and prevention, the ability to act as a proxy server, spam and virus blocking, and much more. Possible deployment scenarios Once you have decided to add a pfSense system to your network, you need to consider how it is going to be deployed on your network. pfSense is suitable for a variety of networks, from small to large ones, and can be employed in a variety of deployment scenarios. In this article, we will cover the following possible uses for pfSense: Perimeter firewall Router Switch Wireless router/wireless access point The most common way to add pfSense to your network is to use it as a perimeter firewall. In this scenario, your Internet connection is connected to one port on the pfSense system, and your local network is connected to another port on the system. The port connected to the Internet is known as the WAN (wide area network) interface, and the port connected to the local network is known as the LAN (local area network) interface. Diagram showing a deployment in which pfSense is the perimeter firewall. If pfSense is your perimeter firewall, you may choose to set it up as a dedicated firewall, or you might want to have it perform the double duty of a firewall and a router. You may also choose to have more than two interfaces in your pfSense system (known as optional interfaces). In order to act as a perimeter firewall, however, a pfSense system requires at least two interfaces: a WAN interface (to connect to outside networks), and a LAN interface (to connect to the local network). In more complex network setups, your pfSense system may have to exchange routing information with other routers on the network. There are two types of protocols for exchanging such information: distance vector protocols obtain their routing information by exchanging information with neighboring routers; Routers that use link-state protocols to build a map of the network in order to calculate the shortest path to another router, with each router calculating distances independently. pfSense is capable of running both types of protocols. Packages are available for distance vector protocols such as RIP and RIPv2, and link-state protocols such as Border Gateway Protocol (BGP). Another common deployment scenario is to set up pfSense as a router. In a home or SOHO environment, firewall and router functions are often performed by the same device. In mid-sized to large networks, however, the router is a device separate from that of the perimeter firewall. In larger networks, which have several network segments, pfSense can be used to connect these segments. In corporate-type environments, these are often used in conjunction, which allows a single network interface card (NIC) to operate in multiple broadcast domains via 802.1q tagging. VLANs are often used with the ever-popular router on a stick configuration, in which the router has a single physical connection to a switch, with the single Ethernet interface divided into multiple VLANs, and the router forwarding packets between the VLANs. One of the advantages of this setup is that it only requires a single port, and, as a result, it allows us to use pfSense with systems on when adding another NIC would be cumbersome or even impossible: for example, a laptop or certain thin clients. In most cases, where pfSense is deployed as a router on mid-sized and large networks, it would be used to connect different LAN segments; however, it could also be used as a WAN router. In this case, pfSense's function would be to provide a private WAN connection to the end user. Another possible deployment scenario is to use pfSense as a switch. If you have multiple interfaces on your pfSense system and bridge them together, pfSense can function as a switch. This is a far less common scenario, however, for several reasons: Using pfSense as a switch is generally not cost-effective. You can purchase a 5-port Ethernet switch for less than what it would cost to purchase the hardware for a pfSense system. Buying a commercially available switch will also save you money in the long run, as they likely would consume far less power than whatever computer you would be using to run pfSense. Commercially available switches will likely outperform pfSense, as pfSense will process all packets that pass between ports, while a typical Ethernet switch will handle it locally with dedicated hardware made specifically for passing data between ports quickly. While you can disable filtering entirely in pfSense if you know what you're doing, you will still be limited by the speed of the bus on which your network cards reside, whether it is PCI, PCI-X, or PCI Express (PCI-e). There is also the administrative overhead of using pfSense as a switch. Simple switches are designed to be plug-and-play, and setting up these switches is as easy as plugging in your Ethernet cables and the power cord. Managed switches typically enable you to configure settings at the console and/or through a web interface, but in many cases, configuration is only necessary if you want to modify the operation of the switch. If you use pfSense as a switch, however, some configuration will be required. If none of this intimidates you, then feel free to use pfSense as a switch. While you're not likely to achieve the performance level or cost savings of using a commercially available switch, you will likely learn a great deal about pfSense and networking in the process. Moreover, advances in hardware could make using pfSense as a switch viable at some point in the future. Advances in low-power consumption computers are one factor that could make this possible. Yet another possibility is using pfSense as a wireless router/access point. A sizable proportion of modern networks incorporate some type of wireless connectivity. Connecting to networks wireless is not only easier, but in some cases, running Ethernet cable is not a realistic option. With pfSense, you can add wireless networking capabilities to your system by adding a wireless network card, provided that the network card is supported by FreeBSD. Generally, however, using pfSense as a wireless router or access point is not the best option. Support for wireless network cards in FreeBSD leaves something to be desired. Support for the IEEE's 802.11b and g standards is OK, but support for 802.11n and 802.11ac is not very good. A more likely solution is to buy a wireless router (even if it is one of the aforementioned consumer-grade units), set it up to act solely as an access point, connect it to the LAN port of your pfSense system, and let pfSense act as a Dynamic Host Configuration Protocol (DHCP) server. A typical router will work fine as a dedicated wireless access point, and they are more likely to support the latest wireless networking standards than pfSense. Another possibility is to buy a dedicated wireless access point. These are generally inexpensive and some have such features as multiple SSIDs, which allow you to set up multiple wireless networks (for example, you could have a separate guest network which is completely isolated from other local networks). Using pfSense as a router, in combination with a commercial wireless access point, is likely the least troublesome option. Hardware requirements and sizing guidelines Once you have decided where to deploy pfSense on your network, you should have a clearer idea of what your hardware requirements are. As a minimum, you will need a CPU, motherboard, memory (RAM), some form of disk storage, and at least two network interfaces (unless you are opting for a router on a stick setup, in which case you only need one network interface). You may also need one or more optional interfaces. Minimum specifications The starting point for our discussion on hardware requirements is the pfSense minimum specifications. As of January 2016, the minimum hardware requirements are as follows (these specifications are from the official pfSense site, pfsense.org): CPU – 500 MHz (1 GHz recommended) RAM – 256 MB (1 GB recommended) There are two architectures currently supported by pfSense: i386 (32-bit) and amd64 (64-bit). There are three separate images provided for these architectures: CD, CD on a USB memstick, and embedded. There is also an image for the Netgate RCC-VE 2440 system. A pfSense installation requires at least 1 GB of disk space. If you are installing to an embedded device, you can access the console either by a serial or VGA port. A step-by-step installation guide for the pfSense Live CD can be found on the official pfSense website at: https://doc.pfsense.org/index.php/PfSense_IO_installation_step_by_step. Version 2.3 eliminated the Live CD, which allowed you to try out pfSense without installing it onto other media. If you really want to use the Live CD, however, you could use a pre-2.3 image (version 2.2.6 or earlier). You can always upgrade to the latest version of pfSense after installation. Installation onto either a hard disk drive (HDD) or an SSD is the most common option for a full install of pfSense, whereas embedded installs typically use CF, SD, or USB media. A full install of the current version of pfSense will fit onto a 1 GB drive but will leave little room for installation of packages or for log files. Any activity that requires caching, such as running a proxy server, will also require additional disk space. The last installation option in the table is installation onto an embedded system. For the embedded version, pfSense uses NanoBSD, a tool for installing FreeBSD onto embedded systems. Such an install is ideal for a dedicated appliance (for example, a VPN server), and is geared toward fewer file writes. However, embedded installs cannot run some of the more interesting packages. Hardware sizing guidelines The minimum hardware requirements are general guidelines, and you may want to exceed these minimums based on different factors. It may be useful to consider these factors when determining what CPU, memory, and storage device to use. For the CPU, requirements increase for faster Internet connections. Guidelines for the CPU and network cards can be found at the official pfSense site at http://pfsense.org/hardware/#requirements. The following general guidelines apply: The minimum hardware specifications (Intel/AMD CPU of 500 MHz or greater) are valid up to 20 Mbps. CPU requirements begun to increase at speeds greater than 20 Mbps. Connections of 100 Mbps or faster will require PCI-e network adapters to keep up with the increased network throughput. If you intend to use pfSense to bridge interfaces—for example, if you want to bridge a wireless and wired network, or if you want to use pfSense as a switch—then the PCI bus speed should be considered. The PCI bus can easily become a bottleneck. Therefore, in such scenarios, using PCI-e hardware is the better option, as it offers up to 31.51 GB/s (for PCI-e v. 4.0 on a 16-lane slot) versus 533 MB/s for the fastest conventional PCI buses. If you plan on using pfSense as a VPN server, then you should take into account the effect VPN usage will have on the CPU. Each VPN connection requires the CPU to encrypt traffic, and the more connections there are, the more the CPU will be taxed. Generally, the most cost-effective solution is to use a more powerful CPU. But there are ways to reduce the CPU load from VPN traffic. Soekris has the vpn14x1 product range; these cards offload the CPU of the computing intensive tasks of encryption and compression. AES-NI acceleration of IPsec also significantly reduces the CPU requirements. If you have hundreds of simultaneous captive portal users, you will require slightly more CPU power than you would otherwise. Captive portal usage does not put as much of a load on the CPU as VPN usage, but if you anticipate having a lot of captive portal users, you will want to take this into consideration. If you're not a power user, 256 MB of RAM might be enough for your pfSense system. This, however, would leave little room for the state table (where, as mentioned earlier, active connections are tracked). Each state requires about 1 KB of memory, which is less memory than some consumer-grade routers require, but you still want to be mindful of RAM if you anticipate having a lot of simultaneous connections. The other components of pfSense require 32 to 48 MB of RAM, and possibly more, depending on which features you are using, so you have to subtract that from the available memory in calculating the maximum state table size. RAM Maximum Connections (States) 256 MB ~22,000 connections 512 MB ~46,000 connections 1 GB ~93,000 connections 2 GB ~190,000 connections Installing packages can also increase your RAM requirements; Snort and ntop are two such examples. You should also probably not install packages if you have limited disk space. Proxy servers in particular use up a fair amount of disk space, which is something you should probably consider if you plan on installing a proxy server such as Squid. The amount of disk space, as well as the form of storage you utilize, will likely be dictated by what packages you install, and what forms of logging you will have enabled. Some packages are more taxing on storage than others. Some packages require more disk space than others. Proxies such as Squid store web pages; anti-spam programs such as pfBlocker download lists of blocked IP addresses, and therefore require additional disk space. Proxies also tend to perform a great deal of read and write operations; therefore, if you are going to install a proxy, disk I/O performance is something you should likely take into consideration. You may be tempted to opt for the cheapest NICs. However, inexpensive NICs often have complex drivers that offload most of the processing to the CPU. They can saturate your CPU with interrupt handling, thus causing missed packets. Cheaper network cards typically have smaller buffers (often no more than 300 KB), and when the buffers become full, packets are dropped. In addition, many of them do not support Ethernet frames that are larger than the maximum transmission unit (MTU) of 1500 bytes. NICs that do not support larger frames cannot send or receive jumbo frames (frames with an MTU larger than 1500 bytes), and therefore they cannot take advantage of the performance improvement that using jumbo frames would bring. In addition, such NICs will often have problems with VLAN traffic, since a VLAN tag increases the size of the Ethernet header beyond the traditional size limit. The pfSense project recommends NICs based on Intel chipsets, and there are several reasons why such NICs are considered reliable. They tend to have adequately sized buffers, and do not have problems processing larger frames. Moreover, the drivers tend to be well-written and work well with Unix-based operating systems. For a typical pfSense setup, you will need two network interfaces: one for the WAN and one for the LAN. Each additional subnet (for example, for a guest network) will require an additional interface, as will each additional WAN interface. It should be noted that you don't need an additional card for each interface added; you can buy a multiport network card (most such cards have either 2 or 4 ports). You don't need to buy new NICs for your pfSense system; in fact, it is often economical to buy used NICs, and except in rare cases, the performance level will be the same. If you want to incorporate wireless connectivity into your network, you may consider adding a wireless card to your pfSense system. As mentioned earlier, however, the likely better option is to use pfSense in conjunction with a separate wireless access point. If you do decide to add a wireless card to your system and configure it for use as an access point, you will want to check the FreeBSD hardware compatibility list before making a purchase. Using a laptop You might be wondering if using an old laptop as a pfSense router is a good idea. In many respects, laptops are good candidates for being repurposed into routers. They are small, energy efficient, and when the AC power shuts off, they run on battery power, so they have a built-in uninterruptable power supply (UPS). Moreover, many old laptops can be purchased relatively cheaply at thrift shops and online. There is, however, one critical limitation to using a laptop as a router: in almost all cases, they only have one Ethernet port. Moreover, there is often no realistic way to add another NIC: as there are no expansion slots that will take another NIC (some, however, do have PCMCIA slots that will take a second NIC). There are gigabit USB-to-Ethernet adapters (for USB 3.0), but this is not much of a solution. Such adapters do not have the reliability of traditional NICs. Most laptops do not have Intel NICs either; high-end business laptops are usually the exception to this rule. There is a way to use a laptop with a single Ethernet port as a pfSense router, and that is to configure pfSense using VLANs. As mentioned earlier, VLANs, or virtual LANs, allow us to use a single NIC to serve multiple subnets. Thus, we can set up two VLANs on our single port: virtual LAN #1, which we will use for the WAN interface, and virtual LAN #2, which we will use for the LAN interface. The one disadvantage of this setup is that you must use a managed switch to make this work. Managed switches are switches that can usually be configured and managed as groups, they often have both a command-line and web interface for management, and they often have a wide range of capabilities, such as VLANs. Since unmanaged switches forward traffic to all other ports, they are unsuitable for this setup. You can, however, connect an unmanaged switch to the managed switch to add ports. Keep in mind that managed switches are expensive (more expensive than dual and quad port network cards), and if there are multiple VLANs on a single link, this link can easily become overloaded. In scenarios where you can add a network card, this is usually the better option. If you have an existing laptop, however, a managed switch with VLANs is a workable solution. Introduction to VLANs and DNS Two of the areas in which pfSense excels is in incorporating functionality to implement VLANs and DNS servers. First, let's consider why we would want to implement these. Introduction to VLANs The standard way to partition your network is to use a router to pass traffic between networks, and configure a separate switch (or switches) for each network. In this scenario, there is a one-to-one relationship between the number of network interfaces and the number of physical ports. This works well in many network deployments, especially in small networks. As the network gets larger, however, there are issues with this type of configuration. As the number of users on the network increases, we are faced with a choice of either having more users on each subnet, or increasing the number of subnets (and therefore the number of network interfaces on the router). Both solutions also create new problems: Each subnet makes up a separate broadcast domain. Increasing the number of users on a subnet increases the amount of broadcast traffic, which can bog down our network. Each user on a subnet can use a packet sniffer to sniff network traffic, which creates a security problem. Segmenting the network by adding subnets tends to be costly, as each new subnet requires a separate switch. VLANs offer us a way out of this dilemma with relatively little downside. VLANs allow us to divide traffic on a single network interface (for example, LAN) into several separate networks, by adding a special tag to frames entering the network. This tag, known as an 802.1q tag, identifies which VLAN to which the device belongs. Dividing network traffic in such a way offers several advantages: As each VLAN constitutes a separate broadcast domain, broadcast domains are now smaller, and thus there is less network traffic. Users on one VLAN cannot sniff traffic from another VLAN, even if they are on the same physical interface, thus improving security. Using VLANs requires us to have a managed switch on the interface on which VLANs exist. This is somewhat more expensive than an unmanaged switch, but the cost differential between a managed and unmanaged switch is less than it might be if we had to buy additional switches for new subnets. As a result, VLANs are often the most efficient way of making our networks more scalable. Even if your network is small, it might be advantageous to at least consider implementing a VLAN, as you will likely want to make future growth as seamless as possible. Introduction to DNS The DNS provides a means of converting an easy-to-remember domain name with a numerical (IP) address. It thus provides us with a phone book for the Internet as well as providing a structure that is both hierarchical (there is the root node, which covers all domain names, top-level domains like .com and .net, domain names and subdomain names) and decentralized (the Internet is divided into different zones, and a name server is authoritative for a specific zone). In a home or SOHO environment, we might not need to implement our own DNS server. In these scenarios, we could use our ISP's DNS servers to resolve Internet hostnames. For local hostnames, we could rely on NetBIOS under Windows, the Berkeley Internet Name Domain service (BIND) under Linux (using a configuration that does not require us to run name servers), or osx under Mac OS X. Another option for mapping hostnames to IP addresses on the local network would be to use HOSTS.TXT. This is a text file, which contains a list of hostnames and corresponding IP addresses. But there are certain factors that may prompt us to set up our own DNS server for our networks: We may have chosen to utilize HOSTS.TXT for name resolution, but maintaining the HOSTS.TXT file on each of the hosts on our network may prove to be too difficult. If we have roaming clients, it may even be impossible. If your network is hosting resources that are available externally (for example, an FTP server or a website), and you are constantly making changes to the IP addresses of these resources, you will likely find it much easier to update your own data rather than submit forms to third parties and wait for them to implement the changes. Although your DNS server will only be authoritative for your domains, it can cache DNS data from the rest of the Internet. On your local network, this cached data can be retrieved much faster than DNS data from a remote DNS server. Thus, maintaining your own DNS server should result in faster name resolution. If you anticipate ever having to implement a public DNS server, a private DNS server can be a good learning experience, and if you make mistakes in implementing a private DNS server, the consequences are not as far-reaching as they would be with a public one. Implementing a DNS server with pfSense is relatively easy. By using the DNS resolver, we can have pfSense answer DNS queries from local clients, and we can also have pfSense utilize any currently available DNS servers. We can also use third-party packages such as dns-server (which is a pfSense version of TinyDNS) to add DNS server functionality. The best practices for installation and configuration Once you have chosen your hardware and which version you are going to install, you can download pfSense. Browse to the Downloads section of pfsense.org and select the appropriate computer architecture (32-bit, 64-bit, or Netgate ADI), the appropriate platform (Live CD, memstick, or embedded), and you should be presented with a list of mirrors. Choose the closest one for the best performance. You will also want to download the MD5 checksum file in order to verify the integrity of the downloaded image. Windows has several utilities for displaying MD5 hashes for a file. Under BSD and Linux, generating the MD5 hash is as easy as typing the following command: md5 pfSense-LiveCD-2.2.6-RELEASE-amd64.iso This command would generate the MD5 checksum for the 64-bit Live CD version for pfSense 2.2.6. Compare the resulting hash with the contents of the .md5 file downloaded from the pfSense website. If you are doing a full install from the Live CD or memory stick, then you just need to write the ISO to the target media, boot from either the CD or memory stick, perform some basic configuration, and then invoke the installer. The embedded install is done from a compact flash (CF) card and console data can be sent to either a serial port or the VGA port, depending on which embedded configuration you chose. If you use the serial port version, you will need to connect the embedded system to another computer with a null modem cable. Troubleshooting installation In most cases, you should be able to invoke the pfSense installer and begin installing pfSense onto the system. In some cases, however, pfSense may not boot from the target media, or the system may hang during the boot process. If pfSense is not booting at all, you may want to check to make sure the system is set up to boot from the target media. This can be done by changing the boot sequence in the BIOS settings (which can be accessed during system boot, usually by hitting the Delete key). Most computers also have a means of choosing the boot device on a one-time basis during the boot sequence. Check your motherboard's manual on how to do this. If the system is already set up to boot from the target media, then you may want to verify the integrity of the pfSense image again, or repeat the process of writing the images to the target media. The initial pfSense boot menu when booting from a CD or USB flash drive. If the system hangs during the boot process, there are several options you can try. The first menu that appears, as pfSense boots, has several options. The last two options are Kernel and Configure Boot Options. Kernel allows you to select which kernel to boot from among the available kernels. If you have a reason to suspect that the FreeBSD kernel being used is not compatible with your hardware, you might want to switch to the older version. Configure Boot Options launches a menu (shown in the preceding screenshot) with several useful options. A description of these options can be found at: http://www.freebsd.org/doc/handbook/book.html. Toggling [A]CPI Support to off can help in some cases, as ACPI's hardware discovery and configuration capabilities may cause the pfSense boot process to hang. If turning this off doesn't work, you could try booting in Safe [M]ode, and if all else fails, you can toggle [V]erbose mode to On, which will give you detailed messages while booting. The two options after boot are [R]ecovery, and [I]nstaller. The [R]ecovery mode provides a shell prompt and helps recover from a crash by retrieving config.xml from a crashed hard drive. [I]nstaller allows you to install pfSense onto a hard drive or other media, and gets invoked by default after the timeout period. The installer provides you with the option to either do a quick install or a custom install. In most cases, the quick install option can be used. Invoking the custom install option is only recommended if you want to install pfSense on a drive other than the first drive on the target system, or if you want to install multiple operating systems on the system. It is not likely that either of these situations will apply, unless you are installing pfSense for evaluation purposes (and in such cases, you would probably have an easier time running pfSense on a virtual machine). If you were unable to install pfSense on to the target media, you may have to troubleshoot your system and/or installation media. If you are attempting to install from the CD, your optical drive may be malfunctioning, or the CD may be faulty. You may want to start with a known good bootable disc and see if the system will boot off of it. If it can, then your pfSense disc may be at fault; burning the disc again may solve the problem. If, however, your system cannot boot off the known good disc, then the optical drive itself, or the cables connecting the optical drive to the motherboard, may be at fault. In some cases, however, none of the aforementioned possibilities hold true, and it is possible that the FreeBSD boot loader will not work on the target system. If so, then you could opt to install pfSense on a different system. Another possibility is to install pfSense onto a hard drive on a separate system, then transfer the hard drive into the target system. In order to do this, go through the installation process on another system as you would normally until you get to the Assign Interfaces prompt. When the installer asks if you want to assign VLANS, type n. Type exit at the Assign Interfaces prompt to skip the interface assignment. Proceed through the rest of the installation; then power down the system and transfer the hard drive to the target system. Assuming that the pfSense hard drive is in the boot sequence, the system should boot pfSense and detect the system's hardware correctly. Then you should be able to assign network interfaces. The rest of the configuration can then proceed as usual. If you have not encountered any of these problems, the software should be installed on the target system, and you should get a dialog box telling you to remove the CD from the optical drive tray and press Enter. The system will now reboot, and you will be booting into your new pfSense install for the first time. pfSense configuration Configuration takes place in two phases. Some configuration must be done at the console, including interface configuration and interface IP address assignment. Some configuration steps, such as VLAN and DHCP setup, can be done both at the console and within the web GUI. Configuration from the console On boot, you should eventually see a menu identical to the one seen on the CD version, with the boot multi or single user options and other options. After a timeout period, the boot process will continue and you will get an options menu that is also identical to the CD version, except option 99 (installation option) will not be there. You should select 1 from the menu to begin interface assignment. This is where the network cards installed in the system are given their roles as WAN, LAN, and optional interfaces (OPT1, OPT2, and so on). If you select this option, you will be presented with a list of network interfaces. This list provides four pieces of information: pfSense's device name for the interface (fxp0, em1, and so on) The MAC address of the interface The link state of the interface (up if a link is detected; down otherwise) The manufacturer and model of the interface (Intel PRO 1000, for example) As you are probably aware, generally speaking, no two network cards have the same MAC address, so each of the interfaces in your system should have a unique MAC address. To begin the configuration, press 1 and Enter for the Assign Interfaces option. After that, a prompt will show up for VLAN configuration. Otherwise, type n and press Enter. Keep in mind that you can always configure VLANs later on. The interfaces must be configured, and you will be prompted for the WAN interface first. If you only configure one interface, it will be assigned to the WAN, and you will subsequently be able to login to pfSense through this port. This is not what you would normally want, as the WAN port is typically accessible from the other side of the firewall. Once at least one other interface is configured, you will no longer be able to login to pfSense from the WAN port. Unless you are using VLANs, you will have to set up at least two network interfaces. In pfSense, network interfaces are assigned rather cryptic device names (for example, fxp0, em1 and so on) and it is not always easy to know which ports correspond to particular device names. One way of solving this problem is to use the automatic interface assignment feature. To do this, unplug all network cables from the system and then type a and press Enter to begin auto-detection. The WAN interface is the first interface to be detected, so plug a cable into the port you intend to be the WAN interface. The process is repeated with each successive interface. The LAN interface is configured next, then each of the optional interfaces (OPT1, OPT2). If auto-detection does not work, or you do not want to use it, you can always choose manual configuration. You can always reassign network interfaces later on, so even if you make a mistake on this step, the mistake can be easily fixed. Once you have finished configuration, type y at the Do you want to proceed? prompt, or type n and enter to re-assign the interfaces. Option two on the menu is Set interface(s) IP address, and you will likely want to complete this step as well. When you invoke this option, you will be prompted to specify which interface's IP address is to be set. If you select WAN interface, you will be asked if you want to configure the IP address via DHCP. In most scenarios, this is probably the option you want to choose, especially if pfSense is acting as a firewall. In that case, the WAN interface will receive an IP address from your ISP's DHCP server. For all other interfaces (or if you choose not to use DHCP on the WAN interface), you will be prompted to enter the interface's IPv4 address. The next prompt will ask you for the subnet bit count. In most cases, you'll want to enter 8 if you are using a Class A private address, 16 for Class B, and 24 for Class C, but if you are using classless subnetting (for example, to divide a Class C network into two separate networks), then you will want to set the bit count accordingly. You will also be prompted for the IPv4 gateway address (any interface with a gateway set is a WAN, and pfSense supports multiple WANs); if you are not configuring the WAN interface(s), you can just hit Enter here. Next, you will be prompted to provide the address, subnet bit count, and gateway address for IPv6; if you want your network to fully utilize IPv6 addresses, you should enter them here. We have now configured as much as we need to from the console (actually, we have done more than we have to, since we really only have to configure the WAN interface from the console). The remainder of the configuration can be done from the pfSense web GUI. Configuration from the web GUI The pfSense web GUI can only be accessed from another PC. If the WAN was the only interface assigned during the initial setup, then you will be able to access pfSense through the WAN IP address. Once one of the local interfaces is configured (typically the LAN interface), pfSense can no longer be accessed through the WAN interface. You will, however, be able to access pfSense from the local side of the firewall (typically through the LAN interface). In either case, you can access the web GUI by connecting another computer to the pfSense system, either directly (with a crossover cable) or indirectly (through a switch), and then typing either the WAN or LAN IP address into the connected computer's web browser. The login screen should look similar to the following screenshot: The pfSense 2.3 web GUI login screen. When you initially log in to pfSense, the default username/password combination will be admin/pfsense respectively. On your first login, Setup Wizard will begin automatically. Click on the Next button to begin configuration. The first screen provides a link for information about a pfSense Gold subscription. You can click on the link to sign up, or click on the Next button. On the next screen, you will be prompted to enter the hostname of the router as well as the domain. Hostnames can contain letters, numbers and hyphens, but must begin with a letter. If you have a domain, you can enter it in the appropriate field. In the Primary DNS Server and Secondary DNS Server fields, you can enter your DNS servers. If you are using DHCP for your WAN, you can probably leave these fields blank, as they will usually be assigned automatically by your ISP. If you have alternate DNS servers you wish to use, you can enter them here. I have entered 8.8.8.8 and 8.8.4.4 as the primary and secondary DNS servers (these are two DNS servers run by Google that conveniently have easy to remember IP addresses). You can keep the Override DNS checkbox checked unless you have reason to use DNS servers other than the ones assigned by your ISP. Click on Next when finished. The next screen will prompt you for the Network Time Protocol (NTP) server as well as the local time zone. You can keep the default value for the server hostname for now. For the Timezone field, you should select the zone which matches your location and click on Next. The next screen of the wizard is the WAN configuration page. You will be prompted to select the WAN type. You can select either DHCP (the default type) or Static. If your pfSense system is behind another firewall and it is not going to receive an IP address from an upstream DHCP server, then you probably should choose Static. If pfSense is going to be a perimeter firewall, however, then DHCP is likely the correct setting, since your ISP will probably dynamically assign an IP address (this is not always the case, as you may have an IP address statically assigned to you by your ISP, but it is the more likely scenario). If you are not sure which WAN type to use, you will need to obtain this information from your ISP (the other choices are PPPoE, PPTP, and Static. PPPoE stands for Point-to-Point over Ethernet and PPTP stands for Point-to-Point Tunneling Protocol). The MAC address field allows you to enter a MAC address that is different from the actual MAC address of the WAN interface. This can be useful if your ISP will not recognize an interface with a different MAC address than the device that was previously connected, or if you want to acquire a different IP address (changing the MAC address will cause the upstream DHCP server to assign a different address). If you use this option, make sure the portion of the address reserved for the Organizationally Unique Identifier (OUI) is a valid OUI – in other words, an OUI assigned to a network card manufacturer. (The OUI portion of the address is the first three bytes of a MAC-48 address and the first five bytes of an EUI-48 address.) The next few fields can usually be left blank. Maximum Transmission Unit (MTU) allows you to change the MTU size if necessary. DHCP hostname allows you to send a hostname to your ISP when making a DHCP request, which is useful if your ISP requires this. Besides DHCP and Static, you can select PPTP or PPPoE as your WAN type. If you choose PPPoE, then there will be a field for a PPPoE Username, PPPoE Password, and PPPoE Server Name. The PPPoE dial-on-demand checkbox allows you to connect to your ISP only when a user requests data that requires an Internet connection. PPPoE Idle timeout specifies how long the connection will be kept open after transmitting data when this option is invoked. The Block RFC1918 Private Networks checkbox, if checked, will block registered private networks (as defined by RFC 1918) from connecting to the WAN interface. The Block Bogon Networks option blocks traffic from reserved and/or unassigned IP addresses. For the WAN interface, you should check both options unless you have special reasons for not invoking these options. Click the Next button when you are done. The next screen provides fields in which you can change LAN IP address and subnet mask, but only if you configured the LAN interface previously. You can keep the default, or change it to another value within the private address blocks. You may want to choose an address range other than the very common 192.168.1.x in order to avoid a conflict. Be aware that if you change the LAN IP address value, you will also need to adjust your PC's IP address, or release and renew its DHCP lease when finished with the network interface. You will also have to change the pfSense IP address in your browser to reflect the change. The final screen of the pfSense Setup Wizard allows you to change the admin password, which you should probably do. Enter the password, enter it again for confirmation in the next edit box, and click on Next. On the following screen, there will be a Reload button; click on Reload. This will reload pfSense with the new changes. Once you have completed the wizard, you should have network connectivity. Although there are other means of making changes to pfSense's configuration, if you want to repeat the wizard, you can do so by navigating to System | Setup Wizard. Completion of the wizard will take you to the pfSense dashboard. The pfSense dashboard, redesigned for version 2.3. Configuring additional interfaces By now, both the WAN and LAN interface configuration should be complete. Although additional interface configuration can be done at the console, it can also be done in the web GUI. To add optional interfaces, navigate to Interfaces | assign theInterface assignments tab will show a list of assigned interfaces, and at the bottom of the table, there will be an Available network ports: entry option. There will be a corresponding drop-down box with a list of unassigned network ports. These will have device names such as fxp0, em1, and so on. To assign an unused port, select the port you want to assign from the drop-down box, and click on the + button to the right. The page will reload, and the new interface will be the last entry in the table. The name of the interface will be OPTx, where x equals the number of optional interfaces. By clicking on interface name, you can configure the interface. Nearly all the settings here are similar to the settings that were available on the WAN and LAN configuration pages in the pfSense Setup Wizard. Some of the options under the General Configuration section, that are not available in the setup wizard, are MSS (Maximum Segment Size), and Speed and duplex. Normally, MSS should remain unchanged, although you can change this setting if your Internet connection requires it. If you click on the Advanced button under Speed and duplex, a drop-down box will appear in which you can explicitly set the speed and duplex for the interface. Since virtually all modern network hardware has the capability of automatically selecting the correct speed and duplex, you will probably want to leave this unchanged. If you have selected DHCP as the configuration type, then there are several options in addition to the ones available in the setup wizard. Alias IPv4 address allows you to enter a fixed IP address for the DHCP client. The Reject Leases from field allows you to specify the IP address or subnet of an upstream DHCP server to be ignored. Clicking on the Advanced checkbox in the DHCP client configuration causes several additional options to appear in this section of the page. The first is Protocol Timing, which allows you to control DHCP protocol timings when requesting a lease. You can also choose several presets (FreeBSD, pfSense, Clear, or Saved Cfg) using the radio buttons on the right. The next option in this section is Lease Requirements and Requests. Here you can specify send, request, and require options when requesting a DHCP lease. These options are useful if your ISP requires these options. The last section is Option Modifiers, where you can add DHCP option modifiers, which are applied to an obtained DHCP lease. There is a second checkbox at the top of this section called Config File Override. Checking this box allows you to enter a DHCP client configuration file. If you use this option, you must specify the full absolute path of the file. Starting with pfSense version 2.2.5, there is support for IPv6 with DHCP (DHCP6). If you are running 2.2.5 or above, there will be a section on the page called DHCP6 client configuration. The first setting is Use IPv4 connectivity as parent interface. This allows you to request an IPv6 address over IPv4. The second is Request only an IPv6 prefix. This is useful if your ISP supports Stateless Address Auto Configuration(SLAAC). In this case, instead of the usual procedure in which the DHCP server assigns an IP address to the client, the DHCP server only sends a prefix, and the host may generate its own IP address and test the uniqueness of a generated address in the intended addressing scope. By default, the IPv6 prefix is 64 bits, but you can change that by altering the DHCPv6 Prefix Delegation size in the corresponding drop-down box. The last setting is the Send IPv6 prefix hint, which allows you to request the specified prefix size from your ISP. The advanced DHCP6 client configuration section of the interface configuration page. This section appears if DHCP6 is selected as the IPv6 configuration type. Checking the Advanced checkbox in the heading of this section displays the advanced DHCP 6 options. If you check the Information Only checkbox on the left, pfSense will send requests for stateless DHCPv6 information. You can specify send and request options, just as you can for IPv4. There is also a Script field where you can enter the absolute path to a script that will be invoked on certain conditions. The next options are for the Identity Association Statement checkboxes. The Non-Temporary Address Allocation checkbox results in normal, that is, not temporary, IPv6 addresses to be allocated for the interface. The Prefix Delegation checkbox causes a set of IPv6 prefixes to be allocated from the DHCP server. The next set of options, Authentication Statement, allow you to specify authentication parameters to the DHCP server. The Authname parameter allows you to specify a string, which in turn specifies a set of parameters. The remaining parameters are of limited usefulness in configuring a DHCP6 client, because each has only one allowed value, and leaving them blank will result in only the allowed value being used. If you are curious as to what these values are, here they are: Parameter Allowed value Description Protocol Delayed The DHCPv6 delayed authentication protocol Algorithm hmac-md5, HMAC-MD5, hmacmd5, or HMACMD5 The HMAC-MD5 authentication algorithm rdm Monocounter The replay protection method; only monocounter is available Finally, Key info Statement allows you to enter a secret key. The required fields are key id, which identifies the key, and secret, which provides the shared secret. key name and realm are arbitrary strings and may be omitted. expire may be used to specify an expiration time for the key, but if it is omitted, the key will never expire. The last section on the page is identical to the interface configuration page in the Setup Wizard, and contains the Block Private Networks and Block Bogon Networks checkboxes. Normally, these are checked for WAN interfaces, but not for other interfaces. General setup options You can find several configuration options under System | General Setup. Most of these are identical to settings that can be configured in the Setup Wizard (Hostname, Domain, DNS servers, Timezone, and NTP server). There are two additional settings available. The Language drop-down box allows you to select the web configurator language. Under the Web Configurator section, there is a Theme drop-down box that allows you to select the theme. The default theme of pfSense is perfectly adequate, but you can select another one here. pfSense 2.3 also adds new options to control the look and feel of the web interface; these settings are also found in the Web Configurator section of the General Settings page. The Top Navigation drop-down box allows you to choose whether the top navigation scrolls with the page, or remains anchored at the top as you scroll. The Dashboard Columns option allows you to select the number of columns on the dashboard page (the default is 2). The next set of options is Associated Panels Show/Hide. These options control the appearance of certain panels on the Dashboard and System Logs page. The options are: Available Widgets: Checking this box causes the Available Widgets panel to appear on the Dashboard. Prior to version 2.3, the Available Widgets panel was always visible on the Dashboard. Log Filter: Checking this box causes the Advanced Log Filter panel to appear on the System Logs page. Advanced Log Filter allows you to filter the system logs by time, process, PID and message. Manage Log: Checking this box causes the Manage General Log panel to appear on the System Logs page. The Manage General Log panel allows you to control the display of the logs, how big the log file may be, and the formatting of the log file, among other things. The last option on this page, Left Column Labels, allows you to select/toggle the first item in a group by clicking on the left column if checked. Click on Save at the bottom of the page to save any changes. Advanced setup options Under System | Advanced, there are a number of options that you will probably want to configure before completing the initial setup. There are six separate tabs here, all with multiple options, and we won't cover all of them here, but we will cover the more common ones. The first setting allows you to choose between HTTP and HTTPS for the web configurator. If you plan on making the pfSense web GUI accessible from the WAN side, you will definitely want to choose HTTPS in order to encrypt access to the web GUI. Even if the web GUI will only be accessible over local networks, you probably will want to choose HTTPS. Modern web browsers will complain about the SSL certificate the first time you access the web GUI, but most of them will allow you to create an exception. The next setting, SSL certificate, allows you to choose a certificate from a drop-down list of available certificates. You can choose web Configurator default, or you can add another certificate (by navigating to System | Cert Manager and adding one), and use it instead. The next important setting, also in the Web Configurator section, is the Disable web Configurator anti-lockout rule. If left unchecked, access to the web GUI is always allowed on the LAN (or WAN if the LAN interface has not been assigned), regardless of any user-defined firewall rules. If you check this option and you don't have a user-defined rule to allow access to pfSense, you will lock yourself out of the web GUI. If you are locked out of the web GUI because of firewall rules, there are several options. The easiest option is probably to restore a previous configuration from the console. You can also reset pfSense to factory defaults, but if you don't mind typing in shell commands, there are less drastic options. One is to add an allow all rule on the WAN interface by typing the following command at the console shell prompt (type 8 at the console menu to invoke the shell): pfSsh.php playback enableallowallwan Once you issue this command, you will be able to access the web GUI through the WAN interface. To do so, either connect the WAN port to a network running DHCP (if the WAN uses DHCP), or connect the WAN port to another computer with an IP on the same network (if the WAN has a static IP). Be sure to delete the WAN allow all rule before deploying the system. Another possibility is to temporarily disable the firewall rules with the following shell command: pfctl –d Once you have regained access, you can re-enable the firewall rules with this command: pfctl -e In any case, you want to make sure your firewall rules are configured correctly before invoking the anti-lockout option. You can reset pfSense to factory defaults by selecting 4 from the console menu. If you need to go back to a previous configuration, you can do that by selecting 15 from the console menu; this option will allow you to select from automatically-saved restore points. The next section is Secure Shell; checking the Enable Secure Shell checkbox makes the console accessible via a Secure Shell (SSH) connection. This makes life easier for admins, but it also creates a security concern. Therefore, it is a good idea to change the default SSH port (the default is 22), which you can do in this section. You can add another layer of security by checking the Disable password login for the Secure Shell checkbox. If you invoke this option, you must create authorized SSH keys for each user that requires SSH access. The process for generating SSH keys is different depending on your OS. Under Linux, it is fairly simple. First, enter the following at the command prompt: ssh-keygen –t rsa You will receive the following prompt: Enter file in which to save the key (/home/user/.ssh/id-rsa): The directory in parenthesis will be a subdirectory of your home directory. You can change the directory or press Enter. The next prompt asks you for a passphrase: Enter passphrase (empty for no passphrase): You can enter a passphrase here or just press Enter. You will be prompted to enter the passphrase again, and then the public/private key pair will be generated. The public key will now be saved in a file called id-rsa.pub. Entering SSH keys for a user in the user manager. The next step is adding the newly generated public key to the admin account in pfSense. Open the file in the text editor of your choice and in the web GUI, select the public key and copy it to the clipboard. Then navigate to System | User Manager and click on the Edit user icon for the appropriate user. Scroll down to the Keys section and paste the key into the Authorized SSH keys box. Then click on Save at the bottom of the page. You should now be able to SSH into the admin account without entering the password. Type the following at the command line: ssh pfsense_address –ladmin Here pfsense_address is the IP address of the pfSense system. If you specified a passphrase earlier, you will be prompted to enter it in order to unlock the private key. You will not be prompted for the passphrase on subsequent logins. Once you unlock the private key, you should be logged into the console. The last section of the page, Console options, gives you one more layer of security by allowing you to require a password for console login. Check this checkbox if you want to enable this option, although this could result in being locked out if you forget the password. If this happens, you may still be able to restore access by booting from the live CD and doing a pre-flight install, described in a subsequent section. The next tab, Firewall/NAT, contains a number of important settings relating to pfSense's firewall functionality. Firewall Optimization Options allows you to select the optimization algorithm for the state table. The Normal option is designed for average case scenario network usage. High latency, as the name implies, is for connections in which it is expected that there will be a significant delay between a request and response (a satellite connection is a good example). Aggressive and Conservative are inverses of each other. Aggressive is more aggressive than Normal in dropping idle connections, while Conservative will leave idle connections open longer than Normal would. Obviously, the trade-off here is that if we expire idle connections too soon, legitimate connections may be dropped, while keeping them open too long will be costly from a resource (CPU and memory) standpoint. In the Firewall Advanced section, there is a Disable all packet filtering checkbox. Enabling this option disables all firewall functionality, including NAT. This should be used with caution, but may be useful in troubleshooting. The Firewall maximum settings and Firewall maximum table entries options allow you to specify the maximum number of connections and maximum number of table entries respectively to hold in the system state table. If you leave these entries blank, pfSense will assign reasonable defaults based on the amount of memory your system has. Since increasing the maximum number of connections and/or state table entries will leave less memory for everything else, you will want to invoke these options with caution. The static route filtering checkbox, if checked, will result in firewall rules not taking effect for traffic that enters and leaves through the same interface. This can be useful if you have a static route in which traffic enters pfSense through an interface, but the source of the traffic is not the same as the interface on which it enters. This option does not apply to traffic whose source and destination is the same interface – such traffic is intra-network traffic, and firewall rules would not apply to it whether or not this option was invoked. The next section of the page, Bogon Networks, allows you to select the update frequency of the list of addresses reserved, or not yet assigned, by IANA. If someone is trying to access your network from a newly-assigned IP address, but the Bogon networks list has not yet been updated, they may find themselves blocked. If this is happening on a frequent basis, you may want to change the update frequency. The next tab, Networking, contains a number of IPv6 options. The Allow IPv6 checkbox must be checked in order for IPv6 traffic to pass (it is checked by default). The next option, IPv6 over IPv4 Tunneling, allows you to enable the transitional IPv6 over IPv4. There is also an option called Prefer IPv4 even when IPv6 is available, which will cause IPv4 to be used in cases where a hostname resolves both IPv4 and IPv6 addresses. The next tab is called Miscellaneous. The Proxy Port section allows you to specify URL for a remote proxy server, as well as the proxy port as well as a username and password. The following section, Load Balancing, has two settings. The first setting, Use sticky connections, causes successive connections from the same source to be connected to the same server, instead of directing them to the next web server in the pool, which would be the normal behavior. The timeout period for sticky connections may be adjusted in the adjacent Edit box. The default is 0, so the sticky connection expires as soon as the last connection from the source expires. The second setting, Enable default gateway switching, switches from the default gateway to another available one when the default gateway goes down. This is not necessary in most cases, since it is easier to incorporate redundancy into gateways with gateway groups. The Scheduling section has only one option, but it has significance if you use rule scheduling. Checking the Do not kill connections when schedule expires checkbox will cause connections permitted by the rule to survive even after the time period specified by the schedule expires. Otherwise, pfSense will kill all existing connections when a schedule expires. Upgrading, backing up, and restoring pfSense You can usually upgrade pfSense from one version to another, although the means of upgrading may differ depending on what platform you are using. So long as the firmware is moving from an older version to a newer version, pfSense will work unless otherwise noted. Before you make any changes, you should make an up-to-date backup. In the web GUI, you can back up the configuration by navigating to Diagnostics | Backup/Restore. In the Backup Configuration section of the page, set Backup Area to ALL. Then click on Download Configuration and save the file. Before you upgrade pfSense, it is a good idea to have a plan on how to recover in case the upgrade goes wrong. There is always a chance that an upgrade will leave pfSense in an unusable state. In these cases, it is always helpful to have a backup system available. Also, with advance planning, the firewall can be quickly returned to the previous release. There are three methods for upgrading pfSense. The first is to download the upgrade binaries from the official pfSense site. The same options are available as are available for a full install. Just download the appropriate image, write the image to the target media, and boot the system to be upgraded from the target media. For embedded systems, releases prior to 1.2.3 are not upgradable (in such cases, a full install would be the only way to upgrade), but newer NanoBSD-based embedded images do support upgrades. The second method is to upgrade from the console. From the console menu, select 13 (the Upgrade from Console option). pfSense will check the repositories to see if there is an update, and if there is, how much more disk space is required, and also inform you that upgrading will require a reboot. It will also prompt you to confirm that the upgrade should proceed. Type y and Enter, and the upgrade will proceed. pfSense will also automatically reboot 10 seconds after downloading and installing the upgrade. Rebooting may take slightly longer than it would normally, since pfSense must extract the new binaries from a tarball during the boot sequence. Upgrading pfSense from the console. The third method is the easiest way to upgrade your system: from the web GUI. Navigate to Status | Dashboard (this should also be the screen you see when initially logging into the web GUI). The System Information widget should have a section called Version, and this section should provide: The current version of pfSense Whether an update is available If an update is available, there will be a link to the firmware auto update page; click on this link. (Alternatively, you can access this page by navigating to System | Update and clicking on the System Update tab (note that on versions prior to 2.3, this menu option was called Firmware instead of Update.) If there is an update available, this page will let you know. Choosing a firmware branch from the Update Settings tab of the Update option. The Update Settings tab contains options that may be helpful in some situations. The Firmware Branch section has a drop-down box, allowing you to select either the Stable branch or Development branch. The Dashboard check checkbox allows you to disable the dashboard auto-update check. Once you are satisfied with these settings, you can click on the Confirm button on the System Update tab. The updating process will then begin, starting with the backup (if you chose that option). Upgrading can take as little as 15 minutes, especially if you are upgrading from one minor version to another. If you are upgrading in a production environment, you will want to schedule your upgrade for a suitable time (either during the weekend or after normal working hours). The web GUI will keep you informed of the status of the update process and when it is complete. Another means of updating pfSense in the web GUI is to use the manual update feature. To do so, navigate to System | Update and click on the Manual Update tab. Click on the Enable firmware upload button. When you do this, a new section should appear on the page. The Choose file button launches a file dialog box where you can specify the firmware image file. Once you select the file, click on Open to close out the file dialog box. There is a Perform full backup prior to upgrade checkbox you can check if you want to back up the system, and also an Upgrade firmware button that will start the upgrade process. If the update is successful, the System Information widget on the Dashboard should indicate that you are on the current version of pfSense (or the version to which you upgraded, if you invoked the manual update). If something went wrong and pfSense is not functioning properly, and you made a backup prior to updating, you can restore the old version. Available methods of backing up and restoring pfSense are outlined in the next section. Backing up and restoring pfSense The following screenshot shows the options related to backing up and restoring pfSense: Backup and restore options in pfSense 2.3. You can back up and restore the config.xml file from the web GUI by navigating to Diagnostics | Backup/Restore. The first section, Backup configuration, allows you to back up some or all of the configuration data. There is a drop-down box which allows you to select which areas to backup. There are checkbox options such as do not backup package information, and Encrypt this configuration file. The final checkbox, selected by default, allows you to disable the backup of round robin database (RRD) data, real-time traffic data which you likely will not want to save. The Download Configuration as XML button allows you to save config.xml to a local drive. Restoring the configuration is just as easy. In the Restore configuration section of the page, select the area to restore from the drop-down box and browse to the file by clicking on the Choose File button. Specify whether config.xml is encrypted with the corresponding checkbox, and then click the Restore configuration button. Restoring a configuration with Pre-Flight Install You may find it is necessary to restore an old pfSense configuration. Moreover, it is possible that restoring an old configuration from the console or web GUI as described previously in this article is not possible. In these cases, there is one more possible way of restoring an old configuration, and that is with a Pre-Flight Install (PFI), A PFI essentially involves the following: Copying a backup config.xml file into a directory called conf on a DOS/FAT formatted USB drive. Plugging the USB drive into the system whose configuration is to be restored, and then booting off the Live CD. Installing pfSense from the CD onto the target system. Rebooting the system, and allowing pfSense to boot (off the target media, not the CD). The configuration should now be restored. Another option that is useful if you want to retain your configuration while reinstalling pfSense is to choose the menu option Rescue config.xml during the installation process. This allows you to select and load a configuration file from any storage media attached to the system. Summary The goal of this article was to provide an overview of how to get pfSense up and running. Completion of this article should give you an idea of where to deploy your pfSense system as well as what hardware to utilize. You should also know how to troubleshoot the most common installation problems, and how to do basic system configuration and interface setup for both IPv4 and IPv6 networks. You should know how to configure pfSense for remote access. Finally, you should know how to upgrade, backup, and restore pfSense. Resources for Article: Further resources on this subject: Configuring the essential networking services provided by pfSense [article] pfSense: Configuring NAT and Firewall Rules [article] Upgrading a Home Network to a Small Business System Using pfSense [article]
Read more
  • 0
  • 0
  • 9379

article-image-overview-horizon-view-architecture-and-components
Packt
06 Sep 2016
18 min read
Save for later

An Overview of Horizon View Architecture and Components

Packt
06 Sep 2016
18 min read
In this article by Peter von Oven, author of the book Mastering VMware Horizon 7 - Second Edition, we will introduce you to the architecture and infrastructure components that make up the core VMware Horizon solution, concentrating on the virtual desktop elements of Horizon with Horizon Standard edition, plus the Instant Clone technology that is available in the Horizon Enterprise edition. We are going to concentrate on the core Horizon View functionality of brokering virtual desktop machines that are hosted on a VMware vSphere platform. Throughout the sections of this article we will discuss the role of each of the Horizon View components, explaining how they fit into the overall infrastructure, their role, and the benefits they bring. Once we have explained the high-level concept, we will then take a deeper dive into how that particular component works. As we work through the sections we will also highlight some of the best practices as well as useful hints and tips along the way. We will also cover some of the third-party technologies that integrate and compliment Horizon View, such as antivirus solutions, storage acceleration technologies, and high-end graphics solutions that help deliver a complete end-to-end solution. After reading this article, you will be able to describe each of the components and what part they play within the solution, and why you would use them. (For more resources related to this topic, see here.) Introducing the key Horizon components To start with, we are going to introduce, at a high level, the core infrastructure components and the architecture that make up the Horizon View product. We will start with the high-level architecture, as shown in the following diagram, before going on to drill down into each part in greater detail. All of the VMware Horizon components described are included as part of the licensed product, and the features that are available to you depend on whether you have the Standard Edition, the Advanced Edition, or the Enterprise Edition. It’s also worth remembering that Horizon licensing also includes ESXi and vCenter licensing to support the abilityto deploy the core hosting infrastructure. You can deploy as many ESXi hosts andvCenter servers as you require to host the desktop infrastructure. High-level architectural overview In this section, we will cover the core Horizon View features and functionality for brokeringvirtual desktop machines that are hosted on the VMware vSphere platform. The Horizon View architecture is pretty straightforward to understand, as its foundations lie in the standard VMware vSphere products (ESXi and vCenter). So, if you have the necessary skills and experience of working with this platform, then you are already nearly halfway there. Horizon View builds on the vSphere infrastructure, taking advantage of some of the features of the ESXi hypervisor and vCenter Server. Horizon View requires adding a number of virtual machines to perform the various View roles and functions. An overview of the View architecture for delivery virtual desktops is shown in the following diagram: View components run as applications that are installed on the Microsoft WindowsServer operating system,with the exception of the Access Point which is a hardened Linux appliance, so they could actually run on physical hardware as well.However, there are a great number of benefits available when you run them asvirtual machines, such as delivering HA and DR, as well as the typical cost savingsthat can be achieved through virtualization. The following sections will cover each of these roles/components of the Viewarchitecture in greater detail, starting with the Horizon View Connection Server. The Horizon View Connection Server The Horizon View Connection Server, sometimes referred to as Connection Broker or View Manager, is the central component of the View infrastructure. Its primary role is to connect a user to their virtual desktop by means of performing user authentication and then delivering the appropriate desktop resources based on the user's profile and user entitlement. When logging on to your virtual desktop, it is the Connection Server that you are communicating with How does the Connection Server work? A user will typically connect to their virtual desktop machine from their end point device by launching the View Client, but equally they could use browser-based access. So how does the login process work? Once the View Client has launched (shown as 1 in the diagram on the following page), the user enters the address details of the View Connection Server, which in turn responds (2) by asking them to provide their network login details (their Active Directory (AD) domain username and password). It's worth noting that Horizon View now supports the following different AD Domain functionallevels: Windows Server 2003 Windows Server 2008 and 2008 R2 Windows Server 2012 and 2012 R2 Based on the user’s entitlements, these credentials are authenticated with AD (3) and, if successful, the user is able to continue the logon process. Depending on what they are entitled to, the user could see a launch screen that displays a number of different virtual desktop machine icons that are available for them to login to. These desktop icons represent the desktop poolsthat the user has been entitled to use. A pool is basically a collection of like virtual desktop machines; for example, it could be a pool for the marketing department where the virtual desktop machines contain specific applications/software for that department. Once authenticated, the View Manager or Connection Server makes a call to the vCenter Server (4) to create avirtual desktop machine and then vCenter makes a call (5) to either View Composer (if you areusing linked clones) or will create an Instant Clone using the VM Fork feature of vSphere to start the build process of the virtual desktop if there is not onealready available for the user to login to. When the build process has completed, and the virtual desktop machine is available to the end user,it is displayed/delivered within the View Client window (6) using the chosen display protocol (PCoIP, Blast, or RDP). This process is described pictorially in the following diagram: There are other ways to deploy VDI solutions that do not require a connection broker, although you could argue that strictly speaking this is not a true VDI solution. This is actually what the first VDI solutions looked like, and just allowed a user to connect directly to their own virtual desktop via RDP. If you think about it there are actually some specific use cases for doing just this. For example, if you have a large number of remote branches or offices, you could deploy local infrastructure allowingusers to continue working in the event of a WAN outageor poor network communication between the branch and head office. The infrastructure required would be a sub-set of what you deploy centrally in order to keep costs minimal. It just so happens that VMware have also thought of this use case and have a solution that’s referred to as a Brokerless View, which uses the VMware Horizon View Agent Direct-Connection Plugin. However, don't forget that, in a Horizon View environment, the View Connection Server provides greater functionality and does much more than just connecting users to desktops, as we will see later in this article. As we previously touched on, the Horizon View Connection Server runs as an application on a Windows Server which could be either be a physical or a virtual machine. Running as a virtual machine has many advantages; for example, it means that you can easily add high-availability features, which are critical in this environment, as you could potentially have hundreds or maybe even thousands of virtual desktop machines running on a single host server. Along with brokering the connections between the users and virtual desktop machines, the Connection Server also works with vCenter Server to manage the virtual desktop machines. For example, when using Linked Clones or Instant Clones and powering on virtual desktops, these tasks are initiated by the Connection Server, but they are executed at the vCenter Server level. Now that we have covered what the Connection Server is and how it works, in the next section we are going to look at the requirements you need for it to run. Minimum requirements for the Connection Server To install the View Connection Server, you need to meet the following minimum requirements to run on physical or virtual machines: Hardware requirements: The following table shows the hardware required: Supported operating systems: The View Connection Server must be installed on one of the following operating systems listed in the table below: In the next section we are going to look at the Horizon View Security Server. The Horizon View Security Server The Horizon View Security Server is another component in the architecture and is essentially another version of the View Connection Server but, this time, it sits within your DMZ so that you can allow end users to securely connect to their virtual desktop machine from an external network or the Internet. How does the Security Server work? To start with, the user login process at the beginning is the same as when connecting to a View Connection Server, essentially because the Security Server is just another version of the Connection Server running a subset of the features. The difference being is that you connect to the address of the Security Server.The Security Server sits inside your DMZ and communicates with a Connection Server sitting on the internal network that it ispaired with. So now we have added an extra security layer as the internal Connection Server is not exposed externally, with theidea being that users can now access their virtual desktop machines externally withoutneeding to first connect to a VPN on the network first.The Security Server should not be joined to the Domain. This process is described pictorially in the following diagram: We mentioned previously that the Security Server is paired with a Connection Servers. The pairing is configured by the use of a one-time password during installation. It's a bit like pairing your smart phone with the hands-free kit in your car using Bluetooth. When the user logs in from the View Client, they now use the external URL of the Security Server to access the Connection Server, which in turn authenticates the user against AD. If the Connection Server is configured as a PCoIP gateway, then it will pass the connection and addressing information to the View Client. This connection information will allow the View Client to connect to the Security Server using PCoIP. This is shown in the diagram by the green arrow (1). The Security Server will then forward the PCoIP connection to the virtual desktop machine, (2) creating the connection for the user. The virtual desktop machine is displayed/delivered within the View Client window (3) using the chosen display protocol (PCoIP, Blast, or RDP). The Horizon View Replica Server The Horizon View Replica Server, as the name suggests, is a replica or copy of a View Connection Server and serves two key purposes. The first is that it is used to enable high availability to your HorizonView environment. Having a replica of your View Connection Server means that, if the Connection Server fails, users are still able to connect to their virtualdesktop machines. Secondly, adding Replica Servers allows you to scale up the number of users and virtual desktop connections. An individual instance of a Connection Server can support 2000 connections, so by adding additional Connection Servers allows you to add another 2000 users at a time, up to the maximum of five connection servers and 10,000 users per Horizon View Pod. When deploying a Replica Server, you will need to change the IP address or update the DNSrecord to match this server if you are not using a load balancer. How does the Replica Server work? So, the first question is, what actually gets replicated? The Connection Broker stores all its information relating to the end users, desktop pools, virtual desktopmachines, and other View-related objects, in an Active Directory Application Mode(ADAM) database. Then, using the Lightweight Directory Access Protocol (LDAP) (it uses a method similar to the one AD uses for replication), this View information getscopied from the original Connection Server to the Replica Server. As both, the Connection Server and the Replica Server are now identical to each other, if your Connection Server fails, then you essentially have a backup that steps in and takes over so that end users can still continue to connect to their virtual desktop machines. Just like with the other components, you cannot install the Replica Server role on the same machine that is running as a Connection Server or any of the other Horizon View components. The Horizon View Enrollment Server and True SSO The Horizon View Enrollment Server is the final component that is part of the Horizon View Connection Server installation options, and is selected from the drop-down menu from the installation options screen.So what does the Enrollment Server do? Horizon 7 sees the introduction to a new feature called True SSO. True SSO is a solution that allows a user to authenticate to a Microsoft Windows environment without them having to enter their AD credentials.It integrates into another VMware product, VMware Identity Manager which forms part of both Horizon 7 Advanced and Enterprise Editions. Its job is to sit between the Connection Server and the Microsoft Certificate Authority and to request temporary certificates from the certificate store. This process is described pictorially in the following diagram: A user first logs in to VMware Identity Manager either using their credentials or other authentication methods such as smartcards or biometric devices. Once successfully authenticated, the user will be presented with the virtual desktop machines or hosted applications that they are entitled to use. They can launch any of these by simply double clicking, which by doing so will launch the Horizon View Client as shown by the red arrow (1) in the previous diagram. The user’s credentials will then be passed to the Connection Server (2) which in turn will verify them by sending a Security Assertion Markup Language (SAML)assertion back to the Identity Manager (3). If the user’s credentials are verified, then the Connection Server passes them on to the Enrollment Server (4). The Enrollment Server then makes a request to the Microsoft Certificate Authority (CA) to generate a short-lived, temporary certificate for that user to use (5). With the certificate now generated, the Connection Server presents it to the operating system of the virtual desktop machine (6), which in turn validates with Active Directory as to whether or not the certificate is authentic (7). When the certificate has been authenticated then the user is logged on to theirvirtual desktop machine which will be displayed/delivered to the View Client using the chosen display protocol (8). True SSO is supported with all Horizon 7 supported desktop operating systems for desktops as well Windows Server 2008 R2 and Windows Server 2012 R2. It also supports PCoIP, HTML, and Blast Extreme delivery protocols. VMware Access Point VMware Access Point performs exactly the same functionality as the View Security Server, as shown in the following diagram, however with one key difference. Instead of being a Windows application and another role of the Connection Server, the Access Point is a separate virtual appliance that runs a hardened, locked-down Linux operating system. Although the Access Point appliance delivers pretty much the same functionality as the Security Server, it does not yet completely replace it. Especially if you already have a production deployment that uses the Security Server for external access. You can continue to use this architecture. If you are using the secure tunnel function, PCoIP Secure Gateway, or the Blast Secure Gateway features of the Connection Server, then these features will need to be disabled on the Connection Server if you are using the Access Point. They are all enabled by default on the Access Point appliance. A key difference between the Access Point appliance and the Security Server is in the way it scales. Before you had to pair a Security Server with a Connection Server, which was a limitation, but this is now no longer the case. As such you can now scale to as many Access Point appliances as you need for your environment, with the maximum limit being around 2000 sessions for a single appliance. Adding additional appliances is simply a case of deploying the appliance as appliances don’t depend on other appliances and do not communicate with them. They communicate directly with the Connection Servers. Persistent or non-persistent desktops In this section, we are going to talk about the different types of desktop assignments, and the way a virtual desktop machine is delivered to an end user. This is an important design consideration as the chosen method could potentially impact on the storage requirements (covered in the next section), the hosting infrastructure, and also which technology or solution is used to provision the desktop to the end users. One of the questions that always get asked is whether you should deploy a dedicated (persistent) assignment, or a floating desktop assignment (non-persistent). Desktops can either be individual virtual machines, which are dedicated to a user on a 1:1 basis (as we have in a physical desktop deployment, where each user effectively owns their own desktop), or a user has a new, vanilla desktop that gets provisioned, built, personalized, and then assigned at the time of login. The virtual desktop machine is chosen at random from a pool of available desktops that the end user is entitled to use. The two options are described in more detail as follows: Persistent desktop: Users are allocated a desktop that retains all of their documents, applications, and settings between sessions. The desktop is statically assigned the first time that the user connects and is then used for all subsequent sessions. No other user is permitted access to the desktop. Non-persistent desktop: Users might be connected to different desktops from the pool, each time that they connect. Environmental or user data does not persist between sessions and instead is delivered as the user logs on to their desktop. The desktop is refreshed or reset when the user logs off. In most use cases, a non-persistent configuration is the best option, the key reason is that, in this model, you don't need to build all the desktops upfront for each user. You only need to power on a virtual desktop as and when it's required. All users start with the same basic desktop, which then gets personalized before delivery. This helps with concurrency rates. For example, you might have 5,000 people in your organization, but only 2,000 ever login at the same time; therefore, you only need to have 2,000 virtual desktops available. Otherwise, you would have to build a desktop for each one of the 5,000 users that might ever log in, resulting in more server infrastructure and certainly a lot more storage capacity. We will talk about storage in the next section. The one thing that used to be a bit of a show-stopper for non-persistent desktops was around how to deliver the applications to the virtual desktop machine. Now application layering solutions such as VMware App Volumes is becoming a more main stream technology, the applications can now be delivered on demand as the desktop is built and the user logs in. Another thing that we often see some confusion over is the difference between dedicated and floating desktops, and how linked clones fit in. Just to make it clear, linked clones, full clones, and Instant Clones are not what we are talking about when we refer to dedicated and floating desktops. Cloning operations refer to how a desktop is built and provisioned, whereas the terms persistent and non-persistent refer to how a desktop is assigned to an end user. Dedicated and floating desktops are purely about user assignment and whether a user has a dedicated desktop or one allocated from a pool on-demand. Linked clones and full clones are features of Horizon View, which uses View Composer to create the desktop image for each user from a master or parent image. This means, regardless of having a floating or dedicated desktop assignment, the virtual desktop machine could still be a linked or full clone. So, here's a summary of the benefits: It is operationally efficient: All users start from a single or smaller number of desktop images. Organizations reduce the amount of image and patch management. It is efficient storage-wise: The amount of storage required to host the non-persistent desktop images will be smaller than keeping separate instances of unique user desktop images. In the next sections, we are going to cover an in-depth overview of the cloning technologies available in Horizon 7, starting with Horizon View Composer and linked clones, and the advantages the technology delivers. Summary In this article, we dscussed the Horizon View architecture and the different components that make up the complete solution. We covered the key technologies, such as how linked clones and Instant Clones work to optimize storage, and then introduced some of the features that go toward delivering a great end user experience, such as delivering high-end graphics, unified communications, profile management, and how the protocols deliver the desktop to the end user. Resources for Article: Further resources on this subject: An Introduction to VMware Horizon Mirage [article] Upgrading VMware Virtual Infrastructure Setups [article] Backups in the VMware View Infrastructure [article]
Read more
  • 0
  • 0
  • 11764
Modal Close icon
Modal Close icon