Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7012 Articles
article-image-secure-azure-virtual-network
Gebin George
18 May 2018
7 min read
Save for later

How to secure an Azure Virtual Network

Gebin George
18 May 2018
7 min read
The most common question that anyone asks when they buy a service is, can it be secured? The answer to that question, in this case, is absolutely yes. In this tutorial, we will learn to secure your connection between virtual machines. On top of the security, Microsoft provides for Azure as a vendor, there are some configurations that you can do at your end to increase the level of security to your virtual network. For a higher level of security, you can use the following: NSG: It is like a firewall that controls the inbound and outbound traffic by specifying which traffic is allowed to flow to/from the NIC/subnet Distributed denial of service (DDoS) protection: It is used to prevent DDoS attacks and at the time of writing is in preview This tutorial is an excerpt from the book, Hands-On Networking with Azure, written by Mohamed Waly.  Network Security Groups (NSG) NSG controls the flow of traffic by specifying which traffic is allowed to enter or exit the network. Creating NSG Creating NSG is a pretty straightforward process. To do it, you need to follow these steps: Navigate to Azure portal, and search for network security groups, as shown in the following screenshot: Figure 2.13: Searching for network security groups Once you have clicked on it, a new blade will be opened wherein all the created NSGs are located, as shown in the following screenshot: Figure 2.12: Network security groups blade Click on Add and a new blade will pop up, where you have to specify the following: Name: The name of the NSG Subscription: The subscription, which will be charged for NSG usage Resource group: The resource group within which the NSG will be located as a resource Location: The region where this resource will be created Figure 2.13: Creating an NSG Once you have clicked on Create, the NSG will be created within seconds. Inbound security rules By default, all the subnets and NICs that are not associated with NSG have all the inbound traffic allowed and once they are associated with an NSG, the following inbound security rules are assigned to them as they are a default part of any NSG: AllowVnetInBound: Allows all the inbound traffic that comes from a virtual network AllowAzureLoadBalancerInBound: Allows all the inbound traffic that comes from Load Balancer DenyAllInbound: Denies all the inbound traffic that comes from any source Figure 2.14: Default inbound security rules As shown in the previous screenshot, the rule consists of some properties, such as PRIORITY, NAME, PORT, and so on. It is important to understand what these properties mean for a better understanding of security rules. So, let's go ahead and explain them: PRIORITY: A number assigned to each rule to specify which rule has a higher priority than the other. The lower the number, the higher the priority. You can specify a priority with any number between 100 and 4096. NAME: The name of the rule. The same name cannot be reused within the same network security group. PORT: The allowed port through which the traffic will flow to the network. PROTOCOL: Specify whether the protocol you are using is TCP or UDP. SOURCE and DESTINATION: The source can be any, an IP address range, or a service tag. You can remove the default rules by clicking on Default rules. You can customize your own inbound rules, by following these steps: On the Inbound security rules blade, click on Add. A new blade will pop up, where you have to specify the following: Source: The source can be Any, an IP address range, or a service tag. It specifies the incoming traffic from a specific source IP address range that will be allowed or denied by this rule. Source port ranges: You can provide a single port, such as 80, a port range, such as 1024 - 65535, or a comma-separated list of single ports and/or port ranges, such as 80, 1024 - 65535. This specifies on which ports traffic will be allowed or denied by this rule. Provide an asterisk (*) to allow traffic on any port. Destination: The destination can be Any, an IP address range, or a virtual network. It specifies the outing traffic to a specific destination IP address range that will be allowed or denied by this rule. Destination port ranges: What applies for the source port ranges, applies for the destination port ranges. Protocol: It can be Any, TCP, or UDP. Action: Whether to Allow the rule or to Deny it. Priority: As mentioned earlier, the lower the number, the higher the priority. The priority number must be between 100 - 4096. Name: The name of the rule. Description: The description of the rule, which will help you to differentiate between the rules. In our scenario, I want to allow all the incoming connections to access a website published on a web server located in a virtual network, as shown in the following screenshot: Figure 2.15: Creating an inbound security rule Once you click on OK, the rule will be created. Outbound security rules Outbound security rules are no different than inbound security rules, except inbound rules are meant for inbound traffic and outbound rules are meant for outbound traffic. Otherwise, everything else is similar. Associating the NSG Once you have the NSG created, you can associate it to either an NIC or a subnet. Associating the NSG to an NIC To associate the NSG to an NIC, you need to follow these steps: Navigate to the Network security groups that you have created and then select Network interfaces, as shown in the following screenshot: Figure 2.16: Associated NICs to an NSG Click on Associate. A new blade will pop up, from which you need to select the NIC that you want to associate with the NSG, as shown in the following screenshot: Figure 2.17: NICs to be associated to the NSG Voila! You are done. Associating the NSG to a subnet To associate the NSG to a subnet, you need to follow these steps: Navigate to the Network security groups that you have created and then select Subnets, as shown in the following screenshot: Figure 2.18: Associated subnets to an NSG Click on Associate. A new blade will pop up, where you have to specify the virtual network within which the subnet exists, as shown in the following screenshot: Figure 2.19: Choosing the VNet within which the subnet exists Then, you need to specify which subnet of the VNet you want to associate the NSG to, as shown in the following screenshot: Figure 2.20: Selecting the subnet to which the NSG will be associated Once the subnet is selected, click on OK, and it will take some seconds to get it associated to the NSG. Azure DDoS protection DDoS attacks have spread out lately, by exhausting the application and making it unavailable for use, and you can expect an attack of that type any time. Recently, Microsoft announced the support of Azure DDoS protection as a service for protecting Azure resources, such as Azure VMs, Load Balancers, and Application Gateways. Azure DDoS protection comes in two flavors: Basic: This type has been around for a while as it is already enabled for Azure services to mitigate DDoS attacks. It incurs no charges. Standard: This flavor comes with more enhancements that mitigate attacks, especially for Azure VNet. At the time of writing this book, Azure DDoS protection standard is in preview and it is not available at the portal. You need to request it by filling out a form that is available here. If you found this post useful, do check out the book Hands on Networking with Azure, to design and implement Azure Networking for Azure VMs. Read More The Microsoft Azure Stack Architecture What is Azure API Management?
Read more
  • 0
  • 0
  • 28985

article-image-what-is-multi-layered-software-architecture
Packt Editorial Staff
17 May 2018
7 min read
Save for later

What is a multi layered software architecture?

Packt Editorial Staff
17 May 2018
7 min read
Multi layered software architecture is one of the most popular architectural patterns today. It moderates the increasing complexity of modern applications. It also makes it easier to work in a more agile manner. That's important when you consider the dominance of DevOps and other similar methodologies today. Sometimes called tiered architecture, or n-tier architecture, a multi layered software architecture consists of various layers, each of which corresponds to a different service or integration. Because each layer is separate, making changes to each layer is easier than having to tackle the entire architecture. Let's take a look at how a multi layered software architecture works, and what the advantages and disadvantages of it are. This has been taken from the book Architectural Patterns. Find it here. What does a layered software architecture consist of? Before we get into a multi layered architecture, let's start with the simplest form of layered architecture - three tiered architecture. This is a good place to start because all layered software architecture contains these three elements. These are the foundations: Presentation layer: This is the first and topmost layer which is present in the application. This tier provides presentation services, that is presentation, of content to the end user through GUI. This tier can be accessed through any type of client device like desktop, laptop, tablet, mobile, thin client, and so on. For the content to the displayed to the user, the relevant web pages should be fetched by the web browser or other presentation component which is running in the client device. To present the content, it is essential for this tier to interact with the other tiers that are present preceding it. Application layer: This is the middle tier of this architecture. This is the tier in which the business logic of the application runs. Business logic is the set of rules that are required for running the application as per the guidelines laid down by the organization. The components of this tier typically run on one or more application servers. Data layer: This is the lowest tier of this architecture and is mainly concerned with the storage and retrieval of application data. The application data is typically stored in a database server, file server, or any other device or media that supports data access logic and provides the necessary steps to ensure that only the data is exposed without providing any access to the data storage and retrieval mechanisms. This is done by the data tier by providing an API to the application tier. The provision of this API ensures complete transparency to the data operations which are done in this tier without affecting the application tier. For example, updates or upgrades to the systems in this tier do not affect the application tier of this architecture. The diagram below shows how a simple layered architecture with 3 tiers works:   These three layers are essential. But other layers can be built on top of them. That's when we get into multi layered architecture. It's sometimes called n-tiered architecture because the number of tiers or layers (n) could be anything! It depends on what you need and how much complexity you're able to handle. Multi layered software architecture A multi layered software architecture still has the presentation layer and data layer. It simply splits up and expands the application layer. These additional aspects within the application layer are essentially different services. This means your software should now be more scalable and have extra dimensions of functionality. Of course, the distribution of application code and functions among the various tiers will vary from one architectural design to another, but the concept remains the same. The diagram below illustrates what a multi layered software architecture looks like. As you can see, it's a little more complex that a three-tiered architecture, but it does increase scalability quite significantly: What are the benefits of a layered software architecture? A layered software architecture has a number of benefits - that's why it has become such a popular architectural pattern in recent years. Most importantly, tiered segregation allows you to manage and maintain each layer accordingly. In theory it should greatly simplify the way you manage your software infrastructure. The multi layered approach is particularly good for developing web-scale, production-grade, and cloud-hosted applications very quickly and relatively risk-free. It also makes it easier to update any legacy systems - when you're architecture is broken up into multiple layers, the changes that need to be made should be simpler and less extensive than they might otherwise have to be. When should you use a multi layered software architecture? Clearly, the argument for a multi layered software architecture is pretty clear. However, there are some instances when it is particularly appropriate: If you are building a system in which it is possible to split the application logic into smaller components that could be spread across several servers. This could lead to the design of multiple tiers in the application tier. If the system under consideration requires faster network communications, high reliability, and great performance, then n-tier has the capability to provide that as this architectural pattern is designed to reduce the overhead which is caused by network traffic. An example of a multi layered software architecture We can illustrate the working of an multi layered architecture with the help of an example of a shopping cart web application which is present in all e-commerce sites. The shopping cart web application is used by the e-commerce site user to complete the purchase of items through the e-commerce site. You'd expect the application to have several features that allow the user to: Add selected items to the cart Change the quantity of items in their cart Make payments The client tier, which is present in the shopping cart application, interacts with the end user through a GUI. The client tier also interacts with the application that runs in the application servers present in multiple tiers. Since the shopping cart is a web application, the client tier contains the web browser. The presentation tier present in the shopping cart application displays information related to the services like browsing merchandise, buying them, adding them to the shopping cart, and so on. The presentation tier communicates with other tiers by sending results to the client tier and all other tiers which are present in the network. The presentation tier also makes calls to database stored procedures and web services. All these activities are done with the objective of providing a quick response time to the end user. The presentation tier plays a vital role by acting as a glue which binds the entire shopping cart application together by allowing the functions present in different tiers to communicate with each other and display the outputs to the end user through the web browser. In this multi layered architecture, the business logic which is required for processing activities like calculation of shipping cost and so on are pulled from the application tier to the presentation tier. The application tier also acts as the integration layer and allows the applications to communicate seamlessly with both the data tier and the presentation tier. The last tier which is the data tier is used to maintain data. This layer typically contains database servers. This layer maintains data independent from the application server and the business logic. This approach provides enhanced scalability and performance to the data tier. Read next Microservices and Service Oriented Architecture What is serverless architecture and why should I be interested?
Read more
  • 0
  • 1
  • 107301

article-image-generic-section-single-page-based-website
Savia Lobo
17 May 2018
14 min read
Save for later

How to create a generic reusable section for a single page based website

Savia Lobo
17 May 2018
14 min read
There are countless variations when it comes to different sections that can be incorporated into the design of a single page website. In this tutorial, we will cover how to create a generic section that can be extended to multiple sections. This section provides the ability to display any information your website needs. Single page sections are commonly used to display the following data to the user: Contact form (will be implemented in the next chapter). About us: This can be as simple as a couple of paragraphs talking about the company/individual or more complex with images, even showing the team and their roles. Projects/work: Any work you or the company has done and would like to showcase. They are usually linked to external pages or pop up boxes containing more information about the project. Useful company info such as opening times. These are just some of the many uses for sections in a single page website. A good rule of thumb is that if it can be a page on another website it can most likely be adapted into sections on a single page website. Also, depending on the amount of information a single section has, it could potentially be split into multiple sections. This article is an excerpt taken from the book,' Responsive Web Design by Example', written by Frahaan Hussain. Single page section examples Let's go through some examples of the sections mentioned above. Example 1: Contact form As can be seen, by the contact form from Richman, the elements used are very similar to that of a contact page. A form is used with inputs for the various pieces of information required from the user along with a button for submission: Not all contact forms will have the same fields. Put what you need, it may be more or less, there is no right or wrong answer. Also at the bottom of the section is the company's logo along with some written contact information, which is also very common. Some websites also display a map usually using the Google Maps API; these mainly have a physical presence such as a store. Website link—http://richman-kcm.com/ Example 2: About us This is an excellent example of an about us page that uses the following elements to convey the information: Images: Display the individual's face. Creates a very personal touch to the otherwise digital website. Title: Used to display the individual's name. This can also be an image if you want a fancier title. Simple text: Talks about who the person is and what they do. Icons: Linking to the individual's social media accounts. Website link—http://designedbyfew.com/ Example 3: Projects/work This website shows its work off very elegantly and cleanly using images and little text: It also provides a carousel-like slider to display the work, which is extremely useful for displaying the content bigger without displaying all of it at once and it allows a lot of content for a small section to be used. Website link: http://peeltheorange.com/#recent-work Example 4: Opening times This website uses a background image similar to the introduction section created in the previous chapter and an additional image on top to display the opening times. This can also be achieved using a mixture of text and CSS styling for various facets such as the border. Website link—http://www.mumbaigate.co.uk/ Implementing generic reusable single page section We will now create a generic section that can easily be modified and reused to our single page portfolio website. But we still need some sort of layout/design in mind before we implement the section, let's go with an Our Team style section. What will the Our Team section contain? The Our Team section will be a bit simpler, but it can easily be modified to accommodate the animations and styles displayed on the previously mentioned websites. It will be similar to the following example: Website link—http://demo.themeum.com/html/oxygen/ The preceding example consists of the following elements: Heading Intro text (Lorem Ipsum in this case) Images displaying each member of the team Team member's name Their role Text informing the viewer a little bit about them Social links We will also create our section using a similar layout. We are now finally going to use the column system to its full potential to provide a responsive experience using breakpoints. Creating the Our Team section container First let's implement a simple container, with the title and section introduction text, without any extra elements such as an image. We will then use this to link to our navigation bar. Add the following code to the jumbotron div: Let's go over what the preceding code is doing: Line 9 creates a container that is fluid, allowing it to span the browser's width fully. This can be changed to a regular container if you like. The id will be used very soon to link to the navigation bar. Line 10 creates a row in which our text elements will be stored. Line 11 creates a div that spans all the 12 columns on all screen sizes and centers the text inside of it. Line 12 creates a simple header for the Team section. Line 14 to Line 16 adds introduction text. I have put the first two sentences of "Lorem Ipsum..." inside of it, but you can put anything you like. All of this produces the following result: Anchoring the Team section to the navigation bar We will now link the navigation bar to the Team section. This will allow the user to navigate to the Team section without having to scroll up or down. At the moment, there is no need to scroll up, but when more content is added this can become a problem as a single page website can become quite long. Fortunately, we have already done the heavy lifting with the navigation bar through HTML and JavaScript, phew! First, let's change the name of the second button in the navigation bar to Team. Update the navigation bar like so: The navigation bar will now look as follows: Fantastic, our navigation bar is looking more like what you would see on a real website. Now let's change href to the same ID as the Team section, which was #TeamSection like so: Now when we click on any of the navigation buttons we get no JavaScript errors like we would have in the previous chapter. Also, it automatically scrolls to each section without any extra JavaScript code. Adding team pictures Now let's use images to showcase the team members. I will use the image from the following link for our employees, but in a real website you would obviously use different images: http://res.cloudinary.com/dmliyxggm/image/upload/v1511699813/John_vepwoz.png I have modified the image so all the background is removed and the image is trimmed, so it looks as follows: Up until now, all images that we have used have been stored on other websites such as CDN's, this is great, but the need may arise when the use of a custom image like the previous one is needed. We can either store it on a CDN, which is a very good approach, and I would recommend Cloudinary (http://cloudinary.com/), or we can store it locally, which we will do now. A CDN is a Content Delivery Network that has a sole purpose of delivering content such as images to other websites using the best and fastest servers available to a specific user. I would definitely recommend using one. Create a folder called Images and place the image using the following folder structure: Root CSS Images Team Thumbnails Thumbnails.png Index.php JS SNIPPETS This may seem like overkill, considering we only have one image, but as your website gets more complex you will store more images and having an intelligent folder structure/hierarchy will save an immense amount of time. Add the following code to the first row like so: The code we have added doesn't actually provide any visual changes as it is nothing but empty div classes. But these div classes will serve as structures for each team member and their respective content such as name and social links. We created a new row to group our new div classes. Inside each div we will represent each team member. The classes have been set up to be displayed like so: Extra small screens will only show a single team member on a single row Small and medium screens will show two team members on a single row Large and extra large screens will show four team members on a single row The rows are rows in their literal sense and not the class row. Another way to look at them is as lines. The sizes/breakpoints can easily be changed using the information regarding the grid from this article What Is Bootstrap, Why Do We Use It? Now let's add the team's images, update the previous code like so: The preceding code produces the following result: As you can see, this is not the desired effect we were looking for. As there are no size restrictions on the image, it is displayed at its original size. Which, on some screens, will produce a result similar to the monstrosity you saw before; worry not, this can easily be fixed. Add the classes img-fluid and img-thumbnail to each one of the images like so: The classes we added are designed to provide the following styling: img-fluid: Provides a responsive image that is automatically restricted based on the number of columns and browser size. img-thumbnail: Is more of an optional class, but it is still very useful. It provides a light border around the images to make them pop. This produces the following result: As it can be seen, this is significantly better than our previous result. Depending on the browser/screen size, the positioning will slightly change based on the column breakpoints we specified. As usual, I recommend that you resize the browser to see the different layouts. These images are almost complete; they look fine on most screen sizes, but they aren't actually centered within their respective div. This is evident in larger screen sizes, as can be seen here: It isn't very noticeable, but the problem is there, it can be seen to the right of the last image. You probably could get away without fixing this, but when creating anything, from a website to a game, or even a table, the smallest details are what separate the good websites from the amazing websites. This is a simple idea called the aggregation of marginal gains. Fortunately for us, like many times before, Bootstrap offers functionality to resolve our little problem. Simply add the text-center class, to the row within the div of the images like so:   This now produces the following result: There is one more slight problem that is only noticeable on smaller screens when the images/member containers are stacked on top of each other. The following result is produced: The problem might not jump out at first glance, but look closely at the gaps between the images that are stacked, or I should say, to the lack of a gap. This isn't the end of the world, but again the small details make an immense difference to the look of a website. This can be easily fixed by adding padding to each team member div. First, add a class of teamMemberContainer to each team member div like so: Add the following CSS code to the index.css file to provide a more visible gap through the use of padding: This simple solution now produces the following result: If you want the gap to be bigger, simply increase the value and lower it to reduce the gap. Team member info text The previous section covered quite a lot if you're not 100% on what we did just go back and take a second look. This section will thankfully be very simple as it will incorporate techniques and features we have already covered, to add the following information to each team member: Name Job title Member info text Plus anything else you need Update each team member container with the following code: Let's go over the new code line by line: Line 24 adds a simple header that is intended to display the team member's name. I have chosen an h4 tag, but you can use something bigger or smaller if you like. Line 26 adds the team member's job title, I have used a paragraph element with the Bootstrap class text-muted, which lightens the text color. If you would like more information regarding text styling within Bootstrap, feel free to check out the following link. Line 28 adds a simple paragraph with no extra styling to display some information about the team member. Bootstrap text styling link—https://v4-alpha.getbootstrap.com/utilities/colors/ The code that we just added will produce the following result: As usual, resize your browser to simulate different screen sizes. I use Chrome as my main browser, but Safari has an awesome feature baked right in that allows you to see how your website will run on different browsers/devices, this link will help you use this feature—https://www.tekrevue.com/tip/safari-responsive-design-mode/ Most browsers have a plethora of plugins to aid in this process, but not only does Safari have it built in, it works really well. It all looks fantastic, but again I will nitpick at the gaps. The image is right on top of the team member name text; a small gap would really help improve the visual fidelity. Add a class of teamMemberImage to each image tag as it is demonstrated here: Now add the following code to the index.css file, which will apply a margin of 10px below the image, hence moving all the content down:  Change the margin to suit your needs. This very simple code will produce the following similar yet subtly different and more visually appealing result: Team member social links We have almost completed the Team section, only the social links remain for each team member. I will be using simple images for the social buttons from the following link: https://simplesharebuttons.com/html-share-buttons/ I will also only be adding three social icons, but feel free to add as many or as few as you need. Add the following code to the button of each team member container: Let's go over each new line of code: Line 30 creates a div to store all the social buttons for each team member Line 31 creates a link to Facebook (add your social link in the href) Line 32 adds an image to show the Facebook social link Line 35 creates a link to Google+ (add your social link in the href) Line 36 adds an image to show the Google+ social link Line 39 creates a link to Twitter (add your social link in the href) Line 40 adds an image to show the Twitter social link We have added a class that needs to be implemented, but let's first run our website to see the result without any styling: It looks OK, but the social icons are a bit big, especially if we were to have more icons. Add the following CSS styling to the index.css file: This piece of code simply restricts the social icons size to 50px. Only setting the width causes the height to be automatically calculated, this ensures that any changes to the image that involves a ratio change won't mess up the look of the icons. This now produces the following result: Feel free to change the width to suit your desires. With the social buttons implemented, we are done. If you've enjoyed this tutorial, check out Responsive Web Design by Example, to create a cool blog page, beautiful portfolio site, or professional business site to make them all totally responsive. 5 things to consider when developing an eCommerce website What UX designers can teach Machine Learning Engineers? To start with: Model Interpretability
Read more
  • 0
  • 0
  • 23743

article-image-awk-programming-langauge
Pavan Ramchandani
17 May 2018
9 min read
Save for later

That '70s language: AWK programming

Pavan Ramchandani
17 May 2018
9 min read
AWK is an interpreted programming language designed for text processing and report generation. It is typically used for data manipulation, such as searching for items within data, performing arithmetic operations, and restructuring raw data for generating reports in most Unix-like operating systems. Today, we will explore the AWK philosophy and different types of AWK that exist, starting from its original implementation in 1977 at AT&T's Laboratories, Inc. We will also look at the various implementation areas of AWK in data science today. Using AWK programs, one can handle repetitive text-editing problems with very simple and short programs. It is a pattern-action language; it searches for patterns in a given input and, when a match is found, it performs the corresponding action. The pattern can be made of strings, regular expressions, comparison operations on numbers, fields, variables, and so on. It reads the input files and splits each input line of the file into fields automatically. AWK has most of the well-designed features that every programming language should contain. Its syntax particularly resembles that of the C programming language. It is named after its original three authors: Alfred V. Aho Peter J. Weinberger Brian W. Kernighan AWK is a very powerful, elegant, and simple that every person dealing with text processing should be familiar with. This article is an excerpt from a book written by Shiwang Kalkhanda, titled Learning AWK Programming. This book will introduce you to AWK programming language and get you hands-on working with practical implementation of AWK. Types of AWK The AWK language was originally implemented as an AWK utility on Unix. Today, most Linux distributions provide GNU implementation of AWK (GAWK), and a symlink for AWK is created from the original GAWK binary. The AWK utility can be categorized into the following three types, depending upon the type of interpreter it uses for executing AWK programs: AWK: This is the original AWK interpreter available from AT&T Laboratories. However, it is not used much nowadays and hence it might not be well-maintained. Its limitation is that it splits a line into a maximum 99 fields. It was updated and replaced in the mid-1980s with an enhanced version called New AWK (NAWK). NAWK: This is AT&T's latest development on the AWK interpreter. It is well-maintained by one of the original authors of AWK - Dr. Brian W. Kernighan. GAWK: This is the GNU project's implementation of the AWK programming language. All GNU/Linux distributions are shipped with GAWK by default and hence it is the most popular version of AWK. GAWK interpreter is fully compatible with AWK and NAWK. Beyond these, we also have other, less popular, AWK interpreters and translators, mentioned as follows. These variants are useful in operations when you want to translate your AWK program to C, C++, or Perl: MAWK: Michael Brennan interpreter for AWK. TAWK: Thompson Automation interpreter/compiler/Microsoft Windows DLL for AWK. MKSAWK: Mortice Kern Systems interpreter/compiler/for AWK. AWKCC: An AWK translator to C (might not be well-maintained). AWKC++: Brian Kernighan's AWK translator to C++ (experimental). It can be downloaded from: https://9p.io/cm/cs/who/bwk/awkc++.ps. AWK2C: An AWK translator to C. It uses GNU AWK libraries extensively. A2P: An AWK translator to Perl. It comes with Perl. AWKA: Yet another AWK translator to C (comes with the library), based on MAWK. It can be downloaded from: http://awka.sourceforge.net/download.html. When and where to use AWK AWK is simpler than any other utility for text processing and is available as the default on Unix-like operating systems. However, some people might say Perl is a superior choice for text processing, as AWK is functionally a subset of Perl, but the learning curve for Perl is steeper than that of AWK; AWK is simpler than Perl. AWK programs are smaller and hence quicker to execute. Anybody who knows the Linux command line can start writing AWK programs in no time. Here are a few use cases of AWK: Text processing Producing formatted text reports/labels Performing arithmetic operations on fields of a file Performing string operations on different fields of a file Programs written in AWK are smaller than they would be in other higher-level languages for similar text processing operations. AWK programs are interpreted on a GNU/Linux Terminal and thus avoid the compiling, debugging phase of software development in other languages. Getting started with installation This section describes how to set up the AWK environment on your GNU/Linux system, and we'll also discuss the workflow of AWK. Then, we'll look at different methods for executing AWK programs. Installation on Linux Generally, AWK is installed by default on most GNU/Linux distributions. Using the which command, you can check whether it is installed on your system or not. In case AWK is not installed on your system, you can do so in one of two ways: Using the package manager of the corresponding GNU/Linux system Compiling from the source code Let's take a look at each method in detail in the following sections. Using the package manager Different flavors of GNU/Linux distribution have different package-management utilities. If you are using a Debian-based GNU/Linux distribution, such as Ubuntu, Mint, or Debian, then you can install it using the Advance Package Tool (APT) package manager, as follows: [ shiwang@linux ~ ] $ sudo apt-get update -y [ shiwang@linux ~ ] $ sudo apt-get install gawk -y Similarly, to install AWK on an RPM-based GNU/Linux distribution, such as Fedora, CentOS, or RHEL, you can use the Yellowdog Updator Modified (YUM) package manager, as follows: [ root@linux ~ ] # yum update -y [ root@linux ~ ] # yum install gawk -y For installation of AWK on openSUSE, you can use the zypper (zypper command line) package-management utility, as follows: [ root@linux ~ ] # zypper update -y [ root@linux ~ ] # zypper install gawk -y Once the installation is finished, make sure AWK is accessible through the command line. We can check that using the which command, which will return the absolute path of AWK on our system: [ root@linux ~ ] # which awk /usr/bin/awk You can also use awk --version to find the AWK version on our system: [ root@linux ~ ] # awk --version Compiling from the source code Like every other open source utility, the GNU AWK source code is freely available for download as part of the GNU project. Previously, you saw how to install AWK using the package manager; now, you will see how to install AWK by compiling from its source code on the GNU/Linux distribution. The following steps are applicable to most of the GNU/Linux software for installation: Download the source code from a GNU project ftp site. Here, we will use the wget command line utility to download it, however you are free to choose any other program, such as curl, you feel comfortable with: [ shiwang@linux ~ ] $ wget http://ftp.gnu.org/gnu/gawk/gawk-4.1.3.tar.xz Extract the downloaded source code: [ shiwang@linux ~ ] $ tar xvf gawk-4.1.3.tar.xz Change your working directory and execute the configure file to configure the GAWK as per the working environment of your system: [ shiwang@linux ~ ] $ cd gawk-4.1.3 && ./configure Once the configure command completes its execution successfully, it will generate the make file. Now, compile the source code by executing the make command: [ shiwang@linux ~ ] $ make Type make install to install the programs and any data files and documentation. When installing into a prefix owned by root, it is recommended that the package be configured and built as a regular user, and only the make install phase is executed with root privileges: [ shiwang@linux ~ ] $ sudo make install Upon successful execution of these five steps, you have compiled and installed AWK on your GNU/Linux distribution. You can verify this by executing the which awk command in the Terminal or awk --version: [ root@linux ~ ] # which awk /usr/bin/awk Now you have a working AWK/GAWK installation and we are ready to begin AWK programming, but before that, our next section describes the workflow of the AWK interpreter. If you are running on macOS X, AWK, and not GAWK, would be installed as a default on it. For GAWK installation on macOS X, please refer to MacPorts for GAWK. Workflow of AWK Having a basic knowledge of the AWK interpreter workflow will help you to better understand AWK and will result in more efficient AWK program development. Hence, before getting your hands dirty with AWK programming, you need to understand its internals. The AWK workflow can be summarized as shown in the following figure: Let's take a look at each operation: READ OPERATION: AWK reads a line from the input stream (file, pipe, or stdin) and stores it in memory. It works on text input, which can be a file, the standard input stream, or from a pipe, which it further splits into records and fields: Records: An AWK record is a single, continuous data input that AWK works on. Records are bounded by a record separator, whose value is stored in the RS variable. The default value of RS is set to a newline character. So, the lines of input are considered records for the AWK interpreter. Records are read continuously until the end of the input is reached. Figure 1.2  shows how input data is broken into records and then goes further into how it is split into fields: Fields: Each record can further be broken down into individual chunks called fields. Like records, fields are bounded. The default field separator is any amount of whitespace, including tab and space characters. So by default, lines of input are further broken down into individual words separated by whitespace. You can refer to the fields of a record by a field number, beginning with 1. The last field in each record can be accessed by its number or with the NF special variable, which contains the number of fields in the current record, as shown in Figure 1.3: EXECUTE OPERATION: All AWK commands are applied sequentially on the input (records and fields). By default, AWK executes commands on each record/line. This behavior of AWK can be restricted by the use of patterns. REPEAT OPERATION: The process of read and execute is repeated until the end of the file is reached. The following flowchart depicts the workflow:   We introduced you to the AWK programming language and got ourselves a quick primer to get started with application development. If you found this post is useful, do check out the book Learning AWK Programming to learn more about the intricacies of AWK programming language for text processing. The oldest programming languages in use today What is the difference between functional and object oriented programming? Systems programming with Go in UNIX and Linux  
Read more
  • 0
  • 0
  • 20184

article-image-arrays-lists-dictionaries-unity-3d-game-development
Amarabha Banerjee
16 May 2018
14 min read
Save for later

How to use arrays, lists, and dictionaries in Unity for 3D game development

Amarabha Banerjee
16 May 2018
14 min read
A key ingredient in scripting 3D games with Unity is the ability to work with C# to create arrays, lists, objects and dictionaries within the Unity platform. In this tutorial, we help you to get started with creating arrays, lists, and dictionaries effectively. This article is an excerpt from Learning C# by Developing Games with Unity 2017. Read more here. You can also read the latest edition of the book here.  An array stores a sequential collection of values of the same type, in the simplest terms. We can use arrays to store lists of values in a single variable. Imagine we want to store a number of student names. Simple! Just create a few variables and name them student1, student2, and so on: public string student1 = "Greg"; public string student2 = "Kate"; public string student3 = "Adam"; public string student4 = "Mia"; There's nothing wrong with this. We can print and assign new values to them. The problem starts when you don't know how many student names you will be storing. The name variable suggests that it's a changing element. There is a much cleaner way of storing lists of data. Let's store the same names using a C# array variable type: public string[ ] familyMembers = new string[ ]{"Greg", "Kate", "Adam", "Mia"} ; As you can see, all the preceding values are stored in a single variable called familyMembers. Declaring an array To declare a C# array, you must first say what type of data will be stored in the array. As you can see in the preceding example, we are storing strings of characters. After the type, we have an open square bracket and then immediately a closed square bracket, [ ]. This will make the variable an actual array. We also need to declare the size of the array. It simply means how many places are there in our variable to be accessed. The minimum code required to declare a variable looks similar to this: public string[] myArrayName = new string[4]; The array size is set during assignment. As you have learned before, all code after the variable declaration and the equal sign is an assignment. To assign empty values to all places in the array, simply write the new keyword followed by the type, an open square bracket, a number describing the size of the array, and then a closed square bracket. If you feel confused, give yourself a bit more time. Then you will fully understand why arrays are helpful. Take a look at the following examples of arrays; don't worry about testing how they work yet: string[ ] familyMembers = new string[]{"John", "Amanda", "Chris", "Amber"} ; string[ ] carsInTheGarage = new string[] {"VWPassat", "BMW"} ; int[ ] doorNumbersOnMyStreet = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 }; GameObject[ ] carsInTheScene = GameObject.FindGameObjectsWithTag("car"); As you can see, we can store different types of data as long as the elements in the array are of the same type. You are probably wondering why the last example, shown here, looks different: GameObject[ ] carsInTheScene = GameObject.FindGameObjectsWithTag("car"); In fact, we are just declaring the new array variable to store a collection of GameObject in the scene using the "car" tag. Jump into the Unity scripting documentation and search for GameObject.FindGameObjectsWithTag: As you can see, GameObject.FindGameObjectsWithTag is a special built-in Unity function that takes a string parameter (tag) and returns an array of GameObjects using this tag. Storing items in the List Using a List instead of an array can be so easier to work with in a script. Look at some forum sites related to C# and Unity, and you'll discover that plenty of programmers simply don't use an array unless they have to; they prefer to use a List. It is up to the developer's preference and task. Let's stick to lists for now. Here are the basics of why a List is better and easier to use than an array: An array is of fixed size and unchangeable The size of a List is adjustable You can easily add and remove elements from a List To mimic adding a new element to an array, we would need to create a whole new array with the desired number of elements and then copy the old elements The first thing to understand is that a List has the ability to store any type of object, just like an array. Also, like an array, we must specify which type of object we want a particular List to store. This means that if you want a List of integers of the int type then you can create a List that will store only the int type. Let's go back to the first array example and store the same data in a List. To use a List in C#, you need to add the following line at the beginning of your script: using System.Collections.Generic; As you can see, using Lists is slightly different from using arrays. Line 9 is a declaration and assignment of the familyMembers List. When declaring the list, there is a requirement for a type of objects that you will be storing in the List. Simply write the type between the < > characters. In this case, we are using string. As we are adding the actual elements later in lines 14 to 17, instead of assigning elements in the declaration line, we need to assign an empty List to be stored temporarily in the familyMembers variable. Confused? If so, just take a look at the right-hand side of the equal sign on line 9. This is how you create a new instance of the List for a given type, string for this example: new List<string>(); Lines 14 to 17 are very simple to understand. Each line adds an object at the end of the List, passing the string value in the parentheses. In various documentation, Lists of type look like this: List< T >. Here, T stands for the type of data. This simply means that you can insert any type in place of T and the List will become a list of that specific type. From now on, we will be using it. Common operations with Lists List<T> is very easy to use. There is a huge list of different operations that you can perform with it. We have already spoken about adding an element at the end of a List. Very briefly, let's look at the common ones that we will be possibly using at later stages: Add: This adds an object at the end of List<T>. Remove: This removes the first occurrence of a specific object from List<T>. Clear: This removes all elements from List<T>. Contains: This determines whether an element is in List<T> or not. It is very useful to check whether an element is stored in the list. Insert: This inserts an element into List<T> at the specified index. ToArray: This copies the elements of List<T> to a new array. You don't need to understand all of these at this stage. All I want you to know is that there are many out-of-the-box operations that you can use. If you want to see them all, I encourage you to dive into the C# documentation and search for the List<T> class. List <T> versus arrays Now you are probably thinking, "Okay, which one should I use?" There isn't a general rule for this. Arrays and List<T> can serve the same purpose. You can find a lot of additional information online to convince you to use one or the other. Arrays are generally faster. For what we are doing at this stage, we don't need to worry about processing speeds. Some time from now, however, you might need a bit more speed if your game slows down, so this is good to remember. List<T> offers great flexibility. You don't need to know the size of the list during declaration. There is a massive list of out-of-the-box operations that you can use with List, so it is my recommendation. Array is faster, List<T> is more flexible. Retrieving the data from the Array or List<T> Declaring and storing data in the array or list is very clear to us now. The next thing to learn is how to get stored elements from an array. To get a stored element from the array, write an array variable name followed by square brackets. You must write an int value within the brackets. That value is called an index. The index is simply a position in the array. So, to get the first element stored in the array, we will write the following code: myArray[0]; Unity will return the data stored in the first place in myArray. It works exactly the same way as the return type methods. So, if myArray stores a string value on index 0, that string will be returned to the place where you are calling it. Complex? It's not. Let's show you by example. The index value starts at 0, no 1, so the first element in an array containing 10 elements will be accessible through an index value of 0 and last one through a value of 9. Let's extend the familyMembers example: I want to talk about line 20. The rest of it is pretty obvious for you, isn't it? Line 20 creates a new variable called thirdFamilyMember and assigns the third value stored in the familyMembers list. We are using an index value of 2 instead of 3 because in programming counting starts at 0. Try to memorize this; it is a common mistake made by beginners in programming. Go ahead and click Play. You will see the name Adam being printed in the Unity Console. While accessing objects stored in an array, make sure you use an index value between zero and the size of the array. In simpler words, we cannot access data from index 10 in an array that contains only four objects. Makes sense? Checking the size This is very common; we need to check the size of the array or list. There is a slight difference between a C# array and List<T>. To get the size as an integer value, we write the name of the variable, then a dot, and then Length of an array or Count for List<T>: arrayName.Length: This returns an integer value with the size of the array listName.Count: This returns an integer value with the size of the list As we need to focus on one of the choices here and move on, from now on we will be using List<T>. ArrayList We definitely know how to use lists now. We also know how to declare a new list and add, remove, and retrieve elements. Moreover, you have learned that the data stored in List<T> must be of the same type across all elements. Let's throw a little curveball. ArrayList is basically List<T> without a specified type of data. This means that we can store whatever objects we want. Storing elements of different types is also possible. ArrayList is very flexible. Take a look at the following example to understand what ArrayList can look like: You have probably noticed that ArrayList also supports all common operations, such as .Add(). Lines 12 to 15 add different elements into the array. The first two are of the integer type, the third is a string type, and the last one is a GameObject. All mixed types of elements in one variable! When using ArrayList, you might need to check what type of element is under a specific index to know how to treat it in code. Unity provides a very useful function that you can use on virtually any type of object. Its GetType() method returns the type of the object, not the value. We are using it in lines 18 and 19 to print the types of the second and third elements. Go ahead, write the preceding code, and click Play. You should get the following output in the Console window: Dictionaries When we talk about collection data, we need to mention Dictionaries. A Dictionary is similar to a List. However, instead of accessing a certain element by index value, we use a string called key. The Dictionary that you will probably be using the most often is called Hashtable. Feel free to dive into the C# documentation after reading this chapter to discover all the bits of this powerful class. Here are a few key properties of Hashtable: Hashtable can be resized dynamically, like List<T> and ArrayList Hashtable can store multiple data types at the same type, like ArrayList A public member Hashtable isn't visible in the Unity Inspector panel due to default inspector limitations I want to make sure that you won't feel confused, so I will go straight to a simple example: Accessing values To access a specific key in the Hashtable, you must know the string key the value is stored under. Remember, the key is the first value in the brackets when adding an element to Hashtable. Ideally, you should also know the type of data you are trying to access. In most cases, that would not be an issue. Take a look at this line: Debug.Log((string)personalDetails["firstName"]); We already know that using Debug.Log serves to display a message on the Unity console, so what are we trying to display? A string value (it's one that can contain letters and numbers), then we specify where that value is stored. In this case, the information is stored under Hashtable personalDetails and the content that we want to display is firstName. Now take a look at the script once again and see if you can display the age, remember that the value that we are trying to access here is a number, so we should use int instead of string: Similar to ArrayList, we can store mixed-type data in Hashtable. Unity requires the developer to specify how an accessed element should be treated. To do this, we need to cast the element into a specific data type. The syntax is very simple. There are brackets with the data type inside, followed by the Hashtable variable name. Then, in square brackets, we have to enter the key string the value is stored under. Ufff, confusing! As you can see in the preceding line, we are casting to string (inside brackets). If we were to access another type of data, for example, an integer number, the syntax would look like this: (int)personalDetails["age"]; I hope that this is clear now. If it isn't, why not search for more examples on the Unity forums? How do I know what's inside my Hashtable? Hashtable, by default, isn't displayed in the Unity Inspector panel. You cannot simply look at the Inspector tab and preview all keys and values in your public member Hashtable. We can do this in code, however. You know how to access a value and cast it. What if you are trying to access the value under a key that isn't stored in the Hashtable? Unity will spit out a null reference error and your program is likely to crash. To check whether an element exists in the Hashtable, we can use the .Contains(object) method, passing the key parameter: This determines whether the array contains the item and if so, the code will continue; otherwise, it will stop there, preventing any error. We discussed how to use C# to create arrays, lists, dictionaries and objects in Unity. The code samples and the examples will help you implement these from the platform. Do check out this book Learning C# by Developing Games with Unity 2017  to develop your first interactive 2D and 3D platform game. Read More Unity 2D & 3D game kits simplify Unity game development for beginners How to create non-player Characters (NPC) with Unity 2018 Game Engine Wars: Unity vs Unreal Engine
Read more
  • 0
  • 4
  • 194375

article-image-getting-started-with-google-data-studio-an-intuitive-tool-for-visualizing-bigquery-data
Sugandha Lahoti
16 May 2018
8 min read
Save for later

Getting started with Google Data Studio: An intuitive tool for visualizing BigQuery Data

Sugandha Lahoti
16 May 2018
8 min read
Google Data Studio is one of the most popular tools for visualizing data. It can be used to pull data directly out of Google's suite of marketing tools, including Google Analytics, Google AdWords, and Google Search Console. It also supports connectors for database tools such as PostgreSQL and BigQuery, it can be accessed at datastudio.google.com. In this article, we will learn to visualize BigQuery Data with Google Data Studio. [box type="note" align="" class="" width=""]This article is an excerpt from the book, Learning Google BigQuery, written by Thirukkumaran Haridass and Eric Brown. This book will serve as a comprehensive guide to mastering BigQuery, and utilizing it to get useful insights from your Big Data.[/box] The following steps explain how to get started in Google Data Studio and access BigQuery data from Data Studio: Setting up an account: Account setup is extremely easy for Data Studio. Any user with a Google account is eligible to use all Data Studio features for free: Accessing BigQuery data: Once logged in, the next step is to connect to BigQuery. This can be done by clicking on the DATA SOURCES button on the left-hand-side navigation: You'll be prompted to create a data source by clicking on the large plus sign to the bottom-right of the screen. On the right-hand-side navigation, you'll get a list of all of the connectors available to you. Select BigQuery: At this point, you'll be prompted to select from your projects, shared projects, a custom query, or public datasets. Since you are querying the Google Analytics BigQuery Export test data, select Custom Query. Select the project you would like to use. In the Enter Custom Query prompt, add this query and click on the Connect button on the top right: SELECT trafficsource.medium as Medium, COUNT(visitId) as Visits FROM `google.com:analytics- bigquery.LondonCycleHelmet.ga_sessions_20130910` GROUP BY Medium This query will pull the count of sessions for traffic source mediums for the Google Analytics account that has been exported. The next screen shows the schema of the data source you have created. Here, you can make changes to each field of your data, such as changing text fields to date fields or creating calculated metrics: Click on Create Report. Then click on Add to Report. At this point, you will land on your report dashboard. Here, you can begin to create charts using the data you've just pulled from BigQuery. Icons for all the chart types available are shown near the top of the page. Hover over the chart types and click on the chart labeled Bar Chart; then in the grid, hold your right-click button to draw a rectangle. A bar chart should appear, with the Traffic Source Medium and Visit data from the query you ran: A properties prompt should also show on the right-hand side of the page: Here, a number of properties can be selected for your chart, including the dimension, metric, and many style settings. Once you've completed your first chart, more charts can be added to a single page to show other metrics if needed. For many situations, a single bar graph will answer the question at hand. Some situations may require more exploration. In such cases, an analyst might want to know whether the visit metric influences other metrics such as the number of transactions. A scatterplot with visits on the x axis and transactions on the y axis can be used to easily visualize this relationship. Making a scatterplot in Data Studio The following steps show how to make a scatterplot in Data Studio with the data from BigQuery: Update the original query by adding the transaction metric. In the edit screen of your report, click on the bar chart to bring up the chart options on the right-hand- side navigation. Click on the pencil icon next to the data source titled BigQuery to edit the data source. Click on the left-hand-side arrow icon titled Edit Connection: 3. In the dialog titled Enter Custom Query, add this query: SELECT trafficsource.medium as Medium, COUNT(visitId) as Visits, SUM(totals.transactions) AS Transactions FROM `google.com:analytics- bigquery.LondonCycleHelmet.ga_sessions_20130910` GROUP BY Medium Click on the button titled Reconnect in order to reprocess the query. A prompt should emerge, asking whether you'd like to add a new field titled Transactions. Click on Apply. Click on Done. Once you return to the report edit screen, click on the Scatter Chart button() and use your mouse to draw a square in the report space: The report should autoselect the two metrics you've created. Click on the chart to bring up the chart edit screen on the right-hand-side navigation; then click on the Style tab. Click on the dropdown under the Trendline option and select Linear to add a linear trend line, also known as linear regression line. The graph will default to blue, so use the pencil icon on the right to select red as the line color: Making a map in Data Studio Data Studio includes a map chart type that can be used to create simple maps. In order to create maps, a map dimension will need to be included in your data, along with a metric. Here, we will use the Google BigQuery public dataset for Medicare data. You'll need to create a new data source: Accessing BigQuery data: Once logged in, the next step is to connect to BigQuery. This can be done by clicking on the DATA SOURCES button on the left-hand-side navigation. You'll be prompted to create a data source by clicking on the large plus sign to the bottom-right of the screen. On the right-hand-side navigation, you'll get a list of all of the connectors available to you. Select BigQuery. At this point, you'll be prompted to select from your projects, shared projects, a custom query, or public datasets. Since you are querying the Google Analytics BigQuery Export test data, select Custom Query. Select the project you would like to use. In the Enter Custom Query prompt, add this query and click on the Connect button on the top right: SELECT CONCAT(provider_city,", ",provider_state) city, AVG(average_estimated_submitted_charges) avg_sub_charges FROM `bigquery-public-data.medicare.outpatient_charges_2014` WHERE apc = '0267 - Level III Diagnostic and Screening Ultrasound' GROUP BY 1 ORDER BY 2 desc This query will pull the average of submitted charges for diagnostic ultrasounds by city in the United States. This is the most submitted charge in the 2014 Medicaid data. The next screen shows the schema of the data source you have created. Here, you can make changes to each field of your data, such as changing text fields to date fields or creating calculated metrics: Click on Create Report. Then click on Add to Report. At this point, you will land on your report dashboard. Here, you can begin to create charts using the data you've just pulled from BigQuery. Icons for all the chart types available are shown near the top of the page. Hover over the chart types and click on the chart labeled Map  Chart; then in the grid, hold your right-click button to draw a rectangle. Click on the chart to bring up the Dimension Picker on the right-hand-side navigation, and click on Create New Dimension: Right click on the City dimension and select the Geo type and City subtype. Here, we can also choose other sub-types (Latitude, Longitude, Metro, Country, and so on). Data Studio will plot the top 500 rows of data (in this case, the top 500 cities in the results set). Hovering over each city brings up detailed data: Data Studio can also be used to roll up geographic data. In this case, we'll roll city data up to state data. From the edit screen, click on the map to bring up the Dimension Picker and click on Create New Dimension in the right-hand-side navigation. Right-click on the City dimension and select the Geo type and Region subtype. Google uses the term Region to signify states: Once completed, the map will be rolled up to the state level instead of the city level. This functionality is very handy when data has not been rolled up prior to being inserted into BigQuery: Other features of Data Studio Filtering: Filtering can be added to your visualizations based on dimensions or metrics as long as the data is available in the data source Data joins: Data for multiple sources can be joined to create new, calculated metrics Turnkey integrations with many Google Marketing Suite tools such as Adwords and Search Console We explored various features of Google Data Studio and learnt to use them for visualizing BigQuery data.To know about other third party tools for reporting and visualization purpose such as R and Tableau, check out the book Learning Google BigQuery. Getting Started with Data Storytelling What is Seaborn and why should you use it for data visualization? Pandas is an effective tool to explore and analyze data - Interview Insights
Read more
  • 0
  • 2
  • 38941
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-work-with-classes-in-typescript
Amey Varangaonkar
15 May 2018
8 min read
Save for later

How to work with classes in Typescript

Amey Varangaonkar
15 May 2018
8 min read
If we are developing any application using TypeScript, be it a small-scale or a large-scale application, we will use classes to manage our properties and methods. Prior to ES 2015, JavaScript did not have the concept of classes, and we used functions to create class-like behavior. TypeScript introduced classes as part of its initial release, and now we have classes in ES6 as well. The behavior of classes in TypeScript and JavaScript ES6 closely relates to the behavior of any object-oriented language that you might have worked on, such as C#. This excerpt is taken from the book TypeScript 2.x By Example written by Sachin Ohri. Object-oriented programming in TypeScript Object-oriented programming allows us to represent our code in the form of objects, which themselves are instances of classes holding properties and methods. Classes form the container of related properties and their behavior. Modeling our code in the form of classes allows us to achieve various features of object-oriented programming, which helps us write more intuitive, reusable, and robust code. Features such as encapsulation, polymorphism, and inheritance are the result of implementing classes. TypeScript, with its implementation of classes and interfaces, allows us to write code in an object-oriented fashion. This allows developers coming from traditional languages, such as Java and C#, feel right at home when learning TypeScript. Understanding classes Prior to ES 2015, JavaScript developers did not have any concept of classes; the best way they could replicate the behavior of classes was with functions. The function provides a mechanism to group together related properties and methods. The methods can be either added internally to the function or using the prototype keyword. The following is an example of such a function: function Name (firstName, lastName) { this.firstName = firstName; this.lastName = lastName; this.fullName = function() { return this.firstName + ' ' + this.lastName ; }; } In this preceding example, we have the fullName method encapsulated inside the Name function. Another way of adding methods to functions is shown in the following code snippet with the prototype keyword: function Name (firstName, lastName) { this.firstName = firstName; this.lastName = lastName; } Name.prototype.fullName = function() { return this.firstName + ' ' + this.lastName ; }; These features of functions did solve most of the issues of not having classes, but most of the dev community has not been comfortable with these approaches. Classes make this process easier. Classes provide an abstraction on top of common behavior, thus making code reusable. The following is the syntax for defining a class in TypeScript: The syntax of the class should look very similar to readers who come from an object-oriented background. To define a class, we use a class keyword followed by the name of the class. The News class has three member properties and one method. Each member has a type assigned to it and has an access modifier to define the scope. On line 10, we create an object of a class with the new keyword. Classes in TypeScript also have the concept of a constructor, where we can initialize some properties at the time of object creation. Access modifiers Once the object is created, we can access the public members of the class with the dot operator. Note that we cannot access the author property with the espn object because this property is defined as private. TypeScript provides three types of access modifiers. Public Any property defined with the public keyword will be freely accessible outside the class. As we saw in the previous example, all the variables marked with the public keyword were available outside the class in an object. Note that TypeScript assigns public as a default access modifier if we do not assign any explicitly. This is because the default JavaScript behavior is to have everything public. Private When a property is marked as private, it cannot be accessed outside of the class. The scope of a private variable is only inside the class when using TypeScript. In JavaScript, as we do not have access modifiers, private members are treated similarly to public members. Protected The protected keyword behaves similarly to private, with the exception that protected variables can be accessed in the derived classes. The following is one such example: class base{ protected id: number; } class child extends base{ name: string; details():string{ return `${name} has id: ${this.id}` } } In the preceding code, we extend the child class with the base class and have access to the id property inside the child class. If we create an object of the child class, we will still not have access to the id property outside. Readonly As the name suggests, a property with a readonly access modifier cannot be modified after the value has been assigned to it. The value assigned to a readonly property can only happen at the time of variable declaration or in the constructor. In the above code, line 5 gives an error stating that property name is readonly, and cannot be an assigned value. Transpiled JavaScript from classes While learning TypeScript, it is important to remember that TypeScript is a superset of JavaScript and not a new language on its own. Browsers can only understand JavaScript, so it is important for us to understand the JavaScript that is transpiled by TypeScript. TypeScript provides an option to generate JavaScript based on the ECMA standards. You can configure TypeScript to transpile into ES5 or ES6 (ES 2015) and even ES3 JavaScript by using the flag target in the tsconfig.json file. The biggest difference between ES5 and ES6 is with regard to the classes, let, and const keywords which were introduced in ES6. Even though ES6 has been around for more than a year, most browsers still do not have full support for ES6. So, if you are creating an application that would target older browsers as well, consider having the target as ES5. So, the JavaScript that's generated will be different based on the target setting. Here, we will take an example of class in TypeScript and generate JavaScript for both ES5 and ES6. The following is the class definition in TypeScript: This is the same code that we saw when we introduced classes in the Understanding Classes section. Here, we have a class named News that has three members, two of which are public and one private. The News class also has a format method, which returns a string concatenated from the member variables. Then, we create an object of the News class in line 10 and assign values to public properties. In the last line, we call the format method to print the result. Now let's look at the JavaScript transpiled by TypeScript compiler for this class. ES6 JavaScript ES6, also known as ES 2015, is the latest version of JavaScript, which provides many new features on top of ES5. Classes are one such feature; JavaScript did not have classes prior to ES6. The following is the code generated from the TypeScript class, which we saw previously: If you compare the preceding code with TypeScript code, you will notice minor differences. This is because classes in TypeScript and JavaScript are similar, with just types and access modifiers additional in TypeScript. In JavaScript, we do not have the concept of declaring public members. The author variable, which was defined as private and was initialized at its declaration, is converted to a constructor initialization in JavaScript. If we had not have initialized author, then the produced JavaScript would not have added author in the constructor. ES5 JavaScript ES5 is the most popular JavaScript version supported in browsers, and if you are developing an application that has to support the majority of browser versions, then you need to transpile your code to the ES5 version. This version of JavaScript does not have classes, and hence the transpiled code converts classes to functions, and methods inside the classes are converted to prototypically defined methods on the functions. The following is the code transpiled when we have the target set as ES5 in the TypeScript compiler options: As discussed earlier, the basic difference is that the class is converted to a function. The interesting aspect of this conversion is that the News class is converted to an immediately invoked function expression (IIFE). An IIFE can be identified by the parenthesis at the end of the function declaration, as we see in line 9 in the preceding code snippet. IIFEs cause the function to be executed immediately and help to maintain the correct scope of a function rather than declaring the function in a global scope. Another difference was how we defined the method format in the ES5 JavaScript. The prototype keyword is used to add the additional behavior to the function, which we see here. A couple of other differences you may have noticed include the change of the let keyword to var, as let is not supported in ES5. All variables in ES5 are defined with the var keyword. Also, the format method now does not use a template string, but standard string concatenation to print the output. TypeScript does a good job of transpiling the code to JavaScript while following recommended practices. This helps in making sure we have a robust and reusable code with minimum error cases. If you found this tutorial useful, make sure you check out the book TypeScript 2.x By Example for more hands-on tutorials on how to effectively leverage the power of TypeScript to develop and deploy state-of-the-art web applications. How to install and configure TypeScript Understanding Patterns and Architectures in TypeScript Writing SOLID JavaScript code with TypeScript
Read more
  • 0
  • 0
  • 30675

article-image-what-does-the-structure-of-a-data-mining-architecture-look-like
Packt Editorial Staff
15 May 2018
17 min read
Save for later

What does the structure of a data mining architecture look like?

Packt Editorial Staff
15 May 2018
17 min read
Any good data mining project is built on a robust data mining architecture. Without it, your project might well be time-consuming, overly complicated or simply inaccurate. Whether you're new to data mining or want to re-familiarize yourself with what the structure of a data mining architecture should look like, you've come to the right place. Of course, this is just a guide to what a data mining architecture should look like. You'll need to be aware of how this translates to your needs and situation. This has been taken from Data Mining with R. Find it here. The core components of a data mining architecture Let's first gain a general view on the main components of a data mining architecture. It is basically composed of all of the basic elements you will need to perform the activities described in the previous chapter. As a minimum set of components, the following are usually considered: Data sources Data warehouse Data mining engine User interface Below is a diagram of a data mining architecture. You can see how each of the elements fit together: Before we get into the details of each of the components of a data mining architecture, let's first briefly look at how these components fit together: Data sources: These are all the possible sources of small bits of information to be analyzed. Data sources feed our data warehouses and are fed by the data produced from our activity toward the user interface. Data warehouse: This is where the data is stored when acquired from data sources. Data mining engine: This contains all of the logic and the processes needed to perform the actual data mining activity, taking data from the data warehouse. User interface: The front office of our machine, which allows the user to interact with the data mining engine, creating data that will be stored within the data warehouse and that could become part of the big ocean of data sources. We'll now delve a little deeper into each of these elements, starting with data sources. How data sources fit inside the data mining architecture Data sources are everywhere. This is becoming more and more true everyday thanks to the the internet of things. Now that every kind of object can be connected to the internet, we can collect data from a huge range of new physical sources. This data can come in a form already feasible for being collected and stored within our databases, or in a form that needs to be further modified to become usable for our analyses. We can, therefore, see that between our data sources and the physical data warehouse where they are going to be stored, a small components lies, which is the set of tools and software needed to make data coming from sources storable. We should note something here—we are not talking about data cleaning and data validation. Those activities will be performed later on by our data mining engine which retrieves data from the data warehouse. Types of data sources There are a range of data sources. Each type will require different data modelling techniques. Getting this wrong could seriously hamper your data mining projects, so an awareness of how data sources differ is actually really important. Unstructured data sources Unstructured data sources are data sources missing a logical data model. Whenever you find a data source where no particular logic and structure is defined to collect, store, and expose it, you are dealing with an unstructured data source. The most obvious example of an unstructured data source is a written document. That document has a lot of information in it, but there's no structure that defines and codifies how information is stored. There are some data modeling techniques that can be useful here. There are some that can even derive structured data from unstructured data. This kind of analysis is becoming increasingly popular as companies seek to use 'social listening' to understand sentiment on social media. Structured data sources Structured data sources are highly organized. These kinds of data sources follow a specific data model, and the engine which makes the storing activity is programmed to respect this model. A well-known data model behind structured data is the so-called relational model of data. Following this model, each table has to represent an entity within the considered universe of analysis. Each entity will then have a specific attribute within each column, and a related observation within each row. Finally, each entity can be related to the others through key attributes. We can think of an example of a relational database of a small factory. Within this database, we have a table recording all customers orders and one table recording all shipments. Finally, a table recording the warehouse's movements will be included. Within this database, we will have: The warehouse table linked to the shipment table through the product_code attribute The shipment table linked to the customer table through the shipment_code attribute It can be easily seen that a relevant advantage of this model is the possibility to easily perform queries within tables, and merges between them. The cost to analyze structured data is far lower than the one to be considered when dealing with unstructured data. Key issues of data sources When dealing with data sources and planning their acquisition into your data warehouse, some specific aspects need to be considered: Frequency of feeding: Is the data updated with a frequency feasible for the scope of your data mining activity? Volume of data: Can the volume of data be handled by your system, or it is too much? This is often the case for unstructured data, which tends to occupy more space for a given piece of information. Data format: Is the data format readable by your data warehouse solution, and subsequently, by your data mining engine? A careful evaluation of these three aspects has to be performed before implementing the data acquisition phase, to avoid relevant problems during the project. How databases and data warehouses fit in the data mining architecture What is a data warehouse, and how is it different from a simple database? A data warehouse is a software solution aimed at storing usually great amounts of data properly related among them and indexed through a time-related index. We can better understand this by looking at the data warehouse's cousin: the operational database. These kinds of instruments are usually of small dimensions, and aimed at storing and inquiring data, overwriting old data when new data is available. Data warehouses are therefore usually fed by databases, and stores data from those kinds of sources ensuring a historical depth to them and read-only access from other users and software applications. Moreover, data warehouses are usually employed at a company level, to store, and make available, data from (and to) all company processes, while databases are usually related to one specific process or task. How do you use a data warehouse for your data mining project? You're probably not going to use a data warehouse for your data mining process. More specicially, data will be made available via a data mart. A data mart is a partition or a sub-element of a data warehouse. The data marts are set of data that are feed directly from the data warehouse, and related to a specific company area or process. A real-life example is the data mart created to store data related to default events for the purpose of modeling customers probability of default. This kind of data mart will collect data from different tables within the data warehouse, properly joining them into new tables that will not communicate with the data warehouse one. We can therefore consider the data mart as an extension of the data warehouse. Data warehouses are usually classified into three main categories: One-level architecture where only a simple database is available and the data warehousing activity is performed by the mean of a virtual component Two-level architecture composed of a group of operational databases that are related to different activities, and a proper data warehouse is available Three-level architecture with one or more operational database, a reconciled database and a proper data warehouse Let's now have a closer look to those three different types of data warehouse. One-level database This is for sure the most simple and, in a way, primitive model. Within one level data warehouses, we actually have just one operational database, where data is written and read, mixing those two kinds of activities. A virtual data warehouse layer is then offered to perform inquiry activities. This is a primitive model for the simple reason that it is not able to warrant the appropriate level of segregation between live data, which is the one currently produced from the process, and historical data. This model could therefore produce inaccurate data and even a data loss episode. This model would be particularly dangerous for data mining activity, since it would not ensure a clear segregation between the development environment and the production one. Two-level database This more sophisticated model encompasses a first level of operational databases, for instance, the one employed within marketing, production, and accounting processes, and a proper data warehouse environment. Within this solution, the databases are to be considered like feeding data sources, where the data is produced, possibly validated, and then made available to the data warehouse. The data warehouse will then store and freeze data coming from databases, for instance, with a daily frequency. Every set of data stored within a day will be labeled with a proper attribute showing the date of record. This will later allow us to retrieve records related to a specific time period in a sort of time machine functionality. Going back to our previous probability of default example, this kind of functionality will allow us to retrieve all default events that have occurred within a given time period, constituting the estimation sample for our model. Two-level architecture is an optimal solution for data mining processes, since they will allow us to provide a safe environment, the previously mentioned data mart, to develop data mining activity, without compromising the quality of data residing within the remaining data warehouses and within the operational databases. Three-level database Three-level databases are the most advanced ones. The main difference between them and the two-level ones is the presence of the reconciliation stage, which is performed through Extraction, Transformation, and Load (ETL) instruments. To understand the relevance of such kinds of instruments, we can resort to a practical example once again, and to the one we were taking advantage of some lines previously: the probability of the default model. Imagine we are estimating such kind of model for customers clustered as large corporate, for which public forecasts, outlooks and ratings are made available by financial analyses companies like Moody's, Standard & Poor, and similar. Since this data could be reasonably related to the probability of default of our customers, we would probably be interested in adding them to our estimation database. This can be easily done through the mean of those ETL instruments. These instruments will ensure, within the reconciliation stage, that data gathered from internal sources, such as personal data and default events data, will be properly matched with the external information we have mentioned. Moreover, even within internal data fields only, those instruments will ensure the needed level of quality and coherence among different sources, at least within the data warehouse environment. Data warehouse technologies We are now going to look a bit more closely at the actual technology -  most of which is open source. A proper awareness of their existence and main features should be enough, since you will usually be taking input data from them through an interface provided by your programming language. Nevertheless, knowing what's under the hood is pretty useful... SQL SQL stands for Structured Query Language, and identifies what has been for many years the standard within the field of data storage. The base for this programming language, employed for storing and querying data, are the so-called relational data bases. The theory behind these data bases was first introduced by IBM engineer Edgar F. Codd, and is based on the following main elements: Tables, each of which represent an entity Columns, each of which represent an attribute of the entity Rows, each one representing a record of the entity Key attributes, which permit us to relate two or more tables together, establishing relations between them Starting from these main elements, SQL language provides a concise and effective way to query and retrieve this data. Moreover, basilar data munging operations, such as table merging and filtering, are possible through SQL language. As previously mentioned, SQL and relational databases have formed the vast majority of data warehouse systems around the world for many, many years. A really famous example of SQL-based data storing products is the well-known Microsoft Access software. In this software, behind the familiar user interface, hide SQL codes to store, update, and retrieve user's data. MongoDB While SQL-based products are still very popular, NoSQL technology has been going for a long time now, showing its relevance and effectiveness. Behind this acronym stands all data storing and managing solutions not based on the relational paradigm and its main elements. Among this is the document-oriented paradigm, where data is represented as documents, which are complex virtual objects identified with some kind of code, and without a fixed scheme. A popular product developed following this paradigm is MongoDB. This product stores data, representing it in the JSON format. Data is therefore organized into documents and collections, that is, a set of documents. A basic example of a document is the following: { name: "donald" , surname: "duck", style: "sailor", friends: ["mickey mouse" , "goofy", "daisy"] } As you can see, even from this basic example, the MongoDB paradigm will allow you to easily store data even with a rich and complex structure. Hadoop Hadoop is a leading technology within the field of data warehouse systems, mainly due to its ability to effectively handle large amounts of data. To maintain this ability, Hadoop fully exploits the concept of parallel computing by means of a central master that divides the all needed data related to a job into smaller chunks to be sent to two or more slaves. Those slaves are to be considered as nodes within a network, each of them working separately and locally. They can actually be physically separated pieces of hardware, but even core within a CPU (which is usually considered pseudo-parallel mode). At the heart of Hadoop is the MapReduce programming model. This model, originally conceptualized by Google, consists of a processing layer, and is responsible for moving the data mining activity close to where data resides. This minimizes the time and cost needed to perform computation, allowing for the possibility to scale the process to hundreds and hundreds of different nodes. Read next: Why choose R for your data mining project [link] The data mining engine that drives a data mining architecture The data mining engine is the true heart of our data mining architecture. It consists of tools and software employed to gain insights and knowledge from data acquired from data sources, and stored within data warehouses. What makes a data mining engine? As you should be able to imagine at this point, a good data mining engine is composed of at least three components: An interpreter, able to transmit commands defined within the data mining engine to the computer Some kind of gear between the engine and the data warehouse to produce and handle communication in both directions A set of instructions, or algorithms, needed to perform data mining activities Let's take a look at these components in a little more detail. The interpreter The interpreter carries out instructions coming from a higher-level programming language, and then translates them into instructions understandable from the piece of hardware it is running on, and transmits them to it. Obtaining the interpreter for the language you are going to perform data mining with is usually as simple as obtaining the language itself. In the case of our beloved R language, installing the language will automatically install the interpreter as well. The interface between the engine and the data warehouse If the interpreter was previously introduced, this interface we are talking about within this section is a new character within our story. The interface we are talking about here is a kind of software that enables your programming language to talk with the data warehouse solution you have been provided with for your data mining project. To exemplify the concept, let's consider a setup adopting as a data mining engine, a bunch of R scripts, with their related interpreter, while employing an SQL-based database to store data. In this case, what would be the interface between the engine and the data warehouse? It could be, for instance, the RODBC package, which is a well-established package designed to let R users connect to remote servers, and transfer data from those servers to their R session. By employing this package, it will also be possible to write data to your data warehouse. This packages works exactly like a gear between the R environment and the SQL database. This means you will write your R code, which will then be translated into a readable language from the database and sent to him. For sure, this translation also works the other way, meaning that results coming from your instructions, such as new tables of results from a query, will be formatted in a way that's readable from the R environment and conveniently shown to the user. The data mining algorithms This last element of the engine is the actual core topic of the book you are reading—the data mining algorithms. To help you gain an organic and systematic view of what we have learned so far, we can consider that these algorithms will be the result of the data modelling phase described in the previous chapter in the context of the CRISP-DM methodology description. This will usually not include code needed to perform basic data validation treatments, such as integrity checking and massive merging among data from different sources, since those kind of activities will be performed within the data warehouse environment. This will be especially true in cases of three-level data warehouses, which have a dedicated reconciliation layer. The user interface - the bit that makes the data mining architecture accessible Until now, we have been looking at the back office of our data mining architecture, which is the part not directly visible to its end user. Imagine this architecture is provided to be employed by someone not skilled enough to work on the data mining engine itself; we will need some way to let this user interact with the architecture in the right way, and discover the results of its interaction. This is what a user interface is all about. Clarity and simplicity There's a lot to be said about UI design that site more in the field of design than data analysis. Clearly, those fields are getting blurred as data mining becomes more popular, and as 'self-service' analytics grows as a trend. However, the fundamental elements of a UI is clarity and simplicity. What this means is that it is designed with purpose and usage in mind. What do you want to see? What do you want to be able to do with your data? Ask yourself this question: how many steps you need to perform to reach the objective you want to reach with the product? Imagine evaluating a data mining tool, and particularly, its data import feature. Evaluating the efficiency of the tool in this regard would involve answering the following question: how many steps do I need to perform to import a dataset into my data mining environment? Every piece is important in the data mining architecture When it comes to data mining architecture, it's essential that you don't overlook either part of it. Every component is essential. Of course, like any other data mining project, understanding what your needs are - and the needs of those in your organization - are going to inform how you build each part. But fundamentally the principles behind a robust and reliable data mining architecture will always remain the same. Read more: Expanding Your Data Mining Toolbox [link]
Read more
  • 0
  • 0
  • 11050

article-image-tensorflow-models-mobile-embedded-devices
Savia Lobo
15 May 2018
12 min read
Save for later

How to Build TensorFlow Models for Mobile and Embedded devices

Savia Lobo
15 May 2018
12 min read
TensorFlow models can be used in applications running on mobile and embedded platforms. TensorFlow Lite and TensorFlow Mobile are two flavors of TensorFlow for resource-constrained mobile devices. TensorFlow Lite supports a subset of the functionality compared to TensorFlow Mobile. It results in better performance due to smaller binary size with fewer dependencies. The article covers topics for training a model to integrate TensorFlow into an application. The model can then be saved and used for inference and prediction in the mobile application. [box type="note" align="" class="" width=""]This article is an excerpt from the book Mastering TensorFlow 1.x written by Armando Fandango. This book will help you leverage the power of TensorFlow and Keras to build deep learning models, using concepts such as transfer learning, generative adversarial networks, and deep reinforcement learning.[/box] To learn how to use TensorFlow models on mobile devices, following topics are covered: TensorFlow on mobile platforms TF Mobile in Android apps TF Mobile demo on Android TF Mobile demo on iOS TensorFlow Lite TF Lite demo on Android TF Lite demo on iOS TensorFlow on mobile platforms TensorFlow can be integrated into mobile apps for many use cases that involve one or more of the following machine learning tasks: Speech recognition Image recognition Gesture recognition Optical character recognition Image or text classification Image, text, or speech synthesis Object identification To run TensorFlow on mobile apps, we need two major ingredients: A trained and saved model that can be used for predictions A TensorFlow binary that can receive the inputs, apply the model, produce the predictions, and send the predictions as output The high-level architecture looks like the following figure: The mobile application code sends the inputs to the TensorFlow binary, which uses the trained model to compute predictions and send the predictions back. TF Mobile in Android apps The TensorFlow ecosystem enables it to be used in Android apps through the interface class  TensorFlowInferenceInterface, and the TensorFlow Java API in the jar file libandroid_tensorflow_inference_java.jar. You can either use the jar file from the JCenter, download a precompiled jar from ci.tensorflow.org, or build it yourself. The inference interface has been made available as a JCenter package and can be included in the Android project by adding the following code to the build.gradle file: allprojects  { repositories  { jcenter() } } dependencies  { compile  'org.tensorflow:tensorflow-android:+' } Note : Instead of using the pre-built binaries from the JCenter, you can also build them yourself using Bazel or Cmake by following the instructions at this link: https://github.com/tensorflow/tensorflow/blob/r1.4/ tensorflow/contrib/android/README.md Once the TF library is configured in your Android project, you can call the TF model with the following four steps:  Load the model: TensorFlowInferenceInterface  inferenceInterface  = new  TensorFlowInferenceInterface(assetManager,  modelFilename);  Send the input data to the TensorFlow binary: inferenceInterface.feed(inputName, floatValues,  1,  inputSize,  inputSize,  3);  Run the prediction or inference: inferenceInterface.run(outputNames,  logStats);  Receive the output from the TensorFlow binary: inferenceInterface.fetch(outputName,  outputs); TF Mobile demo on Android In this section, we shall learn about recreating the Android demo app provided by the TensorFlow team in their official repo. The Android demo will install the following four apps on your Android device: TF  Classify: This is an object identification app that identifies the images in the input from the device camera and classifies them in one of the pre-defined classes. It does not learn new types of pictures but tries to classify them into one of the categories that it has already learned. The app is built using the inception model pre-trained by Google. TF  Detect: This is an object detection app that detects multiple objects in the input from the device camera. It continues to identify the objects as you move the camera around in continuous picture feed mode. TF  Stylize: This is a style transfer app that transfers one of the selected predefined styles to the input from the device camera. TF  Speech: This is a speech recognition app that identifies your speech and if it matches one of the predefined commands in the app, then it highlights that specific command on the device screen. Note: The sample demo only works for Android devices with an API level greater than 21 and the device must have a modern camera that supports FOCUS_MODE_CONTINUOUS_PICTURE. If your device camera does not have this feature supported, then you have to add the path submitted to TensorFlow by the author: https://github.com/ tensorflow/tensorflow/pull/15489/files. The easiest way to build and deploy the demo app on your device is using Android Studio. To build it this way, follow these steps:  Install Android Studio. We installed Android Studio on Ubuntu 16.04 from the instructions at the following link: https://developer.android.com/studio/ install.html  Check out the TensorFlow repository, and apply the patch mentioned in the previous tip. Let's assume you checked out the code in the tensorflow folder in your home directory.  Using Android Studio, open the Android project in the path ~/tensorflow/tensorflow/examples/Android.     Your screen will look similar to this:  Expand the Gradle Scripts option from the left bar and then open the  build.gradle file.  In the build.gradle file, locate the def  nativeBuildSystem definition and set it to 'none'. In the version of  the code we checked out, this definition is at line 43: def  nativeBuildSystem  =  'none'  Build the demo and run it on either a real or simulated device. We tested the app on these devices: 7.  You can also build the apk and install the apk file on the virtual or actual connected device. Once the app installs on the device, you will see the four apps we discussed earlier: You can also build the whole demo app from the source using Bazel or Cmake by following the instructions at this link: https://github.com/tensorflow/tensorflow/tree/r1.4/tensorflow/examples/android TF Mobile in iOS apps TensorFlow enables support for iOS apps by following these steps:  Include TF Mobile in your app by adding a file named Profile in the root directory of your project. Add the following content to the Profile: target  'Name-Of-Your-Project' pod  'TensorFlow-experimental'  Run the pod  install command to download and install the TensorFlow Experimental pod.  Run the myproject.xcworkspace command to open the workspace so you can add the      prediction code to your application logic. Note: To create your own TensorFlow binaries for iOS projects, follow the instructions at this link: https://github.com/tensorflow/tensorflow/ tree/master/tensorflow/examples/ios Once the TF library is configured in your iOS project, you can call the TF model with the following four steps:  Load the model: PortableReadFileToProto(file_path,  &tensorflow_graph);  Create a session: tensorflow::Status  s  =  session->Create(tensorflow_graph);  Run the prediction or inference and get the outputs: std::string  input_layer  =  "input"; std::string  output_layer  =  "output"; std::vector<tensorflow::Tensor>  outputs; tensorflow::Status  run_status  =  session->Run( {{input_layer,  image_tensor}}, {output_layer},  {},  &outputs);  Fetch the output data: tensorflow::Tensor*  output  =  &outputs[0]; TF Mobile demo on iOS In order to build the demo on iOS, you need Xcode 7.3 or later. Follow these steps to build the iOS demo apps:  Check out the TensorFlow code in a tensorflow folder in your home directory.  Open a terminal window and execute the following commands from your home folder to download the Inception V1 model, extract the label and graph files, and move these files into the data folders inside the sample app code: $ mkdir -p ~/Downloads $ curl -o ~/Downloads/inception5h.zip https://storage.googleapis.com/download.tensorflow.org/models/incep tion5h.zip && unzip ~/Downloads/inception5h.zip -d ~/Downloads/inception5h $ cp ~/Downloads/inception5h/* ~/tensorflow/tensorflow/examples/ios/benchmark/data/ $ cp ~/Downloads/inception5h/* ~/tensorflow/tensorflow/examples/ios/camera/data/ $ cp ~/Downloads/inception5h/* ~/tensorflow/tensorflow/examples/ios/simple/data/  Navigate to one of the sample folders and download the experimental pod: $ cd ~/tensorflow/tensorflow/examples/ios/camera $ pod install  Open the Xcode workspace: $ open tf_simple_example.xcworkspace  Run the sample app in the device simulator. The sample app will appear with a Run Model button. The camera app requires an Apple device to be connected, while the other two can run in a simulator too. TensorFlow Lite TF Lite is the new kid on the block and still in the developer view at the time of writing this book. TF Lite is a very small subset of TensorFlow Mobile and TensorFlow, so the binaries compiled with TF Lite are very small in size and deliver superior performance. Apart from reducing the size of binaries, TensorFlow employs various other techniques, such as: The kernels are optimized for various device and mobile architectures The values used in the computations are quantized The activation functions are pre-fused It leverages specialized machine learning software or hardware available on the device, such as the Android NN API The workflow for using the models in TF Lite is as follows:  Get the model: You can train your own model or pick a pre-trained model available from different sources, and use the pre-trained as is or retrain it with your own data, or retrain after modifying some parts of the model. As long as you have a trained model in the file with an extension .pb or .pbtxt, you are good to proceed to the next step. We learned how to save the models in the previous chapters.  Checkpoint the model: The model file only contains the structure of the graph, so you need to save the checkpoint file. The checkpoint file contains the serialized variables of the model, such as weights and biases. We learned how to save a checkpoint in the previous chapters.  Freeze the model: The checkpoint and the model files are merged, also known as freezing the graph. TensorFlow provides the freeze_graph tool for this step, which can be executed as follows: $ freeze_graph --input_graph=mymodel.pb --input_checkpoint=mycheckpoint.ckpt --input_binary=true --output_graph=frozen_model.pb --output_node_name=mymodel_nodes  Convert the model: The frozen model from step 3 needs to be converted to TF Lite format with the toco tool provided by TensorFlow: $ toco --input_file=frozen_model.pb --input_format=TENSORFLOW_GRAPHDEF --output_format=TFLITE --input_type=FLOAT --input_arrays=input_nodes --output_arrays=mymodel_nodes --input_shapes=n,h,w,c  The .tflite model saved in step 4 can now be used inside an Android or iOS app that employs the TFLite binary for inference. The process of including the TFLite binary in your app is continuously evolving, so we recommend the reader follows the information at this link to include the TFLite binary in your Android or iOS app: https://github.com/tensorflow/tensorflow/tree/master/ tensorflow/contrib/lite/g3doc Generally, you would use the graph_transforms:summarize_graph tool to prune the model obtained in step 1. The pruned model will only have the paths that lead from input to output at the time of inference or prediction. Any other nodes and paths that are required only for training or for debugging purposes, such as saving checkpoints, are removed, thus making the size of the final model very small. The official TensorFlow repository comes with a TF Lite demo that uses a pre-trained mobilenet to classify the input from the device camera in the 1001 categories. The demo app displays the probabilities of the top three categories. TF Lite Demo on Android To build a TF Lite demo on Android, follow these steps: Install Android Studio. We installed Android Studio on Ubuntu 16.04 from the instructions at the following link: https://developer.android.com/studio/ install.html Check out the TensorFlow repository, and apply the patch mentioned in the previous tip. Let's assume you checked out the code in the tensorflow folder in your home directory. Using Android Studio, open the Android project from the path ~/tensorflow/tensorflow/contrib/lite/java/demo. If it complains about a missing SDK or Gradle components, please install those components and sync Gradle. Build the project and run it on a virtual device with API > 21. We received the following warnings, but the build succeeded. You may want to resolve the warnings if the build fails: Warning:The  Jack  toolchain  is  deprecated  and  will  not run.  To  enable  support  for  Java  8 language  features  built into  the  plugin,  remove  'jackOptions  {  ...  }'  from  your build.gradle  file, and  add android.compileOptions.sourceCompatibility  1.8 android.compileOptions.targetCompatibility  1.8 Note:  Future  versions  of  the  plugin  will  not  support  usage 'jackOptions'  in  build.gradle. To learn  more,  go  to https://d.android.com/r/tools/java-8-support-message.html Warning:The  specified  Android  SDK  Build  Tools  version (26.0.1)  is  ignored,  as  it  is  below  the minimum  supported version  (26.0.2)  for  Android  Gradle  Plugin  3.0.1. Android  SDK  Build  Tools 26.0.2  will  be  used. To  suppress  this  warning,  remove  "buildToolsVersion '26.0.1'"  from  your  build.gradle  file,  as  each  version  of the  Android  Gradle  Plugin  now  has  a  default  version  of the  build  tools. TF Lite demo on iOS In order to build the demo on iOS, you need Xcode 7.3 or later. Follow these steps to build the iOS demo apps:  Check out the TensorFlow code in a  tensorflow folder in your home directory.  Build the TF Lite binary for iOS from the instructions at this link: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite  Navigate to the sample folder and download the pod: $ cd ~/tensorflow/tensorflow/contrib/lite/examples/ios/camera $ pod install  Open the Xcode workspace: $ open tflite_camera_example.xcworkspace  Run the sample app in the device simulator. We learned about using TensorFlow models on mobile applications and devices. TensorFlow provides two ways to run on mobile devices: TF Mobile and TF Lite. We learned how to build TF Mobile and TF Lite apps for iOs and Android. We used TensorFlow demo apps as an example.   If you found this post useful, do check out the book Mastering TensorFlow 1.x  to skill up for building smarter, faster, and efficient machine learning and deep learning systems. The 5 biggest announcements from TensorFlow Developer Summit 2018 Getting started with Q-learning using TensorFlow Implement Long-short Term Memory (LSTM) with TensorFlow  
Read more
  • 0
  • 0
  • 23157

article-image-chat-application-kotlin-node-js-javascript
Sugandha Lahoti
14 May 2018
30 min read
Save for later

Building chat application with Kotlin using Node.js, the powerful Server-side JavaScript platform

Sugandha Lahoti
14 May 2018
30 min read
When one mentions server-side JavaScript technology, Node.js is what comes to our mind first. Node.js is an extremely powerful and robust platform. Using this JavaScript platform, we can build server-side applications very easily. In today's tutorial, we will focus on creating a chat application that uses Kotlin by using the Node.js technology. So, basically, we will transpile Kotlin code to JavaScript on the server side. This article is an excerpt from the book, Kotlin Blueprints, written by Ashish Belagali, Hardik Trivedi, and Akshay Chordiya. With this book, you will get to know the building blocks of Kotlin and best practices when using quality world-class applications. Kotlin is a modern language and is gaining popularity in the JavaScript community day by day. The Kotlin language with modern features and statically typed; is superior to JavaScript. Similar to JavaScript, developers that know Kotlin can use the same language skills on both sides but, they also have the advantage of using a better language. The Kotlin code gets transpiled to JavaScript and that in turn works with the Node.js. This is the mechanism that lets you use the Kotlin code to work with a server-side technology, such as Node.js. Creating a chat application Our chat app will have following functionalities: User can log in by entering their nickname User can see the list of online users User will get notified when a new user joins User can receive chats from anyone User can perform a group chat in a chat room User will receive a notification when any user leaves the chat To visualize the app that we will develop, take a look at the following screenshots. The following screenshot is a page where the user will enter a nickname and gain an entry in our chat app: In the following screen, you can see a chat window and a list of online users: We have slightly configured this application in a different way. We have kept the backend code module and frontend code module separate using the following method: Create a new project named kotlin_node_chat_app Now, create a new Gradle module named backend and select Kotlin (JavaScript) under the libraries and additional information window, and follow the remaining steps. Similarly, also create a Gradle module named webapp. The backend module will contain all the Kotlin code that will be converted into Node.JS code later, and the webapp module will contain all the Kotlin code that will later be converted into the JavaScript code. We have referred to the directory structure from Github. After performing the previous steps correctly, your project will have three build.gradle files. We have highlighted all three files in the project explorer section, as shown in the following screenshot: Setting up the Node.js server We need to initialize our root directory for the node. Execute npm init and it will create package.json. Now our login page is created. To run it, we need to set up the Node.js server. We want to create the server in such a way that by executing npm start, it should start the server. To achieve it, our package.json file should look like the following piece of code: { "name": "kotlin_node_chat_app", "version": "1.0.0", "description": "", "main": "backend/server/app.js", "scripts": { "start": "node backend/server/app.js" }, "author": "Hardik Trivedi", "license": "ISC", "dependencies": { "ejs": "^2.5.7", "express": "^4.16.2", "kotlin": "^1.1.60", "socket.io": "^2.0.4" } } We have specified a few dependencies here as well: EJS to render HTML pages Express.JS as its framework, which makes it easier to deal with Node.js Kotlin, because, ultimately, we want to write our code into Kotlin and want it compiled into the Node.js code Socket.IO to perform chat Execute npm install on the Terminal/Command Prompt and it should trigger the download of all these dependencies. Specifying the output files Now, it's very important where your output will be generated once you trigger the build. For that, build.gradle will help us. Specify the following lines in your module-level build.gradle file. The backend module's build.gradle will have the following lines of code: compileKotlin2Js { kotlinOptions.outputFile = "${projectDir}/server/app.js" kotlinOptions.moduleKind = "commonjs" kotlinOptions.sourceMap = true } The webapp module's build.gradle will have the following lines of code: compileKotlin2Js { kotlinOptions.metaInfo = true kotlinOptions.outputFile = "${projectDir}/js/main.js" kotlinOptions.sourceMap = true kotlinOptions.main = "call" } In both the compileKotlin2Js nodes, kotlinOptions.outputFile plays a key role. This basically tells us that once Kotlin's code gets compiled, it will generate app.js and main.js for Node.js and JavaScript respectively. In the index.ejs file, you should define a script tag to load main.js. It will look something like the following line of code: <script type="text/javascript" src="js/main.js"></script> Along with this, also specify the following two tags: <script type="text/javascript" src="lib/kotlin/kotlin.js"></script> <script type="text/javascript" src="lib/kotlin/kotlinx-html-js.js"> </script> Examining the compilation output The kotlin.js and kotlinx-html-js.js files are nothing but the Kotlin output files. It's not compilation output, but actually transpiled output. The following are output compilations: kotlin.js: This is the runtime and standard library. It doesn't change between applications, and it's tied to the version of Kotlin being used. {module}.js: This is the actual code from the application. All files are compiled into a single JavaScript file that has the same name as the module. {file}.meta.js: This metafile will be used for reflection and other functionalities. Let's assume our final Main.kt file will look like this: fun main(args: Array<String>) { val socket: dynamic = js("window.socket") val chatWindow = ChatWindow { println("here") socket.emit("new_message", it) } val loginWindow = LoginWindow { chatWindow.showChatWindow(it) socket.emit("add_user", it) } loginWindow.showLogin() socket.on("login", { data -> chatWindow.showNewUserJoined(data) chatWindow.showOnlineUsers(data) }) socket.on("user_joined", { data -> chatWindow.showNewUserJoined(data) chatWindow.addNewUsers(data) }) socket.on("user_left", { data -> chatWindow.showUserLeft(data) }) socket.on("new_message", { data -> chatWindow.showNewMessage(data) }) } For this, inside main.js, our main function will look like this: function main(args) { var socket = window.socket; var chatWindow = new ChatWindow(main$lambda(socket)); var loginWindow = new LoginWindow(main$lambda_0(chatWindow, socket)); loginWindow.showLogin(); socket.on('login', main$lambda_1(chatWindow)); socket.on('user_joined', main$lambda_2(chatWindow)); socket.on('user_left', main$lambda_3(chatWindow)); socket.on('new_message', main$lambda_4(chatWindow)); } The actual main.js file will be bulkier because it will have all the code transpiled, including other functions and LoginWindow and ChatWindow classes. Keep a watchful eye on how the Lambda functions are converted into simple JavaScript functions. Lambda functions, for all socket events, are transpiled into the following piece of code: function main$lambda_1(closure$chatWindow) { return function (data) { closure$chatWindow.showNewUserJoined_qk3xy8$(data); closure$chatWindow.showOnlineUsers_qk3xy8$(data); }; } function main$lambda_2(closure$chatWindow) { return function (data) { closure$chatWindow.showNewUserJoined_qk3xy8$(data); closure$chatWindow.addNewUsers_qk3xy8$(data); }; } function main$lambda_3(closure$chatWindow) { return function (data) { closure$chatWindow.showUserLeft_qk3xy8$(data); }; } function main$lambda_4(closure$chatWindow) { return function (data) { closure$chatWindow.showNewMessage_qk3xy8$(data); }; } As can be seen, Kotlin aims to create very concise and readable JavaScript, allowing us to interact with it as needed. Specifying the router We need to write a behavior in the route.kt file. This will let the server know which page to load when any request hits the server. The router.kt file will look like this: fun router() { val express = require("express") val router = express.Router() router.get("/", { req, res -> res.render("index") }) return router } This simply means that whenever a get request with no name approaches the server, it should display an index page to the user. We are told to instruct the framework to refer to the router.kt file by writing the following line of code: app.use("/", router()) Starting the node server Now let's create a server. We should create an app.kt file under the backend module at the backend/src/kotlin path. Refer to the source code to verify. Write the following piece of code in app.kt: external fun require(module: String): dynamic external val process: dynamic external val __dirname: dynamic fun main(args: Array<String>) { println("Server Starting!") val express = require("express") val app = express() val path = require("path") val http = require("http") /** * Get port from environment and store in Express. */ val port = normalizePort(process.env.PORT) app.set("port", port) // view engine setup app.set("views", path.join(__dirname, "../../webapp")) app.set("view engine", "ejs") app.use(express.static("webapp")) val server = http.createServer(app) app.use("/", router()) app.listen(port, { println("Chat app listening on port http://localhost:$port") }) } fun normalizePort(port: Int) = if (port >= 0) port else 7000 These are multiple things to highlight here: external: This is basically an indicator for Kotlin that the line written along with this a pure JavaScript code. Also, when this code gets compiled into the respected language, the compiler understands that the class, function, or property written along with that will be provided by the developer, and so no JavaScript code should be generated for that invocation. The external modifier is automatically applied to nested declarations. For example, consider the following code block. We declare the class as external and automatically all its functions and properties are treated as external: external class Node { val firstChild: Node fun append(child: Node): Node fun removeChild(child: Node): Node // etc } dynamic: You will often see the usage of dynamic while working with JavaScript. Kotlin is a statically typed language, but it still has to interoperate with languages such as JavaScript. To support such use cases with a loosely or untyped programming language, dynamic is useful. It turns off Kotlin's type checker. A value of this type can be assigned to any variable or passed anywhere as a parameter. Any value can be assigned to a variable of dynamic type or passed to a function that takes dynamic as a parameter. Null checks are disabled for such values. require("express"): We typically use ExpressJS with Node.js. It's a framework that goes hand in hand with Node.js. It's designed with the sole purpose of developing web applications. A Node.js developer must be very familiar with it. process.env.PORT: This will find an available port on the server, as simple as that. This line is required if you want to deploy your application on a utility like Heroku. Also, notice the normalizePort function. See how concise it is. The if…else condition is written as an expression. No explicit return keyword is required. Kotlin compiler also identifies that if (port >= 0) port else 7000 will always return Int, hence no explicit return type is required. Smart, isn't it! __dirname: This is always a location where your currently executing script is present. We will use it to create a path to indicate where we have kept our web pages. app.listen(): This is a crucial one. It starts the socket and listens for the incoming request. It takes multiple parameters. Mainly, we will use two parameterized functions, that take the port number and connection callback as an argument. The app.listen() method is identical to http.Server.listen(). In Kotlin, it takes a Lambda function. Now, it's time to kick-start the server. Hit the Gradle by using ./gradlew build. All Kotlin code will get compiled into Node.js code. On Terminal, go to the root directory and execute npm start. You should be able to see the following message on your Terminal/Command Prompt: Creating a login page Now, let's begin with the login page. Along with that, we will have to enable some other settings in the project as well. If you refer to a screenshot that we mentioned at the beginning of the previous section, you can make out that we will have the title, the input filed, and a button as a part of the login page. We will create the page using Kotlin and the entire HTML tree structure, and by applying CSS to them, the will be part of our Kotlin code. For that, you should refer to the Main.kt and LoginWindow files. Creating an index.ejs file We will use EJS (effective JavaScript templating) to render HTML content on the page. EJS and Node.js go hand in hand. It's simple, flexible, easy to debug, and increases development speed. Initially, index.ejs would look like the following code snippet: <!DOCTYPE html> <html> <head> <meta name="viewport" content="width=device-width, initial- scale=1.0"/> </head> <body> <div id="container" class="mainContainer"> </div> </body> </html> The <div> tag will contain all different views, for example, the Login View, Chat Window View, and so on. Using DSL DSL stands for domain-specific language. As the name indicates, it gives you the feeling as if you are writing code in a language using terminology particular to a given domain without being geeky, but then, this terminology is cleverly embedded as a syntax in a powerful language. If you are from the Groovy community, you must be aware of builders. Groovy builders allow you to define data in a semi-declarative way. It's a kind of mini-language of its own. Builders are considered good for generating XML and laying out UI components. The Kotlin DSL uses Lambdas a lot. The DSL in Kotlin is a type-safe builder. It means we can detect compilation errors in IntelliJ's beautiful IDE. The type-check builders are much better than the dynamically typed builders of Groovy. Using kotlinx.html The DSL to build HTML trees is a pluggable dependency. We, therefore, need to set it up and configure it for our project. We are using Gradle as a build tool and Gradle has the best way to manage the dependencies. We will define the following line of code in our build.gradle file to use kotlinx.html: compile("org.jetbrains.kotlinx:kotlinx-html-js:$html_version") Gradle will automatically download this dependency from jcenter(). Build your project from menu Build | Build Project. You can also trigger a build from the terminal/command prompt. To build a project from the Terminal, go to the root directory of your project and then execute ./gradlew build. Now create the index.ejs file under the webapp directory. At this moment, your index.ejs file may look like the following: Inside your LoginWindow class file, you should write the following piece of code: class LoginWindow(val callback: (String) -> Unit) { fun showLogin() { val formContainer = document.getElementById("container") as HTMLDivElement val loginDiv = document.create.div { id = "loginDiv" h3(classes = "title") { +"Welcome to Kotlin Blueprints chat app" } input(classes = "nickNameInput") { id = "nickName" onInputFunction = onInput() maxLength = 16.toString() placeholder = "Enter your nick name" } button(classes = "loginButton") { +"Login" onClickFunction = onLoginButtonClicked() } } formContainer.appendChild(loginDiv) } } Observe how we have provided the ID, input types, and a default ZIP code value. A default ZIP code value is optional. Let's spend some time understanding the previous code. The div, input, button, and h3 all these are nothing but functions. They are basically Lambda functions. The following are the functions that use Lambda as the last parameter. You can call them in different ways: someFunction({}) someFunction("KotlinBluePrints",1,{}) someFunction("KotlinBluePrints",1){} someFunction{} Lambda functions Lambda functions are nothing but functions without a name. We used to call them anonymous functions. A function is basically passed into a parameter of a function call, but as an expression. They are very useful. They save us a lot of time by not writing specific functions in an abstract class or interface. Lambda usage can be as simple the following code snippet, where it seems like we are simply binding a block an invocation of the helloKotlin function: fun main(args: Array<String>) { val helloKotlin={println("Hello from KotlinBlueprints team!")} helloKotlin() } At the same time, lambda can be a bit complex as well, just like the following code block: fun <T> lock(lock: Lock, body: () -> T): T { lock.lock() try { return body() } finally { lock.unlock() } } In the previous function, we are acquiring a lock before executing a function and releasing it when the function gets executed. This way, you can synchronously call a function in a multithreaded environment. So, if we have a use case where we want to execute sharedObject.someCrucialFunction() in a thread-safe environment, we will call the preceding lock function like this: lock(lock,{sharedObject.someCrucialFunction()}) Now, the lambda function is the last parameter of a function call, so it can be easily written like this: lock(lock) { sharedObject.someCrucialFunction() } Look how expressive and easy to understand the code is. We will dig more into the Lambda in the upcoming section. Reading the nickname In the index.ejs page, we will have an input field with the ID nickName when it is rendered. We can simply read the value by writing the following lines of code: val nickName = (document.getElementById("nickName") as? HTMLInputElement)?.value However, to cover more possibilities, we have written it in a slightly different way. We have written it as if we are taking the input as an event. The following code block will continuously read the value that is entered into the nickName input field: private fun onInput(): (Event) -> Unit { return { val input = it.currentTarget as HTMLInputElement when (input.id) { "nickName" -> nickName = input.value "emailId" -> email = input.value } } } Check out, we have used the when function, which is a replacement for the switch case. The preceding code will check whether the ID of the element is nickName or emailId, and, based on that, it will assign the value to the respective objects by reading them from the in-out field. In the app, we will only have the nickname as the input file, but using the preceding approach, you can read the value from multiple input fields. In its simplest form, it looks like this: when (x) { 1 -> print("x == 1") 2 -> print("x == 2") else -> { // Note the block print("x is neither 1 nor 2") } } The when function compares its argument against all branches, top to bottom until some branch condition is met. The when function can be used either as an expression or as a statement. The else branch is evaluated if none of the other branch conditions are satisfied. If when is used as an expression, the else branch is mandatory, unless the compiler can prove that all possible cases are covered with branch conditions. If many cases should be handled in the same way, the branch conditions may be combined with a comma, as shown in the following code: when (x) { 0, 1 -> print("x == 0 or x == 1") else -> print("otherwise") } The following uses arbitrary expressions (not only constants) as branch conditions: when (x) { parseInt(s) -> print("s encodes x") else -> print("s does not encode x") } The following is used to check a value for being in or !in a range or a collection: when (x) { in 1..10 -> print("x is in the range") in validNumbers -> print("x is valid") !in 10..20 -> print("x is outside the range") else -> print("none of the above") } Passing nickname to the server Once our setup is done, we are able to start the server and see the login page. It's time to pass the nickname or server and enter the chat room. We have written a function named onLoginButtonClicked(). The body for this function should like this: private fun onLoginButtonClicked(): (Event) -> Unit { return { if (!nickName.isBlank()) { val formContainer = document.getElementById("loginDiv") as HTMLDivElement formContainer.remove() callback(nickName) } } } The preceding function does two special things: Smart casting Registering a simple callback Smart cast Unlike any other programming language, Kotlin also provides class cast support. The document.getElementById() method returns an Element type if instance. We basically want it to cast into HTMLDivElement to perform some <div> related operation. So, using as, we cast the Element into HTMLDivElement. With the as keyword, it's unsafe casting. It's always better to use as?. On a successful cast, it will give an instance of that class; otherwise, it returns null. So, while using as?, you have to use Kotlin's null safety feature. This gives your app a great safety net onLoginButtonClicked can be refactored by modifying the code a bit. The following code block is the modified version of the function. We have highlighted the modification in bold: private fun onLoginButtonClicked(): (Event) -> Unit { return { if (!nickName.isBlank()) { val formContainer = document.getElementById("loginDiv") as? HTMLDivElement formContainer?.remove() callback(nickName) } } } Registering a callback Oftentimes, we need a function to notify us when something gets done. We prefer callbacks in JavaScript. To write a click event for a button, a typical JavaScript code could look like the following: $("#btn_submit").click(function() { alert("Submit Button Clicked"); }); With Kotlin, it's simple. Kotlin uses the Lambda function to achieve this. For the LoginWindow class, we have passed a callback as a constructor parameter. In the LoginWindow class (val callback: (String) -> Unit), the class header specifies that the constructor will take a callback as a parameter, which will return a string when invoked. To pass a callback, we will write the following line of code: callback(nickName) To consume a callback, we will write code that will look like this: val loginWindow = LoginWindow { chatWindow.showChatWindow(it) socket.emit("add_user", it) } So, when callback(nickName) is called, chatWindow.showChatWindow will get called and the nickname will be passed. Without it, you are accessing nothing but the nickname. Establishing a socket connection We shall be using the Socket.IO library to set up sockets between the server and the clients. Socket.IO takes care of the following complexities: Setting up connections Sending and receiving messages to multiple clients Notifying clients when the connection is disconnected Read more about Socket.IO at https://socket.io/. Setting up Socket.IO We have already specified the dependency for Socket.IO in our package.json file. Look at this file. It has a dependency block, which is mentioned in the following code block: "dependencies": { "ejs": "^2.5.7", "express": "^4.16.2", "kotlin": "^1.1.60", "socket.io": "^2.0.4" } When we perform npm install, it basically downloads the socket-io.js file and keeps node_modules | socket.io inside. We will add this JavaScript file to our index.ejs file. There we can find the following mentioned <script> tag inside the <body> tag: <script type="text/javascript" src="/socket.io/socket.io.js"> </script> Also, initialize socket in the same index.js file like this: <script> window.socket = io(); </script> Listening to events With the Socket.IO library, you should open a port and listen to the request using the following lines of code. Initially, we were directly using app.listen(), but now, we will pass that function as a listener for sockets: val io = require("socket.io").listen(app.listen(port, { println("Chat app listening on port http://localhost:$port") })) The server will listen to the following events and based on those events, it will perform certain tasks: Listen to the successful socket connection with the client Listen for the new user login events Whenever a new user joins the chat, add it to the online users chat list and broadcast it to every client so that they will know that a new member has joined the chat Listen to the request when someone sends a message Receive the message and broadcast it to all the clients so that the client can receive it and show it in the chat window Emitting the event The Socket.IO library works on a simple principle—emit and listen. Clients emit the messages and a listener listens to those messages and performs an action associated with them. So now, whenever a user successfully logs in, we will emit an event named add_user and the server will add it to an online user's list. The following code line emits the message: socket.emit("add_user", it) The following code snippet listens to the message and adds a user to the list: socket.on("add_user", { nickName -> socket.nickname= nickName numOfUsers = numOfUsers.inc() usersList.add(nickName as String) }) The socket.on function will listen to the add_user event and store the nickname in the socket. Incrementing and decrementing operator overloading There are a lot of things operator overloading can do, and we have used quite a few features here. Check out how we increment a count of online users: numOfUsers = numOfUsers.inc() It is a much more readable code compared to numOfUsers = numOfUsers+1, umOfUsers += 1, or numOfUsers++. Similarly, we can decrement any number by using the dec() function. Operator overloading applies to the whole set of unary operators, increment-decrement operators, binary operators, and index access operator. Read more about all of them here. Showing a list of online users Now we need to show the list of online users. For this, we need to pass the list of all online users and the count of users along with it. Using the data class The data class is one of the most popular features among Kotlin developers. It is similar to the concept of the Model class. The compiler automatically derives the following members from all properties declared in the primary constructor: The equals()/hashCode() pair The toString() of the User(name=John, age=42) form The componentN() functions corresponding to the properties in their order of declaration The copy() function A simple version of the data class can look like the following line of code, where name and age will become properties of a class: data class User(val name: String, val age: Int) With this single line and, mainly, with the data keyword, you get equals()/hasCode(), toString() and the benefits of getters and setters by using val/var in the form of properties. What a powerful keyword! Using the Pair class In our app, we have chosen the Pair class to demonstrate its usage. The Pair is also a data class. Consider the following line of code: data class Pair<out A, out B> : Serializable It represents a generic pair of two values. You can look at it as a key–value utility in the form of a class. We need to create a JSON object of a number of online users with the list of their nicknames. You can create a JSON object with the help of a Pair class. Take a look at the following lines of code: val userJoinedData = json(Pair("numOfUsers", numOfUsers), Pair("nickname", nickname), Pair("usersList", usersList)) The preceding JSON object will look like the following piece of code in the JSON format: { "numOfUsers": 3, "nickname": "Hardik Trivedi", "usersList": [ "Hardik Trivedi", "Akshay Chordiya", "Ashish Belagali" ] } Iterating list The user's list that we have passed inside the JSON object will be iterated and rendered on the page. Kotlin has a variety of ways to iterate over the list. Actually, anything that implements iterable can be represented as a sequence of elements that can be iterated. It has a lot of utility functions, some of which are mentioned in the following list: hasNext(): This returns true if the iteration has more elements hasPrevious(): This returns true if there are elements in the iteration before the current element next(): This returns the next element in the iteration nextIndex(): This returns the index of the element that would be returned by a subsequent call to next previous(): This returns the previous element in the iteration and moves the cursor position backward previousIndex(): This returns the index of the element that would be returned by a subsequent call to previous() There are some really useful extension functions, such as the following: .asSequence(): This creates a sequence that returns all elements from this iterator. The sequence is constrained to be iterated only once. .forEach(operation: (T) -> Unit): This performs the given operation on each element of this iterator. .iterator(): This returns the given iterator itself and allows you to use an instance of the iterator in a for the loop. .withIndex(): Iterator<IndexedValue<T>>: This returns an iterator and wraps each value produced by this iterator with the IndexedValue, containing a value, and its index. We have used forEachIndexed; this gives the extracted value at the index and the index itself. Check out the way we have iterated the user list: fun showOnlineUsers(data: Json) { val onlineUsersList = document.getElementById("onlineUsersList") onlineUsersList?.let { val usersList = data["usersList"] as? Array<String> usersList?.forEachIndexed { index, nickName -> it.appendChild(getUserListItem(nickName)) } } } Sending and receiving messages Now, here comes the interesting part: sending and receiving a chat message. The flow is very simple: The client will emit the new_message event, which will be consumed by the server, and the server will emit it in the form of a broadcast for other clients. When the user clicks on Send Message, the onSendMessageClicked method will be called. It sends the value back to the view using callback and logs the message in the chat window. After successfully sending a message, it clears the input field as well. Take a look at the following piece of code: private fun onSendMessageClicked(): (Event) -> Unit { return { if (chatMessage?.isNotBlank() as Boolean) { val formContainer = document.getElementById("chatInputBox") as HTMLInputElement callback(chatMessage!!) logMessageFromMe(nickName = nickName, message = chatMessage!!) formContainer.value = "" } } } Null safety We have defined chatMessage as nullable. Check out the declaration here: private var chatMessage: String? = null Kotlin is, by default, null safe. This means that, in Kotlin, objects cannot be null. So, if you want any object that can be null, you need to explicitly state that it can be nullable. With the safe call operator ?., we can write if(obj !=null) in the easiest way ever. The if (chatMessage?.isNotBlank() == true) can only be true if it's not null, and does not contain any whitespace. We do know how to use the Elvis operator while dealing with null. With the help of the Elvis operator, we can provide an alternative value if the object is null. We have used this feature in our code in a number of places. The following are some of the code snippets that highlight the usage of the safe call operator. Removing the view if not null: formContainer?.remove() Iterating over list if not null: usersList?.forEachIndexed { _, nickName -> it.appendChild(getUserListItem(nickName)) } Appending a child if the div tag is not null: onlineUsersList?.appendChild(getUserListItem (data["nickName"].toString())) Getting a list of all child nodes if the <ul> tag is not null: onlineUsersList?.childNodes Checking whether the string is not null and not blank: chatMessage?.isNotBlank() Force unwraps Sometimes, you will have to face a situation where you will be sure that the object will not be null at the time of accessing it. However, since you have declared nullable at the beginning, you will have to end up using force unwraps. Force unwraps have the syntax of !!. This means you have to fetch the value of the calling object, irrespective of it being nullable. We are explicitly reading the chatMessage value to pass its value in the callback. The following is the code: callback(chatMessage!!) Force unwraps are something we should avoid. We should only use them while dealing with interoperability issues. Otherwise, using them is basically nothing but throwing away Kotlin's beautiful features. Using the let function With the help of Lambda and extension functions, Kotlin is providing yet another powerful feature in the form of let functions. The let() function helps you execute a series of steps on the calling object. This is highly useful when you want to perform some code where the calling object is used multiple times and you want to avoid a null check every time. In the following code block, the forEach loop will only get executed if onlineUsersList is not null. We can refer to the calling object inside the let function using it: fun showOnlineUsers(data: Json) { val onlineUsersList = document.getElementById("onlineUsersList") onlineUsersList?.let { val usersList = data["usersList"] as? Array<String> usersList?.forEachIndexed { _, nickName -> it.appendChild(getUserListItem(nickName)) } } } Named parameter What if we told you that while calling, it's not mandatory to pass the parameter in the same sequence that is defined in the function signature? Believe us. With Kotlin's named parameter feature, it's no longer a constraint. Take a look at the following function that has a nickName parameter and the second parameter is message: private fun logMessageFromMe(nickName: String, message: String) { val onlineUsersList = document.getElementById("chatMessages") val li = document.create.li { div(classes = "sentMessages") { span(classes = "chatMessage") { +message } span(classes = "filledInitialsMe") { +getInitials(nickName) } } } onlineUsersList?.appendChild(li) } If you call a function such as logMessageForMe(mrssage,nickName), it will be a blunder. However, with Kotlin, you can call a function without worrying about the sequence of the parameter. The following is the code for this: fun showNewMessage(data: Json) { logMessage(message = data["message"] as String, nickName = data["nickName"] as String) } Note how the showNewMessage() function is calling it, passing message as the first parameter and nickName as the second parameter. Disconnecting a socket Whenever any user leaves the chat room, we will show other online users a message saying x user left. Socket.IO will send a notification to the server when any client disconnects. Upon receiving the disconnect, the event server will remove the user from the list, decrement the count of online users, and broadcast the event to all clients. The code can look something like this: socket.on("disconnect", { usersList.remove(socket.nicknameas String) numOfUsers = numOfUsers.dec() val userJoinedData = json(Pair("numOfUsers", numOfUsers), Pair("nickName", socket.nickname)) socket.broadcast.emit("user_left", userJoinedData) }) Now, it's the client's responsibility to show the message for that event on the UI. The client will listen to the event and the showUsersLeft function will be called from the ChatWindow class. The following code is used for receiving the user_left broadcast: socket.on("user_left", { data -> chatWindow.showUserLeft(data) }) The following displays the message with the nickname of the user who left the chat and the count of the remaining online users: fun showUserLeft(data: Json) { logListItem("${data["nickName"]} left") logListItem(getParticipantsMessage(data["numOfUsers"] as Int)) } Styling the page using CSS We saw how to build a chat application using Kotlin, but without showing the data on a beautiful UI, the user will not like the web app. We have used some simple CSS to give a rich look to the index.ejs page. The styling code is kept inside webapp/css/ styles.css. However, we have done everything so far entirely and exclusively in Kotlin. So, it's better we apply CSS using Kotlin as well. You may have already observed that there are a few mentions of classes. It's nothing but applying the CSS in a Kotlin way. Take a look at how we have applied the classes while making HTML tree elements using a DSL: fun showLogin() { val formContainer = document.getElementById("container") as HTMLDivElement val loginDiv = document.create.div { id = "loginDiv" h3(classes = "title") { +"Welcome to Kotlin Blueprints chat app" } input(classes = "nickNameInput") { id = "nickName" onInputFunction = onInput() maxLength = 16.toString() placeholder = "Enter your nick name" } button(classes = "loginButton") { +"Login" onClickFunction = onLoginButtonClicked() } } formContainer.appendChild(loginDiv) } We developed an entire chat application using Kotlin. If you liked this extract, read our book Kotlin Blueprints to build a REST API using Kotlin. Read More Top 4 chatbot development frameworks for developers How to build a basic server-side chatbot using Go 5 reasons to choose Kotlin over Java
Read more
  • 0
  • 0
  • 51498
article-image-building-a-microsoft-power-bi-data-model
Amarabha Banerjee
14 May 2018
11 min read
Save for later

Building a Microsoft Power BI Data Model

Amarabha Banerjee
14 May 2018
11 min read
"The data model is what feeds and what powers Power BI." - Kasper de Jonge, Senior Program Manager, Microsoft Data models developed in Power BI Desktop are at the center of Power BI projects, as they expose the interface in support of data exploration and drive the analytical queries visualized in reports and dashboards. Well-designed data models leverage the data connectivity and transformation capabilities to provide an integrated view of distinct business processes and entities. Additionally, data models contain predefined calculations, hierarchies groupings, and metadata to greatly enhance both the analytical power of the dataset and its ease of use. The combination of, Building a Power BI data model, querying and modeling, serves as the foundation for the BI and analytical capabilities of Power BI. In this article, we explore how to design and develop robust data models. Common challenges in dimensional modeling are mapped to corresponding features and approaches in Power BI Desktop, including multiple grains and many-to-many relationships. Examples are also provided to embed business logic and definitions, develop analytical calculations with the DAX language, and configure metadata settings to increase the value and sustainability of models. [box type="note" align="" class="" width=""]Our article is an excerpt from the book Microsoft Power BI Cookbook, written by Brett Powell. This book contains powerful tutorials and techniques to help you with Data Analytics and visualization with Microsoft Power BI.[/box] Designing a multi fact data model Power BI Desktop lends itself to rapid, agile development in which significant value can be obtained quickly despite both imperfect data sources and an incomplete understanding of business requirements and use cases. However, rushing through the design phase can undermine the sustainability of the solution as future needs cannot be met without structural revisions to the model or complex workarounds. A balanced design phase in which fundamental decisions such as DirectQuery versus in-memory are analyzed while a limited prototype model is used to generate visualizations and business feedback can address both short- and long-term needs. This recipe describes a process for designing a multiple fact table data model and identifies some of the primary questions and factors to consider. Setting business expectations Everyone has seen impressive Power BI demonstrations and many business analysts have effectively used Power BI Desktop independently. These experiences may create an impression that integration, rich analytics, and collaboration can be delivered across many distinct systems and stakeholders very quickly or easily. It's important to reign in any unrealistic expectations and confirm feasibility. For example, Power BI Desktop is not an enterprise BI tool like SSIS or SSAS in terms of scalability, version control, features, and configurations. Power BI datasets cannot be incrementally refreshed like partitions in SSAS, and the current 1 GB file limit (after compression) places a hard limit on the amount of data a single model can store. Additionally, if multiple data sources are needed within the model, then DirectQuery models are not an option. Finally, it's critical to distinguish the data model as a platform supporting robust analysis of business processes, not an individual report or dashboard itself. Identify the top pain points and unanswered business questions in the current state. Contrast this input with an assessment of feasibility and complexity (for example, data quality and analytical needs) and Target realistic and sustainable deliverables. How to do it Dimensional modeling best practices and star schema designs are directly applicable to Power BI data models. Short, collaborative modeling sessions can be scheduled with subject matter experts and main stakeholders. With the design of the model in place, an informed decision of the model's data mode (Import or DirectQuery) can be made prior to Development. Four-step dimensional design process Choose the business process The number and nature of processes to include depends on the scale of the sources and scope of the project In this example, the chosen processes are Internet Sales, Reseller Sales and General Ledger Declare the granularity For each business process (or fact) to be modeled from step 1, define the meaning of each row: These should be clear, concise business definitions--each fact table should only contain one grain Consider scalability limitations with Power BI Desktop and balance the needs between detail and history (for example, greater history but lower granularity) Example: One Row per Sales Order Line, One Row per GL Account Balance per fiscal period Separate business processes, such as plan and sales should never be integrated into the same table. Likewise, a single fact table should not contain distinct processes such as shipping and receiving. Fact tables can be related to common dimensions but should never be related to each other in the data model (for example, PO Header and Line level). Identify the dimensions These entities should have a natural relationship with the business process or event at the given granularity Compare the dimension with any existing dimensions and hierarchies in the organization (for example, Store) If so, determine if there's a conflict or if additional columns are required Be aware of the query performance implications with large, high cardinality dimensions such as customer tables with over 2 million rows. It may be necessary to optimize this relationship in the model or the measures and queries that use this relationship. See Chapter 11, Enhancing and Optimizing Existing Power BI Solutions, for more details. Identify the facts These should align with the business processes being modeled: For example, the sum of a quantity or a unique count of a dimension Document the business and technical definition of the primary facts and compare this with any existing reports or metadata repository (for example, Net Sales = Extended Amount - Discounts). Given steps 1-3, you should be able to walk through top business  questions and check whether the planned data model will support it. Example: "What was the variance between Sales and Plan for last month in Bikes?" Any clear gaps require modifying the earlier steps, removing the question from the scope of the data model, or a plan to address the issue with additional logic in the model (M or DAX). Focus only on the primary facts at this stage such as the individual source columns that comprise the cost facts. If the business definition or logic for core fact has multiple steps and conditions, check if the data model will naturally simplify it or if the logic can be developed in the data retrieval to avoid complex measures. Data warehouse and implementation bus matrix The Power BI model should preferably align with a corporate data architecture framework of standard facts and dimensions that can be shared across models. Though consumed into Power BI Desktop, existing data definitions and governance should be observed. Any new facts, dimensions, and measures developed with Power BI should supplement this  architecture. Create a data warehouse bus matrix: A matrix of business processes (facts) and standard dimensions is a primary tool for designing and managing data models and communicating the overall BI architecture. In this example, the business processes selected for the model are Internet Sales, Reseller Sales, and General Ledger. Create an implementation bus matrix: An outcome of the model design process should include a more detailed implementation bus matrix. Clarity and approval of the grain of the fact tables, the definitions of the primary measures, and all dimensions gives confidence when entering the development phase. Power BI queries (M) and analysis logic (DAX) should not be considered a long-term substitute for issues with data quality, master data management, and the data warehouse. If it is necessary to move forward, document the "technical debts" incurred and consider long-term solutions such as Master Data Services (MDS). Choose the dataset storage mode - Import or DirectQuery With the logical design of a model in place, one of the top design questions is whether to implement this model with DirectQuery mode or with the default imported In-Memory mode. In-Memory mode The default in-memory mode is highly optimized for query performance and supports additional modeling and development flexibility with DAX functions. With compression, columnar storage, parallel query plans, and other techniques an import mode model is able to support a large amount of data (for example, 50M rows) and still perform well with complex analysis expressions. Multiple data sources can be accessed and integrated in a single data model and all DAX functions are supported for measures, columns, and role security. However, the import or refresh process must be scheduled and this is currently limited to eight refreshes per day for datasets in shared capacity (48X per day in premium capacity). As an alternative to scheduled refreshes in the Power BI service, REST APIs can be used to trigger a data refresh of a published dataset. For example, an HTTP request to a Power BI REST API calling for the refresh of a dataset can be added to the end of a nightly update or ETL process script such that published Power BI content remains aligned with the source systems. More importantly, it's not currently possible to perform an incremental refresh such as the Current Year rows of a table (for example, a table partition) or only the source rows that have changed. In-Memory mode models must maintain a file size smaller than the current limits (1 GB compressed currently, 10GB expected for Premium capacities by October 2017) and must also manage refresh schedules in the Power BI Service. Both incremental data refresh and larger dataset sizes are identified as planned capabilities of the Microsoft Power BI Premium Whitepaper (May 2017). DirectQuery mode A DirectQuery mode model provides the same semantic layer interface for users and contains the same metadata that drives model behaviors as In-Memory models. The performance of DirectQuery models, however, is dependent on the source system and how this data is presented to the model. By eliminating the import or refresh process, DirectQuery provides a means to expose reports and dashboards to source data as it changes. This also avoids the file size limit of import mode models. However, there are several limitations and restrictions to be aware of with DirectQuery: Only a single database from a single, supported data source can be used in a DirectQuery model. When deployed for widespread use, a high level of network traffic can be generated thus impacting performance. Power BI visualizations will need to query the source system, potentially via an on-premises data gateway. Some DAX functions cannot be used in calculated columns or with role security. Additionally, several common DAX functions are not optimized for DirectQuery performance. Many M query transformation functions cannot be used with DirectQuery. MDX client applications such as Excel are supported but less metadata (for example, hierarchies) is exposed. Given these limitations and the importance of a "speed of thought" user experience with Power BI, DirectQuery should generally only be used on centralized and smaller projects in which visibility to updates of the source data is essential. If a supported DirectQuery system (for example, Teradata or Oracle) is available, the performance of core measures and queries should be tested. Confirm referential integrity in the source database and use the Assume Referential Integrity relationship setting in DirectQuery mode models. This will generate more efficient inner join SQL queries against the source Database. How it works DAX formula and storage engine Power BI Datasets and SQL Server Analysis Services (SSAS) share the same database engine and architecture. Both tools support both Import and DirectQuery data models and both DAX and MDX client applications such as Power BI (DAX) and Excel (MDX). The DAX Query Engine is comprised of a formula and a storage engine for both Import and DirectQuery models. The formula engine produces query plans, requests data from the storage engine, and performs any remaining complex logic not supported by the storage engine against this data such as IF and SWITCH functions In DirectQuery models, the data source database is the storage engine--it receives SQL queries from the formula engine and returns the results to the formula engine. For In- Memory models, the imported and compressed columnar memory cache is the storage engine. We discussed about building data models using Microsoft power BI. If you liked our post, be sure to check out Microsoft Power BI Cookbook to gain more information on using Microsoft power BI for data analysis and visualization. Unlocking the secrets of Microsoft Power BI Microsoft spring updates for PowerBI and PowerApps How to build a live interactive visual dashboard in Power BI with Azure Stream  
Read more
  • 0
  • 0
  • 32451

article-image-this-week-on-packt-hub-11-may-2018
Aarthi Kumaraswamy
11 May 2018
3 min read
Save for later

This week on Packt Hub – 11 May 2018

Aarthi Kumaraswamy
11 May 2018
3 min read
May’s continues on a high note. Plenty of big announcements and major new releases announced in two of the biggest events in the tech world: Google I/O, Microsoft Build and PyCon. Read about them and more in our tech news section. Here’s what you may have missed in the last 7 days – Tech news, insights and tutorials… Tech news Conferences in focus this week Top 5 Google I/O 2018 conference Day 1 Highlights What we learned from Qlik Qonnections 2018 Microsoft Build 2018 Day 1: Azure meets Artificial Intelligence Data news in depth Microsoft Open Sources ML.NET, a cross-platform machine learning framework Linux Foundation launches the Acumos Al Project to make AI accessible Nvidia’s Volta Tensor Core GPU hits performance milestones. But is it the best? Development & programming news in depth Google’s Android Things, developer preview 8: First look Put your game face on! Unity 2018.1 is now available What’s new in Vapor 3, the popular Swift based web framework Xamarin Forms 3, the popular cross-platform UI Toolkit, is here! Windows 10 IoT Core: What you need to know Google Daydream powered Lenovo Mirage solo hits the market GCC 8.1 Standards released! Google open sources Seurat to bring high precision graphics to Mobile VR Cloud & networking news in depth What to expect from vSphere 6.7 What’s new in Wireshark 2.6? Get DevOps eBooks and videos while supporting charity Microsoft’s Azure Container Service (ACS) is now Azure Kubernetes Services (AKS) Red Hat Enterprise Linux 7.5 (RHEL 7.5) now generally available Kali Linux 2018.2 released Tutorials Data tutorials Getting Started with Automated Machine Learning (AutoML) Analyzing CloudTrail Logs using Amazon Elasticsearch Tensor Processing Unit (TPU) 3.0: Google’s answer to cloud-ready Artificial Intelligence Distributed TensorFlow: Working with multiple GPUs and servers Implementing 3 Naive Bayes classifiers in scikit-learn Development & programming tutorials Web development tutorials Getting started with Angular CLI and build your first Angular Component How to implement Internationalization and localization in your Node.js app Programming tutorials How to install and configure TypeScript NativeScript: What is it, and how to set it up Building functional programs with F# Applying Single Responsibility principle from SOLID in .NET Core Unit Testing in .NET Core with Visual Studio 2017 for better code quality Cloud & Networking tutorials How to secure a private cloud using IAM How to create your own AWS CloudTrail How to secure ElasticCache in AWS This week’s opinions, analysis, and insights Data Insights Google News’ AI revolution strikes balance between personalization and the bigger picture Why Drive.ai is going to struggle to disrupt public transport 6 reasons to choose MySQL 8 for designing database solutions Are Recurrent Neural Networks capable of warping time? Development & Programming Insights 8 recipes to master Promises in ECMAScript 2018 Forget C and Java. Learn Kotlin: the next universal programming language
Read more
  • 0
  • 0
  • 3214

article-image-how-to-secure-elasticcache-in-aws
Savia Lobo
11 May 2018
5 min read
Save for later

How to secure ElasticCache in AWS

Savia Lobo
11 May 2018
5 min read
AWS offers services to handle the cache management process. Earlier, we were using Memcached or Redis installed on VM, which was a very complex and tough task to manage in terms of ensuring availability, patching, scalability, and security. [box type="shadow" align="" class="" width=""]This article is an excerpt taken from the book,'Cloud Security Automation'. In this book, you'll learn the basics of why cloud security is important and how automation can be the most effective way of controlling cloud security.[/box] On AWS, we have this service available as ElastiCache. This gives you the option to use any engine (Redis or Memcached) to manage your cache. It's a scalable platform that will be managed by AWS in the backend. ElastiCache provides a scalable and high-performance caching solution. It removes the complexity associated with creating and managing distributed cache clusters using Memcached or Redis. Now, let's look at how to secure ElastiCache. Secure ElastiCache in AWS For enhanced security, we deploy ElastiCache clusters inside VPC. When they are deployed inside VPC, we can use a security group and NACL to add a level of security on the communication ports at network level. Apart from this, there are multiple ways to enable security for ElastiCache. VPC-level security Using a security group at VPC—when we deploy AWS ElastiCache in VPC, it gets associated with a subnet, a security group, and the routing policy of that VPC. Here, we define a rule to communicate with the ElastiCache cluster on a specific port. ElastiCache clusters can also be accessed from on-premise applications using VPN and Direct Connect. Authentication and access control We use IAM in order to implement the authentication and access control on ElastiCache. For authentication, you can have the following identity type: Root user: It's a superuser that is created while setting up an AWS account. It has super administrator privileges for all the AWS services. However, it's not recommended to use the root user to access any of the services. IAM user: It's a user identity in your AWS account that will have a specific set of permissions for accessing the ElastiCache service. IAM role: We also can define an IAM role with a specific set of permissions and associate it with the services that want to access ElastiCache. It basically generates temporary access keys to use ElastiCache. Apart from this, we can also specify federated access to services where we have an IAM role with temporary credentials for accessing the service. To access ElastiCache, service users or services must have a specific set of permissions such as create, modify, and reboot the cluster. For this, we define an IAM policy and associate it with users or roles. Let's see an example of an IAM policy where users will have permission to perform system administration activity for ElastiCache cluster: { "Version": "2012-10-17", "Statement":[{ "Sid": "ECAllowSpecific", "Effect":"Allow", "Action":[ "elasticache:ModifyCacheCluster", "elasticache:RebootCacheCluster", "elasticache:DescribeCacheClusters", "elasticache:DescribeEvents", "elasticache:ModifyCacheParameterGroup", "elasticache:DescribeCacheParameterGroups", "elasticache:DescribeCacheParameters", "elasticache:ResetCacheParameterGroup", "elasticache:DescribeEngineDefaultParameters"], "Resource":"*" } ] } Authenticating with Redis authentication AWS ElastiCache also adds an additional layer of security with the Redis authentication command, which asks users to enter a password before they are granted permission to execute Redis commands on a password-protected Redis server. When we use Redis authentication, there are the following few constraints for the authentication token while using ElastiCache: Passwords must have at least 16 and a maximum of 128 characters Characters such as @, ", and / cannot be used in passwords Authentication can only be enabled when you are creating clusters with the in-transit encryption option enabled The password defined during cluster creation cannot be changed To make the policy harder or more complex, there are the following rules related to defining the strength of a password: A password must include at least three characters of the following character types: Uppercase characters Lowercase characters Digits Non-alphanumeric characters (!, &, #, $, ^, <, >, -) A password must not contain any word that is commonly used A password must be unique; it should not be similar to previous passwords Data encryption AWS ElastiCache and EC2 instances have mechanisms to protect against unauthorized access of your data on the server. ElastiCache for Redis also has methods of encryption for data run-in on Redis clusters. Here, too, you have data-in-transit and data-at-rest encryption methods. Data-in-transit encryption ElastiCache ensures the encryption of data when in transit from one location to another. ElastiCache in-transit encryption implements the following features: Encrypted connections: In this mode, SSL-based encryption is enabled for server and client communication Encrypted replication: Any data moving between the primary node and the replication node are encrypted Server authentication: Using data-in-transit encryption, the client checks the authenticity of a connection—whether it is connected to the right server Client authentication: After using data-in-transit encryption, the server can check the authenticity of the client using the Redis authentication feature Data-at-rest encryption ElastiCache for Redis at-rest encryption is an optional feature that increases data security by encrypting data stored on disk during sync and backup or snapshot operations. However, there are the following few constraints for data-at-rest encryption: It is supported only on replication groups running Redis version 3.2.6. It is not supported on clusters running Memcached. It is supported only for replication groups running inside VPC. Data-at-rest encryption is supported for replication groups running on any node type. During the creation of the replication group, you can define data-at-rest encryption. Data-at-rest encryption once enabled, cannot be disabled. To summarize, we learned how to secure ElastiCache and ensured security for PaaS services, such as database and analytics services. If you've enjoyed reading this article, do check out 'Cloud Security Automation' for hands-on experience of automating your cloud security and governance. How to start using AWS AWS Sydney Summit 2018 is all about IoT AWS Fargate makes Container infrastructure management a piece of cake    
Read more
  • 0
  • 0
  • 50680
article-image-install-configure-typescript
Amey Varangaonkar
11 May 2018
9 min read
Save for later

How to install and configure TypeScript

Amey Varangaonkar
11 May 2018
9 min read
In this tutorial, we will look at the installation process of TypeScript and the editor setup for TypeScript development. Microsoft does well in providing easy-to-perform steps to install TypeScript on all platforms, namely Windows, macOS, and Linux. [box type="shadow" align="" class="" width=""]The following excerpt is taken from the book TypeScript 2.x By Example written by Sachin Ohri. This book presents hands-on examples and projects to learn the fundamental concepts of the popular TypeScript programming language.[/box] Installation of TypeScript TypeScript's official website is the best source to install the latest version. On the website, go to the Download section. There, you will find details on how to install TypeScript. Node.js and Visual Studio are the two most common ways to get it. It supports a host of other editors and has plugins available for them in the same link. We will be installing TypeScript using Node.js and using Visual Studio Code as our primary editor. You can use any editor of your choice and be able to run the applications seamlessly. If you use full-blown Visual Studio as your primary development IDE, then you can use either of the links, Visual Studio 2017 or Visual Studio 2013, to download the TypeScript SDK. Visual Studio does come with a TypeScript compiler but it's better to install it from this link so as to get the latest version. To install TypeScript using Node.js, we will use npm (node package manager), which comes with Node.js. Node.js is a popular JavaScript runtime for building and running server-side JavaScript applications. As TypeScript compiles into JavaScript, Node is an ideal fit for developing server-side applications with the TypeScript language. As mentioned on the website, just running the following command in the Terminal (on macOS) / Command Prompt (on Windows) window will install the latest version: npm install -g typescript To load any package from Node.js, the npm command starts with npm install; the -g flag identifies that we are installing the package globally. The last parameter is the name of the package that we are installing. Once it is installed, you can check the version of TypeScript by running the following command in the Terminal window: tsc -v You can use the following command to get the help for all the other options that are available with tsc: tsc -h TypeScript editors One of the outstanding features of TypeScript is its support for editors. All the editors provide support for language services, thereby providing features such as IntelliSense, statement completion, and error highlighting. If you are coming from a .NET background, then Visual Studio 2013/2015/2017 is a good option for you. Visual Studio does not require any configuration and it's easy to start using TypeScript. As we discussed earlier, just install the SDK and you are good to go. If you are from a Java background, TypeScript supports Eclipse as well. It also supports plugins for Sublime, WebStorm, and Atom, and each of these provides a rich set of features. Visual Studio Code (VS Code) is another good option for an IDE. It's a smaller, lighter version of Visual Studio and primarily used for web application development. VS Code is lightweight and cross-platform, capable of running on Windows, Linux, and macOS. It has an ever-increasing set of plugins to help you write better code, such as TSLint, a static analysis tool to help TypeScript code for readability, maintainability, and error checking. VS Code has a compelling case to be the default IDE for all sorts of web application development. In this post, we will briefly look at the Visual Studio and VS Code setup for TypeScript. Visual Studio Visual Studio is a full-blown IDE provided by Microsoft for all .NET based development, but now Visual Studio also has excellent support for TypeScript with built-in project templates. A TypeScript compiler is integrated into Visual Studio to allow automatic transpiling of code to JavaScript. Visual Studio also has the TypeScript language service integrated to provide IntelliSense and design-time error checking, among other things. With Visual Studio, creating a project with a TypeScript file is as simple as adding a new file with a .ts extension. Visual Studio will provide all the features out of the box. VS Code VS Code is a lightweight IDE from Microsoft used for web application development. VS Code can be installed on Windows, macOS, and Linux-based systems. VS Code can recognize the different type of code files and comes with a huge set of extensions to help in development. You can install VS Code from https://code.visualstudio.com/download. VS Code comes with an integrated TypeScript compiler, so we can start creating projects directly. The following screenshot shows a TypeScript file opened in VS Code: To run the project in VS Code, we need a task runner. VS Code includes multiple task runners which can be configured for the project, such as Gulp, Grunt, and TypeScript. We will be using the TypeScript task runner for our build. VS Code has a Command Palette which allows you to access various different features, such as Build Task, Themes, Debug options, and so on. To open the Command Palette, use Ctrl + Shift + P on a Windows machine or Cmd + Shift + P on a macOS. In the Command Palette, type Build, as shown in the following screenshot, which will show the command to build the project: When the command is selected, VS Code shows an alert, No built task defined..., as follows: We select Configure Build Task and, from all the available options as shown in the following screenshot, choose TypeScript build: This creates a new folder in your project, .vscode and a new file, task.json. This JSON file is used to create the task that will be responsible for compiling TypeScript code in VS Code. TypeScript needs another JSON file (tsconfig.json) to be able to configure compiler options. Every time we run the code, tsc will look for a file with this name and use this file to configure itself. TypeScript is extremely flexible in transpiling the code to JavaScript as per developer requirements, and this is achieved by configuring the compiler options of TypeScript. TypeScript compiler The TypeScript compiler is called tsc and is responsible for transpiling the TypeScript code to JavaScript. The TypeScript compiler is also cross-platform and supported on Windows, macOS, and Linux. To run the TypeScript compiler, there are a couple of options. One is to integrate the compiler in your editor of choice, which we explained in the previous section. In the previous section, we also integrated the TypeScript compiler with VS Code, which allowed us to build our code from the editor itself. All the compiler configurations that we would want to use are added to the tsconfig.json file. Another option is to use tsc directly from the command line / Terminal window. TypeScript's tsc command takes compiler configuration options as parameters and compiles code into JavaScript. For example, create a simple TypeScript file in Notepad and add the following lines of code to it. To create a file as a TypeScript file, we just need to make sure we have the file extension as *.ts: class Editor { constructor(public name: string,public isTypeScriptCompatible : Boolean) {} details() { console.log('Editor: ' + this.name + ', TypeScript installed: ' + this.isTypeScriptCompatible); } } class VisualStudioCode extends Editor{ public OSType: string constructor(name: string,isTypeScriptCompatible : Boolean, OSType: string) { super(name,isTypeScriptCompatible); this.OSType = OSType; } } let VS = new VisualStudioCode('VSCode', true, 'all'); VS.details(); This is the same code example we used in the TypeScript features section of this chapter. Save this file as app.ts (you can give it any name you want, as long as the extension of the file is *.ts). In the command line / Terminal window, navigate to the path where you have saved this file and run the following command: tsc app.ts This command will build the code and the transpile it into JavaScript. The JavaScript file is also saved in the same location where we had TypeScript. If there is any build issue, tsc will show these messages on the command line only. As you can imagine, running the tsc command manually for medium- to large-scale projects is not a productive approach. Hence, we prefer to use an editor that has TypeScript integrated. The following table shows the most commonly used TypeScript compiler configurations. We will be discussing these in detail in upcoming chapters: Compiler option Type Description allowUnusedLabels boolean By default, this flag is false. This option tells the compiler to flag unused labels. alwaysStrict boolean By default, this flag is false. When turned on, this will cause the compiler to compile in strict mode and emit use strict in the source file. module string Specify module code generation: None, CommonJS, AMD, System, UMD, ES6, or ES2015. moduleResolution string Determines how the module is resolved. noImplicitAny boolean This property allows an error to be raised if there is any code which implies data type as any. This flag is recommended to be turned off if you are migrating a JavaScript project to TypeScript in an incremental manner. noImplicitReturn boolean Default value is false; raises an error if not all code paths return a value. noUnusedLocals boolean Reports an error if there are any unused locals in the code. noUnusedParameter boolean Reports an error if there are any unused parameters in the code. outDir string Redirects output structure to the directory. outFile string Concatenates and emits output to a single file. The order of concatenation is determined by the list of files passed to the compiler on the command line along with triple-slash references and imports. See the output file order documentation for more details. removeComments boolean Remove all comments except copyright header comments beginning with /*!. sourcemap boolean Generates corresponding .map file. Target string Specifies ECMAScript target version: ES3(default), ES5, ES6/ES2015, ES2016, ES2017, or ESNext. Watch Runs the compiler in watch mode. Watches input files and triggers recompilation on changes. We saw it is quite easy to set up and configure TypeScript, and we are now ready to get started with our first application! To learn more about writing and compiling your first TypeScript application, make sure you check out the book TypeScript 2.x By Example. Introduction to TypeScript Introducing Object Oriented Programmng with TypeScript Elm and TypeScript – Static typing on the Frontend
Read more
  • 0
  • 1
  • 50226

article-image-getting-started-with-automated-machine-learning-automl
Kunal Chaudhari
10 May 2018
7 min read
Save for later

Anatomy of an automated machine learning algorithm (AutoML)

Kunal Chaudhari
10 May 2018
7 min read
Machine learning has always been dependent on the selection of the right features within a given model; even the selection of the right algorithm. But deep learning changed this. The selection process is now built into the models themselves. Researchers and engineers are now shofting their focus from feature engineering to network engineering. Out of this, AutoML, or meta learning, has become an increasingly important part of deep learning. AutoML is an emerging research topic which aims at auto-selecting the most efficient neural network for a given learning task. In other words, AutoML represents a set of methodologies for learning how to learn efficiently. Consider for instance the tasks of machine translation, image recognition, or game playing. Typically, the models are manually designed by a team of engineers, data scientist, and domain experts. If you consider that a typical 10-layer network can have ~1010 candidate network, you understand how expensive, error prone, and ultimately sub-optimal the process can be. This article is an excerpt from a book written by Antonio Gulli and Amita Kapoor titled TensorFlow 1.x Deep Learning Cookbook. This book is an easy-to-follow guide that lets you explore reinforcement learning, GANs, autoencoders, multilayer perceptrons and more. AutoML with recurrent networks and with reinforcement learning The key idea to tackle this problem is to have a controller network which proposes a child model architecture with probability p, given a particular network given in input. The child is trained and evaluated for the particular task to be solved (say for instance that the child gets accuracy R). This evaluation R is passed back to the controller which, in turn, uses R to improve the next candidate architecture. Given this framework, it is possible to model the feedback from the candidate child to the controller as the task of computing the gradient of p and then scale this gradient by R. The controller can be implemented as a Recurrent Neural Network (see the following figure). In doing so, the controller will tend to privilege iteration after iterations candidate areas of architecture that achieve better R and will tend to assign a lower probability to candidate areas that do not score so well. For instance, a controller recurrent neural network can sample a convolutional network. The controller can predict many hyper-parameters such as filter height, filter width, stride height, stride width, and the number of filters for one layer and then can repeat. Every prediction can be carried out by a softmax classifier and then fed into the next RNN time step as input. This is well expressed by the following images taken from Neural Architecture Search with Reinforcement Learning, Barret Zoph, Quoc V. Le: Predicting hyperparameters is not enough as it would be optimal to define a set of actions to create new layers in the network. This is particularly difficult because the reward function that describes the new layers is most likely not differentiable. This makes it impossible to optimize using standard techniques such as SGD. The solution comes from reinforcement learning. It consists of adopting a policy gradient network. Besides that, parallelism can be used for optimizing the parameters of the controller RNN. Quoc Le & Barret Zoph proposed to adopt a parameter-server scheme where we have a parameter server of S shards, that store the shared parameters for K controller replicas. Each controller replica samples m different child architectures that are trained in parallel as illustrated in the following images, taken from Neural Architecture Search with Reinforcement Learning, Barret Zoph, Quoc V. Le: Quoc and Barret applied AutoML techniques for Neural Architecture Search to the Penn Treebank dataset, a well-known benchmark for language modeling. Their results improve the manually designed networks currently considered the state-of-the-art. In particular, they achieve a test set perplexity of 62.4 on the Penn Treebank, which is 3.6 perplexity better than the previous state-of-the-art model. Similarly, on the CIFAR-10 dataset, starting from scratch, the method can design a novel network architecture that rivals the best human-invented architecture in terms of test set accuracy. The proposed CIFAR-10 model achieves a test error rate of 3.65, which is 0.09 percent better and 1.05x faster than the previous state-of-the-art model that used a similar architectural scheme. Meta-learning blocks In Learning Transferable Architectures for Scalable Image Recognition, Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le, 2017. propose to learn an architectural building block on a small dataset that can be transferred to a large dataset. The authors propose to search for the best convolutional layer (or cell) on the CIFAR-10 dataset and then apply this learned cell to the ImageNet dataset by stacking together more copies of this cell, each with their own parameters. Precisely, all convolutional networks are made of convolutional layers (or cells) with identical structures but different weights. Searching for the best convolutional architectures is therefore reduced to searching for the best cell structures, which is faster more likely to generalize to other problems. Although the cell is not learned directly on ImageNet, an architecture constructed from the best learned cell achieves, among the published work, state-of-the-art accuracy of 82.7 percent top-1 and 96.2 percent top-5 on ImageNet. The model is 1.2 percent better in top-1 accuracy than the best human-invented architectures while having 9 billion fewer FLOPS—a reduction of 28% from the previous state of the art model. What is also important to notice is that the model learned with RNN+RL (Recurrent Neural Networks + Reinforcement Learning) is beating the baseline represented by Random Search (RS) as shown in the figure taken from the paper. In the mean performance of the top-5 and top-25 models identified in RL versus RS, RL is always winning: AutoML and learning new tasks Meta-learning systems can be trained to achieve a large number of tasks and are then tested for their ability to learn new tasks. A famous example of this kind of meta-learning is transfer learning, where networks can successfully learn new image-based tasks from relatively small datasets. However, there is no analogous pre-training scheme for non-vision domains such as speech, language, and text. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks, Chelsea Finn, Pieter Abbeel, Sergey Levine, 2017, proposes a model- agnostic approach names MAML, compatible with any model trained with gradient descent and applicable to a variety of different learning problems, including classification, regression, and reinforcement learning. The goal of meta-learning is to train a model on a variety of learning tasks, such that it can solve new learning tasks using only a small number of training samples. The meta-learner aims at finding an initialization that rapidly adapts to various problems quickly (in a small number of steps) and efficiently (using only a few examples). A model represented by a parametrized function fθ with parameters θ.When adapting to a new task Ti, the model's parameters θ become θi  . In MAML, the updated parameter vector θi  is computed using one or more gradient descent updates on task Ti. For example, when using one gradient update, θ ~ = θ − α∇θLTi (fθ) where LTi is the loss function for the task T and α is a meta-learning parameter. The MAML algorithm is reported in this figure: MAML was able to substantially outperform a number of existing approaches on popular few-shot image classification benchmark. Few shot image is a quite challenging problem aiming at learning new concepts from one or a few instances of that concept. As an example, Human-level concept learning through probabilistic program induction, Brenden M. Lake, Ruslan Salakhutdinov, Joshua B. Tenenbaum, 2015, suggested that humans can learn to identify novel two-wheel vehicles from a single picture such as the one contained in the box as follows: If you enjoyed this excerpt, check out the book TensorFlow 1.x Deep Learning Cookbook, to skill up and implement tricky neural networks using Google's TensorFlow 1.x AmoebaNets: Google’s new evolutionary AutoML AutoML : Developments and where is it heading to What is Automated Machine Learning (AutoML)?
Read more
  • 0
  • 0
  • 21659
Modal Close icon
Modal Close icon