How-To Tutorials

A key ingredient in scripting 3D games with Unity is the ability to work with C# to create arrays, lists, objects and dictionaries within the Unity platform. In this tutorial, we help you to get started with creating arrays, lists, and dictionaries effectively. This article is an excerpt from Learning C# by Developing Games with Unity 2017. Read more here. You can also read the latest edition of the book here.  An array stores a sequential collection of values of the same type, in the simplest terms. We can use arrays to store lists of values in a single variable. Imagine we want to store a number of student names. Simple! Just create a few variables and name them student1, student2, and so on: public string student1 = "Greg"; public string student2 = "Kate"; public string student3 = "Adam"; public string student4 = "Mia"; There's nothing wrong with this. We can print and assign new values to them. The problem starts when you don't know how many student names you will be storing. The name variable suggests that it's a changing element. There is a much cleaner way of storing lists of data. Let's store the same names using a C# array variable type: public string[ ] familyMembers = new string[ ]{"Greg", "Kate", "Adam", "Mia"} ; As you can see, all the preceding values are stored in a single variable called familyMembers. Declaring an array To declare a C# array, you must first say what type of data will be stored in the array. As you can see in the preceding example, we are storing strings of characters. After the type, we have an open square bracket and then immediately a closed square bracket, [ ]. This will make the variable an actual array. We also need to declare the size of the array. It simply means how many places are there in our variable to be accessed. The minimum code required to declare a variable looks similar to this: public string[] myArrayName = new string[4]; The array size is set during assignment. As you have learned before, all code after the variable declaration and the equal sign is an assignment. To assign empty values to all places in the array, simply write the new keyword followed by the type, an open square bracket, a number describing the size of the array, and then a closed square bracket. If you feel confused, give yourself a bit more time. Then you will fully understand why arrays are helpful. Take a look at the following examples of arrays; don't worry about testing how they work yet: string[ ] familyMembers = new string[]{"John", "Amanda", "Chris", "Amber"} ; string[ ] carsInTheGarage = new string[] {"VWPassat", "BMW"} ; int[ ] doorNumbersOnMyStreet = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 }; GameObject[ ] carsInTheScene = GameObject.FindGameObjectsWithTag("car"); As you can see, we can store different types of data as long as the elements in the array are of the same type. You are probably wondering why the last example, shown here, looks different: GameObject[ ] carsInTheScene = GameObject.FindGameObjectsWithTag("car"); In fact, we are just declaring the new array variable to store a collection of GameObject in the scene using the "car" tag. Jump into the Unity scripting documentation and search for GameObject.FindGameObjectsWithTag: As you can see, GameObject.FindGameObjectsWithTag is a special built-in Unity function that takes a string parameter (tag) and returns an array of GameObjects using this tag. Storing items in the List Using a List instead of an array can be so easier to work with in a script. Look at some forum sites related to C# and Unity, and you'll discover that plenty of programmers simply don't use an array unless they have to; they prefer to use a List. It is up to the developer's preference and task. Let's stick to lists for now. Here are the basics of why a List is better and easier to use than an array: An array is of fixed size and unchangeable The size of a List is adjustable You can easily add and remove elements from a List To mimic adding a new element to an array, we would need to create a whole new array with the desired number of elements and then copy the old elements The first thing to understand is that a List has the ability to store any type of object, just like an array. Also, like an array, we must specify which type of object we want a particular List to store. This means that if you want a List of integers of the int type then you can create a List that will store only the int type. Let's go back to the first array example and store the same data in a List. To use a List in C#, you need to add the following line at the beginning of your script: using System.Collections.Generic; As you can see, using Lists is slightly different from using arrays. Line 9 is a declaration and assignment of the familyMembers List. When declaring the list, there is a requirement for a type of objects that you will be storing in the List. Simply write the type between the < > characters. In this case, we are using string. As we are adding the actual elements later in lines 14 to 17, instead of assigning elements in the declaration line, we need to assign an empty List to be stored temporarily in the familyMembers variable. Confused? If so, just take a look at the right-hand side of the equal sign on line 9. This is how you create a new instance of the List for a given type, string for this example: new List<string>(); Lines 14 to 17 are very simple to understand. Each line adds an object at the end of the List, passing the string value in the parentheses. In various documentation, Lists of type look like this: List< T >. Here, T stands for the type of data. This simply means that you can insert any type in place of T and the List will become a list of that specific type. From now on, we will be using it. Common operations with Lists List<T> is very easy to use. There is a huge list of different operations that you can perform with it. We have already spoken about adding an element at the end of a List. Very briefly, let's look at the common ones that we will be possibly using at later stages: Add: This adds an object at the end of List<T>. Remove: This removes the first occurrence of a specific object from List<T>. Clear: This removes all elements from List<T>. Contains: This determines whether an element is in List<T> or not. It is very useful to check whether an element is stored in the list. Insert: This inserts an element into List<T> at the specified index. ToArray: This copies the elements of List<T> to a new array. You don't need to understand all of these at this stage. All I want you to know is that there are many out-of-the-box operations that you can use. If you want to see them all, I encourage you to dive into the C# documentation and search for the List<T> class. List <T> versus arrays Now you are probably thinking, "Okay, which one should I use?" There isn't a general rule for this. Arrays and List<T> can serve the same purpose. You can find a lot of additional information online to convince you to use one or the other. Arrays are generally faster. For what we are doing at this stage, we don't need to worry about processing speeds. Some time from now, however, you might need a bit more speed if your game slows down, so this is good to remember. List<T> offers great flexibility. You don't need to know the size of the list during declaration. There is a massive list of out-of-the-box operations that you can use with List, so it is my recommendation. Array is faster, List<T> is more flexible. Retrieving the data from the Array or List<T> Declaring and storing data in the array or list is very clear to us now. The next thing to learn is how to get stored elements from an array. To get a stored element from the array, write an array variable name followed by square brackets. You must write an int value within the brackets. That value is called an index. The index is simply a position in the array. So, to get the first element stored in the array, we will write the following code: myArray[0]; Unity will return the data stored in the first place in myArray. It works exactly the same way as the return type methods. So, if myArray stores a string value on index 0, that string will be returned to the place where you are calling it. Complex? It's not. Let's show you by example. The index value starts at 0, no 1, so the first element in an array containing 10 elements will be accessible through an index value of 0 and last one through a value of 9. Let's extend the familyMembers example: I want to talk about line 20. The rest of it is pretty obvious for you, isn't it? Line 20 creates a new variable called thirdFamilyMember and assigns the third value stored in the familyMembers list. We are using an index value of 2 instead of 3 because in programming counting starts at 0. Try to memorize this; it is a common mistake made by beginners in programming. Go ahead and click Play. You will see the name Adam being printed in the Unity Console. While accessing objects stored in an array, make sure you use an index value between zero and the size of the array. In simpler words, we cannot access data from index 10 in an array that contains only four objects. Makes sense? Checking the size This is very common; we need to check the size of the array or list. There is a slight difference between a C# array and List<T>. To get the size as an integer value, we write the name of the variable, then a dot, and then Length of an array or Count for List<T>: arrayName.Length: This returns an integer value with the size of the array listName.Count: This returns an integer value with the size of the list As we need to focus on one of the choices here and move on, from now on we will be using List<T>. ArrayList We definitely know how to use lists now. We also know how to declare a new list and add, remove, and retrieve elements. Moreover, you have learned that the data stored in List<T> must be of the same type across all elements. Let's throw a little curveball. ArrayList is basically List<T> without a specified type of data. This means that we can store whatever objects we want. Storing elements of different types is also possible. ArrayList is very flexible. Take a look at the following example to understand what ArrayList can look like: You have probably noticed that ArrayList also supports all common operations, such as .Add(). Lines 12 to 15 add different elements into the array. The first two are of the integer type, the third is a string type, and the last one is a GameObject. All mixed types of elements in one variable! When using ArrayList, you might need to check what type of element is under a specific index to know how to treat it in code. Unity provides a very useful function that you can use on virtually any type of object. Its GetType() method returns the type of the object, not the value. We are using it in lines 18 and 19 to print the types of the second and third elements. Go ahead, write the preceding code, and click Play. You should get the following output in the Console window: Dictionaries When we talk about collection data, we need to mention Dictionaries. A Dictionary is similar to a List. However, instead of accessing a certain element by index value, we use a string called key. The Dictionary that you will probably be using the most often is called Hashtable. Feel free to dive into the C# documentation after reading this chapter to discover all the bits of this powerful class. Here are a few key properties of Hashtable: Hashtable can be resized dynamically, like List<T> and ArrayList Hashtable can store multiple data types at the same type, like ArrayList A public member Hashtable isn't visible in the Unity Inspector panel due to default inspector limitations I want to make sure that you won't feel confused, so I will go straight to a simple example: Accessing values To access a specific key in the Hashtable, you must know the string key the value is stored under. Remember, the key is the first value in the brackets when adding an element to Hashtable. Ideally, you should also know the type of data you are trying to access. In most cases, that would not be an issue. Take a look at this line: Debug.Log((string)personalDetails["firstName"]); We already know that using Debug.Log serves to display a message on the Unity console, so what are we trying to display? A string value (it's one that can contain letters and numbers), then we specify where that value is stored. In this case, the information is stored under Hashtable personalDetails and the content that we want to display is firstName. Now take a look at the script once again and see if you can display the age, remember that the value that we are trying to access here is a number, so we should use int instead of string: Similar to ArrayList, we can store mixed-type data in Hashtable. Unity requires the developer to specify how an accessed element should be treated. To do this, we need to cast the element into a specific data type. The syntax is very simple. There are brackets with the data type inside, followed by the Hashtable variable name. Then, in square brackets, we have to enter the key string the value is stored under. Ufff, confusing! As you can see in the preceding line, we are casting to string (inside brackets). If we were to access another type of data, for example, an integer number, the syntax would look like this: (int)personalDetails["age"]; I hope that this is clear now. If it isn't, why not search for more examples on the Unity forums? How do I know what's inside my Hashtable? Hashtable, by default, isn't displayed in the Unity Inspector panel. You cannot simply look at the Inspector tab and preview all keys and values in your public member Hashtable. We can do this in code, however. You know how to access a value and cast it. What if you are trying to access the value under a key that isn't stored in the Hashtable? Unity will spit out a null reference error and your program is likely to crash. To check whether an element exists in the Hashtable, we can use the .Contains(object) method, passing the key parameter: This determines whether the array contains the item and if so, the code will continue; otherwise, it will stop there, preventing any error. We discussed how to use C# to create arrays, lists, dictionaries and objects in Unity. The code samples and the examples will help you implement these from the platform. Do check out this book Learning C# by Developing Games with Unity 2017  to develop your first interactive 2D and 3D platform game. Read More Unity 2D & 3D game kits simplify Unity game development for beginners How to create non-player Characters (NPC) with Unity 2018 Game Engine Wars: Unity vs Unreal Engine

Google Data Studio is one of the most popular tools for visualizing data. It can be used to pull data directly out of Google's suite of marketing tools, including Google Analytics, Google AdWords, and Google Search Console. It also supports connectors for database tools such as PostgreSQL and BigQuery, it can be accessed at datastudio.google.com. In this article, we will learn to visualize BigQuery Data with Google Data Studio. [box type="note" align="" class="" width=""]This article is an excerpt from the book, Learning Google BigQuery, written by Thirukkumaran Haridass and Eric Brown. This book will serve as a comprehensive guide to mastering BigQuery, and utilizing it to get useful insights from your Big Data.[/box] The following steps explain how to get started in Google Data Studio and access BigQuery data from Data Studio: Setting up an account: Account setup is extremely easy for Data Studio. Any user with a Google account is eligible to use all Data Studio features for free: Accessing BigQuery data: Once logged in, the next step is to connect to BigQuery. This can be done by clicking on the DATA SOURCES button on the left-hand-side navigation: You'll be prompted to create a data source by clicking on the large plus sign to the bottom-right of the screen. On the right-hand-side navigation, you'll get a list of all of the connectors available to you. Select BigQuery: At this point, you'll be prompted to select from your projects, shared projects, a custom query, or public datasets. Since you are querying the Google Analytics BigQuery Export test data, select Custom Query. Select the project you would like to use. In the Enter Custom Query prompt, add this query and click on the Connect button on the top right: SELECT trafficsource.medium as Medium, COUNT(visitId) as Visits FROM `google.com:analytics- bigquery.LondonCycleHelmet.ga_sessions_20130910` GROUP BY Medium This query will pull the count of sessions for traffic source mediums for the Google Analytics account that has been exported. The next screen shows the schema of the data source you have created. Here, you can make changes to each field of your data, such as changing text fields to date fields or creating calculated metrics: Click on Create Report. Then click on Add to Report. At this point, you will land on your report dashboard. Here, you can begin to create charts using the data you've just pulled from BigQuery. Icons for all the chart types available are shown near the top of the page. Hover over the chart types and click on the chart labeled Bar Chart; then in the grid, hold your right-click button to draw a rectangle. A bar chart should appear, with the Traffic Source Medium and Visit data from the query you ran: A properties prompt should also show on the right-hand side of the page: Here, a number of properties can be selected for your chart, including the dimension, metric, and many style settings. Once you've completed your first chart, more charts can be added to a single page to show other metrics if needed. For many situations, a single bar graph will answer the question at hand. Some situations may require more exploration. In such cases, an analyst might want to know whether the visit metric influences other metrics such as the number of transactions. A scatterplot with visits on the x axis and transactions on the y axis can be used to easily visualize this relationship. Making a scatterplot in Data Studio The following steps show how to make a scatterplot in Data Studio with the data from BigQuery: Update the original query by adding the transaction metric. In the edit screen of your report, click on the bar chart to bring up the chart options on the right-hand- side navigation. Click on the pencil icon next to the data source titled BigQuery to edit the data source. Click on the left-hand-side arrow icon titled Edit Connection: 3. In the dialog titled Enter Custom Query, add this query: SELECT trafficsource.medium as Medium, COUNT(visitId) as Visits, SUM(totals.transactions) AS Transactions FROM `google.com:analytics- bigquery.LondonCycleHelmet.ga_sessions_20130910` GROUP BY Medium Click on the button titled Reconnect in order to reprocess the query. A prompt should emerge, asking whether you'd like to add a new field titled Transactions. Click on Apply. Click on Done. Once you return to the report edit screen, click on the Scatter Chart button() and use your mouse to draw a square in the report space: The report should autoselect the two metrics you've created. Click on the chart to bring up the chart edit screen on the right-hand-side navigation; then click on the Style tab. Click on the dropdown under the Trendline option and select Linear to add a linear trend line, also known as linear regression line. The graph will default to blue, so use the pencil icon on the right to select red as the line color: Making a map in Data Studio Data Studio includes a map chart type that can be used to create simple maps. In order to create maps, a map dimension will need to be included in your data, along with a metric. Here, we will use the Google BigQuery public dataset for Medicare data. You'll need to create a new data source: Accessing BigQuery data: Once logged in, the next step is to connect to BigQuery. This can be done by clicking on the DATA SOURCES button on the left-hand-side navigation. You'll be prompted to create a data source by clicking on the large plus sign to the bottom-right of the screen. On the right-hand-side navigation, you'll get a list of all of the connectors available to you. Select BigQuery. At this point, you'll be prompted to select from your projects, shared projects, a custom query, or public datasets. Since you are querying the Google Analytics BigQuery Export test data, select Custom Query. Select the project you would like to use. In the Enter Custom Query prompt, add this query and click on the Connect button on the top right: SELECT CONCAT(provider_city,", ",provider_state) city, AVG(average_estimated_submitted_charges) avg_sub_charges FROM `bigquery-public-data.medicare.outpatient_charges_2014` WHERE apc = '0267 - Level III Diagnostic and Screening Ultrasound' GROUP BY 1 ORDER BY 2 desc This query will pull the average of submitted charges for diagnostic ultrasounds by city in the United States. This is the most submitted charge in the 2014 Medicaid data. The next screen shows the schema of the data source you have created. Here, you can make changes to each field of your data, such as changing text fields to date fields or creating calculated metrics: Click on Create Report. Then click on Add to Report. At this point, you will land on your report dashboard. Here, you can begin to create charts using the data you've just pulled from BigQuery. Icons for all the chart types available are shown near the top of the page. Hover over the chart types and click on the chart labeled Map  Chart; then in the grid, hold your right-click button to draw a rectangle. Click on the chart to bring up the Dimension Picker on the right-hand-side navigation, and click on Create New Dimension: Right click on the City dimension and select the Geo type and City subtype. Here, we can also choose other sub-types (Latitude, Longitude, Metro, Country, and so on). Data Studio will plot the top 500 rows of data (in this case, the top 500 cities in the results set). Hovering over each city brings up detailed data: Data Studio can also be used to roll up geographic data. In this case, we'll roll city data up to state data. From the edit screen, click on the map to bring up the Dimension Picker and click on Create New Dimension in the right-hand-side navigation. Right-click on the City dimension and select the Geo type and Region subtype. Google uses the term Region to signify states: Once completed, the map will be rolled up to the state level instead of the city level. This functionality is very handy when data has not been rolled up prior to being inserted into BigQuery: Other features of Data Studio Filtering: Filtering can be added to your visualizations based on dimensions or metrics as long as the data is available in the data source Data joins: Data for multiple sources can be joined to create new, calculated metrics Turnkey integrations with many Google Marketing Suite tools such as Adwords and Search Console We explored various features of Google Data Studio and learnt to use them for visualizing BigQuery data.To know about other third party tools for reporting and visualization purpose such as R and Tableau, check out the book Learning Google BigQuery. Getting Started with Data Storytelling What is Seaborn and why should you use it for data visualization? Pandas is an effective tool to explore and analyze data - Interview Insights

If we are developing any application using TypeScript, be it a small-scale or a large-scale application, we will use classes to manage our properties and methods. Prior to ES 2015, JavaScript did not have the concept of classes, and we used functions to create class-like behavior. TypeScript introduced classes as part of its initial release, and now we have classes in ES6 as well. The behavior of classes in TypeScript and JavaScript ES6 closely relates to the behavior of any object-oriented language that you might have worked on, such as C#. This excerpt is taken from the book TypeScript 2.x By Example written by Sachin Ohri. Object-oriented programming in TypeScript Object-oriented programming allows us to represent our code in the form of objects, which themselves are instances of classes holding properties and methods. Classes form the container of related properties and their behavior. Modeling our code in the form of classes allows us to achieve various features of object-oriented programming, which helps us write more intuitive, reusable, and robust code. Features such as encapsulation, polymorphism, and inheritance are the result of implementing classes. TypeScript, with its implementation of classes and interfaces, allows us to write code in an object-oriented fashion. This allows developers coming from traditional languages, such as Java and C#, feel right at home when learning TypeScript. Understanding classes Prior to ES 2015, JavaScript developers did not have any concept of classes; the best way they could replicate the behavior of classes was with functions. The function provides a mechanism to group together related properties and methods. The methods can be either added internally to the function or using the prototype keyword. The following is an example of such a function: function Name (firstName, lastName) { this.firstName = firstName; this.lastName = lastName; this.fullName = function() { return this.firstName + ' ' + this.lastName ; }; } In this preceding example, we have the fullName method encapsulated inside the Name function. Another way of adding methods to functions is shown in the following code snippet with the prototype keyword: function Name (firstName, lastName) { this.firstName = firstName; this.lastName = lastName; } Name.prototype.fullName = function() { return this.firstName + ' ' + this.lastName ; }; These features of functions did solve most of the issues of not having classes, but most of the dev community has not been comfortable with these approaches. Classes make this process easier. Classes provide an abstraction on top of common behavior, thus making code reusable. The following is the syntax for defining a class in TypeScript: The syntax of the class should look very similar to readers who come from an object-oriented background. To define a class, we use a class keyword followed by the name of the class. The News class has three member properties and one method. Each member has a type assigned to it and has an access modifier to define the scope. On line 10, we create an object of a class with the new keyword. Classes in TypeScript also have the concept of a constructor, where we can initialize some properties at the time of object creation. Access modifiers Once the object is created, we can access the public members of the class with the dot operator. Note that we cannot access the author property with the espn object because this property is defined as private. TypeScript provides three types of access modifiers. Public Any property defined with the public keyword will be freely accessible outside the class. As we saw in the previous example, all the variables marked with the public keyword were available outside the class in an object. Note that TypeScript assigns public as a default access modifier if we do not assign any explicitly. This is because the default JavaScript behavior is to have everything public. Private When a property is marked as private, it cannot be accessed outside of the class. The scope of a private variable is only inside the class when using TypeScript. In JavaScript, as we do not have access modifiers, private members are treated similarly to public members. Protected The protected keyword behaves similarly to private, with the exception that protected variables can be accessed in the derived classes. The following is one such example: class base{ protected id: number; } class child extends base{ name: string; details():string{ return `${name} has id: ${this.id}` } } In the preceding code, we extend the child class with the base class and have access to the id property inside the child class. If we create an object of the child class, we will still not have access to the id property outside. Readonly As the name suggests, a property with a readonly access modifier cannot be modified after the value has been assigned to it. The value assigned to a readonly property can only happen at the time of variable declaration or in the constructor. In the above code, line 5 gives an error stating that property name is readonly, and cannot be an assigned value. Transpiled JavaScript from classes While learning TypeScript, it is important to remember that TypeScript is a superset of JavaScript and not a new language on its own. Browsers can only understand JavaScript, so it is important for us to understand the JavaScript that is transpiled by TypeScript. TypeScript provides an option to generate JavaScript based on the ECMA standards. You can configure TypeScript to transpile into ES5 or ES6 (ES 2015) and even ES3 JavaScript by using the flag target in the tsconfig.json file. The biggest difference between ES5 and ES6 is with regard to the classes, let, and const keywords which were introduced in ES6. Even though ES6 has been around for more than a year, most browsers still do not have full support for ES6. So, if you are creating an application that would target older browsers as well, consider having the target as ES5. So, the JavaScript that's generated will be different based on the target setting. Here, we will take an example of class in TypeScript and generate JavaScript for both ES5 and ES6. The following is the class definition in TypeScript: This is the same code that we saw when we introduced classes in the Understanding Classes section. Here, we have a class named News that has three members, two of which are public and one private. The News class also has a format method, which returns a string concatenated from the member variables. Then, we create an object of the News class in line 10 and assign values to public properties. In the last line, we call the format method to print the result. Now let's look at the JavaScript transpiled by TypeScript compiler for this class. ES6 JavaScript ES6, also known as ES 2015, is the latest version of JavaScript, which provides many new features on top of ES5. Classes are one such feature; JavaScript did not have classes prior to ES6. The following is the code generated from the TypeScript class, which we saw previously: If you compare the preceding code with TypeScript code, you will notice minor differences. This is because classes in TypeScript and JavaScript are similar, with just types and access modifiers additional in TypeScript. In JavaScript, we do not have the concept of declaring public members. The author variable, which was defined as private and was initialized at its declaration, is converted to a constructor initialization in JavaScript. If we had not have initialized author, then the produced JavaScript would not have added author in the constructor. ES5 JavaScript ES5 is the most popular JavaScript version supported in browsers, and if you are developing an application that has to support the majority of browser versions, then you need to transpile your code to the ES5 version. This version of JavaScript does not have classes, and hence the transpiled code converts classes to functions, and methods inside the classes are converted to prototypically defined methods on the functions. The following is the code transpiled when we have the target set as ES5 in the TypeScript compiler options: As discussed earlier, the basic difference is that the class is converted to a function. The interesting aspect of this conversion is that the News class is converted to an immediately invoked function expression (IIFE). An IIFE can be identified by the parenthesis at the end of the function declaration, as we see in line 9 in the preceding code snippet. IIFEs cause the function to be executed immediately and help to maintain the correct scope of a function rather than declaring the function in a global scope. Another difference was how we defined the method format in the ES5 JavaScript. The prototype keyword is used to add the additional behavior to the function, which we see here. A couple of other differences you may have noticed include the change of the let keyword to var, as let is not supported in ES5. All variables in ES5 are defined with the var keyword. Also, the format method now does not use a template string, but standard string concatenation to print the output. TypeScript does a good job of transpiling the code to JavaScript while following recommended practices. This helps in making sure we have a robust and reusable code with minimum error cases. If you found this tutorial useful, make sure you check out the book TypeScript 2.x By Example for more hands-on tutorials on how to effectively leverage the power of TypeScript to develop and deploy state-of-the-art web applications. How to install and configure TypeScript Understanding Patterns and Architectures in TypeScript Writing SOLID JavaScript code with TypeScript

Any good data mining project is built on a robust data mining architecture. Without it, your project might well be time-consuming, overly complicated or simply inaccurate. Whether you're new to data mining or want to re-familiarize yourself with what the structure of a data mining architecture should look like, you've come to the right place. Of course, this is just a guide to what a data mining architecture should look like. You'll need to be aware of how this translates to your needs and situation. This has been taken from Data Mining with R. Find it here. The core components of a data mining architecture Let's first gain a general view on the main components of a data mining architecture. It is basically composed of all of the basic elements you will need to perform the activities described in the previous chapter. As a minimum set of components, the following are usually considered: Data sources Data warehouse Data mining engine User interface Below is a diagram of a data mining architecture. You can see how each of the elements fit together: Before we get into the details of each of the components of a data mining architecture, let's first briefly look at how these components fit together: Data sources: These are all the possible sources of small bits of information to be analyzed. Data sources feed our data warehouses and are fed by the data produced from our activity toward the user interface. Data warehouse: This is where the data is stored when acquired from data sources. Data mining engine: This contains all of the logic and the processes needed to perform the actual data mining activity, taking data from the data warehouse. User interface: The front office of our machine, which allows the user to interact with the data mining engine, creating data that will be stored within the data warehouse and that could become part of the big ocean of data sources. We'll now delve a little deeper into each of these elements, starting with data sources. How data sources fit inside the data mining architecture Data sources are everywhere. This is becoming more and more true everyday thanks to the the internet of things. Now that every kind of object can be connected to the internet, we can collect data from a huge range of new physical sources. This data can come in a form already feasible for being collected and stored within our databases, or in a form that needs to be further modified to become usable for our analyses. We can, therefore, see that between our data sources and the physical data warehouse where they are going to be stored, a small components lies, which is the set of tools and software needed to make data coming from sources storable. We should note something here—we are not talking about data cleaning and data validation. Those activities will be performed later on by our data mining engine which retrieves data from the data warehouse. Types of data sources There are a range of data sources. Each type will require different data modelling techniques. Getting this wrong could seriously hamper your data mining projects, so an awareness of how data sources differ is actually really important. Unstructured data sources Unstructured data sources are data sources missing a logical data model. Whenever you find a data source where no particular logic and structure is defined to collect, store, and expose it, you are dealing with an unstructured data source. The most obvious example of an unstructured data source is a written document. That document has a lot of information in it, but there's no structure that defines and codifies how information is stored. There are some data modeling techniques that can be useful here. There are some that can even derive structured data from unstructured data. This kind of analysis is becoming increasingly popular as companies seek to use 'social listening' to understand sentiment on social media. Structured data sources Structured data sources are highly organized. These kinds of data sources follow a specific data model, and the engine which makes the storing activity is programmed to respect this model. A well-known data model behind structured data is the so-called relational model of data. Following this model, each table has to represent an entity within the considered universe of analysis. Each entity will then have a specific attribute within each column, and a related observation within each row. Finally, each entity can be related to the others through key attributes. We can think of an example of a relational database of a small factory. Within this database, we have a table recording all customers orders and one table recording all shipments. Finally, a table recording the warehouse's movements will be included. Within this database, we will have: The warehouse table linked to the shipment table through the product_code attribute The shipment table linked to the customer table through the shipment_code attribute It can be easily seen that a relevant advantage of this model is the possibility to easily perform queries within tables, and merges between them. The cost to analyze structured data is far lower than the one to be considered when dealing with unstructured data. Key issues of data sources When dealing with data sources and planning their acquisition into your data warehouse, some specific aspects need to be considered: Frequency of feeding: Is the data updated with a frequency feasible for the scope of your data mining activity? Volume of data: Can the volume of data be handled by your system, or it is too much? This is often the case for unstructured data, which tends to occupy more space for a given piece of information. Data format: Is the data format readable by your data warehouse solution, and subsequently, by your data mining engine? A careful evaluation of these three aspects has to be performed before implementing the data acquisition phase, to avoid relevant problems during the project. How databases and data warehouses fit in the data mining architecture What is a data warehouse, and how is it different from a simple database? A data warehouse is a software solution aimed at storing usually great amounts of data properly related among them and indexed through a time-related index. We can better understand this by looking at the data warehouse's cousin: the operational database. These kinds of instruments are usually of small dimensions, and aimed at storing and inquiring data, overwriting old data when new data is available. Data warehouses are therefore usually fed by databases, and stores data from those kinds of sources ensuring a historical depth to them and read-only access from other users and software applications. Moreover, data warehouses are usually employed at a company level, to store, and make available, data from (and to) all company processes, while databases are usually related to one specific process or task. How do you use a data warehouse for your data mining project? You're probably not going to use a data warehouse for your data mining process. More specicially, data will be made available via a data mart. A data mart is a partition or a sub-element of a data warehouse. The data marts are set of data that are feed directly from the data warehouse, and related to a specific company area or process. A real-life example is the data mart created to store data related to default events for the purpose of modeling customers probability of default. This kind of data mart will collect data from different tables within the data warehouse, properly joining them into new tables that will not communicate with the data warehouse one. We can therefore consider the data mart as an extension of the data warehouse. Data warehouses are usually classified into three main categories: One-level architecture where only a simple database is available and the data warehousing activity is performed by the mean of a virtual component Two-level architecture composed of a group of operational databases that are related to different activities, and a proper data warehouse is available Three-level architecture with one or more operational database, a reconciled database and a proper data warehouse Let's now have a closer look to those three different types of data warehouse. One-level database This is for sure the most simple and, in a way, primitive model. Within one level data warehouses, we actually have just one operational database, where data is written and read, mixing those two kinds of activities. A virtual data warehouse layer is then offered to perform inquiry activities. This is a primitive model for the simple reason that it is not able to warrant the appropriate level of segregation between live data, which is the one currently produced from the process, and historical data. This model could therefore produce inaccurate data and even a data loss episode. This model would be particularly dangerous for data mining activity, since it would not ensure a clear segregation between the development environment and the production one. Two-level database This more sophisticated model encompasses a first level of operational databases, for instance, the one employed within marketing, production, and accounting processes, and a proper data warehouse environment. Within this solution, the databases are to be considered like feeding data sources, where the data is produced, possibly validated, and then made available to the data warehouse. The data warehouse will then store and freeze data coming from databases, for instance, with a daily frequency. Every set of data stored within a day will be labeled with a proper attribute showing the date of record. This will later allow us to retrieve records related to a specific time period in a sort of time machine functionality. Going back to our previous probability of default example, this kind of functionality will allow us to retrieve all default events that have occurred within a given time period, constituting the estimation sample for our model. Two-level architecture is an optimal solution for data mining processes, since they will allow us to provide a safe environment, the previously mentioned data mart, to develop data mining activity, without compromising the quality of data residing within the remaining data warehouses and within the operational databases. Three-level database Three-level databases are the most advanced ones. The main difference between them and the two-level ones is the presence of the reconciliation stage, which is performed through Extraction, Transformation, and Load (ETL) instruments. To understand the relevance of such kinds of instruments, we can resort to a practical example once again, and to the one we were taking advantage of some lines previously: the probability of the default model. Imagine we are estimating such kind of model for customers clustered as large corporate, for which public forecasts, outlooks and ratings are made available by financial analyses companies like Moody's, Standard & Poor, and similar. Since this data could be reasonably related to the probability of default of our customers, we would probably be interested in adding them to our estimation database. This can be easily done through the mean of those ETL instruments. These instruments will ensure, within the reconciliation stage, that data gathered from internal sources, such as personal data and default events data, will be properly matched with the external information we have mentioned. Moreover, even within internal data fields only, those instruments will ensure the needed level of quality and coherence among different sources, at least within the data warehouse environment. Data warehouse technologies We are now going to look a bit more closely at the actual technology -  most of which is open source. A proper awareness of their existence and main features should be enough, since you will usually be taking input data from them through an interface provided by your programming language. Nevertheless, knowing what's under the hood is pretty useful... SQL SQL stands for Structured Query Language, and identifies what has been for many years the standard within the field of data storage. The base for this programming language, employed for storing and querying data, are the so-called relational data bases. The theory behind these data bases was first introduced by IBM engineer Edgar F. Codd, and is based on the following main elements: Tables, each of which represent an entity Columns, each of which represent an attribute of the entity Rows, each one representing a record of the entity Key attributes, which permit us to relate two or more tables together, establishing relations between them Starting from these main elements, SQL language provides a concise and effective way to query and retrieve this data. Moreover, basilar data munging operations, such as table merging and filtering, are possible through SQL language. As previously mentioned, SQL and relational databases have formed the vast majority of data warehouse systems around the world for many, many years. A really famous example of SQL-based data storing products is the well-known Microsoft Access software. In this software, behind the familiar user interface, hide SQL codes to store, update, and retrieve user's data. MongoDB While SQL-based products are still very popular, NoSQL technology has been going for a long time now, showing its relevance and effectiveness. Behind this acronym stands all data storing and managing solutions not based on the relational paradigm and its main elements. Among this is the document-oriented paradigm, where data is represented as documents, which are complex virtual objects identified with some kind of code, and without a fixed scheme. A popular product developed following this paradigm is MongoDB. This product stores data, representing it in the JSON format. Data is therefore organized into documents and collections, that is, a set of documents. A basic example of a document is the following: { name: "donald" , surname: "duck", style: "sailor", friends: ["mickey mouse" , "goofy", "daisy"] } As you can see, even from this basic example, the MongoDB paradigm will allow you to easily store data even with a rich and complex structure. Hadoop Hadoop is a leading technology within the field of data warehouse systems, mainly due to its ability to effectively handle large amounts of data. To maintain this ability, Hadoop fully exploits the concept of parallel computing by means of a central master that divides the all needed data related to a job into smaller chunks to be sent to two or more slaves. Those slaves are to be considered as nodes within a network, each of them working separately and locally. They can actually be physically separated pieces of hardware, but even core within a CPU (which is usually considered pseudo-parallel mode). At the heart of Hadoop is the MapReduce programming model. This model, originally conceptualized by Google, consists of a processing layer, and is responsible for moving the data mining activity close to where data resides. This minimizes the time and cost needed to perform computation, allowing for the possibility to scale the process to hundreds and hundreds of different nodes. Read next: Why choose R for your data mining project [link] The data mining engine that drives a data mining architecture The data mining engine is the true heart of our data mining architecture. It consists of tools and software employed to gain insights and knowledge from data acquired from data sources, and stored within data warehouses. What makes a data mining engine? As you should be able to imagine at this point, a good data mining engine is composed of at least three components: An interpreter, able to transmit commands defined within the data mining engine to the computer Some kind of gear between the engine and the data warehouse to produce and handle communication in both directions A set of instructions, or algorithms, needed to perform data mining activities Let's take a look at these components in a little more detail. The interpreter The interpreter carries out instructions coming from a higher-level programming language, and then translates them into instructions understandable from the piece of hardware it is running on, and transmits them to it. Obtaining the interpreter for the language you are going to perform data mining with is usually as simple as obtaining the language itself. In the case of our beloved R language, installing the language will automatically install the interpreter as well. The interface between the engine and the data warehouse If the interpreter was previously introduced, this interface we are talking about within this section is a new character within our story. The interface we are talking about here is a kind of software that enables your programming language to talk with the data warehouse solution you have been provided with for your data mining project. To exemplify the concept, let's consider a setup adopting as a data mining engine, a bunch of R scripts, with their related interpreter, while employing an SQL-based database to store data. In this case, what would be the interface between the engine and the data warehouse? It could be, for instance, the RODBC package, which is a well-established package designed to let R users connect to remote servers, and transfer data from those servers to their R session. By employing this package, it will also be possible to write data to your data warehouse. This packages works exactly like a gear between the R environment and the SQL database. This means you will write your R code, which will then be translated into a readable language from the database and sent to him. For sure, this translation also works the other way, meaning that results coming from your instructions, such as new tables of results from a query, will be formatted in a way that's readable from the R environment and conveniently shown to the user. The data mining algorithms This last element of the engine is the actual core topic of the book you are reading—the data mining algorithms. To help you gain an organic and systematic view of what we have learned so far, we can consider that these algorithms will be the result of the data modelling phase described in the previous chapter in the context of the CRISP-DM methodology description. This will usually not include code needed to perform basic data validation treatments, such as integrity checking and massive merging among data from different sources, since those kind of activities will be performed within the data warehouse environment. This will be especially true in cases of three-level data warehouses, which have a dedicated reconciliation layer. The user interface - the bit that makes the data mining architecture accessible Until now, we have been looking at the back office of our data mining architecture, which is the part not directly visible to its end user. Imagine this architecture is provided to be employed by someone not skilled enough to work on the data mining engine itself; we will need some way to let this user interact with the architecture in the right way, and discover the results of its interaction. This is what a user interface is all about. Clarity and simplicity There's a lot to be said about UI design that site more in the field of design than data analysis. Clearly, those fields are getting blurred as data mining becomes more popular, and as 'self-service' analytics grows as a trend. However, the fundamental elements of a UI is clarity and simplicity. What this means is that it is designed with purpose and usage in mind. What do you want to see? What do you want to be able to do with your data? Ask yourself this question: how many steps you need to perform to reach the objective you want to reach with the product? Imagine evaluating a data mining tool, and particularly, its data import feature. Evaluating the efficiency of the tool in this regard would involve answering the following question: how many steps do I need to perform to import a dataset into my data mining environment? Every piece is important in the data mining architecture When it comes to data mining architecture, it's essential that you don't overlook either part of it. Every component is essential. Of course, like any other data mining project, understanding what your needs are - and the needs of those in your organization - are going to inform how you build each part. But fundamentally the principles behind a robust and reliable data mining architecture will always remain the same. Read more: Expanding Your Data Mining Toolbox [link]

TensorFlow models can be used in applications running on mobile and embedded platforms. TensorFlow Lite and TensorFlow Mobile are two flavors of TensorFlow for resource-constrained mobile devices. TensorFlow Lite supports a subset of the functionality compared to TensorFlow Mobile. It results in better performance due to smaller binary size with fewer dependencies. The article covers topics for training a model to integrate TensorFlow into an application. The model can then be saved and used for inference and prediction in the mobile application. [box type="note" align="" class="" width=""]This article is an excerpt from the book Mastering TensorFlow 1.x written by Armando Fandango. This book will help you leverage the power of TensorFlow and Keras to build deep learning models, using concepts such as transfer learning, generative adversarial networks, and deep reinforcement learning.[/box] To learn how to use TensorFlow models on mobile devices, following topics are covered: TensorFlow on mobile platforms TF Mobile in Android apps TF Mobile demo on Android TF Mobile demo on iOS TensorFlow Lite TF Lite demo on Android TF Lite demo on iOS TensorFlow on mobile platforms TensorFlow can be integrated into mobile apps for many use cases that involve one or more of the following machine learning tasks: Speech recognition Image recognition Gesture recognition Optical character recognition Image or text classification Image, text, or speech synthesis Object identification To run TensorFlow on mobile apps, we need two major ingredients: A trained and saved model that can be used for predictions A TensorFlow binary that can receive the inputs, apply the model, produce the predictions, and send the predictions as output The high-level architecture looks like the following figure: The mobile application code sends the inputs to the TensorFlow binary, which uses the trained model to compute predictions and send the predictions back. TF Mobile in Android apps The TensorFlow ecosystem enables it to be used in Android apps through the interface class  TensorFlowInferenceInterface, and the TensorFlow Java API in the jar file libandroid_tensorflow_inference_java.jar. You can either use the jar file from the JCenter, download a precompiled jar from ci.tensorflow.org, or build it yourself. The inference interface has been made available as a JCenter package and can be included in the Android project by adding the following code to the build.gradle file: allprojects  { repositories  { jcenter() } } dependencies  { compile  'org.tensorflow:tensorflow-android:+' } Note : Instead of using the pre-built binaries from the JCenter, you can also build them yourself using Bazel or Cmake by following the instructions at this link: https://github.com/tensorflow/tensorflow/blob/r1.4/ tensorflow/contrib/android/README.md Once the TF library is configured in your Android project, you can call the TF model with the following four steps:  Load the model: TensorFlowInferenceInterface  inferenceInterface  = new  TensorFlowInferenceInterface(assetManager,  modelFilename);  Send the input data to the TensorFlow binary: inferenceInterface.feed(inputName, floatValues,  1,  inputSize,  inputSize,  3);  Run the prediction or inference: inferenceInterface.run(outputNames,  logStats);  Receive the output from the TensorFlow binary: inferenceInterface.fetch(outputName,  outputs); TF Mobile demo on Android In this section, we shall learn about recreating the Android demo app provided by the TensorFlow team in their official repo. The Android demo will install the following four apps on your Android device: TF  Classify: This is an object identification app that identifies the images in the input from the device camera and classifies them in one of the pre-defined classes. It does not learn new types of pictures but tries to classify them into one of the categories that it has already learned. The app is built using the inception model pre-trained by Google. TF  Detect: This is an object detection app that detects multiple objects in the input from the device camera. It continues to identify the objects as you move the camera around in continuous picture feed mode. TF  Stylize: This is a style transfer app that transfers one of the selected predefined styles to the input from the device camera. TF  Speech: This is a speech recognition app that identifies your speech and if it matches one of the predefined commands in the app, then it highlights that specific command on the device screen. Note: The sample demo only works for Android devices with an API level greater than 21 and the device must have a modern camera that supports FOCUS_MODE_CONTINUOUS_PICTURE. If your device camera does not have this feature supported, then you have to add the path submitted to TensorFlow by the author: https://github.com/ tensorflow/tensorflow/pull/15489/files. The easiest way to build and deploy the demo app on your device is using Android Studio. To build it this way, follow these steps:  Install Android Studio. We installed Android Studio on Ubuntu 16.04 from the instructions at the following link: https://developer.android.com/studio/ install.html  Check out the TensorFlow repository, and apply the patch mentioned in the previous tip. Let's assume you checked out the code in the tensorflow folder in your home directory.  Using Android Studio, open the Android project in the path ~/tensorflow/tensorflow/examples/Android.     Your screen will look similar to this:  Expand the Gradle Scripts option from the left bar and then open the  build.gradle file.  In the build.gradle file, locate the def  nativeBuildSystem definition and set it to 'none'. In the version of  the code we checked out, this definition is at line 43: def  nativeBuildSystem  =  'none'  Build the demo and run it on either a real or simulated device. We tested the app on these devices: 7.  You can also build the apk and install the apk file on the virtual or actual connected device. Once the app installs on the device, you will see the four apps we discussed earlier: You can also build the whole demo app from the source using Bazel or Cmake by following the instructions at this link: https://github.com/tensorflow/tensorflow/tree/r1.4/tensorflow/examples/android TF Mobile in iOS apps TensorFlow enables support for iOS apps by following these steps:  Include TF Mobile in your app by adding a file named Profile in the root directory of your project. Add the following content to the Profile: target  'Name-Of-Your-Project' pod  'TensorFlow-experimental'  Run the pod  install command to download and install the TensorFlow Experimental pod.  Run the myproject.xcworkspace command to open the workspace so you can add the      prediction code to your application logic. Note: To create your own TensorFlow binaries for iOS projects, follow the instructions at this link: https://github.com/tensorflow/tensorflow/ tree/master/tensorflow/examples/ios Once the TF library is configured in your iOS project, you can call the TF model with the following four steps:  Load the model: PortableReadFileToProto(file_path,  &tensorflow_graph);  Create a session: tensorflow::Status  s  =  session->Create(tensorflow_graph);  Run the prediction or inference and get the outputs: std::string  input_layer  =  "input"; std::string  output_layer  =  "output"; std::vector<tensorflow::Tensor>  outputs; tensorflow::Status  run_status  =  session->Run( {{input_layer,  image_tensor}}, {output_layer},  {},  &outputs);  Fetch the output data: tensorflow::Tensor*  output  =  &outputs[0]; TF Mobile demo on iOS In order to build the demo on iOS, you need Xcode 7.3 or later. Follow these steps to build the iOS demo apps:  Check out the TensorFlow code in a tensorflow folder in your home directory.  Open a terminal window and execute the following commands from your home folder to download the Inception V1 model, extract the label and graph files, and move these files into the data folders inside the sample app code: $ mkdir -p ~/Downloads $ curl -o ~/Downloads/inception5h.zip https://storage.googleapis.com/download.tensorflow.org/models/incep tion5h.zip && unzip ~/Downloads/inception5h.zip -d ~/Downloads/inception5h $ cp ~/Downloads/inception5h/* ~/tensorflow/tensorflow/examples/ios/benchmark/data/ $ cp ~/Downloads/inception5h/* ~/tensorflow/tensorflow/examples/ios/camera/data/ $ cp ~/Downloads/inception5h/* ~/tensorflow/tensorflow/examples/ios/simple/data/  Navigate to one of the sample folders and download the experimental pod: $ cd ~/tensorflow/tensorflow/examples/ios/camera $ pod install  Open the Xcode workspace: $ open tf_simple_example.xcworkspace  Run the sample app in the device simulator. The sample app will appear with a Run Model button. The camera app requires an Apple device to be connected, while the other two can run in a simulator too. TensorFlow Lite TF Lite is the new kid on the block and still in the developer view at the time of writing this book. TF Lite is a very small subset of TensorFlow Mobile and TensorFlow, so the binaries compiled with TF Lite are very small in size and deliver superior performance. Apart from reducing the size of binaries, TensorFlow employs various other techniques, such as: The kernels are optimized for various device and mobile architectures The values used in the computations are quantized The activation functions are pre-fused It leverages specialized machine learning software or hardware available on the device, such as the Android NN API The workflow for using the models in TF Lite is as follows:  Get the model: You can train your own model or pick a pre-trained model available from different sources, and use the pre-trained as is or retrain it with your own data, or retrain after modifying some parts of the model. As long as you have a trained model in the file with an extension .pb or .pbtxt, you are good to proceed to the next step. We learned how to save the models in the previous chapters.  Checkpoint the model: The model file only contains the structure of the graph, so you need to save the checkpoint file. The checkpoint file contains the serialized variables of the model, such as weights and biases. We learned how to save a checkpoint in the previous chapters.  Freeze the model: The checkpoint and the model files are merged, also known as freezing the graph. TensorFlow provides the freeze_graph tool for this step, which can be executed as follows: $ freeze_graph --input_graph=mymodel.pb --input_checkpoint=mycheckpoint.ckpt --input_binary=true --output_graph=frozen_model.pb --output_node_name=mymodel_nodes  Convert the model: The frozen model from step 3 needs to be converted to TF Lite format with the toco tool provided by TensorFlow: $ toco --input_file=frozen_model.pb --input_format=TENSORFLOW_GRAPHDEF --output_format=TFLITE --input_type=FLOAT --input_arrays=input_nodes --output_arrays=mymodel_nodes --input_shapes=n,h,w,c  The .tflite model saved in step 4 can now be used inside an Android or iOS app that employs the TFLite binary for inference. The process of including the TFLite binary in your app is continuously evolving, so we recommend the reader follows the information at this link to include the TFLite binary in your Android or iOS app: https://github.com/tensorflow/tensorflow/tree/master/ tensorflow/contrib/lite/g3doc Generally, you would use the graph_transforms:summarize_graph tool to prune the model obtained in step 1. The pruned model will only have the paths that lead from input to output at the time of inference or prediction. Any other nodes and paths that are required only for training or for debugging purposes, such as saving checkpoints, are removed, thus making the size of the final model very small. The official TensorFlow repository comes with a TF Lite demo that uses a pre-trained mobilenet to classify the input from the device camera in the 1001 categories. The demo app displays the probabilities of the top three categories. TF Lite Demo on Android To build a TF Lite demo on Android, follow these steps: Install Android Studio. We installed Android Studio on Ubuntu 16.04 from the instructions at the following link: https://developer.android.com/studio/ install.html Check out the TensorFlow repository, and apply the patch mentioned in the previous tip. Let's assume you checked out the code in the tensorflow folder in your home directory. Using Android Studio, open the Android project from the path ~/tensorflow/tensorflow/contrib/lite/java/demo. If it complains about a missing SDK or Gradle components, please install those components and sync Gradle. Build the project and run it on a virtual device with API > 21. We received the following warnings, but the build succeeded. You may want to resolve the warnings if the build fails: Warning:The  Jack  toolchain  is  deprecated  and  will  not run.  To  enable  support  for  Java  8 language  features  built into  the  plugin,  remove  'jackOptions  {  ...  }'  from  your build.gradle  file, and  add android.compileOptions.sourceCompatibility  1.8 android.compileOptions.targetCompatibility  1.8 Note:  Future  versions  of  the  plugin  will  not  support  usage 'jackOptions'  in  build.gradle. To learn  more,  go  to https://d.android.com/r/tools/java-8-support-message.html Warning:The  specified  Android  SDK  Build  Tools  version (26.0.1)  is  ignored,  as  it  is  below  the minimum  supported version  (26.0.2)  for  Android  Gradle  Plugin  3.0.1. Android  SDK  Build  Tools 26.0.2  will  be  used. To  suppress  this  warning,  remove  "buildToolsVersion '26.0.1'"  from  your  build.gradle  file,  as  each  version  of the  Android  Gradle  Plugin  now  has  a  default  version  of the  build  tools. TF Lite demo on iOS In order to build the demo on iOS, you need Xcode 7.3 or later. Follow these steps to build the iOS demo apps:  Check out the TensorFlow code in a  tensorflow folder in your home directory.  Build the TF Lite binary for iOS from the instructions at this link: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite  Navigate to the sample folder and download the pod: $ cd ~/tensorflow/tensorflow/contrib/lite/examples/ios/camera $ pod install  Open the Xcode workspace: $ open tflite_camera_example.xcworkspace  Run the sample app in the device simulator. We learned about using TensorFlow models on mobile applications and devices. TensorFlow provides two ways to run on mobile devices: TF Mobile and TF Lite. We learned how to build TF Mobile and TF Lite apps for iOs and Android. We used TensorFlow demo apps as an example.   If you found this post useful, do check out the book Mastering TensorFlow 1.x  to skill up for building smarter, faster, and efficient machine learning and deep learning systems. The 5 biggest announcements from TensorFlow Developer Summit 2018 Getting started with Q-learning using TensorFlow Implement Long-short Term Memory (LSTM) with TensorFlow  

When one mentions server-side JavaScript technology, Node.js is what comes to our mind first. Node.js is an extremely powerful and robust platform. Using this JavaScript platform, we can build server-side applications very easily. In today's tutorial, we will focus on creating a chat application that uses Kotlin by using the Node.js technology. So, basically, we will transpile Kotlin code to JavaScript on the server side. This article is an excerpt from the book, Kotlin Blueprints, written by Ashish Belagali, Hardik Trivedi, and Akshay Chordiya. With this book, you will get to know the building blocks of Kotlin and best practices when using quality world-class applications. Kotlin is a modern language and is gaining popularity in the JavaScript community day by day. The Kotlin language with modern features and statically typed; is superior to JavaScript. Similar to JavaScript, developers that know Kotlin can use the same language skills on both sides but, they also have the advantage of using a better language. The Kotlin code gets transpiled to JavaScript and that in turn works with the Node.js. This is the mechanism that lets you use the Kotlin code to work with a server-side technology, such as Node.js. Creating a chat application Our chat app will have following functionalities: User can log in by entering their nickname User can see the list of online users User will get notified when a new user joins User can receive chats from anyone User can perform a group chat in a chat room User will receive a notification when any user leaves the chat To visualize the app that we will develop, take a look at the following screenshots. The following screenshot is a page where the user will enter a nickname and gain an entry in our chat app: In the following screen, you can see a chat window and a list of online users: We have slightly configured this application in a different way. We have kept the backend code module and frontend code module separate using the following method: Create a new project named kotlin_node_chat_app Now, create a new Gradle module named backend and select Kotlin (JavaScript) under the libraries and additional information window, and follow the remaining steps. Similarly, also create a Gradle module named webapp. The backend module will contain all the Kotlin code that will be converted into Node.JS code later, and the webapp module will contain all the Kotlin code that will later be converted into the JavaScript code. We have referred to the directory structure from Github. After performing the previous steps correctly, your project will have three build.gradle files. We have highlighted all three files in the project explorer section, as shown in the following screenshot: Setting up the Node.js server We need to initialize our root directory for the node. Execute npm init and it will create package.json. Now our login page is created. To run it, we need to set up the Node.js server. We want to create the server in such a way that by executing npm start, it should start the server. To achieve it, our package.json file should look like the following piece of code: { "name": "kotlin_node_chat_app", "version": "1.0.0", "description": "", "main": "backend/server/app.js", "scripts": { "start": "node backend/server/app.js" }, "author": "Hardik Trivedi", "license": "ISC", "dependencies": { "ejs": "^2.5.7", "express": "^4.16.2", "kotlin": "^1.1.60", "socket.io": "^2.0.4" } } We have specified a few dependencies here as well: EJS to render HTML pages Express.JS as its framework, which makes it easier to deal with Node.js Kotlin, because, ultimately, we want to write our code into Kotlin and want it compiled into the Node.js code Socket.IO to perform chat Execute npm install on the Terminal/Command Prompt and it should trigger the download of all these dependencies. Specifying the output files Now, it's very important where your output will be generated once you trigger the build. For that, build.gradle will help us. Specify the following lines in your module-level build.gradle file. The backend module's build.gradle will have the following lines of code: compileKotlin2Js { kotlinOptions.outputFile = "${projectDir}/server/app.js" kotlinOptions.moduleKind = "commonjs" kotlinOptions.sourceMap = true } The webapp module's build.gradle will have the following lines of code: compileKotlin2Js { kotlinOptions.metaInfo = true kotlinOptions.outputFile = "${projectDir}/js/main.js" kotlinOptions.sourceMap = true kotlinOptions.main = "call" } In both the compileKotlin2Js nodes, kotlinOptions.outputFile plays a key role. This basically tells us that once Kotlin's code gets compiled, it will generate app.js and main.js for Node.js and JavaScript respectively. In the index.ejs file, you should define a script tag to load main.js. It will look something like the following line of code: <script type="text/javascript" src="js/main.js"></script> Along with this, also specify the following two tags: <script type="text/javascript" src="lib/kotlin/kotlin.js"></script> <script type="text/javascript" src="lib/kotlin/kotlinx-html-js.js"> </script> Examining the compilation output The kotlin.js and kotlinx-html-js.js files are nothing but the Kotlin output files. It's not compilation output, but actually transpiled output. The following are output compilations: kotlin.js: This is the runtime and standard library. It doesn't change between applications, and it's tied to the version of Kotlin being used. {module}.js: This is the actual code from the application. All files are compiled into a single JavaScript file that has the same name as the module. {file}.meta.js: This metafile will be used for reflection and other functionalities. Let's assume our final Main.kt file will look like this: fun main(args: Array<String>) { val socket: dynamic = js("window.socket") val chatWindow = ChatWindow { println("here") socket.emit("new_message", it) } val loginWindow = LoginWindow { chatWindow.showChatWindow(it) socket.emit("add_user", it) } loginWindow.showLogin() socket.on("login", { data -> chatWindow.showNewUserJoined(data) chatWindow.showOnlineUsers(data) }) socket.on("user_joined", { data -> chatWindow.showNewUserJoined(data) chatWindow.addNewUsers(data) }) socket.on("user_left", { data -> chatWindow.showUserLeft(data) }) socket.on("new_message", { data -> chatWindow.showNewMessage(data) }) } For this, inside main.js, our main function will look like this: function main(args) { var socket = window.socket; var chatWindow = new ChatWindow(main$lambda(socket)); var loginWindow = new LoginWindow(main$lambda_0(chatWindow, socket)); loginWindow.showLogin(); socket.on('login', main$lambda_1(chatWindow)); socket.on('user_joined', main$lambda_2(chatWindow)); socket.on('user_left', main$lambda_3(chatWindow)); socket.on('new_message', main$lambda_4(chatWindow)); } The actual main.js file will be bulkier because it will have all the code transpiled, including other functions and LoginWindow and ChatWindow classes. Keep a watchful eye on how the Lambda functions are converted into simple JavaScript functions. Lambda functions, for all socket events, are transpiled into the following piece of code: function main$lambda_1(closure$chatWindow) { return function (data) { closure$chatWindow.showNewUserJoined_qk3xy8$(data); closure$chatWindow.showOnlineUsers_qk3xy8$(data); }; } function main$lambda_2(closure$chatWindow) { return function (data) { closure$chatWindow.showNewUserJoined_qk3xy8$(data); closure$chatWindow.addNewUsers_qk3xy8$(data); }; } function main$lambda_3(closure$chatWindow) { return function (data) { closure$chatWindow.showUserLeft_qk3xy8$(data); }; } function main$lambda_4(closure$chatWindow) { return function (data) { closure$chatWindow.showNewMessage_qk3xy8$(data); }; } As can be seen, Kotlin aims to create very concise and readable JavaScript, allowing us to interact with it as needed. Specifying the router We need to write a behavior in the route.kt file. This will let the server know which page to load when any request hits the server. The router.kt file will look like this: fun router() { val express = require("express") val router = express.Router() router.get("/", { req, res -> res.render("index") }) return router } This simply means that whenever a get request with no name approaches the server, it should display an index page to the user. We are told to instruct the framework to refer to the router.kt file by writing the following line of code: app.use("/", router()) Starting the node server Now let's create a server. We should create an app.kt file under the backend module at the backend/src/kotlin path. Refer to the source code to verify. Write the following piece of code in app.kt: external fun require(module: String): dynamic external val process: dynamic external val __dirname: dynamic fun main(args: Array<String>) { println("Server Starting!") val express = require("express") val app = express() val path = require("path") val http = require("http") /** * Get port from environment and store in Express. */ val port = normalizePort(process.env.PORT) app.set("port", port) // view engine setup app.set("views", path.join(__dirname, "../../webapp")) app.set("view engine", "ejs") app.use(express.static("webapp")) val server = http.createServer(app) app.use("/", router()) app.listen(port, { println("Chat app listening on port http://localhost:$port") }) } fun normalizePort(port: Int) = if (port >= 0) port else 7000 These are multiple things to highlight here: external: This is basically an indicator for Kotlin that the line written along with this a pure JavaScript code. Also, when this code gets compiled into the respected language, the compiler understands that the class, function, or property written along with that will be provided by the developer, and so no JavaScript code should be generated for that invocation. The external modifier is automatically applied to nested declarations. For example, consider the following code block. We declare the class as external and automatically all its functions and properties are treated as external: external class Node { val firstChild: Node fun append(child: Node): Node fun removeChild(child: Node): Node // etc } dynamic: You will often see the usage of dynamic while working with JavaScript. Kotlin is a statically typed language, but it still has to interoperate with languages such as JavaScript. To support such use cases with a loosely or untyped programming language, dynamic is useful. It turns off Kotlin's type checker. A value of this type can be assigned to any variable or passed anywhere as a parameter. Any value can be assigned to a variable of dynamic type or passed to a function that takes dynamic as a parameter. Null checks are disabled for such values. require("express"): We typically use ExpressJS with Node.js. It's a framework that goes hand in hand with Node.js. It's designed with the sole purpose of developing web applications. A Node.js developer must be very familiar with it. process.env.PORT: This will find an available port on the server, as simple as that. This line is required if you want to deploy your application on a utility like Heroku. Also, notice the normalizePort function. See how concise it is. The if…else condition is written as an expression. No explicit return keyword is required. Kotlin compiler also identifies that if (port >= 0) port else 7000 will always return Int, hence no explicit return type is required. Smart, isn't it! __dirname: This is always a location where your currently executing script is present. We will use it to create a path to indicate where we have kept our web pages. app.listen(): This is a crucial one. It starts the socket and listens for the incoming request. It takes multiple parameters. Mainly, we will use two parameterized functions, that take the port number and connection callback as an argument. The app.listen() method is identical to http.Server.listen(). In Kotlin, it takes a Lambda function. Now, it's time to kick-start the server. Hit the Gradle by using ./gradlew build. All Kotlin code will get compiled into Node.js code. On Terminal, go to the root directory and execute npm start. You should be able to see the following message on your Terminal/Command Prompt: Creating a login page Now, let's begin with the login page. Along with that, we will have to enable some other settings in the project as well. If you refer to a screenshot that we mentioned at the beginning of the previous section, you can make out that we will have the title, the input filed, and a button as a part of the login page. We will create the page using Kotlin and the entire HTML tree structure, and by applying CSS to them, the will be part of our Kotlin code. For that, you should refer to the Main.kt and LoginWindow files. Creating an index.ejs file We will use EJS (effective JavaScript templating) to render HTML content on the page. EJS and Node.js go hand in hand. It's simple, flexible, easy to debug, and increases development speed. Initially, index.ejs would look like the following code snippet: <!DOCTYPE html> <html> <head> <meta name="viewport" content="width=device-width, initial- scale=1.0"/> </head> <body> <div id="container" class="mainContainer"> </div> </body> </html> The <div> tag will contain all different views, for example, the Login View, Chat Window View, and so on. Using DSL DSL stands for domain-specific language. As the name indicates, it gives you the feeling as if you are writing code in a language using terminology particular to a given domain without being geeky, but then, this terminology is cleverly embedded as a syntax in a powerful language. If you are from the Groovy community, you must be aware of builders. Groovy builders allow you to define data in a semi-declarative way. It's a kind of mini-language of its own. Builders are considered good for generating XML and laying out UI components. The Kotlin DSL uses Lambdas a lot. The DSL in Kotlin is a type-safe builder. It means we can detect compilation errors in IntelliJ's beautiful IDE. The type-check builders are much better than the dynamically typed builders of Groovy. Using kotlinx.html The DSL to build HTML trees is a pluggable dependency. We, therefore, need to set it up and configure it for our project. We are using Gradle as a build tool and Gradle has the best way to manage the dependencies. We will define the following line of code in our build.gradle file to use kotlinx.html: compile("org.jetbrains.kotlinx:kotlinx-html-js:$html_version") Gradle will automatically download this dependency from jcenter(). Build your project from menu Build | Build Project. You can also trigger a build from the terminal/command prompt. To build a project from the Terminal, go to the root directory of your project and then execute ./gradlew build. Now create the index.ejs file under the webapp directory. At this moment, your index.ejs file may look like the following: Inside your LoginWindow class file, you should write the following piece of code: class LoginWindow(val callback: (String) -> Unit) { fun showLogin() { val formContainer = document.getElementById("container") as HTMLDivElement val loginDiv = document.create.div { id = "loginDiv" h3(classes = "title") { +"Welcome to Kotlin Blueprints chat app" } input(classes = "nickNameInput") { id = "nickName" onInputFunction = onInput() maxLength = 16.toString() placeholder = "Enter your nick name" } button(classes = "loginButton") { +"Login" onClickFunction = onLoginButtonClicked() } } formContainer.appendChild(loginDiv) } } Observe how we have provided the ID, input types, and a default ZIP code value. A default ZIP code value is optional. Let's spend some time understanding the previous code. The div, input, button, and h3 all these are nothing but functions. They are basically Lambda functions. The following are the functions that use Lambda as the last parameter. You can call them in different ways: someFunction({}) someFunction("KotlinBluePrints",1,{}) someFunction("KotlinBluePrints",1){} someFunction{} Lambda functions Lambda functions are nothing but functions without a name. We used to call them anonymous functions. A function is basically passed into a parameter of a function call, but as an expression. They are very useful. They save us a lot of time by not writing specific functions in an abstract class or interface. Lambda usage can be as simple the following code snippet, where it seems like we are simply binding a block an invocation of the helloKotlin function: fun main(args: Array<String>) { val helloKotlin={println("Hello from KotlinBlueprints team!")} helloKotlin() } At the same time, lambda can be a bit complex as well, just like the following code block: fun <T> lock(lock: Lock, body: () -> T): T { lock.lock() try { return body() } finally { lock.unlock() } } In the previous function, we are acquiring a lock before executing a function and releasing it when the function gets executed. This way, you can synchronously call a function in a multithreaded environment. So, if we have a use case where we want to execute sharedObject.someCrucialFunction() in a thread-safe environment, we will call the preceding lock function like this: lock(lock,{sharedObject.someCrucialFunction()}) Now, the lambda function is the last parameter of a function call, so it can be easily written like this: lock(lock) { sharedObject.someCrucialFunction() } Look how expressive and easy to understand the code is. We will dig more into the Lambda in the upcoming section. Reading the nickname In the index.ejs page, we will have an input field with the ID nickName when it is rendered. We can simply read the value by writing the following lines of code: val nickName = (document.getElementById("nickName") as? HTMLInputElement)?.value However, to cover more possibilities, we have written it in a slightly different way. We have written it as if we are taking the input as an event. The following code block will continuously read the value that is entered into the nickName input field: private fun onInput(): (Event) -> Unit { return { val input = it.currentTarget as HTMLInputElement when (input.id) { "nickName" -> nickName = input.value "emailId" -> email = input.value } } } Check out, we have used the when function, which is a replacement for the switch case. The preceding code will check whether the ID of the element is nickName or emailId, and, based on that, it will assign the value to the respective objects by reading them from the in-out field. In the app, we will only have the nickname as the input file, but using the preceding approach, you can read the value from multiple input fields. In its simplest form, it looks like this: when (x) { 1 -> print("x == 1") 2 -> print("x == 2") else -> { // Note the block print("x is neither 1 nor 2") } } The when function compares its argument against all branches, top to bottom until some branch condition is met. The when function can be used either as an expression or as a statement. The else branch is evaluated if none of the other branch conditions are satisfied. If when is used as an expression, the else branch is mandatory, unless the compiler can prove that all possible cases are covered with branch conditions. If many cases should be handled in the same way, the branch conditions may be combined with a comma, as shown in the following code: when (x) { 0, 1 -> print("x == 0 or x == 1") else -> print("otherwise") } The following uses arbitrary expressions (not only constants) as branch conditions: when (x) { parseInt(s) -> print("s encodes x") else -> print("s does not encode x") } The following is used to check a value for being in or !in a range or a collection: when (x) { in 1..10 -> print("x is in the range") in validNumbers -> print("x is valid") !in 10..20 -> print("x is outside the range") else -> print("none of the above") } Passing nickname to the server Once our setup is done, we are able to start the server and see the login page. It's time to pass the nickname or server and enter the chat room. We have written a function named onLoginButtonClicked(). The body for this function should like this: private fun onLoginButtonClicked(): (Event) -> Unit { return { if (!nickName.isBlank()) { val formContainer = document.getElementById("loginDiv") as HTMLDivElement formContainer.remove() callback(nickName) } } } The preceding function does two special things: Smart casting Registering a simple callback Smart cast Unlike any other programming language, Kotlin also provides class cast support. The document.getElementById() method returns an Element type if instance. We basically want it to cast into HTMLDivElement to perform some <div> related operation. So, using as, we cast the Element into HTMLDivElement. With the as keyword, it's unsafe casting. It's always better to use as?. On a successful cast, it will give an instance of that class; otherwise, it returns null. So, while using as?, you have to use Kotlin's null safety feature. This gives your app a great safety net onLoginButtonClicked can be refactored by modifying the code a bit. The following code block is the modified version of the function. We have highlighted the modification in bold: private fun onLoginButtonClicked(): (Event) -> Unit { return { if (!nickName.isBlank()) { val formContainer = document.getElementById("loginDiv") as? HTMLDivElement formContainer?.remove() callback(nickName) } } } Registering a callback Oftentimes, we need a function to notify us when something gets done. We prefer callbacks in JavaScript. To write a click event for a button, a typical JavaScript code could look like the following: $("#btn_submit").click(function() { alert("Submit Button Clicked"); }); With Kotlin, it's simple. Kotlin uses the Lambda function to achieve this. For the LoginWindow class, we have passed a callback as a constructor parameter. In the LoginWindow class (val callback: (String) -> Unit), the class header specifies that the constructor will take a callback as a parameter, which will return a string when invoked. To pass a callback, we will write the following line of code: callback(nickName) To consume a callback, we will write code that will look like this: val loginWindow = LoginWindow { chatWindow.showChatWindow(it) socket.emit("add_user", it) } So, when callback(nickName) is called, chatWindow.showChatWindow will get called and the nickname will be passed. Without it, you are accessing nothing but the nickname. Establishing a socket connection We shall be using the Socket.IO library to set up sockets between the server and the clients. Socket.IO takes care of the following complexities: Setting up connections Sending and receiving messages to multiple clients Notifying clients when the connection is disconnected Read more about Socket.IO at https://socket.io/. Setting up Socket.IO We have already specified the dependency for Socket.IO in our package.json file. Look at this file. It has a dependency block, which is mentioned in the following code block: "dependencies": { "ejs": "^2.5.7", "express": "^4.16.2", "kotlin": "^1.1.60", "socket.io": "^2.0.4" } When we perform npm install, it basically downloads the socket-io.js file and keeps node_modules | socket.io inside. We will add this JavaScript file to our index.ejs file. There we can find the following mentioned <script> tag inside the <body> tag: <script type="text/javascript" src="/socket.io/socket.io.js"> </script> Also, initialize socket in the same index.js file like this: <script> window.socket = io(); </script> Listening to events With the Socket.IO library, you should open a port and listen to the request using the following lines of code. Initially, we were directly using app.listen(), but now, we will pass that function as a listener for sockets: val io = require("socket.io").listen(app.listen(port, { println("Chat app listening on port http://localhost:$port") })) The server will listen to the following events and based on those events, it will perform certain tasks: Listen to the successful socket connection with the client Listen for the new user login events Whenever a new user joins the chat, add it to the online users chat list and broadcast it to every client so that they will know that a new member has joined the chat Listen to the request when someone sends a message Receive the message and broadcast it to all the clients so that the client can receive it and show it in the chat window Emitting the event The Socket.IO library works on a simple principle—emit and listen. Clients emit the messages and a listener listens to those messages and performs an action associated with them. So now, whenever a user successfully logs in, we will emit an event named add_user and the server will add it to an online user's list. The following code line emits the message: socket.emit("add_user", it) The following code snippet listens to the message and adds a user to the list: socket.on("add_user", { nickName -> socket.nickname= nickName numOfUsers = numOfUsers.inc() usersList.add(nickName as String) }) The socket.on function will listen to the add_user event and store the nickname in the socket. Incrementing and decrementing operator overloading There are a lot of things operator overloading can do, and we have used quite a few features here. Check out how we increment a count of online users: numOfUsers = numOfUsers.inc() It is a much more readable code compared to numOfUsers = numOfUsers+1, umOfUsers += 1, or numOfUsers++. Similarly, we can decrement any number by using the dec() function. Operator overloading applies to the whole set of unary operators, increment-decrement operators, binary operators, and index access operator. Read more about all of them here. Showing a list of online users Now we need to show the list of online users. For this, we need to pass the list of all online users and the count of users along with it. Using the data class The data class is one of the most popular features among Kotlin developers. It is similar to the concept of the Model class. The compiler automatically derives the following members from all properties declared in the primary constructor: The equals()/hashCode() pair The toString() of the User(name=John, age=42) form The componentN() functions corresponding to the properties in their order of declaration The copy() function A simple version of the data class can look like the following line of code, where name and age will become properties of a class: data class User(val name: String, val age: Int) With this single line and, mainly, with the data keyword, you get equals()/hasCode(), toString() and the benefits of getters and setters by using val/var in the form of properties. What a powerful keyword! Using the Pair class In our app, we have chosen the Pair class to demonstrate its usage. The Pair is also a data class. Consider the following line of code: data class Pair<out A, out B> : Serializable It represents a generic pair of two values. You can look at it as a key–value utility in the form of a class. We need to create a JSON object of a number of online users with the list of their nicknames. You can create a JSON object with the help of a Pair class. Take a look at the following lines of code: val userJoinedData = json(Pair("numOfUsers", numOfUsers), Pair("nickname", nickname), Pair("usersList", usersList)) The preceding JSON object will look like the following piece of code in the JSON format: { "numOfUsers": 3, "nickname": "Hardik Trivedi", "usersList": [ "Hardik Trivedi", "Akshay Chordiya", "Ashish Belagali" ] } Iterating list The user's list that we have passed inside the JSON object will be iterated and rendered on the page. Kotlin has a variety of ways to iterate over the list. Actually, anything that implements iterable can be represented as a sequence of elements that can be iterated. It has a lot of utility functions, some of which are mentioned in the following list: hasNext(): This returns true if the iteration has more elements hasPrevious(): This returns true if there are elements in the iteration before the current element next(): This returns the next element in the iteration nextIndex(): This returns the index of the element that would be returned by a subsequent call to next previous(): This returns the previous element in the iteration and moves the cursor position backward previousIndex(): This returns the index of the element that would be returned by a subsequent call to previous() There are some really useful extension functions, such as the following: .asSequence(): This creates a sequence that returns all elements from this iterator. The sequence is constrained to be iterated only once. .forEach(operation: (T) -> Unit): This performs the given operation on each element of this iterator. .iterator(): This returns the given iterator itself and allows you to use an instance of the iterator in a for the loop. .withIndex(): Iterator<IndexedValue<T>>: This returns an iterator and wraps each value produced by this iterator with the IndexedValue, containing a value, and its index. We have used forEachIndexed; this gives the extracted value at the index and the index itself. Check out the way we have iterated the user list: fun showOnlineUsers(data: Json) { val onlineUsersList = document.getElementById("onlineUsersList") onlineUsersList?.let { val usersList = data["usersList"] as? Array<String> usersList?.forEachIndexed { index, nickName -> it.appendChild(getUserListItem(nickName)) } } } Sending and receiving messages Now, here comes the interesting part: sending and receiving a chat message. The flow is very simple: The client will emit the new_message event, which will be consumed by the server, and the server will emit it in the form of a broadcast for other clients. When the user clicks on Send Message, the onSendMessageClicked method will be called. It sends the value back to the view using callback and logs the message in the chat window. After successfully sending a message, it clears the input field as well. Take a look at the following piece of code: private fun onSendMessageClicked(): (Event) -> Unit { return { if (chatMessage?.isNotBlank() as Boolean) { val formContainer = document.getElementById("chatInputBox") as HTMLInputElement callback(chatMessage!!) logMessageFromMe(nickName = nickName, message = chatMessage!!) formContainer.value = "" } } } Null safety We have defined chatMessage as nullable. Check out the declaration here: private var chatMessage: String? = null Kotlin is, by default, null safe. This means that, in Kotlin, objects cannot be null. So, if you want any object that can be null, you need to explicitly state that it can be nullable. With the safe call operator ?., we can write if(obj !=null) in the easiest way ever. The if (chatMessage?.isNotBlank() == true) can only be true if it's not null, and does not contain any whitespace. We do know how to use the Elvis operator while dealing with null. With the help of the Elvis operator, we can provide an alternative value if the object is null. We have used this feature in our code in a number of places. The following are some of the code snippets that highlight the usage of the safe call operator. Removing the view if not null: formContainer?.remove() Iterating over list if not null: usersList?.forEachIndexed { _, nickName -> it.appendChild(getUserListItem(nickName)) } Appending a child if the div tag is not null: onlineUsersList?.appendChild(getUserListItem (data["nickName"].toString())) Getting a list of all child nodes if the <ul> tag is not null: onlineUsersList?.childNodes Checking whether the string is not null and not blank: chatMessage?.isNotBlank() Force unwraps Sometimes, you will have to face a situation where you will be sure that the object will not be null at the time of accessing it. However, since you have declared nullable at the beginning, you will have to end up using force unwraps. Force unwraps have the syntax of !!. This means you have to fetch the value of the calling object, irrespective of it being nullable. We are explicitly reading the chatMessage value to pass its value in the callback. The following is the code: callback(chatMessage!!) Force unwraps are something we should avoid. We should only use them while dealing with interoperability issues. Otherwise, using them is basically nothing but throwing away Kotlin's beautiful features. Using the let function With the help of Lambda and extension functions, Kotlin is providing yet another powerful feature in the form of let functions. The let() function helps you execute a series of steps on the calling object. This is highly useful when you want to perform some code where the calling object is used multiple times and you want to avoid a null check every time. In the following code block, the forEach loop will only get executed if onlineUsersList is not null. We can refer to the calling object inside the let function using it: fun showOnlineUsers(data: Json) { val onlineUsersList = document.getElementById("onlineUsersList") onlineUsersList?.let { val usersList = data["usersList"] as? Array<String> usersList?.forEachIndexed { _, nickName -> it.appendChild(getUserListItem(nickName)) } } } Named parameter What if we told you that while calling, it's not mandatory to pass the parameter in the same sequence that is defined in the function signature? Believe us. With Kotlin's named parameter feature, it's no longer a constraint. Take a look at the following function that has a nickName parameter and the second parameter is message: private fun logMessageFromMe(nickName: String, message: String) { val onlineUsersList = document.getElementById("chatMessages") val li = document.create.li { div(classes = "sentMessages") { span(classes = "chatMessage") { +message } span(classes = "filledInitialsMe") { +getInitials(nickName) } } } onlineUsersList?.appendChild(li) } If you call a function such as logMessageForMe(mrssage,nickName), it will be a blunder. However, with Kotlin, you can call a function without worrying about the sequence of the parameter. The following is the code for this: fun showNewMessage(data: Json) { logMessage(message = data["message"] as String, nickName = data["nickName"] as String) } Note how the showNewMessage() function is calling it, passing message as the first parameter and nickName as the second parameter. Disconnecting a socket Whenever any user leaves the chat room, we will show other online users a message saying x user left. Socket.IO will send a notification to the server when any client disconnects. Upon receiving the disconnect, the event server will remove the user from the list, decrement the count of online users, and broadcast the event to all clients. The code can look something like this: socket.on("disconnect", { usersList.remove(socket.nicknameas String) numOfUsers = numOfUsers.dec() val userJoinedData = json(Pair("numOfUsers", numOfUsers), Pair("nickName", socket.nickname)) socket.broadcast.emit("user_left", userJoinedData) }) Now, it's the client's responsibility to show the message for that event on the UI. The client will listen to the event and the showUsersLeft function will be called from the ChatWindow class. The following code is used for receiving the user_left broadcast: socket.on("user_left", { data -> chatWindow.showUserLeft(data) }) The following displays the message with the nickname of the user who left the chat and the count of the remaining online users: fun showUserLeft(data: Json) { logListItem("${data["nickName"]} left") logListItem(getParticipantsMessage(data["numOfUsers"] as Int)) } Styling the page using CSS We saw how to build a chat application using Kotlin, but without showing the data on a beautiful UI, the user will not like the web app. We have used some simple CSS to give a rich look to the index.ejs page. The styling code is kept inside webapp/css/ styles.css. However, we have done everything so far entirely and exclusively in Kotlin. So, it's better we apply CSS using Kotlin as well. You may have already observed that there are a few mentions of classes. It's nothing but applying the CSS in a Kotlin way. Take a look at how we have applied the classes while making HTML tree elements using a DSL: fun showLogin() { val formContainer = document.getElementById("container") as HTMLDivElement val loginDiv = document.create.div { id = "loginDiv" h3(classes = "title") { +"Welcome to Kotlin Blueprints chat app" } input(classes = "nickNameInput") { id = "nickName" onInputFunction = onInput() maxLength = 16.toString() placeholder = "Enter your nick name" } button(classes = "loginButton") { +"Login" onClickFunction = onLoginButtonClicked() } } formContainer.appendChild(loginDiv) } We developed an entire chat application using Kotlin. If you liked this extract, read our book Kotlin Blueprints to build a REST API using Kotlin. Read More Top 4 chatbot development frameworks for developers How to build a basic server-side chatbot using Go 5 reasons to choose Kotlin over Java

"The data model is what feeds and what powers Power BI." - Kasper de Jonge, Senior Program Manager, Microsoft Data models developed in Power BI Desktop are at the center of Power BI projects, as they expose the interface in support of data exploration and drive the analytical queries visualized in reports and dashboards. Well-designed data models leverage the data connectivity and transformation capabilities to provide an integrated view of distinct business processes and entities. Additionally, data models contain predefined calculations, hierarchies groupings, and metadata to greatly enhance both the analytical power of the dataset and its ease of use. The combination of, Building a Power BI data model, querying and modeling, serves as the foundation for the BI and analytical capabilities of Power BI. In this article, we explore how to design and develop robust data models. Common challenges in dimensional modeling are mapped to corresponding features and approaches in Power BI Desktop, including multiple grains and many-to-many relationships. Examples are also provided to embed business logic and definitions, develop analytical calculations with the DAX language, and configure metadata settings to increase the value and sustainability of models. [box type="note" align="" class="" width=""]Our article is an excerpt from the book Microsoft Power BI Cookbook, written by Brett Powell. This book contains powerful tutorials and techniques to help you with Data Analytics and visualization with Microsoft Power BI.[/box] Designing a multi fact data model Power BI Desktop lends itself to rapid, agile development in which significant value can be obtained quickly despite both imperfect data sources and an incomplete understanding of business requirements and use cases. However, rushing through the design phase can undermine the sustainability of the solution as future needs cannot be met without structural revisions to the model or complex workarounds. A balanced design phase in which fundamental decisions such as DirectQuery versus in-memory are analyzed while a limited prototype model is used to generate visualizations and business feedback can address both short- and long-term needs. This recipe describes a process for designing a multiple fact table data model and identifies some of the primary questions and factors to consider. Setting business expectations Everyone has seen impressive Power BI demonstrations and many business analysts have effectively used Power BI Desktop independently. These experiences may create an impression that integration, rich analytics, and collaboration can be delivered across many distinct systems and stakeholders very quickly or easily. It's important to reign in any unrealistic expectations and confirm feasibility. For example, Power BI Desktop is not an enterprise BI tool like SSIS or SSAS in terms of scalability, version control, features, and configurations. Power BI datasets cannot be incrementally refreshed like partitions in SSAS, and the current 1 GB file limit (after compression) places a hard limit on the amount of data a single model can store. Additionally, if multiple data sources are needed within the model, then DirectQuery models are not an option. Finally, it's critical to distinguish the data model as a platform supporting robust analysis of business processes, not an individual report or dashboard itself. Identify the top pain points and unanswered business questions in the current state. Contrast this input with an assessment of feasibility and complexity (for example, data quality and analytical needs) and Target realistic and sustainable deliverables. How to do it Dimensional modeling best practices and star schema designs are directly applicable to Power BI data models. Short, collaborative modeling sessions can be scheduled with subject matter experts and main stakeholders. With the design of the model in place, an informed decision of the model's data mode (Import or DirectQuery) can be made prior to Development. Four-step dimensional design process Choose the business process The number and nature of processes to include depends on the scale of the sources and scope of the project In this example, the chosen processes are Internet Sales, Reseller Sales and General Ledger Declare the granularity For each business process (or fact) to be modeled from step 1, define the meaning of each row: These should be clear, concise business definitions--each fact table should only contain one grain Consider scalability limitations with Power BI Desktop and balance the needs between detail and history (for example, greater history but lower granularity) Example: One Row per Sales Order Line, One Row per GL Account Balance per fiscal period Separate business processes, such as plan and sales should never be integrated into the same table. Likewise, a single fact table should not contain distinct processes such as shipping and receiving. Fact tables can be related to common dimensions but should never be related to each other in the data model (for example, PO Header and Line level). Identify the dimensions These entities should have a natural relationship with the business process or event at the given granularity Compare the dimension with any existing dimensions and hierarchies in the organization (for example, Store) If so, determine if there's a conflict or if additional columns are required Be aware of the query performance implications with large, high cardinality dimensions such as customer tables with over 2 million rows. It may be necessary to optimize this relationship in the model or the measures and queries that use this relationship. See Chapter 11, Enhancing and Optimizing Existing Power BI Solutions, for more details. Identify the facts These should align with the business processes being modeled: For example, the sum of a quantity or a unique count of a dimension Document the business and technical definition of the primary facts and compare this with any existing reports or metadata repository (for example, Net Sales = Extended Amount - Discounts). Given steps 1-3, you should be able to walk through top business  questions and check whether the planned data model will support it. Example: "What was the variance between Sales and Plan for last month in Bikes?" Any clear gaps require modifying the earlier steps, removing the question from the scope of the data model, or a plan to address the issue with additional logic in the model (M or DAX). Focus only on the primary facts at this stage such as the individual source columns that comprise the cost facts. If the business definition or logic for core fact has multiple steps and conditions, check if the data model will naturally simplify it or if the logic can be developed in the data retrieval to avoid complex measures. Data warehouse and implementation bus matrix The Power BI model should preferably align with a corporate data architecture framework of standard facts and dimensions that can be shared across models. Though consumed into Power BI Desktop, existing data definitions and governance should be observed. Any new facts, dimensions, and measures developed with Power BI should supplement this  architecture. Create a data warehouse bus matrix: A matrix of business processes (facts) and standard dimensions is a primary tool for designing and managing data models and communicating the overall BI architecture. In this example, the business processes selected for the model are Internet Sales, Reseller Sales, and General Ledger. Create an implementation bus matrix: An outcome of the model design process should include a more detailed implementation bus matrix. Clarity and approval of the grain of the fact tables, the definitions of the primary measures, and all dimensions gives confidence when entering the development phase. Power BI queries (M) and analysis logic (DAX) should not be considered a long-term substitute for issues with data quality, master data management, and the data warehouse. If it is necessary to move forward, document the "technical debts" incurred and consider long-term solutions such as Master Data Services (MDS). Choose the dataset storage mode - Import or DirectQuery With the logical design of a model in place, one of the top design questions is whether to implement this model with DirectQuery mode or with the default imported In-Memory mode. In-Memory mode The default in-memory mode is highly optimized for query performance and supports additional modeling and development flexibility with DAX functions. With compression, columnar storage, parallel query plans, and other techniques an import mode model is able to support a large amount of data (for example, 50M rows) and still perform well with complex analysis expressions. Multiple data sources can be accessed and integrated in a single data model and all DAX functions are supported for measures, columns, and role security. However, the import or refresh process must be scheduled and this is currently limited to eight refreshes per day for datasets in shared capacity (48X per day in premium capacity). As an alternative to scheduled refreshes in the Power BI service, REST APIs can be used to trigger a data refresh of a published dataset. For example, an HTTP request to a Power BI REST API calling for the refresh of a dataset can be added to the end of a nightly update or ETL process script such that published Power BI content remains aligned with the source systems. More importantly, it's not currently possible to perform an incremental refresh such as the Current Year rows of a table (for example, a table partition) or only the source rows that have changed. In-Memory mode models must maintain a file size smaller than the current limits (1 GB compressed currently, 10GB expected for Premium capacities by October 2017) and must also manage refresh schedules in the Power BI Service. Both incremental data refresh and larger dataset sizes are identified as planned capabilities of the Microsoft Power BI Premium Whitepaper (May 2017). DirectQuery mode A DirectQuery mode model provides the same semantic layer interface for users and contains the same metadata that drives model behaviors as In-Memory models. The performance of DirectQuery models, however, is dependent on the source system and how this data is presented to the model. By eliminating the import or refresh process, DirectQuery provides a means to expose reports and dashboards to source data as it changes. This also avoids the file size limit of import mode models. However, there are several limitations and restrictions to be aware of with DirectQuery: Only a single database from a single, supported data source can be used in a DirectQuery model. When deployed for widespread use, a high level of network traffic can be generated thus impacting performance. Power BI visualizations will need to query the source system, potentially via an on-premises data gateway. Some DAX functions cannot be used in calculated columns or with role security. Additionally, several common DAX functions are not optimized for DirectQuery performance. Many M query transformation functions cannot be used with DirectQuery. MDX client applications such as Excel are supported but less metadata (for example, hierarchies) is exposed. Given these limitations and the importance of a "speed of thought" user experience with Power BI, DirectQuery should generally only be used on centralized and smaller projects in which visibility to updates of the source data is essential. If a supported DirectQuery system (for example, Teradata or Oracle) is available, the performance of core measures and queries should be tested. Confirm referential integrity in the source database and use the Assume Referential Integrity relationship setting in DirectQuery mode models. This will generate more efficient inner join SQL queries against the source Database. How it works DAX formula and storage engine Power BI Datasets and SQL Server Analysis Services (SSAS) share the same database engine and architecture. Both tools support both Import and DirectQuery data models and both DAX and MDX client applications such as Power BI (DAX) and Excel (MDX). The DAX Query Engine is comprised of a formula and a storage engine for both Import and DirectQuery models. The formula engine produces query plans, requests data from the storage engine, and performs any remaining complex logic not supported by the storage engine against this data such as IF and SWITCH functions In DirectQuery models, the data source database is the storage engine--it receives SQL queries from the formula engine and returns the results to the formula engine. For In- Memory models, the imported and compressed columnar memory cache is the storage engine. We discussed about building data models using Microsoft power BI. If you liked our post, be sure to check out Microsoft Power BI Cookbook to gain more information on using Microsoft power BI for data analysis and visualization. Unlocking the secrets of Microsoft Power BI Microsoft spring updates for PowerBI and PowerApps How to build a live interactive visual dashboard in Power BI with Azure Stream  

May’s continues on a high note. Plenty of big announcements and major new releases announced in two of the biggest events in the tech world: Google I/O, Microsoft Build and PyCon. Read about them and more in our tech news section. Here’s what you may have missed in the last 7 days – Tech news, insights and tutorials… Tech news Conferences in focus this week Top 5 Google I/O 2018 conference Day 1 Highlights What we learned from Qlik Qonnections 2018 Microsoft Build 2018 Day 1: Azure meets Artificial Intelligence Data news in depth Microsoft Open Sources ML.NET, a cross-platform machine learning framework Linux Foundation launches the Acumos Al Project to make AI accessible Nvidia’s Volta Tensor Core GPU hits performance milestones. But is it the best? Development & programming news in depth Google’s Android Things, developer preview 8: First look Put your game face on! Unity 2018.1 is now available What’s new in Vapor 3, the popular Swift based web framework Xamarin Forms 3, the popular cross-platform UI Toolkit, is here! Windows 10 IoT Core: What you need to know Google Daydream powered Lenovo Mirage solo hits the market GCC 8.1 Standards released! Google open sources Seurat to bring high precision graphics to Mobile VR Cloud & networking news in depth What to expect from vSphere 6.7 What’s new in Wireshark 2.6? Get DevOps eBooks and videos while supporting charity Microsoft’s Azure Container Service (ACS) is now Azure Kubernetes Services (AKS) Red Hat Enterprise Linux 7.5 (RHEL 7.5) now generally available Kali Linux 2018.2 released Tutorials Data tutorials Getting Started with Automated Machine Learning (AutoML) Analyzing CloudTrail Logs using Amazon Elasticsearch Tensor Processing Unit (TPU) 3.0: Google’s answer to cloud-ready Artificial Intelligence Distributed TensorFlow: Working with multiple GPUs and servers Implementing 3 Naive Bayes classifiers in scikit-learn Development & programming tutorials Web development tutorials Getting started with Angular CLI and build your first Angular Component How to implement Internationalization and localization in your Node.js app Programming tutorials How to install and configure TypeScript NativeScript: What is it, and how to set it up Building functional programs with F# Applying Single Responsibility principle from SOLID in .NET Core Unit Testing in .NET Core with Visual Studio 2017 for better code quality Cloud & Networking tutorials How to secure a private cloud using IAM How to create your own AWS CloudTrail How to secure ElasticCache in AWS This week’s opinions, analysis, and insights Data Insights Google News’ AI revolution strikes balance between personalization and the bigger picture Why Drive.ai is going to struggle to disrupt public transport 6 reasons to choose MySQL 8 for designing database solutions Are Recurrent Neural Networks capable of warping time? Development & Programming Insights 8 recipes to master Promises in ECMAScript 2018 Forget C and Java. Learn Kotlin: the next universal programming language

AWS offers services to handle the cache management process. Earlier, we were using Memcached or Redis installed on VM, which was a very complex and tough task to manage in terms of ensuring availability, patching, scalability, and security. [box type="shadow" align="" class="" width=""]This article is an excerpt taken from the book,'Cloud Security Automation'. In this book, you'll learn the basics of why cloud security is important and how automation can be the most effective way of controlling cloud security.[/box] On AWS, we have this service available as ElastiCache. This gives you the option to use any engine (Redis or Memcached) to manage your cache. It's a scalable platform that will be managed by AWS in the backend. ElastiCache provides a scalable and high-performance caching solution. It removes the complexity associated with creating and managing distributed cache clusters using Memcached or Redis. Now, let's look at how to secure ElastiCache. Secure ElastiCache in AWS For enhanced security, we deploy ElastiCache clusters inside VPC. When they are deployed inside VPC, we can use a security group and NACL to add a level of security on the communication ports at network level. Apart from this, there are multiple ways to enable security for ElastiCache. VPC-level security Using a security group at VPC—when we deploy AWS ElastiCache in VPC, it gets associated with a subnet, a security group, and the routing policy of that VPC. Here, we define a rule to communicate with the ElastiCache cluster on a specific port. ElastiCache clusters can also be accessed from on-premise applications using VPN and Direct Connect. Authentication and access control We use IAM in order to implement the authentication and access control on ElastiCache. For authentication, you can have the following identity type: Root user: It's a superuser that is created while setting up an AWS account. It has super administrator privileges for all the AWS services. However, it's not recommended to use the root user to access any of the services. IAM user: It's a user identity in your AWS account that will have a specific set of permissions for accessing the ElastiCache service. IAM role: We also can define an IAM role with a specific set of permissions and associate it with the services that want to access ElastiCache. It basically generates temporary access keys to use ElastiCache. Apart from this, we can also specify federated access to services where we have an IAM role with temporary credentials for accessing the service. To access ElastiCache, service users or services must have a specific set of permissions such as create, modify, and reboot the cluster. For this, we define an IAM policy and associate it with users or roles. Let's see an example of an IAM policy where users will have permission to perform system administration activity for ElastiCache cluster: { "Version": "2012-10-17", "Statement":[{ "Sid": "ECAllowSpecific", "Effect":"Allow", "Action":[ "elasticache:ModifyCacheCluster", "elasticache:RebootCacheCluster", "elasticache:DescribeCacheClusters", "elasticache:DescribeEvents", "elasticache:ModifyCacheParameterGroup", "elasticache:DescribeCacheParameterGroups", "elasticache:DescribeCacheParameters", "elasticache:ResetCacheParameterGroup", "elasticache:DescribeEngineDefaultParameters"], "Resource":"*" } ] } Authenticating with Redis authentication AWS ElastiCache also adds an additional layer of security with the Redis authentication command, which asks users to enter a password before they are granted permission to execute Redis commands on a password-protected Redis server. When we use Redis authentication, there are the following few constraints for the authentication token while using ElastiCache: Passwords must have at least 16 and a maximum of 128 characters Characters such as @, ", and / cannot be used in passwords Authentication can only be enabled when you are creating clusters with the in-transit encryption option enabled The password defined during cluster creation cannot be changed To make the policy harder or more complex, there are the following rules related to defining the strength of a password: A password must include at least three characters of the following character types: Uppercase characters Lowercase characters Digits Non-alphanumeric characters (!, &, #, $, ^, <, >, -) A password must not contain any word that is commonly used A password must be unique; it should not be similar to previous passwords Data encryption AWS ElastiCache and EC2 instances have mechanisms to protect against unauthorized access of your data on the server. ElastiCache for Redis also has methods of encryption for data run-in on Redis clusters. Here, too, you have data-in-transit and data-at-rest encryption methods. Data-in-transit encryption ElastiCache ensures the encryption of data when in transit from one location to another. ElastiCache in-transit encryption implements the following features: Encrypted connections: In this mode, SSL-based encryption is enabled for server and client communication Encrypted replication: Any data moving between the primary node and the replication node are encrypted Server authentication: Using data-in-transit encryption, the client checks the authenticity of a connection—whether it is connected to the right server Client authentication: After using data-in-transit encryption, the server can check the authenticity of the client using the Redis authentication feature Data-at-rest encryption ElastiCache for Redis at-rest encryption is an optional feature that increases data security by encrypting data stored on disk during sync and backup or snapshot operations. However, there are the following few constraints for data-at-rest encryption: It is supported only on replication groups running Redis version 3.2.6. It is not supported on clusters running Memcached. It is supported only for replication groups running inside VPC. Data-at-rest encryption is supported for replication groups running on any node type. During the creation of the replication group, you can define data-at-rest encryption. Data-at-rest encryption once enabled, cannot be disabled. To summarize, we learned how to secure ElastiCache and ensured security for PaaS services, such as database and analytics services. If you've enjoyed reading this article, do check out 'Cloud Security Automation' for hands-on experience of automating your cloud security and governance. How to start using AWS AWS Sydney Summit 2018 is all about IoT AWS Fargate makes Container infrastructure management a piece of cake    

In this tutorial, we will look at the installation process of TypeScript and the editor setup for TypeScript development. Microsoft does well in providing easy-to-perform steps to install TypeScript on all platforms, namely Windows, macOS, and Linux. [box type="shadow" align="" class="" width=""]The following excerpt is taken from the book TypeScript 2.x By Example written by Sachin Ohri. This book presents hands-on examples and projects to learn the fundamental concepts of the popular TypeScript programming language.[/box] Installation of TypeScript TypeScript's official website is the best source to install the latest version. On the website, go to the Download section. There, you will find details on how to install TypeScript. Node.js and Visual Studio are the two most common ways to get it. It supports a host of other editors and has plugins available for them in the same link. We will be installing TypeScript using Node.js and using Visual Studio Code as our primary editor. You can use any editor of your choice and be able to run the applications seamlessly. If you use full-blown Visual Studio as your primary development IDE, then you can use either of the links, Visual Studio 2017 or Visual Studio 2013, to download the TypeScript SDK. Visual Studio does come with a TypeScript compiler but it's better to install it from this link so as to get the latest version. To install TypeScript using Node.js, we will use npm (node package manager), which comes with Node.js. Node.js is a popular JavaScript runtime for building and running server-side JavaScript applications. As TypeScript compiles into JavaScript, Node is an ideal fit for developing server-side applications with the TypeScript language. As mentioned on the website, just running the following command in the Terminal (on macOS) / Command Prompt (on Windows) window will install the latest version: npm install -g typescript To load any package from Node.js, the npm command starts with npm install; the -g flag identifies that we are installing the package globally. The last parameter is the name of the package that we are installing. Once it is installed, you can check the version of TypeScript by running the following command in the Terminal window: tsc -v You can use the following command to get the help for all the other options that are available with tsc: tsc -h TypeScript editors One of the outstanding features of TypeScript is its support for editors. All the editors provide support for language services, thereby providing features such as IntelliSense, statement completion, and error highlighting. If you are coming from a .NET background, then Visual Studio 2013/2015/2017 is a good option for you. Visual Studio does not require any configuration and it's easy to start using TypeScript. As we discussed earlier, just install the SDK and you are good to go. If you are from a Java background, TypeScript supports Eclipse as well. It also supports plugins for Sublime, WebStorm, and Atom, and each of these provides a rich set of features. Visual Studio Code (VS Code) is another good option for an IDE. It's a smaller, lighter version of Visual Studio and primarily used for web application development. VS Code is lightweight and cross-platform, capable of running on Windows, Linux, and macOS. It has an ever-increasing set of plugins to help you write better code, such as TSLint, a static analysis tool to help TypeScript code for readability, maintainability, and error checking. VS Code has a compelling case to be the default IDE for all sorts of web application development. In this post, we will briefly look at the Visual Studio and VS Code setup for TypeScript. Visual Studio Visual Studio is a full-blown IDE provided by Microsoft for all .NET based development, but now Visual Studio also has excellent support for TypeScript with built-in project templates. A TypeScript compiler is integrated into Visual Studio to allow automatic transpiling of code to JavaScript. Visual Studio also has the TypeScript language service integrated to provide IntelliSense and design-time error checking, among other things. With Visual Studio, creating a project with a TypeScript file is as simple as adding a new file with a .ts extension. Visual Studio will provide all the features out of the box. VS Code VS Code is a lightweight IDE from Microsoft used for web application development. VS Code can be installed on Windows, macOS, and Linux-based systems. VS Code can recognize the different type of code files and comes with a huge set of extensions to help in development. You can install VS Code from https://code.visualstudio.com/download. VS Code comes with an integrated TypeScript compiler, so we can start creating projects directly. The following screenshot shows a TypeScript file opened in VS Code: To run the project in VS Code, we need a task runner. VS Code includes multiple task runners which can be configured for the project, such as Gulp, Grunt, and TypeScript. We will be using the TypeScript task runner for our build. VS Code has a Command Palette which allows you to access various different features, such as Build Task, Themes, Debug options, and so on. To open the Command Palette, use Ctrl + Shift + P on a Windows machine or Cmd + Shift + P on a macOS. In the Command Palette, type Build, as shown in the following screenshot, which will show the command to build the project: When the command is selected, VS Code shows an alert, No built task defined..., as follows: We select Configure Build Task and, from all the available options as shown in the following screenshot, choose TypeScript build: This creates a new folder in your project, .vscode and a new file, task.json. This JSON file is used to create the task that will be responsible for compiling TypeScript code in VS Code. TypeScript needs another JSON file (tsconfig.json) to be able to configure compiler options. Every time we run the code, tsc will look for a file with this name and use this file to configure itself. TypeScript is extremely flexible in transpiling the code to JavaScript as per developer requirements, and this is achieved by configuring the compiler options of TypeScript. TypeScript compiler The TypeScript compiler is called tsc and is responsible for transpiling the TypeScript code to JavaScript. The TypeScript compiler is also cross-platform and supported on Windows, macOS, and Linux. To run the TypeScript compiler, there are a couple of options. One is to integrate the compiler in your editor of choice, which we explained in the previous section. In the previous section, we also integrated the TypeScript compiler with VS Code, which allowed us to build our code from the editor itself. All the compiler configurations that we would want to use are added to the tsconfig.json file. Another option is to use tsc directly from the command line / Terminal window. TypeScript's tsc command takes compiler configuration options as parameters and compiles code into JavaScript. For example, create a simple TypeScript file in Notepad and add the following lines of code to it. To create a file as a TypeScript file, we just need to make sure we have the file extension as *.ts: class Editor { constructor(public name: string,public isTypeScriptCompatible : Boolean) {} details() { console.log('Editor: ' + this.name + ', TypeScript installed: ' + this.isTypeScriptCompatible); } } class VisualStudioCode extends Editor{ public OSType: string constructor(name: string,isTypeScriptCompatible : Boolean, OSType: string) { super(name,isTypeScriptCompatible); this.OSType = OSType; } } let VS = new VisualStudioCode('VSCode', true, 'all'); VS.details(); This is the same code example we used in the TypeScript features section of this chapter. Save this file as app.ts (you can give it any name you want, as long as the extension of the file is *.ts). In the command line / Terminal window, navigate to the path where you have saved this file and run the following command: tsc app.ts This command will build the code and the transpile it into JavaScript. The JavaScript file is also saved in the same location where we had TypeScript. If there is any build issue, tsc will show these messages on the command line only. As you can imagine, running the tsc command manually for medium- to large-scale projects is not a productive approach. Hence, we prefer to use an editor that has TypeScript integrated. The following table shows the most commonly used TypeScript compiler configurations. We will be discussing these in detail in upcoming chapters: Compiler option Type Description allowUnusedLabels boolean By default, this flag is false. This option tells the compiler to flag unused labels. alwaysStrict boolean By default, this flag is false. When turned on, this will cause the compiler to compile in strict mode and emit use strict in the source file. module string Specify module code generation: None, CommonJS, AMD, System, UMD, ES6, or ES2015. moduleResolution string Determines how the module is resolved. noImplicitAny boolean This property allows an error to be raised if there is any code which implies data type as any. This flag is recommended to be turned off if you are migrating a JavaScript project to TypeScript in an incremental manner. noImplicitReturn boolean Default value is false; raises an error if not all code paths return a value. noUnusedLocals boolean Reports an error if there are any unused locals in the code. noUnusedParameter boolean Reports an error if there are any unused parameters in the code. outDir string Redirects output structure to the directory. outFile string Concatenates and emits output to a single file. The order of concatenation is determined by the list of files passed to the compiler on the command line along with triple-slash references and imports. See the output file order documentation for more details. removeComments boolean Remove all comments except copyright header comments beginning with /*!. sourcemap boolean Generates corresponding .map file. Target string Specifies ECMAScript target version: ES3(default), ES5, ES6/ES2015, ES2016, ES2017, or ESNext. Watch Runs the compiler in watch mode. Watches input files and triggers recompilation on changes. We saw it is quite easy to set up and configure TypeScript, and we are now ready to get started with our first application! To learn more about writing and compiling your first TypeScript application, make sure you check out the book TypeScript 2.x By Example. Introduction to TypeScript Introducing Object Oriented Programmng with TypeScript Elm and TypeScript – Static typing on the Frontend

Machine learning has always been dependent on the selection of the right features within a given model; even the selection of the right algorithm. But deep learning changed this. The selection process is now built into the models themselves. Researchers and engineers are now shofting their focus from feature engineering to network engineering. Out of this, AutoML, or meta learning, has become an increasingly important part of deep learning. AutoML is an emerging research topic which aims at auto-selecting the most efficient neural network for a given learning task. In other words, AutoML represents a set of methodologies for learning how to learn efficiently. Consider for instance the tasks of machine translation, image recognition, or game playing. Typically, the models are manually designed by a team of engineers, data scientist, and domain experts. If you consider that a typical 10-layer network can have ~1010 candidate network, you understand how expensive, error prone, and ultimately sub-optimal the process can be. This article is an excerpt from a book written by Antonio Gulli and Amita Kapoor titled TensorFlow 1.x Deep Learning Cookbook. This book is an easy-to-follow guide that lets you explore reinforcement learning, GANs, autoencoders, multilayer perceptrons and more. AutoML with recurrent networks and with reinforcement learning The key idea to tackle this problem is to have a controller network which proposes a child model architecture with probability p, given a particular network given in input. The child is trained and evaluated for the particular task to be solved (say for instance that the child gets accuracy R). This evaluation R is passed back to the controller which, in turn, uses R to improve the next candidate architecture. Given this framework, it is possible to model the feedback from the candidate child to the controller as the task of computing the gradient of p and then scale this gradient by R. The controller can be implemented as a Recurrent Neural Network (see the following figure). In doing so, the controller will tend to privilege iteration after iterations candidate areas of architecture that achieve better R and will tend to assign a lower probability to candidate areas that do not score so well. For instance, a controller recurrent neural network can sample a convolutional network. The controller can predict many hyper-parameters such as filter height, filter width, stride height, stride width, and the number of filters for one layer and then can repeat. Every prediction can be carried out by a softmax classifier and then fed into the next RNN time step as input. This is well expressed by the following images taken from Neural Architecture Search with Reinforcement Learning, Barret Zoph, Quoc V. Le: Predicting hyperparameters is not enough as it would be optimal to define a set of actions to create new layers in the network. This is particularly difficult because the reward function that describes the new layers is most likely not differentiable. This makes it impossible to optimize using standard techniques such as SGD. The solution comes from reinforcement learning. It consists of adopting a policy gradient network. Besides that, parallelism can be used for optimizing the parameters of the controller RNN. Quoc Le & Barret Zoph proposed to adopt a parameter-server scheme where we have a parameter server of S shards, that store the shared parameters for K controller replicas. Each controller replica samples m different child architectures that are trained in parallel as illustrated in the following images, taken from Neural Architecture Search with Reinforcement Learning, Barret Zoph, Quoc V. Le: Quoc and Barret applied AutoML techniques for Neural Architecture Search to the Penn Treebank dataset, a well-known benchmark for language modeling. Their results improve the manually designed networks currently considered the state-of-the-art. In particular, they achieve a test set perplexity of 62.4 on the Penn Treebank, which is 3.6 perplexity better than the previous state-of-the-art model. Similarly, on the CIFAR-10 dataset, starting from scratch, the method can design a novel network architecture that rivals the best human-invented architecture in terms of test set accuracy. The proposed CIFAR-10 model achieves a test error rate of 3.65, which is 0.09 percent better and 1.05x faster than the previous state-of-the-art model that used a similar architectural scheme. Meta-learning blocks In Learning Transferable Architectures for Scalable Image Recognition, Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le, 2017. propose to learn an architectural building block on a small dataset that can be transferred to a large dataset. The authors propose to search for the best convolutional layer (or cell) on the CIFAR-10 dataset and then apply this learned cell to the ImageNet dataset by stacking together more copies of this cell, each with their own parameters. Precisely, all convolutional networks are made of convolutional layers (or cells) with identical structures but different weights. Searching for the best convolutional architectures is therefore reduced to searching for the best cell structures, which is faster more likely to generalize to other problems. Although the cell is not learned directly on ImageNet, an architecture constructed from the best learned cell achieves, among the published work, state-of-the-art accuracy of 82.7 percent top-1 and 96.2 percent top-5 on ImageNet. The model is 1.2 percent better in top-1 accuracy than the best human-invented architectures while having 9 billion fewer FLOPS—a reduction of 28% from the previous state of the art model. What is also important to notice is that the model learned with RNN+RL (Recurrent Neural Networks + Reinforcement Learning) is beating the baseline represented by Random Search (RS) as shown in the figure taken from the paper. In the mean performance of the top-5 and top-25 models identified in RL versus RS, RL is always winning: AutoML and learning new tasks Meta-learning systems can be trained to achieve a large number of tasks and are then tested for their ability to learn new tasks. A famous example of this kind of meta-learning is transfer learning, where networks can successfully learn new image-based tasks from relatively small datasets. However, there is no analogous pre-training scheme for non-vision domains such as speech, language, and text. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks, Chelsea Finn, Pieter Abbeel, Sergey Levine, 2017, proposes a model- agnostic approach names MAML, compatible with any model trained with gradient descent and applicable to a variety of different learning problems, including classification, regression, and reinforcement learning. The goal of meta-learning is to train a model on a variety of learning tasks, such that it can solve new learning tasks using only a small number of training samples. The meta-learner aims at finding an initialization that rapidly adapts to various problems quickly (in a small number of steps) and efficiently (using only a few examples). A model represented by a parametrized function fθ with parameters θ.When adapting to a new task Ti, the model's parameters θ become θi  . In MAML, the updated parameter vector θi  is computed using one or more gradient descent updates on task Ti. For example, when using one gradient update, θ ~ = θ − α∇θLTi (fθ) where LTi is the loss function for the task T and α is a meta-learning parameter. The MAML algorithm is reported in this figure: MAML was able to substantially outperform a number of existing approaches on popular few-shot image classification benchmark. Few shot image is a quite challenging problem aiming at learning new concepts from one or a few instances of that concept. As an example, Human-level concept learning through probabilistic program induction, Brenden M. Lake, Ruslan Salakhutdinov, Joshua B. Tenenbaum, 2015, suggested that humans can learn to identify novel two-wheel vehicles from a single picture such as the one contained in the box as follows: If you enjoyed this excerpt, check out the book TensorFlow 1.x Deep Learning Cookbook, to skill up and implement tricky neural networks using Google's TensorFlow 1.x AmoebaNets: Google’s new evolutionary AutoML AutoML : Developments and where is it heading to What is Automated Machine Learning (AutoML)?

Log management and analysis for many organizations start and end with just three letters: E, L, and K, which stands for Elasticsearch, Logstash, and Kibana. In today's tutorial, we will learn about analyzing CloudTrail logs which are E, L and K. [box type="shadow" align="" class="" width=""]This tutorial is an excerpt from the book AWS Administration - The Definitive Guide - Second Edition written by Yohan Wadia. This book will help you enhance your application delivery skills with the latest AWS services while also securing and monitoring the environment workflow.[/box] The three open-sourced products are essentially used together to aggregate, parse, search and visualize logs at an enterprise scale: Logstash: Logstash is primarily used as a log collection tool. It is designed to collect, parse, and store logs originating from multiple sources, such as applications, infrastructure, operating systems, tools, services, and so on. Elasticsearch: With all the logs collected in one place, you now need a query engine to filter and search through these logs for particular events. That's exactly where Elasticsearch comes into play. Elasticsearch is basically a search server based on the popular information retrieval software library, Lucene. It provides a distributed, full-text search engine along with a RESTful web interface for querying your logs. Kibana: Kibana is an open source data visualization plugin, used in conjunction with Elasticsearch. It provides you with the ability to create and export your logs into various visual graphs, such as bar charts, scatter graphs, pie charts, and so on. You can easily download and install each of these components in your AWS environment, and get up and running with your very own ELK stack in a matter of hours! Alternatively, you can also leverage AWS own Elasticsearch service! Amazon Elasticsearch is a managed ELK service that enables you to quickly deploy operate, and scale an ELK stack as per your requirements. Using Amazon Elasticsearch, you eliminate the need for installing and managing the ELK stack's components on your own, which in the long run can be a painful experience. For this particular use case, we will leverage a simple CloudFormation template that will essentially set up an Amazon Elasticsearch domain to filter and visualize the captured CloudTrail Log files, as depicted in the following diagram: To get started, log in to the CloudFormation dashboard, at https://console.aws.amazon.com/cloudformation. Next, select the option Create Stack to bring up the CloudFormation template selector page. Paste http://s3.amazonaws.com/concurrencylabs-cfn-templates/cloudtrail-es-cluster/cloudtrail-es-cluster.json in, the Specify an Amazon S3 template URL field, and click on Next to continue. In the Specify Details page, provide a suitable Stack name and fill out the following required parameters: AllowedIPForEsCluster: Provide the IP address that will have access to the nginx proxy and, in turn, have access to your Elasticsearch cluster. In my case, I've provided my laptop's IP. Note that you can change this IP at a later stage, by visiting the security group of the nginx proxy once it has been created by the CloudFormation template. CloudTrailName: Name of the CloudTrail that we set up at the beginning of this chapter. KeyName: You can select a key-pair for obtaining SSH to your nginx proxy instance: LogGroupName: The name of the CloudWatch Log Group that will act as the input to our Elasticsearch cluster. ProxyInstanceTypeParameter: The EC2 instance type for your proxy instance. Since this is a demonstration, I've opted for the t2.micro instance type. Alternatively, you can select a different instance type as well. Once done, click on Next to continue. Review the settings of your stack and hit Create to complete the process. The stack takes a good few minutes to deploy as a new Elasticsearch domain is created. You can monitor the progress of the deployment by either viewing the CloudFormation's Output tab or, alternatively, by viewing the Elasticsearch dashboard. Note that, for this deployment, a default t2.micro.elasticsearch instance type is selected for deploying Elasticsearch. You should change this value to a larger instance type before deploying the stack for production use. You can view information on Elasticsearch Supported Instance Types at http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-supported-instance-types.html. With the stack deployed successfully, copy the Kibana URL from the CloudFormation Output tab: "KibanaProxyEndpoint": "http://<NGINX_PROXY>/_plugin/kibana/" The Kibana UI may take a few minutes to load. Once it is up and running, you will need to configure a few essential parameters before you can actually proceed. Select Settings and hit the Indices option. Here, fill in the following details: Index contains time-based events: Enable this checkbox to index time-based events Use event times to create index names: Enable this checkbox as well Index pattern interval: Set the Index pattern interval to Daily from the drop-down list Index name of pattern: Type [cwl-]YYYY.MM.DD in to this field Time-field name: Select the @timestamp value from the drop-down list Once completed, hit Create to complete the process. With this, you should now start seeing logs populate on to Kibana's dashboard. Feel free to have a look around and try out the various options and filters provided by Kibana:   Phew! That was definitely a lot to cover! But wait, there's more! AWS provides yet another extremely useful governance and configuration management service AWS Config, know more from this book AWS Administration - The Definitive Guide - Second Edition. The Cloud and the DevOps Revolution Serverless computing wars: AWS Lambdas vs Azure Functions

It won’t be wrong to say that the first day of the ongoing Google I/O 2018 conference was largely dominated by Artificial Intelligence. CEO Sundar Pichai didn’t hide his excitement in giving us a sneak-peek of a whole host of features across various products driven by AI, which Google plan to roll out to the masses in the coming days. One of the biggest announcements was the unveiling of the next-gen silicon chip - the Tensor Processing Unit 3.0. The TPU has been central to Google’s AI market dominance strategy since its inception in 2016, and the latest iteration of this custom-made silicon chip promises to deliver faster and more powerful machine learning capabilities. What’s new in TPU 3.0? TPU is Google’s premium AI hardware offering for its cloud platform, with the objective of making it easier to run machine learning systems in a fast, cheap and easy manner. In his I/O 2018 keynote, Sundar Pichai declared that TPU 3.0 will be 8x more powerful than its predecessor. [embed]https://www.youtube.com/watch?v=ogfYd705cRs&t=5750[/embed] A TPU 3.0 pod is expected to crunch numbers at approximately 100 petaflops, as compared to 11.5 petaflops delivered by TPU 2.0. Pichai did not comment about the precision of the processing in these benchmarks - something which can make a lot of difference in real-world applications. Not a lot of other TPU 3.0 features were disclosed. The chip is still only for Google’s use for powering their applications. It also powers the Google Cloud Platform to handle customers’ workloads. High performance - but at a cost? An important takeaway from Pichai’s announcement is that TPU 3.0 is expected to be power-hungry - so much so that Google’s data centers deploying the chips now require liquid cooling to take care of the heat dissipation problem. This is not necessarily a good thing, as the need for dedicated cooling systems will only increase as Google scale up their infrastructure. A few analysts and experts, including Patrick Moorhead, tech analyst and founder of Moor Insights & Strategy, have raised concerns about this on twitter. [embed]https://twitter.com/PatrickMoorhead/status/993905641390391296[/embed] [embed]https://twitter.com/ryanshrout/status/993906310197506048[/embed] [embed]https://twitter.com/LostInBrittany/status/993904650888724480[/embed] The TPU is keeping up with Google’s growing AI needs The evolution of the Tensor Processing Unit has been rather interesting. When TPU was initially released way back in 2016, it was just a simple math accelerator used for training models, supporting barely close to 8 software instructions. However, Google needed more computing power to keep up with their neural networks to power their applications on the cloud. TPU 2.0 supported single-precision floating calculations and added 8GB of HBM (High Bandwidth Memory) for faster, more improved performance. With 3.0, TPU have stepped up another notch, delivering the power and performance needed to process data and run their AI models effectively. The significant increase in the processing capability from 11 petaflops to more than 100 petaflops is a clear step in this direction. Optimized for Tensorflow - the most popular machine learning framework out there - it is clear that TPU 3.0 will have an important role to play as Google look to infuse AI into all their major offerings. A proof of this is some of the smart features that were announced in the conference - the smart compose option in Gmail, improved Google Assistant, Gboard, Google Duplex, and more. TPU 3.0 was needed, with the competition getting serious It comes as no surprise to anyone that almost all the major tech giants are investing in cloud-ready AI technology. These companies are specifically investing in hardware to make machine learning faster and more efficient, to make sense of the data at scale, and give intelligent predictions which are used to improve their operations. There are quite a few examples to demonstrate this. Facebook’s infrastructure is being optimized for the Caffe2 and Pytorch frameworks, designed to process the massive information it handles on a day to day basis. Intel have come up with their neural network processors in a bid to redefine AI. It is also common knowledge that even the cloud giants like Amazon want to build an efficient cloud infrastructure powered by Artificial Intelligence. Just a few days back, Microsoft previewed their Project Brainwave in the Build 2018 conference, claiming super-fast Artificial Intelligence capabilities which rivaled Google’s very own TPU. We can safely infer that Google needed a TPU 3.0 like hardware to join the elite list of prime enablers of Artificial Intelligence in the cloud, empowering efficient data management and processing. Check out our coverage of Google I/O 2018, for some exciting announcements on other Google products in store for the developers and Android fans. Read also AI chip wars: Is Brainwave Microsoft’s Answer to Google’s TPU? How machine learning as a service is transforming cloud The Deep Learning Framework Showdown: TensorFlow vs CNTK

In this article, we look at securing the private cloud using IAM. For IAM, OpenStack uses the Keystone project. Keystone provides the identity, token, catalog, and policy services, which are used specifically by OpenStack services. It is organized as a group of internal services exposed on one or many endpoints. For example, an authentication call validates the user and project credentials with the identity service. [box type="shadow" align="" class="" width=""]This article is an excerpt from the book,'Cloud Security Automation'. In this book, you'll learn how to work with OpenStack security modules and learn how private cloud security functions can be automated for better time and cost-effectiveness.[/box] Authentication Authentication is an integral part of an OpenStack deployment and so we must be careful about the system design. Authentication is the process of confirming a user's identity, which means that a user is actually who they claim to be. For example, providing a username and a password when logging into a system. Keystone supports authentication using the username and password, LDAP, and external authentication methods. After successful authentication, the identity service provides the user with an authorization token, which is further used for subsequent service requests. Transport Layer Security (TLS) provides authentication between services and users using X.509 certificates. The default mode for TLS is server-side only authentication, but we can also use certificates for client authentication. However, in authentication, there can also be the case where a hacker is trying to access the console by guessing your username and password. If we have not enabled the policy to handle this, it can be disastrous. For this, we can use the Failed Login Policy, which states that a maximum number of attempts are allowed for a failed login; after that, the account is blocked for a certain number of hours and the user will also get a notification about it. However, the identity service provided in Keystone does not provide a method to limit access to accounts after repeated unsuccessful login attempts. For this, we need to rely on an external authentication system that blocks out an account after a configured number of failed login attempts. Then, the account might only be unlocked with further side-channel intervention, or on request, or after a certain duration. We can use detection techniques to the fullest only when we have a prevention method available to save them from damage. In the detection process, we frequently review the access control logs to identify unauthorized attempts to access accounts. During the review of access control logs, if we find any hints of a brute force attack (where the user tries to guess the username and password to log in to the system), we can define a strong username and password or block the source of the attack (IP) through firewall rules. When we define firewall rules on Keystone node, it restricts the connection, which helps to reduce the attack surface. Apart from this, reviewing access control logs also helps to examine the account activity for unusual logins and suspicious actions, so that we can take corrective actions such as disabling the account. To increase the level of security, we can also utilize MFA for network access to the privileged user accounts. Keystone supports external authentication services through the Apache web server that can provide this functionality. Servers can also enforce client-side authentication using certificates. This will help to get rid of brute force and phishing attacks that may compromise administrator passwords. Authentication methods – internal and external Keystone stores user credentials in a database or may use an LDAP-compliant directory server. The Keystone identity database can be kept separate from databases used by other OpenStack services to reduce the risk of a compromise of the stored credentials. When we use the username and password to authenticate, identity does not apply policies for password strength, expiration, or failed authentication attempts. For this, we need to implement external authentication services. To integrate an external authentication system or organize an existing directory service to manage users account management, we can use LDAP. LDAP simplifies the integration process. In OpenStack authentication and authorization, the policy may be delegated to another service. For example, an organization that is going to deploy a private cloud and already has a database of employees and users in an LDAP system. Using this LDAP as an authentication authority, requests to the Identity service (Keystone) are transferred to the LDAP system, which allows or denies requests based on its policies. After successful authentication, the identity service generates a token for access to the authorized services. Now, if the LDAP has already defined attributes for the user such as the admin, finance, and HR departments, these must be mapped into roles and groups within identity for use by the various OpenStack services. We need to define this mapping into Keystone node configuration files stored at /etc/keystone/keystone.conf. Keystone must not be allowed to write to the LDAP used for authentication outside of the OpenStack Scope, as there is a chance to allow a sufficiently privileged Keystone user to make changes to the LDAP directory, which is not desirable from a security point of view. This can also lead to unauthorized access of other information and resources. So, if we have other authentication providers such as LDAP or Active Directory, then user provisioning always happens at other authentication provider systems. For external authentication, we have the following methods: MFA: The MFA service requires the user to provide additional layers of information for authentication such as a one-time password token or X.509 certificate (called MFA token). Once MFA is implemented, the user will have to enter the MFA token after putting the user ID and password in for a successful login. Password policy enforcement: Once the external authentication service is in place, we can define the strength of the user passwords to conform to the minimum standards for length, diversity of characters, expiration, or failed login attempts. Keystone also supports TLS-based client authentication. TLS client authentication provides an additional authentication factor, apart from the username and password, which provides greater reliability on user identification. It reduces the risk of unauthorized access when usernames and passwords are compromised. However, TLS-based authentication is not cost effective as we need to have a certificate for each of the clients. Authorization Keystone also provides the option of groups and roles. Users belong to groups where a group has a list of roles. All of the OpenStack services, such as Cinder, Glance, nova, and Horizon, reference the roles of the user attempting to access the service. OpenStack policy enforcers always consider the policy rule associated with each resource and use the user’s group or role, and their association, to determine and allow or deny the service access. Before configuring roles, groups, and users, we should document your required access control policies for the OpenStack installation. The policies must be as per the regulatory or legal requirements of the organization. Additional changes to the access control configuration should be done as per the formal policies. These policies must include the conditions and processes for creating, deleting, disabling, and enabling accounts, and for assigning privileges to the accounts. One needs to review these policies from time to time and ensure that the configuration is in compliance with the approved policies. For user creation and administration, there must be a user created with the admin role in Keystone for each OpenStack service. This account will provide the service with the authorization to authenticate users. Nova (compute) and Swift (object storage) can be configured to use the Identity service to store authentication information. For the test environment, we can have tempAuth, which records user credentials in a text file, but it is not recommended for the production environment. The OpenStack administrator must protect sensitive configuration files from unauthorized modification with mandatory access control frameworks such as SELinux or DAC. Also, we need to protect the Keystone configuration files, which are stored at /etc/keystone/keystone.conf, and also the X.509 certificates. It is recommended that cloud admin users must authenticate using the identity service (Keystone) and an external authentication service that supports two-factor authentication. Getting authenticated with two-factor authentication reduces the risk of compromised passwords. It is also recommended in the NIST guideline called NIST 800-53 IA-2(1). Which defines MFA for network access to privileged accounts, when one factor is provided by a separate device from the system being accessed. Policy, tokens, and domains In OpenStack, every service defines the access policies for its resources in a policy file, where a resource can be like an API access, it can create and attach Cinder volume, or it can create an instance. The policy rules are defined in JSON format in a file called policy.json. Only administrators can modify the service-based policy.json file, to control the access to the various resources. However, one has to also ensure that any changes to the access control policies do not unintentionally breach or create an option to breach the security of any resource. Any changes made to policy.json are applied immediately and it does not need any service restart. After a user is authenticated, a token is generated for authorization and access to an OpenStack environment. A token can have a variable lifespan, but the default value is 1 hour. It is also recommended to lower the lifespan of the token to a certain level so that within the specified timeframe the internal service can complete the task. If the token expires before task completion, the system can be unresponsive. Keystone also supports token revocation. For this, it uses an API to revoke a token and to list the revoked tokens. In OpenStack Newton release, there are four supported token types: UUID, PKI, PKIZ, and fernet. After the OpenStack Ocata release, there are two supported token types: UUID and fernet. We'll see all of these token types in detail here: UUID: These tokens are persistent tokens. UUID tokens are 32 bytes in length, which must be persisted in the backend. They are stored in the Keystone backend, along with the metadata for authentication. All of the clients must pass their UUID token to the Keystone (identity service) in order to validate it. PKI and PKIZ: These are signed documents that contain the authentication content, as well as the service catalog. The difference between the PKI and PKIZ is that PKIZ tokens are compressed to help mitigate the size issues of PKI (sometimes PKI tokens becomes very long). Both of these tokens have become obsolete after the Ocata release. The length of PKI and PKIZ tokens typically exceeds 1,600 bytes. The Identity service uses public and private key pairs and certificates in order to create and validate these tokens. Fernet: These tokens are the default supported token provider for OpenStack Pike Release. It is a secure messaging format explicitly designed for use in API tokens. They are nonpersistent, lightweight (fall in the range of 180 to 240 bytes), and reduce the operational overhead. Authentication and authorization metadata is neatly bundled into a message-packed payload, which is then encrypted and signed in as a fernet token. In the OpenStack, the Keystone Service domain is a high-level container for projects, users, and groups. Domains are used to centrally manage all Keystone-based identity components. Compute, storage, and other resources can be logically grouped into multiple projects, which can further be grouped under a master account. Users of different domains can be represented in different authentication backends and have different attributes that must be mapped to a single set of roles and privileges in the policy definitions to access the various service resources. Domain-specific authentication drivers allow the identity service to be configured for multiple domains, using domain-specific configuration files stored at keystone.conf. Federated identity Federated identity enables you to establish trusts between identity providers and the cloud environment (OpenStack Cloud). It gives you secure access to cloud resources using your existing identity. You do not need to remember multiple credentials to access your applications. Now, the question is, what is the reason for using federated identity? This is answered as follows: It enables your security team to manage all of the users (cloud or noncloud) from a single identity application It enables you to set up different identity providers on the basis of the application that somewhere creates an additional workload for the security team and leads the security risk as well It gives ease of life to users by proving them a single credential for all of the apps so that they can save the time they spend on the forgot password page Federated identity enables you to have a single sign-on mechanism. We can implement it using SAML 2.0. To do this, you need to run the identity service provider under Apache. We learned about securing your private cloud and the authentication process therein. If you've enjoyed this article, do check out 'Cloud Security Automation' for a hands-on experience of automating your cloud security and governance. Top 5 cloud security threats to look out for in 2018 Cloud Security Tips: Locking Your Account Down with AWS Identity Access Manager (IAM)

When it comes to Angular development, there are some things that are good to know and some things that we need to know to embark on our great journey. One of the things that is good to know is semantic versioning. This is good to know because it is the way the Angular team has chosen to deal with changes. This will hopefully make it easier to find the right solutions to future app development challenges when you go to https://angular.io/ or Stack Overflow and other sites to search for solutions. In this tutorial, we will discuss Angular components and few practical examples to help you get real world understanding of Angular components. This article is an excerpt from the book Learning Angular Second edition, written by Christoffer Noring & Pablo Deeleman. Web components Web components is a concept that encompasses four technologies designed to be used together to build feature elements with a higher level of visual expressivity and reusability, thereby leading to a more modular, consistent, and maintainable web. These four technologies are as follows: Templates: These are pieces of HTML that structure the content we aim to render. Custom elements: These templates not only contain traditional HTML elements, but also the custom wrapper items that provide further presentation elements or API functionalities. Shadow DOM: This provides a sandbox to encapsulate the CSS layout rules and JavaScript behaviors of each custom element. HTML imports: HTML is no longer constrained to host HTML elements, but to other HTML documents as well. In theory, an Angular component is indeed a custom element that contains a template to host the HTML structure of its layout, the latter being governed by a scoped CSS style sheet encapsulated within a shadow DOM container. Let's try to rephrase this in plain English. Think of the range input control type in HTML5. It is a handy way to give our users a convenient input control for entering a value ranging between two predefined boundaries. If you have not used it before, insert the following piece of markup in a blank HTML template and load it in your browser: <input id="mySlider" type="range" min="0" max="100" step="10"> You will see a nice input control featuring a horizontal slider in your browser. Inspecting such control with the browser developer tools will unveil a concealed set of HTML tags that were not present at the time you edited your HTML template. There you have an example of shadow DOM in action, with an actual HTML template governed by its own encapsulated CSS with advanced dragging functionality. You will probably agree that it would be cool to do that yourself. Well, the good news is that Angular gives you the tool set required for delivering this very same functionality, to build our own custom elements (input controls, personalized tags, and self-contained widgets). We can feature the inner HTML markup of our choice and our very own style sheet that is not affected (nor is impacted) by the CSS of the page hosting our component. Why TypeScript over other syntaxes to code Angular apps? Angular applications can be coded in a wide variety of languages and syntaxes: ECMAScript 5, Dart, ECMAScript 6, TypeScript, or ECMAScript 7. TypeScript is a typed superset of ECMAScript 6 (also known as ECMAScript 2015) that compiles to plain JavaScript and is widely supported by modern OSes. It features a sound object-oriented design and supports annotations, decorators, and type checking. The reason why we picked (and obviously recommend) TypeScript as the syntax of choice for instructing how to develop Angular applications is based on the fact that Angular itself is written in this language. Being proficient in TypeScript will give the developer an enormous advantage when it comes to understanding the guts of the framework. On the other hand, it is worth remarking that TypeScript's support for annotations and type introspection turns out to be paramount when it comes to managing dependency injection and type binding between components with a minimum code footprint. Check out the book, Learning Angular 2nd edition, to learn how to do this. Ultimately, you can carry out your Angular projects in plain ECMAScript 6 syntax if that is your preference. Even the examples provided in this book can be easily ported to ES6 by removing type annotations and interfaces, or replacing the way dependency injection is handled in TypeScript with the most verbose ES6 way. For the sake of brevity, we will only cover examples written in TypeScript. We recommend its usage because of its higher expressivity thanks to type annotations, and its neat way of approaching dependency injection based on type introspection out of such type annotations. Setting up our workspace with Angular CLI There are different ways to get started, either using the Angular quickstart repository on the https://angular.io/ site, or installing the scaffolding tool Angular CLI, or, you could use Webpack to set up your project. It is worth pointing out that the standard way of creating a new Angular project is through using Angular CLI and scaffold your project. Systemjs, used by the quickstart repository, is something that used to be the default way of building Angular projects. It is now rapidly diminishing, but it is still a valid way of setting up an Angular project. Setting up a frontend project today is more cumbersome than ever. We used to just include the necessary script with our JavaScript code and a link tag for our CSS and img tag for our [SN1] assets and so on. Life used to be simple. Then frontend development became more ambitious and we started splitting up our code in modules, we started using preprocessors for both our code and CSS. All in all, our projects became more complicated and we started to rely on build systems such as Grunt, Gulp, Webpack, and so on. Most developers out there are not huge fans of configuration, they just want to focus on building apps. Modern browsers, however, do more to support the latest ECMAScript standard and some browsers have even started to support modules, which are resolved at runtime. This is far from being widely supported though. In the meantime, we still have to rely on tools for bundling and module support. Setting up a project with leading frameworks such as React or Angular can be quite difficult. You need to know what libraries to import and ensure that files are processed in the correct order, which leads us to the topic of scaffolding tools. For AngularJS, it was quite popular to use Yeoman to scaffold up a new application quickly and get a lot of nice things preconfigured. React has a scaffolder tool called create-react-app, which you probably have saved and it saves countless hours for React developers. Scaffolder tools becomes almost a necessity as complexity grows, but also where every hour counts towards producing business value rather than fighting configuration problems. The main motivation behind creating the Angular CLI tool was to help developers focus on app building and not so much on configuration. Essentially, with a simple command, you should be able to scaffold an application, add a new construct to it, run tests, or create a production grade bundle. Angular CLI supports all that. Prerequisites for installing Angular CLI What you need to get started is to have Git and Node.js installed. Node.js will also install something called NPM, a node package manager that you will use later to install files you need for your project. After this is done, you are ready to set up your Angular application. You can find installation files to Node.js. The easiest way to have it installed is to go to the site: Installing Node.js will also install something called NPM, Node Package Manager, which you will need to install dependencies and more. The Angular CLI requires Node 6.9.0 and NPM 3 or higher. Currently on the site, you can choose between an LTS version and the current version. The LTS version should be enough. Angular CLI Installation Installing the Angular CLI is as easy as running the following command in your Terminal: npm install -g @angular/cli On some systems, you may need to have elevated permissions to do so; in that case, run your Terminal window as an administrator and on Linux/macOS instead run the command like this: sudo npm install -g @angular/cli Building your first app Once the Angular CLI is in place the time has come to create your first project. To do so place yourself in a directory of your choice and type the following: ng new <give it a name here> Type the following: ng new TodoApp This will create a directory called TodoApp. After you have run the preceding command, there are two things you need to do to see your app in a browser: Navigate to the just created directory Serve up the application This will be accomplished by the following commands: cd TodoApp npm start At this point, open up your browser on http://localhost:4200 and you should see the following: Testing your app The Angular CLI doesn't just come with code that makes your app work. It also comes with code that sets up testing and includes a test. Running the said test is as easy as typing the following in the Terminal: You should see the following: How come this works? Let's have a look at the package.json file that was just created and the scripts tag. Everything specified here can be run using the following syntax: npm run <key> In some cases, it is not necessary to type run and it will be enough to just type: npm <key> This is the case with the start and test commands. The following listing makes it clear that it is possible to run more commands than start and test that we just learned about: "scripts": { "ng": "ng", "start": "ng serve", "build": "ng build", "test": "ng test", "lint": "ng lint", "e2e": "ng e2e" } So far we have learned how to install the Angular CLI. Using the Angular CLI we have learned to:    Scaffold a new project.    Serve up the project and see it displayed in a browser.    Run tests. That is quite an accomplishment. We will revisit the Angular CLI in a later chapter as it is a very competent tool, capable of a lot more. Hello Angular We are about to take the first trembling steps into building our first component. The Angular CLI has already scaffolded our project and thereby carried out a lot of heavy lifting. All we need to do is to create new file and starting filling it with content. The million dollar question is what to type? So let's venture into building our first component. There are three steps you need to take in creating a component. Those are:    Import the component decorator construct.    Decorate a class with a component decorator.    Add a component to its module (this might be in two different places). Creating the component First off, let's import the component decorator: import { Component } from '@angular/core'; Then create the class for your component: class AppComponent { title:string = 'hello app'; } Then decorate your class using the Component decorator: @Component({ selector: 'app', template: `<h1>{{ title }}</h1>` }) export class AppComponent { title: string = 'hello app'; } We give the Component decorator, which is function, an object literal as an input parameter. The object literal consists at this point of the selector and template keys, so let's explain what those are. Selector A selector is what it should be referred to if used in a template somewhere else. As we call it app, we would refer to it as: <app></app> Template/templateUrl The template or templateUrl is your view. Here you can write HTML markup. Using the  template keyword, in our object literal, means we get to define the HTML markup in the same file as the component class. Were we to use templateUrl, we would then place our HTML markup in a separate file. The preceding  example also lists the following double curly braces, in the markup: <h1>{{ title }}</h1> This will be treated as an interpolation and the expression will be replaced with the value of AppComponent's title field. The component, when rendered, will therefore look like this: hello app Telling the module Now we need to introduce a completely new concept, an Angular module. All types of constructs that you create in Angular should be registered with a module. An Angular module serves as a facade to the outside world and it  is nothing more than a class that is decorated by the decorate @NgModule. Just like the @Component decorator, the @NgModule decorator takes an object literal as an input parameter. To register our component with our Angular module, we need to give the object literal the property declarations. The declarations property is of a type array and by adding our component to that array we are registering it with the Angular module. The following code shows the creation of an Angular module and the component being registered with it by being added to declarations keyword array: import { AppComponent } from './app.component'; @NgModule({ declarations: [AppComponent] }) export class AppModule {} At this point, our Angular module knows about the component. We need to add one more property to our module, bootstrap. The bootstrap keyword states that whatever is placed in here serves as the entry component for the entire application. Because we only have one component, so far, it makes sense to register our component with this bootstrap keyword: @NgModule({ declarations:  [AppComponent], bootstrap: [AppComponent] }) export class AppModule {} It's definitely possible to have more than one entry component, but the usual scenario is that there is only one. For any future components, however, we will only need to add them to the declarations property, to ensure the module knows about them. So far we have created a component and an Angular module and registered the component with said the module. We don't really have a working application yet, as there is one more step we need to take. We need to set up the bootstrapping. To summarize, we have shown how to get started with the Angular CLI and create your first Angular component efficiently. If you are interested to know more, check out Learning Angular Second edition, to get your way through Angular and create dynamic applications with it. Building Components Using Angular Why switch to Angular for web development – Interview Insights 8 built-in Angular Pipes in Angular 4 that you should know