Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7018 Articles
article-image-android-virtual-device-manager
Packt
06 Feb 2015
8 min read
Save for later

Android Virtual Device Manager

Packt
06 Feb 2015
8 min read
This article written by Belén Cruz Zapata, the author of the book Android Studio Essentials, teaches us the uses of the AVD Manager tool. It introduces us to the Google Play services. (For more resources related to this topic, see here.) The Android Virtual Device Manager (AVD Manager) is an Android tool accessible from Android Studio to manage the Android virtual devices that will be executed in the Android emulator. To open the AVD Manager from Android Studio, navigate to the Tools | Android | AVD Manager menu option. You can also click on the shortcut from the toolbar. The AVD Manager displays the list of the existing virtual devices. Since we have not created any virtual device, initially the list will be empty. To create our first virtual device, click on the Create Virtual Device button to open the configuration dialog. The first step is to select the hardware configuration of the virtual device. The hardware definitions are listed on the left side of the window. Select one of them, like the Nexus 5, to examine its details on the right side as shown in the following screenshot. Hardware definitions can be classified into one of these categories: Phone, Tablet, Wear or TV. We can also configure our own hardware device definitions from the AVD Manager. We can create a new definition using the New Hardware Profile button. The Clone Device button creates a duplicate of an existing device. Click on the New Hardware Profile button to examine the existing configuration parameters. The most important parameters that define a device are: Device Name: Name of the device. Screensize: Screen size in inches. This value determines the size category of the device. Type a value of 4.0 and notice how the Size value (on the right side) is normal. Now type a value of 7.0 and the Size field changes its value to large. This parameter along with the screen resolution also determines the density category. Resolution: Screen resolution in pixels. This value determines the density category of the device. Having a screen size of 4.0 inches, type a value of 768 x 1280 and notice how the density value is 400 dpi. Change the screen size to 6.0 inches and the density value changes to hdpi. Now change the resolution to 480 x 800 and the density value is mdpi. RAM: RAM memory size of the device. Input: Indicate if the home, back, or menu buttons of the device are available via software or hardware. Supported device states: Check the allowed states. Cameras: Select if the device has a front camera or a back camera. Sensors: Sensors available in the device: accelerometer, gyroscope, GPS, and proximity sensor. Default Skin: Select additional hardware controls. Create a new device with a screen size of 4.7 inches, a resolution of 800 x 1280, a RAM value of 500 MiB, software buttons, and both portrait and landscape states enabled. Name it as My Device. Click on the Finish button. The hardware definition has been added to the list of configurations. Click on the Next button to continue the creation of a new virtual device. The next step is to select the virtual device system image and the target Android platform. Each platform has its architecture, so the system images that are installed on your system will be listed along with the rest of the images that can be downloaded (Show downloadable system images box checked). Download and select one of the images of the Lollipop release and click on the Next button. Finally, the last step is to verify the configuration of the virtual device. Enter the name of the Android Virtual Device in the AVD Name field. Give the virtual device a meaningful name to recognize it easily, such as AVD_nexus5_api21. Click on the Show Advanced Settings button. The settings that we can configure for the virtual device are the following: Emulation Options: The Store a snapshot for faster startup option saves the state of the emulator in order to load faster the next time. The Use Host GPU tries to accelerate the GPU hardware to run the emulator faster. Custom skin definition: Select if additional hardware controls are displayed in the emulator. Memory and Storage: Select the memory parameters of the virtual device. Let the default values, unless a warning message is shown; in this case, follow the instructions of the message. For example, select 1536M for the RAM memory and 64 for the VM Heap. The Internal Storage can also be configured. Select for example: 200 MiB. Select the size of the SD Card or select a file to behave as the SD card. Device: Select one of the available device configurations. These configurations are the ones we tested in the layout editor preview. Select the Nexus 5 device to load its parameters in the dialog. Target: Select the device Android platform. We have to create one virtual device with the minimum platform supported by our application and another virtual device with the target platform of our application. For this first virtual device, select the target platform, Android 4.4.2 - API Level 19. CPU/ABI: Select the device architecture. The value of this field is set when we select the target platform. Each platform has its architecture, so if we do not have it installed, the following message will be shown; No system images installed for this target. To solve this, open the SDK Manager and search for one of the architectures of the target platform, ARM EABI v7a System Image or Intel x86 Atom System Image. Keyboard: Select if a hardware keyboard is displayed in the emulator. Check it. Skin: Select if additional hardware controls are displayed in the emulator. You can select the Skin with dynamic hardware controls option. Front Camera: Select if the emulator has a front camera or a back camera. The camera can be emulated or can be real by the use of a webcam from the computer. Select None for both cameras. Keyboard: Select if a hardware keyboard is displayed in the emulator. Check it. Network: Select the speed of the simulated network and select the delay in processing data across the network. The new virtual device is now listed in the AVD Manager. Select the recently created virtual device to enable the remaining actions: Start: Run the virtual device. Edit: Edit the virtual device configuration. Duplicate: Creates a new device configuration displaying the last step of the creation process. You can change its configuration parameters and then verify the new device. Wipe Data: Removes the user files from the virtual device. Show on Disk: Opens the virtual device directory on your system. View Details: Open a dialog detailing the virtual device characteristics. Delete: Delete the virtual device. Click on the Start button. The emulator will be opened as shown in the following screenshot. Wait until it is completely loaded, and then you will be able to try it. In Android Studio, open the main layout with the graphical editor and click on the list of the devices. As the following screenshot shows, our custom device definition appears and we can select it to preview the layout: Navigation Editor The Navigation Editor is a tool to create and structure the layouts of the application using a graphical viewer. To open this tool navigate to the Tools | Android | Navigation Editor menu. The tool opens a file in XML format named main.nvg.xml. This file is stored in your project at /.navigation/app/raw/. Since there is only one layout and one activity in our project, the navigation editor only shows this main layout. If you select the layout, detailed information about it is displayed on the right panel of the editor. If you double-click on the layout, the XML layout file will be opened in a new tab. We can create a new activity by right-mouse clicking on the editor and selecting the New Activity option. We can also add transitions from the controls of a layout by shift clicking on a control and then dragging to the target activity. Open the main layout and create a new button with the label Open Activity: <Button        android_id="@+id/button_open"        android_layout_width="wrap_content"        android_layout_height="wrap_content"        android_layout_below="@+id/button_accept"        android_layout_centerHorizontal="true"        android_text="Open Activity" /> Open the Navigation Editor and add a second activity. Now the navigation editor displays both activities as the next screenshot shows. Now we can add the navigation between them. Shift-drag from the new button of the main activity to the second activity. A blue line and a pink circle have been added to represent the new navigation. Select the navigation relationship to see its details on the right panel as shown in the following screenshot. The right panel shows the source the activity, the destination activity and the gesture that triggers the navigation. Now open our main activity class and notice the new code that has been added to implement the recently created navigation. The onCreate method now contains the following code: findViewById(R.id.button_open).setOnClickListener( new View.OnClickListener() { @Override public void onClick(View v) { MainActivity.this.startActivity( new Intent(MainActivity.this, Activity2.class)); } }); This code sets the onClick method of the new button, from where the second activity is launched. Summary This article thought us about the Navigation Editor tool. It also showed how to integrate the Google Play services with a project in Android Studio. In this article, we got acquainted to the AVD Manager tool. Resources for Article: Further resources on this subject: Android Native Application API [article] Creating User Interfaces [article] Android 3.0 Application Development: Multimedia Management [article]
Read more
  • 0
  • 0
  • 14118

article-image-2018-data-science-part-2-of-3
Savia Lobo
04 Jan 2018
7 min read
Save for later

2018 new year resolutions to thrive in the Algorithmic World - Part 2 of 3

Savia Lobo
04 Jan 2018
7 min read
In our first resolution, we talked about learning the building blocks of data science i.e developing your technical skills. In this second resolution, we walk you through steps to stay relevant in your field and how to dodge jobs that have a high possibility of getting automated in the near future. 2nd Resolution: Stay relevant in your field even as job automation is on the rise (Time investment: half an hour every day, 2 hours on weekends) Once you have got your fundamentals right, it is important to stay relevant through continuous learning and reskilling. In addition to honing your technical skills, you must also deepen your domain expertise and keep adding to your portfolio of soft skills to stay ahead of not the just human competition but also to thrive in an automated job market. We list below some simple ways to do all these in a systematic manner. All it requires is a commitment of half an hour to one hour of your time daily for your professional development. 1. Commit to and execute a daily learning-practice-participation ritual Here are some ways to stay relevant. Follow data science blogs and podcasts relevant to your area of interest. Here are some of our favorites: Data Science 101, the journey of a data scientist The Data Skeptic for a healthy dose of scientific skepticism Data Stories for data visualization This Week in Machine Learning & AI for informative discussions with prominent people in the data science/machine learning community Linear Digressions, a podcast co-hosted by a data scientist and a software engineer attempting to make data science accessible You could also follow individual bloggers/vloggers in this space like Siraj Raval, Sebastian Raschka, Denny Britz, Rodney Brookes, Corinna Cortes, Erin LeDell Newsletters are a great way to stay up-to-date and to get a macro-level perspective. You don’t have to spend an awful lot of time doing the research yourself on many different subtopics. So, subscribe to useful newsletters on data science. You can subscribe to our newsletter here. It is a good idea to subscribe to multiple newsletters on your topic of interest to get a balanced and comprehensive view of the topic. Try to choose newsletters that have distinct perspectives, are regular and are published by people passionate about the topic. Twitter gives a whole new meaning to ‘breaking news’. Also, it is a great place to follow contemporary discussions on topics of interest where participation is open to all. When done right, it can be a gold mine for insights and learning. But often it is too overwhelming as it is viewed as a broadcasting marketing tool. Follow your role models in data science on Twitter. Or you could follow us on Twitter @PacktDataHub for curated content from key data science influencers and our own updates about the world of data science. You could also click here to keep a track of 737 twitter accounts most followed by the members of the NIPS2017 community. Quora, Reddit, Medium, and StackOverflow are great places to learn about topics in depth when you have a specific question in mind or a narrow focus area. They help you get multiple informed opinions on topics. In other words, when you choose a topic worth learning, these are great places to start. Follow them up by reading books on the topic and also by reading the seminal papers to gain a robust technical appreciation. Create a Github account and participate in Kaggle competitions. Nothing sticks as well as learning by doing. You can also browse into Data Helpers, a site voluntarily set up by Angela Bass where interested data science people can offer to help newcomers with their queries on entering the required field and anything else. 2. Identify your strengths and interests to realign your career trajectory OK, now that you have got your daily learning routine in place, it is time to think a little more strategically about your career trajectory, goals and eventually the kind of work you want to be doing. This means: Getting out of jobs that can be automated Developing skills that augment or complement AI driven tasks Finding your niche and developing deep domain expertise that AI will find hard to automate in the near future Here are some ideas to start thinking about some of the above ideas. The first step is to assess your current job role and understand how likely it is to get automated. If you are in a job that has well-defined routines and rules to follow, it is quite likely to go the AI job apocalypse route. Eg: data entry, customer support that follows scripts, invoice processing, template-based software testing or development etc. Even “creative” job such as content summarization, news aggregation, template-based photo-editing/video editing etc fall in this category. In the world of data professionals, jobs like data cleaning, database optimization, feature generation, even model building (gasp!) among others could head the same way given the right incentives. Choose today to transition out of jobs that may not exist in the next 10 years. Then instead of hitting the panic button, invest in redefining your skills in a way that would be helpful in the long run. If you are a data professional, skills such as data interpretation, data-driven storytelling,  data pipeline architecture and engineering, feature engineering, and others that require a high level of human judgment skills are least likely to be replicated by machines anytime soon. By mastering skills that complement AI driven tasks and jobs, you should be able to present yourself as a lucrative option to potential employers in a highly competitive job market space.    In addition to reskilling, try to find your niche and dive deep. By niche, we mean, if you are a data scientist, choose a specific technical aspect in your field, something that interests you. It could be anything from computer vision to NLP to even a class of algorithms like neural nets or a type of problem that machine learning solves such as recommender systems or classification systems. It could even be a specific phase of a data science project such as data visualization or data pipeline engineering. Master your niche while keeping up with what’s happening in other related areas. Next, understand where your strengths lie. In other words, what your expertise is, what industry or domain do you understand well or have amassed experience in. For instance, NLP, a subset of machine learning abilities, can be applied to customer reviews to mine useful insights, perform sentiment analysis, build recommendation systems in conjunction with predictive modeling among other things. In order to build an NLP model to mine some kind of insights from customer feedback, we must have some idea of what we are looking for. Your domain expertise can be of great value here. If you are in the publishing business, you would know what keywords matter most in reviews and more importantly why they matter and how to convert the findings into actionable insights - aspects that your model or even a machine learning engineer outside your industry may not understand or appreciate. Take the case of Brendan Frey and the team of researchers at Deep Genomics as a real-world example. They applied AI and machine learning (their niche expertise) to build a neural network to identify pathological mutations in genes (their domain expertise). Their knowledge of how genes get created and how they work, what a mutation looks like etc helped them feed the features and hyperparameters into their model. Similarly, you can pick up any of your niche skills and apply them in whichever field you find interesting and worthwhile. Based on your domain knowledge and area of expertise, it could range from sorting a person into a Hogwarts house because you are a Harry Potter fan to sorting them into potential patients with a high likelihood to develop diabetes because you have a background in biotechnology.   This brings us to the next resolution where we cover aspects related to how your work will come to define you and why it matters that you choose your projects well.   
Read more
  • 0
  • 0
  • 14114

article-image-introduction-scala
Packt
01 Nov 2016
8 min read
Save for later

Introduction to Scala

Packt
01 Nov 2016
8 min read
In this article by Diego Pacheco, the author of the book, Building applications with Scala, we will see the following topics: Writing a program for Scala Hello World using the REPL Scala language – the basics Scala variables – var and val Creating immutable variables (For more resources related to this topic, see here.) Scala Hello World using the REPL Let's get started. Go ahead, open your terminal, and type $ scala in order to open the Scala REPL. Once the REPL is open, you can just type "Hello World". By doing this, you are performing two operations – eval and print. The Scala REPL will create a variable called res0 and store your string there, and then it will print the content of the res0 variable. Scala REPL Hello World program $ scala Welcome to Scala 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_77). Type in expressions for evaluation. Or try :help. scala> "Hello World" res0: String = Hello World scala> Scala is a hybrid language, which means it is both object-oriented (OO) and functional. You can create classes and objects in Scala. Next, we will create a complete Hello World application using classes. Scala OO Hello World program $ scala Welcome to Scala 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_77). Type in expressions for evaluation. Or try :help. scala> object HelloWorld { | def main(args:Array[String]) = println("Hello World") | } defined object HelloWorld scala> HelloWorld.main(null) Hello World scala> First things first, you need to realize that we use the word object instead of class. The Scala language has different constructs, compared with Java. Object is a Singleton in Scala. It's the same as you code the Singleton pattern in Java. Next, we see the word def that is used in Scala to create functions. In this program, we create the main function just as we do in Java, and we call the built-in function, println, in order to print the String Hello World. Scala imports some java objects and packages by default. Coding in Scala does not require you to type, for instance, System.out.println("Hello World"), but you can if you want to, as shown in the following:. $ scala Welcome to Scala 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_77). Type in expressions for evaluation. Or try :help. scala> System.out.println("Hello World") Hello World scala> We can and we will do better. Scala has some abstractions for a console application. We can write this code with less lines of code. To accomplish this goal, we need to extend the Scala class App. When we extend from App, we are performing inheritance, and we don't need to define the main function. We can just put all the code on the body of the class, which is very convenient, and which makes the code clean and simple to read. Scala HelloWorld App in the Scala REPL $ scala Welcome to Scala 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_77). Type in expressions for evaluation. Or try :help. scala> object HelloWorld extends App { | println("Hello World") | } defined object HelloWorld scala> HelloWorld object HelloWorld scala> HelloWorld.main(null) Hello World scala> After coding the HelloWorld object in the Scala REPL, we can ask the REPL what HelloWorld is and, as you might realize, the REPL answers that HelloWorld is an object. This is a very convenient Scala way to code console applications because we can have a Hello World application with just three lines of code. Sadly, the same program in Java requires way more code, as you will see in the next section. Java is a great language for performance, but it is a verbose language compared with Scala. Java Hello World application package scalabook.javacode.chap1; public class HelloWorld { public static void main(String args[]){ System.out.println("Hello World"); } } The Java application required six lines of code, while in Scala, we were able to do the same with 50% less code(three lines of code). This is a very simple application; when we are coding complex applications, the difference gets bigger as a Scala application ends up with way lesser code than that of Java. Remember that we use an object in Scala in order to have a Singleton(Design Pattern that makes sure you have just one instance of a class), and if we want to do the same in Java, the code would be something like this: package scalabook.javacode.chap1; public class HelloWorldSingleton { private HelloWorldSingleton(){} private static class SingletonHelper{ private static final HelloWorldSingleton INSTANCE = new HelloWorldSingleton(); } public static HelloWorldSingleton getInstance(){ return SingletonHelper.INSTANCE; } public void sayHello(){ System.out.println("Hello World"); } public static void main(String[] args) { getInstance().sayHello(); } } It's not just about the size of the code, but it is all about consistency and the language providing more abstractions for you. If you write less code, you will have less bugs in your software at the end of the day. Scala language – the basics Scala is a statically typed language with a very expressive type system, which enforces abstractions in a safe yet coherent manner. All values in Scala are Java objects (but primitives that are unboxed at runtime) because at the end of the day, Scala runs on the Java JVM. Scala enforces immutability as a core functional programing principle. This enforcement happens in multiple aspects of the Scala language, for instance, when you create a variable, you do it in an immutable way, and when you use a collection, you use an immutable collection. Scala also lets you use mutable variables and mutable structures, but it favors immutable ones by design. Scala variables – var and val When you are coding in Scala, you create variables using either the var operator or the val operator. The var operator allows you to create mutable states, which is fine as long as you make it local, stick to the core functional programing principles, and avoid mutable shared state. Using var in the Scala REPL $ scala Welcome to Scala 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_77). Type in expressions for evaluation. Or try :help. scala> var x = 10 x: Int = 10 scala> x res0: Int = 10 scala> x = 11 x: Int = 11 scala> x res1: Int = 11 scala> However, Scala has a more interesting construct called val. Using the val operator makes your variables immutable, which means that you can't change their values after you set them. If you try to change the value of a val variable in Scala, the compiler will give you an error. As a Scala developer, you should use val as much as possible because that's a good functional programing mindset, and it will make your programs better and more correct. In Scala, everything is an object; there are no primitives – the var and val rules apply for everything, be it Int, String, or even a class. Using val in the Scala REPL $ scala Welcome to Scala 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_77). Type in expressions for evaluation. Or try :help. scala> val x = 10 x: Int = 10 scala> x res0: Int = 10 scala> x = 11 <console>:12: error: reassignment to val x = 11 ^ scala> x res1: Int = 10 scala> Creating immutable variables Right. Now let's see how we can define the most common types in Scala, such as Int, Double, Boolean, and String. Remember that you can create these variables using val or var, depending on your requirement. Scala variable types at the Scala REPL $ scala Welcome to Scala 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_77). Type in expressions for evaluation. Or try :help. scala> val x = 10 x: Int = 10 scala> val y = 11.1 y: Double = 11.1 scala> val b = true b: Boolean = true scala> val f = false f: Boolean = false scala> val s = "A Simple String" s: String = A Simple String scala> For these variables, we did not define the type. The Scala language figures it out for us. However, it is possible to specify the type if you want. In Scala, the type comes after the name of the variable, as shown in the following section. Scala variables with explicit typing at the Scala REPL $ scala Welcome to Scala 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_77). Type in expressions for evaluation. Or try :help. scala> val x:Int = 10 x: Int = 10 scala> val y:Double = 11.1 y: Double = 11.1 scala> val s:String = "My String " s: String = "My String " scala> val b:Boolean = true b: Boolean = true scala> Summary In this article, we learned about some basic constructs and concepts of the Scala language, with functions, collections, and OO in Scala. Resources for Article: Further resources on this subject: Making History with Event Sourcing [article] Creating Your First Plug-in [article] Content-based recommendation [article]
Read more
  • 0
  • 0
  • 14114

article-image-configuring-hbase
Packt
25 Jan 2016
14 min read
Save for later

Configuring HBase

Packt
25 Jan 2016
14 min read
In this article by Ruchir Choudhry, the author of the book HBase High Performance Cookbook, we will cover the configuration and deployment of HBase. (For more resources related to this topic, see here.) Introduction HBase is an open source, nonrelational, column-oriented distributed database modeled after Google's Cloud BigTable and written in Java. It is developed as part of Apache Software Foundation's Apache Hadoop project, and it runs on top of Hadoop Distributed File System (HDFS), providing BigTable-like capabilities for Hadoop. It's a column-oriented database, which is empowered by a fault-tolerant distributed file structure knows as HDFS. In addition to this, it also provides advanced features, such as auto sharding, load balancing, in-memory caching, replication, compression, near real-time lookups, strong consistency (using multiversions), block caches, and bloom filters for real-time queries and an array of client APIs. Throughout the chapter, we will discuss how to effectively set up mid and large size HBase clusters on top of the Hadoop and HDFS framework. This article will help you set up an HBase on a fully distributed cluster. For the cluster setup, we will consider redhat-6.2 Linux 2.6.32-220.el6.x86_64 #1 SMP Wed Nov 9 08:03:13 EST 2011 x86_64 x86_64 GNU/Linux, which will have six nodes. Configuration and Deployment Before we start HBase in a fully distributed mode, we will first be setting up Hadoop-2.4.0 in a distributed mode, and then, on top of a Hadoop cluster, we will set up HBase because it stores data in Hadoop Distributed File System (HDFS). Check the permissions of the users; HBase must have the ability to create a directory. Let's create two directories in which the data for NameNode and DataNode will reside: drwxrwxr-x 2 app app 4096 Jun 19 22:22 NameNodeData drwxrwxr-x 2 app app 4096 Jun 19 22:22 DataNodeData -bash-4.1$ pwd /u/HbaseB/hadoop-2.4.0 -bash-4.1$ ls -lh total 60K drwxr-xr-x 2 app app 4.0K Mar 31 08:49 bin drwxrwxr-x 2 app app 4.0K Jun 19 22:22 DataNodeData drwxr-xr-x 3 app app 4.0K Mar 31 08:49 etc Getting Ready Following are the steps to install and configure HBase: The first step to start is to choose a Hadoop cluster. Then, get the hardware details required for it. Get the software required to perform the setup. Get the OS required to do the setup. Perform the configuration steps. We will require the following components for NameNode: Components Details Type of systems An operating system redhat-6.2 Linux 2.6.32-220.el6.x86_64 #1 SMP Wed Nov 9 08:03:13 EST 2011 x86_64 x86_64 GNU/Linux, or other standard linux kernel.   Hardware/CPUS 16 to 24 CPUS cores. NameNode /Secondry NameNode. Hardware/RAM 64 to 128 GB. In special cases, 128 GB to 512 GB RAM. NameNode/Secondry NameNodes. Hardware/storage Both NameNode servers should have highly reliable storage for their namespace storage and edit log journaling. Typically, hardware RAID and/or reliable network storage are justifiable options. Note that the previous commands including an onsite disk replacement option in your support contract so that a failed RAID disk can be replaced quickly. NameNode/Secondry Namenodes.   RAID: Raid is nothing but a Random Access Inexpensive Drive or Independent Disk; there are many levels of RAID drives, but for Master or NameNode, RAID-1 will be enough. JBOD: This stands for Just a Bunch of Disk. The design is to have multiple hard drives stacked over each other with no redundancy. The calling software needs to take care of the failure and redundancy. In essence, it works as a single logical volume. The following screenshot shows the working mechanism of RAID and JBOD: Before we start for the cluster setup, a quick recap of the Hadoop setup is essential, with brief descriptions. How to do it… Let's create a directory where you will have all the software components to be downloaded: For simplicity, let's take this as /u/HbaseB. Create different users for different purposes. The format will be user/group; this is essentially required to differentiate various roles for specific purposes: HDFS/Hadoop: This is for the handling of Hadoop-related setups Yarn/Hadoop: This is for Yarn-related setups HBase/Hadoop Pig/Hadoop Hive/Hadoop Zookeeper/Hadoop HCat/Hadoop Set up directories for the Hadoop cluster: let's assume /u as a shared mount point; we can create specific directories, which will be used for specific purposes: -bash-4.1$ ls -ltr total 32 drwxr-xr-x 9 app app 4096 Oct 7 2013 hadoop-2.2.0 drwxr-xr-x 10 app app 4096 Feb 20 10:58 zookeeper-3.4.6 drwxr-xr-x 15 app app 4096 Apr 5 08:44 pig-0.12.1 drwxrwxr-x 7 app app 4096 Jun 30 00:57 hbase-0.98.3-hadoop2 drwxrwxr-x 8 app app 4096 Jun 30 00:59 apache-hive-0.13.1-bindrwxrwxr-x 7 app app 4096 Jun 30 01:04 mahout-distribution-0.9 Make sure that you have adequate privileges in the folder to add, edit, and execute a command. Also, you must set up password-less communication between different machines, such as from the name node to DataNode and from HBase Master to all the region server nodes. Refer to this webpage to learn how to do this: http://www.debian-administration.org/article/152/Password-less_logins_with_OpenSSH. Here, we will list the procedure to achieve the end result of the recipe. This section will follow a numbered bullet form. We do not need to explain the reason we are following a procedure. Numbered single sentences will do fine. Let's assume there is a /u directory and you have downloaded the entire stack of software from /u/HbaseB/hadoop-2.2.0/etc/hadoop/; look for the core-site.xml file. Place the following lines in this file: configuration> <property> <name>fs.default.name</name> <value>hdfs://mynamenode-hadoop:9001</value> <description>The name of the default file system. </description> </property> </configuration> You can specify a port that you want to use; it should not clash with the ports that are already in use by the system for various purposes. A quick look at this link can provide more specific details about this; complete detail on this topic is out of the scope of this book. You can refer to http://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers. Save the file. This helps us create a master/NameNode directory. Now let's move on to set up secondary nodes. Edit /u/HbaseB/hadoop-2.4.0/etc/hadoop/ and look for the core-site.xml file: <configuration> <property> <name>fs.checkpoint.dir</name> <value>/u/dn001/hadoop/hdf/secdn /u/dn002/hadoop/hdfs/secdn </value> <description>A comma separated list of paths. Use the list of directories from $FS_CHECKPOINT_DIR. example, /u/dn001/hadoop/hdf/secdn,/u/dn002/hadoop/hdfs/secd n </description> </property> </configuration> The separation of the directory structure is for the purpose of the clean separation of the hdfs block separation and to keep the configurations as simple as possible. This also allows us to do proper maintenance. Now let's move toward changing the setup for hdfs; the file location will be /u/HbaseB/hadoop-2.4.0/etc/hadoop/hdfs-site.xmlfor NameNode: <property> <name>dfs.name.dir</name> <value> /u/nn01/hadoop/hdfs/nn/u/nn02/hadoop/hdfs/nn </value> <description> Comma separated list of path, Use the list of directories </description> </property> for DataNode: <property> <name>dfs.data.dir</name> <value>/u/dnn01/hadoop/hdfs/dn,/u/dnn02/hadoop/hdfs/dn </value> <description>Comma separated list of path, Use the list of directories </description> </property> Now let's go for NameNode for the HTTP address or to NameNode using the HTTP protocol: <property> <name>dfs.http.address</name> <value>namenode.full.hostname:50070</value> <description>Enter your NameNode hostname for http access. </description> </property> The HTTP address for the secondary NameNode is as follows: <property> <name>dfs.secondary.http.address</name> <value> secondary.namenode.full.hostname:50090 </value> <description> Enter your Secondary NameNode hostname. </description> </property> We can go for an HTTPS setup for NameNode as well, but let's keep this optional for now: Now let's look for the Yarn setup in the /u/HbaseB/ hadoop-2.2.0/etc/hadoop/ yarn-site.xml file: For the resource tracker that's a part of the Yarn resource manager, execute the following code: <property> <name>yarn.resourcemanager.resourcetracker.address</name> <value>yarnresourcemanager.full.hostname:8025</value> <description>Enter your yarn Resource Manager hostname.</description> </property> For the resource schedule that's part of the Yarn resource scheduler, execute the following code: <property> <name>yarn.resourcemanager.scheduler.address</name> <value>resourcemanager.full.hostname:8030</value> <description>Enter your ResourceManager hostname</description> </property> For scheduler address, execute the following code: <property> <name>yarn.resourcemanager.address</name> <value>resourcemanager.full.hostname:8050</value> <description>Enter your ResourceManager hostname.</description> </property> For scheduler admin address, execute the following code: <property> <name>yarn.resourcemanager.admin.address</name> <value>resourcemanager.full.hostname:8041</value> <description>Enter your ResourceManager hostname.</description> </property> To set up the local directory, execute the following code: <property> <name>yarn.nodemanager.local-dirs</name> <value>/u/dnn01/hadoop/hdfs /yarn,/u/dnn02/hadoop/hdfs/yarn </value> <description>Comma separated list of paths. Use the list of directories from,.</description> </property> To set up the log location, execute the following code: <property> <name>yarn.nodemanager.logdirs</name> <value>/u/var/log/hadoop/yarn</value> <description>Use the list of directories from $YARN_LOG_DIR. <description> </property> This completes the configuration changes required for Yarn Now let's make the changes for MapReduce. Open /u/HbaseB/ hadoop-2.2.0/etc/hadoop/mapred-site.xml. Now let's place this configuration setup in mapred-site.xml and place this between <configuration></configuration>: <property> <name>mapreduce.jobhistory.address</name> <value>jobhistoryserver.full.hostname:10020</value> <description>Enter your JobHistoryServer hostname.</description> </property> Once we have configured MapReduce, we can move on to configuring HBase. Let's go to the /u/HbaseB/hbase-0.98.3-hadoop2/conf path and open the hbase-site.xml file. You will see a template that has <configuration></configurations>. We need to add the following lines between the starting and ending tags: <property> <name>hbase.rootdir</name> <value>hdfs://hbase.namenode.full.hostname:8020/apps/hbase/data</value> <description> Enter the HBase NameNode server hostname</description> </property> <property> <!—this id for binding address --> <name>hbase.master.info.bindAddress</name> <value>$hbase.master.full.hostname</value> <description>Enter the HBase Master server hostname</description> </property> This competes the HBase changes. ZooKeeper: Now let's focus on the setup of ZooKeeper. In distributed a environment, let's go to /u/HbaseB/zookeeper-3.4.6/conf locations, rename zoo_sample.cfg to zoo.cfg, and place the details as follows: yourzooKeeperserver.1=zoo1:2888:3888 yourZooKeeperserver.2=zoo2:2888:3888 If you want to test this setup locally, use different port combinations. Atomic broadcasting is an atomic messaging system that keeps all the servers in sync and provides reliable delivery, total orders, casual orders, and so on. Region servers: Before concluding, let's go to the region server setup process. Go to the /u/HbaseB/hbase-0.98.3-hadoop2/conf folder and edit the regionserver file. Specify the region servers accordingly: RegionServer1 RegionServer2 RegionServer3 RegionServer4 Copy all the configuration files of Hbase and ZooKeeper to the relative host dedicated for Hbase and ZooKeeper. Let's quickly validate the setup that we worked on: Sudo su $HDFS_USER /u/HbaseB/hadoop-2.2.0/bin/hadoop namenode -format /u/HbaseB/hadoop-2.4.0/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR start namenode Now let's go to the secondary nodes: Sudo su $HDFS_USER /u/HbaseB/hadoop-2.2.0/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR start secondarynamenode Now let's perform all the steps for DataNode: Sudo su $HDFS_USER /u/HbaseB/hadoop-2.2.0/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR start datanode Test 01> See if you can reach from your browser http://namenode.full.hostname:50070 Test 02> sudo su $HDFS_USER /u/HbaseB/hadoop-2.2.0/sbin/hadoop dfs -copyFromLocal /tmp/hello.txt /u/HbaseB/hadoop-2.2.0/sbin/hadoop dfs –ls you must see hello.txt once the command executes. Test 03> Browse http://datanode.full.hostname:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=/&nnaddr=$datanode.full.hostname:8020 you should see the details on the datanode. Validate the Yarn and MapReduce setup by following these steps: Execute the command from Resource Manager: <login as $YARN_USER and source the directories.sh companion script> /u/HbaseB/hadoop-2.2.0/sbin /yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager Execute the command from Node Manager <login as $YARN_USER and source the directories.sh companion script> /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager Execute the following commands: hadoop fs -mkdir /app-logs hadoop fs -chown $YARN_USER /app-logs hadoop fs -chmod 1777 /app-logs Execute MapReduce Sudo su $HDFS_USER /u/HbaseB/hadoop-2.2.0/sbin/hadoop fs -mkdir -p /mapred/history/done_intermediate /u/HbaseB/hadoop-2.2.0/sbin/hadoop fs -chmod -R 1777 /mapred/history/done_intermediate /u/HbaseB/hadoop-2.2.0/sbin/hadoop fs -mkdir -p /mapred/history/done /u/HbaseB/hadoop-2.2.0/sbin/hadoop fs -chmod -R 1777 /mapred/history/done /u/HbaseB/hadoop-2.2.0/sbin/hadoop fs -chown -R mapred /mapred export HADOOP_LIBEXEC_DIR=/u/HbaseB/hadoop-2.2.0/libexec/ export HADOOP_MAPRED_HOME=/=/u/HbaseB/hadoop-2.2.0/hadoop-mapreduceexport HADOOP_MAPRED_LOG_DIR==/u/HbaseB/hadoop-2.2.0//mapred Start the jobhistory servers: <login as $MAPRED_USER and source the directories.sh companion script> /u/HbaseB/hadoop-2.2.0/sbin/mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR Test 01: from the browser or from curl use the link to browse. http://resourcemanager.full.hostname:8088/ Test 02: Sudo su $HDFS_USER /u/HbaseB/hadoop-2.2.0/bin/hadoop jar /u/HbaseB/hadoop-2.2.0/hadoop-mapreduce/hadoop-mapreduce-examples-2.0.2.1-alpha.jar teragen 100 /test/10gsort/input /u/HbaseB/hadoop-2.2.0/bin/hadoop jar /u/HbaseB/hadoop-2.2.0/hadoop-mapreduce/hadoop-mapreduce-examples-2.0.2.1-alpha.jar Validate the HBase setup:     Login as $HDFS_USER /u/HbaseB/hadoop-2.2.0/bin/hadoop fs –mkdir /apps/hbase /u/HbaseB/hadoop-2.2.0/bin/hadoop fs –chown –R /apps/hbase      Now login as $HBASE_USER /u/HbaseB/hbase-0.98.3-hadoop2/bin/hbas-daemon.sh –-config $HBASE_CONF_DIR start master this will start the master node      Now let’s move to HBase Region server nodes: /u/HbaseB/hbase-0.98.3-hadoop2/bin/hbase-daemon.sh –config $HBASE_CONF_DIR start regionservers this will start the regionservers For single machine direct sudo ./hbase master start can also be used. Please check the logs in case of any logs. Now lets login using Sudo su- $HBASE_USER ./hbase shell will connect us to the hbase to the master. Validate the ZooKeeper setup: -bash-4.1$ sudo ./zkServer.sh start JMX enabled by default Using config: /u/HbaseB/zookeeper-3.4.6/bin/../conf/zoo.cfg Starting zookeeper ... STARTED You can also pipe the log to the ZooKeeper logs. /u/logs//u/HbaseB/zookeeper-3.4.6/zoo.out 2>&1 Summary In this article, we learned how to configure and set up HBase. We set up HBase to store data in Hadoop Distributed File System. We explored the working structure of RAID and JBOD and the differences between both filesystems. Resources for Article: Further resources on this subject: Understanding the HBase Ecosystem[article] The HBase's Data Storage[article] HBase Administration, Performance Tuning[article]
Read more
  • 0
  • 0
  • 14107

article-image-deploying-html5-applications-gnome
Packt
28 May 2013
10 min read
Save for later

Deploying HTML5 Applications with GNOME

Packt
28 May 2013
10 min read
(For more resources related to this topic, see here.) Before we start Most of the discussions in this article require a moderate knowledge of HTML5, JSON, and common client-side JavaScript programming. One particular exercise uses JQuery and JQuery Mobile to show how a real HTML5 application will be implemented. Embedding WebKit What we need to learn first is how to embed a WebKit layout engine inside our GTK+ application. Embedding WebKit means we can use HTML and CSS as our user interface instead of GTK+ or Clutter. Time for action – embedding WebKit With WebKitGTK+, this is a very easy task to do; just follow these steps: Create an empty Vala project without GtkBuilder and no license. Name it hello-webkit. Modify configure.ac to include WebKitGTK+ into the project. Find the following line of code in the file: PKG_CHECK_MODULES(HELLO_WEBKIT, [gtk+-3.0]) Remove the previous line and replace it with the following one: PKG_CHECK_MODULES(HELLO_WEBKIT, [gtk+-3.0 webkitgtk-3.0]) Modify Makefile.am inside the src folder to include WebKitGTK into the Vala compilation pipeline. Find the following lines of code in the file: hello_webkit_VALAFLAGS = --pkg gtk+-3.0 Remove it and replace it completely with the following lines: hello_webkit_VALAFLAGS = --vapidir . --pkg gtk+-3.0 --pkg webkit-1.0 --pkglibsoup-2.4 Fill the hello_webkit.vala file inside the src folder with the following lines: using GLib;using Gtk;using WebKit;public class Main : WebView{public Main (){load_html_string("<h1>Hello</h1>","/");}static int main (string[] args){Gtk.init (ref args);var webView = new Main ();var window = new Gtk.Window();window.add(webView);window.show_all ();Gtk.main ();return 0;}} Copy the accompanying webkit-1.0.vapi file into the src folder. We need to do this, unfortunately, because the webkit-1.0.vapi file distributed with many distributions is still using GTK+ Version 2. Run it, you will see a window with the message Hello, as shown in the following screenshot: What just happened? What we need to do first is to include WebKit into our namespace, so we can use all the functions and classes from it. using WebKit; Our class is derived from the WebView widget. It is an important widget in WebKit, which is capable of showing a web page. Showing it means not only parsing and displaying the DOM properly, but that it's capable to run the scripts and handle the styles referred to by the document. The derivation declaration is put in the class declaration as shown next: public class Main : WebView In our constructor, we only load a string and parse it as an HTML document. The string is Hello, styled with level 1 heading. After the execution of the following line, WebKit will parse and display the presentation of the HTML5 code inside its body: public Main (){load_html_string("<h1>Hello</h1>","/");} In our main function, what we need to do is create a window to put our WebView widget into. After adding the widget, we need to call the show_all() function in order to display both the window and the widget. static int main (string[] args){Gtk.init (ref args);var webView = new Main ();var window = new Gtk.Window();window.add(webView); The window content now only has a WebView widget as its sole displaying widget. At this point, we no longer use GTK+ to show our UI, but it is all written in HTML5. Runtime with JavaScriptCore An HTML5 application is, most of the time, accompanied by client-side scripts that are written in JavaScript and a set of styling definition written in CSS3. WebKit already provides the feature of running client-side JavaScript (running the script inside the web page) with a component called JavaScriptCore, so we don't need to worry about it. But how about the connection with the GNOME platform? How to make the client-side script access the GNOME objects? One approach is that we can expose our objects, which are written in Vala so that they can be used by the client-side JavaScript. This is where we will utilize JavaScriptCore. We can think of this as a frontend and backend architecture pattern. All of the code of business process which touch GNOME will reside in the backend. They are all written in Vala and run by the main process. On the opposite side, the frontend, the code is written in JavaScript and HTML5, and is run by WebKit internally. The frontend is what the user sees while the backend is what is going on behind the scene. Consider the following diagram of our application. The backend part is grouped inside a grey bordered box and run in the main process. The frontend is outside the box and run and displayed by WebKit. From the diagram, we can see that the frontend creates an object and calls a function in the created object. The object we create is not defined in the client side, but is actually created at the backend. We ask JavaScriptCore to act as a bridge to connect the object created at the backend to be made accessible by the frontend code. To do this, we wrap the backend objects with JavaScriptCore class and function definitions. For each object we want to make available to frontend, we need to create a mapping in the JavaScriptCore side. In the following diagram, we first map the MyClass object, then the helloFromVala function, then the intFromVala, and so on: Time for action – calling the Vala object from the frontend Now let's try and create a simple client-side JavaScript code and call an object defined at the backend: Create an empty Vala project, without GtkBuilder and no license. Name it hello-jscore. Modify configure.ac to include WebKitGTK+ exactly like our previous experiment. Modify Makefile.am inside the src folder to include WebKitGTK+ and JSCore into the Vala compilation pipeline. Find the following lines of code in the file: hello_jscore_VALAFLAGS = --pkg gtk+-3.0 Remove it and replace it completely with the following lines: hello_jscore_VALAFLAGS = --vapidir . --pkg gtk+-3.0 --pkg webkit-1.0 --pkglibsoup-2.4 --pkg javascriptcore Fill the hello_jscore.vala file inside the src folder with the following lines of code: using GLib;using Gtk;using WebKit;using JSCore;public class Main : WebView{public Main (){load_html_string("<h1>Hello</h1>" +"<script>alert(HelloJSCore.hello())</script>","/");window_object_cleared.connect ((frame, context) => {setup_js_class ((JSCore.GlobalContext) context);});}public static JSCore.Value helloFromVala (Context ctx,JSCore.Object function,JSCore.Object thisObject,JSCore.Value[] arguments,out JSCore.Value exception) {exception = null;var text = new String.with_utf8_c_string ("Hello fromJSCore");return new JSCore.Value.string (ctx, text);}static const JSCore.StaticFunction[] js_funcs = {{ "hello", helloFromVala, PropertyAttribute.ReadOnly },{ null, null, 0 }};static const ClassDefinition js_class = {0, // versionClassAttribute.None, // attribute"HelloJSCore", // classNamenull, // parentClassnull, // static valuesjs_funcs, // static functionsnull, // initializenull, // finalizenull, // hasPropertynull, // getPropertynull, // setPropertynull, // deletePropertynull, // getPropertyNamesnull, // callAsFunctionnull, // callAsConstructornull, // hasInstancenull // convertToType};void setup_js_class (GlobalContext context) {var theClass = new Class (js_class);var theObject = new JSCore.Object (context, theClass,context);var theGlobal = context.get_global_object ();var id = new String.with_utf8_c_string ("HelloJSCore");theGlobal.set_property (context, id, theObject,PropertyAttribute.None, null);}static int main (string[] args){Gtk.init (ref args);var webView = new Main ();var window = new Gtk.Window();window.add(webView);window.show_all ();Gtk.main ();return 0;}} Copy the accompanied webkit-1.0.vapi and javascriptcore.vapi files into the src folder. The javascriptcore.vapi file is needed because some distributions do not have this .vapi file in their repositories. Run the application. The following output will be displayed: What just happened? The first thing we do is include the WebKit and JavaScriptCore namespaces. Note, in the following code snippet, that the JavaScriptCore namespace is abbreviated as JSCore: using WebKit;using JSCore; In the Main function, we load HTML content into the WebView widget. We display a level 1 heading and then call the alert function. The alert function displays a string returned by the hello function inside the HelloJSCore class, as shown in the following code: public Main (){load_html_string("<h1>Hello</h1>" +"<script>alert(HelloJSCore.hello())</script>","/"); In the preceding code snippet, we can see that the client-side JavaScript code is as follows: alert(HelloJSCore.hello()) And we can also see that we call the hello function from the HelloJSCore class as a static function. It means that we don't instantiate the HelloJSCore object before calling the hello function. In WebView, we initialize the class defined in the Vala class when we get the window_object_cleared signal. This signal is emitted whenever a page is cleared. The initialization is done in setup_js_class and this is also where we pass the JSCore global context into. The global context is where JSCore keeps the global variables and functions. It is accessible by every code. window_object_cleared.connect ((frame, context) => {setup_js_class ((JSCore.GlobalContext)context);}); The following snippet of code contains the function, which we want to expose to the clientside JavaScript. The function just returns a Hello from JSCore string message: public static JSCore.Value helloFromVala (Context ctx,JSCore.Object function,JSCore.Object thisObject,JSCore.Value[] arguments,out JSCore.Value exception) {exception = null;var text = new String.with_utf8_c_string ("Hello from JSCore");return new JSCore.Value.string (ctx, text);} Then we need to put a boilerplate code that is needed to expose the function and other members of the class. The first part of the code is the static function index. This is the mapping between the exposed function and the name of the function defined in the wrapper. In the following example, we map the hello function, which can be used in the client side, with the helloFromVala function defined in the code. The index is then ended with null to mark the end of the array: static const JSCore.StaticFunction[] js_funcs = {{ "hello", helloFromVala, PropertyAttribute.ReadOnly },{ null, null, 0 }}; The next part of the code is the class definition. It is about the structure that we have to fill, so that JSCore would know about the class. All of the fields are filled with null, except for those we want to make use of. In this example, we use the static function for the hello function. So we fill the static function field with js_funcs, which we defined in the preceding code snippet: static const ClassDefinition js_class = {0, // versionClassAttribute.None, // attribute"HelloJSCore", // classNamenull, // parentClassnull, // static valuesjs_funcs, // static functionsnull, // initializenull, // finalizenull, // hasPropertynull, // getPropertynull, // setPropertynull, // deletePropertynull, // getPropertyNamesnull, // callAsFunctionnull, // callAsConstructornull, // hasInstancenull // convertToType}; After that, in the the setup_js_class function, we set up the class to be made available in the JSCore global context. First, we create JSCore.Class with the class definition structure we filled previously. Then, we create an object of the class, which is created in the global context. Last but not least, we assign the object with a string identifier, which is HelloJSCore. After executing the following code, we will be able to refer HelloJSCore on the client side: void setup_js_class (GlobalContext context) {var theClass = new Class (js_class);var theObject = new JSCore.Object (context, theClass,context);var theGlobal = context.get_global_object ();var id = new String.with_utf8_c_string ("HelloJSCore");theGlobal.set_property (context, id, theObject,PropertyAttribute.None, null);}
Read more
  • 0
  • 0
  • 14101

article-image-fat-2018-conference-session-5-summary-fat-recommenders-etc
Savia Lobo
24 Feb 2018
6 min read
Save for later

FAT* 2018 Conference Session 5 Summary on FAT Recommenders, Etc.

Savia Lobo
24 Feb 2018
6 min read
This session of FAT 2018 is about Recommenders, etc. Recommender systems are algorithmic tools for identifying items of interest to users. They are usually deployed to help mitigate information overload. Internet-scale item spaces offer many more choices than humans can process, diminishing the quality of their decision-making abilities. Recommender systems alleviate this problem by allowing users to more quickly focus on items likely to match their particular tastes. They are deployed across the modern Internet, suggesting products in e-commerce sites, movies and music in streaming media platforms, new connections on social networks, and many more types of items. This session explains what Fairness, Accountability, and Transparency means in the context of recommendation. The session also includes a paper that talks about predictive policing, which is defined as ‘Given historical crime incident data for a collection of regions, decide how to allocate patrol officers to areas to detect crime.’ The Conference on Fairness, Accountability, and Transparency (FAT), which would be held on the 23rd and 24th of February, 2018 is a multi-disciplinary conference that brings together researchers and practitioners interested in fairness, accountability, and transparency in socio-technical systems. The FAT 2018 conference will witness 17 research papers, 6 tutorials, and 2 keynote presentations from leading experts in the field. This article covers research papers pertaining to the 5th session that is dedicated to FAT Recommenders, etc. Paper 1: Runaway Feedback Loops in Predictive Policing Predictive policing systems are increasingly being used to determine how to allocate police across a city in order to best prevent crime. To update the model, discovered crime data (e.g., arrest counts) are used. Such systems have been empirically shown to be susceptible to runaway feedback loops, where police are repeatedly sent back to the same neighborhoods regardless of the true crime rate. This paper is in response to this system, where the authors have developed a mathematical model of predictive policing that proves why this feedback loop occurs.The paper also empirically shows how this model exhibits such problems, and demonstrates ways to change the inputs to a predictive policing system (in a black-box manner) so the runaway feedback loop does not occur, allowing the true crime rate to be learned. Key takeaways: The results stated in the paper establish a link between the degree to which runaway feedback causes problems and the disparity in crime rates between areas. The paper also demonstrates ways in which reported incidents of crime (reported by residents) and discovered incidents of crime (directly observed by police officers dispatched as a result of the predictive policing algorithm) interact. In this paper, the authors have used the theory of urns (a common framework in reinforcement learning) to analyze existing methods for predictive policing. There are formal as well as empirical results which shows why these methods will not work. Subsequently, the authors have also provided remedies that can be used directly with these methods in a black-box fashion that improve their behavior, and provide theoretical justification for these remedies. Paper 2: All The Cool Kids, How Do They Fit In? Popularity and Demographic Biases in Recommender Evaluation and Effectiveness There have been many advances in the information retrieval evaluation, which demonstrate the importance of considering the distribution of effectiveness across diverse groups of varying sizes. This paper addresses this question, ‘do users of different ages or genders obtain similar utility from the system, particularly if their group is a relatively small subset of the user base?’ The authors have applied this consideration to recommender systems, using offline evaluation and a utility-based metric of recommendation effectiveness to explore whether different user demographic groups experience similar recommendation accuracy. The paper shows that there are demographic differences in measured recommender effectiveness across two data sets containing different types of feedback in different domains; these differences sometimes, but not always, correlate with the size of the user group in question. Demographic effects also have a complex— and likely detrimental—interaction with popularity bias, a known deficiency of recommender evaluation. Key takeaways: The paper presents an empirical analysis of the effectiveness of collaborative filtering recommendation strategies, stratified by the gender and age of the users in the data set. The authors applied widely-used recommendation techniques across two domains, musical artists and movies, using publicly-available data. The paper explains whether recommender systems produced equal utility for users of different demographic groups. The authors made use of publicly available datasets, they compared the utility, as measured with nDCG, for users grouped by age and gender. Regardless of the recommender strategy considered, they found significant differences for the nDCG among demographic groups. Paper 3: Recommendation Independence In this paper the authors have showcased new methods that can deal with variance of recommendation outcomes without increasing the computational complexity. These methods can more strictly remove the sensitive information, and experimental results demonstrate that the new algorithms can more effectively eliminate the factors that undermine fairness. Additionally, the paper also explores potential applications for independence enhanced recommendation, and discuss its relation to other concepts, such as recommendation diversity. Key takeaways from the paper: The authors have developed new independence-enhanced recommendation models that can deal with the second moment of distributions without sacrificing computational efficiency. The paper also explores applications in which recommendation independence would be useful, and reveal the relation of independence to the other concepts in recommendation research. It also presents the concept of recommendation independence, and discuss how the concept would be useful for solving real-world problems. Paper 4: Balanced Neighborhoods for Multi-sided Fairness in Recommendation In this paper, the authors examine two different cases of fairness-aware recommender systems: consumer-centered and provider-centered. The paper explores the concept of a balanced neighborhood as a mechanism to preserve personalization in recommendation while enhancing the fairness of recommendation outcomes. It shows that a modified version of the Sparse Linear Method (SLIM) can be used to improve the balance of user and item neighborhoods, with the result of achieving greater outcome fairness in real-world datasets with minimal loss in ranking performance. Key takeaways: In this paper, the authors examine applications in which fairness with respect to consumers and to item providers is important. They have shown that variants of the well-known sparse linear method (SLIM) can be used to negotiate the tradeoff between fairness and accuracy. This paper also introduces the concept of multisided fairness, relevant in multisided platforms that serve a matchmaking function. It demonstrates that the concept of balanced neighborhoods in conjunction with the well-known sparse linear method can be used to balance personalization with fairness considerations. If you’ve missed our summaries on the previous sessions, visit the article links to be on track. Session 1: Online Discrimination and Privacy Session 2: Interpretability and Explainability Session 3: Fairness in Computer Vision and NLP Session 4: Fair Classification
Read more
  • 0
  • 0
  • 14068
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at €18.99/month. Cancel anytime
article-image-configuring-mysql-linked-server-sql-server-2008
Packt Editorial Staff
16 Aug 2009
7 min read
Save for later

Configuring a MySQL linked server on SQL Server 2008

Packt Editorial Staff
16 Aug 2009
7 min read
Linking servers provides an elegant solution when you are faced with running queries against databases on distributed servers or looking at your distributed assets on disparate databases. This article by Dr. Jay Krishnaswamy explains how to set up a MySQL linked server on SQL Server 2008 Enterprise. Configuring a linked MySQL server as well as querying a table on the MySQL linked server is described. The reader would benefit reviewing the first article on this series on MySQL Servers. Introduction MS SQL servers always provided remote access to servers through RPC mechanisms, but they had the limitation of being confined to invoking procedures on remote locations. A linked server (a virtual server) may be considered a more flexible way of achieving the same thing, with the added benefits of remote table access and distributed queries. Microsoft manages the link mechanism via OLE DB technology. Specifically, an OLE DB data source points to the specific database that can be accessed using OLEDB. In this article, we will be creating a MySQL linked server on SQL Server 2008 and querying a database [TestMove] table shown in the next listing. In reviewing the previous article it may be noticed that the Employees tables were moved to MySQL database TestMove. In running the commands from the mysql> prompt it is assumed that the MySQL Server has been started. Listing 1: employees table in TestMove mysql> show tables; +--------------------+ | Tables_in_testmove | +--------------------+ | employees | +--------------------+ 1 row in set (0.09 sec) mysql> Creating an ODBC DSN for MySQL In the previous article on MySQL Servers cited earlier, a DSN was created for moving data. Essentially the same DSN can be used. Herein follows a brief review of the DSN MySQL_Link created along the same lines as in the previously referenced article. The ODBC driver used for creating this ODBC DSN is the one installed on the machine when the MySQL Server was installed as shown. The final interactive window where you may test the connectivity is shown in the next figure. You may notice that the database Testmove has been named in the ODBC DSN. The name MySQL_LINK is the ODBC DSN. When you close the window after clicking the OK button, an ODBC DSN item will be added to the System DSN tab of the ODBC wizard as shown. Steps to create a MySQL linked server from Management Studio Right click the Linked Servers node to display a drop-down menu as shown in the next figure. Click on New Linked Server...item. This brings up the New Linked Server window as shown. The window is all empty except for a default Provider. The very first thing to do is to provide a name for this linked server. Herein it is LINKED_ MYSQL. We will be using one of the providers [Microsoft OLE DB Provider for ODBC] supported by SQL Server 2008. You may access the list of OLE DB Providers in the Providers sub-folder of the Linked Servers. Provide the other details as shown in the next figure. Make sure they are entered exactly as shown or according to how you have created the database on MySQL Server. Click on the Security list item under General to the left. This opens the 'Security' page of the New Linked Server wizard as shown. Change the login from the default "Be made without using a security context" to "Be made using this security context". Enter remote login. In this case, it is "root" for the remote login and the password is the one used during the ODBC DSN (also the password for server authentication) creation. Make no changes to the Server Options page. Click OK. This creates a linked server Linked_MySQL as shown expanded in the Linked Server's node as shown. You may need to right-click and refresh the Linked Servers' node to see the new linked server. As you can see in the figure, the 'User' tables are not displayed.     Running Queries and reviewing results Running system stored procedures can provide various levels of information and the database can be queried using the four-part syntax and the openquery() method. Information on the linked server It is easy to find how the linked server is configured using system stored procedure sp_linkedsrvlogin on the SQL Server 2008. Open a Query window from File | New | Query Current Connection to open the query window and type in the following statement. The next figure shows the statement as well as the returned values. SQL Server 2008 querying has the intellisense report and this must be put to good use. Exec sp_linkedsrvlogin This shows all servers both local and remote as shown in the next figure. Information about the tables on the remote server can also be accessed by running a stored procedure. Executing the stored procedure sp_tables_ex as shown in the next figure (which displays the statement and the result of executing the stored procedure) can be used to obtain table information. Querying the table on the database Data in the table on the linked server can be queried using the openquery() function. The syntax for this function shown next is very simple. openquery ('linked server', 'query') The next figure shows the result of running the openquery() function on the Linked_MySQL linked server. Although it should be possible to query the linked server using the four-part syntax as in: Select * from LINKED_MYSQL...employees The above statement returns an error. This is probably a limitation of a combination of MSDASQL and the ODBC driver which does not provide the schema information correctly(this is just the author's opinion). Are Remote Procedure Calls (RPC) allowed? The easiest way to test this is to send out a call by running the following query against the linked server. Execute('Select FirstName, LastName from employees') at Linked_MYSQL If the linked server is not configured for RPC, then the result you get by running the above query is as shown in the next figure. Turn on RPC Earlier on we skipped the Server Options page of the linked server. Back in the Management Studio right click linked server LINKED_MYSQL and from the drop-down choose to look at the properties at the bottom of the list. This brings up the LINKED_MYSQL properties window as shown. Click on Server Options. In the Server Options page change the values of RPC and RPCOUT to true, default for both being false. Now run the query that produced the error previously. The result is displayed in the next figure. You might have noted that only two columns were returned from the employees table. This was deliberate as trying to get all the column would produce an error due partly to the data types of data stored in the table and their compatibility with MSDASQL and the ODBC driver (Again, an author's opinion). Creating Linked Server using TSQL While the linked server can be created using the built-in wizard of the Management Studio, it can also be created using TSQL statements as in the following listing (run both statements, the first one creates the linked server and the second the logins). Listing 2:  Exec master.dbo.sp_addlinkedserver @server=N'MySQlbyCode', @srvprodcut=N'MySQL', @provider=N'MSDASQL', @datasrc=N'MySQL_link' Exec master.dbo.sp_addlinkedserverlogin @server=N'MySQlbyCode', @locallogin=NULL, @rmtuser=N'root', @rmtpassword=N'<your password>' @rmtsrvname=N'localhost' Summary The article described the steps involved in configuring a MySql Linked server on SQL Server 2008 using the built-in New Linked Server wizard as well as TSQL. Method to query the linked server as well as enabling RPC were described. If you have read this article you may be interested in the following: MySQL Data Transfer using SQL Server Integration Services (SSIS) Transferring Data from MS Access 2003 to SQL Server 2008 Exporting data from MS Access 2003 to MySQL
Read more
  • 0
  • 0
  • 14067

article-image-co-pilot-microsoft-fabric-for-power-bi
Sagar Lad
23 Aug 2023
8 min read
Save for later

Co-Pilot & Microsoft Fabric for Power BI

Sagar Lad
23 Aug 2023
8 min read
IntroductionMicrosoft's data platform solution for the modern era is called Fabric. Microsoft's three primary data analytics tools:  Power BI, Azure Data Factory, and Azure Synapse all covered under Fabric. Advanced artificial intelligence capabilities built on machine learning and natural language processing (NLP) are made available to Power BI customers through Copilot. In this article, we will deep dive into how co-pilot and Microsoft Fabric will transform the way we develop and work with Power BI.Co-Pilot and Fabric with Power BIThe urgent requirement for businesses to turn their data into value is something that both Microsoft Fabric and Copilot aspire to address. Big Data continues to fall short of its initial promises even after years have passed. Every year, businesses generate more data, yet a recent IBM study found that 90% of this data is never successfully exploited for any kind of strategic purpose. So, more data does not mean more value or business insight. Data fragmentation and poor data quality are the key obstacles to releasing the value of data. These problems are what Microsoft hopes to address with Microsoft Fabric, a human-centric, end-to-end analytics product that brings together all of an organization's data and analytics in one place. Copilot has now been integrated into Power BI. Large multi-modal artificial intelligence models based on natural language processing have gained attention since the publication of ChatGPT. Beyond casuistry, Microsoft Fabric and Copilot share a trait in that they each want to transform the Power BI user interface.●       Microsoft Fabric and Power BIMicrosoft Fabric is just Synapse and Power BI together. By combining the benefits of the Power BI SaaS platform with the various Synapse workload types, Microsoft Fabric creates an environment that is more cohesive, integrated, and easier to use for all of the associated profiles. However, Power BI Premium users will get access to new opportunities for data science, data engineering, etc. Power BI will continue to function as it does right now. Data analysts and Power BI developers are not required to begin using Synapse Data Warehouse if they do not want to. Microsoft wants to combine all of its data offerings into one, called Fabric, just like it did with Office 365:Image 1: Microsoft Fabric (Source: Microsoft)Let’s understand in detail how Microsoft Fabric will make life easier for Power BI developers.1.     Data IngestionThere are various methods by which we can connect to data sources in Fabric in order to consume data. For example, utilising Spark notebooks or pipelines, for instance. This may be unknown to the Power BI realm, though.                                                       Image 2: Data Transformation in Power BI Instead, we can ingest the data using dataflows gen2, which will save it on OneLake in the proper format.2.     Ad Hoc Query One or more dataflows successfully published and refreshed will show in the workspace along with a number of other artifacts. The SQL Endpoint artifact is one of them. We can begin creating on-demand SQL queries and saving them as views after you open them. As an alternative, we can also create visual queries which will enable us to familiarise ourselves with the data flow diagram view. Above all, however, is the fact that this interface shares many characteristics with Power BI Data Marts, making it a familiar environment for those familiar with Power BI:   Image 3: Power BI - One Data Lake Hub3.     Data ModellingWith the introduction of web modelling for Power BI, we can introduce new metrics and start establishing linkages between different tables right away in this interface. The default workspace where the default dataset is kept will automatically contain the data model. The new storage option Direct Lake is advantageous for the datasets created in this manner via the cloud interface. By having just one copy of data in OneLake, this storage style prevents data duplication and unnecessary data refreshes.●       Co-Pilot and Power BI Copilot, a new artificial intelligence framework for Power BI is an offering from Microsoft. CoPilot is Power BI's expensive multimodal artificial intelligence model that is built on natural language processing. It might be compared to the ChatGPT of Power BI. Users will be able to ask inquiries about data, generate graphics, and DAX measures by providing a brief description of what they need thanks to the addition of Copilot to Power BI. For instance, it demonstrates how a brief statement of the user's preferences for the report:"Add a table of the top 500 MNC IT Companies by total sales to my model”.The DAX code required to generate measures and tables is generated automatically by the algorithm.Copilot enables:●       Power BI reports can be created and customized to provide insights.●       Create and improve DAX computations.●       Inquire about your data.●       Publish narrative summaries.●       Ease of Use●       Faster Time to Market Key Features of the Power BI Copilot are as follows: ●       Automated report generationCopilot can create well-designed dashboards, data narratives, and interactive components automatically, saving time and effort compared to manually creating reports.●       Conversational language interfaceWe can use everyday language to express data requests and inquiries, making it simpler to connect with your data and gain insights. ●        Real-time analyticsCopilot's real-time analytics capabilities can be used by Power BI customers to view data and react swiftly to shifts and trends. Let’s look at the step-by-step process on how to use Copilot for Power BI:Step 1: Open Power BI and go to the Copilot tab screen,Step 2:  Type a query pertaining to the data for example to produce a financial report or pick from a list of suggestions that Copilot has automatically prepared for you.Step 3: Copilot sorts through and analyses data to provide the information.Step 4: Copilot compiles a visually stunning report, successfully converting complex data into easily comprehended, practical information.Step 5: Investigate data even more by posing queries, writing summaries to present to stakeholders, and more. There are also a few limitations to using the Copilot features with Power BI: ●       Reliability for the recommendationsAll programming languages that are available in public sources have been taught to Copilot, ensuring the quality of its proposals. The quantity of the training dataset that is accessible for that language, however, may have an impact on the quality of the suggestions. APL, Erlang, and other specialized programming languages' suggestions won't be as useful as those for more widely used ones like Python, Java, etc.●       Privacy and security issuesThere are worries that the model, which was trained on publicly accessible code, can unintentionally recommend code fragments that have security flaws or were intended to be private.●       Dependence on comments and namingThe user is responsible for accuracy because the AI can provide suggestions that are more accurate when given specific comments and descriptive variable names.●       Lack of original solutions and inability to creatively solve problems. Unlike a human developer, the tool is unable to do either. It can only make code suggestions based on the training data.●       Inefficient codebaseThe tool is not designed for going through and comprehending big codebases. It works best when recommending code for straightforward tasks.ConclusionThe combination of Microsoft Copilot and Fabric with Power BI has the ability to completely alter the data modelling field. It blends sophisticated generative AI with data to speed up the discovery and sharing of insights by everyone. By enabling both data engineers and non-technical people to examine data using AI models, it is transforming Power BI into a human-centered analytics platform. Author Bio: Sagar Lad is a Cloud Data Solution Architect with a leading organization and has deep expertise in designing and building Enterprise-grade Intelligent Azure Data and Analytics Solutions. He is a published author, content writer, Microsoft Certified Trainer, and C# Corner MVP.Medium , Amazon , LinkedIn   
Read more
  • 0
  • 0
  • 14058

article-image-start-treating-your-infrastructure-code
Packt
26 Dec 2016
18 min read
Save for later

Start Treating your Infrastructure as Code

Packt
26 Dec 2016
18 min read
In this article by Veselin Kantsev, the author of the book Implementing DevOps on AWS. Ladies and gentlemen, put your hands in the atmosphere for Programmable Infrastructure is here! Perhaps Infrastructure-as-Code(IaC) is not an entirely new concept considering how long Configuration Management has been around. Codifying server, storage and networking infrastructure and their relationships however is a relatively recent tendency brought about by the rise of cloud computing. But let us leave Config Management for later and focus our attention on that second aspect of IaC. You would recall from the previous chapter some of the benefits of storing all-the-things as code: Code can be kept under version control Code can be shared/collaborated on easily Code doubles as documentation Code is reproducible (For more resources related to this topic, see here.) That last point was a big win for me personally. Automated provisioning helped reduce the time it took to deploy a full-featured cloud environment from four hours down to one and the occurrences of human error to almost zero (one shall not be trusted with an input field). Being able to rapidly provision resources becomes a significant advantage when a team starts using multiple environments in parallel and needs those brought up or down on-demand. In this article we examine in detail how to describe (in code) and deploy one such environment on AWS with minimal manual interaction. For implementing IaC in the Cloud, we will look at two tools or services: Terraform and CloudFormation. We will go through examples on how to: Configure the tool Write an IaC template Deploy a template Deploy subsequent changes to the template Delete a template and remove the provisioned infrastructure For the purpose of these examples, let us assume our application requires a Virtual Private Cloude(VPC) which hosts a Relational Database Services(RDS) back-end and a couple of Elastic Compute Cloud (EC2) instances behind an Elastic Load Balancing (ELB). We will keep most components behind Network Address Translation (NAT), allowing only the load-balancer to be accessed externally. IaC using Terraform One of the tools that can help deploy infrastructure on AWS is HashiCorp's Terraform (https://www.terraform.io). HashiCorp is that genius bunch which gave us Vagrant, Packer and Consul. I would recommend you look up their website if you have not already. Using Terraform (TF), we will be able to write a template describing an environment, perform a dry run to see what is about to happen and whether it is expected, deploy the template and make any late adjustments where necessary—all of this without leaving the shell prompt. Configuration Firstly, you will need to have a copy of TF (https://www.terraform.io/downloads.html) on your machine and available on the CLI. You should be able to query the currently installed version, which in my case is 0.6.15: $ terraform --version Terraform v0.6.15 Since TF makes use of the AWS APIs, it requires a set of authentication keys and some level of access to your AWS account. In order to deploy the examples in this article you could create a new Identity and Access Management (IAM) user with the following permissions: "autoscaling:CreateAutoScalingGroup", "autoscaling:CreateLaunchConfiguration", "autoscaling:DeleteLaunchConfiguration", "autoscaling:Describe*", "autoscaling:UpdateAutoScalingGroup", "ec2:AllocateAddress", "ec2:AssociateAddress", "ec2:AssociateRouteTable", "ec2:AttachInternetGateway", "ec2:AuthorizeSecurityGroupEgress", "ec2:AuthorizeSecurityGroupIngress", "ec2:CreateInternetGateway", "ec2:CreateNatGateway", "ec2:CreateRoute", "ec2:CreateRouteTable", "ec2:CreateSecurityGroup", "ec2:CreateSubnet", "ec2:CreateTags", "ec2:CreateVpc", "ec2:Describe*", "ec2:ModifySubnetAttribute", "ec2:RevokeSecurityGroupEgress", "elasticloadbalancing:AddTags", "elasticloadbalancing:ApplySecurityGroupsToLoadBalancer", "elasticloadbalancing:AttachLoadBalancerToSubnets", "elasticloadbalancing:CreateLoadBalancer", "elasticloadbalancing:CreateLoadBalancerListeners", "elasticloadbalancing:Describe*", "elasticloadbalancing:ModifyLoadBalancerAttributes", "rds:CreateDBInstance", "rds:CreateDBSubnetGroup", "rds:Describe*" Please refer:${GIT_URL}/Examples/Chapter-2/Terraform/iam_user_policy.json One way to make the credentials of the IAM user available to TF is by exporting the following environment variables: $ export AWS_ACCESS_KEY_ID='user_access_key' $ export AWS_SECRET_ACCESS_KEY='user_secret_access_key' This should be sufficient to get us started. Template design Before we get to coding, here are some of the rules: You could choose to write a TF template as a single large file or a combination of smaller ones. Templates can be written in pure JSON or TF's own format. Terraform will look for files with extensions .tfor .tf.json in a given folder and load these in alphabetical order. TF templates are declarative, hence the order in which resources appear in them does not affect the flow of execution. A Terraform template generally consists of three sections: resources, variables and outputs. As mentioned in the preceding section, it is a matter of personal preference how you arrange these, however, for better readability I suggest we make use of the TF format and write each section to a separate file. Also, while the file extensions are of importance, the file names are up to you. Resources In a way, this file holds the main part of a template, as the resources represent the actual components that end up being provisioned. For example, we will be using a VPC resource, RDS, an ELB one and a few others. Since template elements can be written in any order, Terraform determines the flow of execution by examining any references that it finds (for example a VPC should exist before an ELB which is said to belong to it is created). Alternatively, explicit flow control attributes such as the depends_on are used, as we will observe shortly. To find out more, let us go through the contents of the resources.tf file. Please refer to: ${GIT_URL}/Examples/Chapter-2/Terraform/resources.tf First we tell Terraform what provider to use for our infrastructure: # Set a Provider provider "aws" { region = "${var.aws-region}" } You will notice that no credentials are specified, since we set those as environment variables earlier. Now we can add the VPC and its networking components: # Create a VPC resource "aws_vpc""terraform-vpc" { cidr_block = "${var.vpc-cidr}" tags { Name = "${var.vpc-name}" } } # Create an Internet Gateway resource "aws_internet_gateway""terraform-igw" { vpc_id = "${aws_vpc.terraform-vpc.id}" } # Create NAT resource "aws_eip""nat-eip" { vpc = true } So far we have declared the VPC, its Internet and NAT gateways plus a set of public and private subnets with matching routing tables. It will help clarify the syntax if we examined some of those resource blocks, line by line: resource "aws_subnet""public-1" { The first argument is the type of the resource followed by an arbitrary name. vpc_id = "${aws_vpc.terraform-vpc.id}" The aws_subnet resource named public-1 has a property vpc_id which refers to the id attribute of a different resource of type aws_vpc named terraform-vpc. Such references to other resources implicitly define the execution flow, that is to say the VPC needs to exist before the subnet can be created. cidr_block = "${cidrsubnet(var.vpc-cidr, 8, 1)}" We will talk more about variables in a moment, but the format is var.var_name. Here we use the cidrsubnet function with the vpc-cidr variable which returns a cidr_block to be assigned to the public-1 subnet. Please refer to the Terraform documentation for this and other useful functions. Next we add a RDS to the VPC: resource "aws_db_instance""terraform" { identifier = "${var.rds-identifier}" allocated_storage = "${var.rds-storage-size}" storage_type= "${var.rds-storage-type}" engine = "${var.rds-engine}" engine_version = "${var.rds-engine-version}" instance_class = "${var.rds-instance-class}" username = "${var.rds-username}" password = "${var.rds-password}" port = "${var.rds-port}" vpc_security_group_ids = ["${aws_security_group.terraform-rds.id}"] db_subnet_group_name = "${aws_db_subnet_group.rds.id}" } Here we see mostly references to variables with a few calls to other resources. Following the RDS is an ELB: resource "aws_elb""terraform-elb" { name = "terraform-elb" security_groups = ["${aws_security_group.terraform-elb.id}"] subnets = ["${aws_subnet.public-1.id}", "${aws_subnet.public-2.id}"] listener { instance_port = 80 instance_protocol = "http" lb_port = 80 lb_protocol = "http" } tags { Name = "terraform-elb" } } Lastly we define the EC2 auto scaling group and related resources: resource "aws_launch_configuration""terraform-lcfg" { image_id = "${var.autoscaling-group-image-id}" instance_type = "${var.autoscaling-group-instance-type}" key_name = "${var.autoscaling-group-key-name}" security_groups = ["${aws_security_group.terraform-ec2.id}"] user_data = "#!/bin/bash n set -euf -o pipefail n exec 1>>(logger -s -t $(basename $0)) 2>&1 n yum -y install nginx; chkconfig nginx on; service nginx start" lifecycle { create_before_destroy = true } } resource "aws_autoscaling_group""terraform-asg" { name = "terraform" launch_configuration = "${aws_launch_configuration.terraform-lcfg.id}" vpc_zone_identifier = ["${aws_subnet.private-1.id}", "${aws_subnet.private-2.id}"] min_size = "${var.autoscaling-group-minsize}" max_size = "${var.autoscaling-group-maxsize}" load_balancers = ["${aws_elb.terraform-elb.name}"] depends_on = ["aws_db_instance.terraform"] tag { key = "Name" value = "terraform" propagate_at_launch = true } } The user_data shell script above will install and start NGINX onto the EC2 node(s). Variables We have made great use of variables to define our resources, making the template as re-usable as possible. Let us now look inside variables.tf to study these further. Similarly to the resources list, we start with the VPC : Please refer to:${GIT_URL}/Examples/Chapter-2/Terraform/variables.tf variable "aws-region" { type = "string" description = "AWS region" } variable "aws-availability-zones" { type = "string" description = "AWS zones" } variable "vpc-cidr" { type = "string" description = "VPC CIDR" } variable "vpc-name" { type = "string" description = "VPC name" } The syntax is: variable "variable_name" { variable properties } Where variable_name is arbitrary, but needs to match relevant var.var_name references made in other parts of the template. For example, variable aws-region will satisfy the ${var.aws-region} reference we made earlier when describing the region of the provider aws resource. We will mostly use string variables, however there is another useful type called map which can hold lookup tables. Maps are queried in a similar way to looking up values in a hash/dict (Please see: https://www.terraform.io/docs/configuration/variables.html). Next comes RDS: variable "rds-identifier" { type = "string" description = "RDS instance identifier" } variable "rds-storage-size" { type = "string" description = "Storage size in GB" } variable "rds-storage-type" { type = "string" description = "Storage type" } variable "rds-engine" { type = "string" description = "RDS type" } variable "rds-engine-version" { type = "string" description = "RDS version" } variable "rds-instance-class" { type = "string" description = "RDS instance class" } variable "rds-username" { type = "string" description = "RDS username" } variable "rds-password" { type = "string" description = "RDS password" } variable "rds-port" { type = "string" description = "RDS port number" } Finally, EC2: variable "autoscaling-group-minsize" { type = "string" description = "Min size of the ASG" } variable "autoscaling-group-maxsize" { type = "string" description = "Max size of the ASG" } variable "autoscaling-group-image-id" { type="string" description = "EC2 AMI identifier" } variable "autoscaling-group-instance-type" { type = "string" description = "EC2 instance type" } variable "autoscaling-group-key-name" { type = "string" description = "EC2 ssh key name" } We now have the type and description of all our variables defined in variables.tf, however no values have been assigned to them yet. Terraform is quite flexible with how this can be done. We could: Assign default values directly in variables.tf variable "aws-region" { type = "string"description = "AWS region"default = 'us-east-1' } Not assign a value to a variable, in which case Terraform will prompt for it at run time Pass a -var 'key=value' argument(s) directly to the Terraform command, like so: -var 'aws-region=us-east-1' Store key=value pairs in a file Use environment variables prefixed with TF_VAR, as in TF_VAR_ aws-region Using a key=value pairs file proves to be quite convenient within teams, as each engineer can have a private copy (excluded from revision control). If the file is named terraform.tfvars it will be read automatically by Terraform, alternatively -var-file can be used on the command line to specify a different source. Below is the content of our sample terraform.tfvars file: Please refer to:${GIT_URL}/Examples/Chapter-2/Terraform/terraform.tfvars autoscaling-group-image-id = "ami-08111162" autoscaling-group-instance-type = "t2.nano" autoscaling-group-key-name = "terraform" autoscaling-group-maxsize = "1" autoscaling-group-minsize = "1" aws-availability-zones = "us-east-1b,us-east-1c" aws-region = "us-east-1" rds-engine = "postgres" rds-engine-version = "9.5.2" rds-identifier = "terraform-rds" rds-instance-class = "db.t2.micro" rds-port = "5432" rds-storage-size = "5" rds-storage-type = "gp2" rds-username = "dbroot" rds-password = "donotusethispassword" vpc-cidr = "10.0.0.0/16" vpc-name = "Terraform" A point of interest is aws-availability-zones, it holds multiple values which we interact with using the element and split functions as seen in resources.tf. Outputs The third, mostly informational part of our template contains the Terraform Outputs. These allow for selected values to be returned to the user when testing, deploying or after a template has been deployed. The concept is similar to how echo statements are commonly used in shell scripts to display useful information during execution. Let us add outputs to our template by creating an outputs.tf file: Please refer to:${GIT_URL}/Examples/Chapter-2/Terraform/outputs.tf output "VPC ID" { value = "${aws_vpc.terraform-vpc.id}" } output "NAT EIP" { value = "${aws_nat_gateway.terraform-nat.public_ip}" } output "ELB URI" { value = "${aws_elb.terraform-elb.dns_name}" } output "RDS Endpoint" { value = "${aws_db_instance.terraform.endpoint}" } To configure an output you simply reference a given resource and its attribute. As shown in preceding code, we have chosen the ID of the VPC, the Elastic IP address of the NAT gateway, the DNS name of the ELB and the Endpoint address of the RDS instance. The Outputs section completes the template in this example. You should now have four files in your template folder: resources.tf, variables.tf, terraform.tfvars and outputs.tf. Operations We shall examine five main Terraform operations: Validating a template Testing (dry-run) Initial deployment Updating a deployment Removal of a deployment In the following command line examples, terraform is run within the folder which contains the template files. Validation Before going any further, a basic syntax check should be done with the terraform validate command. After renaming one of the variables in resources.tf, validate returns an unknown variable error: $ terraform validate Error validating: 1 error(s) occurred: * provider config 'aws': unknown variable referenced: 'aws-region-1'. define it with 'variable' blocks Once the variable name has been corrected, re-running validate returns no output, meaning OK. Dry-run The next step is to perform a test/dry-run execution with terraform plan, which displays what would happen during an actual deployment. The command returns a colour coded list of resources and their properties or more precisely: $ terraform plan Resources are shown in alphabetical order for quick scanning. Green resources will be created (or destroyed and then created if an existing resource exists), yellow resources are being changed in-place, and red resources will be destroyed. To literally get the picture of what the to-be-deployed infrastructure looks like, you could use terraform graph: $ terraform graph > my_graph.dot DOT files can be manipulated with the Graphviz open source software (Please see : http://www.graphviz.org) or many online readers/converters. Below is a portion of a larger graph representing the template we designed earlier: Deployment If you are happy with the plan and graph, the template can now be deployed using terraform apply: $ terraform apply aws_eip.nat-eip: Creating... allocation_id: "" =>"<computed>" association_id: "" =>"<computed>" domain: "" =>"<computed>" instance: "" =>"<computed>" network_interface: "" =>"<computed>" private_ip: "" =>"<computed>" public_ip: "" =>"<computed>" vpc: "" =>"1" aws_vpc.terraform-vpc: Creating... cidr_block: "" =>"10.0.0.0/16" default_network_acl_id: "" =>"<computed>" default_security_group_id: "" =>"<computed>" dhcp_options_id: "" =>"<computed>" enable_classiclink: "" =>"<computed>" enable_dns_hostnames: "" =>"<computed>" Apply complete! Resources: 22 added, 0 changed, 0 destroyed. The state of your infrastructure has been saved to the following path. This state is required to modify and destroy your infrastructure, so keep it safe. To inspect the complete state use the terraform show command. State path: terraform.tfstate Outputs: ELB URI = terraform-elb-xxxxxx.us-east-1.elb.amazonaws.com NAT EIP = x.x.x.x RDS Endpoint = terraform-rds.xxxxxx.us-east-1.rds.amazonaws.com:5432 VPC ID = vpc-xxxxxx At the end of a successful deployment, you will notice the Outputs we configured earlier and a message about another important part of Terraform – the state file.(Please refer to: https://www.terraform.io/docs/state/): Terraform stores the state of your managed infrastructure from the last time Terraform was run. By default this state is stored in a local file named terraform.tfstate, but it can also be stored remotely, which works better in a team environment. Terraform uses this local state to create plans and make changes to your infrastructure. Prior to any operation, Terraform does a refresh to update the state with the real infrastructure. In a sense, the state file contains a snapshot of your infrastructure and is used to calculate any changes when a template has been modified. Normally you would keep the terraform.tfstate file under version control alongside your templates. In a team environment however, if you encounter too many merge conflicts you can switch to storing the state file(s) in an alternative location such as S3 (Please see: https://www.terraform.io/docs/state/remote/index.html). Allow a few minutes for the EC2 node to fully initialize then try loading the ELB URI from the preceding Outputs in your browser. You should be greeted by NGINX as shown in the following screenshot: Updates As per Murphy 's Law, as soon as we deploy a template, a change to it will become necessary. Fortunately, all that is needed for this is to update and re-deploy the given template. Let us say we need to add a new rule to the ELB security group (shown in bold below). Update the resource "aws_security_group""terraform-elb" block in resources.tf: resource "aws_security_group""terraform-elb" { name = "terraform-elb" description = "ELB security group" vpc_id = "${aws_vpc.terraform-vpc.id}" ingress { from_port = "80" to_port = "80" protocol = "tcp" cidr_blocks = ["0.0.0.0/0"] } ingress { from_port = "443" to_port = "443" protocol = "tcp" cidr_blocks = ["0.0.0.0/0"] } egress { from_port = 0 to_port = 0 protocol = "-1" cidr_blocks = ["0.0.0.0/0"] } } Verify what is about to change $ terraform plan ... ~ aws_security_group.terraform-elb ingress.#: "1" =>"2" ingress.2214680975.cidr_blocks.#: "1" =>"1" ingress.2214680975.cidr_blocks.0: "0.0.0.0/0" =>"0.0.0.0/0" ingress.2214680975.from_port: "80" =>"80" ingress.2214680975.protocol: "tcp" =>"tcp" ingress.2214680975.security_groups.#: "0" =>"0" ingress.2214680975.self: "0" =>"0" ingress.2214680975.to_port: "80" =>"80" ingress.2617001939.cidr_blocks.#: "0" =>"1" ingress.2617001939.cidr_blocks.0: "" =>"0.0.0.0/0" ingress.2617001939.from_port: "" =>"443" ingress.2617001939.protocol: "" =>"tcp" ingress.2617001939.security_groups.#: "0" =>"0" ingress.2617001939.self: "" =>"0" ingress.2617001939.to_port: "" =>"443" Plan: 0 to add, 1 to change, 0 to destroy. Deploy the change: $ terraform apply ... aws_security_group.terraform-elb: Modifying... ingress.#: "1" =>"2" ingress.2214680975.cidr_blocks.#: "1" =>"1" ingress.2214680975.cidr_blocks.0: "0.0.0.0/0" =>"0.0.0.0/0" ingress.2214680975.from_port: "80" =>"80" ingress.2214680975.protocol: "tcp" =>"tcp" ingress.2214680975.security_groups.#: "0" =>"0" ingress.2214680975.self: "0" =>"0" ingress.2214680975.to_port: "80" =>"80" ingress.2617001939.cidr_blocks.#: "0" =>"1" ingress.2617001939.cidr_blocks.0: "" =>"0.0.0.0/0" ingress.2617001939.from_port: "" =>"443" ingress.2617001939.protocol: "" =>"tcp" ingress.2617001939.security_groups.#: "0" =>"0" ingress.2617001939.self: "" =>"0" ingress.2617001939.to_port: "" =>"443" aws_security_group.terraform-elb: Modifications complete ... Apply complete! Resources: 0 added, 1 changed, 0 destroyed. Some update operations can be destructive (Please refer: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-update-behaviors.html.You should always check the CloudFormation documentation on the resource you are planning to modify to see whether a change is going to cause any interruption. Terraform provides some protection via the prevent_destroy lifecycle property (Please refer: https://www.terraform.io/docs/configuration/resources.html#prevent_destroy). Removal This is a friendly reminder to always remove AWS resources after you are done experimenting with them to avoid any unexpected charges. Before performing any delete operations, we will need to grant such privileges to the (terraform) IAM user we created in the beginning of this article. As a shortcut, you could temporarily attach the Aministrato rAccess managed policy to the user via the AWS Console as shown in the following figure: To remove the VPC and all associated resources that we created as part of this example, we will use terraform destroy: $ terraform destroy Do you really want to destroy? Terraform will delete all your managed infrastructure. There is no undo. Only 'yes' will be accepted to confirm. Enter a value: yes Terraform asks for a confirmation then proceeds to destroy resources, ending with: Apply complete! Resources: 0 added, 0 changed, 22 destroyed. Next, we remove the temporary admin access we granted to the IAM user by detaching the Aministrator Access managed policy as shown in the following screenshot: Then verify that the VPC is no longer visible in the AWS Console. Summary In this aticlewe looked at the importance and usefulness of Infrastructure as Code and ways to implement it using Terraform or AWS CloudFormation. We examined the structure and individual components of both a Terraform and a CF template then practiced deploying those onto AWS using the CLI.I trust that the examples we went through have demonstrated the benefits and immediate gains from the practice of deploying infrastructure as code. So far however, we have only done half the job. With the provisioning stage completed, you would naturally want to start configuring your infrastructure. Resources for Article: Further resources on this subject: Provision IaaS with Terraform [article] Ansible – An Introduction [article] Design with Spring AOP [article]
Read more
  • 0
  • 0
  • 14052

article-image-types-variables-and-function-techniques
Packt
16 Feb 2016
39 min read
Save for later

Types, Variables, and Function Techniques

Packt
16 Feb 2016
39 min read
This article is an introduction to the syntax used in the TypeScript language to apply strong typing to JavaScript. It is intended for readers that have not used TypeScript before, and covers the transition from standard JavaScript to TypeScript. We will cover the following topics in this article: Basic types and type syntax: strings, numbers, and booleans Inferred typing and duck-typing Arrays and enums The any type and explicit casting Functions and anonymous functions Optional and default function parameters Argument arrays Function callbacks and function signatures Function scoping rules and overloads (For more resources related to this topic, see here.) Basic types JavaScript variables can hold a number of data types, including numbers, strings, arrays, objects, functions, and more. The type of an object in JavaScript is determined by its assignment–so if a variable has been assigned a string value, then it will be of type string. This can, however, introduce a number of problems in our code. JavaScript is not strongly typed JavaScript objects and variables can be changed or reassigned on the fly. As an example of this, consider the following JavaScript code: var myString = "test"; var myNumber = 1; var myBoolean = true; We start by defining three variables, named myString, myNumber and myBoolean. The myString variable is set to a string value of "test", and as such will be of type string. Similarly, myNumber is set to the value of 1, and is therefore of type number, and myBoolean is set to true, making it of type boolean. Now let's start assigning these variables to each other, as follows: myString = myNumber; myBoolean = myString; myNumber = myBoolean; We start by setting the value of myString to the value of myNumber (which is the numeric value of 1). We then set the value of myBoolean to the value of myString, (which would now be the numeric value of 1). Finally, we set the value of myNumber to the value of myBoolean. What is happening here, is that even though we started out with three different types of variables—a string, a number, and a boolean—we are able to reassign any of these variables to one of the other types. We can assign a number to a string, a string to boolean, or a boolean to a number. While this type of assignment in JavaScript is legal, it shows that the JavaScript language is not strongly typed. This can lead to unwanted behaviour in our code. Parts of our code may be relying on the fact that a particular variable is holding a string, and if we inadvertently assign a number to this variable, our code may start to break in unexpected ways. TypeScript is strongly typed TypeScript, on the other hand, is a strongly typed language. Once you have declared a variable to be of type string, you can only assign string values to it. All further code that uses this variable must treat it as though it has a type of string. This helps to ensure that code that we write will behave as expected. While strong typing may not seem to be of any use with simple strings and numbers—it certainly does become important when we apply the same rules to objects, groups of objects, function definitions and classes. If you have written a function that expects a string as the first parameter and a number as the second, you cannot be blamed, if someone calls your function with a boolean as the first parameter and something else as the second. JavaScript programmers have always relied heavily on documentation to understand how to call functions, and the order and type of the correct function parameters. But what if we could take all of this documentation and include it within the IDE? Then, as we write our code, our compiler could point out to us—automatically—that we were using objects and functions in the wrong way. Surely this would make us more efficient, more productive programmers, allowing us to generating code with fewer errors? TypeScript does exactly that. It introduces a very simple syntax to define the type of a variable or a function parameter to ensure that we are using these objects, variables, and functions in the correct manner. If we break any of these rules, the TypeScript compiler will automatically generate errors, pointing us to the lines of code that are in error. This is how TypeScript got its name. It is JavaScript with strong typing - hence TypeScript. Let's take a look at this very simple language syntax that enables the "Type" in TypeScript. Type syntax The TypeScript syntax for declaring the type of a variable is to include a colon (:), after the variable name, and then indicate its type. Consider the following TypeScript code: var myString : string = "test"; var myNumber: number = 1; var myBoolean : boolean = true; This code snippet is the TypeScript equivalent of our preceding JavaScript code. We can now see an example of the TypeScript syntax for declaring a type for the myString variable. By including a colon and then the keyword string (: string), we are telling the compiler that the myString variable is of type string. Similarly, the myNumber variable is of type number, and the myBoolean variable is of type boolean. TypeScript has introduced the string, number and boolean keywords for each of these basic JavaScript types. If we attempt to assign a value to a variable that is not of the same type, the TypeScript compiler will generate a compile-time error. Given the variables declared in the preceding code, the following TypeScript code will generate some compile errors: myString = myNumber; myBoolean = myString; myNumber = myBoolean; TypeScript build errors when assigning incorrect types The TypeScript compiler is generating compile errors, because we are attempting to mix these basic types. The first error is generated by the compiler because we cannot assign a number value to a variable of type string. Similarly, the second compile error indicates that we cannot assign a string value to a variable of type boolean. Again, the third error is generated because we cannot assign a boolean value to a variable of type number. The strong typing syntax that the TypeScript language introduces, means that we need to ensure that the types on the left-hand side of an assignment operator (=) are the same as the types on the right-hand side of the assignment operator. To fix the preceding TypeScript code, and remove the compile errors, we would need to do something similar to the following: myString = myNumber.toString(); myBoolean = (myString === "test"); if (myBoolean) { myNumber = 1; } Our first line of code has been changed to call the .toString() function on the myNumber variable (which is of type number), in order to return a value that is of type string. This line of code, then, does not generate a compile error because both sides of the equal sign are of the same type. Our second line of code has also been changed so that the right hand side of the assignment operator returns the result of a comparison, myString === "test", which will return a value of type boolean. The compiler will therefore allow this code, because both sides of the assignment resolve to a value of type boolean. The last line of our code snippet has been changed to only assign the value 1 (which is of type number) to the myNumber variable, if the value of the myBoolean variable is true. Anders Hejlsberg describes this feature as "syntactic sugar". With a little sugar on top of comparable JavaScript code, TypeScript has enabled our code to conform to strong typing rules. Whenever you break these strong typing rules, the compiler will generate errors for your offending code. Inferred typing TypeScript also uses a technique called inferred typing, in cases where you do not explicitly specify the type of your variable. In other words, TypeScript will find the first usage of a variable within your code, figure out what type the variable is first initialized to, and then assume the same type for this variable in the rest of your code block. As an example of this, consider the following code: var myString = "this is a string"; var myNumber = 1; myNumber = myString; We start by declaring a variable named myString, and assign a string value to it. TypeScript identifies that this variable has been assigned a value of type string, and will, therefore, infer any further usages of this variable to be of type string. Our second variable, named myNumber has a number assigned to it. Again, TypeScript is inferring the type of this variable to be of type number. If we then attempt to assign the myString variable (of type string) to the myNumber variable (of type number) in the last line of code, TypeScript will generate a familiar error message: error TS2011: Build: Cannot convert 'string' to 'number' This error is generated because of TypeScript's inferred typing rules. Duck-typing TypeScript also uses a method called duck-typing for more complex variable types. Duck-typing means that if it looks like a duck, and quacks like a duck, then it probably is a duck. Consider the following TypeScript code: var complexType = { name: "myName", id: 1 }; complexType = { id: 2, name: "anotherName" }; We start with a variable named complexType that has been assigned a simple JavaScript object with a name and id property. On our second line of code, we can see that we are re-assigning the value of this complexType variable to another object that also has an id and a name property. The compiler will use duck-typing in this instance to figure out whether this assignment is valid. In other words, if an object has the same set of properties as another object, then they are considered to be of the same type. To further illustrate this point, let's see how the compiler reacts if we attempt to assign an object to our complexType variable that does not conform to this duck-typing: var complexType = { name: "myName", id: 1 }; complexType = { id: 2 }; complexType = { name: "anotherName" }; complexType = { address: "address" }; The first line of this code snippet defines our complexType variable, and assigns to it an object that contains both an id and name property. From this point, TypeScript will use this inferred type on any value we attempt to assign to the complexType variable. On our second line of code, we are attempting to assign a value that has an id property but not the name property. On the third line of code, we again attempt to assign a value that has a name property, but does not have an id property. On the last line of our code snippet, we have completely missed the mark. Compiling this code will generate the following errors: error TS2012: Build: Cannot convert '{ id: number; }' to '{ name: string; id: number; }': error TS2012: Build: Cannot convert '{ name: string; }' to '{ name: string; id: number; }': error TS2012: Build: Cannot convert '{ address: string; }' to '{ name: string; id: number; }': As we can see from the error messages, TypeScript is using duck-typing to ensure type safety. In each message, the compiler gives us clues as to what is wrong with the offending code – by explicitly stating what it is expecting. The complexType variable has both an id and a name property. To assign a value to the complexType variable, then, this value will need to have both an id and a name property. Working through each of these errors, TypeScript is explicitly stating what is wrong with each line of code. Note that the following code will not generate any error messages: var complexType = { name: "myName", id: 1 }; complexType = { name: "name", id: 2, address: "address" }; Again, our first line of code defines the complexType variable, as we have seen previously, with an id and a name property. Now, look at the second line of this example. The object we are using actually has three properties: name, id, and address. Even though we have added a new address property, the compiler will only check to see if our new object has both an id and a name. Because our new object has these properties, and will therefore match the original type of the variable, TypeScript will allow this assignment through duck-typing. Inferred typing and duck-typing are powerful features of the TypeScript language – bringing strong typing to our code, without the need to use explicit typing, that is, a colon : and then the type specifier syntax. Arrays Besides the base JavaScript types of string, number, and boolean, TypeScript has two other data types: Arrays and enums. Let's look at the syntax for defining arrays. An array is simply marked with the [] notation, similar to JavaScript, and each array can be strongly typed to hold a specific type as seen in the code below: var arrayOfNumbers: number[] = [1, 2, 3]; arrayOfNumbers = [3, 4, 5]; arrayOfNumbers = ["one", "two", "three"]; On the first line of this code snippet, we are defining an array named arrayOfNumbers, and further specify that each element of this array must be of type number. The second line then reassigns this array to hold some different numerical values. The last line of this snippet, however, will generate the following error message: error TS2012: Build: Cannot convert 'string[]' to 'number[]': This error message is warning us that the variable arrayOfNumbers is strongly typed to only accept values of type number. Our code tries to assign an array of strings to this array of numbers, and is therefore, generating a compile error. The any type All this type checking is well and good, but JavaScript is flexible enough to allow variables to be mixed and matched. The following code snippet is actually valid JavaScript code: var item1 = { id: 1, name: "item 1" }; item1 = { id: 2 }; Our first line of code assigns an object with an id property and a name property to the variable item1. The second line then re-assigns this variable to an object that has an id property but not a name property. Unfortunately, as we have seen previously, TypeScript will generate a compile time error for the preceding code: error TS2012: Build: Cannot convert '{ id: number; }' to '{ id: number; name: string; }' TypeScript introduces the any type for such occasions. Specifying that an object has a type of any in essence relaxes the compiler's strict type checking. The following code shows how to use the any type: var item1 : any = { id: 1, name: "item 1" }; item1 = { id: 2 }; Note how our first line of code has changed. We specify the type of the variable item1 to be of type : any so that our code will compile without errors. Without the type specifier of : any, the second line of code, would normally generate an error. Explicit casting As with any strongly typed language, there comes a time where you need to explicitly specify the type of an object. An object can be cast to the type of another by using the < > syntax. This is not a cast in the strictest sense of the word; it is more of an assertion that is used at runtime by the TypeScript compiler. Any explicit casting that you use will be compiled away in the resultant JavaScript and will not affect the code at runtime. Let's modify our previous code snippet to use explicit casting: var item1 = <any>{ id: 1, name: "item 1" }; item1 = { id: 2 }; Note that on the first line of this snippet, we have now replaced the : any type specifier on the left hand side of the assignment, with an explicit cast of <any> on the right hand side. This snippet of code is telling the compiler to explicitly cast, or to explicitly treat the { id: 1, name: "item 1" } object on the right-hand side as a type of any. So the item1 variable, therefore, also has the type of any (due to TypeScript's inferred typing rules). This then allows us to assign an object with only the { id: 2 } property to the variable item1 on the second line of code. This technique of using the < > syntax on the right hand side of an assignment, is called explicit casting. While the any type is a necessary feature of the TypeScript language – its usage should really be limited as much as possible. It is a language shortcut that is necessary to ensure compatibility with JavaScript, but over-use of the any type will quickly lead to coding errors that will be difficult to find. Rather than using the type any, try to figure out the correct type of the object you are using, and then use this type instead. We use an acronym within our programming teams: S.F.I.A.T. (pronounced sviat or sveat). Simply Find an Interface for the Any Type. While this may sound silly – it brings home the point that the any type should always be replaced with an interface – so simply find it. Just remember that by actively trying to define what an object's type should be, we are building strongly typed code, and therefore protecting ourselves from future coding errors and bugs. Enums Enums are a special type that has been borrowed from other languages such as C#, and provide a solution to the problem of special numbers. An enum associates a human-readable name for a specific number. Consider the following code: enum DoorState { Open, Closed, Ajar } In this code snippet, we have defined an enum called DoorState to represent the state of a door. Valid values for this door state are Open, Closed, or Ajar. Under the hood (in the generated JavaScript), TypeScript will assign a numeric value to each of these human-readable enum values. In this example, the DoorState.Open enum value will equate to a numeric value of 0. Likewise, the enum value DoorState.Closed will be equate to the numeric value of 1, and the DoorState.Ajar enum value will equate to 2. Let's have a quick look at how we would use these enum values: window.onload = () => { var myDoor = DoorState.Open; console.log("My door state is " + myDoor.toString()); }; The first line within the window.onload function creates a variable named myDoor, and sets its value to DoorState.Open. The second line simply logs the value of myDoor to the console. The output of this console.log function would be: My door state is 0 This clearly shows that the TypeScript compiler has substituted the enum value of DoorState.Open with the numeric value 0. Now let's use this enum in a slightly different way: window.onload = () => { var openDoor = DoorState["Closed"]; console.log("My door state is " + openDoor.toString()); }; This code snippet uses a string value of "Closed" to lookup the enum type, and assign the resulting enum value to the openDoor variable. The output of this code would be: My door state is 1 This sample clearly shows that the enum value of DoorState.Closed is the same as the enum value of DoorState["Closed"], because both variants resolve to the numeric value of 1. Finally, let's have a look at what happens when we reference an enum using an array type syntax: window.onload = () => { var ajarDoor = DoorState[2]; console.log("My door state is " + ajarDoor.toString()); }; Here, we assign the variable openDoor to an enum value based on the 2nd index value of the DoorState enum. The output of this code, though, is surprising: My door state is Ajar You may have been expecting the output to be simply 2, but here we are getting the string "Ajar" – which is a string representation of our original enum name. This is actually a neat little trick – allowing us to access a string representation of our enum value. The reason that this is possible is down to the JavaScript that has been generated by the TypeScript compiler. Let's have a look, then, at the closure that the TypeScript compiler has generated: var DoorState; (function (DoorState) { DoorState[DoorState["Open"] = 0] = "Open"; DoorState[DoorState["Closed"] = 1] = "Closed"; DoorState[DoorState["Ajar"] = 2] = "Ajar"; })(DoorState || (DoorState = {})); This strange looking syntax is building an object that has a specific internal structure. It is this internal structure that allows us to use this enum in the various ways that we have just explored. If we interrogate this structure while debugging our JavaScript, we will see the internal structure of the DoorState object is as follows: DoorState {...} [prototype]: {...} [0]: "Open" [1]: "Closed" [2]: "Ajar" [prototype]: [] Ajar: 2 Closed: 1 Open: 0 The DoorState object has a property called "0", which has a string value of "Open". Unfortunately, in JavaScript the number 0 is not a valid property name, so we cannot access this property by simply using DoorState.0. Instead, we must access this property using either DoorState[0] or DoorState["0"]. The DoorState object also has a property named Open, which is set to the numeric value 0. The word Open IS a valid property name in JavaScript, so we can access this property using DoorState["Open"], or simply DoorState.Open, which equate to the same property in JavaScript. While the underlying JavaScript can be a little confusing, all we need to remember about enums is that they are a handy way of defining an easily remembered, human-readable name to a special number. Using human-readable enums, instead of just scattering various special numbers around in our code, also makes the intent of the code clearer. Using an application wide value named DoorState.Open or DoorState.Closed is far simpler than remembering to set a value to 0 for Open, 1 for Closed, and 3 for ajar. As well as making our code more readable, and more maintainable, using enums also protects our code base whenever these special numeric values change – because they are all defined in one place. One last note on enums – we can set the numeric value manually, if needs be: enum DoorState { Open = 3, Closed = 7, Ajar = 10 } Here, we have overridden the default values of the enum to set DoorState.Open to 3, DoorState.Closed to 7, and DoorState.Ajar to 10. Const enums With the release of TypeScript 1.4, we are also able to define const enums as follows: const enum DoorStateConst { Open, Closed, Ajar } var myState = DoorStateConst.Open; These types of enums have been introduced largely for performance reasons, and the resultant JavaScript will not contain the full closure definition for the DoorStateConst enum as we saw previously. Let's have a quick look at the JavaScript that is generated from this DoorStateConst enum: var myState = 0 /* Open */; Note how we do not have a full JavaScript closure for the DoorStateConst at all. The compiler has simply resolved the DoorStateConst.Open enum to its internal value of 0, and removed the const enum definition entirely. With const enums, we therefore cannot reference the internal string value of an enum, as we did in our previous code sample. Consider the following example: // generates an error console.log(DoorStateConst[0]); // valid usage console.log(DoorStateConst["Open"]); The first console.log statement will now generate a compile time error – as we do not have the full closure available with the property of [0] for our const enum. The second usage of this const enum is valid, however, and will generate the following JavaScript: console.log(0 /* "Open" */); When using const enums, just keep in mind that the compiler will strip away all enum definitions and simply substitute the numeric value of the enum directly into our JavaScript code. Functions JavaScript defines functions using the function keyword, a set of braces, and then a set of curly braces. A typical JavaScript function would be written as follows: function addNumbers(a, b) { return a + b; } var result = addNumbers(1, 2); var result2 = addNumbers("1", "2"); This code snippet is fairly self-explanatory; we have defined a function named addNumbers that takes two variables and returns their sum. We then invoke this function, passing in the values of 1 and 2. The value of the variable result would then be 1 + 2, which is 3. Now have a look at the last line of code. Here, we are invoking the addNumbers function, passing in two strings as arguments, instead of numbers. The value of the variable result2 would then be a string, "12". This string value seems like it may not be the desired result, as the name of the function is addNumbers. Copying the preceding code into a TypeScript file would not generate any errors, but let's insert some type rules to the preceding JavaScript to make it more robust: function addNumbers(a: number, b: number): number { return a + b; }; var result = addNumbers(1, 2); var result2 = addNumbers("1", "2"); In this TypeScript code, we have added a :number type to both of the parameters of the addNumbers function (a and b), and we have also added a :number type just after the ( ) braces. Placing a type descriptor here means that the return type of the function itself is strongly typed to return a value of type number. In TypeScript, the last line of code, however, will cause a compilation error: error TS2082: Build: Supplied parameters do not match any signature of call target: This error message is generate because we have explicitly stated that the function should accept only numbers for both of the arguments a and b, but in our offending code, we are passing two strings. The TypeScript compiler, therefore, cannot match the signature of a function named addNumbers that accepts two arguments of type string. Anonymous functions The JavaScript language also has the concept of anonymous functions. These are functions that are defined on the fly and don't specify a function name. Consider the following JavaScript code: var addVar = function(a, b) { return a + b; }; var result = addVar(1, 2); This code snippet defines a function that has no name and adds two values. Because the function does not have a name, it is known as an anonymous function. This anonymous function is then assigned to a variable named addVar. The addVar variable, then, can then be invoked as a function with two parameters, and the return value will be the result of executing the anonymous function. In this case, the variable result will have a value of 3. Let's now rewrite the preceding JavaScript function in TypeScript, and add some type syntax, in order to ensure that the function only accepts two arguments of type number, and returns a value of type number: var addVar = function(a: number, b: number): number { return a + b; } var result = addVar(1, 2); var result2 = addVar("1", "2"); In this code snippet, we have created an anonymous function that accepts only arguments of type number for the parameters a and b, and also returns a value of type number. The types for both the a and b parameters, as well as the return type of the function, are now using the :number syntax. This is another example of the simple "syntactic sugar" that TypeScript injects into the language. If we compile this code, TypeScript will reject the code on the last line, where we try to call our anonymous function with two string parameters: error TS2082: Build: Supplied parameters do not match any signature of call target: Optional parameters When we call a JavaScript function that has is expecting parameters, and we do not supply these parameters, then the value of the parameter within the function will be undefined. As an example of this, consider the following JavaScript code: var concatStrings = function(a, b, c) { return a + b + c; } console.log(concatStrings("a", "b", "c")); console.log(concatStrings("a", "b")); Here, we have defined a function called concatStrings that takes three parameters, a, b, and c, and simply returns the sum of these values. If we call this function with all three parameters, as seen in the second last line of this snipped, we will end up with the string "abc" logged to the console. If, however, we only supply two parameters, as seen in the last line of this snippet, the string "abundefined" will be logged to the console. Again, if we call a function and do not supply a parameter, then this parameter, c in our case, will be simply undefined. TypeScript introduces the question mark ? syntax to indicate optional parameters. Consider the following TypeScript function definition: var concatStrings = function(a: string, b: string, c?: string) { return a + b + c; } console.log(concatStrings("a", "b", "c")); console.log(concatStrings("a", "b")); console.log(concatStrings("a")); This is a strongly typed version of the original concatStrings JavaScript function that we were using previously. Note the addition of the ? character in the syntax for the third parameter: c?: string. This indicates that the third parameter is optional, and therefore, all of the preceding code will compile cleanly, except for the last line. The last line will generate an error: error TS2081: Build: Supplied parameters do not match any signature of call target. This error is generated because we are attempting to call the concatStrings function with only a single parameter. Our function definition, though, requires at least two parameters, with only the third parameter being optional. The optional parameters must be the last parameters in the function definition. You can have as many optional parameters as you want, as long as non-optional parameters precede the optional parameters. Default parameters A subtle variant on the optional parameter function definition, allows us to specify the value of a parameter if it is not passed in as an argument from the calling code. Let's modify our preceding function definition to use an optional parameter: var concatStrings = function(a: string, b: string, c: string = "c") { return a + b + c; } console.log(concatStrings("a", "b", "c")); console.log(concatStrings("a", "b")); This function definition has now dropped the ? optional parameter syntax, but instead has assigned a value of "c" to the last parameter: c:string = "c". By using default parameters, if we do not supply a value for the final parameter named c, the concatStrings function will substitute the default value of "c" instead. The argument c, therefore, will not be undefined. The output of the last two lines of code will both be "abc". Note that using the default parameter syntax will automatically make the parameter optional. The arguments variable The JavaScript language allows a function to be called with a variable number of arguments. Every JavaScript function has access to a special variable, named arguments, that can be used to retrieve all arguments that have been passed into the function. As an example of this, consider the following JavaScript code: function testParams() { if (arguments.length > 0) { for (var i = 0; i < arguments.length; i++) { console.log("Argument " + i + " = " + arguments[i]); } } } testParams(1, 2, 3, 4); testParams("first argument"); In this code snippet, we have defined a function name testParams that does not have any named parameters. Note, though, that we can use the special variable, named arguments, to test whether the function was called with any arguments. In our sample, we can simply loop through the arguments array, and log the value of each argument to the console, by using an array indexer : arguments[i]. The output of the console.log calls are as follows: Argument 0 = 1 Argument 1 = 2 Argument 2 = 3 Argument 3 = 4 Argument 0 = first argument So, how do we express a variable number of function parameters in TypeScript? The answer is to use what are called rest parameters, or the three dots (…) syntax. Here is the equivalent testParams function, expressed in TypeScript: function testParams(...argArray: number[]) { if (argArray.length > 0) { for (var i = 0; i < argArray.length; i++) { console.log("argArray " + i + " = " + argArray[i]); console.log("arguments " + i + " = " + arguments[i]); } } } testParams(1); testParams(1, 2, 3, 4); testParams("one", "two"); Note the use of the …argArray: number[] syntax for our testParams function. This syntax is telling the TypeScript compiler that the function can accept any number of arguments. This means that our usages of this function, i.e. calling the function with either testParams(1) or testParams(1,2,3,4), will both compile correctly. In this version of the testParams function, we have added two console.log lines, just to show that the arguments array can be accessed by either the named rest parameter, argArray[i], or through the normal JavaScript array, arguments[i]. The last line in this sample will, however, generate a compile error, as we have defined the rest parameter to only accept numbers, and we are attempting to call the function with strings. The the subtle difference between using argArray and arguments is the inferred type of the argument. Since we have explicitly specified that argArray is of type number, TypeScript will treat any item of the argArray array as a number. However, the internal arguments array does not have an inferred type, and so will be treated as the any type. We can also combine normal parameters along with rest parameters in a function definition, as long as the rest parameters are the last to be defined in the parameter list, as follows: function testParamsTs2(arg1: string, arg2: number, ...ArgArray: number[]) { } Here, we have two normal parameters named arg1 and arg2 and then an argArray rest parameter. Mistakenly placing the rest parameter at the beginning of the parameter list will generate a compile error. Function callbacks One of the most powerful features of JavaScript–and in fact the technology that Node was built on–is the concept of callback functions. A callback function is a function that is passed into another function. Remember that JavaScript is not strongly typed, so a variable can also be a function. This is best illustrated by having a look at some JavaScript code: function myCallBack(text) { console.log("inside myCallback " + text); } function callingFunction(initialText, callback) { console.log("inside CallingFunction"); callback(initialText); } callingFunction("myText", myCallBack); Here, we have a function named myCallBack that takes a parameter and logs its value to the console. We then define a function named callingFunction that takes two parameters: initialText and callback. The first line of this funciton simply logs "inside CallingFunction" to the console. The second line of the callingFunction is the interesting bit. It assumes that the callback argument is in fact a function, and invokes it. It also passes the initialText variable to the callback function. If we run this code, we will get two messages logged to the console, as follows: inside CallingFunction inside myCallback myText But what happens if we do not pass a function as a callback? There is nothing in the preceding code that signals to us that the second parameter of callingFunction must be a function. If we inadvertently called the callingFunction function with a string, instead of a function as the second parameter as follows: callingFunction("myText", "this is not a function"); We would get a JavaScript runtime error: 0x800a138a - JavaScript runtime error: Function expected Defensive minded programmers, however, would first check whether the callback parameter was in fact a function before invoking it, as follows: function callingFunction(initialText, callback) { console.log("inside CallingFunction"); if (typeof callback === "function") { callback(initialText); } else { console.log(callback + " is not a function"); } } callingFunction("myText", "this is not a function"); Note the third line of this code snippet, where we check the type of the callback variable before invoking it. If it is not a function, we then log a message to the console. On the last line of this snippet, we are executing the callingFunction, but this time passing a string as the second parameter. The output of the code snipped would be: inside CallingFunction this is not a function is not a function When using function callbacks, then, JavaScript programmers need to do two things; firstly, understand which parameters are in fact callbacks and secondly, code around the invalid use of callback functions. Function signatures The TypeScript "syntactic sugar" that enforces strong typing, is not only intended for variables and types, but for function signatures as well. What if we could document our JavaScript callback functions in code, and then warn users of our code when they are passing the wrong type of parameter to our functions ? TypeScript does this through function signatures. A function signature introduces a fat arrow syntax, () =>, to define what the function should look like. Let's re-write the preceding JavaScript sample in TypeScript: function myCallBack(text: string) { console.log("inside myCallback " + text); } function callingFunction(initialText: string, callback: (text: string) => void) { callback(initialText); } callingFunction("myText", myCallBack); callingFunction("myText", "this is not a function"); Our first function definition, myCallBack now strongly types the text parameter to be of type string. Our callingFunction function has two parameters; initialText, which is of type string, and callback, which now has the new function signature syntax. Let's look at this function signature more closely: callback: (text: string) => void What this function definition is saying, is that the callback argument is typed (by the : syntax) to be a function, using the fat arrow syntax () =>. Additionally, this function takes a parameter named text that is of type string. To the right of the fat arrow syntax, we can see a new TypeScript basic type, called void. Void is a keyword to denote that a function does not return a value. So, the callingFunction function will only accept, as its second argument, a function that takes a single string parameter and returns nothing. Compiling the preceding code will correctly highlight an error in the last line of the code snippet, where we passing a string as the second parameter, instead of a callback function: error TS2082: Build: Supplied parameters do not match any signature of call target: Type '(text: string) => void' requires a call signature, but type 'String' lacks one Given the preceding function signature for the callback function, the following code would also generate compile time errors: function myCallBackNumber(arg1: number) { console.log("arg1 = " + arg1); } callingFunction("myText", myCallBackNumber); Here, we are defining a function named myCallBackNumber, that takes a number as its only parameter. When we attempt to compile this code, we will get an error message indicating that the callback parameter, which is our myCallBackNumber function, also does not have the correct function signature: Call signatures of types 'typeof myCallBackNumber' and '(text: string) => void' are incompatible. The function signature of myCallBackNumber would actually be (arg1:number) => void, instead of the required (text: string) => void, hence the error. In function signatures, the parameter name (arg1 or text) does not need to be the same. Only the number of parameters, their types, and the return type of the function need to be the same. This is a very powerful feature of TypeScript — defining in code what the signatures of functions should be, and warning users when they do not call a function with the correct parameters. As we saw in our introduction to TypeScript, this is most significant when we are working with third-party libraries. Before we are able to use third-party functions, classes, or objects in TypeScript, we need to define what their function signatures are. These function definitions are put into a special type of TypeScript file, called a declaration file, and saved with a .d.ts extension. Function callbacks and scope JavaScript uses lexical scoping rules to define the valid scope of a variable. This means that the value of a variable is defined by its location within the source code. Nested functions have access to variables that are defined in their parent scope. As an example of this, consider the following TypeScript code: function testScope() { var testVariable = "myTestVariable"; function print() { console.log(testVariable); } } console.log(testVariable); This code snippet defines a function named testScope. The variable testVariable is defined within this function. The print function is a child function of testScope, so it has access to the testVariable variable. The last line of the code, however, will generate a compile error, because it is attempting to use the variabletestVariable, which is lexically scoped to be valid only inside the body of the testScope function: error TS2095: Build: Could not find symbol 'testVariable'. Simple, right? A nested function has access to variables depending on its location within the source code. This is all well and good, but in large JavaScript projects, there are many different files and many areas of the code are designed to be re-usable. Let's take a look at how these scoping rules can become a problem. For this sample, we will use a typical callback scenario—using jQuery to execute an asynchronous call to fetch some data. Consider the following TypeScript code: var testVariable = "testValue"; function getData() { var testVariable_2 = "testValue_2"; $.ajax( { url: "/sample_json.json" success: (data, status, jqXhr) => { console.log("success : testVariable is " + testVariable); console.log("success : testVariable_2 is" + testVariable_2); }, error: (message, status, stack) => { alert("error " + message); } } ); } getData(); In this code snippet, we are defining a variable named testVariable and setting its value. We then define a function called getData. The getData function sets another variable called testVariable_2, and then calls the jQuery $.ajax function. The $.ajax function is configured with three properties: url, success, and error. The url property is a simple string that points to a sample_json.json file in our project directory. The success property is an anonymous function callback, that simply logs the values of testVariable and testVariable_2 to the console. Finally, the error property is also an anonymous function callback, that simply pops up an alert. This code runs as expected, and the success function will log the following results to the console: success : testVariable is :testValue success : testVariable_2 is :testValue_2 So far so good. Now, let's assume that we are trying to refactor the preceding code, as we are doing quite a few similar $.ajax calls, and want to reuse the success callback function elsewhere. We can easily switch out this anonymous function, and create a named function for our success callback, as follows: var testVariable = "testValue"; function getData() { var testVariable_2 = "testValue_2"; $.ajax( { url: "/sample_json.json", success: successCallback, error: (message, status, stack) => { alert("error " + message); } } ); } function successCallback(data, status, jqXhr) { console.log("success : testVariable is :" + testVariable); console.log("success : testVariable_2 is :" + testVariable_2); } getData(); In this sample, we have created a new function named successCallback with the same parameters as our previous anonymous function. We have also modified the $.ajax call to simply pass this function in, as a callback function for the success property: success: successCallback. If we were to compile this code now, TypeScript would generate an error, as follows: error TS2095: Build: Could not find symbol ''testVariable_2''. Since we have changed the lexical scope of our code, by creating a named function, the new successCallback function no longer has access the variable testVariable_2. It is fairly easy to spot this sort of error in a trivial example, but in larger projects, and when using third-party libraries, these sorts of errors become more difficult to track down. It is, therefore, worth mentioning that when using callback functions, we need to understand this lexical scope. If your code expects a property to have a value, and it does not have one after a callback, then remember to have a look at the context of the calling code. Function overloads As JavaScript is a dynamic language, we can often call the same function with different argument types. Consider the following JavaScript code: function add(x, y) { return x + y; } console.log("add(1,1)=" + add(1,1)); console.log("add(''1'',''1'')=" + add("1", "1")); console.log("add(true,false)=" + add(true, false)); Here, we are defining a simple add function that returns the sum of its two parameters, x and y. The last three lines of this code snippet simply log the result of the add function with different types: two numbers, two strings, and two boolean values. If we run this code, we will see the following output: add(1,1)=2 add('1','1')=11 add(true,false)=1 TypeScript introduces a specific syntax to indicate multiple function signatures for the same function. If we were to replicate the preceding code in TypeScript, we would need to use the function overload syntax: function add(arg1: string, arg2: string): string; function add(arg1: number, arg2: number): number; function add(arg1: boolean, arg2: boolean): boolean; function add(arg1: any, arg2: any): any { return arg1 + arg2; } console.log("add(1,1)=" + add(1, 1)); console.log("add(''1'',''1'')=" + add("1", "1")); console.log("add(true,false)=" + add(true, false)); The first line of this code snippet specifies a function overload signature for the add function that accepts two strings and returns a string. The second line specifies another function overload that uses numbers, and the third line uses booleans. The fourth line contains the actual body of the function and uses the type specifier of any. The last three lines of this snippet show how we would use these function signatures, and are similar to the JavaScript code that we have been using previously. There are three points of interest in the preceding code snippet. Firstly, none of the function signatures on the first three lines of the snippet actually have a function body. Secondly, the final function definition uses the type specifier of any and eventually includes the function body. The function overload syntax must follow this structure, and the final function signature, that includes the body of the function must use the any type specifier, as anything else will generate compile-time errors. The third point to note, is that we are limiting the add function, by using these function overload signatures, to only accept two parameters that are of the same type. If we were to try and mix our types; for example, if we call the function with a boolean and a string, as follows: console.log("add(true,''1'')", add(true, "1")); TypeScript would generate compile errors: error TS2082: Build: Supplied parameters do not match any signature of call target: error TS2087: Build: Could not select overload for ''call'' expression. This seems to contradict our final function definition though. In the original TypeScript sample, we had a function signature that accepted (arg1: any, arg2: any); so, in theory, this should be called when we try to add a boolean and a number. The TypeScript syntax for function overloads, however, does not allow this. Remember that the function overload syntax must include the use of the any type for the function body, as all overloads eventually call this function body. However, the inclusion of the function overloads above the function body indicates to the compiler that these are the only signatures that should be available to the calling code. Summary To learn more about TypeScript, the following books published by Packt Publishing (https://www.packtpub.com/) are recommended: Learning TypeScript (https://www.packtpub.com/web-development/learning-typescript) TypeScript Essentials (https://www.packtpub.com/web-development/typescript-essentials) Resources for Article: Further resources on this subject: Introduction to TypeScript[article] Writing SOLID JavaScript code with TypeScript[article] JavaScript Execution with Selenium[article]
Read more
  • 0
  • 0
  • 14049
article-image-how-optimize-put-requests
Packt
04 Sep 2015
8 min read
Save for later

How to Optimize PUT Requests

Packt
04 Sep 2015
8 min read
In this article by Naoya Hashiotmo, author of the book Amazon S3 Cookbook, explains how to optimize PUT requests, it would be effective to use multipart uploads because it can aggregate throughput by parallelizing PUT requests and uploading a large object into parts. It is recommended that the size of each part should be between 25 and 50 MB for higher networks and 10 MB for mobile networks. (For more resources related to this topic, see here.) Amazon S3 is a highly-scalable, reliable, and low-latency data storage service at a very low cost, designed for mission-critical and primary data storage. It provides the Amazon S3 APIs to simplify your programming tasks. S3 performance optimization is composed of several factors, for example, which region to choose to reduce latency, considering the naming scheme and optimizing the put and get operations. Multipart upload consists of three-step processes; the first step is initiating the upload, next is uploading the object parts, and finally, after uploading all the parts, the multipart upload is finished. The following methods are currently supported to upload objects with multipart upload: AWS SDK for Android AWS SDK for iOS AWS SDK for Java AWS SDK for JavaScript AWS SDK for PHP AWS SDK for Python AWS SDK for Ruby AWS SDK for .NET REST API AWS CLI In order to try multipart upload and see how much it aggregates throughput, we use AWS SDK for Node.js and S3 via NPM (package manager for Node.js). AWS CLI also supports multipart upload. When you use the AWS CLI s3 or s3api subcommand to upload an object, the object is automatically uploaded via multipart requests. Getting ready You need to complete the following set up in advance: Sign up on AWS and be able to access S3 with your IAM credentials Install and set up AWS CLI in your PC or use Amazon Linux AMI Install Node.js It is recommended that you score the benchmark from your local PC or if you use the EC2 instance, you should launch an instance and create an S3 bucket in different regions. For example, if you launch an instance in the Asia Pacific Tokyo region, you should create an S3 bucket in the US standard region. The reason is that the latency between EC2 and S3 is very low, and it is hard to see the difference. How to do it… We upload a 300 GB file in an S3 bucket over HTTP in two ways; one is to use a multipart upload and the other is not to use multipart upload to compare the time. To clearly see how the performance differs, I launched an instance and created an S3 bucket in different regions as follows: EC2 instance: Asia Pacific Tokyo Region (ap-northeast-1) S3 bucket: US Standard region (us-east-1) First, we install the S3 Node.js module via npm, create a dummy file, upload the object into a bucket using a sample Node.js script without enabling multipart upload, and then do the same enabling multipart upload, so that we can see how multipart upload performs the operation. Now, let's move on to the instructions: Install s3 via the npm command: $ cdaws-nodejs-sample/ $ npm install s3 Create a 300 GB dummy file: $ file=300mb.dmp $ dd if=/dev/zero of=${file} bs=10M count=30 Put the following script and save the script as s3_upload.js: // Load the SDK var AWS = require('aws-sdk'); var s3 = require('s3'); var conf = require('./conf'); // Load parameters var client = s3.createClient({ maxAsyncS3: conf.maxAsyncS3, s3RetryCount: conf.s3RetryCount, s3RetryDelay: conf.s3RetryDelay, multipartUploadThreshold: conf.multipartUploadThreshold, multipartUploadSize: conf.multipartUploadSize, }); var params = { localFile: conf.localFile, s3Params: { Bucket: conf.Bucket, Key: conf.localFile, }, }; // upload objects console.log("## s3 Parameters"); console.log(conf); console.log("## Begin uploading."); var uploader = client.uploadFile(params); uploader.on('error', function(err) { console.error("Unable to upload:", err.stack); }); uploader.on('progress', function() { console.log("Progress", uploader.progressMd5Amount,uploader.progressAmount, uploader.progressTotal); }); uploader.on('end', function() { console.log("## Finished uploading."); }); Create a configuration file and save the file conf.js in the same directory as s3_upload.js: exports.maxAsyncS3 = 20; // default value exports.s3RetryCount = 3; // default value exports.s3RetryDelay = 1000; // default value exports.multipartUploadThreshold = 20971520; // default value exports.multipartUploadSize = 15728640; // default value exports.Bucket = "your-bucket-name"; exports.localFile = "300mb.dmp"; exports.Key = "300mb.dmp"; How it works… First of all, let's try uploading a 300 GB object using multipart upload, and then upload the same file without using multipart upload. You can upload an object and see how long it takes by typing the following command: $ time node s3_upload.js ## s3 Parameters { maxAsyncS3: 20, s3RetryCount: 3, s3RetryDelay: 1000, multipartUploadThreshold: 20971520, multipartUploadSize: 15728640, localFile: './300mb.dmp', Bucket: 'bucket-sample-us-east-1', Key: './300mb.dmp' } ## Begin uploading. Progress 0 16384 314572800 Progress 0 32768 314572800 … Progress 0 314572800 314572800 Progress 0 314572800 314572800 ## Finished uploading. real 0m16.111s user 0m4.164s sys 0m0.884s As it took about 16 seconds to upload the object, the transfer rate was 18.75 MB/sec. Then, let's change the following parameters in the configuration (conf.js) as follows and see the result. The 300 GB object is uploaded through only one S3 client and exports.maxAsyncS3 = 1; exports. multipartUploadThreshold = 2097152000; exports.maxAsyncS3 = 1; exports.s3RetryCount = 3; // default value exports.s3RetryDelay = 1000; // default value exports.multipartUploadThreshold = 2097152000; exports.multipartUploadSize = 15728640; // default value exports.Bucket = "your-bucket-name"; exports.localFile = "300mb.dmp"; exports.Key = "300mb.dmp"; Let's see the result after changing the parameters in the configuration (conf.js): $ time node s3_upload.js ## s3 Parameters … ## Begin uploading. Progress 0 16384 314572800 … Progress 0 314572800 314572800 ## Finished uploading. real 0m41.887s user 0m4.196s sys 0m0.728s As it took about 42 seconds to upload the object, the transfer rate was 7.14 MB/sec. Now, let's quickly check each parameter, and then get to the conclusion; maxAsyncS3 defines the maximum number of simultaneous requests that S3 clients are open to Amazon S3. The default value is 20. s3RetryCount defines the number of retries when a request fails. The default value is 3. s3RetryDelay is how many milliseconds S3 clients will wait when a request fails. The default value is 1000. multipartUploadThreshold defines the size of uploading objects via multipart requests. The object will be uploaded via multipart request, if you choose an object that is greater than the size you specified. The default value is 20 MB, the minimum is 5 MB, and the maximum is 5 GB. multipartUploadSize defines the size for each part when uploaded via the multipart request. The default value is 15 MB, the minimum is 5 MB, and the maximum is 5 GB. The following table shows the speed test score with different parameters: maxAsyncS3 1 20 20 40 30 s3RetryCount 3 3 3 3 3 s3RetryDelay 1000 1000 1000 1000 1000 multipartUploadThreshold 2097152000 20971520 20971520 20971520 20971520 multipartUploadSize 15728640 15728640 31457280 15728640 10728640 Time (seconds) 41.88 16.11 17.41 16.37 9.68 Transfer Rate (MB) 7.51 19.53 18.07 19.22 32.50 In conclusion, multipart upload is effective for optimizing the PUT operation, aggregating throughput. However, you need to consider the following: Benchmark your scenario and evaluate the number of retry count, delay, parts, and the multipart upload size based on the networks that your application belongs to. There's more… Multipart upload specification There are limits to using multipart upload. The following table shows the specification of multipart upload: Item Specification Maximum object size 5 TB Maximum number of parts per upload 10,000 Part numbers 1 to 10,000 (inclusive) Part size 5 MB to 5 GB, last part can be more than 5 MB Maximum number of parts returned for a list of parts request 1,000 Maximum number of multipart uploads returned in a list of multipart uploads request 1,000 Multipart upload and charging If you initiate multipart upload and abort the request, Amazon S3 deletes the upload artifacts and any parts you have uploaded and you are not charged for the bills. However, you are charged for all storage, bandwidth, and requests for the multipart upload requests and the associated parts of an object after the operation is completed. The point is you are charged when a multipart upload is completed (not aborted). See also Multipart Upload Overview https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html AWS SDK for Node.js http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/node-intro.htm Node.js S3 package npm https://www.npmjs.com/package/s3 Amazon Simple Storage Service: Introduction to Amazon S3 http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html (PFC403) Maximizing Amazon S3 Performance | AWS re:Invent 2014 http://www.slideshare.net/AmazonWebServices/pfc403-maximizing-amazon-s3-performance-aws-reinvent-2014 AWS re:Invent 2014 | (PFC403) Maximizing Amazon S3 Performance https://www.youtube.com/watch?v=_FHRzq7eHQc Summary In this article, we learned how to optimize the PUT requests and uploading a large object into parts. Resources for Article: Further resources on this subject: Amazon Web Services[article] Achieving High-Availability on AWS Cloud[article] Architecture and Component Overview [article]
Read more
  • 0
  • 0
  • 14028

article-image-8-tech-companies-and-18-governments-sign-the-christchurch-call-to-curb-online-extremism-the-us-backs-off
Sugandha Lahoti
16 May 2019
6 min read
Save for later

8 tech companies and 18 governments sign the Christchurch Call to curb online extremism; the US backs off

Sugandha Lahoti
16 May 2019
6 min read
Yesterday, New Zealand Prime Minister, Jacinda Ardern, and French President, Emmanuel Macron brought together world governments and leaders from the tech sector to adopt the Christchurch Call. This call is a non-binding agreement to eliminate terrorist and violent extremist content online after the Christ church terror attack, which took place March 15 and killed 51 Muslim worshipers in Christchurch mosques in New Zealand. Tech companies scrambled to take action due to the speed and volume of content which was live streamed on Facebook and then subsequently uploaded, reuploaded and shared by the users worldwide. “The Christchurch Call is a commitment by Governments and tech companies to eliminate terrorist and violent extremist content online. It rests on the conviction that a free, open and secure internet offers extraordinary benefits to society. Respect for freedom of expression is fundamental. However, no one has the right to create and share terrorist and violent extremist content online.”, reads the call. These tech companies and governments will also be developing tools to prevent the upload of terrorist and violent extremist content; countering the roots of violent extremism; increasing transparency around the removal and detection of content, and reviewing how companies’ algorithms to direct users to violent extremist content. Representatives from Facebook, Google, Microsoft, Twitter, and other tech giants have also agreed to identify "appropriate checks on live streaming, aimed at reducing the risk of disseminating terrorist and violent extremist content online". Two days ago in a statement, Facebook declared that from now on they will start restricting users from using Facebook Live if they break certain rules-including their Dangerous Organizations and Individuals policy. In addition to signing the Christchurch Call, Amazon, Facebook, Google, Twitter, and Microsoft are publishing nine steps that they will take to address the abuse of technology to spread terrorist and violent extremist content. These nine steps include five individual actions that each company is committing to take and a further four collaborative actions they’ll take together. Terms of Use user reporting of terrorist and violent extremist content; enhancing technology; live streaming, and transparency reports. Also included were four collaborative actions such as shared technology development, crisis protocols, educating the public about terrorist and extremist violent content online and combating hate and bigotry. Also, to be noted, Twitter CEO Jack Dorsey was the only social media CEO to attend the Call. Facebook’s Mark Zuckerberg was a no show with Facebook’s head of global affairs Nick Clegg attending the call. Microsoft was represented by President Brad Smith while Wikimedia was represented by Wikipedia founder Jimmy Wales. Senior Vice President for Global Affairs Kent Walker participated on behalf of Google. https://twitter.com/Policy/status/1128693729181749248 The signing of the Christchurch Call was organized around a meeting of digital ministers from the Group of 7 nations this week in Paris. Until now 18 countries and tech organizations have joined the call, pledging to counter the drivers of terrorism and violent extremism, and ensure effective enforcement of applicable laws. The Call was adopted at the meeting by France, New Zealand, Canada, Indonesia, Ireland, Jordan, Norway, Senegal, the UK, and the European Commission. Other countries who have adopted the Call but were not at the meeting are Australia, Germany, India, Italy, Japan, the Netherlands, Spain, and Sweden. Sri Lanka has not signed the agreement. In April, Sri Lanka has massive bombings on churches and hotels, which killed 321 people and led to the Sri Lankan government issuing a blanket social media ban in the weeks to come for fear of them inciting further violence, fear and spread of misinformation. Sri Lanka’s junior minister for defense, Ruwan Wijewardene, in an initial investigation had revealed the bombings were related to New Zealand mosque attack. However, Ardern denied those claims stating that the New Zealand government was not aware of any such intelligence. Interestingly, the U.S also refused to join other nations to pledge the Christchurch Call. A statement from the White House reads, “While the United States is not currently in a position to join the endorsement, we continue to support the overall goals reflected in the Call. We will continue to engage governments, industry, and civil society to counter terrorist content on the Internet,” “We continue to be proactive in our efforts to counter terrorist content online while also continuing to respect freedom of expression and freedom of the press,” the statement reads. “Further, we maintain that the best tool to defeat terrorist speech is productive speech, and thus we emphasize the importance of promoting credible, alternative narratives as the primary means by which we can defeat terrorist messaging.” Critics shot down White House’s refusal to pledge to the non-binding Christchurch Call. https://twitter.com/cwarzel/status/1128701486144282624 https://twitter.com/MarkMellman/status/1128691586001326082 https://twitter.com/StollmeyerEU/status/1128782543585775616 https://twitter.com/JamilSmith/status/1128742138903056385 The Christchurch call is the first time when governments and tech companies have jointly agreed to a set of commitments and ongoing collaboration to make the internet safer. “We can be proud of what we have started with the adoption of the Christchurch Call. We’ve taken practical steps to try and stop what we experienced in Christchurch from happening again” Jacinda Ardern said. “From here, I will work alongside others signed up to the Christchurch Call to bring more partners on board, and develop a range of practical initiatives to ensure the pledge we have made today is delivered”, she added. Digital ministers of the Group of 7 nations, who signed the Christ Church call are also meeting today to discuss an upcoming charter that would cover broader territory than the Christchurch Call on toxic content and tech regulation in general. https://twitter.com/LeoVaradkar/status/1128771650391093250 https://twitter.com/adamihad/status/1128298612734201858 Although critics praised the call for its use of existing laws against extremist and terrorist content and its insistence that legal and platform content regulation measures comply with intelligent human rights law, they also listed certain shortcomings. https://twitter.com/mediamorphis/status/1128773160357302273 Read the full text of Christchurch call here. Google and Facebook working hard to clean image after the media backlash from the Christchurch terrorist attack Facebook tightens rules around live streaming in response to the Christchurch terror attack How social media enabled and amplified the Christchurch terrorist attack
Read more
  • 0
  • 0
  • 14025

article-image-ngi0-consortium-to-award-grants-worth-5-6-million-euro-to-open-internet-projects-that-promote-inclusivity-privacy-interoperability-and-data-protection
Bhagyashree R
03 Dec 2018
3 min read
Save for later

NGI0 Consortium to award grants worth 5.6 million euro to open internet projects that promote inclusivity, privacy, interoperability, and data protection

Bhagyashree R
03 Dec 2018
3 min read
NLnet Foundation, on Saturday, announced that they are now taking in submissions for project proposals for projects that will deliver “potential break-through contributions to the open internet”. These project will be judged on their technical merits, strategic relevance to the Next Generation Internet and overall value for money. For this, they have created separate themes such as NGI Zero PET and NGI Zero Discovery under which you can list your projects. The foundation will be investing 5.6 million euro on small to medium-size R&D grants towards improving search and discovery and privacy and trust enhancing technologies from 2018 to 2021. They are seeking project proposals between 5.000 and 50.000 euros, with the chances to scale them up if there is proven potential. Deadline for submitting these proposals is February 1st, 2019 12:00 CET. NLnet Foundation supports the open internet and the privacy and security of internet users. The foundation helps independent organizations and people that contribute to an open information society by providing them microgrants, advice, and access to a global network. Next Generation Internet (NGI): Creating open, trustworthy, and reliable internet for all The European Commission launched the NGI initiative in 2016 aiming to make the internet an interoperable platform ecosystem. This future internet will respect human and societal values such as openness, inclusivity, transparency, privacy, cooperation, and protection of data. NGI wants to make the internet more human-centric while also driving the adoption of advanced concepts and methodologies in domains such as artificial intelligence, Internet of Things, interactive technologies, and more. To achieve these goals NLnet has launched projects like NGI Zero Discovery and NGI Zero Privacy and Trust Enhancing Technologies (PET). NGI Zero Discovery aims to provide individual researchers and developers an agile, effective, and low-threshold funding mechanism. This project will help researchers and developers bring in new ideas that contribute to the establishment of the Next Generation Internet. These new projects will be made available as free/libre/open source software. NGI Zero PET is the sister project of NGI Zero Discovery. The objective of this project is to equip people with new technologies that will provide them better privacy. NLnet on their website said these investments are for helping researchers and developers towards creating an open internet: “Trust is one of the key drivers for the Next Generation Internet, and an adequate level of privacy is a non-negotiable requirement for that. We want to assist independent researchers and developers to create powerful new technology and to help them put it in the hands of future generations as building blocks for a fair and democratic society and an open economy that benefits all.” To read more, check out the NLnet Foundation’s official website. The State of Mozilla 2017 report focuses on internet health and user privacy Tim Berners-Lee plans to decentralize the web with ‘Solid’, an open-source project for “personal empowerment through data” Has the EU just ended the internet as we know it?
Read more
  • 0
  • 0
  • 14021
article-image-adobe-firefly-feature-deep-dive
Joseph Labrecque
23 Aug 2023
9 min read
Save for later

Adobe Firefly Feature Deep Dive

Joseph Labrecque
23 Aug 2023
9 min read
Adobe FireflyAdobe Firefly is a new set of generative AI tools which can be accessed via https://firefly.adobe.com/ by anyone with an Adobe ID. To learn more about Firefly… have a look at their FAQ.  Image 1: Adobe FireflyFor more information about Firefly, have a look at the previous articles in this series:       Animating Adobe Firefly Content with Adobe Animate       Exploring Text to Image with Adobe Firefly       Generating Text Effects with Adobe FireflyIn this article, we’ll be exploring some of the more detailed features of Firefly in general. While we will be doing so from the perspective of the text-to-image module, much of what we cover will be applicable to other modules and procedures as well.Before moving on to the visual controls and options… let’s consider accessibility. Here is what Adobe has to say about accessibility within Firefly:Firefly is committed to providing accessible and inclusive features to all individuals, including users working with assistive devices such as speech recognition software and screen readers. Firefly is continuously enhanced to strive to meet the needs of all types of users, including individuals with visual, hearing, cognitive, motor, or other impairments, and is designed to conform to worldwide accessibility standards. -- AdobeYou can use the following keyboard shortcuts across the Firefly interface to navigate and control the software in a non-visual way:       Tab: navigates between user interface controls.       Space/Enter: activates buttons.       Enter: activates links.       Arrow Keys: navigates between options.       Space: selects options.As with most accessibility concerns and practices, these additional controls within Firefly can benefit those users who are not otherwise impaired as well – similar to sight-enabled users making use of captions when watching video-based content.For our exploration of the various additional controls and options within Firefly, we’ll start off with a generated set of images based on a prompt. To review how to achieve this, have a look at the article “Exploring Text to Image with Adobe Firefly”.Choose one of the generated images to work with and hover your mouse across the image to reveal a set of controls.Image 2: Image Overlay OptionsWe will explore each of these options one by one as we continue along with this article.Rating and Feedback OptionsAdobe is very open to feedback with Firefly. One reason is to get general user feedback to improve the experience of using the product… and the other is to influence the generative models so that users receive the output that is expected.Giving a simple thumbs-up or thumbs-down is the most basic level of feedback and is meant to rate the results of your prompt.Image 3: Rating the generated resultsOnce you provide a thumbs-up or thumbs-down… the overlay changes to request additional feedback. You don’t necessarily need to provide more feedback – but clicking on the Feedback button will allow you to go more in-depth in terms of why you provided the initial rating.Image 4: Additional feedback promptClicking the Feedback button will summon a much larger overlay where you can make choices via a checkbox as to why you rated the results the way you did. You also have the option to put a little note in here as well.Image 5: Additional feedback formClicking the Submit Feedback button or the Cancel button will close the overlay and bring you back to the experience.Additionally, there is an option to Report the image to Adobe. This is always a negative action – meaning that you find the results offensive or inappropriate in some way.Image 6: Report promptClicking on the Report option will summon a similar form to that of additional feedback, but the options will, of course, be different.Image 7: Report feedback formHere, you can report via a checkbox and add an optional note as part of the report. Adobe has committed to making sure that violence and things like copyrighted or trademarked characters and such are not generated by Firefly.For instance, if you use a prompt such as “Micky Mouse murdering a construction worker with a chainsaw”… you will receive a message like the following:Image 8: Firefly will not render trademarked characters or violenceWith Adobe is being massively careful in filtering certain words right now… I do hope in the future that users will be able to selectively choose exclusions in place of a general list of censored terms as exists now. While the prompt above is meant to be absurd – there are legitimate artistic reasons for many of the word categories which are currently banned.General Image ControlsThe controls in this section include some of the most used in Firefly at the moment – including the ability to download your generated image.Image 9: Image optionsWe have the following controls exposed, from left to right they are named:       Options       Download       FavoriteOptionsStarting at the left-hand side of this group of controls, we begin with an ellipse that represents Options which, when clicked, will summon a small overlay with additional choices.Image 10: Expanded optionsThe menu that appears includes the following items:1.     Submit to Firefly gallery2.     Use as a reference image3.     Copy to the clipboardLet’s examine each of these in detail.You may have noticed that the main navigation of the Firefly website includes a number of options: Home, Gallery, Favorites, About, and FAQ. The Gallery section contains generated images that users have submitted to be featured on this page.Clicking the Submit to Firefly gallery option will summon a submission overlay through which you can request that your image is included in the Gallery.Image 11: Firefly Gallery submissionSimply read over the details and click Continue or Cancel to return.The second item, Use as reference image, brings up a small overlay that includes the selected image to use as a reference along with a strength slider.Image 12: Reference image sliderMoving the slider to the left will favor the reference image and moving it to the right will favor the raw prompt instead. You must click the Generate button after adjusting the slider to see its effect.The final option is Copy to clipboard – which does exactly as you’d expect. Note that Content Credentials are applied in this case just the same as they are when downloading an image. You can read more about this feature in the Firefly FAQ.DownloadBack up to the set of three controls, the middle option allows you to initiate a Download of the selected image. As Firefly begins preparing the image for download, a small overlay dialog appears.Image 13: Download applies content credentials – similar to the Copy to clipboard optionFirefly applies metadata to any generated image in the form of content credentials and the image download process begins. We’ve covered exactly what this means in previous articles. The image is then downloaded to your local file system.FavoriteClicking the Favorite control will add the generated image to your Firefly Favorites so that you can return to the generated set of images for further manipulation or to download later on.Image 14: Adding a favoriteThe Favorite control works as a toggle. Once you declare a favorite, the heart icon will appear filled and the control will allow you to un-favorite the selected image instead.That covers the main set of controls which overlay the right of your image – but there is a smaller set of controls on the left that we must explore as well.Additional Manipulation OptionsThe alternative set of controls numbers only two – but they are both very powerful. To the left is the Show similar control and to the right is Generative fill.Image 15: Show similar and Generative fill controlsClicking upon the Show similar control will retain the particular, chosen image while regenerating the other three to be more in conformity with the image specified.Image 16: Show similar will refresh the other three imagesAs you can see when comparing the sets of images in the figures above and below… you can have great influence over your set of generated images through this control.Image 17: The original image stays the sameThe final control we will examine in this article is Generative fill. It is located right next to the Show similar control.The generative fill view presents us with a separate view and a number of all-new tools for making selections in order to add or remove content from our images.Image 18: Generative fill brings you to a different view altogetherGenerative fill is actually its own proper procedure in Adobe Firefly… and we’ll explore how to use this feature in full - in the next article! Author BioJoseph Labrecque is a Teaching Assistant Professor, Instructor of Technology, University of Colorado Boulder / Adobe Education Leader / Partner by DesignJoseph is a creative developer, designer, and educator with nearly two decades of experience creating expressive web, desktop, and mobile solutions. He joined the University of Colorado Boulder College of Media, Communication, and Information as faculty with the Department of Advertising, Public Relations, and Media Design in Autumn 2019. His teaching focuses on creative software, digital workflows, user interaction, and design principles and concepts. Before joining the faculty at CU Boulder, he was associated with the University of Denver as adjunct faculty and as a senior interactive software engineer, user interface developer, and digital media designer.Labrecque has authored a number of books and video course publications on design and development technologies, tools, and concepts through publishers which include LinkedIn Learning (Lynda.com), Peachpit Press, and Adobe. He has spoken at large design and technology conferences such as Adobe MAX and for a variety of smaller creative communities. He is also the founder of Fractured Vision Media, LLC; a digital media production studio and distribution vehicle for a variety of creative works.Joseph is an Adobe Education Leader and member of Adobe Partners by Design. He holds a bachelor’s degree in communication from Worcester State University and a master’s degree in digital media studies from the University of Denver.Author of the book: Mastering Adobe Animate 2023 
Read more
  • 0
  • 0
  • 14018

article-image-using-urls-access-nnmi-objects
Packt
24 Dec 2010
14 min read
Save for later

Using URLs to Access NNMi Objects

Packt
24 Dec 2010
14 min read
  HP Network Node Manager 9: Getting Started Manage your network effectively with NNMi Install, customize, and expand NNMi functionality by developing custom features Integrate NNMi with other management tools, such as HP SW Operations Manager, Network Automation, Cisco Works, Business Availability center, UCMDB, and many others Navigate between incidents and maps to reduce troubleshooting time Screenshots and step-by-step instructions to customize NNMi in the way you want Instructions in the book are valid for version 8 as well         Read more about this book       URLs that are listed in this section can be grouped into a few groups (refer to the preceding screenshot): Generic URLs. Workspace-related URLs. Form-related URLs. Menu item-related URLs. As an example, we can provide a quick access to menu items for our team. For example, building a smart-phone application that displays critical events and allows user to run ping, trace route, or other commands for faster troubleshooting (refer to the next diagram). The following section lists these URLs and provides tables of parameters, which can be used for building URLs. Note: If our NNMi is configured to use the https protocol, we need to modify our URLs in this article accordingly, as URLs here are provided for http access. Generic URLs Generic URLs that can be used for obtaining information about NNMi are: View launching URL: To launch the NNMi console, use the following URL:http://<server_name>:<port_number>/nnm/launch?cmd=showMain. If we are passing by using a username and password within the URL, we need to use the following syntax: http://<server_name>:<port_number>/nnm/launch?cmd=showMain&j_username=<account_name>&j_password=<account_password>. To confirm whether NNMi is running: The following URL should be used: http:/<serverName>:<portNumber>/nnm/launch?cmd=isRunning. Using this URL, we will receive a confirmation response on whether or not NNMi is running. Otherwise, a browser error message will be sent saying that the URL is unreachable. Workspace-related URLs This URL group is related to NNMi views. Here is a list of available URLs. Launching view The following table presents the required attributes and their values for launching views. Attributes Values Description Use following URL to show general view: http://<serverName>:<portNumber>/nnm/launch?cmd=showView&objtype=<x> objtype Incident Node Interface IP Address IP Subnet NodeGroup InterfaceGroup Incidents workspace, All Incidents table view. Inventory workspace, Nodes table view. Inventory workspace, Interfaces table view. Inventory workspace, IP Addresses table view. Inventory workspace, IP Subnets table view. Inventory workspace, Node Groups table view. Inventory workspace, Interface Groups table view. Use the following URL to show general view for a specific node group, filtered by node group name, id, or uuid: http://<serverName>:<portNumber>/nnm/launch?cmd=showView&objtype= <x>&nodegroup=<Name> nodegroup nodegroup nodegroupid     nodegroupuuid The Name attribute value of Node Group*. The ID number of node group. This ID is unique across all NNMi database. Use nnmconfigexport.ovpl tool to see ID number of node group. Universally Unique Object ID of the node group. Use nnmconfigexport.ovpl tool to see ID number of node group. Use following URL to show general view for a specific node group, filtered by interface group name or id: http://<serverName>:<portNumber>/nnm/launch?cmd=showView&objtype=<x>&ifgroup=<Name> ifgroup ifgroup ifgroupid     ifgroupduuid The name attribute value of interface group*. The ID number of interface group. This ID is unique across all NNMi database. Use nnmconfigexport.ovpl tool to see id number of interface group. Universally Unique Object ID of the interface group. Use nnmconfigexport.ovpl tool to see id number of interface group. Use following URL to show general view with custom window settings: http://<serverName>:<portNumber>/nnm/launch?cmd=showView&objtype=<x>&menus=<true|false>&newWindow=<true|false>&envattrs=<name1=value>;<name2=value> menus [true|false] True: Show the view menus. False: Hide the view menus. The default value is true. newWindow [true|false] True: To display view in new window. False: To display view within same browser window. Default value is false. envattrs   This is session-specific attributes and they are stored by following syntax: <name=value>.     * If the node group name has spaces in it, use one of the following symbols to represent a "space" character (without brackets): "%20", "+", or " " (space symbol). Consider the following example. To open a window with incidents, which are related to the node group Europe, run the following URL (assuming, that NNMi is installed on node "box1" and uses port number "8080"): http://box1:8080/nnm/launch?cmd=showView&objtype=Incident&nodegroup=Europe Launching an Incident view The following table presents the required attributes and their values for launching incident-specific views. Attributes Values Description Use following URL to show incident view, filtered by specific incident attributes: http://<serverName>:<portNumber>/nnm/launch?cmd=showView&view=<x> view allIncidentsTableView allOpenIncidentsTableView closedKeyIncidentsTableView closedRCIncidentTableView customIncidentTableView incidentsByNatureTableView incidentsByFamilyTableView keyIncidentsByLifecycleStateTableView myIncidentTableView nnm6x7xIncidentTableView nnm6x7xIncidentByCategoryTableView openKeyIncidentsTableView openKeyIncidentsByCategoryTableView openKeyIncidentsByFamilyTableView openKeyIncidentsByPriorityTableView openKeyIncidentsBySeverityTableView openRCIncidentsByCategoryTableView openRCIncidentsByFamilyTableView openRCIncidentsByPriorityTableView openRCIncidentsBySeverityTableView openRCIncidentTableView allRCIncidentTableView RCIncidentsByLifecycleStateTableView serviceImpactIncidentTableView snmpTrapsIncidentTableView snmpTrapsIncidentByFamilyTableView streamCorrelationIncidentTableView unassignedKeyIncidentsTableView unassignedIncidentTableView All incidents All open incidents Closed Key Incidents Closed Root Cause Incidents Custom Incidents Incidents by Correlation Nature Incidents by Family Key Incidents by Lifecycle State My Open Incidents NNM 6.x / 7.x Events NNM 6.x/7.x Event by Category Open Key Incidents Open Key Incidents by Category Open Key Incidents by Family Open Key Incidents by Priority Open Key Incidents by Severity Open Root Cause by Category Open Root Cause by Family Open Root Cause by Priority Open Root Cause by Severity Open Root Cause Incidents Root Cause Incidents Root Cause by Lifecycle State Service Impact Incidents SNMP Traps SNMP Traps by Family Stream Correlation Incidents Unassigned Open Key Incidents Unassigned Root Cause Incidents Use following URL to show Incident view, filtered for specific node group: http://<serverName>:<portNumber>/nnm/launch?cmd=showView&view=<x>&nodegroup_var=<Name> See the preceding table for attribute description. Use following URL to show Incident view with custom window settings: http://<serverName>:<portNumber>/nnm/launch?cmd=showView&view=<x>&menus_var=<true|false>&newWindow=<true|false>&envattrs=<name1=value>;<name2=value> See the preceding table for attribute description.   Launching a Topology Maps Workspace view Here is a list of Topology Maps that can be opened using the following URLs: Consider the following example. Here are the links to open a few groups on NNMi server "box1" (port number 8080): Open Routers node groups: http://box1:8080/nnm/launch?cmd=showNodeGroup&name=Routers Open Networking Infrastructure devices: http://box1:8080/nnm/launch?cmd=showView&objtype= Node&nodegroup=Networking%20Infrastructure%20Devices Launching a Monitoring Workspace view The following table presents the required attributes and their values for launching Monitoring workspace-specific views. Attributes Values Description Use following URL to show monitoring view, filtered by specific attributes: http://<serverName>:<portNumber>/nnm/launch?cmd=showView&view=<x> view criticalComponentHealthTableView criticalInterfaceTableView criticalNodeTableView nonNormalInterfaceTableView nonNormalNodeTableView notRespondingIPAddressTableView nodesByStatusTableView componentHealthByStatusTableView interfacesByStatusTableView interfacesByAdministrativeStateTableView interfacesByOperationalStateTableView IPAddressesByStateTableView interfacePerformanceTableView routerRedundancyGroupsTableView nodeGroupsStatusTableView Critical Component Health Critical Interfaces Critical Nodes Non-Normal Interfaces Non-Normal Nodes Not Responding Addresses Nodes by Status Component Health by Status Interfaces by Status Interfaces by Administrative State Interfaces by Operational State IP Addresses by State Interface Performance Router Redundancy Groups Node Groups Use following URL to show monitoring view, filtered by node group: http://<serverName>:<portNumber>/nnm/launch?cmd=showView&view=<x>&nodegroup=<Name> Refer to the table under the Launching view section for attribute description. Use following URL to show general view, filtered by interface group: http://<serverName>:<portNumber>/nnm/launch?cmd=showView&view=<x>&ifgroup=<Name> Refer to the table under the Launching view section for attribute description. Use following URL to show monitoring view with custom window settings: http://<serverName>:<portNumber>/nnm/launch?cmd=showView&view=<x>&menus=<true|false>&newWindow=<true|false>&envattrs=<name1=value>;<name2=value> Refer to the table under the Launching view section for attribute description.   Launching a Troubleshooting Workspace view Troubleshooting workspace contains the following four different views: Layer 2 Neighbor view Layer 3 Neighbor view Path view Node Group Map view Use the following URL to show the Layer 2 Neighbor view: http://<serverName>:<portNumber>/nnm/launch?cmd=showLayer2Neighbors. Attributes   Values Description Use following URL to show Layer 2 Neighbor view by specified number of hops: http://<serverName>:<portNumber>/nnm/launch?cmd=showLayer2Neighbors&nodename=<x>&hops=<#> nodename nodename The source node's name or IP address hops 1 - 9 Number of hops Use following URL to show Layer 2 Neighbor view with custom window parameters: http://<serverName>:<portNumber>/nnm/launch?cmd=showLayer2Neighbors&menus=<true/false>&newWindow=<true/false>&envattrs=<name1=value>;<name2=value> Refer to the table under the Launching view section for attribute description.   Use the following URL to show Layer 3 Neighbor view: http://<serverName>:<portNumber>/nnm/launch?cmd= showLayer3Neighbors. Attributes Values Description Use following URL to show Layer 3 Neighbor view for specified node by specified number of hops: http://<serverName>:<portNumber>/nnm/launch?cmd=showLayer3Neighbors&nodename=<x>&hops=<#> nodename See the previous table for attribute description. hops See the previous table for attribute description. menus [true|false] Show (true) or hide (false) menu and window toolbar. Default value is "true".   Use the following URL to show Layer 3 Neighbor view with the specified window parameters: http://<serverName>:<portNumber>/nnm/launch?cmd=showLayer3Neighbors&menus=<true/false>&newWindow=<true/false>&envattrs=<name1=value>;<name2=value>. Use the following URL to show Path view: http://<serverName>:<portNumber>/nnm/launch?cmd=showPath. Attributes Values Description Use the following URL to show Path view by specified source and destination nodes: http://<serverName>:<portNumber>/nnm/launch?cmd=showPath&src=<x>&dest=<y> src src Source node's hostname or IPv4 address. dest dest Destination node's hostname or IPv4 address. Use the following URL to show Path view with custom window parameters: http://<serverName>:<portNumber>/nnm/launch?cmd=showPath&menus=<true/false>&newWindow=<true/false>&envattrs=<name1=value>;<name2= value> Attributes Values Description   Use following URL to list nodes belonging to a specified node group: http://<serverName>:<portNumber>/nnm/launch?cmd=showNodeGroup&name=<x> name name The Name attribute value from the Node Group form   Use the following URL to list nodes belonging to a specified node group with custom window parameters: http://<serverName>:<portNumber>/nnm/launch?cmd=showNodeGroup&name=<x>&menus=<true/false>&newWindow=<true/false>&envattrs=<name1=value>;<name2=value>. Launching an Inventory Workspace view The following table presents the required attributes and their values for launching Inventory workspace-specific views. Attributes Values Description Use the following URL to show a view, specified in the view list below in this table: http://<serverName>:<portNumber>/nnm/launch?cmd=showView&view=<x> view allNodesTableView allInterfacesTableView allIPAddressTableView allIPSubnetsTableView allVlansTableView allLayer2ConnectionsTableView nodesByDeviceCategoryTableView interfacesByIfTypeTableView customNodeTableView customInterfaceTableView customIPAddressTableView routerRedundancyGroupsTableView nodeGroupsTableView interfaceGroupsTableView allManagementStationsTableView Nodes Interfaces IP Addresses IP Subnets VLANs Layer 2 Connections Nodes by Device Category Interfaces by IfType Custom Nodes Custom Interfaces Custom IP Addresses Router Redundancy Groups Node Groups Interface Groups Management Stations Use the following URL to show a view, specified in the view list below in this table for specified node group: http://<serverName>:<portNumber>/nnm/launch?cmd=showView&view=<x>&nodegroup=<Name>. Refer to the table under the Launching view section for attribute description. nodegroup nodegroup nodegroupid nodegroupuuid Refer to the table under the Launching view section for attribute description. Use the following URL to show a view, specified in the view list below in this table for specified interface group: http://<serverName>:<portNumber>/nnm/launch?cmd=showView&view=<x>&interfacegroup=<Name> interfacegroup ifgroup ifgroupid ifgroupuuid Refer to the table under the Launching view section for attribute description.   Use the following URL to show a view, specified in the view list above in a table with custom window settings: http://<serverName>:<portNumber>/nnm/launch?cmd=showView&view=<x>&menus= <true/false>&newWindow=<true/false>&envattrs=<name1=value>;<name2=value>. Launching Management Mode Workspace views The following table presents the required attributes and their values for launching Management Mode workspace-specific views: Attributes Values Description Use the following URL to show a view, specified in the view list below in this table: http://<serverName>:<portNumber>/nnm/launch?cmd=showView&view=<x> view allNodesTableView allInterfacesTableView allIPAddressTableView allIPSubnetsTableView allVlansTableView allLayer2ConnectionsTableView nodesByDeviceCategoryTableView interfacesByIfTypeTableView customNodeTableView customInterfaceTableView customIPAddressTableView routerRedundancyGroupsTableView nodeGroupsTableView interfaceGroupsTableView allManagementStationsTableView Nodes Interfaces IP Addresses IP Subnets VLANs Layer 2 Connections Nodes by Device Category Interfaces by IfType Custom Nodes Custom Interfaces Custom IP Addresses Router Redundancy Groups Node Groups Interface Groups Management Stations Use the following URL to show a view, specified in the view list below in this table for specified node group: http://<serverName>:<portNumber>/nnm/launch?cmd=showView&view=<x>&nodegroup=<Name>. Refer to the table under the Launching view section for attribute description. nodegroup nodegroup nodegroupid nodegroupuuid Refer to the table under the Launching view section for attribute description. Use the following URL to show a view, specified in the view list below in this table for specified interface group: http://<serverName>:<portNumber>/nnm/launch?cmd=showView&view= <x>&interfacegroup=<Name> interfacegroup ifgroup ifgroupid ifgroupuuid Refer to the table under the Launching view section for attribute description. Use the following URL to show a view, from the Management Mode workspace, with specified window settings: http://<serverName>:<portNumber>/nnm/launch?cmd=showView&view=<x>&menus=<true/false>&newWindow=<true/false>&envattrs=<name1=value>;<name2=value>. Launching a Configuration Workspace view The following table presents the required attributes and their values for launching the Configuration workspace-specific views: Attributes Values Description Use the following URL to show a view from Configuration workspace for a view specified below: http://<serverName>:<portNumber>/nnm/launch?cmd=showView&view=<x>. view nodeGroupsTableView allNodeGroupMapSettingsTableView interfaceGroupsTableView allManagementStationsTableView ramsServerTableView allURLActionInfosTableView allAccountsTableView allIfTypesTableView allDeviceProfilesTableView Node Groups Node Group Map Settings Interface Groups Management Stations RAMS Servers URL Actions User Accounts and Roles IfTypes Device Profiles   Use the following URL to show a view, from Configuration workspace, with specified window parameters: http://<serverName>:<portNumber>/nnm/launch?cmd=showView&view=<x>&menus=<true/false>&newWindow=<true/false>&envattrs=<name1=value>;<name2=value>.  
Read more
  • 0
  • 0
  • 14016
Modal Close icon
Modal Close icon