Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7019 Articles
article-image-managing-public-and-private-groups
Packt
19 Nov 2014
11 min read
Save for later

Managing public and private groups

Packt
19 Nov 2014
11 min read
In this article by Andrew Mallett, the author of CentOS System Administration Essentials, we will look at how we can manage public and private groups and set quotas. The Red Hat and, therefore, the CentOS user management systems deploy a private user group system. Each user created will also belong to an eponymous primary group; in other words, creating a user bob will also create a group bob, to which the user will be the only member. (For more resources related to this topic, see here.) Linux groups Firstly, we have to understand a little about Linux groups. A user has both a primary group and secondary groups. User ID and group ID (UID/GID) are used with the permission management structure in Linux. Every file in any filesystem will be owned by a user and a group by means of storing the UID and GID in the files metadata. Permissions can be assigned to the user, group, or others. Each user has one UID and GID but belongs to just one group, which is a little restrictive, so users additionally have secondary groups. Users can change their current GID to one from their secondary groups using the /usr/bin/newgrp command, effectively switching their GID. In practice, this is not required and leads us to describing the differences between the users' primary group and secondary groups. When creating a new file, the users UID and their current GID are used to create the ownership of the new file. If a user creates a new file, he/she will be the owner of that file and the file will be group owned by his/her own private group, creating an inherently secure system without the need of user intervention. Secondary groups are used in all other situations when accessing resources that currently exist. Users present all of their secondary groups when accessing a resource. In this way, a file that is readable by the users group but not to others will be accessible to a user whose GID is set to his/her own private group, but the list of secondary groups to which they belong includes the users group. When assessing a user's ID, setting the /usr/bin/id command can be very useful. Run without any options or arguments and the output will display your own associated IDs. In the following screenshot, we can see that the user andrew belongs to only the private user group and has no additional secondary group memberships: $ id We will use the same command but this time we will use the user, u1, as an argument. The output will show the associated IDs of that account; this command can be run as a standard user: $ id u1 From the following screenshot, we can see that the user, u1, has the primary group or GID assigned to the private group u1; however, u1 additionally belongs to the users group. With the current IDs in place for the user u1, any new file created will be group owned by GID 501 (u1), but u1 can access any resource accessible to the users and u1 groups without any additional action on u1's part. From an administrative perspective, we need to make sure we assign the correct secondary IDs to our users. The same cannot be said for the first example that we looked at. The user, andrew, currently belongs only to andrew's private group, so he can only access resources where permissions are set to: Their UID (andrew) Their private GID (andrew) Others The user account andrew does not have access to permissions assigned to the users group in the same way that the user u1 does. Adding users to groups We can now see that the user u1 has the desired access to resources shared with the users groups, but what about andrew? How can we help here? If the user already exists and we need to add him/her to a public group, then we can use the usermod command to add the user to an additional group. When we add andrew to the users group, we will also want to maintain any existing secondary groups' memberships. Run the following command: # usermod -G users andrew If we choose to run the preceding command, then andrew would be added to the users groups but, along with his primary group, this would be his only secondary group membership. In other words, if andrew belongs to multiple secondary groups, the -G option overwrites this group list, which is not a good thing. The command ID can display current secondary groups with the -G option: # id -G andrew If we combine the two commands together, then we can effectively append the users groups to the current group list of andrew. To do this, additionally, we have to translate the spaces in the group list supplied by the command ID into commas: # usermod -G$(id -G andrew | tr ' ' ','),users The commands in the parenthesis are evaluated first. The id command creates a space-separated list of secondary groups, and the tr command will translate the spaces to commas (in this case). The group list we supply to usermod needs to be comma delimited but can use group names or IDs. More simply though, we can use the append option to usermod as shown in the following code example: # usermod -a -G users andrew When creating new users, we can simply specify the secondary groups the user should belong to. We don't need to concern ourselves with the existing group membership: # useradd -G users u4# id u4 From the following output, we can see that the new user, u4, is created and added to the secondary group users. Evaluating private group usage You do not need to use private groups schemes. They are the default, but as with all defaults, we can specify options to modify this. Using the -N option with useradd will not create the private groups and, if not specified, the user's primary group or GID will be the users groups. Let's execute the following commands: # useradd -N u5# id u5 The output is shown in the following screenshot, and we see that the users' primary group is the users group: The only security issue that we may need to contend with is that now, by default, any file created by the user u5 will be group owned by a shared group. Depending on the circumstances, this may be not desirable; however, having all files private to the user by default is no more desirable either. This is up to the administration team deciding which model suits the organizational needs best. Getent The /usr/bin/getent command will display a list of entries, Get Entries. The entries are resolved by Name Service Switch Libraries, which are configured in the /etc/nsswitch.conf file. This file has a list of databases and libraries that will be used to access those databases. For example, we could use the getent passwd command to display all users, or getent group to display all groups. We could extend this though to commands such as getent hosts to display host file entries and getent aliases to display user aliases on the system. The nsswitch.conf file will define the libraries used to access the passwd database. On a standard CentOS system, /etc/passwd is often the only local file, but an enterprise system could include Lightweight Directory Access Protocol (LDAP) modules. We search the /etc/nsswitch file for the passwd database using grep: # grep passwd /etc/nsswitch.conf We can see that on my system, we just use the local files to resolve user names: The getent command is a very useful way to quickly list users or groups on your system, and the output can be filtered or sorted as required with the grep and sort commands. For example, if we want to see all configured groups on our system that start with the letter u and have only one additional character in their names, we can use the following command: # getent group | grep 'u.:' | sort The following screenshot shows this command: Quotas In almost all areas of user management, we have to assign disk space quotas of some description in order to give the responsibility of disk space management back to the user. If we do not, then the user would never know the struggles that we have to face in providing them with unlimited disk space. Allowing the user to see that their space is filling up then may prompt them to carry out a little housekeeping. In Linux, disk quotas are applied to the mount points; if you want to limit a user's space in their home directory, then the /home directory will need to be in its own partition. If it is part of the root filesystem, then a user's space will be restricted to all directories within that partition. Quota restrictions are implemented using tools from the quota package. You can use the yum command to verify that it is installed: $ yum list quota If the output of the command indicates that it is available rather than installed, then install the quota with: # yum install quota Setting quotas My system includes a partition for /home and has the quota package installed. We now need to set the correct mount options for the /home partition. Currently, it does not include quotas. To enable this, we will edit the /etc/fstab file and mount options for the /home partition. The following two mount options should be added to enable journal quotas for a selected partition: usrjquota=aquota.user,jqfmt=vfsv0 The userjquota=aquota.user part specifies the quota file, and jqfmt=vfsv0 specifies the quota format. The line in question is shown in the following screenshot: We have enabled journal-based user quotas as we are using ext4, a journal-based filesystem. User space restriction is checked when writing the journal rather than waiting until the changes are flushed to disk. We also set the format of the journal quotas. To make these settings effective, we can remount the /home partition using the following command: # mount -o remount /home We will now need to initialize the quota database; this was referenced in the mount options as aquota.user and will reside at the root of the partition where quotas are enabled. Enabling quotas on a filesystem may take some time, depending on the amount of data in the filesystem: #quotacheck -muv /home Using these options with the /sbin/quotacheck command, we can set the following options: -m: This indicates not to remount as read-only during an operation -u: This is for user quotas -v: This is the verbose output /home: This is the partition to work with, or use -a for all quota-enabled partitions It may be worth adding the quotacheck commands and options into your crontab to ensure that quotacheck is run perhaps once a day. Even though journal quotas are more reliable than traditional quotas, there is no harm in re-evaluating file space used to ensure that the data maintained is accurate. Quotas can be set with the edquota or setquota command; I prefer the setquota command, but traditionally edquota is taught to new administrators. The /usr/sbin/edquota command takes you into your editor to make the changes, whereas /usr/sbin/setquota sets the quota directly from the command line: # setquota -u u1 20000 25000 0 0 /home The preceding command will set the quota for the user u1. Giving the user a soft limit, just a warning when they exceed 20 M (20 x 1k blocks) and implementing a hard limit of 25 M, where the user cannot save any more data in /home. I have not limited the user u1 with either soft or hard limits to the number of files they may have, just the space they use. The /usr/sbin/repquota command can be used to display disk usage: # repquota -uv /home The output from my system is shown in the following screenshot: Summary The big task for this article was to become more accustomed to the vagaries of CentOS group management and being able to properly differentiate between the primary group and secondary groups of a user. During this process, we took the time to evaluate the use of public and private group schemes and the use of the -N option to disable the user's private group during user creation. It was not long before we found ourselves in the depths of /etc/nsswitch.conf and the getent command (get entries). From here, we got down straight to business implementing user disk limits or quotas. Resources for Article: Further resources on this subject: Installing CentOS [article] Creating a sample C#.NET application [article] Introducing SproutCore [article]
Read more
  • 0
  • 0
  • 1835

article-image-using-leap-motion-controller-arduino
Packt
19 Nov 2014
18 min read
Save for later

Using the Leap Motion Controller with Arduino

Packt
19 Nov 2014
18 min read
This article by Brandon Sanders, the author of the book Mastering Leap Motion, focuses on what he specializes in—hardware. While normal applications are all fine and good, he finds it much more gratifying if a program he writes has an impact in the physical world. (For more resources related to this topic, see here.) One of the most popular hobbyist hardware solutions, as I'm sure you know, is the Arduino. This cute little blue board from Italy brought the power of micro controllers to the masses. Throughout this article, we're going to work on integrating a Leap Motion Controller with an Arduino board via a simplistic program; the end goal is to make the built-in LED on an Arduino board blink either slower or faster depending on how far a user's hand is away from the Leap. While this is a relatively simple task, it's a great way to demonstrate how you can connect something like the Leap to an external piece of hardware. From there, it's only a hop, skip, and jump to control robots and other cool things with the Leap! This project will follow the client-server model of programming: we'll be writing a simple Java server which will be run from a computer, and a C++ client which will run on an Arduino board connected to the computer. The server will be responsible for retrieving Leap Motion input and sending it to the client, while the client will be responsible for making an LED blink based on data received from the server. Before we begin, I'd like to note that you can download the completed (and working) project from GitHub at https://github.com/Mizumi/Mastering-Leap-Motion-Chapter-9-Project-Leapduino. A few things you'll need Before you begin working on this tutorial, there are a few things you're going to need: A computer (for obvious reasons). A Leap Motion Controller. An Arduino of some kind. This tutorial is based around the Uno model, but other similar models like the Mega should work just as well. A USB cable to connect your Arduino to your computer. Optionally, the Eclipse IDE (this tutorial will assume you're using Eclipse for the sake of readability and instruction). Setting up the environment First off, you're going to need a copy of the Leap Motion SDK so that you can add the requisite library jar files and DLLs to the project. If you don't already have it, you can get a copy of the SDK from https://www.developer.leapmotion.com/downloads/. Next, you're going to need the Java Simple Serial Connector (JSSC) library and the Arduino IDE. You can download the library JAR file for JSSC from GitHub at https://github.com/scream3r/java-simple-serial-connector/releases. Once the download completes, extract the JAR file from the downloaded ZIP folder and store it somewhere safe; you'll need it later on in this tutorial. You can then proceed to download the Arduino IDE from their official website at http://arduino.cc/en/Main/Software. If you're on Windows, you will be able to download a Windows installer file which will automagically install the entire IDE on to your computer. On the other hand, Mac and Linux users will need to instead download .zip or .tgz files and then extract them manually, running the executable binary from the extracted folder contents. Setting up the project To set up our project, perform the following steps: The first thing we're going to do is create a new Java project. This can be easily achieved by opening up Eclipse (to reiterate for the third time, this tutorial will assume you're using Eclipse) and heading over to File -> New -> Java Project. You will then be greeted by a project creation wizard, where you'll be prompted to choose a name for the project (I used Leapduino). Click on the Finish button when you're done. My current development environment is based around the Eclipse IDE for Java Developers, which can be found at http://www.eclipse.org/downloads. The instructions that follow will use Eclipse nomenclature and jargon, but they will still be usable if you're using something else (like NetBeans). Once the project is created, navigate to it in the Package Explorer window. You'll want to go ahead and perform the following actions: Create a new package for the project by right-clicking on the src folder for your project in the Package Explorer and then navigating to New | Package in the resulting tooltip. You can name it whatever you like; I personally called mine com.mechakana.tutorials. You'll now want to add three files to our newly-created package: Leapduino.java, LeapduinoListener.java, and RS232Protocol.java. To create a new file, simply right-click on the package and then navigate to New | Class. Create a new folder in your project by right-clicking on the project name in the Package Explorer and then navigating to New | Folder in the resulting tooltip. For the purposes of this tutorial, please name it Leapduino. Now add one file to your newly created folder: Leapduino.ino. This file will contain all of the code that we're going to upload to the Arduino. With all of our files created, we need to add the libraries to the project. Go ahead and create a new folder at the root directory of your project, called lib. Within the lib folder, you'll want to place the jssc.jar file that you downloaded earlier, along with the LeapJava.jar file from the Leap Motion SDK. Then, you will want to add the appropriate Leap.dll and LeapJava.dll files for your platform to the root of your project. Finally, you'll need to modify your Java build path to link the LeapJava.jar and jssc.jar files to your project. This can be achieved by right-clicking on your project in the Package Explorer (within Eclipse) and navigating to Build Path… | Configure Build Path…. From there, go to the Libraries tab and click on Add JARs…, selecting the two aforementioned JAR files (LeapJava.jar and jssc.jar). When you're done, your project should look similar to the following screenshot: And you're done; now to write some code! Writing the Java side of things With everything set up and ready to go, we can start writing some code. First off, we're going to write the RS232Protocol class, which will allow our application to communicate with any Arduino board connected to the computer via a serial (RS-232) connection. This is where the JSSC library will come into play, allowing us to quickly and easily write code that would otherwise be quite lengthy (and not fun). Fun fact RS-232 is a standard for serial communications and transmission of data. There was a time when it was a common feature on a personal computer, used for modems, printers, mice, hard drives, and so on. With time, though, the Universal Serial Bus (USB) technology replaced RS-232 for many of those roles. Despite this, today's industrial machines, scientific equipment and (of course) robots still make heavy usage of this protocol due to its light weight and ease of use; the Arduino is no exception! Go ahead and open up the RS232Protocol.java file which we created earlier, and enter the following: package com.mechakana.tutorials; import jssc.SerialPort; import jssc.SerialPortEvent; import jssc.SerialPortEventListener; import jssc.SerialPortException; public class RS232Protocol { //Serial port we're manipulating. private SerialPort port; //Class: RS232Listener public class RS232Listener implements SerialPortEventListener {    public void serialEvent(SerialPortEvent event)    {      //Check if data is available.      if (event.isRXCHAR() && event.getEventValue() > 0)      {        try        {          int bytesCount = event.getEventValue();          System.out.print(port.readString(bytesCount));        }                 catch (SerialPortException e) { e.printStackTrace(); }      }    } } //Member Function: connect public void connect(String newAddress) {    try    {      //Set up a connection.      port = new SerialPort(newAddress);         //Open the new port and set its parameters.      port.openPort();      port.setParams(38400, 8, 1, 0);               //Attach our event listener.      port.addEventListener(new RS232Listener());    }     catch (SerialPortException e) { e.printStackTrace(); } } //Member Function: disconnect public void disconnect() {    try { port.closePort(); }     catch (SerialPortException e) { e.printStackTrace(); } } //Member Function: write public void write(String text) {    try { port.writeBytes(text.getBytes()); }     catch (SerialPortException e) { e.printStackTrace(); } } } All in all, RS232Protocol is a simple class—there really isn't a whole lot to talk about here! However, I'd love to point your attention to one interesting part of the class: public class RS232Listener implements SerialPortEventListener { public void serialEvent(SerialPortEvent event) { /*code*/ } } You might have found it rather odd that we didn't create a function for reading from the serial port—we only created a function for writing to it. This is because we've opted to utilize an event listener, the nested RS232Listener class. Under normal operating conditions, this class's serialEvent function will be called and executed every single time new information is received from the port. When this happens, the function will print all of the incoming data out to the user's screen. Isn't that nifty? Moving on, our next class is a familiar one—LeapduinoListener, a simple Listener implementation. This class represents the meat of our program, receiving Leap Motion tracking data and then sending it over our serial port to the connected Arduino. Go ahead and open up LeapduinoListener.java and enter the following code: package com.mechakana.tutorials; import com.leapmotion.leap.*; public class LeapduinoListener extends Listener {   //Serial port that we'll be using to communicate with the Arduino. private RS232Protocol serial; //Constructor public LeapduinoListener(RS232Protocol serial) {    this.serial = serial; } //Member Function: onInit public void onInit(Controller controller) {    System.out.println("Initialized"); } //Member Function: onConnect public void onConnect(Controller controller) {    System.out.println("Connected"); } //Member Function: onDisconnect public void onDisconnect(Controller controller) {    System.out.println("Disconnected"); } //Member Function: onExit public void onExit(Controller controller) {    System.out.println("Exited"); } //Member Function: onFrame public void onFrame(Controller controller) {    //Get the most recent frame.    Frame frame = controller.frame();    //Verify a hand is in view.    if (frame.hands().count() > 0)    {      //Get some hand tracking data.      int hand = (int) (frame.hands().frontmost().palmPosition().getY());      //Send the hand pitch to the Arduino.      serial.write(String.valueOf(hand));      //Give the Arduino some time to process our data.      try { Thread.sleep(30); }      catch (InterruptedException e) { e.printStackTrace(); }    } } } In this class, we've got the basic Leap Motion API onInit, onConnect, onDisconnect, onExit, and onFrame functions. Our onFrame function is fairly straightforward: we get the most recent frame, verify a hand is within view, retrieve its y axis coordinates (height from the Leap Motion Controller) and then send it off to the Arduino via our instance of the RS232Protocol class (which gets assigned during initialization). The remaining functions simply print text out to the console telling us when the Leap has initialized, connected, disconnected, and exited (respectively). And now, for our final class on the Java side of things: Leapduino! This class is a super basic main class that simply initializes the RS232Protocol class and the LeapduinoListener—that's it! Without further ado, go on ahead and open up Leapduino.java and enter the following code: package com.mechakana.tutorials; import com.leapmotion.leap.Controller; public class Leapduino { //Main public static final void main(String args[]) {      //Initialize serial communications.    RS232Protocol serial = new RS232Protocol();    serial.connect("COM4");    //Initialize the Leapduino listener.    LeapduinoListener leap = new LeapduinoListener(serial);    Controller controller = new Controller();    controller.addListener(leap); } } Like all of the classes so far, there isn't a whole lot to say here. That said, there is one line that you must absolutely be aware of, since it can change depending on how you're Arduino is connected: serial.connect("COM4"); Depending on which port Windows chose for your Arduino when it connected to your computer (more on that next), you will need to modify the COM4 value in the above line to match the port your Arduino is on. Examples of values you'll probable use are COM3, COM4, and COM5. And with that, the Java side of things is complete. If you run this project right now, most likely all you'll see will be two lines of output: Initialized and Connected. If you want to see anything else happen, you'll need to move on to the next section and get the Arduino side of things working. Writing the Arduino side of things With our Java coding done, it's time to write some good-old C++ for the Arduino. If you were able to use the Windows installer for Arduino, simply navigate to the Leapduino.ino file in your Eclipse project explorer and double click on it. If you had to extract the entire Arduino IDE and store it somewhere instead of running a simple Windows installer, navigate to it and launch the Arduino.exe file. From there, select File | Open, navigate to the Leapduino.ino file on your computer and double click on it. You will now be presented with a screen similar to the one here: This is the wonderful Arduino IDE—a minimalistic and straightforward text editor and compiler for the Arduino microcontrollers. On the top left of the IDE, you'll find two circular buttons: the check mark verifies (compiles) your code to make sure it works, and the arrow deploys your code to the Arduino board connected to your computer. On the bottom of the IDE, you'll find the compiler output console (the black box), and on the very bottom right you'll see a line of text telling you which Arduino model is connected to your computer, and on what port (I have an Arduino Uno on COM4 in the preceding screenshot). As is typical for many IDEs and text editors, the big white area in the middle is where your code will go. So without further ado, let's get started with writing some code! Input all of the text shown here into the Arduino IDE: //Most Arduino boards have an LED pre-wired to pin 13. int led = 13; //Current LED state. LOW is off and HIGH is on. int ledState = LOW; //Blink rate in milliseconds. long blinkRate = 500; //Last time the LED was updated. long previousTime = 0; //Function: setup void setup() { //Initialize the built-in LED (assuming the Arduino board has one) pinMode(led, OUTPUT); //Start a serial connection at a baud rate of 38,400. Serial.begin(38400); } //Function: loop void loop() { //Get the current system time in milliseconds. unsigned long currentTime = millis(); //Check if it's time to toggle the LED on or off. if (currentTime - previousTime >= blinkRate) {    previousTime = currentTime;       if (ledState == LOW) ledState = HIGH;    else ledState = LOW;       digitalWrite(led, ledState); } //Check if there is serial data available. if (Serial.available()) {    //Wait for all data to arrive.    delay(20);       //Our data.    String data = "";       //Iterate over all of the available data and compound it into      a string.    while (Serial.available())      data += (char) (Serial.read());       //Set the blink rate based on our newly-read data.    blinkRate = abs(data.toInt() * 2);       //A blink rate lower than 30 milliseconds won't really be      perceptable by a human.    if (blinkRate < 30) blinkRate = 30;       //Echo the data.    Serial.println("Leapduino Client Received:");    Serial.println("Raw Leap Data: " + data + " | Blink Rate (MS):      " + blinkRate); } } Now, let's go over the contents. The first few lines are basic global variables, which we'll be using throughout the program (the comments do a good job of describing them, so we won't go into much detail here). The first function, setup, is an Arduino's equivalent of a constructor; it's called only once, when the Arduino is first turned on. Within the setup function, we initialize the built-in LED (most Arduino boards have an LED pre-wired to pin 13) on the board. We then initialize serial communications at a baud rate of 38,400 bits per second—this will allow our board to communicate with the computer later on. Fun fact The baud rate (abbreviated as Bd in some diagrams) is the unit for symbol rate or modulation rate in symbols or pulses per second. Simply put, on serial ports, the baud rate controls how many bits a serial port can send per second—the higher the number, the faster a serial port can communicate. The question is, why don't we set a ridiculously high rate? Well, the higher you go with the baud rate, the more likely it is for there to be data loss—and we all know data loss just isn't good. For many applications, though, a baud rate of 9,600 to 38,400 bits per second is sufficient. Moving on to the second function, loop is the main function in any Arduino program, which is repeatedly called while the Arduino is turned on. Due to this functionality, many programs will treat any code within this function as if it were inside a while (true) loop. In loop, we start off by getting the current system time (in milliseconds) and then comparing it to our ideal blink rate for the LED. If the time elapsed since our last blink exceeds the ideal blink rate, we'll go ahead and toggle the LED on or off accordingly. We then proceed to check if any data has been received over the serial port. If it has, we'll proceed to wait for a brief period of time, 20 milliseconds, to make sure all data has been received. At that point, our code will proceed to read in all of the data, parse it for an integer (which will be our new blink rate), and then echo the data back out to the serial port for diagnostics purposes. As you can see, an Arduino program (or sketch, as they are formally known) is quite simple. Why don't we test it out? Deploying and testing the application With all of the code written, it's time to deploy the Arduino side of things to the, well, Arduino. The first step is to simply open up your Leapduino.ino file in the Arduino IDE. Once that's done, navigate to Tools | Board and select the appropriate option for your Arduino board. In my case, it's an Arduino Uno. At this point, you'll want to verify that you have an Arduino connected to your computer via a USB cable—after all, we can't deploy to thin air! At this point, once everything is ready, simply hit the Deploy button in the top-left of the IDE, as seen here: If all goes well, you'll see the following output in the console after 15 or so seconds: And with that, your Arduino is ready to go! How about we test it out? Keeping your Arduino plugged into your computer, go on over to Eclipse and run the project we just made. Once it's running, try moving your hand up and down over your Leap Motion controller; if all goes well, you'll see the following output from within the console in Eclipse: All of that data is coming directly from the Arduino, not your Java program; isn't that cool? Now, take a look at your Arduino while you're doing this; you should notice that the built-in LED (circled in the following image, labelled L on the board itself) will begin to blink slower or faster depending on how close your hand gets to the Leap. Circled in red: the built-in L LED on an Arduino Uno, wired to pin 13 by default. With this, you've created a simple Leap Motion application for use with an Arduino. From here, you could go on to make an Arduino-controlled robotic arm driven by coordinates from the Leap, or maybe an interactive light show. The possibilities are endless, and this is just the (albeit extremely, extremely simple) tip of the iceberg. Summary In this article, you had a lengthy look at some things you can do with the Leap Motion Controller and hardware such as Arduino. If you have any questions, I encourage you to contact me directly at brandon@mechakana.com. You can also visit my website, http://www.mechakana.com, for more technological goodies and tutorials. Resources for Article: Further resources on this subject: Major SDK components [Article] 2D Twin-stick Shooter [Article] What's Your Input? [Article]
Read more
  • 0
  • 0
  • 36076

Packt
19 Nov 2014
25 min read
Save for later

A Peek Under the Hood – Facts, Types, and Providers

Packt
19 Nov 2014
25 min read
This article is written by Felix Frank, the author of Puppet Essentials. Before you are introduced to the missing language concepts that you will need to use Puppet effectively for bigger projects, there is some background that we should cover first. Don't worry, it won't be all dry theory—most of the important parts of Puppet are relevant to your daily business. (For more resources related to this topic, see here.) These elementary topics will be thoroughly explored in the following sections: Summarizing systems with Facter Understanding the type system Substantiating the model with providers Putting it all together Summarizing systems with Facter Configuration management is quite a dynamic problem. In other words, the systems that need configuration are mostly moving targets. In some situations, system administrators or operators get lucky and work with large quantities of 100 percent uniform hardware and software. In most cases, however, mostly, the landscape of servers and other computing nodes is rather heterogeneous, at least in subtle ways. Even in unified networks, there are likely multiple generations of machines, with small or larger differences required for their respective configurations. For example, a common task for Puppet is to handle the configuration of system monitoring. Your business logic will likely dictate warning thresholds for gauges such as the system load value. However, those thresholds can rarely be static. On a two-processor virtual machine, a system load of 10 represents a crippling overload, while the same value can be absolutely acceptable for a busy DBMS server that has cutting-edge hardware of the largest dimensions. Another important factor can be software platforms. Your infrastructure might span multiple distributions of Linux, or alternate operating systems such as BSD, Solaris, or Windows, each with different ways of handling certain scenarios. Imagine, for example, that you want Puppet to manage some content of the fstab file. On your rare Solaris system, you would have to make sure that Puppet targets the /etc/vfstab file instead of /etc/fstab. It is usually not a good idea to interact with the fstab file in your manifest directly. This example will be rounded off in the section concerning providers. Puppet strives to present you with a unified way of managing all of your infrastructure. It obviously needs a means to allow your manifests to adapt to different kinds of circumstances on the agent machines. This includes their operating system, hardware layout, and many other details. Keep in mind that generally, the manifests have to be compiled on the master machine. There are several conceivable ways to implement a solution for this particular problem. A direct approach would be a language construct that allows the master to send a piece of shell script (or other code) to the agent and receive its output in return. The following is pseudocode however; there are no backtick expressions in the Puppet DSL: if `grep -c ^processor /proc/cpuinfo` > 2 {   $load_warning = 4}else {   $load_warning = 2} This solution would be powerful but expensive. The master would need to call back to the agent whenever the compilation process encounters such an expression. Writing manifests that were able to cope if such a command had returned an error code would be strenuous, and Puppet would likely end up resembling a quirky scripting engine. When using puppet apply instead of the master, such a feature would pose less of a problem, and it is indeed available in the form of the generate function, which works just like the backticks in the pseudocode mentioned previously. The commands are always run on the compiling node though, so this is less useful with an agent/master than apply. Puppet uses a different approach. It relies on a secondary system called Facter, which has the sole purpose of examining the machine on which it is run. It serves a list of well-known variable names and values, all according to the system on which it runs. For example, an actual Puppet manifest that needs to form a condition upon the number of processors on the agent will use this expression: if $processorcount > 4 { … } Facter's variables are called facts, and processorcount is such a fact. The fact values are gathered by the agent just before it requests its catalog from the master. All fact names and values are transferred to the master as part of the request. They are available in the manifest as variables. Facts are available to manifests that are used with puppet apply too, of course. You can test this very simply: puppet apply -e 'notify { "I am $fqdn and have $processorcount CPUs": }' Accessing and using fact values You have already seen an example use of the processorcount fact. In the manifest, each fact value is available as a global variable value. That is why you can just use the $processorcount expression where you need it. You will often see conventional uses such as $::processorcount or $::ipaddress. Prefixing the fact name with double colons was a good idea in older Puppet versions before 3.0. The official Style Guide at https://docs.puppetlabs.com/guides/style_guide.html#namespacing-variables is outdated in this regard and still recommends this. The prefix is no longer necessary. Some helpful facts have already been mentioned. The processorcount fact might play a role for your configuration. When configuring some services, you will want to use the machine's ipaddress value in a configuration file or as an argument value: file {   '/etc/mysql/conf.d/bind-address':       ensure => 'file',       mode  => '644',       content => "[mysqld]\nbind-address=$ipaddress\n",} Besides the hostname, your manifest can also make use of the fully qualified domain name (FQDN) of the agent machine. The agent will use the value of its fqdn fact as the name of its certificate (clientcert) by default. The master receives both these values. Note that the agent can override the fqdn value to any name, whereas the clientcert value is tied to the signed certificate that the agent uses. Sometimes, you will want the master to pass sensitive information to individual nodes. The manifest must identify the agent by its clientcert fact and never use fqdn or hostname instead, for the reason mentioned. An examples is shown in the following code: file {   '/etc/my-secret':      ensure => 'file',       mode   => '600',       owner => 'root',       source =>           "puppet:///modules/secrets/$clientcert/key",} There is a whole group of facts to describe the operating system. Each fact is useful in different situations. The operatingsystem fact takes values such as Debian or CentOS: if $operatingsystem != 'Ubuntu' {   package {       'avahi-daemon':           ensure => absent   }} If your manifest will behave identical for RHEL, CentOS, and Fedora (but not on Debian and Ubuntu), you will make use of the osfamily fact instead: if $osfamily == 'RedHat' {   $kernel_package = 'kernel'} The operatingsystemrelease fact allows you to tailor your manifests to differences between the versions of your OS: if $operatingsystem == 'Debian' {   if versioncmp($operatingsystemrelease, '7.0') >= 0 {       $ssh_ecdsa_support = true   }} Facts such as macaddress, the different SSH host keys, fingerprints, and others make it easy to use Puppet for keeping inventory of your hardware. There is a slew of other useful facts. Of course, the collection will not suit every possible need of every user out there. That is why Facter comes readily extendible. Extending Facter with custom facts Technically, nothing is stopping you from adding your own fact code right next to the core facts, either by maintaining your own Facter package, or even deploying the Ruby code files to your agents directly through Puppet management. However, Puppet offers a much more convenient alternative in the form of custom facts. For now, just create a Ruby file at /etc/puppet/modules/hello_world/lib/facter/hello.rb on the master machine. Puppet will recognize this as a custom fact of the name hello. The inner workings of Facter are very straightforward and goal oriented. There is one block of Ruby code for each fact, and the return value of the block becomes the fact value. Many facts are self-sufficient, but others will rely on the values of one or more basic facts. For example, the method for determining the IP address(es) of the local machine is highly dependent upon the operating system. The hello fact is very simple though: Facter.add(:hello) dosetcode { "Hello, world!" }end The return value of the setcode block is the string, Hello, world!, and you can use this fact as $hello in a Puppet manifest. Before Facter Version 2.0, each fact had a string value. If a code block returns another value, such as an array or hash, Facter 1.x will convert it to a string. The result is not useful in many cases. For this historic reason, there are facts such as ipaddress_eth0 and ipaddress_lo, instead of (or in addition to) a proper hash structure with interface names and addresses. It is important for the pluginsync option to be enabled on the agent side. This has been the default for a long time and should not require any customization. The agent will synchronize all custom facts whenever checking in to the master. They are permanently available on the agent machine after that. You can then retrieve the hello fact from the command line using facter -p hello. By just invoking facter without an argument, you request a list of all fact names and values. When testing your custom facts from the command line, you need to invoke facter with the -p or --puppet option. Puppet itself will always include the custom facts. This article will not cover all aspects of Facter's API, but there is one facility that is quite essential. Many of your custom facts will only be useful on Unix-like systems, and others will only be useful on your Windows boxen. You can retrieve such values using a construct like the following: if Facter.value(:kernel) != "windows"nilelse# actual fact code hereend This would be quite tedious and repetitive though. Instead, you can invoke the confine method within the Facter.add(name) { … } block: Facter.add(:msvs_version) doconfine :kernel => :windowssetcode do   # …endend You can confine a fact to several alternative values as well: confine :kernel => [ :linux, :sunos ] Finally, if a fact does make sense in different circumstances, but requires drastically different code in each respective case, you can add the same fact several times, each with a different set of confine values. Core facts such as ipaddress use this often: Facter.add(:ipaddress) doconfine :kernel => :linux…endFacter.add(:ipaddress) doconfine :kernel => %w{FreeBSD OpenBSD Darwin DragonFly}…end… You can confine facts based on any combination of other facts, not just kernel. It is a very popular choice, though. The operatingsystem or osfamily fact can be more appropriate in certain situations. Technically, you can even confine some of your facts to certain processorcount values and so forth. Simplifying things using external facts If writing and maintaining Ruby code is not desirable in your team for any reason, you might prefer to use an alternative that allows shell scripts, or really any kind of programming language, or even static data with no programming involved at all. Facter allows this in the form of external facts. Creating an external fact is similar to the process for regular custom facts, with the following distinctions: Facts are produced by standalone executables or files with static data, which the agent must find in /etc/facter/facts.d The data is not just a string value, but an arbitrary number of key=value pairs instead The data need not use the ini file notation style—the key/value pairs can also be in the YAML or JSON format. The following external facts hold the same data: # site-facts.txtworkgroup=CT4Site2domain_psk=nm56DxLp% The facts can be written in the YAML format in the following way: # site-facts.yamlworkgroup: CT4Site2domain_psk: nm56DxLp% In the JSON format, facts can be written as follows: # site-facts.json{ 'workgroup': 'CT4Site2', 'domain_psk': 'nm56DxLp%' } The deployment of the external facts works simply through file resources in your Puppet manifest: file {   '/etc/facter/facts.d/site-facts.yaml':       ensure => 'file',       source => 'puppet:///…',} With newer versions of Puppet and Facter, external facts will be automatically synchronized just like custom fact, if they are found in facts.d/* in any module (for example, /etc/puppet/modules/hello_world/facts.d/hello.sh). This is not only more convenient, but has a large benefit: when Puppet must fetch an external fact through a file resource instead, its fact value(s) are not yet available while the catalog is being compiled. The pluginsync mechanism, on the other hand, makes sure that all synced facts are available before manifest compilation starts. When facts are not static and cannot be placed in a txt or YAML file, you can make the file executable instead. It will usually be a shell script, but the implementation is of no consequence; it is just important that properly formatted data is written to the standard output. You can simplify the hello fact this way, in /etc/puppet/modules/hello_world/facts.d/hello: #!/bin/sh echo hello=Hello, world\! For executable facts, the ini styled key=value format is the only supported one. YAML or JSON are not eligible in this context. Goals of Facter The whole structure and philosophy of Facter serves the goal of allowing for platform-agnostic usage and development. The same collection of facts (roughly) is available on all supported platforms. This allows Puppet users to keep a coherent development style through manifests for all those different systems. Facter forms a layer of abstraction over the characteristics of both hardware and software. It is an important piece of Puppet's platform-independent architecture. Another piece that was mentioned before is the type and provider subsystem. Types and providers are explored in greater detail in the following sections. Understanding the type system Each resource represents a piece of state on the agent system. It has a resource type, a name (or a title), and a list of attributes. An attribute can either be property or parameter. Between the two of them, properties represent distinct pieces of state, and parameters merely influence Puppet's actions upon the property values. Let's examine resource types in more detail and understand their inner workings. This is not only important when extending Puppet with resource types of your own. It also helps you anticipate the action that Puppet will take, given your manifest, and get a better understanding of both the master and the agent. First, we take a closer look at the operational structure of Puppet, with its pieces and phases. The agent performs all its work in discreet transactions. A transaction is started every time under any of the following circumstances: The background agent process activates and checks in to the master An agent process is started with the --onetime or --test option A local manifest is compiled using puppet apply The transaction always passes several stages: Gathering fact values to form the actual catalog request. Receiving the compiled catalog from the master. Prefetching of current resource states. Validation of the catalog's content. Synchronization of the system with the property values from the catalog. Facter was explained in the previous section. The resource types become important during compilation and then throughout the rest of the agent transaction. The master loads all resource types to perform some basic checking—it basically makes sure that the types of resources it finds in the manifests do exist and that the attribute names fit the respective type. The resource type's life cycle on the agent side Once the compilation has succeeded, the master hands out the catalog and the agent enters the catalog validation phase. Each resource type can define some Ruby methods that ensure that the passed values make sense. This happens on two levels of granularity: each attribute can validate its input value, and then the resource as a whole can be checked for consistency. One example for attribute value validation can be found in the ssh_authorized_key resource type. A resource of this type fails if its key value contains a whitespace character, because SSH keys cannot comprise multiple strings. Validation of whole resources happens with the cron type for example. It makes sure that the time fields make sense together. The following resource would not pass, because special times such as @midgnight cannot be combined with numeric fields: cron {   'invalid-resource':       command => 'rm -rf /',       special => 'midnight',       weekday => [ '2', '5' ],} Another task during this phase is the transformation of input values to more suitable internal representations. The resource type code refers to this as a munge action. Typical examples of munging are the removal of leading and trailing whitespace from string values, or the conversion of array values to an appropriate string format—this can be a comma-separated list, but for search paths, the separator should be a colon instead. Other kinds of values will use different representations. Next up is the prefetching phase. Some resource types allow the agent to create an internal list of resource instances that are present on the system. For example, this is possible (and makes sense) for installed packages—Puppet can just invoke the package manager to produce the list. For other types, such as file, this would not be prudent. Creating a list of all reachable paths in the whole filesystem can be arbitrarily expensive, depending on the system on which the agent is running. Finally, the agent starts walking its internal graph of interdependent resources. Each resource is brought in sync if necessary. This happens separately for each individual property, for the most part. The ensure property, for types that support it, is a notable exception. It is expected to manage all other properties on its own—when a resource is changed from absent to present through its ensure property (in other words, the resource is getting newly created), this action should bring all other properties in sync as well. There are some notable aspects of the whole agent process. For one, attributes are handled independently. Each can define its own methods for the different phases. There are quite a number of hooks, which allow a resource type author to add a lot of flexibility to the model. For aspiring type authors, skimming through the core types can be quite inspirational. You will be familiar with many attributes, through using them in your manifests and studying their hooks will offer quite some insight. It is also worth noting that the whole validation process is performed by the agent, not the master. This is beneficial in terms of performance. The master saves a lot of work, which gets distributed to the network of agents (which scales with your needs automatically). Substantiating the model with providers At the start of this article, you learned about Facter and how it works like a layer of abstraction over the supported platforms. This unified information base is one of Puppet's most important means to achieve its goal of operating system independence. Another one is the DSL, of course. Finally, Puppet also needs a method to transparently adapt its behavior to the respective platform on which each agent runs. In other words, depending on the characteristics of the computing environment, the agent needs to switch between different implementations for its resources. This is not unlike object-oriented programming—the type system provides a unified interface, like an abstract base class. The programmer need not worry what specific class is being referenced, as long as it correctly implements all required methods. In this analogy, Puppet's providers are the concrete classes that implement the abstract interface. For a practical example, look at package management. Different flavors of UNIX-like operating systems have their own implementation. The most prevalent Puppet platforms use apt and yum, respectively, but can (and sometimes must) also manage their packages through dpkg and rpm. Other platforms use tools such as emerge, zypper, fink, and a slew of other things. There are even packages that exist apart from the operating system software base, handled through gem, pip, and other language-specific package management tools. For each of these management tools, there is a provider for the package type. Many of these tools allow the same set of operations—install and uninstall a package and update a package to a specific version. The latter is not universally possible though. For example, dpkg can only ever install the local package that is specified on the command line, with no other version to choose. There are also some distinct features that are unique to specific tools, or supported by only a few. Some management systems can hold packages at specific versions. Some use different states for uninstalled versus purged packages. Some have a notion of virtual packages. There are some more examples. Because of this potential diversity (which is not limited to package management systems), Puppet providers can opt for features. The set of features is resource type specific. All providers for a type can support one or more of the same group of features. For the package type, there are features such as versionable, purgeable, holdable, and so forth. You can set ensure => purged on any package resource like so: package {   'haproxy':       ensure => 'purged'} However, if you are managing the HAproxy package through rpm, Puppet will fail to make any sense of that, because rpm has no notion of a purged state, and therefore the purgeable feature is missing from the rpm provider. Trying to use an unsupported feature will usually produce an error message. Some attributes (such as install_options) might just get ignored by Puppet instead. The official documentation on the Puppet Labs website holds a complete list of the core resource types and all their built-in providers, along with the respective feature matrices. It is very easy to find suitable providers and their capabilities; the documentation is at https://docs.puppetlabs.com/references/latest/type.html. Providerless resource types There are some resource types that use no providers, but they are rare among the core types. Most of the interesting management tasks that Puppet makes easy just work differently among operating systems, and providers enable this in a most elegant fashion. Even for straightforward tasks that are the same on all platforms, there might be a provider. For example, there is a host type to manage entries in the /etc/hosts file. Its syntax is universal, so the code can technically just be implemented in the type. However, there are actual abstract base classes for certain kinds of providers in the Puppet code base. One of them makes it very easy to build providers that edit files if those files consist of single-line records with ordered fields. Therefore, it makes sense to implement a provider for the host type and base it on this provider class. For the curious, this is what a host resource looks like: host { 'puppet':   ip => '10.144.12.100',   host_aliases => [ 'puppet.example.net', 'master' ]} Summarizing types and providers Puppet's resource types and their providers work together to form a solid abstraction layer over software configuration details. The type system is an extendable basis for Puppet's powerful DSL. It forms an elaborate interface for the polymorphous provider layer. The providers flexibly implement the actual management actions that Puppet is supposed to perform. They map the necessary synchronization steps to commands and system interactions. Many providers cannot satisfy every nuance that the resource type models. The feature system takes care of these disparities in a transparent fashion. Putting it all together Reading this far, you might have gotten the impression that this article is a rather odd mix of topics. While types and providers do belong closely together, the whole introduction to Facter might seem out of place in their context. This is deceptive however: facts do play a vital part in the type/provider structure. They are essential for Puppet to make good choices among providers. Let's look at an example from the Extending Facter with custom facts section once more. It was about fstab entries and the difference of Solaris, where those are found in /etc/vfstab instead of /etc/fstab. That section suggested a manifest that adapts according to a fact value. As you can imagine now, Puppet has a resource type to manage fstab content: the mount type. However, for the small deviation of a different file path, there is no dedicated mount provider for Solaris. There is actually just one provider for all platforms, but on Solaris, it behaves differently. It does this by resolving Facter's osfamily value. The following code example was adapted from the actual provider code: case Facter.value(:osfamily)when "Solaris"fstab = "/etc/vfstab"elsefstab = "/etc/fstab"end In other cases, Puppet should use thoroughly different providers on different platforms, though. Package management is a classic example. On a Red Hat-like platform, you will want Puppet to use the yum provider in virtually all cases. It can be sensible to use rpm, and even apt might be available. However, if you tell Puppet to make sure a package is installed, you expect it to install it using yum if necessary. This is obviously a common theme. Certain management tasks need to be performed in different environments, with very different toolchains. In such cases, it is quite clear which provider would be best suited. To make this happen, a provider can declare itself the default if a condition is met. In the case of yum, it is the following: defaultfor :operatingsystem => [:fedora, :centos, :redhat] The conditions are based around fact values. If the operatingsystem value for a given agent is among the listed, yum will consider itself the default package provider. The operatingsystem and osfamily facts are the most popular choices for such queries in providers, but any fact is eligible. In addition to marking themselves as being default, there is more filtering of providers that relies on fact values. Providers can also confine themselves to certain combinations of values. For example, the yum alternative, zypper, confines itself to SUSE Linux distributions: confine :operatingsystem => [:suse, :sles, :sled, :opensuse] This provider method works just like the confine method in Facter, which was discussed earlier in this article. The provider will not even be seen as valid if the respective facts on the agent machine have none of the white-listed values. If you find yourself looking through code for some core providers, you will notice confinement (and even declaring default providers) on feature values, although there is no Facter fact of that name. These features are not related to provider features either. They are from another layer of introspection similar to Facter, but hardcoded into the Puppet agent. These agent features are a number of flags that identify some system properties that need not be made available to manifests in the form of facts. For example, the posix provider for the exec type becomes the default in the presence of the corresponding feature: defaultfor :feature => :posix You will find that some providers forgo the confine method altogether, as it is not mandatory for correct agent operation. Puppet will also identify unsuitable providers when looking for their necessary operating system commands. For example, the pw provider for certain BSD flavors does not bother with a confine statement. It only declares its one required command: commands :pw => "pw" Agents that find no pw binary in their search path will not try and use this provider at all. This concludes the little tour of the inner workings of types and providers with the help of Facter. Summary Puppet gathers information about all agent systems using Facter. The information base consists of a large number of independent bits, called facts. Manifests can query the values of those facts to adapt to the respective agents that trigger their compilation. Puppet also uses facts to choose among providers, the work horses that make the abstract resource types functional across the wide range of supported platforms. The resource types not only completely define the interface that Puppet exposes in the DSL, they also take care of all validation of input values, transformations that must be performed before handing values to the providers and other related tasks. The providers encapsulate all knowledge of actual operating systems and their respective toolchains. They implement the functionality that the resource types describe. The Puppet model's configurations apply to platforms, which vary from one another, so not every facet of every resource type can make sense for all agents. By exposing only the supported features, a provider can express such limitations. Resources for Article: Further resources on this subject: Module, Facts, Types and Reporting tools in Puppet [article] Quick start – Using the core Puppet resource types [article] Designing Puppet Architectures [article]
Read more
  • 0
  • 0
  • 1309

article-image-function-passing
Packt
19 Nov 2014
6 min read
Save for later

Function passing

Packt
19 Nov 2014
6 min read
In this article by Simon Timms, the author of the book, Mastering JavaScript Design Patterns, we will cover function passing. In functional programming languages, functions are first-class citizens. Functions can be assigned to variables and passed around just like you would with any other variable. This is not entirely a foreign concept. Even languages such as C had function pointers that could be treated just like other variables. C# has delegates and, in more recent versions, lambdas. The latest release of Java has also added support for lambdas, as they have proven to be so useful. (For more resources related to this topic, see here.) JavaScript allows for functions to be treated as variables and even as objects and strings. In this way, JavaScript is functional in nature. Because of JavaScript's single-threaded nature, callbacks are a common convention and you can find them pretty much everywhere. Consider calling a function at a later date on a web page. This is done by setting a timeout on the window object as follows: setTimeout(function(){alert("Hello from the past")}, 5 * 1000); The arguments for the set timeout function are a function to call and a time to delay in milliseconds. No matter the JavaScript environment in which you're working, it is almost impossible to avoid functions in the shape of callbacks. The asynchronous processing model of Node.js is highly dependent on being able to call a function and pass in something to be completed at a later date. Making calls to external resources in a browser is also dependent on a callback to notify the caller that some asynchronous operation has completed. In basic JavaScript, this looks like the following code: var xmlhttp = new XMLHttpRequest()xmlhttp.onreadystatechange=function()if (xmlhttp.readyState==4 &&xmlhttp.status==200){//process returned data}};xmlhttp.open("GET", http://some.external.resource, true); xmlhttp.send(); You may notice that we assign onreadystatechange before we even send the request. This is because assigning it later may result in a race condition in which the server responds before the function is attached to the ready state change. In this case, we've used an inline function to process the returned data. Because functions are first class citizens, we can change this to look like the following code: var xmlhttp;function requestData(){xmlhttp = new XMLHttpRequest()xmlhttp.onreadystatechange=processData;xmlhttp.open("GET", http://some.external.resource, true); xmlhttp.send();}function processData(){if (xmlhttp.readyState==4 &&xmlhttp.status==200){   //process returned data}} This is typically a cleaner approach and avoids performing complex processing in line with another function. However, you might be more familiar with the jQuery version of this, which looks something like this: $.getJSON('http://some.external.resource', function(json){//process returned data}); In this case, the boiler plate of dealing with ready state changes is handled for you. There is even convenience provided for you should the request for data fail with the following code: $.ajax('http://some.external.resource',{ success: function(json){   //process returned data},error: function(){   //process failure},dataType: "json"}); In this case, we've passed an object into the ajax call, which defines a number of properties. Amongst these properties are function callbacks for success and failure. This method of passing numerous functions into another suggests a great way of providing expansion points for classes. Likely, you've seen this pattern in use before without even realizing it. Passing functions into constructors as part of an options object is a commonly used approach to providing extension hooks in JavaScript libraries. Implementation In Westeros, the tourism industry is almost nonextant. There are great difficulties with bandits killing tourists and tourists becoming entangled in regional conflicts. Nonetheless, some enterprising folks have started to advertise a grand tour of Westeros in which they will take those with the means on a tour of all the major attractions. From King's Landing to Eyrie, to the great mountains of Dorne, the tour will cover it all. In fact, a rather mathematically inclined member of the tourism board has taken to calling it a Hamiltonian tour, as it visits everywhere once. The HamiltonianTour class provides an options object that allows the definition of an options object. This object contains the various places to which a callback can be attached. In our case, the interface for it would look something like the following code: export class HamiltonianTourOptions{onTourStart: Function;onEntryToAttraction: Function;onExitFromAttraction: Function;onTourCompletion: Function;} The full HamiltonianTour class looks like the following code: var HamiltonianTour = (function () {function HamiltonianTour(options) {   this.options = options;}HamiltonianTour.prototype.StartTour = function () {   if (this.options.onTourStart&&typeof (this.options.onTourStart)    === "function")   this.options.onTourStart();   this.VisitAttraction("King's Landing");   this.VisitAttraction("Winterfell");   this.VisitAttraction("Mountains of Dorne");   this.VisitAttraction("Eyrie");   if (this.options.onTourCompletion&&typeof    (this.options.onTourCompletion) === "function")   this.options.onTourCompletion();}; HamiltonianTour.prototype.VisitAttraction = function (AttractionName) {   if (this.options.onEntryToAttraction&&typeof    (this.options.onEntryToAttraction) === "function")   this.options.onEntryToAttraction(AttractionName);    //do whatever one does in a Attraction   if (this.options.onExitFromAttraction&&typeof    (this.options.onExitFromAttraction) === "function")   this.options.onExitFromAttraction(AttractionName);};return HamiltonianTour;})(); You can see in the highlighted code how we check the options and then execute the callback as needed. This can be done by simply using the following code: var tour = new HamiltonianTour({onEntryToAttraction: function(cityname){console.log("I'm delighted to be in " + cityname)}});tour.StartTour(); The output of the preceding code will be: I'm delighted to be in King's LandingI'm delighted to be in WinterfellI'm delighted to be in Mountains of DorneI'm delighted to be in Eyrie Summary In this article, we have learned about function passing. Passing functions is a great approach to solving a number of problems in JavaScript and tends to be used extensively by libraries such as jQuery and frameworks such as Express. It is so commonly adopted that using it provides to added barriers no your code's readability. Resources for Article: Further resources on this subject: Creating Java EE Applications [article] Meteor.js JavaScript Framework: Why Meteor Rocks! [article] Dart with JavaScript [article]
Read more
  • 0
  • 0
  • 11069

article-image-look-responsive-design-frameworks
Packt
19 Nov 2014
11 min read
Save for later

A look into responsive design frameworks

Packt
19 Nov 2014
11 min read
In this article, by Thoriq Firdaus author of Responsive Web Design by Example Beginner's Guide Second Edition we will look into responsive web design which is one of the most discussed topics among the web design and development community. So I believe many of you have heard about it to certain extend. (For more resources related to this topic, see here.) Ethan Marcotte was the one who coined the term "Responsive Web Design". He suggests in his article, Responsive Web Design, that the web should seamlessly adjust and adapt to the environment where the users view the website, rather than addressing it exclusively for a specific platform. In other words, the website should be responsive; it should be presentable at any screen size and regardless of the platform in which the website is viewed. Take Time website as an example, the web page fits nicely in a desktop browser with large screen size and also in a mobile browser with limited viewable area. The layout shifts and adapts as the viewport size changes. As you can see from the following screenshot, the header background color turned into dark grey, the image is scaled down proportionally, and the Tap bar appears where Time hides the Latest news, Magazine and Videos section: Yet, building a responsive website could be very tedious work. There are many measurements to consider when building a responsive website, one of which would be creating the responsive grid. Grid helps us to build websites with proper alignment. If you have ever used 960.gs framework, which is one of the popular CSS Frameworks, you should’ve experienced how easy is to organize the web page layout by adding preset classes like grid_1 or push_1 in the elements. However, 960.gs grid is set in fixed unit, pixel (px), which is not applicable when it comes to building a responsive website. We need a Framework with the grid set in percentage (%) unit to build responsive websites; we need a Responsive Framework. A Responsive Framework provides the building blocks to build responsive websites. Generally, it includes the classes to assemble a responsive grid, the basic styles for typography and form inputs, and a few styles to address various browser quirks. Some frameworks even go further with a series of styles for creating common design patterns and Web User Interface such as buttons, navigation bars, and image slider. These predefined styles allow us to develop responsive websites faster with less of the hassle. And the following are a few other reasons why using a Responsive Framework is a favorable option to build responsive websites: Browser Compatibility: Assuring consistency of a web page on different browsers is really painful and more distressing than developing the website itself. But, with a framework, we can minimize the work to address Browser Compatibility issues. The framework developers most likely have tested the framework in various desktop browsers and mobile browsers with the most constrained environment prior to releasing it publicly. Documentation: A framework, in general, also comes with comprehensive documentation that records the bits and pieces on using the framework. The documentation would be very helpful for entry users to begin to study the framework. It is also a great advantage when we are working with a team. We can refer to the documentation to get everyone on the same page and follow the standard code of writing conventions. Community and Extensions: Some popular frameworks like Bootstrap and Foundation have an active community that helps addressing the bugs in the framework and extends the functionality. The jQuery UI Bootstrap is perhaps a good example, in this case. jQuery UI Bootstrap is a collection styles for jQuery UI widgets to match the feel and look of Bootstrap’s original theme. It’s now a common to find free WordPress and Joomla themes that are based using these frameworks. The Responsive.gs framework Responsive.gs is a lightweight responsive framework with merely 1kb of size when compressed. Responsive.gs is based on a width of 940px, and made in three variant of grids: 12, 16, and 24 columns. What’s more, Responsive.gs is shipped with Box Sizing polyfill that enables CSS3 box-sizing in Internet Explorer 8 to Internet Explorer 6, and make it decently presentable in those browsers. Polyfill is a piece code that enables certain web features and capabilities that are not built in the browser natively; usually, it addresses to the older version of Internet Explorer. For example, you can use HTML5 Shiv so that new HTML5 elements, such as <header>, <footer>, and <nav>, are recognized in Internet Explorer 8 to Internet Explorer 6. CSS Box model HTML elements, which are categorized as a block-level element, are essentially a box drawn with the content width, height, margin, padding, and border through CSS. Prior to CSS3, we have been facing a constraint when specifying a box. For instance, when we specify a <div> with width and height of 100px, as follows: div {width: 100px;height: 100px;} The browser will render the div as 100px, square box. However, this will only be true if the padding and border have not been added in. Since a box has four sides, a padding of 10px (padding: 10px;) will actually add 20px to the width and height — 10px for each side, as follows. While it takes up space on the page, the element's margin is space reserved outside the element rather than part of the element itself; thus, if we give an element a background color, the margin area will not take on that color. CSS3 Box sizing CSS3 introduced a new property called box-sizing that lets us to specify how the browser should calculate the CSS box model. There are a couple of values that we can apply within the box-sizing property, which are: content-box: this is the default value of the box model. This value specifies the padding and the border box's thickness outside the specified width and height of the content, as we have demonstrated in the preceding section. border-box: this value will do the opposite; it includes the padding and the border box as the width and height of the box. padding-box: at the time of writing this article, this value is experimental and has just been added recently. This value specifies the box dimensions. Let’s take our preceding as our example, but this time we will set the box-sizing model to border-box. As mentioned in the table above, the border-box value will retain the box’s width and the height for 100px regardless of the padding and border addition. The following illustration shows a comparison between the outputs of the two different values, the content-box (the default) and the border-box. The Bootstrap framework Bootstrap was originally built by Mark Otto and was initially only intended for internal use in Twitter. Short story, Bootstrap was then launched for free for public consumption. Bootstrap has long been associated with Twitter, but since the author has departed from Twitter and Bootstrap itself has grown beyond his expectations..... Date back to the initial development, the responsive feature was not yet added, it was then added in version 2 along with the increasing demand for creating responsive websites. Bootstrap also comes with a lot more added features as compared to Responsive.gs. It is packed with preset user interface styles, which comprise of common User Interfaces used on websites such as buttons, navigation bars, pagination, and forms so you don’t have to create them from scratch again when starting off a new project. On top of that, Bootstrap is also powered with some custom jQuery plugins like image slider, carousel, popover and modal box. You can use and customize Bootstrap in many ways. You can directly customize Bootstrap theme and components directly through the CSS style sheets, the Bootstrap Customization page, or the Bootstrap LESS variables and mixins, which are used to generate the style sheets. The Foundation framework Foundation is a framework created by ZURB, a design agency based in California. Similar to Bootstrap, Foundation is beyond just a responsive CSS framework; it is shipped with preset grid, components, and a number of jQuery plugins to present interactive features. Some high-profile brands have built their website using of Foundation such as McAfee, which is one the most respectable brands for computer anti-virus. Foundation style sheet is powered by Sass, a Ruby-based CSS Pre-processor. There are many complaint that the code in responsive frameworks is excessive; since a framework like Bootstrap is used widely, it has to cover every design scenario and thus it comes with some extra styles that you might not need for your website. Fortunately, we can easily minimize this issue by using the right tools like CSS Preprocessors and following a proper workflow. And speaking the truth, there isn’t a perfect solution, and certainly using a framework isn’t for everyone. It all comes down to your need, your website need, and in particular your client needs and budgets. In reality, you will have to weigh these factors to decide whether you will go with responsive framework or not. Jem Kremer has an extensive discussion on this regard in her article: Responsive Design Frameworks: Just Because You Can, Should You? A brief Introduction to CSS preprocessors Both Bootstrap and Foundation uses CSS Pre-processors to generate their style sheets. Bootstrap uses LESS — though the official support for Sass has just been released recently. Foundation, on the contrary, uses Sass as the only way to generate its style sheets. CSS pre-processor is not an entirely new language. If you have known CSS, you should be accustomed to CSS Pre-preprocessor immediately. CSS Pre-processor simply extends CSS by allowing the use of programming features like Variables, Functions, and Operations. Below is an example of how we write CSS with LESS syntax. @color: #f3f3f3;body {background-color: @color;}p {color: darken(@color, 50%);} When the above code is compiled, it takes the @color variable that we have defined and place the value in the output, as follows. body {background-color: #f3f3f3;}p {color: #737373;} The variable is reusable throughout the style sheet that enables us to retain style consistency and make the style sheet more maintainable. Delve into responsive web design Our discussion on Responsive Web Design herein, though essential, is merely a tip of the iceberg. There are so much more about Responsive Web Design than what have recently covered in the preceding sections. I would suggest that you take your time to get yourself more insight and apprehension on Responsive Web Design including the concept, the technicalities, and some constraints. The following are some of the best recommendations of reference to follow: Also a good place to start Responsive Web Design by Rachel Shillcock. Don’t Forget the Viewport Meta Tag by Ian Yates. How To Use CSS3 Media Queries To Create a Mobile Version of Your Website by Rachel Andrew. Read about the future standard on responsive image using HTML5 Picture Element Responsive Images Done Right: A Guide To <picture> And srcset by Eric Portis a roundup of methods of making data table responsive. Responsive web design inspiration sources Now before we jump down into the next Chapters and start off building responsive websites, it may be a good idea to spend some time looking for ideas and inspiration of responsive websites; to see how they are built, and how the layout is organized in desktop browsers as well as in mobile browsers. It’s a common thing for websites to be redesigned from time to time to stay fresh. So herein, instead of making a pile of website screenshots, which may no longer be relevant in the next several months because of the redesign, we’re better going straight to the websites that curates websites, and following is the places to go: MediaQueries Awwwards CSS Awards WebDesignServed Bootstrap Expo Zurb Responsive Summary Using a framework is the easier and faster way to build responsive websites up and running rather than building everything from scratch on our own. Alas, as mentioned, using a framework also has some negative sides. If it is not done properly, the end result could all go wrong. The website could be stuffed and stuck with unnecessary styles and JavaScript, which at the end makes the website load slowly and hard to maintain. We need to set up the right tools that, not only will they facilitate the projects, but they also help us making the website more easily maintainable. Resources for Article:  Further resources on this subject: Linking Dynamic Content from External Websites [article] Building Responsive Image Sliders [article] Top Features You Need to Know About – Responsive Web Design [article]
Read more
  • 0
  • 0
  • 9964

article-image-dart-javascript
Packt
18 Nov 2014
12 min read
Save for later

Dart with JavaScript

Packt
18 Nov 2014
12 min read
In this article by Sergey Akopkokhyants, author of Mastering Dart, we will combine the simplicity of jQuery and the power of Dart in a real example. (For more resources related to this topic, see here.) Integrating Dart with jQuery For demonstration purposes, we have created the js_proxy package to help the Dart code to communicate with jQuery. It is available on the pub manager at https://pub.dartlang.org/packages/js_proxy. This package is layered on dart:js and has a library of the same name and sole class JProxy. An instance of the JProxy class can be created via the generative constructor where we can specify the optional reference on the proxied JsObject: JProxy([this._object]); We can create an instance of JProxy with a named constructor and provide the name of the JavaScript object accessible through the dart:js context as follows: JProxy.fromContext(String name) { _object = js.context[name]; } The JProxy instance keeps the reference on the proxied JsObject class and makes all the manipulation on it, as shown in the following code: js.JsObject _object;    js.JsObject get object => _object; How to create a shortcut to jQuery? We can use JProxy to create a reference to jQuery via the context from the dart:js library as follows: var jquery = new JProxy.fromContext('jQuery'); Another very popular way is to use the dollar sign as a shortcut to the jQuery variable as shown in the following code: var $ = new JProxy.fromContext('jQuery'); Bear in mind that the original jQuery and $ variables from JavaScript are functions, so our variables reference to the JsFunction class. From now, jQuery lovers who moved to Dart have a chance to use both the syntax to work with selectors via parentheses. Why JProxy needs a method call? Usually, jQuery send a request to select HTML elements based on IDs, classes, types, attributes, and values of their attributes or their combination, and then performs some action on the results. We can use the basic syntax to pass the search criteria in the jQuery or $ function to select the HTML elements: $(selector) Dart has syntactic sugar method call that helps us to emulate a function and we can use the call method in the jQuery syntax. Dart knows nothing about the number of arguments passing through the function, so we use the fixed number of optional arguments in the call method. Through this method, we invoke the proxied function (because jquery and $ are functions) and returns results within JProxy: dynamic call([arg0 = null, arg1 = null, arg2 = null,    arg3 = null, arg4 = null, arg5 = null, arg6 = null,    arg7 = null, arg8 = null, arg9 = null]) { var args = []; if (arg0 != null) args.add(arg0); if (arg1 != null) args.add(arg1); if (arg2 != null) args.add(arg2); if (arg3 != null) args.add(arg3); if (arg4 != null) args.add(arg4); if (arg5 != null) args.add(arg5); if (arg6 != null) args.add(arg6); if (arg7 != null) args.add(arg7); if (arg8 != null) args.add(arg8); if (arg9 != null) args.add(arg9); return _proxify((_object as js.JsFunction).apply(args)); } How JProxy invokes jQuery? The JProxy class is a proxy to other classes, so it marks with the @proxy annotation. We override noSuchMethod intentionally to call the proxied methods and properties of jQuery when the methods or properties of the proxy are invoked. The logic flow in noSuchMethod is pretty straightforward. It invokes callMethod of the proxied JsObject when we invoke the method on proxy, or returns a value of property of the proxied object if we call the corresponding operation on proxy. The code is as follows: @override dynamic noSuchMethod(Invocation invocation) { if (invocation.isMethod) {    return _proxify(_object.callMethod(      symbolAsString(invocation.memberName),      _jsify(invocation.positionalArguments))); } else if (invocation.isGetter) {    return      _proxify(_object[symbolAsString(invocation.memberName)]); } else if (invocation.isSetter) {    throw new Exception('The setter feature was not implemented      yet.'); } return super.noSuchMethod(invocation); } As you might remember, all map or Iterable arguments must be converted to JsObject with the help of the jsify method. In our case, we call the _jsify method to check and convert passed arguments aligned with a called function, as shown in the following code: List _jsify(List params) { List res = []; params.forEach((item) {    if (item is Map || item is List) {      res.add(new js.JsObject.jsify(item));    } else {      res.add(item);    } }); return res; } Before return, the result must be passed through the _proxify function as follows: dynamic _proxify(value) {    return value is js.JsObject ? new JProxy(value) : value; } This function wraps all JsObject within a JProxy class and passes other values as it is. An example project Now create the jquery project, open the pubspec.yaml file, and add js_proxy to the dependencies. Open the jquery.html file and make the following changes: <!DOCTYPE html> <html> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <title>jQuery</title> <link rel="stylesheet" href="jquery.css"> </head> <body> <h1>Jquery</h1> <p>I'm a paragraph</p> <p>Click on me to hide</p> <button>Click me</button> <div class="container"> <div class="box"></div> </div> </body> <script src="//code.jquery.com/jquery-1.11.0.min.js"></script> <script type="application/dart" src="jquery.dart"></script> <script src="packages/browser/dart.js"></script> </html> This project aims to demonstrate that: Communication is easy between Dart and JavaScript The syntax of the Dart code could be similar to the jQuery code In general, you may copy the JavaScript code, paste it in the Dart code, and probably make slightly small changes. How to get the jQuery version? It's time to add js_proxy in our code. Open jquery.dart and make the following changes: import 'dart:html'; import 'package:js_proxy/js_proxy.dart'; /** * Shortcut for jQuery. */ var $ = new JProxy.fromContext('jQuery'); /** * Shortcut for browser console object. */ var console = window.console; main() { printVersion(); } /** * jQuery code: * *   var ver = $().jquery; *   console.log("jQuery version is " + ver); * * JS_Proxy based analog: */ printVersion() { var ver = $().jquery; console.log("jQuery version is " + ver); } You should be familiar with jQuery and console shortcuts yet. The call to jQuery with empty parentheses returns JProxy and contains JsObject with reference to jQuery from JavaScript. The jQuery object has a jQuery property that contains the current version number, so we reach this one via noSuchMethod of JProxy. Run the application, and you will see the following result in the console: jQuery version is 1.11.1 Let's move on and perform some actions on the selected HTML elements. How to perform actions in jQuery? The syntax of jQuery is based on selecting the HTML elements and it also performs some actions on them: $(selector).action(); Let's select a button on the HTML page and fire the click event as shown in the following code: /** * jQuery code: * *   $("button").click(function(){ *     alert('You click on button'); *   }); * * JS_Proxy based analog: */ events() { // We remove 'function' and add 'event' here $("button").click((event) {    // Call method 'alert' of 'window'    window.alert('You click on button'); }); } All we need to do here is just remove the function keyword, because anonymous functions on Dart do not use it and add the event parameter. This is because this argument is required in the Dart version of the event listener. The code calls jQuery to find all the HTML button elements to add the click event listener to each of them. So when we click on any button, a specified alert message will be displayed. On running the application, you will see the following message: How to use effects in jQuery? The jQuery supports animation out of the box, so it sounds very tempting to use it from Dart. Let's take an example of the following code snippet: /** * jQuery code: * *   $("p").click(function() { *     this.hide("slow",function(){ *       alert("The paragraph is now hidden"); *     }); *   }); *   $(".box").click(function(){ *     var box = this; *     startAnimation(); *     function startAnimation(){ *       box.animate({height:300},"slow"); *       box.animate({width:300},"slow"); *       box.css("background-color","blue"); *       box.animate({height:100},"slow"); *       box.animate({width:100},"slow",startAnimation); *     } *   }); * * JS_Proxy based analog: */ effects() { $("p").click((event) {    $(event['target']).hide("slow",(){      window.alert("The paragraph is now hidden");    }); }); $(".box").click((event) {    var box = $(event['target']);    startAnimation() {      box.animate({'height':300},"slow");      box.animate({'width':300},"slow");      box.css("background-color","blue");      box.animate({'height':100},"slow");      box.animate({'width':100},"slow",startAnimation);    };    startAnimation(); }); } This code finds all the paragraphs on the web page to add a click event listener to each one. The JavaScript code uses the this keyword as a reference to the selected paragraph to start the hiding animation. The this keyword has a different notion on JavaScript and Dart, so we cannot use it directly in anonymous functions on Dart. The target property of event keeps the reference to the clicked element and presents JsObject in Dart. We wrap the clicked element to return a JProxy instance and use it to call the hide method. The jQuery is big enough and we have no space in this article to discover all its features, but you can find more examples at https://github.com/akserg/js_proxy. What are the performance impacts? Now, we should talk about the performance impacts of using different approaches across several modern web browsers. The algorithm must perform all the following actions: It should create 10000 DIV elements Each element should be added into the same DIV container Each element should be updated with one style All elements must be removed one by one This algorithm must be implemented in the following solutions: The clear jQuery solution on JavaScript The jQuery solution calling via JProxy and dart:js from Dart The clear Dart solution based on dart:html We implemented this algorithm on all of them, so we have a chance to compare the results and choose the champion. The following HTML code has three buttons to run independent tests, three paragraph elements to show the results of the tests, and one DIV element used as a container. The code is as follows: <div>  <button id="run_js" onclick="run_js_test()">Run JS</button> <button id="run_jproxy">Run JProxy</button> <button id="run_dart">Run Dart</button> </div>   <p id="result_js"></p> <p id="result_jproxy"></p> <p id="result_dart"></p> <div id="container"></div> The JavaScript code based on jQuery is as follows: function run_js_test() { var startTime = new Date(); process_js(); var diff = new Date(new Date().getTime() –    startTime.getTime()).getTime(); $('#result_js').text('jQuery tooks ' + diff +    ' ms to process 10000 HTML elements.'); }     function process_js() { var container = $('#container'); // Create 10000 DIV elements for (var i = 0; i < 10000; i++) {    $('<div>Test</div>').appendTo(container); } // Find and update classes of all DIV elements $('#container > div').css("color","red"); // Remove all DIV elements $('#container > div').remove(); } The main code registers the click event listeners and the call function run_dart_js_test. The first parameter of this function must be investigated. The second and third parameters are used to pass the selector of the result element and test the title: void main() { querySelector('#run_jproxy').onClick.listen((event) {    run_dart_js_test(process_jproxy, '#result_jproxy', 'JProxy'); }); querySelector('#run_dart').onClick.listen((event) {    run_dart_js_test(process_dart, '#result_dart', 'Dart'); }); } run_dart_js_test(Function fun, String el, String title) { var startTime = new DateTime.now(); fun(); var diff = new DateTime.now().difference(startTime); querySelector(el).text = '$title tooks ${diff.inMilliseconds} ms to process 10000 HTML elements.'; } Here is the Dart solution based on JProxy and dart:js: process_jproxy() { var container = $('#container'); // Create 10000 DIV elements for (var i = 0; i < 10000; i++) {    $('<div>Test</div>').appendTo(container.object); } // Find and update classes of all DIV elements $('#container > div').css("color","red"); // Remove all DIV elements $('#container > div').remove(); } Finally, a clear Dart solution based on dart:html is as follows: process_dart() { // Create 10000 DIV elements var container = querySelector('#container'); for (var i = 0; i < 10000; i++) {    container.appendHtml('<div>Test</div>'); } // Find and update classes of all DIV elements querySelectorAll('#container > div').forEach((Element el) {    el.style.color = 'red'; }); // Remove all DIV elements querySelectorAll('#container > div').forEach((Element el) {    el.remove(); }); } All the results are in milliseconds. Run the application and wait until the web page is fully loaded. Run each test by clicking on the appropriate button. My result of the tests on Dartium, Chrome, Firefox, and Internet Explorer are shown in the following table: Web browser jQuery framework jQuery via JProxy Library dart:html Dartium 2173 3156 714 Chrome 2935 6512 795 Firefox 2485 5787 582 Internet Explorer 12262 17748 2956 Now, we have the absolute champion—the Dart-based solution. Even the Dart code compiled in the JavaScript code to be executed in Chrome, Firefox, and Internet Explorer works quicker than jQuery (four to five times) and much quicker than dart:js and JProxy class-based solution (four to ten times). Summary This article showed you how to use Dart and JavaScript together to build web applications. It listed problems and solutions you can use to communicate between Dart and JavaScript and the existing JavaScript program. We compared jQuery, JProxy, and dart:js and cleared the Dart code based on the dart:html solutions to identify who is quicker than others. Resources for Article: Further resources on this subject: Handling the DOM in Dart [article] Dart Server with Dartling and MongoDB [article] Handle Web Applications [article]
Read more
  • 0
  • 0
  • 6627
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-plot-function
Packt
18 Nov 2014
17 min read
Save for later

The plot function

Packt
18 Nov 2014
17 min read
In this article by L. Felipe Martins, the author of the book, IPython Notebook Essentials, has discussed about the plot() function, which is an important aspect of matplotlib, an IPython library for production of publication-quality graphs. (For more resources related to this topic, see here.) The plot() function is the workhorse of the matplotlib library. In this section, we will explore the line-plotting and formatting capabilities included in this function. To make things a bit more concrete, let's consider the formula for logistic growth, as follows: This model is frequently used to represent growth that shows an initial exponential phase, and then is eventually limited by some factor. The examples are the population in an environment with limited resources and new products and/or technological innovations, which initially attract a small and quickly growing market but eventually reach a saturation point. A common strategy to understand a mathematical model is to investigate how it changes as the parameters defining it are modified. Let's say, we want to see what happens to the shape of the curve when the parameter b changes. To be able to do what we want more efficiently, we are going to use a function factory. This way, we can quickly create logistic models with arbitrary values for r, a, b, and c. Run the following code in a cell: def make_logistic(r, a, b, c):    def f_logistic(t):        return a / (b + c * exp(-r * t))    return f_logistic The function factory pattern takes advantage of the fact that functions are first-class objects in Python. This means that functions can be treated as regular objects: they can be assigned to variables, stored in lists in dictionaries, and play the role of arguments and/or return values in other functions. In our example, we define the make_logistic() function, whose output is itself a Python function. Notice how the f_logistic() function is defined inside the body of make_logistic() and then returned in the last line. Let's now use the function factory to create three functions representing logistic curves, as follows: r = 0.15 a = 20.0 c = 15.0 b1, b2, b3 = 2.0, 3.0, 4.0 logistic1 = make_logistic(r, a, b1, c) logistic2 = make_logistic(r, a, b2, c) logistic3 = make_logistic(r, a, b3, c) In the preceding code, we first fix the values of r, a, and c, and define three logistic curves for different values of b. The important point to notice is that logistic1, logistic2, and logistic3 are functions. So, for example, we can use logistic1(2.5) to compute the value of the first logistic curve at the time 2.5. We can now plot the functions using the following code: tmax = 40 tvalues = linspace(0, tmax, 300) plot(tvalues, logistic1(tvalues)) plot(tvalues, logistic2(tvalues)) plot(tvalues, logistic3(tvalues)) The first line in the preceding code sets the maximum time value, tmax, to be 40. Then, we define the set of times at which we want the functions evaluated with the assignment, as follows: tvalues = linspace(0, tmax, 300) The linspace() function is very convenient to generate points for plotting. The preceding code creates an array of 300 equally spaced points in the interval from 0 to tmax. Note that, contrary to other functions, such as range() and arange(), the right endpoint of the interval is included by default. (To exclude the right endpoint, use the endpoint=False option.) After defining the array of time values, the plot() function is called to graph the curves. In its most basic form, it plots a single curve in a default color and line style. In this usage, the two arguments are two arrays. The first array gives the horizontal coordinates of the points being plotted, and the second array gives the vertical coordinates. A typical example will be the following function call: plot(x,y) The variables x and y must refer to NumPy arrays (or any Python iterable values that can be converted into an array) and must have the same dimensions. The points plotted have coordinates as follows: x[0], y[0] x[1], y[1] x[2], y[2] … The preceding command will produce the following plot, displaying the three logistic curves: You may have noticed that before the graph is displayed, there is a line of text output that looks like the following: [<matplotlib.lines.Line2D at 0x7b57c50>] This is the return value of the last call to the plot() function, which is a list (or with a single element) of objects of the Line2D type. One way to prevent the output from being shown is to enter None as the last row in the cell. Alternatively, we can assign the return value of the last call in the cell to a dummy variable: _dummy_ = plot(tvalues, logistic3(tvalues)) The plot() function supports plotting several curves in the same function call. We need to change the contents of the cell that are shown in the following code and run it again: tmax = 40 tvalues = linspace(0, tmax, 300) plot(tvalues, logistic1(tvalues),      tvalues, logistic2(tvalues),      tvalues, logistic3(tvalues)) This form saves some typing but turns out to be a little less flexible when it comes to customizing line options. Notice that the text output produced now is a list with three elements: [<matplotlib.lines.Line2D at 0x9bb6cc0>, <matplotlib.lines.Line2D at 0x9bb6ef0>, <matplotlib.lines.Line2D at 0x9bb9518>] This output can be useful in some instances. For now, we will stick with using one call to plot() for each curve, since it produces code that is clearer and more flexible. Let's now change the line options in the plot and set the plot bounds. Change the contents of the cell to read as follows: plot(tvalues, logistic1(tvalues),      linewidth=1.5, color='DarkGreen', linestyle='-') plot(tvalues, logistic2(tvalues),      linewidth=2.0, color='#8B0000', linestyle=':') plot(tvalues, logistic3(tvalues),      linewidth=3.5, color=(0.0, 0.0, 0.5), linestyle='--') axis([0, tmax, 0, 11.]) None Running the preceding command lines will produce the following plots: The options set in the preceding code are as follows: The first curve is plotted with a line width of 1.5, with the HTML color of DarkGreen, and a filled-line style The second curve is plotted with a line width of 2.0, colored with the RGB value given by the hexadecimal string '#8B0000', and a dotted-line style The third curve is plotted with a line width of 3.0, colored with the RGB components, (0.0, 0.0, 0.5), and a dashed-line style Notice that there are different ways of specifying a fixed color: a HTML color name, a hexadecimal string, or a tuple of floating-point values. In the last case, the entries in the tuple represent the intensity of the red, green, and blue colors, respectively, and must be floating-point values between 0.0 and 1.0. A complete list of HTML name colors can be found at http://www.w3schools.com/html/html_colornames.asp. Editor's Tip: For more insights on colors, check out https://dgtl.link/colors Line styles are specified by a symbolic string. The allowed values are shown in the following table: Symbol string Line style '-' Solid (the default) '--' Dashed ':' Dotted '-.' Dash-dot 'None', '', or '' Not displayed After the calls to plot(), we set the graph bounds with the function call: axis([0, tmax, 0, 11.]) The argument to axis() is a four-element list that specifies, in this order, the maximum and minimum values of the horizontal coordinates, and the maximum and minimum values of the vertical coordinates. It may seem non-intuitive that the bounds for the variables are set after the plots are drawn. In the interactive mode, matplotlib remembers the state of the graph being constructed, and graphics objects are updated in the background after each command is issued. The graph is only rendered when all computations in the cell are done so that all previously specified options take effect. Note that starting a new cell clears all the graph data. This interactive behavior is part of the matplotlib.pyplot module, which is one of the components imported by pylab. Besides drawing a line connecting the data points, it is also possible to draw markers at specified points. Change the graphing commands indicated in the following code snippet, and then run the cell again: plot(tvalues, logistic1(tvalues),      linewidth=1.5, color='DarkGreen', linestyle='-',      marker='o', markevery=50, markerfacecolor='GreenYellow',      markersize=10.0) plot(tvalues, logistic2(tvalues),      linewidth=2.0, color='#8B0000', linestyle=':',      marker='s', markevery=50, markerfacecolor='Salmon',      markersize=10.0) plot(tvalues, logistic3(tvalues),      linewidth=2.0, color=(0.0, 0.0, 0.5), linestyle='--',      marker = '*', markevery=50, markerfacecolor='SkyBlue',      markersize=12.0) axis([0, tmax, 0, 11.]) None Now, the graph will look as shown in the following figure: The only difference from the previous code is that now we added options to draw markers. The following are the options we use: The marker option specifies the shape of the marker. Shapes are given as symbolic strings. In the preceding examples, we use 'o' for a circular marker, 's' for a square, and '*' for a star. A complete list of available markers can be found at http://matplotlib.org/api/markers_api.html#module-matplotlib.markers. The markevery option specifies a stride within the data points for the placement of markers. In our example, we place a marker after every 50 data points. The markercolor option specifies the color of the marker. The markersize option specifies the size of the marker. The size is given in pixels. There are a large number of other options that can be applied to lines in matplotlib. A complete list is available at http://matplotlib.org/api/artist_api.html#module-matplotlib.lines. Adding a title, labels, and a legend The next step is to add a title and labels for the axes. Just before the None line, add the following three lines of code to the cell that creates the graph: title('Logistic growth: a={:5.2f}, c={:5.2f}, r={:5.2f}'.format(a, c, r)) xlabel('$t$') ylabel('$N(t)=a/(b+ce^{-rt})$') In the first line, we call the title() function to set the title of the plot. The argument can be any Python string. In our example, we use a formatted string: title('Logistic growth: a={:5.2f}, b={:5.2f}, r={:5.2f}'.format(a, c, r)) We use the format() method of the string class. The formats are placed between braces, as in {:5.2f}, which specifies a floating-point format with five spaces and two digits of precision. Each of the format specifiers is then associated sequentially with one of the data arguments of the method. A full documentation covering the details of string formatting is available at https://docs.python.org/2/library/string.html. The axis labels are set in the calls: xlabel('$t$') ylabel('$N(t)=a/(b+ce^{-rt})$') As in the title() functions, the xlabel() and ylabel() functions accept any Python string. Note that in the '$t$' and '$N(t)=a/(b+ce^{-rt}$' strings, we use LaTeX to format the mathematical formulas. This is indicated by the dollar signs, $...$, in the string. After the addition of a title and labels, our graph looks like the following: Next, we need a way to identify each of the curves in the picture. One way to do that is to use a legend, which is indicated as follows: legend(['b={:5.2f}'.format(b1),        'b={:5.2f}'.format(b2),        'b={:5.2f}'.format(b3)]) The legend() function accepts a list of strings. Each string is associated with a curve in the order they are added to the plot. Notice that we are again using formatted strings. Unfortunately, the preceding code does not produce great results. The legend, by default, is placed in the top-right corner of the plot, which, in this case, hides part of the graph. This is easily fixed using the loc option in the legend function, as shown in the following code: legend(['b={:5.2f}'.format(b1),        'b={:5.2f}'.format(b2),        'b={:5.2f}'.format(b3)], loc='upper left') Running this code, we obtain the final version of our logistic growth plot, as follows: The legend location can be any of the strings: 'best', 'upper right', 'upper left', 'lower left', 'lower right', 'right', 'center left', 'center right', 'lower center', 'upper center', and 'center'. It is also possible to specify the location of the legend precisely with the bbox_to_anchor option. To see how this works, modify the code for the legend as follows: legend(['b={:5.2f}'.format(b1),        'b={:5.2f}'.format(b2),        'b={:5.2f}'.format(b3)], bbox_to_anchor=(0.9,0.35)) Notice that the bbox_to_anchor option, by default, uses a coordinate system that is not the same as the one we specified for the plot. The x and y coordinates of the box in the preceding example are interpreted as a fraction of the width and height, respectively, of the whole figure. A little trial-and-error is necessary to place the legend box precisely where we want it. Note that the legend box can be placed outside the plot area. For example, try the coordinates (1.32,1.02). The legend() function is quite flexible and has quite a few other options that are documented at http://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.legend. Text and annotations In this subsection, we will show how to add annotations to plots in matplotlib. We will build a plot demonstrating the fact that the tangent to a curve must be horizontal at the highest and lowest points. We start by defining the function associated with the curve and the set of values at which we want the curve to be plotted, which is shown in the following code: f = lambda x: (x**3 - 6*x**2 + 9*x + 3) / (1 + 0.25*x**2) xvalues = linspace(0, 5, 200) The first line in the preceding code uses a lambda expression to define the f() function. We use this approach here because the formula for the function is a simple, one-line expression. The general form of a lambda expression is as follows: lambda <arguments> : <return expression> This expression by itself creates an anonymous function that can be used in any place that a function object is expected. Note that the return value must be a single expression and cannot contain any statements. The formula for the function may seem unusual, but it was chosen by trial-and-error and a little bit of calculus so that it produces a nice graph in the interval from 0 to 5. The xvalues array is defined to contain 200 equally spaced points on this interval. Let's create an initial plot of our curve, as shown in the following code: plot(xvalues, f(xvalues), lw=2, color='FireBrick') axis([0, 5, -1, 8]) grid() xlabel('$x$') ylabel('$f(x)$') title('Extreme values of a function') None # Prevent text output Most of the code in this segment is explained in the previous section. The only new bit is that we use the grid() function to draw a grid. Used with no arguments, the grid coincides with the tick marks on the plot. As everything else in matplotlib, grids are highly customizable. Check the documentation at http://matplotlib.org/1.3.1/api/pyplot_api.html#matplotlib.pyplot.grid. When the preceding code is executed, the following plot is produced: Note that the curve has a highest point (maximum) and a lowest point (minimum). These are collectively called the extreme values of the function (on the displayed interval, this function actually grows without bounds as x becomes large). We would like to locate these on the plot with annotations. We will first store the relevant points as follows: x_min = 3.213 f_min = f(x_min) x_max = 0.698 f_max = f(x_max) p_min = array([x_min, f_min]) p_max = array([x_max, f_max]) print p_min print p_max The variables, x_min and f_min, are defined to be (approximately) the coordinates of the lowest point in the graph. Analogously, x_max and f_max represent the highest point. Don't be concerned with how these points were found. For the purposes of graphing, even a rough approximation by trial-and-error would suffice. Now, add the following code to the cell that draws the plot, right below the title() command, as shown in the following code: arrow_props = dict(facecolor='DimGray', width=3, shrink=0.05,              headwidth=7) delta = array([0.1, 0.1]) offset = array([1.0, .85]) annotate('Maximum', xy=p_max+delta, xytext=p_max+offset,          arrowprops=arrow_props, verticalalignment='bottom',          horizontalalignment='left', fontsize=13) annotate('Minimum', xy=p_min-delta, xytext=p_min-offset,          arrowprops=arrow_props, verticalalignment='top',          horizontalalignment='right', fontsize=13) Run the cell to produce the plot shown in the following diagram: In the code, start by assigning the variables arrow_props, delta, and offset, which will be used to set the arguments in the calls to annotate(). The annotate() function adds a textual annotation to the graph with an optional arrow indicating the point being annotated. The first argument of the function is the text of the annotation. The next two arguments give the locations of the arrow and the text: xy: This is the point being annotated and will correspond to the tip of the arrow. We want this to be the maximum/minimum points, p_min and p_max, but we add/subtract the delta vector so that the tip is a bit removed from the actual point. xytext: This is the point where the text will be placed as well as the base of the arrow. We specify this as offsets from p_min and p_max using the offset vector. All other arguments of annotate() are formatting options: arrowprops: This is a Python dictionary containing the arrow properties. We predefine the dictionary, arrow_props, and use it here. Arrows can be quite sophisticated in matplotlib, and you are directed to the documentation for details. verticalalignment and horizontalalignment: These specify how the arrow should be aligned with the text. fontsize: This signifies the size of the text. Text is also highly configurable, and the reader is directed to the documentation for details. The annotate() function has a huge number of options; for complete details of what is available, users should consult the documentation at http://matplotlib.org/1.3.1/api/pyplot_api.html#matplotlib.pyplot.annotate for the full details. We now want to add a comment for what is being demonstrated by the plot by adding an explanatory textbox. Add the following code to the cell right after the calls to annotate(): bbox_props = dict(boxstyle='round', lw=2, fc='Beige') text(2, 6, 'Maximum and minimum pointsnhave horizontal tangents',      bbox=bbox_props, fontsize=12, verticalalignment='top') The text()function is used to place text at an arbitrary position of the plot. The first two arguments are the position of the textbox, and the third argument is a string containing the text to be displayed. Notice the use of 'n' to indicate a line break. The other arguments are configuration options. The bbox argument is a dictionary with the options for the box. If omitted, the text will be displayed without any surrounding box. In the example code, the box is a rectangle with rounded corners, with a border width of 2 pixels and the face color, beige. As a final detail, let's add the tangent lines at the extreme points. Add the following code: plot([x_min-0.75, x_min+0.75], [f_min, f_min],      color='RoyalBlue', lw=3) plot([x_max-0.75, x_max+0.75], [f_max, f_max],      color='RoyalBlue', lw=3) Since the tangents are segments of straight lines, we simply give the coordinates of the endpoints. The reason to add the code for the tangents at the top of the cell is that this causes them to be plotted first so that the graph of the function is drawn at the top of the tangents. This is the final result: The examples we have seen so far only scratch the surface of what is possible with matplotlib. The reader should read the matplotlib documentation for more examples. Summary In this article, we learned how to use matplotlib to produce presentation-quality plots. We covered two-dimensional plots and how to set plot options, and annotate and configure plots. You also learned how to add labels, titles, and legends. Edited on July 27, 2018 to replace a broken reference link. Resources for Article: Further resources on this subject: Installing NumPy, SciPy, matplotlib, and IPython [Article] SciPy for Computational Geometry [Article] Fast Array Operations with NumPy [Article]
Read more
  • 0
  • 0
  • 10842

article-image-searching-and-resolving-conflicts
Packt
18 Nov 2014
11 min read
Save for later

Searching and Resolving Conflicts

Packt
18 Nov 2014
11 min read
This article, by Eric Pidoux, author of Git Best Practices Guide, covers a part of Git that you will definitely meet: conflicts. How can we resolve them? (For more resources related to this topic, see here.) While working together as a team on a project, you will work on the same files. The pull command won't work because there are conflicts, and you might have tried some Git commands and things got bad. In this chapter, we will find solutions to these conflicts and see how we can fix them. We will cover the following topics: Finding content inside your Git repository Stashing your changes Fixing errors by practical examples Finding content inside your repository Sometimes, you will need to find something inside all your files. You can, of course, find it with the search feature of your OS, but Git already knows all your files. Searching file content To search text inside your files, simply use the following command: Erik@server:~$ git grep "Something to find" Erik@server:~$ git grep -n body Master:Website.Index.html:4:       <bodyMaster:Website.Index.html:12:       </body> It will display every match to the given keyword inside your code. All lines use the [commitref]:[filepath]:[linenumber]:[matchingcontent] pattern. Notice that [commitref] isn't displayed on all Git versions. You can also specify the commit references that grep will use to search the keyword: Erik@server:~$ git grep -n body d32lf56 p88e03d HEAD~3 Master:Website.Index.html:4:       <body> Master:Website.Index.html:12:       </body> In this case, grep will look into the d32lf56, p88e03d, and third commit starting by the head pointer. Your repository has to be encoded in UTF-8; otherwise, the grep command won't work. Git allows you to use regex inside the search feature by replacing somethingToFind with a regex. You can use the logical operators (or and and), as shown in the following command: Erik@server:~$ git grep -e myRegex1 --or -e myRegex2 Erik@server:~$ git grep -e myRegex1 --and -e myRegex2 Let's see this with an example. We only have a test.html page inside our last commit, and we want to find whether or not there is a word with an uppercase alphabetic value and numeric values: Erik@server:~$ git grep -e [A-Z] --and -e [0-9] HEAD Master:Website.Test.html:6:       TEST01 With the grep command, you can delve deeper, but it's not necessary to discuss this topic here because you won't use it every day! Showing the current status The git status command is helpful if you have to analyze your repository: Erik@server:~$ git status # On branch master # Your branch is ahead of 'origin/master' by 2 commits # (use "git push" to publish your local commits) # Changes not staged for commit: #   (use "git add<file>..." to update what will be committed) #   (use "git checkout -- <file>..." to discard changes in working directory) # # modified:   myFile1 # modified:   myFile2 # # Untracked files: #   (use "git add<file>..." to include in what will be committed) # # newFile.txt no changes added to commit (use "git add" and/or "git commit -a") Git analyzes the local repository in comparison to the remote repository. In this case, you have to add newFile.txt, commit myFile1 and myFile2, and push them to the remote repository. Exploring the repository history The best way to explore the past commits inside your repository is to use the git log command. For this part, we will assume that there are only two commits. To display all commits, use the following commands: Erik@server:~$ git log --all Commit xxxxxxxxxxx Author: Jim <jim@mail.com> Date: Sun Jul 20 15:10:12 2014 -0300 Fix front bugs on banner   Commit xxxxxxxxxxx Author: Erik <erik@mail.com> Date: Sat Jul 19 07:06:14 2014 -0300 Add the crop feature on website backend This is probably not what you want. After several days of work, you will have plenty of these commits, so how will you filter it? The power of the git log command is that you can quickly find anything in all commits. Let's go for a quick overview of what Git is able to find. We will start by finding the last commit: Erik@server:~$ git log -1 Commit xxxxxxxxxxx Author: Jim <jim@mail.com> Date: Sun Jul 20 15:10:12 2014 -0300 Fix front bugs on banner The number after the git log command indicates that it is the first commit from Head. Too easy! Let's try to find what the last commit of Erik is: Erik@server:~$ git log --author=Erik -1 Commit xxxxxxxxxxx Author: Erik <erik@mail.com> Date: Sat Jul 19 07:06:14 2014 -0300 Add the crop feature on website backend Now, let's find it between two dates: Erik@server:~$ git log --author=Erik --before "2014-07-20" --after "2014-07-18" Commit xxxxxxxxxxx Author: Erik <erik@mail.com> Date: Sat Jul 19 07:06:14 2014 -0300 Add the crop feature on website backend As I told you earlier, there are a lot of parameters to the git log command. You can see all of them using the git help log command. The stat parameter is really useful: Erik@server:~$ git log --author=Jim --stat Commit xxxxxxxxxxx Author: Jim <jim@mail.com> Date: Sun Jul 20 15:10:12 2014 -0300 Fix front bugs on banner   index.php | 1 +      1 file changed, 1 insertion(+) This parameter allows you to view a summary of the changes made in each commit. If you want to see the full changes, try the -p parameter. Remember that the git log command has a file parameter to restrict the search to the git log [file] file. Viewing changes There are two ways to see changes in a repository: git diff and git show. The git diff command lets you see the changes that are not committed. For example, we have an index.phpfile and replace the file content by a line. Just before the lines, you will see a plus (+) or minus (-) sign. The + sign means that content was added and the – sign denotes that it was removed: Erik@server:~$ git diff diff --git a/index.php b/index.php indexb4d22ea..748ebb2 100644 --- a/index.php +++ b/index.php @@ -1,11 +1 @@ -<html> - -<head> -<title>Git is great!</title> -</head> -<body> -<?php - echo 'Git is great'; -?> -</body> -</html> +<b> I added a line</b> If you want to analyze a commit, I suggest you to use the git show command. It will display the full list of changes of the commit: Erik@server:~$ git show commitId There is a way to do the opposite, that is, to display commits for a file with git blame: Erik@server:~$ git blameindex.php e4bac680 (Erik 2014-07-20 19:00:47 +0200 1) <b> I added a line</b> Cleaning your mistakes The first thing to know is that you can always clean your mistake with Git. Sometimes this will be hard or painful for your code, but you can do it! Let's start this section with how to remove untracked files: Erik@server:~$ git clean -n The –n option will make a dry-run (it's always important to see what will happen before you regret it). If you want to also remove directories and hidden files, use this one: Erik@server:~$ git clean -fdx With these options, you will delete new directories (-d) and hidden files (-x) and be able to force them (-f). The git reset command The git reset command will allow you to go back to a previous state (for example, commit). The git reset command has three options (soft, hard, or mixed, by default). In general, the git reset command's aim is to take the current branch, reset it to point somewhere else, and possibly bring the index and work tree along. More concretely, if the master branch (currently checked out) looks like the first row (in the following figure) and you want it to point to B and not C, you will use this command: Erik@server:~$ git reset B The following diagram shows exactly what happened with the previous command. The HEAD pointer was reset from C to B: The following table explains what the options really move: Option Head pointer Working tree Staging area Soft Yes No No Mixed Yes No Yes Hard Yes Yes Yes The three options that you can provide on the reset command can be easily explained: --hard: This option is the simplest. It will restore the content to the given commit. All the local changes will be erased. The git reset --hard command means git reset --hard HEAD, which will reset your files to the previous version and erase your local changes. --mixed: This option resets the index, but not the work tree. It will reset your local files, but the differences found during the process will be marked as local modifications if you analyze them using git status. It's very helpful if you make some bugs on previous commits and want to keep your local changes. --soft: This option will keep all your files, such as mixed, intact. If you use git status, it will appear as changes to commit. You can use this option when you have not committed files as expected, but your work is correct. So you just have to recommit it the way you want. The git reset command doesn't remove untracked files; use git clean instead. Canceling a commit The git revert command allows you to "cancel" your last unpushed commit. I used quotes around cancel because Git doesn't drop the commit; it creates a new commit that executes the opposite of your commit. A pushed commit is irreversible, so you cannot change it. Firstly, let's have a look at the last commits: Erik@server:~$ git log commite4bac680c5818c70ced1205cfc46545d48ae687e Author: Eric Pidoux Date:   Sun Jul 20 19:00:47 2014 +0200 replace all commit0335a5f13b937e8367eff35d78c259cf2c4d10f7 Author: Eric Pidoux Date:   Sun Jul 20 18:23:06 2014 +0200 commitindex.php We want to cancel the 0335… commit: Erik@server:~$ git revert 0335a5f13 Canceling this commit isn't necessary to enter the full commit ID, but just the first characters. Git will find it, but you will have to enter at least six characters to be sure that there isn't another commit that starts with the same characters. Solving merge conflicts When you are working with several branches, a conflict will probably occur while merging them. It appears if two commits from different branches modify the same content and Git isn't able to merge them. If it occurs, Git will mark the conflict and you have to resolve it. For example, Jim modified the index.html file on a feature branch and Erik has to edit it on another branch. When Erik merges the two branches, the conflict occurs. Git will tell you to edit the file to resolve the conflict. In this file, you will find the following: <<<<<<< HEAD Changes from Erik ======= Changes from Jim >>>>>>> b2919weg63bfd125627gre1911c8b08127c85f8 The <<<<<<< characters indicate the start of the merge conflict, the ====== characters indicate the break points used for comparison, and >>>>>>> indicate the end of the conflict. To resolve a conflict, you have to analyze the differences between the two changes and merge them manually. Don't forget to delete the signs added by Git. After resolving it, simply commit the changes. If your merge conflict is too complicated to resolve because you can't easily find the differences, Git provides a useful tool to help you. Git's diff helps you to find differences: Diff --git erik/mergetestjim/mergetest Index.html 88h3d45..92f62w 130634 --- erik/mergetest +++ jim/mergetest @@ -1,3 +1,4 @@ <body> +I added this code between This is the file content -I added a third line of code +And this is the last one So, what happened? The command displays some lines with the changes, with the + mark coming from origin/master; those marked with – are from your local repository, and of course, the lines without a mark are common to both repositories. Summary In this article, we covered all tips and commands that are useful to fix mistakes, resolve conflicts, search inside the commit history, and so on. Resources for Article: Further resources on this subject: Configuration [Article] Parallel Dimensions – Branching with Git [Article] Issues and Wikis in GitLab [Article]
Read more
  • 0
  • 0
  • 1544

article-image-deployment-and-post-deployment
Packt
17 Nov 2014
30 min read
Save for later

Deployment and Post Deployment

Packt
17 Nov 2014
30 min read
In this article by Shalabh Aggarwal, the author of Flask Framework Cookbook, we will talk about various application-deployment techniques, followed by some monitoring tools that are used post-deployment. (For more resources related to this topic, see here.) Deployment of an application and managing the application post-deployment is as important as developing it. There can be various ways of deploying an application, where choosing the best way depends on the requirements. Deploying an application correctly is very important from the points of view of security and performance. There are multiple ways of monitoring an application after deployment of which some are paid and others are free to use. Using them again depends on requirements and features offered by them. Each of the tools and techniques has its own set of features. For example, adding too much monitoring to an application can prove to be an extra overhead to the application and the developers as well. Similarly, missing out on monitoring can lead to undetected user errors and overall user dissatisfaction. Hence, we should choose the tools wisely and they will ease our lives to the maximum. In the post-deployment monitoring tools, we will discuss Pingdom and New Relic. Sentry is another tool that will prove to be the most beneficial of all from a developer's perspective. Deploying with Apache First, we will learn how to deploy a Flask application with Apache, which is, unarguably, the most popular HTTP server. For Python web applications, we will use mod_wsgi, which implements a simple Apache module that can host any Python applications that support the WSGI interface. Remember that mod_wsgi is not the same as Apache and needs to be installed separately. Getting ready We will start with our catalog application and make appropriate changes to it to make it deployable using the Apache HTTP server. First, we should make our application installable so that our application and all its libraries are on the Python load path. This can be done using a setup.py script. There will be a few changes to the script as per this application. The major changes are mentioned here: packages=[    'my_app',    'my_app.catalog', ], include_package_data=True, zip_safe = False, First, we mentioned all the packages that need to be installed as part of our application. Each of these needs to have an __init__.py file. The zip_safe flag tells the installer to not install this application as a ZIP file. The include_package_data statement reads from a MANIFEST.in file in the same folder and includes any package data mentioned here. Our MANIFEST.in file looks like: recursive-include my_app/templates * recursive-include my_app/static * recursive-include my_app/translations * Now, just install the application using the following command: $ python setup.py install Installing mod_wsgi is usually OS-specific. Installing it on a Debian-based distribution should be as easy as just using the packaging tool, that is, apt or aptitude. For details, refer to https://code.google.com/p/modwsgi/wiki/InstallationInstructions and https://github.com/GrahamDumpleton/mod_wsgi. How to do it… We need to create some more files, the first one being app.wsgi. This loads our application as a WSGI application: activate_this = '<Path to virtualenv>/bin/activate_this.py' execfile(activate_this, dict(__file__=activate_this))   from my_app import app as application import sys, logging logging.basicConfig(stream = sys.stderr) As we perform all our installations inside virtualenv, we need to activate the environment before our application is loaded. In the case of system-wide installations, the first two statements are not needed. Then, we need to import our app object as application, which is used as the application being served. The last two lines are optional, as they just stream the output to the the standard logger, which is disabled by mod_wsgi by default. The app object needs to be imported as application, because mod_wsgi expects the application keyword. Next comes a config file that will be used by the Apache HTTP server to serve our application correctly from specific locations. The file is named apache_wsgi.conf: <VirtualHost *>      WSGIScriptAlias / <Path to application>/flask_catalog_deployment/app.wsgi      <Directory <Path to application>/flask_catalog_deployment>        Order allow,deny        Allow from all    </Directory>   </VirtualHost> The preceding code is the Apache configuration, which tells the HTTP server about the various directories where the application has to be loaded from. The final step is to add the apache_wsgi.conf file to apache2/httpd.conf so that our application is loaded when the server runs: Include <Path to application>/flask_catalog_deployment/ apache_wsgi.conf How it works… Let's restart the Apache server service using the following command: $ sudo apachectl restart Open up http://127.0.0.1/ in the browser to see the application's home page. Any errors coming up can be seen at /var/log/apache2/error_log (this path can differ depending on OS). There's more… After all this, it is possible that the product images uploaded as part of the product creation do not work. For this, we should make a small modification to our application's configuration: app.config['UPLOAD_FOLDER'] = '<Some static absolute path>/flask_test_uploads' We opted for a static path because we do not want it to change every time the application is modified or installed. Now, we will include the path chosen in the preceding code to apache_wsgi.conf: Alias /static/uploads/ "<Some static absolute path>/flask_test_uploads/" <Directory "<Some static absolute path>/flask_test_uploads">    Order allow,deny    Options Indexes    Allow from all    IndexOptions FancyIndexing </Directory> After this, install the application and restart apachectl. See also http://httpd.apache.org/ https://code.google.com/p/modwsgi/ http://wsgi.readthedocs.org/en/latest/ https://pythonhosted.org/setuptools/setuptools.html#setting-the-zip-safe-flag Deploying with uWSGI and Nginx For those who are already aware of the usefulness of uWSGI and Nginx, there is not much that can be explained. uWSGI is a protocol as well as an application server and provides a complete stack to build hosting services. Nginx is a reverse proxy and HTTP server that is very lightweight and capable of handling virtually unlimited requests. Nginx works seamlessly with uWSGI and provides many under-the-hood optimizations for better performance. Getting ready We will use our application from the last recipe, Deploying with Apache, and use the same app.wsgi, setup.py, and MANIFEST.in files. Also, other changes made to the application's configuration in the last recipe will apply to this recipe as well. Disable any other HTTP servers that might be running, such as Apache and so on. How to do it… First, we need to install uWSGI and Nginx. On Debian-based distributions such as Ubuntu, they can be easily installed using the following commands: # sudo apt-get install nginx # sudo apt-get install uWSGI You can also install uWSGI inside a virtualenv using the pip install uWSGI command. Again, these are OS-specific, so refer to the respective documentations as per the OS used. Make sure that you have an apps-enabled folder for uWSGI, where we will keep our application-specific uWSGI configuration files, and a sites-enabled folder for Nginx, where we will keep our site-specific configuration files. Usually, these are already present in most installations in the /etc/ folder. If not, refer to the OS-specific documentations to figure out the same. Next, we will create a file named uwsgi.ini in our application: [uwsgi] http-socket   = :9090 plugin   = python wsgi-file = <Path to application>/flask_catalog_deployment/app.wsgi processes   = 3 To test whether uWSGI is working as expected, run the following command: $ uwsgi --ini uwsgi.ini The preceding file and command are equivalent to running the following command: $ uwsgi --http-socket :9090 --plugin python --wsgi-file app.wsgi Now, point your browser to http://127.0.0.1:9090/; this should open up the home page of the application. Create a soft link of this file to the apps-enabled folder mentioned earlier using the following command: $ ln -s <path/to/uwsgi.ini> <path/to/apps-enabled> Before moving ahead, edit the preceding file to replace http-socket with socket. This changes the protocol from HTTP to uWSGI (read more about it at http://uwsgi-docs.readthedocs.org/en/latest/Protocol.html). Now, create a new file called nginx-wsgi.conf. This contains the Nginx configuration needed to serve our application and the static content: location /{    include uwsgi_params;    uwsgi_pass 127.0.0.1:9090; } location /static/uploads/{    alias <Some static absolute path>/flask_test_uploads/; } In the preceding code block, uwsgi_pass specifies the uWSGI server that needs to be mapped to the specified location. Create a soft link of this file to the sites-enabled folder mentioned earlier using the following command: $ ln -s <path/to/nginx-wsgi.conf> <path/to/sites-enabled> Edit the nginx.conf file (usually found at /etc/nginx/nginx.conf) to add the following line inside the first server block before the last }: include <path/to/sites-enabled>/*; After all of this, reload the Nginx server using the following command: $ sudo nginx -s reload Point your browser to http://127.0.0.1/ to see the application that is served via Nginx and uWSGI. The preceding instructions of this recipe can vary depending on the OS being used and different versions of the same OS can also impact the paths and commands used. Different versions of these packages can also have some variations in usage. Refer to the documentation links provided in the next section. See also Refer to http://uwsgi-docs.readthedocs.org/en/latest/ for more information on uWSGI. Refer to http://nginx.com/ for more information on Nginx. There is a good article by DigitalOcean on this. I advise you to go through this to have a better understanding of the topic. It is available at https://www.digitalocean.com/community/tutorials/how-to-deploy-python-wsgi-applications-using-uwsgi-web-server-with-nginx. To get an insight into the difference between Apache and Nginx, I think the article by Anturis at https://anturis.com/blog/nginx-vs-apache/ is pretty good. Deploying with Gunicorn and Supervisor Gunicorn is a WSGI HTTP server for Unix. It is very simple to implement, ultra light, and fairly speedy. Its simplicity lies in its broad compatibility with various web frameworks. Supervisor is a monitoring tool that controls various child processes and handles the starting/restarting of these child processes when they exit abruptly due to some reason. It can be extended to control the processes via the XML-RPC API over remote locations without logging in to the server (we won't discuss this here as it is out of the scope of this book). One thing to remember is that these tools can be used along with the other tools mentioned in the applications in the previous recipe, such as using Nginx as a proxy server. This is left to you to try on your own. Getting ready We will start with the installation of both the packages, that is, gunicorn and supervisor. Both can be directly installed using pip: $ pip install gunicorn $ pip install supervisor How to do it… To check whether the gunicorn package works as expected, just run the following command from inside our application folder: $ gunicorn -w 4 -b 127.0.0.1:8000 my_app:app After this, point your browser to http://127.0.0.1:8000/ to see the application's home page. Now, we need to do the same using Supervisor so that this runs as a daemon and will be controlled by Supervisor itself rather than human intervention. First of all, we need a Supervisor configuration file. This can be achieved by running the following command from virtualenv. Supervisor, by default, looks for an etc folder that has a file named supervisord.conf. In system-wide installations, this folder is /etc/, and in virtualenv, it will look for an etc folder in virtualenv and then fall back to /etc/: $ echo_supervisord_conf > etc/supervisord.conf The echo_supervisord_conf program is provided by Supervisor; it prints a sample config file to the location specified. This command will create a file named supervisord.conf in the etc folder. Add the following block in this file: [program:flask_catalog] command=<path/to/virtualenv>/bin/gunicorn -w 4 -b 127.0.0.1:8000 my_app:app directory=<path/to/virtualenv>/flask_catalog_deployment user=someuser # Relevant user autostart=true autorestart=true stdout_logfile=/tmp/app.log stderr_logfile=/tmp/error.log Make a note that one should never run the applications as a root user. This is a huge security flaw in itself as the application crashes, which can harm the OS itself. How it works… Now, run the following commands: $ supervisord $ supervisorctl status flask_catalog   RUNNING   pid 40466, uptime 0:00:03 The first command invokes the supervisord server, and the next one gives a status of all the child processes. The tools discussed in this recipe can be coupled with Nginx to serve as a reverse proxy server. I suggest that you try it by yourself. Every time you make a change to your application and then wish to restart Gunicorn in order for it to reflect the changes, run the following command: $ supervisorctl restart all You can also give specific processes instead of restarting everything: $ supervisorctl restart flask_catalog See also http://gunicorn-docs.readthedocs.org/en/latest/index.html http://supervisord.org/index.html Deploying with Tornado Tornado is a complete web framework and a standalone web server in itself. Here, we will use Flask to create our application, which is basically a combination of URL routing and templating, and leave the server part to Tornado. Tornado is built to hold thousands of simultaneous standing connections and makes applications very scalable. Tornado has limitations while working with WSGI applications. So, choose wisely! Read more at http://www.tornadoweb.org/en/stable/wsgi.html#running-wsgi-apps-on-tornado-servers. Getting ready Installing Tornado can be simply done using pip: $ pip install tornado How to do it… Next, create a file named tornado_server.py and put the following code in it: from tornado.wsgi import WSGIContainer from tornado.httpserver import HTTPServer from tornado.ioloop import IOLoop from my_app import app   http_server = HTTPServer(WSGIContainer(app)) http_server.listen(5000) IOLoop.instance().start() Here, we created a WSGI container for our application; this container is then used to create an HTTP server, and the application is hosted on port 5000. How it works… Run the Python file created in the previous section using the following command: $ python tornado_server.py Point your browser to http://127.0.0.1:5000/ to see the home page being served. We can couple Tornado with Nginx (as a reverse proxy to serve static content) and Supervisor (as a process manager) for the best results. It is left for you to try this on your own. Using Fabric for deployment Fabric is a command-line tool in Python; it streamlines the use of SSH for application deployment or system-administration tasks. As it allows the execution of shell commands on remote servers, the overall process of deployment is simplified, as the whole process can now be condensed into a Python file, which can be run whenever needed. Therefore, it saves the pain of logging in to the server and manually running commands every time an update has to be made. Getting ready Installing Fabric can be simply done using pip: $ pip install fabric We will use the application from the Deploying with Gunicorn and Supervisor recipe. We will create a Fabric file to perform the same process to the remote server. For simplicity, let's assume that the remote server setup has been already done and all the required packages have also been installed with a virtualenv environment, which has also been created. How to do it… First, we need to create a file called fabfile.py in our application, preferably at the application's root directory, that is, along with the setup.py and run.py files. Fabric, by default, expects this filename. If we use a different filename, then it will have to be explicitly specified while executing. A basic Fabric file will look like: from fabric.api import sudo, cd, prefix, run   def deploy_app():    "Deploy to the server specified"    root_path = '/usr/local/my_env'      with cd(root_path):        with prefix("source %s/bin/activate" % root_path):            with cd('flask_catalog_deployment'):                run('git pull')                run('python setup.py install')              sudo('bin/supervisorctl restart all') Here, we first moved into our virtualenv, activated it, and then moved into our application. Then, the code is pulled from the Git repository, and the updated application code is installed using setup.py install. After this, we restarted the supervisor processes so that the updated application is now rendered by the server. Most of the commands used here are self-explanatory, except prefix, which wraps all the succeeding commands in its block with the command provided. This means that the command to activate virtualenv will run first and then all the commands in the with block will execute with virtualenv activated. The virtualenv will be deactivated as soon as control goes out of the with block. How it works… To run this file, we need to provide the remote server where the script will be executed. So, the command will look something like: $ fab -H my.remote.server deploy_app Here, we specified the address of the remote host where we wish to deploy and the name of the method to be called from the fab script. There's more… We can also specify the remote host inside our fab script, and this can be good idea if the deployment server remains the same most of the times. To do this, add the following code to the fab script: from fabric.api import settings   def deploy_app_to_server():    "Deploy to the server hardcoded"    with settings(host_string='my.remote.server'):        deploy_app() Here, we have hardcoded the host and then called the method we created earlier to start the deployment process. S3 storage for file uploads Amazon explains S3 as the storage for the Internet that is designed to make web-scale computing easier for developers. S3 provides a very simple interface via web services; this makes storage and retrieval of any amount of data very simple at any time from anywhere on the Internet. Until now, in our catalog application, we saw that there were issues in managing the product images uploaded as a part of the creating process. The whole headache will go away if the images are stored somewhere globally and are easily accessible from anywhere. We will use S3 for the same purpose. Getting ready Amazon offers boto, a complete Python library that interfaces with Amazon Web Services via web services. Almost all of the AWS features can be controlled using boto. It can be installed using pip: $ pip install boto How to do it… Now, we should make some changes to our existing catalog application to accommodate support for file uploads and retrieval from S3. First, we need to store the AWS-specific configuration to allow boto to make calls to S3. Add the following statements to the application's configuration file, that is, my_app/__init__.py: app.config['AWS_ACCESS_KEY'] = 'Amazon Access Key' app.config['AWS_SECRET_KEY'] = 'Amazon Secret Key' app.config['AWS_BUCKET'] = 'flask-cookbook' Next, we need to change our views.py file: from boto.s3.connection import S3Connection This is the import that we need from boto. Next, replace the following two lines in create_product(): filename = secure_filename(image.filename) image.save(os.path.join(app.config['UPLOAD_FOLDER'], filename)) Replace these two lines with: filename = image.filename conn = S3Connection(    app.config['AWS_ACCESS_KEY'], app.config['AWS_SECRET_KEY'] ) bucket = conn.create_bucket(app.config['AWS_BUCKET']) key = bucket.new_key(filename) key.set_contents_from_file(image) key.make_public() key.set_metadata(    'Content-Type', 'image/' + filename.split('.')[-1].lower() ) The last change will go to our product.html template, where we need to change the image src path. Replace the original img src statement with the following statement: <img src="{{ 'https://s3.amazonaws.com/' + config['AWS_BUCKET'] + '/' + product.image_path }}"/> How it works… Now, run the application as usual and create a product. When the created product is rendered, the product image will take a bit of time to come up as it is now being served from S3 (and not from a local machine). If this happens, then the integration with S3 has been successfully done. Deploying with Heroku Heroku is a cloud application platform that provides an easy and quick way to build and deploy web applications. Heroku manages the servers, deployment, and related operations while developers spend their time on developing applications. Deploying with Heroku is pretty simple with the help of the Heroku toolbelt, which is a bundle of some tools that make deployment with Heroku a cakewalk. Getting ready We will proceed with the application from the previous recipe that has S3 support for uploads. As mentioned earlier, the first step will be to download the Heroku toolbelt, which can be downloaded as per the OS from https://toolbelt.heroku.com/. Once the toolbelt is installed, a certain set of commands will be available at the terminal; we will see them later in this recipe. It is advised that you perform Heroku deployment from a fresh virtualenv where we have only the required packages for our application installed and nothing else. This will make the deployment process faster and easier. Now, run the following command to log in to your Heroku account and sync your machined SSH key with the server: $ heroku login Enter your Heroku credentials. Email: shalabh7777@gmail.com Password (typing will be hidden): Authentication successful. You will be prompted to create a new SSH key if one does not exist. Proceed accordingly. Remember! Before all this, you need to have a Heroku account on available on https://www.heroku.com/. How to do it… Now, we already have an application that needs to be deployed to Heroku. First, Heroku needs to know the command that it needs to run while deploying the application. This is done in a file named Procfile: web: gunicorn -w 4 my_app:app Here, we will tell Heroku to run this command to run our web application. There are a lot of different configurations and commands that can go into Procfile. For more details, read the Heroku documentation. Heroku needs to know the dependencies that need to be installed in order to successfully install and run our application. This is done via the requirements.txt file: Flask==0.10.1 Flask-Restless==0.14.0 Flask-SQLAlchemy==1.0 Flask-WTF==0.10.0 Jinja2==2.7.3 MarkupSafe==0.23 SQLAlchemy==0.9.7 WTForms==2.0.1 Werkzeug==0.9.6 boto==2.32.1 gunicorn==19.1.1 itsdangerous==0.24 mimerender==0.5.4 python-dateutil==2.2 python-geoip==1.2 python-geoip-geolite2==2014.0207 python-mimeparse==0.1.4 six==1.7.3 wsgiref==0.1.2 This file contains all the dependencies of our application, the dependencies of these dependencies, and so on. An easy way to generate this file is using the pip freeze command: $ pip freeze > requirements.txt This will create/update the requirements.txt file with all the packages installed in virtualenv. Now, we need to create a Git repo of our application. For this, we will run the following commands: $ git init $ git add . $ git commit -m "First Commit" Now, we have a Git repo with all our files added. Make sure that you have a .gitignore file in your repo or at a global level to prevent temporary files such as .pyc from being added to the repo. Now, we need to create a Heroku application and push our application to Heroku: $ heroku create Creating damp-tor-6795... done, stack is cedar http://damp-tor-6795.herokuapp.com/ | git@heroku.com:damp-tor- 6795.git Git remote heroku added $ git push heroku master After the last command, a whole lot of stuff will get printed on the terminal; this will indicate all the packages being installed and finally, the application being launched. How it works… After the previous commands have successfully finished, just open up the URL provided by Heroku at the end of deployment in a browser or run the following command: $ heroku open This will open up the application's home page. Try creating a new product with an image and see the image being served from Amazon S3. To see the logs of the application, run the following command: $ heroku logs There's more… There is a glitch with the deployment we just did. Every time we update the deployment via the git push command, the SQLite database gets overwritten. The solution to this is to use the Postgres setup provided by Heroku itself. I urge you to try this by yourself. Deploying with AWS Elastic Beanstalk In the last recipe, we saw how deployment to servers becomes easy with Heroku. Similarly, Amazon has a service named Elastic Beanstalk, which allows developers to deploy their application to Amazon EC2 instances as easily as possible. With just a few configuration options, a Flask application can be deployed to AWS using Elastic Beanstalk in a couple of minutes. Getting ready We will start with our catalog application from the previous recipe, Deploying with Heroku. The only file that remains the same from this recipe is requirement.txt. The rest of the files that were added as a part of that recipe can be ignored or discarded for this recipe. Now, the first thing that we need to do is download the AWS Elastic Beanstalk command-line tool library from the Amazon website (http://aws.amazon.com/code/6752709412171743). This will download a ZIP file that needs to be unzipped and placed in a suitable place, preferably your workspace home. The path of this tool should be added to the PATH environment so that the commands are available throughout. This can be done via the export command as shown: $ export PATH=$PATH:<path to unzipped EB CLI package>/eb/linux/python2.7/ This can also be added to the ~/.profile or ~/.bash_profile file using: export PATH=$PATH:<path to unzipped EB CLI package>/eb/linux/python2.7/ How to do it… There are a few conventions that need to be followed in order to deploy using Beanstalk. Beanstalk assumes that there will be a file called application.py, which contains the application object (in our case, the app object). Beanstalk treats this file as the WSGI file, and this is used for deployment. In the Deploying with Apache recipe, we had a file named app.wgsi where we referred our app object as application because apache/mod_wsgi needed it to be so. The same thing happens here too because Amazon, by default, deploys using Apache behind the scenes. The contents of this application.py file can be just a few lines as shown here: from my_app import app as application import sys, logging logging.basicConfig(stream = sys.stderr) Now, create a Git repo in the application and commit with all the files added: $ git init $ git add . $ git commit -m "First Commit" Make sure that you have a .gitignore file in your repo or at a global level to prevent temporary files such as .pyc from being added to the repo. Now, we need to deploy to Elastic Beanstalk. Run the following command to do this: $ eb init The preceding command initializes the process for the configuration of your Elastic Beanstalk instance. It will ask for the AWS credentials followed by a lot of other configuration options needed for the creation of the EC2 instance, which can be selected as needed. For more help on these options, refer to http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_Python_flask.html. After this is done, run the following command to trigger the creation of servers, followed by the deployment of the application: $ eb start Behind the scenes, the preceding command creates the EC2 instance (a volume), assigns an elastic IP, and then runs the following command to push our application to the newly created server for deployment: $ git aws.push This will take a few minutes to complete. When done, you can check the status of your application using the following command: $ eb status –verbose Whenever you need to update your application, just commit your changes using the git and push commands as follows: $ git aws.push How it works… When the deployment process finishes, it gives out the application URL. Point your browser to it to see the application being served. Yet, you will find a small glitch with the application. The static content, that is, the CSS and JS code, is not being served. This is because the static path is not correctly comprehended by Beanstalk. This can be simply fixed by modifying the application's configuration on your application's monitoring/configuration page in the AWS management console. See the following screenshots to understand this better: Click on the Configuration menu item in the left-hand side menu. Notice the highlighted box in the preceding screenshot. This is what we need to change as per our application. Open Software Settings. Change the virtual path for /static/, as shown in the preceding screenshot. After this change is made, the environment created by Elastic Beanstalk will be updated automatically, although it will take a bit of time. When done, check the application again to see the static content also being served correctly. Application monitoring with Pingdom Pingdom is a website-monitoring tool that has the USP of notifying you as soon as your website goes down. The basic idea behind this tool is to constantly ping the website at a specific interval, say, 30 seconds. If a ping fails, it will notify you via an e-mail, SMS, tweet, or push notifications to mobile apps, which inform that your site is down. It will keep on pinging at a faster rate until the site is back up again. There are other monitoring features too, but we will limit ourselves to uptime checks in this book. Getting ready As Pingdom is a SaaS service, the first step will be to sign up for an account. Pingdom offers a free trial of 1 month in case you just want to try it out. The website for the service is https://www.pingdom.com. We will use the application deployed to AWS in the Deploying with AWS Elastic Beanstalk recipe to check for uptime. Here, Pingdom will send an e-mail in case the application goes down and will send an e-mail again when it is back up. How to do it… After successful registration, create a check for time. Have a look at the following screenshot: As you can see, I already added a check for the AWS instance. To create a new check, click on the ADD NEW button. Fill in the details asked by the form that comes up. How it works… After the check is successfully created, try to break the application by consciously making a mistake somewhere in the code and then deploying to AWS. As soon as the faulty application is deployed, you will get an e-mail notifying you of this. This e-mail will look like: Once the application is fixed and put back up again, the next e-mail should look like: You can also check how long the application has been up and the downtime instances from the Pingdom administration panel. Application performance management and monitoring with New Relic New Relic is an analytics software that provides near real-time operational and business analytics related to your application. It provides deep analytics on the behavior of the application from various aspects. It does the job of a profiler as well as eliminating the need to maintain extra moving parts in the application. It actually works in a scenario where our application sends data to New Relic rather than New Relic asking for statistics from our application. Getting ready We will use the application from the last recipe, which is deployed to AWS. The first step will be to sign up with New Relic for an account. Follow the simple signup process, and upon completion and e-mail verification, it will lead to your dashboard. Here, you will have your license key available, which we will use later to connect our application to this account. The dashboard should look like the following screenshot: Here, click on the large button named Reveal your license key. How to do it… Once we have the license key, we need to install the newrelic Python library: $ pip install newrelic Now, we need to generate a file called newrelic.ini, which will contain details regarding the license key, the name of our application, and so on. This can be done using the following commands: $ newrelic-admin generate-config LICENSE-KEY newrelic.ini In the preceding command, replace LICENSE-KEY with the actual license key of your account. Now, we have a new file called newrelic.ini. Open and edit the file for the application name and anything else as needed. To check whether the newrelic.ini file is working successfully, run the following command: $ newrelic-admin validate-config newrelic.ini This will tell us whether the validation was successful or not. If not, then check the license key and its validity. Now, add the following lines at the top of the application's configuration file, that is, my_app/__init__.py in our case. Make sure that you add these lines before anything else is imported: import newrelic.agent newrelic.agent.initialize('newrelic.ini') Now, we need to update the requirements.txt file. So, run the following command: $ pip freeze > requirements.txt After this, commit the changes and deploy the application to AWS using the following command: $ git aws.push How it works… Once the application is successfully updated on AWS, it will start sending statistics to New Relic, and the dashboard will have a new application added to it. Open the application-specific page, and a whole lot of statistics will come across. It will also show which calls have taken the most amount of time and how the application is performing. You will also see multiple tabs that correspond to a different type of monitoring to cover all the aspects. Summary In this article, we have seen the various techniques used to deploy and monitor Flask applications. Resources for Article: Further resources on this subject: Understanding the Python regex engine [Article] Exploring Model View Controller [Article] Plotting Charts with Images and Maps [Article]
Read more
  • 0
  • 0
  • 2187

article-image-posting-reviews-ratings-and-photos
Packt
17 Nov 2014
15 min read
Save for later

Posting Reviews, Ratings, and Photos

Packt
17 Nov 2014
15 min read
In this article, by Hussein Nasser, the author of the book Building Web Applications with ArcGIS, we will learn how to perform editing on services by adding three features, posting reviews, ratings, and uploading pictures for the restaurant. (For more resources related to this topic, see here.) Configuring enterprise Geodatabase Unfortunately, editing process requires some more configurations to be done, and the current service, wouldn't do the trick. The reason is that the service is using a local database file; however, editing GIS data on the Web requires either an enterprise database running on a database management server such as the SQL server, or Oracle, or an ArcGIS Online feature service. Setting up the enterprise geodatabase server is out of the scope of this book. However, you can grab my book Learning ArcGIS Geodatabase, Packt Publishing, and walk through the step-by-step full guide to set up your own enterprise geodatabase with Microsoft SQL Server Express. If you have an existing enterprise geodatabase server, you can use it. I will be using SQL Server Express 2012 SP1. Connecting to the Geodatabase First, we need to establish a connection to the enterprise server. For that we will use a username that has full editing capabilities. I will be using the system administrator (SA) user sa for the SQL server. Note that the SA user has full access to the database; we have used it in this book for simplicity to avoid assigning privileges. In an ideal situation, you would create a user and assign him the correct permissions to read or write into the subject tables. My book, Learning ArcGIS Geodatabase, Packt Publishing, addresses all these issues thoroughly. Follow these steps to establish a connection to your enterprise geodatabase: Open ArcCatalog and expand Database Connections in the Catalog Tree panel. Double-click on Add Database Connection. In the Database Platform option select SQL Server (or your database provider). Type the name of the server hosting the database; in my case, it is the same server arcgismachine. Select Database authentication in Authentication Type and provide the database credentials for the sa user or any user who has administrator privileges on the database. You can use the SDE user as well. Select your Database from the drop-down list and click on OK as shown in the following screenshot: Rename the connection to sa@arcgismachine.sde. We are going to reference this later in the article. Don't close ArcCatalog yet, we will be needing it. You can learn more about ArcGIS database connections and how to create them against different databases from the link http://qr.net/arcgisdb. Copying Bestaurants' data to the server Now that we have our connection ready, we need to copy the Bestaurants data into our new database. Follow these steps: From Catalog Tree, right-click on the Folder connection and select Connect to Folder. Browse to C:2955OT, the folder we created in the book Building Web Application with ArcGIS, and click on OK. This will allow us to access our Bestaurants data. From Folder Connection, expand C:2955OT and browse to 2955OT_01_FilesBestaurants.gdb. Click on Bestaurants.gdb, use the Shift key to select Belize_Landbase and Food_and_Drinks. Then right-click and select Copy as shown in the following screenshot: Double-click on the ags@arcgismachine.sde connection to open it. Right-click on the connection and select Paste. You will be prompted with the following dialog box that will show you what will be copied. Note that related data was also imported. This will paste the data to our new database. After the copying is completed, close ArcCatalog. Publishing feature service Our old service won't work in this article; the reason is that it was pointing to a local database which does not support editing on ArcGIS for Server. That is why we migrated our Bestaurants data to the enterprise geodatabase. It is time to publish a brand new service; it will look the same but will just behave differently. First, we need to open our Belize.mxd map document and point our new database. Second, we will register the database with ArcGIS for Server; finally, we will publish the service. Setting the Source to the Enterprise Geodatabase In order to publish the new service, we have to first create a map document which points to the enterprise geodatabase. Follow these steps to do so: Browse to and open 2955OT_05_FilesBelize.mxd with ArcMap. You can simply double-click on the file. Next, we set the source of our layers from the Table of Contents in ArcMap. Click on List by Source as shown in this screenshot: Double-click on the Food_and_Drinks layer to open the Layer properties. In the Layer Properties window, click on Set Data Source. From the Data Source dialog, browse to the sa@arcgismachine.sde connection and select the sdedb.DBO.Food_and_Drinks object, and then click on Add as illustrated in this screenshot: Click OK to apply the changes. Do the same for the rest of the objects, Landbase and VENUES_REVIEW, in your map selecting the matching objects in the destination connection. The final source list should look like the following, nothing pointing to the local data. Keep ArcMap open for the next step. You can also modify the data sources in an ArcMap document using ArcCatalog. In ArcCatalog, navigate to the map document in Catalog Tree, right-click on the map document, and choose the Set Data Sources option from the pop-up menu. Using this allows you to set the data source for individual and/or all layers at one time. Publishing the map document Now that we have our map document ready, it is time for us to publish it. However, we still need to perform one more step, which is database registration. This step will cause the data to persist in the server, which makes it ready for editing. Follow these steps to publish the map document: From the File menu, point to Share As and click on Service. Select Overwrite an existing service, because we want to replace our old Bestaurants service. Click on Next. Select the Bestaurants service and click on Continue. From the Service Editor window, click on Capabilities. Check the Feature Access capability to enable editing on this service as shown in the following screenshot: Click on Analyze. This will show an error that we have to fix. The error is Feature service requires a registered database. This error can be solved by right-clicking on it and selecting Show Data Store Registration Page, as shown in the following screenshot: In the Data Store window, click on the plus sign to add a new database. In the Register Database window, enter in Bestaurants in the Name textbox. Click on Import and select the sa@arcgismachine connection. Make sure the Same as publisher database connection option is checked as shown in the following screenshot: Click on OK in the Register Database window and the Data Store window. Click on Analyse again; this time you shouldn't get any errors. Go ahead and Publish your service. Close ArcMap. You can read more about registering databases at: http://qr.net/registerdb. Testing the web application with the new service We have updated our Bestaurants service, but it should work fine so far. Hence, just make sure that we run your application under http://arcgismachine/mybestaurants.html. You can find the latest code at 2955OT_05_FilesCodebestaurants01_beforeediting.html, copy it to c:inetpubwwwroot and rename it accordingly. Adding ArcGIS's editing capabilities Currently, the food and drinks layer is being fetched as a read-only map service located at the URL: http://arcgismachine:6080/arcgis/rest/services/Bestaurants/MapServer/0. The only difference is that it points to the enterprise geodatabase. You can see the following code in mybestaurants.html which confirms that: //load the food and drinks layer into an objectlyr_foodanddrinks = new esri.layers.FeatureLayer("http://arcgismachine:6080/arcgis/rest/services/Bestaurants/MapServer/0", { outFields: ["*"] }); We won't be able to edit MapServerlayer, we have to use FeatureServer. We will show how to change this in the next section. Adding or updating records in a service with ArcGIS JavaScript API is simple. We only need to use the applyEdits method on the FeatureLayer object. This method accepts a record and some parameters for the response; what we are interested in is to insert a record. The following code shows how to prepare a record with three fields for this function: varnewrecord = {attributes:{   FIELD1: VALUE1,   FIELD2: VALUE2,   FIELD3: VALUE3}}; For instance, if I want to create a new record that has rating and review, I populate them as follows: varnewrecord = {attributes:{   REVIEW: "This is a sample review",   RATING: 2,   USER: "Hussein"}}; And to add this record, we simply call applyEdits on the corresponding layer and pass the newrecord object as shown in the following code snippet: var layer = new esri.layers.FeatureLayer("URL");layer.applyEdits([newrecord], null, null, null, null); Posting reviews and ratings The first thing we have to add is to point our food and drinks layer to its feature server instead of the map server to allow editing. This can be achieved by using the following URL instead, simply replace MapServer with FeatureServer: http://arcgismachine:6080/arcgis/rest/services/Bestaurants/FeatureServer/0 Follow these steps to perform the first change towards editing our service: Edit mybestaurants.html. Find the food and drinks layer initialization and point it to the feature service instead using the following code snippet: //load the food and drinks layer into an objectlyr_foodanddrinks = new esri.layers.FeatureLayer ("http://arcgismachine:6080/arcgis/rest/services /Bestaurants/FeatureServer/0", { outFields: ["*"] }); The review and rating fields can be found in the VENUES_REVIEW table. So we can add a single record that has a review and a rating, and send it to the service. However, we need to prepare the necessary controls that will eventually populate and add a record to the reviews table. Let's modify the ShowResults function of our query so that each restaurant shows two textboxes: one for review and one for the rating. We will also add a button so that we can call the addnewreview()function that will add the review. Each control will be identified with the object ID of the restaurant as shown in the following code: //display the reatingresulthtml = resulthtml + "<b>Rating:</b> " + record.attributes["RATING"];//create a place holder for each review to be populated laterresulthtml = resulthtml + "<div id = 'review" + record.attributes["OBJECTID"] + "'></div>";//create a place holder for each attachment picture to be populated laterresulthtml = resulthtml + "<div id = 'picture" + record.attributes["OBJECTID"] + "'></div>";//create text box for the review marked with the objectidresulthtml = resulthtml + "<br>Review: <input type = 'text' id = 'txtreview" + record.attributes["OBJECTID"] + "'>";//another one for the rating.resulthtml = resulthtml + "<br>Rating: <input type = 'text' id = 'txtrating" + record.attributes["OBJECTID"] + "'>"; //and a button to call the function addnewreviewresulthtml = resulthtml + "<br><input type = 'button' value = 'Add' onclick = 'addnewreview(" + record.attributes["OBJECTID"] + ")'>"; Now, we need to write the addnewreview function. This function accepts an object ID and adds a record matching that object to the reviews table. I have placed the three empty variables: object ID, review, and rating, and prepared the template to write a new record. I also created a feature layer of our VENUES_REVIEW table: functionaddnewreview(oid){varobjectid;var review;var rating;varnewReview ={   attributes:   {     VENUE_OBJECTID: objectid,     REVIEW: review,     RATING: rating   }};//open the review tablevarreviewtable = new esri.layers.FeatureLayer("http://arcgismachine:6080/arcgis/rest/services/Bestaurants/FeatureServer/2");//apply edits and pass the review recordreviewtable.applyEdits([newReview], null, null, null, null);} You might have guessed how to obtain the review, rating, and object ID. The object ID is passed, so that is easy. The review can be obtained by searching for txtreview + objectid. A similar search can be used for rating. Let's also add a message to see if things went fine: functionaddnewreview(oid){varobjectid = oid;var review =document.getElementById('txtreview' +    oid).value;var rating = document.getElementById('txtrating' +    oid).value;varnewReview ={   attributes:   {     VENUE_OBJECTID: objectid,     REVIEW: review,     RATING: rating   }};//open the review tablevarreviewtable = new esri.layers.FeatureLayer   ("http://arcgismachine:6080/arcgis/rest/services   /Bestaurants/FeatureServer/2");//apply edits and pass the review recordreviewtable.applyEdits([newReview], null, null,   null, null);alert("Review has been added");} You can also add the user to your record in a similar way: varnewReview ={attributes:   OBJECTID: objectid,   REVIEW: review,   USER_: "Hussein"   RATING: rating}}; It is time for us to save and run our new application. Do a search on Fern, then write a review, add a rating, and then click on Add. This should add a record to Fern Diner. You can always check if your record is added or not from ArcCatalog. This can be seen in the following screenshot: You can find the latest code at 2955OT_05_FilesCodebestaurants02_addreviews.html. Uploading pictures Uploading attachments to a service can be achieved by calling the addAttachment method on the feature layer object. However, we have to make some changes in our ShowResults function to ask the user to browse for a file. For that, we will need to use the file's HTML object, but we have to encapsulate it in a form tagged by the object ID of the restaurant we want to upload the pictures for. The file object should be named attachment so that the addAttachment method can find it. Follow these steps to add the upload pictures' logic: Edit the mybestaurants.html file and add the following code to your ShowResults function: //browse for a picture for this restaurantresulthtml = resulthtml + "<form id = 'frm" + record.attributes["OBJECTID"] + "'><input type = 'file' name = 'attachment'/></form>";//and a button to call the function addnewreviewresulthtml = resulthtml + "<br><input type = 'button' value = 'Add' onclick = 'addnewreview(" + record.attributes["OBJECTID"] + ")'>";    //new lineresulthtml = resulthtml + "<br><br>"; We will make it so that when the user clicks on Add, the attachment is also added along with the review for simplicity. The AddAttachment function takes the object ID of the restaurant which you want to upload the picture to, and a form HTML element which contains the file element named attachment: functionaddnewreview(oid){varobjectid = oid;var review = document.getElementById('txtreview' +    oid).value;var rating = document.getElementById('txtrating' +    oid).value;varnewReview ={   attributes:   {     VENUE_OBJECTID: objectid,     USER_: "Hussein",     REVIEW: review,     RATING: rating   }};//open the review tablevarreviewtable = new esri.layers.FeatureLayer("http://arcgismachine:6080/arcgis/rest/services   /Bestaurants/FeatureServer/2");//apply edits and pass the review recordreviewtable.applyEdits   ([newReview], null, null, null, null);//add attachmentlyr_foodanddrinks.addAttachment(oid,    document.getElementById("frm" + oid) , null, null);alert("Review and picture has been added");} Save and run the code. Search for Haulze Restaurant. This one doesn't have an attachment, so go ahead, write a review, and upload a picture. Run the query again and you should see your picture. This is shown in the following screenshot: The final code can be found at 2955OT_05_FilesCodebestaurants03_uploadpicture.html. The final touches This is where we add some web enhancements to the application, things that you as a web developer can do. We will update the status bar, make part of the page scrollable, change the rating into star icons, and do some fine-tuning of the interface. I have already implemented these changes in the final version of the application, which can be found at 2955OT_05_FilesCodebestaurantsfinal.html. These changes don't have anything to do with ArcGIS development; it is pure HTML and JavaScript. The final application should look as shown in the following screenshot: Summary In this article, we have put the final touches to the Bestaurants ArcGIS web application by adding rating, reviews, and uploading pictures to the ArcGIS service. We have learned that editing can only be done for the feature services on the data hosted on enterprise geodatabases that is why we had to set up our own. We have copied the data to a new server, and modified the source document to point to that server. Then we republished the service with feature-access capability to enable editing. We finally added the necessary JavaScript API code to write reviews and upload pictures to the website. With these features, we have completed the Bestaurants project requirements. This is the end of this book but it is only the beginning of great potential applications that you will be developing using the skill set you acquired in the course of this journey. You can now be confident in pursuing more advanced ArcGIS JavaScript APIs from the Esri website (resources.arcgis.com) which is a good place to start. There are hundreds of methods and functions in the API. However, keep in mind that you only need to learn what you really need and require to. We have managed to complete an entire website with a handful of APIs. Take the next project, analyze the requirements, see what APIs you need, and learn them. That is my advice. My inbox is always ready for suggestions and thoughts, and of course, questions. Resources for Article: Further resources on this subject: Enterprise Geodatabase [article] Server Logs [article] Adding Graphics to the Map [article]
Read more
  • 0
  • 0
  • 2168
article-image-cross-site-request-forgery
Packt
17 Nov 2014
9 min read
Save for later

Cross-site Request Forgery

Packt
17 Nov 2014
9 min read
In this article by Y.E Liang, the author of JavaScript Security, we will cover cross-site forgery. This topic is not exactly new. In this article, we will go deeper into cross-site forgery and learn the various techniques of defending against it. (For more resources related to this topic, see here.) Introducing cross-site request forgery Cross-site request forgery (CSRF) exploits the trust that a site has in a user's browser. It is also defined as an attack that forces an end user to execute unwanted actions on a web application in which the user is currently authenticated. Examples of CSRF We will now take a look at a basic CSRF example: Go to the source code and change the directory. Run the following command: python xss_version.py Remember to start your MongoDB process as well. Next, open external.html found in templates, in another host, say http://localhost:8888. You can do this by starting the server, which can be done by running pyton xss_version.py –port=8888, and then visiting http://loaclhost:8888/todo_external. You will see the following screenshot: Adding a new to-do item Click on Add To Do, and fill in a new to-do item, as shown in the following screenshot: Adding a new to-do item and posting it Next, click on Submit. Going back to your to-do list app at http://localhost:8000/todo and refreshing it, you will see the new to-do item added to the database, as shown in the following screenshot: To-do item is added from an external app; this is dangerous! To attack the to-do list app, all we need to do is add a new item that contains a line of JavaScript, as shown in the following screenshot: Adding a new to do for the Python version Now, click on Submit. Then, go back to your to-do app at http://localhost:8000/todo, and you will see two subsequent alerts, as shown in the following screenshot: Successfully injected JavaScript part 1 So here's the first instance where CSRF happens: Successfully injected JavaScript part 2 Take note that this can happen to the other backend written in other languages as well. Now go to your terminal, turn off the Python server backend, and change the directory to node/. Start the node server by issuing this command: node server.js This time around, the server is running at http://localhost:8080, so remember to change the $.post() endpoint to http://localhost:8080 instead of http://localhost:8000 in external.html, as shown in the following code:    function addTodo() {      var data = {        text: $('#todo_title').val(),        details:$('#todo_text').val()      }      // $.post('http://localhost:8000/api/todos', data,      function(result) {      $.post('http://localhost:8080/api/todos', data,      function(result) {        var item = todoTemplate(result.text, result.details);        $('#todos').prepend(item);        $("#todo-form").slideUp();      })    } The line changed is found at addTodo(); the highlighted code is the correct endpoint for this section. Now, going back to external.html, add a new to-do item containing JavaScript, as shown in the following screenshot: Trying to inject JavaScript into a to-do app based on Node.js As usual, submit the item. Go to http://localhost:8080/api/ and refresh; you should see two alerts (or four alerts if you didn't delete the previous ones). The first alert is as follows: Successfully injected JavaScript part 1 The second alert is as follows: Successfully injected JavaScript part 1 Now that we have seen what can happen to our app if we suffered a CSRF attack, let's think about how such attacks can happen. Basically, such attacks can happen when our API endpoints (or URLs accepting the requests) are not protected at all. Attackers can exploit such vulnerabilities by simply observing which endpoints are used and attempt to exploit them by performing a basic HTTP POST operation to it. Basic defense against CSRF attacks If you are using modern frameworks or packages, the good news is that you can easily protect against such attacks by turning on or making use of CSRF protection. For example, for server.py, you can turn on xsrf_cookie by setting it to True, as shown in the following code: class Application(tornado.web.Application):    def __init__(self):        handlers = [            (r"/api/todos", Todos),            (r"/todo", TodoApp)          ]        conn = pymongo.Connection("localhost")        self.db = conn["todos"]        settings = dict(            xsrf_cookies=True,            debug=True,            template_path=os.path.join(os.path.dirname(__file__),            "templates"),            static_path=os.path.join(os.path.dirname(__file__),            "static")        )        tornado.web.Application.__init__(self, handlers, **settings) Note the highlighted line, where we set xsrf_cookies=True. Have a look at the following code snippet: var express   = require('express'); var bodyParser = require('body-parser'); var app       = express(); var session   = require('cookie-session'); var csrf   = require('csrf');   app.use(csrf()); app.use(bodyParser()); The highlighted lines are the new lines (compared to server.js) to add in CSRF protection. Now that both backends are equipped with CSRF protection, you can try to make the same post from external.html. You will not be able to make any post from external.html. For example, you can open Chrome's developer tool and go to Network. You will see the following: POST forbidden On the terminal, you will see a 403 error from our Python server, which is shown in the following screenshot: POST forbidden from the server side Other examples of CSRF CSRF can also happen in many other ways. In this section, we'll cover the other basic examples on how CSRF can happen. CSRF using the <img> tags This is a classic example. Consider the following instance: <img src=http://yousite.com/delete?id=2 /> Should you load a site that contains this img tag, chances are that a piece of data may get deleted unknowingly. Now that we have covered the basics of preventing CSRF attacks through the use of CSRF tokens, the next question you may have is: what if there are times when you need to expose an API to an external app? For example, Facebook's Graph API, Twitter's API, and so on, allow external apps not only to read, but also write data to their system. How do we prevent malicious attacks in this situation? We'll cover this and more in the next section. Other forms of protection Using CSRF tokens may be a convenient way to protect your app from CSRF attacks, but it can be a hassle at times. As mentioned in the previous section, what about the times when you need to expose an API to allow mobile access? Or, your app is growing so quickly that you want to accelerate that growth by creating a Graph API of your own. How do you manage it then? In this section, we will go quickly over the techniques for protection. Creating your own app ID and app secret – OAuth-styled Creating your own app ID and app secret is similar to what the major Internet companies are doing right now: we require developers to sign up for developing accounts and to attach an application ID and secret key for each of the apps. Using this information, the developers will need to exchange OAuth credentials in order to make any API calls, as shown in the following screenshot: Google requires developers to sign up, and it assigns the client ID On the server end, all you need to do is look for the application ID and secret key; if it is not present, simply reject the request. Have a look at the following screenshot: The same thing with Facebook; Facebook requires you to sign up, and it assigns app ID and app secret Checking the Origin header Simply put, you want to check where the request is coming from. This is a technique where you can check the Origin header. The Origin header, in layman's terms, refers to where the request is coming from. There are at least two use cases for the usage of the Origin header, which are as follows: Assuming your endpoint is used internally (by your own web application) and checking whether the requests are indeed made from the same website, that is, your website. If you are creating an endpoint for external use, such as those similar to Facebook's Graph API, then you can make those developers register the website URL where they are going to use the API. If the website URL does not match with the one that is being registered, you can reject this request. Note that the Origin header can also be modified; for example, an attacker can provide a header that is modified. Limiting the lifetime of the token Assuming that you are generating your own tokens, you may also want to limit the lifetime of the token, for instance, making the token valid for only a certain time period if the user is logged in to your site. Similarly, your site can make this a requirement in order for the requests to be made; if the token does not exist, HTTP requests cannot be made. Summary In this article, we covered the basic forms of CSRF attacks and how to defend against it. Note that these security loopholes can come from both the frontend and server side. Resources for Article: Further resources on this subject: Implementing Stacks using JavaScript [Article] Cordova Plugins [Article] JavaScript Promises – Why Should I Care? [Article]
Read more
  • 0
  • 0
  • 12359

article-image-building-beowulf-cluster
Packt
17 Nov 2014
19 min read
Save for later

Building a Beowulf Cluster

Packt
17 Nov 2014
19 min read
A Beowulf cluster is nothing more than a bunch of computers interconnected by Ethernet and running with a Linux or BSD operating system. A key feature is the communication over IP (Internet Protocol) that distributes problems among the boards. The entity of the boards or computers is called a cluster and each board or computer is called a node. In this article, written by Andreas Joseph Reichel, the author of Building a BeagleBone Black Super Cluster, we will first see what is really required for each board to run inside a cluster environment. You will see examples of how to build a cheap and scalable cluster housing and how to modify an ATX power supply in order to use it as a power source. I will then explain the network interconnection of the Beowulf cluster and have a look at its network topology. The article concludes with an introduction to the microSD card usage for installation images and additional swap space as well as external network storage. The following topics will be covered: Describing the minimally required equipment Building a scalable housing Modifying an ATX power source Introducing the Beowulf network topology Managing microSD cards Using external network storage We will first start with a closer look at the utilization of a single BBB and explain the minimal hardware configuration required. (For more resources related to this topic, see here.) Minimal configuration and optional equipment BBB is a single-board computer that has all the components needed to run Linux distributions that support ARMhf platforms. Due to the very powerful network utilities that come with Linux operating systems, it is not necessary to install a mouse or keyboard. Even a monitor is not required in order to install and configure a new BBB. First, we will have a look at the minimal configuration required to use a single board over a network. Minimal configuration A very powerful interface of Linux operating systems is its standard support for SSH. SSH is the abbreviation of Secure Shell, and it enables users to establish an authenticated and encrypted network connection to a remote PC that provides a Shell. Its command line can then be utilized to make use of the PC without any local monitor or keyboard. SSH is the secure replacement for the telnet service. The following diagram shows you the typical configuration of a local area network using SSH for the remote control of a BBB board: The minimal configuration for the SSH control SSH is a key feature of Linux and comes preinstalled on most distributions. If you use Microsoft ® Windows™ as your host operating system, you will require additional software such as putty, which is an SSH client that is available at http://www.putty.org. On Linux and Mac OS, there is usually an SSH client already installed, which can be started using the ssh command. Using a USB keyboard It is practical for several boards to be configured using the same network computer and an SSH client. However, if a system does not boot up, it can be hard for a beginner to figure out the reason. If you get stuck with such a problem and don't find a solution using SSH, or the SSH login is not possible for some reason anymore, it might be helpful to use a local keyboard and a local monitor to control the problematic board such as a usual PC. Installing a keyboard is possible with the onboard USB host port. A very practical way is to use a wireless keyboard and mouse combination. In this case, you only need to plug the wireless control adapter into the USB host port. Using the HDMI adapter and monitor The BBB board supports high definition graphics and, therefore, uses a mini HDMI port for the video output. In order to use a monitor, you need an adapter for mini HDMI to HDMI, DVI, or VGA, respectively. Building a scalable board-mounting system The following image shows you the finished board housing with its key components as well as some installed BBBs. Here, a indicates the threaded rod with the straw as the spacer, b indicates BeagleBone Black, c indicates the Ethernet cable, d indicates 3.5" hard disc cooling fans, e indicates the 5 V power cable, and f indicates the plate with drilled holes. The finished casing with installed BBBs One of the most important things that you have to consider before building a super computer is the space you require. It is not only important to provide stable and practical housing for some BBB boards, but also to keep in mind that you might want to upgrade the system to more boards in the future. This means that you require a scalable system that is easy to upgrade. Also, you need to keep in mind that every single board requires its own power and has to be accessible by hand (reset, boot-selection, and the power button as well as the memory card, and so on). The networking cables also need some place depending on their lengths. There are also flat Ethernet cables that need less space. The tidier the system is built, the easier it will be to track down errors or exchange faulty boards, cables, or memory cards. However, there is a more important point. Although the BBB boards are very power-efficient, they get quite warm depending on their utilization. If you have 20 boards stacked onto each other and do not provide sufficient space for air flow, your system will overheat and suffer from data loss or malfunctions. Insufficient air flow can result in the burning of devices and other permanent hardware damage. Please remember that I'm is not liable for any damages resulting from an insufficient cooling system. Depending on your taste, you can spend a lot of money on your server housing and put some lights inside and make it glow like a Christmas tree. However, I will show you very cheap housing, which is easy and fast to build and still robust enough, scalable, and practical to use. Board-holding rods The key idea of my board installation is to use the existing corner holes of the BBB boards and attach the boards on four rods in order to build a horizontal stack. This stack is then held by two side plates and a base plate. Usually, when I experiment and want to build a prototype, it is helpful not to predefine every single measurement, and then invest money into the sawing and cutting of these parts. Instead, I look around in some hobby markets and see what they have and think about whether I can use these parts. However, drilling some holes is not unavoidable. When you get to drilling holes and using screws and threads, you might know or not know that there are two different systems. One is the metric system and the other is the English system. The BBB board has four holes and their size fits to 1/8" in the English or M3 in the metric system. According to the international standard, this article will only name metric dimensions. For easy and quick installation of the boards, I used four M3 threaded rods that are obtainable at model making or hobby shops. I got mine at Conrad Electronic. For the base plates, I went to a local raw material store. The following diagram shows you the mounting hole positions for the side walls with the dimensions of BBB (dashed line). The measurements are given for the English and metric system. The mounting hole's positions Board spacers As mentioned earlier, it is important to leave enough space between the boards in order to provide finger access to the buttons and, of course, for airflow. First, I mounted each board with eight nuts. However, when you have 16 boards installed and want to uninstall the eighth board from the left, then it will take you a lot of time and nerves to get the nuts along the threaded rods. A simple solution with enough stability is to use short parts of straws. You can buy some thick drinking straws and cut them into equally long parts, each of two or three centimeters in length. Then, you can put them between the boards onto the threaded rods in order to use them as spacers. Of course, this is not the most stable way, but it is sufficient, cheap, and widely available. Cooling system One nice possibility I found for cooling the system is to use hard disk fans. They are not so cheap but I had some lying around for years. Usually, they are mounted to the lower side of 3.5" hard discs, and their width is approximately the length of one BBB. So, they are suitable for the base plate of our casing and can provide enough air flow to cool the whole system. I installed two with two fans each for eight boards and a third one for future upgrades. The following image shows you my system with eight boards installed: A board housing with BBBs and cooling system Once you have built housing with a cooling system, you can install your boards. The next step will be the connection of each board to a power source as well as the network interconnection. Both are described in the following sections. Using a low-cost power source I have seen a picture on the Web where somebody powered a dozen older Beagle Boards with a lot of single DC adapters and built everything into a portable case. The result was a huge mess of cables. You should always try to keep your cables well organized in order to save space and improve the cooling performance. Using an ATX power supply with a cable tree can save you a lot of money compared to buying several standalone power supplies. They are stable and can also provide some protection for hardware, which cheap DC adapters don't always do. In the following section, I will explain the power requirements and how to modify an ATX power supply to fit our needs. Power requirements If you do not use an additional keyboard and mouse and only onboard flash memory, one board needs around 500 mA at 5 V voltage, which gives you a total power of 2.5 Watts for one board. Depending on the installed memory card or other additional hardware, you might need more. Please note that using the Linux distribution described in this article is not compatible with the USB-client port power supply. You have to use the 5 V power jack. Power cables If you want to use an ATX power supply, then you need to build an adapter from the standard PATA or SATA power plugs to a low voltage plug that fits the 5 V jack of the board. You need a 5.5/2.1 mm low voltage plug and they are obtainable from VOLTCRAFT with cables already attached. I got mine from Conrad Electronics (item number 710344). Once you have got your power cables, you can build a small distribution box. Modifying the ATX power supply ATX power supplies are widely available and power-efficient. They cost around 60 dollars, providing more than 500 Watts of output power. For our purpose, we will only need most power on the 5 V rail and some for fans on the 12 V rail. It is not difficult to modify an ATX supply. The trick is to provide the soft-on signal, because ATX supplies are turned on from the mainboard via a soft-on signal on the green wire. If this green wire is connected to the ground, it turns on. If the connection is lost, it turns off. The following image shows you which wires of the ATX mainboard plug have to be cut and attached to a manual switch in order to build a manual on/off switch: The ATX power plug with a green and black wire, as indicated by the red circle, cut and soldered to a switch As we are using the 5 V and most probably the 12 V rail (for the cooling fans) of the power supply, it is not necessary to add resistors. If the output voltage of the supply is far too low, this means that not enough current is flowing for its internal regulation circuitry. If this happens, you can just add a 1/4 Watt 200 Ohms resistor between any +5 V (red) and GND (neighboring black) pin to drain a current of 25 mA. This should never happen when driving the BBB boards, as their power requirements are much higher and the supply should regulate well. The following image shows you the power cable distribution box. I soldered the power cables together with the cut ends of a PATA connector to a PCB board. The power cable distribution box What could happen is that the resistance of one PATA wire is too high and the voltage drop leads to a supply voltage of below 4.5 Volts. If that happens, some of the BBB boards will not power up. Either you need to retry booting these boards separately by their power button later when all others are booted up, or you need to use two PATA wires instead of one to decrease the resistance. Please have a look if this is possible with your power supply and if the two 5 V lines you want to connect do not belong to different regulation circuitries. Setting up the network backbone To interconnect BBB boards via Ethernet, we need a switch or a hub. There is a difference in the functionality between a switch and a hub: With hubs, computers can communicate with each other. Every computer is connected to the hub with a separate Ethernet cable. The hub is nothing more than a multiport repeater. This means that it just repeats all the information it receives for all other ports, and every connected PC has to decide whether the data is for it or not. This produces a lot of network traffic and can slow down the speed. Switches in comparison can control the flow of network traffic based on the address information in each packet. It learns which traffic packets are received by which PC and then forwards them only to the proper port. This allows simultaneous communication across the switch and improves the bandwidth. This is the reason why switches are the preferred choice of network interconnection for our BBB Beowulf cluster. The following table summarizes the main differences between a hub and a switch:   hub Switch Traffic control no yes Bandwidth low high I bought a 24-port Ethernet switch on eBay with 100 Megabit/s ports. This is enough for the BBB boards. The total bandwidth of the switch is 2.4 Gigabit/s. The network topology The typical network topology is a star configuration. This means that every BBB board has its own connection to the switch, and the switch itself is connected to the local area network (LAN). On most Beowulf clusters, there is one special board called the master node. This master node is used to provide the bridge between the cluster and the rest of the LAN. All users (if there are more persons that use the cluster) log in to the master node, and it is only responsible for user management and starting the correct programs on specified nodes. It usually doesn't contribute to any calculation tasks. However, as BBB only has one network connector, it is not possible to use it as a bridge, because a bridge requires two network ports: One connected to the LAN. The other connected to the switch of the cluster. Because of this, we only define one node as the master node, providing some special software features but also contributing to the calculations of the cluster. This way, all BBBs contribute to the overall calculation power, and we do not need any special hardware to build a network bridge. Regarding security, we can manage everything with SSH login rules and the kernel firewall, if required. The following diagram shows you the network topology used in this article. Every BBB has its own IP address, and you have to reserve the required amount of IP addresses in your LAN. They do not have to be successive; however, it makes it easier if you note down every IP for every board. You can give the boards hostnames such as node1, node2, node3, and so on to make them easier to follow. The network topology The RJ45 network cables There is only one thing you have to keep in mind regarding RJ45 Ethernet cables and 100 Megabit/s transmission speed. There are crossover cables and normal ones. The crossover cables have crossed lines regarding data transmission and receiving. This means that one cable can be used to connect two PCs without a hub or switch. Most modern switches can detect when data packets collide, which means when they are received on the transmitting ports and then automatically switch over the lines again. This feature is called auto MDI-X or auto-uplink. If you have a newer switch, you don't need to pay attention to which sort of cable you buy. Usually, normal RJ45 cables without crossover are the preferred choice. The Ethernet multiport switch As described earlier, we use an Ethernet switch rather than a hub. When buying a switch, you have to decide how many ports you want. For future upgrades, you can also buy an 8-port switch, for example, and later, if you want to go from seven boards (one port for the uplink) to 14 boards, you can upgrade with a second 8-port switch and connect both to the LAN. If you want to build a big system from the beginning, you might want to buy a 24-port switch or an even bigger one. The following image shows you my 24-port Ethernet switch with some connected RJ45 cables below the board housing: A 24-port cluster switch The storage memory One thing you might want to think of in the beginning is the amount of space you require for applications and data. The standard version of BBB has 2 GB flash memory onboard and newer ones have 4 GB. A critical feature of computational nodes is the amount of RAM they have installed. On BBB, this is only 512 MB. If you are of the opinion that this is not enough for your tasks, then you can extend the RAM by installing Linux on an external SD card and create a swap partition on it. However, you have to keep in mind that the external swap space is much slower than the DDR3 memory (MB/s compared to GB/s). If the software is nicely programmed, data can always be sufficiently distributed on the nodes, and each node does not need much RAM. However, with more complicated libraries and tasks, you might want to upgrade some day. Installing images on microSD cards For the installation of Linux, we will need Linux Root File System Images on the microSD card. It is always a good idea to keep these cards for future repair or extension purposes. I keep one installation SD for the master node and one for all slave nodes. When I upgrade the system to more slave nodes, I can just insert the installation SD and easily incorporate the new system with a few commands. The swap space on an SD card Usually, it should be possible to boot Linux from the internal memory and utilize the external microSD card solely as the swap space. However, I had problems utilizing the additional space as it was not properly recognized by the system. I obtained best results when booting from the same card I want the swap partition on. The external network storage To reduce the size of the used software, it is always a good idea to compile it dynamically. Each node you want to use for computations has to start the same program. Programs have to be accessible by each node, which means that you would have to install every program on every node. This is a lot of work when adding additional boards and is a waste of memory in general. To circumvent this problem, I use external network storage on the basis of Samba. The master node can then access the Samba share and create a share for all the client nodes by itself. This way, each node has access to the same software and data, and upgrades can be performed easily. Also, the need for local storage memory is reduced. Important libraries that have to be present on each local filesystem can be introduced by hard links pointing to the network storage location. The following image shows you the storage system of my BBB cluster: The storage topology Some of you might worry when I chose Samba over NFS and think that a 100 Megabit networking is too slow for cluster computations. First of all, I chose Samba because I was used to it and it is well known to most hobbyists. It is very easy to install, and I have used it for over 10 years. Only thing you have to keep in mind is that using Samba will cause your filesystem to treat capital and small letters equally. So, your Linux filenames (ext2, ext3, ext4, and so on) will behave like FAT/NTFS filenames. Regarding the network bandwidth, a double value will require 8 bytes of memory and thus, you can transfer a maximum of 2.4 billion double values per second on a hub with 24 ports and 100 Megabit/s. Additionally, libraries are optimized to keep the network talk as low as possible and solve as much as possible on the local CPU memory system. Thus, for most applications, the construction as described earlier will be sufficient. Summary In this article, you were introduced to the whole cluster concept regarding its hardware and interconnection. You were shown a working system configuration using only the minimally required amount of equipment and also some optional possibilities. A description of very basic housing including a cooling system was given as an example for a cheap yet nicely scalable possibility to mount the boards. You also learned how to build a cost-efficient power supply using a widely available ATX supply, and you were shown how to modify it to power several BBBs. Finally, you were introduced to the network topology and the purpose of network switches. A short description about the used storage system ended this article. If you interconnect everything as described in this article, it means that you have created the hardware basis of a super computer cluster. Resources for Article: Further resources on this subject: Protecting GPG Keys in BeagleBone [article] Making the Unit Very Mobile - Controlling Legged Movement [article] Pulse width modulator [article]
Read more
  • 0
  • 0
  • 11210

article-image-web-application-testing
Packt
14 Nov 2014
15 min read
Save for later

Web Application Testing

Packt
14 Nov 2014
15 min read
This article is written by Roberto Messora, the author of the Web App Testing Using Knockout.JS book. This article will give you an overview of various design patterns used in web application testing. It will also tech you web development using jQuery. (For more resources related to this topic, see here.) Presentation design patterns in web application testing The Web has changed a lot since HTML5 has made its appearance. We are witnessing a gradual shift from a classical full server-side web development, to a new architectural asset that moves much of the application logic to the client-side. The general objective is to deliver rich internet applications (commonly known as RIA) with a desktop-like user experience. Think about web applications such as Gmail or Facebook: if you maximize your browser, they look like complete desktop applications in terms of usability, UI effects, responsiveness, and richness. Once we have established that testing is a pillar of our solutions, we need to understand which is the best way to proceed, in terms of software architecture and development. In this regard, it's very important to determine the very basic design principles that allow a proper approach to unit testing. In fact, even though HTML5 is a recent achievement, HTML in general and JavaScript are technologies that have been in use for quite some time. The problem here is that many developers tend to approach the modern web development in the same old way. This is a grave mistake because, back in time, client-side JavaScript development was a lot underrated and mostly confined to simple UI graphic management. Client-side development is historically driven by libraries such as Prototype, jQuery, and Dojo, whose primary feature is DOM (HTML Document Object Model, in other words HTML markup) management. They can work as-is in small web applications, but as soon as these grow in complexity, code base starts to become unmanageable and unmaintainable. We can't really think that we can continue to develop JavaScript in the same way we did 10 years ago. In those days, we only had to dynamically apply some UI transformations. Today we have to deliver complete working applications. We need a better design, but most of all we need to reconsider client-side JavaScript development and apply the advanced design patterns and principles. jQuery web application development JavaScript is the programming language of the web, but its native DOM API is something rudimental. We have to write a lot of code to manage and transform HTML markup to bring UI to life with some dynamic user interaction. Also not full standardization means that the same code can work differently (or not work at all) in different browsers. Over the past years, developers decided to resolve this situation: JavaScript libraries, such as Prototype, jQuery and Dojo have come to light. jQuery is one of the most known open-source JavaScript libraries, which was published for the first time in 2006. Its huge success is mainly due to: A simple and detailed API that allows you to manage HTML DOM elements Cross-browser support Simple and effective extensibility Since its appearance, it's been used by thousands of developers as the foundation library. A large amount of JavaScript code all around the world has been built with keeping jQuery in mind. jQuery ecosystem grew up very quickly and nowadays there are plenty of jQuery plugins that implement virtually everything related to the web development. Despite its simplicity, a typical jQuery web application is virtually untestable. There are two main reasons: User interface items are tightly coupled with the user interface logic User interface logic spans inside event handler callback functions The real problem is that everything passes through a jQuery reference, which is a jQuery("something") call. This means that we will always need a live reference of the HTML page, otherwise these calls will fail, and this is also true for a unit test case. We can't think about testing a piece of user interface logic running an entire web application! Large jQuery applications tend to be monolithic because jQuery itself allows callback function nesting too easily, and doesn't really promote any particular design strategy. The result is often spaghetti code. jQuery is a good option if you want to develop some specific custom plugin, also we will continue to use this library for pure user interface effects and animations, but we need something different to maintain a large web application logic. Presentation design patterns To move a step forward, we need to decide what's the best option in terms of testable code. The main topic here is the application design, in other words, how we can build our code base following a general guideline with keeping testability in mind. In software engineering there's nothing better than not to reinvent the wheel, we can rely on a safe and reliable resource: design patterns. Wikipedia provides a good definition for the term design pattern (http://en.wikipedia.org/wiki/Software_design_pattern): In software engineering, a design pattern is a general reusable solution to a commonly occurring problem within a given context in software design. A design pattern is not a finished design that can be transformed directly into source or machine code. It is a description or template for how to solve a problem that can be used in many different situations. Patterns are formalized best practices that the programmer can use to solve common problems when designing an application or system. There are tens of specific design patterns, but we also need something that is related to the presentation layer because this is where a JavaScript web application belongs to. The most important aspect in terms of design and maintainability of a JavaScript web application is a clear separation between the user interface (basically, the HTML markup) and the presentation logic. (The JavaScript code that turns a web page dynamic and responsive to user interaction.) This is what we learned digging into a typical jQuery web application. At this point, we need to identify an effective implementation of a presentation design pattern and use it in our web applications. In this regard, I have to admit that the JavaScript community has done an extraordinary job in the last two years: up to the present time, there are literally tens of frameworks and libraries that implement a particular presentation design pattern. We only have to choose the framework that fits our needs, for example, we can start taking a look at MyTodo MVC website (http://todomvc.com/): this is an open source project that shows you how to build the same web application using a different library each time. Most of these libraries implement a so-called MV* design pattern (also Knockout.JS does). MV* means that every design pattern belongs to a broader family with a common root: Model-View-Controller. The MVC pattern is one of the oldest and most enduring architectural design patterns: originally designed by Trygve Reenskaug working on Smalltalk-80 back in 1979, it has been heavily refactored since then. Basically, the MVC pattern enforces the isolation of business data (Models) from user interfaces (Views), with a third component (Controllers) that manages the logic and user-input. It can be described as (Addy Osmani, Learning JavaScript Design Patterns, http://addyosmani.com/resources/essentialjsdesignpatterns/book/#detailmvc): A Model represented domain-specific data and was ignorant of the user-interface (Views and Controllers). When a model changed, it would inform its observers A View represented the current state of a Model. The Observer pattern was used for letting the View know whenever the Model was updated or modified Presentation was taken care of by the View, but there wasn't just a single View and Controller - a View-Controller pair was required for each section or element being displayed on the screen The Controllers role in this pair was handling user interaction (such as key-presses and actions e.g. clicks), making decisions for the View This general definition has slightly changed over the years, not only to adapt its implementation to different technologies and programming languages, but also because changes have been made to the Controller part. Model-View-Presenter, Model-View-ViewModel are the most known alternatives to the MVC pattern. MV* presentation design patterns are a valid answer to our need: an architectural design guideline that promotes the separation of concerns and isolation, the two most important factors that are needed for software testing. In this way, we can separately test models, views, and the third actor whatever it is (a Controller, Presenter, ViewModel, etc.). On the other hand, adopting a presentation design pattern doesn't mean at all that we cease to use jQuery. jQuery is a great library, we will continue to add its reference to our pages, but we will also integrate its use wisely in a better design context. Knockout.JS and Model-View-ViewModel Knockout.JS is one of the most popular JavaScript presentation libraries, it implements the Model-View-ViewModel design pattern. The most important concepts that feature Knockout:JS are: An HTML fragment (or an entire page) is considered as a View. A View is always associated with a JavaScript object called ViewModel: this is a code representation of the View that contains the data (model) to be shown (in the form of properties) and the commands that handle View events triggered by the user (in the form of methods). The association between View and ViewModel is built around the concept of data-binding, a mechanism that provides automatic bidirectional synchronization: In the View, it's declared placing the data-bind attributes into DOM elements, the attributes' value must follow a specific syntax that specifies the nature of the association and the target ViewModel property/method. In the ViewModel, methods are considered as commands and properties that are defined as special objects called observables: their main feature is the capability to notify every state modification A ViewModel is a pure-code representation of the View: it contains data to show and commands that handle events triggered by the user. It's important to remember that a ViewModel shouldn't have any knowledge about the View and the UI: pure-code representation means that a ViewModel shouldn't contain any reference to HTML markup elements (buttons, textboxes, and so on), but only pure JavaScript properties and methods. Model-View-ViewModel's objective is to promote a clear separation between View and ViewModel, this principle is called Separation of Concerns. Why is this so important? The answer is quite easy: because, in this way a developer can achieve a real separation of responsibilities: the View is only responsible for presenting data to the user and react to her/his inputs, the ViewModel is only responsible for holding the data and providing the presentation logic. The following diagram from Microsoft MSDN depicts the existing relationships between the three pattern actors very well (http://msdn.microsoft.com/en-us/library/ff798384.aspx): Thinking about a web application in these terms leads to a ViewModel development without any reference to DOM elements' IDs or any other markup related code as in the classic jQuery style. The two main reasons behind this are: As the web application becomes more complex, the number of DOM elements increases and is not uncommon to reach a point where it becomes very difficult to manage all those IDs with the typical jQuery fluent interface style: the JavaScript code base turns into a spaghetti code nightmare very soon. A clear separation between View and ViewModel allows a new way of working: JavaScript developers can concentrate on the presentation logic, UX experts on the other hand, can provide an HTML markup that focuses on the user interaction and how a web application will look. The two groups can work quite independently and agree on the basic contact points using the data-bind tag attributes. The key feature of a ViewModel is the observable object: a special object that is capable to notify its state modifications to any subscribers. There are three types of observable objects: The basic observable that is based on JavaScript data types (string, number, and so on) The computed observable that is dependent on other observables or computed observables The observable array that is a standard JavaScript array, with a built-in change notification mechanism On the View-side, we talk about declarative data-binding because we need to place the data-bind attributes inside HTML tags, and specify what kind of binding is associated to a ViewModel property/command. MVVM and unit testing Why a clear separation between the user interface and presentation logic is a real benefit? There are several possible answers, but, if we want to remain in the unit testing context, we can assert that we can apply proper unit testing specifications to the presentation logic, independently, from the concrete user interface. In Model-View-ViewModel, the ViewModel is a pure-code representation of the View. The View itself must remain a thin and simple layer, whose job is to present data and receive the user interaction. This is a great scenario for unit testing: all the logic in the presentation layer is located in the ViewModel, and this is a JavaScript object. We can definitely test almost everything that takes place in the presentation layer. Ensuring a real separation between View and ViewModel means that we need to follow a particular development procedure: Think about a web application page as a composition of sub-views: we need to embrace the divide et impera principle when we build our user interface, the more sub-views are specific and simple, the more we can test them easily. Knockout.JS supports this kind of scenario very well. Write a class for every View and a corresponding class for its ViewModel: the first one is the starting point to instantiate the ViewModel and apply bindings, after all, the user interface (the HTML markup) is what the browser loads initially. Keep each View class as simple as possible, so simple that it might not even need be tested, it should be just a container for:     Its ViewModel instance     Sub-View instances, in case of a bigger View that is a composition of smaller ones     Pure user interface code, in case of particular UI JavaScript plugins that cannot take place in the ViewModel and simply provide graphical effects/enrichments (in other words they don't change the logical functioning) If we look carefully at a typical ViewModel class implementation, we can see that there are no HTML markup references: no tag names, no tag identifiers, nothing. All of these references are present in the View class implementation. In fact, if we were to test a ViewModel that holds a direct reference to an UI item, we also need a live instance of the UI, otherwise accessing that item reference would cause a null reference runtime error during the test. This is not what we want, because it is very difficult to test a presentation logic having to deal with a live instance of the user interface: there are many reasons, from the need of a web server that delivers the page, to the need of a separate instance of a web browser to load the page. This is not very different from debugging a live page with Mozilla Firebug or Google Chrome Developer Tools, our objective is the test automation, but also we want to run the tests easily and quickly in isolation: we don't want to run the page in any way! An important application asset is the event bus: this is a global object that works as an event/message broker for all the actors that are involved in the web page (Views and ViewModels). Event bus is one of the alternative forms of the Event Collaboration design pattern (http://martinfowler.com/eaaDev/EventCollaboration.html): Multiple components work together by communicating with each other by sending events when their internal state changes (Marting Fowler) The main aspect of an event bus is that: The sender is just broadcasting the event, the sender does not need to know who is interested and who will respond, this loose coupling means that the sender does not have to care about responses, allowing us to add behaviour by plugging new components (Martin Fowler) In this way, we can maintain all the different components of a web page that are completely separated: every View/ViewModel couple sends and receives events, but they don't know anything about all the other couples. Again, every ViewModel is completely decoupled from its View (remember that the View holds a reference to the ViewModel, but not the other way around) and in this case, it can trigger some events in order to communicate something to the View. Concerning unit testing, loose coupling means that we can test our presentation logic a single component at a time, simply ensuring that events are broadcasted when they need to. Event buses can also be mocked so we don't need to rely on concrete implementation. In real-world development, the production process is an iterative task. Usually, we need to: Define a View markup skeleton, without any data-bind attributes. Start developing classes for the View and the ViewModel, which are empty at the beginning. Start developing the presentation logic, adding observables to the ViewModel and their respective data bindings in the View. Start writing test specifications. This process is repetitive, adds more presentation logic at every iteration, until we reach the final result. Summary In this article, you learned about web development using jQuery, presentation design patters, and unit testing using MVVM. Resources for Article: Further resources on this subject: Big Data Analysis [Article] Advanced Hadoop MapReduce Administration [Article] HBase Administration, Performance Tuning [Article]
Read more
  • 0
  • 0
  • 2624
article-image-hbases-data-storage
Packt
13 Nov 2014
9 min read
Save for later

The HBase's Data Storage

Packt
13 Nov 2014
9 min read
In this article by Nishant Garg author of HBase Essentials, we will look at HBase's data storage from its architectural view point. (For more resources related to this topic, see here.) For most of the developers or users, the preceding topics are not of big interest, but for an administrator, it really makes sense to understand how underlying data is stored or replicated within HBase. Administrators are the people who deal with HBase, starting from its installation to cluster management (performance tuning, monitoring, failure, recovery, data security and so on). Let's start with data storage in HBase first. Data storage In HBase, tables are split into smaller chunks that are distributed across multiple servers. These smaller chunks are called regions and the servers that host regions are called RegionServers. The master process handles the distribution of regions among RegionServers, and each RegionServer typically hosts multiple regions. In HBase implementation, the HRegionServer and HRegion classes represent the region server and the region, respectively. HRegionServer contains the set of HRegion instances available to the client and handles two types of files for data storage: HLog (the write-ahead log file, also known as WAL) HFile (the real data storage file) In HBase, there is a system-defined catalog table called hbase:meta that keeps the list of all the regions for user-defined tables. In older versions prior to 0.96.0, HBase had two catalog tables called-ROOT- and .META. The -ROOT- table was used to keep track of the location of the .META table. Version 0.96.0 onwards, the -ROOT- table is removed. The .META table is renamed as hbase:meta. Now, the location of .META is stored in Zookeeper. The following is the structure of the hbase:meta table. Key—the region key of the format ([table],[region start key],[region id]). A region with an empty start key is the first region in a table. The values are as follows: info:regioninfo(serialized the HRegionInfo instance for this region) info:server(server:port of the RegionServer containing this region) info:serverstartcode(start time of the RegionServer process that contains this region) When the table is split, two new columns will be created as info:splitA and info:splitB. These columns represent the two newly created regions. The values for these columns are also serialized as HRegionInfo instances. Once the split process is complete, the row that contains the old region information is deleted. In the case of data reading, the client application first connects to ZooKeeper and looks up the location of the hbase:meta table. For the next client, the HTable instance queries the hbase:meta table and finds out the region that contains the rows of interest and also locate the region server that is serving the identified region. The information about the region and region server is then cached by the client application for future interactions and avoids the lookup process. If the region is reassigned by the load balancer process or if the region server has expired, fresh lookup is done on the hbase:meta catalog table to get the new location of the user table region and cache is updated accordingly. At the object level, the HRegionServer class is responsible to create a connection with the region by creating HRegion objects. This HRegion instance sets up a store instance that has one or more StoreFile instances (wrapped around HFile) and MemStore. MemStore accumulates the data edits as it happens and buffers them into the memory. This is also important for accessing the recent edits of table data. As shown in the preceding diagram, the HRegionServer instance (the region server) contains the map of HRegion instances (regions) and also has an HLog instance that represents the WAL. There is a single block cache instance at the region-server level, which holds data from all the regions hosted on that region server. A block cache instance is created at the time of the region server startup and it can have an implementation of LruBlockCache, SlabCache, or BucketCache. The block cache also supports multilevel caching; that is, a block cache might have first-level cache, L1, as LruBlockCache and second-level cache, L2, as SlabCache or BucketCache. All these cache implementations have their own way of managing the memory; for example, LruBlockCache is like a data structure and resides on the JVM heap whereas the other two types of implementation also use memory outside of the JVM heap. HLog (the write-ahead log – WAL) In the case of writing the data, when the client calls HTable.put(Put), the data is first written to the write-ahead log file (which contains actual data and sequence number together represented by the HLogKey class) and also written in MemStore. Writing data directly into MemStrore can be dangerous as it is a volatile in-memory buffer and always open to the risk of losing data in case of a server failure. Once MemStore is full, the contents of the MemStore are flushed to the disk by creating a new HFile on the HDFS. While inserting data from the HBase shell, the flush command can be used to write the in-memory (memstore) data to the store files. If there is a server failure, the WAL can effectively retrieve the log to get everything up to where the server was prior to the crash failure. Hence, the WAL guarantees that the data is never lost. Also, as another level of assurance, the actual write-ahead log resides on the HDFS, which is a replicated filesystem. Any other server having a replicated copy can open the log. The HLog class represents the WAL. When an HRegion object is instantiated, the single HLog instance is passed on as a parameter to the constructor of HRegion. In the case of an update operation, it saves the data directly to the shared WAL and also keeps track of the changes by incrementing the sequence numbers for each edits. WAL uses a Hadoop SequenceFile, which stores records as sets of key-value pairs. Here, the HLogKey instance represents the key, and the key-value represents the rowkey, column family, column qualifier, timestamp, type, and value along with the region and table name where data needs to be stored. Also, the structure starts with two fixed-length numbers that indicate the size and value of the key. The following diagram shows the structure of a key-value pair: The WALEdit class instance takes care of atomicity at the log level by wrapping each update. For example, in the case of a multicolumn update for a row, each column is represented as a separate KeyValue instance. If the server fails after updating few columns to the WAL, it ends up with only a half-persisted row and the remaining updates are not persisted. Atomicity is guaranteed by wrapping all updates that comprise multiple columns into a single WALEdit instance and writing it in a single operation. For durability, a log writer's sync() method is called, which gets the acknowledgement from the low-level filesystem on each update. This method also takes care of writing the WAL to the replication servers (from one datanode to another). The log flush time can be set to as low as you want, or even be kept in sync for every edit to ensure high durability but at the cost of performance. To take care of the size of the write ahead log file, the LogRoller instance runs as a background thread and takes care of rolling log files at certain intervals (the default is 60 minutes). Rolling of the log file can also be controlled based on the size and hbase.regionserver.logroll.multiplier. It rotates the log file when it becomes 90 percent of the block size, if set to 0.9. HFile (the real data storage file) HFile represents the real data storage file. The files contain a variable number of data blocks and fixed number of file info blocks and trailer blocks. The index blocks records the offsets of the data and meta blocks. Each data block contains a magic header and a number of serialized KeyValue instances. The default size of the block is 64 KB and can be as large as the block size. Hence, the default block size for files in HDFS is 64 MB, which is 1,024 times the HFile default block size but there is no correlation between these two blocks. Each key-value in the HFile is represented as a low-level byte array. Within the HBase root directory, we have different files available at different levels. Write-ahead log files represented by the HLog instances are created in a directory called WALs under the root directory defined by the hbase.rootdir property in hbase-site.xml. This WALs directory also contains a subdirectory for each HRegionServer. In each subdirectory, there are several write-ahead log files (because of log rotation). All regions from that region server share the same HLog files. In HBase, every table also has its own directory created under the data/default directory. This data/default directory is located under the root directory defined by the hbase.rootdir property in hbase-site.xml. Each table directory contains a file called .tableinfo within the .tabledesc folder. This .tableinfo file stores the metadata information about the table, such as table and column family schemas, and is represented as the serialized HTableDescriptor class. Each table directory also has a separate directory for every region comprising the table, and the name of this directory is created using the MD5 hash portion of a region name. The region directory also has a .regioninfo file that contains the serialized information of the HRegionInfo instance for the given region. Once the region exceeds the maximum configured region size, it splits and a matching split directory is created within the region directory. This size is configured using the hbase.hregion.max.filesize property or the configuration done at the column-family level using the HColumnDescriptor instance. In the case of multiple flushes by the MemStore, the number of files might get increased on this disk. The compaction process running in the background combines the files to the largest configured file size and also triggers region split. Summary In this article, we have learned about the internals of HBase and how it stores the data. Resources for Article: Further resources on this subject: Big Data Analysis [Article] Advanced Hadoop MapReduce Administration [Article] HBase Administration, Performance Tuning [Article]
Read more
  • 0
  • 0
  • 5620

article-image-2d-twin-stick-shooter
Packt
11 Nov 2014
21 min read
Save for later

2D Twin-stick Shooter

Packt
11 Nov 2014
21 min read
This article written by John P. Doran, the author of Unity Game Development Blueprints, teaches us how to use Unity to prepare a well formed game. It also gives people experienced in this field a chance to prepare some great stuff. (For more resources related to this topic, see here.) The shoot 'em up genre of games is one of the earliest kinds of games. In shoot 'em ups, the player character is a single entity fighting a large number of enemies. They are typically played with a top-down perspective, which is perfect for 2D games. Shoot 'em up games also exist with many categories, based upon their design elements. Elements of a shoot 'em up were first seen in the 1961 Spacewar! game. However, the concept wasn't popularized until 1978 with Space Invaders. The genre was quite popular throughout the 1980s and 1990s and went in many different directions, including bullet hell games, such as the titles of the Touhou Project. The genre has recently gone through a resurgence in recent years with games such as Bizarre Creations' Geometry Wars: Retro Evolved, which is more famously known as a twin-stick shooter. Project overview Over the course of this article, we will be creating a 2D multidirectional shooter game similar to Geometry Wars. In this game, the player controls a ship. This ship can move around the screen using the keyboard and shoot projectiles in the direction that the mouse is points at. Enemies and obstacles will spawn towards the player, and the player will avoid/shoot them. This article will also serve as a refresher on a lot of the concepts of working in Unity and give an overview of the recent addition of native 2D tools into Unity. Your objectives This project will be split into a number of tasks. It will be a simple step-by-step process from beginning to end. Here is the outline of our tasks: Setting up the project Creating our scene Adding in player movement Adding in shooting functionality Creating enemies Adding GameController to spawn enemy waves Particle systems Adding in audio Adding in points, score, and wave numbers Publishing the game Prerequisites Before we start, we will need to get the latest Unity version, which you can always get by going to http://unity3d.com/unity/download/ and downloading it there: At the time of writing this article, the version is 4.5.3, but this project should work in future versions with minimal changes. Navigate to the preceding URL, and download the Chapter1.zip package and unzip it. Inside the Chapter1 folder, there are a number of things, including an Assets folder, which will have the art, sound, and font files you'll need for the project as well as the Chapter_1_Completed.unitypackage (this is the complete article package that includes the entire project for you to work with). I've also added in the complete game exported (TwinstickShooter Exported) as well as the entire project zipped up in the TwinstickShooter Project.zip file. Setting up the project At this point, I have assumed that you have Unity freshly installed and have started it up. With Unity started, go to File | New Project. Select Project Location of your choice somewhere on your hard drive, and ensure you have Setup defaults for set to 2D. Once completed, select Create. At this point, we will not need to import any packages, as we'll be making everything from scratch. It should look like the following screenshot: From there, if you see the Welcome to Unity pop up, feel free to close it out as we won't be using it. At this point, you will be brought to the general Unity layout, as follows: Again, I'm assuming you have some familiarity with Unity before reading this article; if you would like more information on the interface, please visit http://docs.unity3d.com/Documentation/Manual/LearningtheInterface.html. Keeping your Unity project organized is incredibly important. As your project moves from a small prototype to a full game, more and more files will be introduced to your project. If you don't start organizing from the beginning, you'll keep planning to tidy it up later on, but as deadlines keep coming, things may get quite out of hand. This organization becomes even more vital when you're working as part of a team, especially if your team is telecommuting. Differing project structures across different coders/artists/designers is an awful mess to find yourself in. Setting up a project structure at the start and sticking to it will save you countless minutes of time in the long run and only takes a few seconds, which is what we'll be doing now. Perform the following steps: Click on the Create drop-down menu below the Project tab in the bottom-left side of the screen. From there, click on Folder, and you'll notice that a new folder has been created inside your Assets folder. After the folder is created, you can type in the name for your folder. Once done, press Enter for the folder to be created. We need to create folders for the following directories:      Animations      Prefabs      Scenes      Scripts      Sprites If you happen to create a folder inside another folder, you can simply drag-and-drop it from the left-hand side toolbar. If you need to rename a folder, simply click on it once and wait, and you'll be able to edit it again. You can also use Ctrl + D to duplicate a folder if it is selected. Once you're done with the aforementioned steps, your project should look something like this: Creating our scene Now that we have our project set up, let's get started with creating our player: Double-click on the Sprites folder. Once inside, go to your operating system's browser window, open up the Chapter 1/Assets folder that we provided, and drag the playerShip.png file into the folder to move it into our project. Once added, confirm that the image is Sprite by clicking on it and confirming from the Inspector tab that Texture Type is Sprite. If it isn't, simply change it to that, and then click on the Apply button. Have a look at the following screenshot: If you do not want to drag-and-drop the files, you can also right-click within the folder in the Project Browser (bottom-left corner) and select Import New Asset to select a file from a folder to bring it in. The art assets used for this tutorial were provided by Kenney. To see more of their work, please check out www.kenney.nl. Next, drag-and-drop the ship into the scene (the center part that's currently dark gray). Once completed, set the position of the sprite to the center of the Screen (0, 0) by right-clicking on the Transform component and then selecting Reset Position. Have a look at the following screenshot: Now, with the player in the world, let's add in a background. Drag-and-drop the background.png file into your Sprites folder. After that, drag-and-drop a copy into the scene. If you put the background on top of the ship, you'll notice that currently the background is in front of the player (Unity puts newly added objects on top of previously created ones if their position on the Z axis is the same; this is commonly referred to as the z-order), so let's fix that. Objects on the same Z axis without sorting layer are considered to be equal in terms of draw order; so just because a scene looks a certain way this time, when you reload the level it may look different. In order to guarantee that an object is in front of another one in 2D space is by having different Z values or using sorting layers. Select your background object, and go to the Sprite Renderer component from the Inspector tab. Under Sorting Layer, select Add Sorting Layer. After that, click on the + icon for Sorting Layers, and then give Layer 1 a name, Background. Now, create a sorting layer for Foreground and GUI. Have a look at the following screenshot: Now, place the player ship on the foreground and the background by selecting the object once again and then setting the Sorting Layer property via the drop-down menu. Now, if you play the game, you'll see that the ship is in front of the background, as follows: At this point, we can just duplicate our background a number of times to create our full background by selecting the object in the Hierarchy, but that is tedious and time-consuming. Instead, we can create all of the duplicates by either using code or creating a tileable texture. For our purposes, we'll just create a texture. Delete the background sprite by left-clicking on the background object in the Hierarchy tab on the left-hand side and then pressing the Delete key. Then select the background sprite in the Project tab, change Texture Type in the Inspector tab to Texture, and click on Apply. Now let's create a 3D cube by selecting Game Object | Create Other | Cube from the top toolbar. Change the object's name from Cube to Background. In the Transform component, change the Position to (0, 0, 1) and the Scale to (100, 100, 1). If you are using Unity 4.6 you will need to go to Game Object | 3D Object | Cube to create the cube. Since our camera is at 0, 0, -10 and the player is at 0, 0, 0, putting the object at position 0, 0, 1 will put it behind all of our sprites. By creating a 3D object and scaling it, we are making it really large, much larger than the player's monitor. If we scaled a sprite, it would be one really large image with pixelation, which would look really bad. By using a 3D object, the texture that is applied to the faces of the 3D object is repeated, and since the image is tileable, it looks like one big continuous image. Remove Box Collider by right-clicking on it and selecting Remove Component. Next, we will need to create a material for our background to use. To do so, under the Project tab, select Create | Material, and name the material as BackgroundMaterial. Under the Shader property, click on the drop-down menu, and select Unlit | Texture. Click on the Texture box on the right-hand side, and select the background texture. Once completed, set the Tiling property's x and y to 25. Have a look at the following screenshot: In addition to just selecting from the menu, you can also drag-and-drop the background texture directly onto the Texture box, and it will set the property. Tiling tells Unity how many times the image should repeat in the x and y positions, respectively. Finally, go back to the Background object in Hierarchy. Under the Mesh Renderer component, open up Materials by left-clicking on the arrow, and change Element 0 to our BackgroundMaterial material. Consider the following screenshot: Now, when we play the game, you'll see that we now have a complete background that tiles properly. Scripting 101 In Unity, the behavior of game objects is controlled by the different components that are attached to them in a form of association called composition. These components are things that we can add and remove at any time to create much more complex objects. If you want to do anything that isn't already provided by Unity, you'll have to write it on your own through a process we call scripting. Scripting is an essential element in all but the simplest of video games. Unity allows you to code in either C#, Boo, or UnityScript, a language designed specifically for use with Unity and modelled after JavaScript. For this article, we will use C#. C# is an object-oriented programming language—an industry-standard language similar to Java or C++. The majority of plugins from Asset Store are written in C#, and code written in C# can port to other platforms, such as mobile, with very minimal code changes. C# is also a strongly-typed language, which means that if there is any issue with the code, it will be identified within Unity and will stop you from running the game until it's fixed. This may seem like a hindrance, but when working with code, I very much prefer to write correct code and solve problems before they escalate to something much worse. Implementing player movement Now, at this point, we have a great-looking game, but nothing at all happens. Let's change that now using our player. Perform the following steps: Right-click on the Scripts folder you created earlier, click on Create, and select the C# Script label. Once you click on it, a script will appear in the Scripts folder, and it should already have focus and should be asking you to type a name for the script—call it PlayerBehaviour. Double-click on the script in Unity, and it will open MonoDevelop, which is an open source integrated development environment (IDE) that is included with your Unity installation. After MonoDevelop has loaded, you will be presented with the C# stub code that was created automatically for you by Unity when you created the C# script. Let's break down what's currently there before we replace some of it with new code. At the top, you will see two lines: using UnityEngine;using System.Collections; The engine knows that if we refer to a class that isn't located inside this file, then it has to reference the files within these namespaces for the referenced class before giving an error. We are currently using two namespaces. The UnityEngine namespace contains interfaces and class definitions that let MonoDevelop know about all the addressable objects inside Unity. The System.Collections namespace contains interfaces and classes that define various collections of objects, such as lists, queues, bit arrays, hash tables, and dictionaries. We will be using a list, so we will change the line to the following: using System.Collections.Generic; The next line you'll see is: public class PlayerBehaviour : MonoBehaviour { You can think of a class as a kind of blueprint for creating a new component type that can be attached to GameObjects, the objects inside our scenes that start out with just a Transform and then have components added to them. When Unity created our C# stub code, it took care of that; we can see the result, as our file is called PlayerBehaviour and the class is also called PlayerBehaviour. Make sure that your .cs file and the name of the class match, as they must be the same to enable the script component to be attached to a game object. Next up is the: MonoBehaviour section of the code. The : symbol signifies that we inherit from a particular class; in this case, we'll use MonoBehaviour. All behavior scripts must inherit from MonoBehaviour directly or indirectly by being derived from it. Inheritance is the idea of having an object to be based on another object or class using the same implementation. With this in mind, all the functions and variables that existed inside the MonoBehaviour class will also exist in the PlayerBehaviour class, because PlayerBehaviour is MonoBehaviour. For more information on the MonoBehaviour class and all the functions and properties it has, check out http://docs.unity3d.com/ScriptReference/MonoBehaviour.html. Directly after this line, we will want to add some variables to help us with the project. Variables are pieces of data that we wish to hold on to for one reason or another, typically because they will change over the course of a program, and we will do different things based on their values. Add the following code under the class definition: // Movement modifier applied to directional movement.public float playerSpeed = 2.0f;// What the current speed of our player isprivate float currentSpeed = 0.0f;/** Allows us to have multiple inputs and supports keyboard,* joystick, etc.*/public List<KeyCode> upButton;public List<KeyCode> downButton;public List<KeyCode> leftButton;public List<KeyCode> rightButton;// The last movement that we've madeprivate Vector3 lastMovement = new Vector3(); Between the variable definitions, you will notice comments to explain what each variable is and how we'll use it. To write a comment, you can simply add a // to the beginning of a line and everything after that is commented upon so that the compiler/interpreter won't see it. If you want to write something that is longer than one line, you can use /* to start a comment, and everything inside will be commented until you write */ to close it. It's always a good idea to do this in your own coding endeavors for anything that doesn't make sense at first glance. For those of you working on your own projects in teams, there is an additional form of commenting that Unity supports, which may make your life much nicer: XML comments. They take up more space than the comments we are using, but also document your code for you. For a nice tutorial about that, check out http://unitypatterns.com/xml-comments/. In our game, the player may want to move up using either the arrow keys or the W key. You may even want to use something else. Rather than restricting the player to just having one button, we will store all the possible ways to go up, down, left, or right in their own container. To do this, we are going to use a list, which is a holder for multiple objects that we can add or remove while the game is being played. For more information on lists, check out http://msdn.microsoft.com/en-us/library/6sh2ey19(v=vs.110).aspx One of the things you'll notice is the public and private keywords before the variable type. These are access modifiers that dictate who can and cannot use these variables. The public keyword means that any other class can access that property, while private means that only this class will be able to access this variable. Here, currentSpeed is private because we want our current speed not to be modified or set anywhere else. But, you'll notice something interesting with the public variables that we've created. Go back into the Unity project and drag-and-drop the PlayerBehaviour script onto the playerShip object. Before going back to the Unity project though, make sure that you save your PlayerBehaviour script. Not saving is a very common mistake made by people working with MonoDevelop. Have a look at the following screenshot: You'll notice now that the public variables that we created are located inside Inspector for the component. This means that we can actually set those variables inside Inspector without having to modify the code, allowing us to tweak values in our code very easily, which is a godsend for many game designers. You may also notice that the names have changed to be more readable. This is because of the naming convention that we are using with each word starting with a capital letter. This convention is called CamelCase (more specifically headlessCamelCase). Now change the Size of each of the Button variables to 2, and fill in the Element 0 value with the appropriate arrow and Element 1 with W for up, A for left, S for down, and D for right. When this is done, it should look something like the following screenshot: Now that we have our variables set, go back to MonoDevelop for us to work on the script some more. The line after that is a function definition for a method called Start; it isn't a user method but one that belongs to MonoBehaviour. Where variables are data, functions are the things that modify and/or use that data. Functions are self-contained modules of code (enclosed within braces, { and }) that accomplish a certain task. The nice thing about using a function is that once a function is written, it can be used over and over again. Functions can be called from inside other functions: void Start () {} Start is only called once in the lifetime of the behavior when the game starts and is typically used to initialize data. If you're used to other programming languages, you may be surprised that initialization of an object is not done using a constructor function. This is because the construction of objects is handled by the editor and does not take place at the start of gameplay as you might expect. If you attempt to define a constructor for a script component, it will interfere with the normal operation of Unity and can cause major problems with the project. However, for this behavior, we will not need to use the Start function. Perform the following steps: Delete the Start function and its contents. The next function that we see included is the Update function. Also inherited from MonoBehaviour, this function is called for every frame that the component exists in and for each object that it's attached to. We want to update our player ship's rotation and movement every turn. Inside the Update function (between { and }), put the following lines of code: // Rotate player to face mouse Rotation(); // Move the player's body Movement(); Here, I called two functions, but these functions do not exist, because we haven't created them yet. Let's do that now! Below the Update function and before }, put the following function to close the class: // Will rotate the ship to face the mouse.void Rotation(){// We need to tell where the mouse is relative to the// playerVector3 worldPos = Input.mousePosition;worldPos = Camera.main.ScreenToWorldPoint(worldPos);/*   * Get the differences from each axis (stands for   * deltaX and deltaY)*/float dx = this.transform.position.x - worldPos.x;float dy = this.transform.position.y - worldPos.y;// Get the angle between the two objectsfloat angle = Mathf.Atan2(dy, dx) * Mathf.Rad2Deg;/*   * The transform's rotation property uses a Quaternion,   * so we need to convert the angle in a Vector   * (The Z axis is for rotation for 2D).*/Quaternion rot = Quaternion.Euler(new Vector3(0, 0, angle + 90));// Assign the ship's rotationthis.transform.rotation = rot;} Now if you comment out the Movement line and run the game, you'll notice that the ship will rotate in the direction in which the mouse is. Have a look at the following screenshot: Below the Rotation function, we now need to add in our Movement function the following code: // Will move the player based off of keys pressedvoid Movement(){// The movement that needs to occur this frameVector3 movement = new Vector3();// Check for inputmovement += MoveIfPressed(upButton, Vector3.up);movement += MoveIfPressed(downButton, Vector3.down);movement += MoveIfPressed(leftButton, Vector3.left);movement += MoveIfPressed(rightButton, Vector3.right);/*   * If we pressed multiple buttons, make sure we're only   * moving the same length.*/movement.Normalize ();// Check if we pressed anythingif(movement.magnitude > 0){   // If we did, move in that direction   currentSpeed = playerSpeed;   this.transform.Translate(movement * Time.deltaTime *                           playerSpeed, Space.World);   lastMovement = movement;}else{   // Otherwise, move in the direction we were going   this.transform.Translate(lastMovement * Time.deltaTime                           * currentSpeed, Space.World);   // Slow down over time   currentSpeed *= .9f;}} Now inside this function I've created another function called MoveIfPressed, so we'll need to add that in as well. Below this function, add in the following function as well: /** Will return the movement if any of the keys are pressed,* otherwise it will return (0,0,0)*/Vector3 MoveIfPressed( List<KeyCode> keyList, Vector3 Movement){// Check each key in our listforeach (KeyCode element in keyList){   if(Input.GetKey(element))   {     /*       * It was pressed so we leave the function       * with the movement applied.     */    return Movement; }}// None of the keys were pressed, so don't need to movereturn Vector3.zero;} Now, save your file and move back into Unity. Save your current scene as Chapter_1.unity by going to File | Save Scene. Make sure to save the scene to our Scenes folder we created earlier. Run the game by pressing the play button. Have a look at the following screenshot: Now you'll see that we can move using the arrow keys or the W A S D keys, and our ship will rotate to face the mouse. Great! Summary This article talks about the 2D twin-stick shooter game. It helps to bring you to familiarity with the game development features in Unity. Resources for Article: Further resources on this subject: Components in Unity [article] Customizing skin with GUISkin [article] What's Your Input? [article]
Read more
  • 0
  • 1
  • 15217
Modal Close icon
Modal Close icon