Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7019 Articles
article-image-internet-connected-smart-water-meter
Packt
22 Sep 2015
13 min read
Save for later

Internet Connected Smart Water Meter

Packt
22 Sep 2015
13 min read
In this article by Pradeeka Seneviratne, author of the book Internet of Things with Arduino Blueprints, goes on to say that for many years and even now, water meter readings are collected manually. To do this, a person has to visit the location where the water meter is installed. In this article, we learn how to make a smart water meter with an LCD screen that has the ability to connect to the Internet wirelessly and serve meter readings to the utility company as well as the consumer. (For more resources related to this topic, see here.) In this article, we will: Learn about water flow meters and its basic operation Learn how to mount and plumb a water flow meter to the pipeline Read and count water flow sensor pulses Calculate water flow rate and volume Learn about LCD displays and connecting with Arduino Convert a water flow meter to a simple web server and serve meter readings over the Internet Prerequisites The following are the prerequisites: One Arduino UNO board (The latest version is REV 3) One Arduino Wi-Fi Shield (The latest version is REV 3) One Adafruit Liquid flow meter or a similar one One Hitachi HD44780 DRIVER compatible LCD Screen (16x2) One 10K ohm resistor One 10K ohm potentiometer Few Jumper wires with male and female headers (https://www.sparkfun.com/products/9140) Water Flow Meters The heart of a water flow meter consists of a Hall Effect sensor that outputs pulses for magnetic field changes. Inside the housing, there is a small pinwheel with a permanent magnet attached. When the water flows through the housing, the pinwheel begins to spin and the magnet attached to it passes very close to the Hall Effect sensor in every cycle. The Hall Effect sensor is covered with a separate plastic housing to protect it from the water. The result generates an electric pulse that transitions from low voltage to high voltage, or high voltage to low voltage, depending on the attached permanent magnet's polarity. The resulting pulse can be read and counted using Arduino. For this project, we will be using Adafruit Liquid Flow Meter. You can visit the product page at http://www.adafruit.com/products/828. The following image shows Adafruit Liquid Flow Meter: This image is taken from http://www.adafruit.com/products/828 Pinwheel attached inside the water flow meter A little bit about Plumbing Typically, the direction of the water flow is indicated by an arrow mark on top of the water flow meter's enclosure. Also, you can mount the water flow meter either horizontally or vertically according to its specifications. Some water flow meters can mount both horizontally and vertically. You can install your water flow meter to a half-inch pipeline using normal BSP pipe connectors. The outer diameter of the connector is 0.78" and the inner thread size is half an inch. The water flow meter has threaded ends on both sides. Connect the threaded side of the PVC connectors to both ends of the water flow meter. Use the thread seal tape to seal the connection, and then connect the other ends to an existing half-inch pipe line using PVC pipe glue or solvent cement. Make sure to connect the water flow meter with the pipeline in the correct direction. See the arrow mark on top of the water flow meter for flow direction. BNC Pipeline Connector made by PVC Securing the connection between Water Flow Meter and BNC Pipe Connector using Thread seal PVC Solvent cement used to secure the connection between pipeline and BNC pipe connector. Wiring the water flow meter with Arduino The water flow meter that we are using with this project has three wires, which are as follows: The red wire indicates the positive terminal The black wire indicates the Negative terminal The yellow wire indicates the DATA terminal All three wire ends are connected to a JST connector. Always refer to the datasheet before connecting them with the microcontroller and the power source. Use jumper wires with male and female headers as follows: Connect the positive terminal of the water flow meter to Arduino 5V. Connect the negative terminal of the water flow meter to Arduino GND. Connect the DATA terminal of the water flow meter to Arduino digital pin 2 through a 10K ohm resistor. You can directly power the water flow sensor using Arduino since most of the residential type water flow sensors operate under 5V and consume a very low amount of current. You can read the product manual for more information about the supply voltage and supply current range to save your Arduino from high current consumption by the water flow sensor. If your water flow sensor requires a supply current of more than 200mA or a supply voltage of more than 5V to function correctly, use a separate power source with it. The following image illustrates jumper wires with male and female headers: Reading pulses Water flow meter produces and outputs digital pulses according to the amount of water flowing through it that can be detected and counted using Arduino. According to the data sheet, the water flow meter that we are using for this project will generate approximately 450 pulses per liter. So 1 pulse approximately equals to [1000 ml/450 pulses] 2.22 ml. These values can be different depending on the speed of the water flow and the mounting polarity. Arduino can read digital pulses by generating the water flow meter through the DATA line. Rising edge and falling edge There are two type of pulses, which are as follows: Positive-going pulse: In an idle state, the logic level is normally LOW. It goes to HIGH state, stays at HIGH state for time t, and comes back to LOW state. Negative-going pulse: In an idle state, the logic level is normally HIGH. It goes LOW state, stays at LOW state for time t, and comes back to HIGH state. The rising edge and falling edge of a pulse are vertical. The transition from LOW state to HIGH state is called RISING EDGE and the transition from HIGH state to LOW state is called falling EDGE. You can capture digital pulses using rising edge or falling edge, and in this project, we will be using the rising edge. Reading and counting pulses with Arduino In the previous section, you have attached the water flow meter to Arduino. The pulse can be read by digital pin 2 and the interrupt 0 is attached to digital pin 2. The following sketch counts pulses per second and displays on the Arduino Serial Monitor. Using Arduino IDE, upload the following sketch into your Arduino board: int pin = 2; volatile int pulse; const int pulses_per_litre=450; void setup() { Serial.begin(9600); pinMode(pin, INPUT); attachInterrupt(0, count_pulse, RISING); } void loop() { pulse=0; interrupts(); delay(1000); noInterrupts(); Serial.print("Pulses per second: "); Serial.println(pulse); } void count_pulse() { pulse++; } Calculating the water flow rate The water flow rate is the amount of water flowing at a given time and can be expressed in gallons per second or liters per second. The number of pulses generated per liter of water flowing through the sensor can be found in the water flow sensor's specification sheet. Let's say m. So, you can count the number of pulses generated by the sensor per second, Let's say n. Thus, the water flow rate R can be expressed as follows: The water flow rate is measured in liters per second. Also, you can calculate the water flow rate in liters per minute as follows: For example, if your water flow sensor generates 450 pulses for one liter of water flowing through it and you get 10 pulses for the first second, then the elapsed water flow rate is 10/450 = 0.022 liters per second or 0.022 * 1000 = 22 milliliters per second. Using your Arduino IDE, upload the following sketch into your Arduino board. It will output water flow rate in liters per second on the Arduino Serial Monitor. int pin = 2; volatile int pulse; const int pulses_per_litre=450; void setup() { Serial.begin(9600); pinMode(pin, INPUT); attachInterrupt(0, count_pulse, RISING); } void loop() { pulse=0; interrupts(); delay(1000); noInterrupts(); Serial.print("Pulses per second: "); Serial.println(pulse); Serial.print("Water flow rate: "); Serial.print(pulse/pulses_per_litre); Serial.println("litres per second"); } void count_pulse() { pulse++; } Calculating water flow volume Water flow volume can be calculated by adding all the flow rates per second of a minute and can be expressed as follows: Volume = ∑ Flow Rates The following Arduino sketch will calculate and output the total water volume since startup. Upload the sketch into your Arduino board using Arduino IDE. int pin = 2; volatile int pulse; float volume = 0; float flow_rate =0; const int pulses_per_litre=450; void setup() { Serial.begin(9600); pinMode(pin, INPUT); attachInterrupt(0, count_pulse, RISING); } void loop() { pulse=0; volume=0; interrupts(); delay(1000); noInterrupts(); Serial.print("Pulses per second: "); Serial.println(pulse); flow_rate = pulse/pulses_per_litre; Serial.print("Water flow rate: "); Serial.print(flow_rate); Serial.println("litres per second"); volume = volume + flow_rate; Serial.print("Volume: "); Serial.print(volume); Serial.println(" litres"); } void count_pulse() { pulse++; } To measure the accurate water flow rate and volume, the water flow meter will need careful calibration. The sensor inside the water flow meter is not a precision sensor, and the pulse rate does vary a bit depending on the flow rate, fluid pressure, and sensor orientation. Adding an LCD screen to the water meter You can add an LCD screen to your water meter to display readings rather than displaying them on the Arduino serial monitor. You can then disconnect your water meter from the computer after uploading the sketch onto your Arduino. Using a Hitachi HD44780 driver compatible LCD screen and Arduino LiquidCrystal library, you can easily integrate it with your water meter. Typically, this type of LCD screen has 16 interface connectors. The display has 2 rows and 16 columns, so each row can display up to 16 characters. Wire your LCD screen with Arduino as shown in the preceding diagram. Use the 10K potentiometer to control the contrast of the LCD screen. Perform the following steps to connect your LCD screen with your Arduino: LCD RS pin to digital pin 8 LCD Enable pin to digital pin 7 LCD D4 pin to digital pin 6 LCD D5 pin to digital pin 5 LCD D6 pin to digital pin 4 LCD D7 pin to digital pin 3 Wire a 10K pot to +5V and GND, with its wiper (output) to LCD screens VO pin (pin3). Now, upload the following sketch into your Arduino board using Arduino IDE, and then remove the USB cable from your computer. Make sure the water is flowing through the water meter and press the Arduino reset button. You can see number of pulses per second, water flow rate per second, and the total water volume from the beginning of the time displayed on the LCD screen. #include <LiquidCrystal.h> int pin = 2; volatile int pulse; float volume = 0; float flow_rate =0; const int pulses_per_litre=450; // initialize the library with the numbers of the interface pins LiquidCrystal lcd(8, 7, 6, 5, 4, 3); void setup() { Serial.begin(9600); pinMode(pin, INPUT); attachInterrupt(0, count_pulse, RISING); // set up the LCD's number of columns and rows: lcd.begin(16, 2); // Print a message to the LCD. lcd.print("Welcome"); } void loop() { pulse=0; volume=0; interrupts(); delay(1000); noInterrupts(); lcd.setCursor(0, 0); lcd.print("Pulses/s: "); lcd.print(pulse); flow_rate = pulse/pulses_per_litre; lcd.setCursor(0, 1); lcd.print(flow_rate,DEC); lcd.print(" l/s"); volume = volume + flow_rate; lcd.setCursor(0, 8); lcd.print(volume, DEC); lcd.println(" l"); } void count_pulse() { pulse++; } Converting your water meter to a web server In the previous steps, you have learned how to display your water flow sensor's readings, and calculate water flow rate and total volume on the Arduino serial monitor. In this step, we learn about integrating a simple web server to your water flow sensor and remotely read your water flow sensor's readings. You can make a wireless web server with Arduino Wi-Fi shield or Ethernet connected web server with the Arduino Ethernet shield. Remove all the wires you have connected to your Arduino in the previous sections in this article. Stack the Arduino Wi-Fi shield on the Arduino board using wire-wrap headers. Make sure the Wi-Fi shield is properly seated on the Arduino board. Now reconnect the wires from water flow sensor to the Wi-Fi shield. Use the same pin numbers as in previous step. Connect 9V DC power supply to the Arduino board. Connect your Arduino to your PC using the USB cable and upload the following sketch. Once the upload is complete, remove your USB cable from the water flow meter. Upload the following Arduino sketch into your Arduino board using Arduino IDE: #include <SPI.h> #include <WiFi.h> char ssid[] = "yourNetwork"; char pass[] = "secretPassword"; int keyIndex = 0; int pin = 2; volatile int pulse; float volume = 0; float flow_rate =0; const int pulses_per_litre=450; int status = WL_IDLE_STATUS; WiFiServer server(80); void setup() { Serial.begin(9600); while (!Serial) { ; } if (WiFi.status() == WL_NO_SHIELD) { Serial.println("WiFi shield not present"); while(true); } // attempt to connect to Wifi network: while ( status != WL_CONNECTED) { Serial.print("Attempting to connect to SSID: "); Serial.println(ssid); status = WiFi.begin(ssid, pass); delay(10000); } server.begin(); } void loop() { WiFiClient client = server.available(); if (client) { Serial.println("new client"); boolean currentLineIsBlank = true; while (client.connected()) { if (client.available()) { char c = client.read(); Serial.write(c); if (c == 'n' &&currentLineIsBlank) { client.println("HTTP/1.1 200 OK"); client.println("Content-Type: text/html"); client.println("Connection: close"); client.println("Refresh: 5"); client.println(); client.println("<!DOCTYPE HTML>"); client.println("<html>"); if (WiFi.status() != WL_CONNECTED) { client.println("Couldn't get a wifi connection"); while(true); } else { //print meter readings on web page pulse=0; volume=0; interrupts(); delay(1000); noInterrupts(); client.print("Pulses per second: "); client.println(pulse); flow_rate = pulse/pulses_per_litre; client.print("Water flow rate: "); client.print(flow_rate); client.println("litres per second"); volume = volume + flow_rate; client.print("Volume: "); client.print(volume); client.println(" litres"); //end } client.println("</html>"); break; } if (c == 'n') { currentLineIsBlank = true; } else if (c != 'r') { currentLineIsBlank = false; } } } delay(1); client.stop(); Serial.println("client disconnected"); } } void count_pulse() { pulse++; } Open the water valve and make sure the water flows through the meter. Click on the RESET button on the WiFi shield. In your web browser, type your WiFi shield's IP address and press Enter. You can see your water flow sensor's flow rate and total volume on the web page. The page refreshes every 5 seconds to display the updated information. Summary In this article, you gained hands-on experience and knowledge about water flow sensors and counting pulses while calculating and displaying them. Finally, you made a simple web server to allow users to read the water meter through the Internet. You can apply this to any type of liquid, but make sure to select the correct flow sensor because some liquids react chemically with the material the sensor is made of. You can search on Google and find which flow sensors support your preferred liquid type. Resources for Article: Further resources on this subject: Getting Started with Arduino[article] Arduino Development [article] Prototyping Arduino Projects using Python [article]
Read more
  • 0
  • 0
  • 17336

article-image-introduction-penetration-testing-and-kali-linux
Packt
22 Sep 2015
4 min read
Save for later

Introduction to Penetration Testing and Kali Linux

Packt
22 Sep 2015
4 min read
 In this article by Juned A Ansari, author of the book, Web Penetration Testing with Kali Linux, Second Edition, the author wants us to learn about the following topics: Introduction to penetration testing An Overview of Kali Linux Using Tor for penetration testing (For more resources related to this topic, see here.) Introduction to penetration testing Penetration testing or Ethical hacking is a proactive way of testing your web applications by simulating an attack that's similar to a real attack that could occur on any given day. We will use the tools provided in Kali Linux to accomplish this. Kali Linux is the rebranded version of Backtrack and is now based on Debian-derived Linux distribution. It comes preinstalled with a large list of popular hacking tools that are ready to use with all the prerequisites installed. We will dwell deep into the tools that would help Pentest web applications, and also attack websites in a lab vulnerable to major flaws found in real world web applications. An Overview of Kali Linux Kali Linux is security-focused Linux distribution based on Debian. It's a rebranded version of the famous Linux distribution known as Backtrack, which came with a huge repository of open source hacking tools for network, wireless, and web application penetration testing. Although Kali Linux contains most of the tools from Backtrack, the main aim of Kali Linux is to make it portable so that it can be installed on devices based on the ARM architectures, such as tablets and Chromebook, which makes the tools available at your disposal with much ease. Using open source hacking tools comes with a major drawback. They contain a whole lot of dependencies when installed on Linux, and they need to be installed in a predefined sequence; authors of some tools have not released accurate documentation, which makes our life difficult. Kali Linux simplifies this process; it contains many tools preinstalled with all the dependencies and are in ready-to-use condition so that you can pay more attention for the actual attack and not on installing the tool. Updates for tools installed in Kali Linux are more frequently released, which helps you to keep the tools up to date. A noncommercial toolkit that has all the major hacking tools preinstalled to test real-world networks and applications is a dream of every ethical hacker and the authors of Kali Linux make every effort to make our life easy, which enables us to spend more time on finding the actual flaws rather than building a toolkit. Using Tor for penetration testing The main aim of a penetration test is to hack into a web application in a way that a real-world malicious hacker would do it. Tor provides an interesting option to emulate the steps that a black hat hacker uses to protect his identity and location. Although an ethical hacker trying to improve the security of a web application should be not be concerned about hiding his location, Tor will give an additional option of testing the edge security systems such as network firewalls, web application firewalls, and IPS devices. Black hat hackers try every method to protect their location and true identity; they do not use a permanent IP address and constantly change it to fool cybercrime investigators. You will find port scanning request from a different range of IP addresses, and the actual exploitation having the source IP address that you edge security systems are logging for the first time. With the necessary written approval from the client, you can use Tor to emulate an attacker by connecting to the web application from an unknown IP address that the system does not usually see connections from. Using Tor makes it more difficult to trace back the intrusion attempt to the actual attacker. Tor uses a virtual circuit of interconnected network relays to bounce encrypted data packets. The encryption is multilayered and the final network relay releasing the data to the public Internet cannot identify the source of the communication as the entire packet was encrypted and only a part of it is decrypted at each node. The destination computer sees the final exit point of the data packet as the source of the communication, thus protecting the real identify and location of the user. The following figure shows the working of Tor: Summary This article served as an introduction to penetration testing of web application and Kali Linux. At the end, we looked at how to use Tor for penetration testing. Resources for Article: Further resources on this subject: An Introduction to WEP[article] WLAN Encryption Flaws[article] What is Kali Linux [article]
Read more
  • 0
  • 0
  • 24387

article-image-putting-function-functional-programming
Packt
22 Sep 2015
27 min read
Save for later

Putting the Function in Functional Programming

Packt
22 Sep 2015
27 min read
 In this article by Richard Reese, the author of the book Learning Java Functional Programming, we will cover lambda expressions in more depth. We will explain how they satisfy the mathematical definition of a function and how we can use them in supporting Java applications. In this article, you will cover several topics, including: Lambda expression syntax and type inference High-order, pure, and first-class functions Referential transparency Closure and currying (For more resources related to this topic, see here.) Our discussions cover high-order functions, first-class functions, and pure functions. Also examined are the concepts of referential transparency, closure, and currying. Examples of nonfunctional approaches are followed by their functional equivalent where practical. Lambda expressions usage A lambda expression can be used in many different situations, including: Assigned to a variable Passed as a parameter Returned from a function or method We will demonstrate how each of these are accomplished and then elaborate on the use of functional interfaces. Consider the forEach method supported by several classes and interfaces, including the List interface. In the following example, a List interface is created and the forEach method is executed against it. The forEach method expects an object that implements the Consumer interface. This will display the three cartoon character names: List<String> list = Arrays.asList("Huey", "Duey", "Luey"); list.forEach(/* Implementation of Consumer Interface*/); More specifically, the forEach method expects an object that implements the accept method, the interface's single abstract method. This method's signature is as follows: void accept(T t) The interface also has a default method, andThen, which is passed and returns an instance of the Consumer interface. We can use any of three different approaches for implementing the functionality of the accept method: Use an instance of a class that implements the Consumer interface Use an anonymous inner class Use a lambda expression We will demonstrate each method so that it will be clear how each technique works and why lambda expressions will often result in a better solution. We will start with the declaration of a class that implements the Consumer interface as shown next: public class ConsumerImpl<T> implements Consumer<T> { @Override public void accept(T t) { System.out.println(t); } } We can then use it as the argument of the forEach method: list.forEach(new ConsumerImpl<>()); Using an explicit class allows us to reuse the class or its objects whenever an instance is needed. The second approach uses an anonymous inner function as shown here: list.forEach(new Consumer<String>() { @Override public void accept(String t) { System.out.println(t); } }); This was a fairly common approach used prior to Java 8. It avoids having to explicitly declare and instantiate a class, which implements the Consumer interface. A simple statement that uses a lambda expression is shown next: list.forEach(t->System.out.println(t)); The lambda expression accepts a single argument and returns void. This matches the signature of the Consumer interface. Java 8 is able to automatically perform this matching process. This latter technique obviously uses less code, making it more succinct than the other solutions. If we desire to reuse this lambda expression elsewhere, we could have assigned it to a variable first and then used it in the forEach method as shown here: Consumer consumer = t->System.out.println(t); list.forEach(consumer); Anywhere a functional interface is expected, we can use a lambda expression. Thus, the availability of a large number of functional interfaces will enable the frequent use of lambda expressions and programs that exhibit a functional style of programming. While developers can define their own functional interfaces, which we will do shortly, Java 8 has added a large number of functional interfaces designed to support common operations. Most of these are found in the java.util.function package. We will use several of these throughout the book and will elaborate on their purpose, definition, and use as we encounter them. Functional programming concepts in Java In this section, we will examine the underlying concept of functions and how they are implemented in Java 8. This includes high-order, first-class, and pure functions. A first-class function is a function that can be used where other first-class entities can be used. These types of entities include primitive data types and objects. Typically, they can be passed to and returned from functions and methods. In addition, they can be assigned to variables. A high-order function either takes another function as an argument or returns a function as the return value. Languages that support this type of function are more flexible. They allow a more natural flow and composition of operations. Pure functions have no side effects. The function does not modify nonlocal variables and does not perform I/O. High-order functions We will demonstrate the creation and use of the high-order function using an imperative and a functional approach to convert letters of a string to lowercase. The next code sequence reuses the list variable, developed in the previous section, to illustrate the imperative approach. The for-each statement iterates through each element of the list using the String class' toLowerCase method to perform the conversion: for(String element : list) { System.out.println(element.toLowerCase()); } The output will be each name in the list displayed in lowercase, each on a separate line. To demonstrate the use of a high-order function, we will create a function called, processString, which is passed a function as the first parameter and then apply this function to the second parameter as shown next:   public String processString(Function<String,String> operation,String target) { return operation.apply(target); } The function passed will be an instance of the java.util.function package's Function interface. This interface possesses an accept method that is passed one data type and returns a potentially different data type. With our definition, it is passed String and returns String. In the next code sequence, a lambda expression using the toLowerCase method is passed to the processString method. As you may remember, the forEach method accepts a lambda expression, which matches the Consumer interface's accept method. The lambda expression passed to the processString method matches the Function interface's accept method. The output is the same as produced by the equivalent imperative implementation. list.forEach(s ->System.out.println( processString(t->t.toLowerCase(), s))); We could have also used a method reference as show next: list.forEach(s ->System.out.println( processString(String::toLowerCase, s))); The use of the high-order function may initially seem to be a bit convoluted. We needed to create the processString function and then pass either a lambda expression or a method reference to perform the conversion. While this is true, the benefit of this approach is flexibility. If we needed to perform a different string operation other than converting the target string to lowercase, we will need to essentially duplicate the imperative code and replace toLowerCase with a new method such as toUpperCase. However, with the functional approach, all we need to do is replace the method used as shown next: list.forEach(s ->System.out.println(processString(t- >t.toUpperCase(), s))); This is simpler and more flexible. A lambda expression can also be passed to another lambda expression. Let's consider another example where high-order functions can be useful. Suppose we need to convert a list of one type into a list of a different type. We might have a list of strings that we wish to convert to their integer equivalents. We might want to perform a simple conversion or perhaps we might want to double the integer value. We will use the following lists:   List<String> numberString = Arrays.asList("12", "34", "82"); List<Integer> numbers = new ArrayList<>(); List<Integer> doubleNumbers = new ArrayList<>(); The following code sequence uses an iterative approach to convert the string list into an integer list:   for (String num : numberString) { numbers.add(Integer.parseInt(num)); } The next sequence uses a stream to perform the same conversion: numbers.clear(); numberString .stream() .forEach(s -> numbers.add(Integer.parseInt(s))); There is not a lot of difference between these two approaches, at least from a number of lines perspective. However, the iterative solution will only work for the two lists: numberString and numbers. To avoid this, we could have written the conversion routine as a method. We could also use lambda expression to perform the same conversion. The following two lambda expression will convert a string list to an integer list and from a string list to an integer list where the integer has been doubled:   Function<List<String>, List<Integer>> singleFunction = s -> { s.stream() .forEach(t -> numbers.add(Integer.parseInt(t))); return numbers; }; Function<List<String>, List<Integer>> doubleFunction = s -> { s.stream() .forEach(t -> doubleNumbers.add( Integer.parseInt(t) * 2)); return doubleNumbers; }; We can apply these two functions as shown here: numbers.clear(); System.out.println(singleFunction.apply(numberString)); System.out.println(doubleFunction.apply(numberString)); The output follows: [12, 34, 82] [24, 68, 164] However, the real power comes from passing these functions to other functions. In the next code sequence, a stream is created consisting of a single element, a list. This list contains a single element, the numberString list. The map method expects a Function interface instance. Here, we use the doubleFunction function. The list of strings is converted to integers and then doubled. The resulting list is displayed: Arrays.asList(numberString).stream() .map(doubleFunction) .forEach(s -> System.out.println(s)); The output follows: [24, 68, 164] We passed a function to a method. We could easily pass other functions to achieve different outputs. Returning a function When a value is returned from a function or method, it is intended to be used elsewhere in the application. Sometimes, the return value is used to determine how subsequent computations should proceed. To illustrate how returning a function can be useful, let's consider a problem where we need to calculate the pay of an employee based on the numbers of hours worked, the pay rate, and the employee type. To facilitate the example, start with an enumeration representing the employee type: enum EmployeeType {Hourly, Salary, Sales}; The next method illustrates one way of calculating the pay using an imperative approach. A more complex set of computation could be used, but these will suffice for our needs: public float calculatePay(int hoursWorked, float payRate, EmployeeType type) { switch (type) { case Hourly: return hoursWorked * payRate; case Salary: return 40 * payRate; case Sales: return 500.0f + 0.15f * payRate; default: return 0.0f; } } If we assume a 7 day workweek, then the next code sequence shows an imperative way of calculating the total number of hours worked: int hoursWorked[] = {8, 12, 8, 6, 6, 5, 6, 0}; int totalHoursWorked = 0; for (int hour : hoursWorked) { totalHoursWorked += hour; } Alternatively, we could have used a stream to perform the same operation as shown next. The Arrays class's stream method accepts an array of integers and converts it into a Stream object. The sum method is applied fluently, returning the number of hours worked: totalHoursWorked = Arrays.stream(hoursWorked).sum(); The latter approach is simpler and easier to read. To calculate and display the pay, we can use the following statement which, when executed, will return 803.25.    System.out.println( calculatePay(totalHoursWorked, 15.75f, EmployeeType.Hourly)); The functional approach is shown next. A calculatePayFunction method is created that is passed by the employee type and returns a lambda expression. This will compute the pay based on the number of hours worked and the pay rate. This lambda expression is based on the BiFunction interface. It has an accept method that takes two arguments and returns a value. Each of the parameters and the return type can be of different data types. It is similar to the Function interface's accept method, except that it is passed two arguments instead of one. The calculatePayFunction method is shown next. It is similar to the imperative's calculatePay method, but returns a lambda expression: public BiFunction<Integer, Float, Float> calculatePayFunction( EmployeeType type) { switch (type) { case Hourly: return (hours, payRate) -> hours * payRate; case Salary: return (hours, payRate) -> 40 * payRate; case Sales: return (hours, payRate) -> 500f + 0.15f * payRate; default: return null; } } It can be invoked as shown next: System.out.println( calculatePayFunction(EmployeeType.Hourly) .apply(totalHoursWorked, 15.75f)); When executed, it will produce the same output as the imperative solution. The advantage of this approach is that the lambda expression can be passed around and executed in different contexts. First-class functions To demonstrate first-class functions, we use lambda expressions. Assigning a lambda expression, or method reference, to a variable can be done in Java 8. Simply declare a variable of the appropriate function type and use the assignment operator to do the assignment. In the following statement, a reference variable to the previously defined BiFunction-based lambda expression is declared along with the number of hours worked: BiFunction<Integer, Float, Float> calculateFunction; int hoursWorked = 51; We can easily assign a lambda expression to this variable. Here, we use the lambda expression returned from the calculatePayFunction method: calculateFunction = calculatePayFunction(EmployeeType.Hourly); The reference variable can then be used as shown in this statement: System.out.println( calculateFunction.apply(hoursWorked, 15.75f)); It produces the same output as before. One shortcoming of the way an hourly employee's pay is computed is that overtime pay is not handled. We can add this functionality to the calculatePayFunction method. However, to further illustrate the use of reference variables, we will assign one of two lambda expressions to the calculateFunction variable based on the number of hours worked as shown here: if(hoursWorked<=40) { calculateFunction = (hours, payRate) -> 40 * payRate; } else { calculateFunction = (hours, payRate) -> hours*payRate + (hours-40)*1.5f*payRate; } When the expression is evaluated as shown next, it returns a value of 1063.125: System.out.println( calculateFunction.apply(hoursWorked, 15.75f)); Let's rework the example developed in the High-order functions section, where we used lambda expressions to display the lowercase values of an array of string. Part of the code has been duplicated here for your convenience: list.forEach(s ->System.out.println( processString(t->t.toLowerCase(), s))); Instead, we will use variables to hold the lambda expressions for the Consumer and Function interfaces as shown here: Consumer<String> consumer; consumer = s -> System.out.println(toLowerFunction.apply(s)); Function<String,String> toLowerFunction; toLowerFunction= t -> t.toLowerCase(); The declaration and initialization could have been done with one statement for each variable. To display all of the names, we simply use the consumer variable as the argument of the forEach method: list.forEach(consumer); This will display the names as before. However, this is much easier to read and follow. The ability to use lambda expressions as first-class entities makes this possible. We can also assign method references to variables. Here, we replaced the initialization of the function variable with a method reference: function = String::toLowerCase; The output of the code will not change. The pure function The pure function is a function that has no side effects. By side effects, we mean that the function does not modify nonlocal variables and does not perform I/O. A method that squares a number is an example of a pure method with no side effects as shown here: public class SimpleMath { public static int square(int x) { return x * x; } } Its use is shown here and will display the result, 25: System.out.println(SimpleMath.square(5)); An equivalent lambda expression is shown here: Function<Integer,Integer> squareFunction = x -> x*x; System.out.println(squareFunction.apply(5)); The advantages of pure functions include the following: They can be invoked repeatedly producing the same results There are no dependencies between functions that impact the order they can be executed They support lazy evaluation They support referential transparency We will examine each of these advantages in more depth. Support repeated execution Using the same arguments will produce the same results. The previous square operation is an example of this. Since the operation does not depend on other external values, re-executing the code with the same arguments will return the same results. This supports the optimization technique call memoization. This is the process of caching the results of an expensive execution sequence and retrieving them when they are used again. An imperative technique for implementing this approach involves using a hash map to store values that have already been computed and retrieving them when they are used again. Let's demonstrate this using the square function. The technique should be used for those functions that are compute intensive. However, using the square function will allow us to focus on the technique. Declare a cache to hold the previously computed values as shown here: private final Map<Integer, Integer> memoizationCache = new HashMap<>(); We need to declare two methods. The first method, called doComputeExpensiveSquare, does the actual computation as shown here. A display statement is included only to verify the correct operation of the technique. Otherwise, it is not needed. The method should only be called once for each unique value passed to it. private Integer doComputeExpensiveSquare(Integer input) { System.out.println("Computing square"); return 2 * input; } A second method is used to detect when a value is used a subsequent time and return the previously computed value instead of calling the square method. This is shown next. The containsKey method checks to see if the input value has already been used. If it hasn't, then the doComputeExpensiveSquare method is called. Otherwise, the cached value is returned. public Integer computeExpensiveSquare(Integer input) { if (!memoizationCache.containsKey(input)) { memoizationCache.put(input, doComputeExpensiveSquare(input)); } return memoizationCache.get(input); } The use of the technique is demonstrated with the next code sequence: System.out.println(computeExpensiveSquare(4)); System.out.println(computeExpensiveSquare(4)); The output follows, which demonstrates that the square method was only called once: Computing square 16 16 The problem with this approach is the declaration of a hash map. This object may be inadvertently used by other elements of the program and will require the explicit declaration of new hash maps for each memoization usage. In addition, it does not offer flexibility in handling multiple memoization. A better approach is available in Java 8. This new approach wraps the hash map in a class and allows easier creation and use of memoization. Let's examine a memoization class as adapted from http://java.dzone.com/articles/java-8-automatic-memoization. It is called Memoizer. It uses ConcurrentHashMap to cache value and supports concurrent access from multiple threads. Two methods are defined. The doMemoize method returns a lambda expression that does all of the work. The memorize method creates an instance of the Memoizer class and passes the lambda expression implementing the expensive operation to the doMemoize method. The doMemoize method uses the ConcurrentHashMap class's computeIfAbsent method to determine if the computation has already been performed. If the value has not been computed, it executes the Function interface's apply method against the function argument: public class Memoizer<T, U> { private final Map<T, U> memoizationCache = new ConcurrentHashMap<>(); private Function<T, U> doMemoize(final Function<T, U> function) { return input -> memoizationCache.computeIfAbsent(input, function::apply); } public static <T, U> Function<T, U> memoize(final Function<T, U> function) { return new Memoizer<T, U>().doMemoize(function); } } A lambda expression is created for the square operation: Function<Integer, Integer> squareFunction = x -> { System.out.println("In function"); return x * x; }; The memoizationFunction variable will hold the lambda expression that is subsequently used to invoke the square operations: Function<Integer, Integer> memoizationFunction = Memoizer.memoize(squareFunction); System.out.println(memoizationFunction.apply(2)); System.out.println(memoizationFunction.apply(2)); System.out.println(memoizationFunction.apply(2)); The output of this sequence follows where the square operation is performed only once: In function 4 4 4 We can easily use the Memoizer class for a different function as shown here: Function<Double, Double> memoizationFunction2 = Memoizer.memoize(x -> x * x); System.out.println(memoizationFunction2.apply(4.0)); This will square the number as expected. Functions that are recursive present additional problems. Eliminating dependencies between functions When dependencies between functions are eliminated, then more flexibility in the order of execution is possible. Consider these Function and BiFunction declarations, which define simple expressions for computing hourly, salaried, and sales type pay, respectively: BiFunction<Integer, Double, Double> computeHourly = (hours, rate) -> hours * rate; Function<Double, Double> computeSalary = rate -> rate * 40.0; BiFunction<Double, Double, Double> computeSales = (rate, commission) -> rate * 40.0 + commission; These functions can be executed, and their results are assigned to variables as shown here: double hourlyPay = computeHourly.apply(35, 12.75); double salaryPay = computeSalary.apply(25.35); double salesPay = computeSales.apply(8.75, 2500.0); These are pure functions as they do not use external values to perform their computations. In the following code sequence, the sum of all three pays are totaled and displayed: System.out.println(computeHourly.apply(35, 12.75) + computeSalary.apply(25.35) + computeSales.apply(8.75, 2500.0)); We can easily reorder their execution sequence or even execute them concurrently, and the results will be the same. There are no dependencies between the functions that restrict them to a specific execution ordering. Supporting lazy evaluation Continuing with this example, let's add an additional sequence, which computes the total pay based on the type of employee. The variable, hourly, is set to true if we want to know the total of the hourly employee pay type. It will be set to false if we are interested in salary and sales-type employees: double total = 0.0; boolean hourly = ...; if(hourly) { total = hourlyPay; } else { total = salaryPay + salesPay; } System.out.println(total); When this code sequence is executed with an hourly value of false, there is no need to execute the computeHourly function since it is not used. The runtime system could conceivably choose not to execute any of the lambda expressions until it knows which one is actually used. While all three functions are actually executed in this example, it illustrates the potential for lazy evaluation. Functions are not executed until needed. Referential transparency Referential transparency is the idea that a given expression is made up of subexpressions. The value of the subexpression is important. We are not concerned about how it is written or other details. We can replace the subexpression with its value and be perfectly happy. With regards to pure functions, they are said to be referentially transparent since they have same effect. In the next declaration, we declare a pure function called pureFunction: Function<Double,Double> pureFunction = t -> 3*t; It supports referential transparency. Consider if we declare a variable as shown here: int num = 5; Later, in a method we can assign a different value to the variable: num = 6; If we define a lambda expression that uses this variable, the function is no longer pure: Function<Double,Double> impureFunction = t -> 3*t+num; The function no longer supports referential transparency. Closure in Java The use of external variables in a lambda expression raises several interesting questions. One of these involves the concept of closures. A closure is a function that uses the context within which it was defined. By context, we mean the variables within its scope. This sometimes is referred to as variable capture. We will use a class called ClosureExample to illustrate closures in Java. The class possesses a getStringOperation method that returns a Function lambda expression. This expression takes a string argument and returns an augmented version of it. The argument is converted to lowercase, and then its length is appended to it twice. In the process, both an instance variable and a local variable are used. In the implementation that follows, the instance variable and two local variables are used. One local variable is a member of the getStringOperation method and the second one is a member of the lambda expression. They are used to hold the length of the target string and for a separator string: public class ClosureExample { int instanceLength; public Function<String,String> getStringOperation() { final String seperator = ":"; return target -> { int localLength = target.length(); instanceLength = target.length(); return target.toLowerCase() + seperator + instanceLength + seperator + localLength; }; } } The lambda expression is created and used as shown here: ClosureExample ce = new ClosureExample(); final Function<String,String> function = ce.getStringOperation(); System.out.println(function.apply("Closure")); Its output follows: closure:7:7 Variables used by the lambda expression are restricted in their use. Local variables or parameters cannot be redefined or modified. These variables need to be effectively final. That is, they must be declared as final or not be modified. If the local variable and separator, had not been declared as final, the program would still be executed properly. However, if we tried to modify the variable later, then the following syntax error would be generated, indicating such variable was not permitted within a lambda expression: local variables referenced from a lambda expression must be final or effectively final If we add the following statements to the previous example and remove the final keyword, we will get the same syntax error message: function = String::toLowerCase; Consumer<String> consumer = s -> System.out.println(function.apply(s)); This is because the function variable is used in the Consumer lambda expression. It also needs to be effectively final, but we tried to assign a second value to it, the method reference for the toLowerCase method. Closure refers to functions that enclose variable external to the function. This permits the function to be passed around and used in different contexts. Currying Some functions can have multiple arguments. It is possible to evaluate these arguments one-by-one. This process is called currying and normally involves creating new functions, which have one fewer arguments than the previous one. The advantage of this process is the ability to subdivide the execution sequence and work with intermediate results. This means that it can be used in a more flexible manner. Consider a simple function such as: f(x,y) = x + y The evaluation of f(2,3) will produce a 5. We could use the following, where the 2 is "hardcoded": f(2,y) = 2 + y If we define: g(y) = 2 + y Then the following are equivalent: f(2,y) = g(y) = 2 + y Substituting 3 for y we get: f(2,3) = g(3) = 2 + 3 = 5 This is the process of currying. An intermediate function, g(y), was introduced which we can pass around. Let's see, how something similar to this can be done in Java 8. Start with a BiFunction method designed for concatenation of strings. A BiFunction method takes two parameters and returns a single value: BiFunction<String, String, String> biFunctionConcat = (a, b) -> a + b; The use of the function is demonstrated with the following statement: System.out.println(biFunctionConcat.apply("Cat", "Dog")); The output will be the CatDog string. Next, let's define a reference variable called curryConcat. This variable is a Function interface variable. This interface is based on two data types. The first one is String and represents the value passed to the Function interface's accept method. The second data type represents the accept method's return type. This return type is defined as a Function instance that is passed a string and returns a string. In other words, the curryConcat function is passed a string and returns an instance of a function that is passed and returns a string. Function<String, Function<String, String>> curryConcat; We then assign an appropriate lambda expression to the variable: curryConcat = (a) -> (b) -> biFunctionConcat.apply(a, b); This may seem to be a bit confusing initially, so let's take it one piece at a time. First of all, the lambda expression needs to return a function. The lambda expression assigned to curryConcat follows where the ellipses represent the body of the function. The parameter, a, is passed to the body: (a) ->...; The actual body follows: (b) -> biFunctionConcat.apply(a, b); This is the lambda expression or function that is returned. This function takes two parameters, a and b. When this function is created, the a parameter will be known and specified. This function can be evaluated later when the value for b is specified. The function returned is an instance of a Function interface, which is passed two parameters and returns a single value. To illustrate this, define an intermediate variable to hold this returned function: Function<String,String> intermediateFunction; We can assign the result of executing the curryConcat lambda expression using it's apply method as shown here where a value of Cat is specified for the a parameter: intermediateFunction = curryConcat.apply("Cat"); The next two statements will display the returned function: System.out.println(intermediateFunction); System.out.println(curryConcat.apply("Cat")); The output will look something similar to the following: packt.Chapter2$$Lambda$3/798154996@5305068a packt.Chapter2$$Lambda$3/798154996@1f32e575 Note that these are the values representing this functions as returned by the implied toString method. They are both different, indicating that two different functions were returned and can be passed around. Now that we have confirmed a function has been returned, we can supply a value for the b parameter as shown here: System.out.println(intermediateFunction.apply("Dog")); The output will be CatDog. This illustrates how we can split a two parameter function into two distinct functions, which can be evaluated when desired. They can be used together as shown with these statements: System.out.println(curryConcat.apply("Cat").apply("Dog")); System.out.println(curryConcat.apply("Flying ").apply("Monkeys")); The output of these statements is as follows: CatDog Flying Monkeys We can define a similar operation for doubles as shown here: Function<Double, Function<Double, Double>> curryAdd = (a) -> (b) -> a * b; System.out.println(curryAdd.apply(3.0).apply(4.0)); This will display 12.0 as the returned value. Currying is a valuable approach useful when the arguments of a function need to be evaluated at different times. Summary In this article, we investigated the use of lambda expressions and how they support the functional style of programming in Java 8. When possible, we used examples to contrast the use of classes and methods against the use of functions. This frequently led to simpler and more maintainable functional implementations. We illustrated how lambda expressions support the functional concepts of high-order, first-class, and pure functions. Examples were used to help clarify the concept of referential transparency. The concepts of closure and currying are found in most functional programming languages. We provide examples of how they are supported in Java 8. Lambda expressions have a specific syntax, which we examined in more detail. Also, there are several variations of the function that can be used to support the expression in the form, which we illustrated. Lambda expressions are based on functional interfaces using type inference. It is important to understand how to create functional interfaces and to know what standard functional interfaces are available in Java 8. Resources for Article: Further resources on this subject: An Introduction to Mastering JavaScript Promises and Its Implementation in Angular.js[article] Finding Peace in REST[article] Introducing JAX-RS API [article]
Read more
  • 0
  • 0
  • 23805

article-image-editor-tool-prefabs-and-main-menu
Packt
22 Sep 2015
19 min read
Save for later

Editor Tool, Prefabs, and Main Menu

Packt
22 Sep 2015
19 min read
In this article by Edward Kyle Langley, author of the book Learning Unity iOS Game Development, we will learn that the player has the ability to send input to the device, and we will handle this by manipulating the player character GameObject. We also set up some game logic so that the player character can interact with positive and negative world objects, such as Coins and Obstacles. To further develop the sense of a complete game, we need to create the pieces of the game world that represent a floor that the player will run on. (For more resources related to this topic, see here.) To create these pieces, we will create a Unity EditorWindow class that will help us create grids that will represent the ground the player runs on and the dirt below it. Traditionally, you would have to place each sprite one at a time. With this editor tool, we will be able to crate bigger boxes in a grid based on our settings. After we have our editor tool running, we will begin to create the prefabs that will hold multiple GameObjects and their components in a single file. Finally, we will write the code needed to move the floor and ground pieces below the player character, simulating the character as running forward. To summarize, in this article, we will cover the following topics: Writing a Unity C# class that extends EditorWindow, which allows you to input settings and sprite files that will give you a box grid and simplify the level pieces creation Creating the game-related prefabs so that you have grouped files in an easy-to-use file Building the main menu user interface with Unity's UI tools, including buttons for achievements, leaderboards, and store purchases Use the prefabs we made in the C# script. This will move the level pieces of prefabs under the player character, simulating movement. We will also go through the steps to get the final aspects of the iOS integration function and set up the main menu UI so that the player can navigate between playing the game, view at leaderboards /achievements, and have the option to purchase "remove iAds" for the cost of ten thousand coins or 99 cents. Making the Sprite Tile Editor Tool The Unity engine is incredibly flexible for all the aspects of game development, including creating custom editor tools to help fast track the more tedious aspects of development. In our case, it will be beneficial to have a tool that creates a root GameObject that will then create children GameObjects in a grid. This will be spaced out by the size of the sprite component they have attached. For example, if you were to place say 24 GameObjects one at a time, it could take some time to make sure that all are snapped correctly together. With our tool, we will be able to select the X value and the Y value for the grid, the sprite that represents the ground, and the sprite that represents the dirt below the ground. Perform the following steps: To begin with, navigate to the Assets folder. Right-click on this folder and select Create and then New Folder. Name this folder Level. Right-click on the new Level folder and select Import New Asset. Right-click on the Script folder, select Create and then C# Script. Name the script SpriteTiler. The SpriteTiler C# class Double-click on the SpriteTiler C# file to open it. Change the file so that it looks similar to the following code: using UnityEngine; using UnityEditor; using System.Collections; public class SpriteTiler : EditorWindow { } The big changes from the normally generated code file is the addition to using UnityEditor, changing the inherited class to EditorWindow, and removing the Start() and Update() functions. Global variables We now want to add the global variables for this class. Add the following code in the class block:   // Grid settings to make tiled by public float GridXSlider = 1; public float GridYSlider = 1; // Sprites for both the ground and dirt public Sprite TileGroundSprite; public Sprite TileDirtSprite; // Name of the GameObject that holds our tiled Objects public string TileSpriteRootGameObjectName = "Tiled Object"; The GridXSlider and GridYSlider class will be used to generate our grid, X being left to right and Y being top down. For example, if you had X set to five and Y set to three, the grid would generate columns of five elements and rows of three elements or five sprites long and three sprites down. The TileGroundSprite and TileDirtSprite sprite files will make up the ground and dirt levels. TileSpriteRootGameObjectName is the GameObject name that will hold the GameObjects children that have the sprite components. This is editable by you so that you can choose the name of the GameObject that gets created to avoid having the default new GameObject for each one made. The MenuItem creation Next, we need to create the MenuItem function. This will represent the Editor selection drop-down list so that we can use our tool. Add the following function to the SpriteTiler class under the global variables:    // Menu option to bring up Sprite Tiler window [MenuItem("RushRunner/Sprite Tile")] public static void OpenSpriteTileWindow() { EditorWindow.GetWindow< SpriteTiler > ( true, "Sprite Tiler" ); } As this class extends EditorWindow, and the preceding function is declared as MenuItem, it will create a dropdown in the Editor named RushRunner. This will hold a selection called Sprite Tile: You can name the dropdown and selection anything you like by changing the string that is passed into MenuItem, such as MyEditorTool or Editor Tool Name. If you save the SpiteTiler.cs file and go back to Unity and allow the engine to compile, you will be able to click on the SpriteTile button under RushRunner. This will create a editor window named Sprite Tiler. The OnGUI function Next, we need to add the function that will be used to draw all the windows GUI elements or the fields that we will use to get the settings to make the grid. Under our OpenSpriteTileWindow function, add the following code: // Called to render GUI frames and elements void OnGUI() { } OnGUI is the function that will draw our GUI elements to the window. This allows you to manipulate these GUI elements so that we have values to use when we create the GameObject grid and its GameObjects children with sprite components. The GUILayout and OnGUI setup To begin with the OnGUI function, we want to add the GUI elements to the window. In the OnGUI function, add the following code:   // Setting for GameObject name that holds our tiled Objects GUILayout.Label("Tile Level Object Name", EditorStyles .boldLabel); TileSpriteRootGameObjectName = GUILayout.TextField( TileSpriteRootGameObjectName, 25 ); // Slider for X grid value (left to right) GUILayout.Label("X: " + GridXSlider, EditorStyles. boldLabel); GridXSlider = GUILayout.HorizontalScrollbar( GridXSlider, 1.0f, 0.0f, 30.0f ); GridXSlider = (int)GridXSlider; // Slider for Y grid value(up to down) GUILayout.Label("Y: " + GridYSlider, EditorStyles. boldLabel); GridYSlider = GUILayout.HorizontalScrollbar(GridYSlider, 1.0f, 0.0f, 30.0f); GridYSlider = (int)GridYSlider; // File chose to be our Ground Sprite GUILayout.Label("Sprite Ground File", EditorStyles. boldLabel); TileGroundSprite = EditorGUILayout.ObjectField (TileGroundSprite, typeof(Sprite), true) as Sprite; // File chose to be our Dirt Sprite GUILayout.Label("Sprite Dirt File", EditorStyles. boldLabel); TileDirtSprite = EditorGUILayout.ObjectField (TileDirtSprite, typeof(Sprite), true) as Sprite; GUILayout.Label is a function that creates a text label in the window we are using. Its first use is to let the user know that the next setting is for Tile Level Object Name: the name of the root GameObject that will hold children GameObjects with Sprite components. By default, this is set to Tiled Object, although we allow the user to change it. In order to allow the user to change it, we need to give them a TextField parameter to input a new string. We do this by telling that TileSpriteRootGameObjectName is equal to the GUILayout.TextField setting. As this is used in OnGUI, anything the user inputs will change the value of TileSpriteRootGameObjectName. We will use this later when the user wants to create the GameObject. We then need to create two HorizontalSlider GUI elements so that we can get values from them that represent the X and Y values of the grid. Similar to TextField, we can start each of the HorizontalSlider elements with GUILayout.Label. This describes what the slider is for. We will then assign the GridXSlider and GridYSlider values to what the HorizontalSlider element is set to, which is one by default. As the user adjusts the sliders, the GridXSlider and GridYSlider values will change so that when the user clicks on a button to create the GameObject, we will have a reference to the values that they want to use for the grid. After HorizontalSliders, we want to have ObjectFields so that the user can search for and assign sprite files that will represent the ground and dirt of the grid. EditorGUILayout.ObjectField takes a reference to the object you want to assign when the user selects one, the type of object that ObjectField wants, and if ObjectField takes SceneObjects. As we want this ObjectField to be for sprites, we will set the type of object to typeof( Sprite ) and then cast the result that is assigned to TileGroundSprite or TileDirtSprite to the sprite by using as Sprite. The OnGUI create tiled button In order to know when the user wants to create the root GameObject and its grid of children GameObjects, we will need a button. Add the following code under the last GUI Elements: // If butt "Create Tiled" is clicked if (GUILayout.Button("Create Tiled")) { // If the Grid settings are both zero, // send notification to user if (GridXSlider == 0 && GridYSlider == 0) { ShowNotification(new GUIContent("Must have either X or Y grid set to a value greater than 0")); return; } // if Dirt and Ground Sprite exist if (TileDirtSprite != null && TileGroundSprite !=null) { // If the Sprites sizes dont match, // send notifcation to user if (TileDirtSprite.bounds.size.x != TileGroundSprite. bounds.size.x || TileDirtSprite.bounds.size.y != TileGroundSprite.bounds.size.y) { ShowNotification(new GUIContent("Both Sprites must be of matching size.")); return; } // Create GameObject and tiled // Objects with user settings CreateSpriteTiledGameObject(GridXSlider, GridYSlider, TileGroundSprite, TileDirtSprite, TileSpriteRoot GameObjectName); } else { // If either Dirt or Ground Sprite dont exist, // send notifcation to user ShowNotification( new GUIContent( "Must have Dirt and Ground Sprite selected." ) ); return; } } The first condition we have set is the GUILayout.Button( "Create Tiled" ) function. The Button function will return true as soon as it is clicked on, but it will still render to the window if false. This means that although the button is not active, it'll still be seen by the user. As some settings will create a scenario that is not ideal for the concept of our SpriteTiler, we first want to make sure that the settings are in line with what we have designed the tool to perform. We will first check whether GridXSlider and GridYSlider are set to zero. If both of these values are set to zero, the grid won't create anything, and as the concept of the tool is to create a grid of children sprites, we will tell the user that they must have a selection above zero for either GridXSlider or GridYSlider. We then check whether TileDirtSprite and TileGroundSprite have a value. If either of these values are null, the settings are not complete. This results in you telling the user that Dirt and Ground sprites need a selection. If the user has set Dirt and Ground sprites to something, but their sizing is not the same, such as one being 32 x 32 and the other being 64 x 64, we will tell the user that both the sprites need to be of the same size. If we didn't check for this, the grid wouldn't align correctly, creating negative results and making the tool not function as we want it to. If the user settings are in order, we will call the CreateSpriteTiledGameObject function and pass GridXSlider, GridYSlixer, TileGroundSprite, TileDirtSprite, and TileSpriteRootGameObjectName. The CreateSpriteTiledGameObject function This function is designed to take the user settings and create the grid from them. Add the following function under the OnGUI function: // Create GameObject and tiled childen based on user settings public static void CreateSpriteTiledGameObject(float GridXSlider, float GridYSlider, Sprite SpriteGroundFile, Sprite SpriteDirtFile, string RootObjectName) { // Store size of Sprite float spriteX = SpriteGroundFile.bounds.size.x; float spriteY = SpriteGroundFile.bounds.size.y; // Create the root GameObject which will hold children that tile GameObject rootObject = new GameObject( ); // Set position in world to 0,0,0 rootObject.transform.position = new Vector3( 0.0f, 0.0f, 0.0f ); // Name it based on user settings rootObject.name = RootObjectName; // Create starting values for while loop int currentObjectCount = 0; int currentColumn = 0; int currentRow = 0; Vector3 currentLocation = new Vector3( 0.0f, 0.0f, 0.0f ); // Continue loop until all rows // and columns have been filled while (currentRow < GridYSlider) { // Create a child GameObject, set its parent to root, // name it, and offset its location based on current location GameObject gridObject = new GameObject( ); gridObject.transform.SetParent( rootObject.transform ); gridObject.name = RootObjectName + "_" + currentObjectCount; gridObject.transform.position = currentLocation; // Give child gridObject a SpriteRenderer and set sprite on CurrentRow SpriteRenderer gridRenderer = gridObject.AddComponent <SpriteRenderer>( ); gridRenderer.sprite = ( currentRow == 0 ) ? SpriteGroundFile : SpriteDirtFile; // Give the gridObject a BoxCollider gridObject.AddComponent<BoxCollider2D>(); // Offset currentLocation for next gridObject to use currentLocation.x += spriteX; // Increment current column by one currentColumn++; // If the current collumn is greater than the X slider if (currentColumn >= GridXSlider) { // Reset column, incrmement row, reset x location // and offset y location downwards currentColumn = 0; currentRow++; currentLocation.x = 0; currentLocation.y -= spriteY; } // Add to currentObjectCount for naming of // gridObject children. currentObjectCount++; } } To start with, we must first have the X and Y sizes of the sprite we want to create so that we can offset the location of the children GameObjects that were created. As we originally checked to make sure that both sprites are of the same size, it doesn't matter which sprite object we get the size from. In our case, we will use SpriteGroundFile. We will then move the rootObject position to 0X, 0Y, and 0Z so that it is in the center of our scene. This can be set to anything you like, although when rootObject and its children get created, it is easier to find it at the center of the scene world. After it has been moved, we can set its name to the setting that the user had entered or Tiled Object (the default one). Once we have rootObject set up, we can create its children GameObjects. To start this cycle, we will need a few variables to reference and change: currentObjectCount: This specifies the total number of children that will be created. This increments for each one created. currentColumn: This denotes the current column we are on in the row. currentRow: This specifies the current row we are on. currentLocation: This denotes the current location that the children GameObject will use and sets its position too. This is changed after each new child is created based on the X or Y setting of the sprite size. Now that we have our rootObject and the variables we need to create the children, we can use while loop. A while loop is a loop that will continue until its condition fails. In our case, we will check whether currentRow is less than the GridYSlider value. As soon as currentRow is equal to or greater than GridYSlider, the loop will stop because the condition failed. The reason we will look at currentRow is that for each column created, we can reset its value to zero and increment currentRow by one. This means that each row will hold as many columns as were set by the GridXSlider value, and we know that the grid is complete when currentRow is equal or greater than GridYSlider. For example, if we had a grid setting of 3X and 3Y, the first row will hold three columns. When the first row is done, the row changes to two and adds three more columns. In the last row, it completes three more columns and then the while condition fails because the row value is equal to GridYSlider. In each loop of the while loop, we start by creating gridObject. We set this grid object parent to that of rootObject, set its name to RootObjectName, and concatenate an underscore, followed by currentObjectCount and then set the gridObject position to the currentLocation value, which will change based on the size of the sprite and the column/row. We will then add a SpriteRenderer component to gridObject and assign a sprite to it. We will change the sprite based on whether currentRow is equal to zero or not. If it is, in the first row, we will set the sprite to SpriteGroundFile. If currentRow is not equal to zero, we will set the sprite to SpriteDirtFile. The ternary operator is a sort of shorthand for if → else. If the condition is true, we will set the value to what is behind the question mark. If the condition is false, we will set the value based on what's behind the colon. The question mark represents if, whereas the colon represents else. The ternary operator is as follows: Value = ( condtion == true ) ? ifTrue : elseNotTrue; Once we have the sprite assigned to the SpriteRenderer component of gridObject, we can assign a BoxCollider2D component, which will make itself the same size as the sprite. If we were to add the BoxCollider2D component to SpriteRenderer, it would be the default size of 1, 1, 1, which would be too big. We will then offset currentLocation by the spriteX size, so the next gridObject will offset the size of the spriteX size. The currentColumn value is incremented by one, and we then check whether currentColumn is greater than or equal to the GridXSlider value. If it is, we know that we need to start the next row. To do this, we reset currentColumn to zero, increment currentRow by one, set the currentLocation.x value to zero, and offset currentLocation.y by negative spriteY size. This not only results in an offset location down, but also resets the X value to zero, making it possible for the columns to be created again; just down the size of spriteY. Finally, we increment currentObjectCount by one. Building the main menu UI The main menu UI will be its own Canvas GameObject. We will then handle the main menu and the game UI via the GameInfo class. We will also use the GameInfo class to manage button presses and the iOS integration. In Hierarchy, right-click and select UI and then click on Canvas. Name this new Canvas GameObject MenuUI. Let's start by adding five buttons to achievements, playing, leaderboards, remove iAds, and restore purchase. Right-click on the new MenuUI GameObject, navigate to UI, and left-click on Button. Do this four more times, so there are a total of five buttons that are children of the MenuUI GameObject. Name the buttons and text children as follows: PlayButton, PlayText LeaderboardButton, LeaderboardText AchievementButton, AchievementText RemoveAdsButton, RemoveAdsText RestorePurchaseButton, RestorePurchaseText Adding button images Next, we need to import the art that will be used for the main menu UI. In the Assets | UI folder, right-click and select Import New Asset. Select all the new images in the Assets | UI folder and change their settings as follows: Filter Mode: Trilinear Max Size: 256 Format: Truecolor PlayButton Select PlayButton in Hierarchy and search for Inspector. Change its settings as follows: Anchor: Bottom Center Pos X: 0 Pos Y: 115 Pos Z: 0 Width: 128 Height: 128 Source Image: MenuButton Now, select PlayButtonText. In the Inspector window, change its settings as follows: Text: Play Font: Arial Font Style: Bold Font Size: 36 Alignment: Center LeaderboardButton Select LeaderboardButton in the Hierarchy tab and search for Inspector. Change its settings as follows: Anchor: Bottom Center Pos X: 135 Pos Y: 115 Pos Z: 0 Width: 128 Height: 128 Source Image: MenuButton Select LeaderboardText. In the Inspector window, change its settings to: Text: Leaderboards Font: Arial Font Style: Bold Font Size: 17 Alignment: Center AchievementButton Select AchievementButton. In Hierarchy, search for Inspector. Change its settings as follows: Anchor: Bottom Center Pos X: -135 Pos Y: 115 Pos Z: 0 Width: 128 Height: 128 Source Image: MenuButton Now, select AchievementText and then in Inspector, change its settings to: Text: Achievements Font: Arial Font Style: Bold Font Size: 17 Alignment: Center RemoveAdsButton Select RemoveAdsButton in the Hierarchy tab and navigate to Inspector. Change its settings as follows: Anchor: Bottom Center Pos X: -64 Pos Y: 55 Pos Z: 0 Width: 96 Height: 42 Source Image: RestartButton Now, select RemoveAdsText and then in the Inspector window, change its settings as shown here: Text: Remove iAds Font: Arial Font Style: Bold Font Size: 12 Alignment: Center RestorePurchaseButton Let's select RestorePurchaseButton in the Hierarchy tab and search for Inspector. Change its settings as follows: Anchor: Bottom Center Pos X: 64 Pos Y: 55 Pos Z: 0 Width: 96 Height: 42 Source Image: RestartButton Now, select RestorePurchaseText and then in the Inspector window, change its settings as follows: Text: Restore Purchase Font: Arial Font Style: Bold Font Size: 14 Alignment: Center You should now have a button layout that looks similar to the following image: Summary In this article, we discussed how to create a Unity editor tool and a grid of GameObjects. These were laid out by the size of the sprites you chose and were flexible enough to use with your own settings. We also created prefabs for all of our bigger GameObjects, which could hold all of their components in a neat package. We also covered the basics of how to create a game for iOS and utilize its GameCenter features. Feel free to explore these features and add to them. Adding more store purchases, achievements, and leaderboards is simply repeating the steps that we have already done. Resources for Article: Further resources on this subject: Components in Unity[article] Saying Hello to Unity and Android [article] Unity Networking – The Pong Game [article]
Read more
  • 0
  • 0
  • 2328

article-image-implementing-decision-trees
Packt
22 Sep 2015
4 min read
Save for later

Implementing Decision Trees

Packt
22 Sep 2015
4 min read
 In this article by the author, Sunila Gollapudi, of this book, Practical Machine Learning, we will outline a business problem that can be addressed by building a decision tree-based model, and see how it can be implemented in Apache Mahout, R, Julia, Apache Spark, and Python. This can happen many, many times. So, building a website or an app will take a bit longer than it used to. (For more resources related to this topic, see here.) Implementing decision trees Here, we will explore implementing decision trees using various frameworks and tools. The R example We will use the rpart and ctree packages in R to build decision tree-based models: Import the packages for data import and decision tree libraries as shown here: Start data manipulation: Create a categorical variable on Sales and append to the existing dataset as shown here: Using random functions, split data into training and testing datasets; Fit the tree model with training data and check how the model is working with testing data, measure the error: Prune the tree; Plotting the pruned tree will look like the following: The Spark example Java-based example using MLib is shown here: import java.util.HashMap; import scala.Tuple2; import org.apache.spark.api.java.JavaPairRDD; import org.apache.spark.api.java.JavaRDD; import org.apache.spark.api.java.JavaSparkContext; import org.apache.spark.api.java.function.Function; import org.apache.spark.api.java.function.PairFunction; import org.apache.spark.mllib.regression.LabeledPoint; import org.apache.spark.mllib.tree.DecisionTree; import org.apache.spark.mllib.tree.model.DecisionTreeModel; import org.apache.spark.mllib.util.MLUtils; import org.apache.spark.SparkConf; SparkConf sparkConf = new SparkConf().setAppName("JavaDecisionTree"); JavaSparkContext sc = new JavaSparkContext(sparkConf); // Load and parse the data file. String datapath = "data/mllib/sales.txt"; JavaRDD<LabeledPoint> data = MLUtils.loadLibSVMFile(sc.sc(), datapath).toJavaRDD(); // Split the data into training and test sets (30% held out for testing) JavaRDD<LabeledPoint>[] splits = data.randomSplit(new double[]{0.7, 0.3}); JavaRDD<LabeledPoint> trainingData = splits[0]; JavaRDD<LabeledPoint> testData = splits[1]; // Set parameters. // Empty categoricalFeaturesInfo indicates all features are continuous. Integer numClasses = 2; Map<Integer, Integer> categoricalFeaturesInfo = new HashMap<Integer, Integer>(); String impurity = "gini"; Integer maxDepth = 5; Integer maxBins = 32; // Train a DecisionTree model for classification. final DecisionTreeModel model = DecisionTree.trainClassifier(trainingData, numClasses, categoricalFeaturesInfo, impurity, maxDepth, maxBins); // Evaluate model on test instances and compute test error JavaPairRDD<Double, Double> predictionAndLabel = testData.mapToPair(new PairFunction<LabeledPoint, Double, Double>() { @Override public Tuple2<Double, Double> call(LabeledPoint p) { return new Tuple2<Double, Double>(model.predict(p.features()), p.label()); } }); Double testErr = 1.0 * predictionAndLabel.filter(new Function<Tuple2<Double, Double>, Boolean>() { @Override public Boolean call(Tuple2<Double, Double> pl) { return !pl._1().equals(pl._2()); } }).count() / testData.count(); System.out.println("Test Error: " + testErr); System.out.println("Learned classification tree model:n" + model.toDebugString()); The Julia example We will use the DecisionTree package in Julia as shown here; julia> Pkg.add("DecisionTree")julia> using DecisionTree We will use the RDatasets package to load the dataset for the example in context; julia> Pkg.add("RDatasets"); using RDatasets julia> sales = data("datasets", "sales"); julia> features = array(sales[:, 1:4]); # use matrix() for Julia v0.2 julia> labels = array(sales[:, 5]); # use vector() for Julia v0.2 julia> stump = build_stump(labels, features); julia> print_tree(stump) Feature 3, Threshold 3.0 L-> price : 50/50 R-> shelvelock : 50/100 Pruning the tree julia> length(tree) 11 julia> pruned = prune_tree(tree, 0.9); julia> length(pruned) 9 Summary In this article, we implemented decision trees using R, Spark, and Julia. Resources for Article: Further resources on this subject: An overview of common machine learning tasks[article] How to do Machine Learning with Python[article] Modeling complex functions with artificial neural networks [article]
Read more
  • 0
  • 0
  • 37536

Packt
22 Sep 2015
16 min read
Save for later

R ─ Classification and Regression Trees

Packt
22 Sep 2015
16 min read
"The classifiers most likely to be the best are the random forest (RF) versions, the best of which (implemented in R and accessed via caret), achieves 94.1 percent of the maximum accuracy overcoming 90 percent in the 84.3 percent of the data sets."                                                                          – Fernández-Delgado et al (2014) "You can't see the forest for the trees!"                                                                                                     – An old saying (For more resources related to this topic, see here.) In this article by Cory Lesmeister, the author of Mastering Machine Learning with R, the first item of discussion is the basic decision tree, which is both simple to build and understand. However, the single decision tree method does not perform as well as the other methods such as support vector machines or neural networks. Therefore, we will discuss the creation of multiple, sometimes hundreds of, different trees with their individual results combined, leading to a single overall prediction. The first quote written above is from Fernández-Delgado et al in the Journal of Machine Learning Research and is meant to set the stage that the techniques in this article are quite powerful, particularly when used for the classification problems. Certainly, they are not always the best solution, but they do provide a good starting point. Regression trees For an understanding of the tree-based methods, it is probably easier to start with a quantitative outcome and then move on to how it works on a classification problem. The essence of a tree is that the features are partitioned, starting with the first split that improves the residual sum of squares the most. These binary splits continue until the termination of the tree. Each subsequent split/partition is not done on the entire dataset but only on the portion of the prior split that it falls under. This top-down process is referred as recursive partitioning. It is also a process that is greedy, a term you may stumble on in reading about the machine learning methods. Greedy means that in each split in the process, the algorithm looks for the greatest reduction in the residual sum of squares without a regard to how well it will perform in the later partitions. The result is that you may end up with a full tree of unnecessary branches, leading to a low bias but high variance. To control this effect, you need to appropriately prune the tree to an optimal size after building a full tree. The following figure provides a visual of the technique in action. The data is hypothetical with 30 observations, a response ranging from 1 to 10, and two predictor features, both ranging in value from 0 to 10 named X1 and X2. The tree has three splits that lead to four terminal nodes. Each split is basically an if or then statement or uses an R syntax, ifelse(). In the first split, if X1 < 3.5, then the response is split into 4 observations with an average value of 2.4 and the remaining 26 observations. This left branch of 4 observations is a terminal node as any further splits would not substantially improve the residual sum of squares. The predicted value for the 4 observations in that partition of the tree becomes the average. The next split is at X2 < 4 and finally X1 < 7.5. An advantage of this method is that it can handle the highly nonlinear relationships; but can you see a couple of potential problems? The first issue is that an observation is given the average of the terminal node that it falls under. This can hurt the overall predictive performance (high bias). Conversely, if you keep partitioning the data further and further to achieve a low bias, high variance can become an issue. As with the other methods, you can use cross-validation to select the appropriate tree size. Regression Tree with 3 splits and 4 terminal nodes and the corresponding node average and number of observations. Classification trees Classification trees operate under the same principal as regression trees except that the splits are not determined by the residual sum of squares but an error rate. The error rate used is not what you would expect, where the calculation is simply misclassified observations divided by the total observations. As it turns out, when it comes to tree splitting, a misclassification rate by itself may lead to a situation where you can gain information with a further split but not improve the misclassification rate. Let's look at an example. Suppose we have a node—let's call it N0 where you have 7 observations labeled No and 3 observations labeled Yes, and we say that the misclassified rate is 30 percent. With this in mind, let's calculate a common alternative error measure called Gini index. The formula for a single node Gini index is as follows: Gini = 1 – (probability of Class 1)2 – (probability of Class 2)2. For N0, the Gini is 1 - (.7)2 - (.3)2, which is equal to 0.42, versus the misclassification rate of 30 percent. Taking this example further, we will now create an N1 node with 3 of Class 1 and none of Class 2 along with N2, which has 4 observations from Class 1 and 3 from Class 2. Now, the overall misclassification rate for this branch of the tree is still 30 percent, but look at the following to see how the overall Gini index has improved: Gini(N1) = 1 – (3/3)2 – (0/3)2 = 0. Gini(N2) = 1 – (4/7)2 – (3/7)2 = 0.49. The new Gini index = (proportion of N1 x Gini(N1)) + (proportion of N2 x Gini(N2)) which is equal to (.3 x 0) + (.7 x 0.49) or 0.343. By doing a split on a surrogate error rate, we actually improved our model impurity by reducing it from 0.42 to 0.343, whereas the misclassification rate did not change. This is the methodology used by the rpart() package. Random forest To greatly improve our model's predictive ability, we can produce numerous trees and combine the results. The random forest technique does this by applying two different tricks in the model development. The first is the use of bootstrap aggregation or bagging as it is called. In bagging, an individual tree is built on a sample of dataset, roughly two-thirds of the total observations. It is important to note that the remaining one-third is referred to as Out of Bag(OOB). This is repeated for dozens or hundreds of times and the results are averaged. Each of these trees is grown and not pruned based on any error measure and this means that the variance of each of these individual trees is high. However, by averaging the results, you can reduce the variance without increasing the bias. The next thing that the random forest brings to the table is that concurrently with the random sample of the data, it also takes a random sample of the input features at each split. In the randomForest package, we will use the default random number of the sampled predictors, which is the square root of the total predictors for classification problems and total predictors divided by 3 for regression. The number of predictors that the algorithm randomly chooses at each split can be changed via the model tuning process. By doing this random sampling of the features at each split and incorporating it into the methodology, you mitigate the effect of a highly correlated predictor in becoming the main driver in all of your bootstrapped trees and preventing you from reducing the variance that you hoped to achieve with bagging. The subsequent averaging of the trees that are less correlated to each other than if you only performed bagging, is more generalizable and more robust to outliers. Gradient boosting The boosting methods can become extremely complicated for you to learn and understand, but you should keep in mind about what is fundamentally happening behind the curtain. The main idea is to build an initial model of some kind (linear, spline, tree, and so on.) called the base-learner, examine the residuals, and fit a model based on these residuals around the so-called loss function. A loss function is merely the function that measures the discrepancy between the model and desired prediction, for example, a squared error for the regression or the logistic function for the classification. The process continues until it reaches some specified stopping criterion. This is like the student who takes a practice exam and gets 30 out of 100 questions wrong and as a result, studies only those 30 questions that were missed. The next practice exam they get 10 out of these 30 wrong and so only focus on these 10 questions and so on. If you would like to explore the theory behind this further, a great resource for you is available in Frontiers in Neurorobotics, Gradient boosting machines, a tutorial, Natekin A., Knoll A., (2013), at http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3885826/. As previously mentioned, boosting can be applied to many different base learners, but here we will only focus on the specifics of tree-based learning. Each tree iteration is small and we will determine how small it is with one of the tuning parameters referred to as interaction depth. In fact, it may be as small as one split, which is referred to as a stump. Trees are sequentially fit to the residuals according to the loss function up to the number of trees that we specified (our stopping criterion). There is another tuning parameter that we will need to identify and that is shrinkage. You can think of shrinkage as the rate at which your model is learning generally and specifically, as the contribution of each tree or stump to the model. This learning rate acts as a regularization parameter. The other thing about our boosting algorithm is that it is stochastic, meaning that it adds randomness by taking a random sample of our data at each tree. Introducing some randomness to a boosted model usually improves the accuracy and speed and reduces overfitting (Friedman 2002). As you may have guessed, tuning these parameters can be quite a challenge. These parameters can interact with each other and if you just tinker with one without considering the other, your model may actually perform worse. The caret package will help us in this endeavor. Business case The overall business objective in this situation is to see if we can improve the predictive ability for some of the cases. For regression, we will visit the prostate cancer data. For classification purposes, we will utilize both the breast cancer biopsy data and Pima Indian Diabetes data. Both random forests and boosting will be applied to all the three datasets. The simple tree method will be used only on the breast and prostate cancer sets. Regression tree We will jump right into the prostate data set, but first let's load the necessary R package, as follows: > library(rpart) #classification and regression trees > library(partykit) #treeplots > library(MASS) #breast and pima indian data > library(ElemStatLearn) #prostate data > library(randomForest) #random forests > library(gbm) #gradient boosting > library(caret) #tune hyper-parameter First, we will do regression on the prostate data. This involves calling the dataset, coding the gleason score as an indicator variable using the ifelse() function, and creating a test and training set. The training set will be pros.train and the test set will be pros.test, as follows: > data(prostate) > prostate$gleason = ifelse(prostate$gleason == 6, 0, 1) > pros.train = subset(prostate, train==TRUE)[,1:9] > pros.test = subset(prostate, train==FALSE)[,1:9] To build a regression tree on the training data, we will use the following rpart() function from R's party package. The syntax is quite similar to what we used in the other modeling techniques: > tree.pros <- rpart(lpsa~., data=pros.train) We can call this object using the print() function and cptable and then examine the error per split to determine the optimal number of splits in the tree, as follows: > print(tree.pros$cptable) CP nsplit rel error xerror xstd 1 0.35852251 0 1.0000000 1.0364016 0.1822698 2 0.12295687 1 0.6414775 0.8395071 0.1214181 3 0.11639953 2 0.5185206 0.7255295 0.1015424 4 0.05350873 3 0.4021211 0.7608289 0.1109777 5 0.01032838 4 0.3486124 0.6911426 0.1061507 6 0.01000000 5 0.3382840 0.7102030 0.1093327 This is a very important table to analyze. The first column labeled CP is the cost complexity parameter, which states that the second column, nsplit, is the number of splits in the tree. The rel error column stands for relative errors and is the residual sum of squares for that number of splits divided by the residual sum of squares for no splits (RSS(k)/RSS(0). Both xerror and xstd are based on a ten-fold cross-validation with xerror being the average error and xstd being the standard deviation of the cross-validation process. We can see that four splits produced slightly less errors using cross-validation while five splits produced the lowest error on the full dataset. You can examine this using the plotcp() function, as follows: > plotcp(tree.pros) The following is the output of the preceding command: The plot shows us the relative error by the tree size with the corresponding error bars. The horizontal line on the plot is the upper limit of the lowest standard error. Selecting the tree size 5, which is four splits, we can build a new tree object where xerror is minimized by pruning our tree accordingly—first creating an object for cp associated with the pruned tree from the table. Then the prune() function handles the rest as follows: > cp = min(tree.pros$cptable[5,]) > prune.tree.pros <- prune(tree.pros, cp = cp) With this done, you can plot and compare the full and pruned trees. The tree plots produced by the partykit package are much better than those produced by the party package. You can simply use the as.party() function as a wrapper in the plot() function: > plot(as.party(tree.pros)) The output of the preceding command is as follows: > plot(as.party(prune.tree.pros)) The following is the output of the preceding command: Note that the splits are exactly the same in the two trees with the exception of the last split, which includes the age variable for the full tree. Interestingly, both the first and second splits in the tree are related to the log of cancer volume lcavol. These plots are quite informative as they show the splits, nodes, observations per node, and box plots of the outcome that we are trying to predict. Let's see how well the pruned tree performs on the test data. What we will do is create an object of the predicted values using the predict() function by incorporating the test data. Then, we will calculate the errors as the predicted values minus the actual values and finally the mean of the squared errors, as follows: > party.pros.test <- predict(prune.tree.pros, newdata=pros.test) > rpart.resid = party.pros.test - pros.test$lpsa #calculate residuals > mean(rpart.resid^2) #caluclate MSE [1] 0.5267748 One can look at the tree plots that we produced and easily explain what are the primary drivers behind the response. As mentioned in the introduction, the trees are easy to interpret and explain, which, in many cases, may be more important than the accuracy. Classification tree For the classification problem, we will prepare the breast cancer data. After loading the data, you will delete the patient ID, rename the features, eliminate the few missing values, and then create the train/test datasets, as follows: > data(biopsy) > biopsy <- biopsy[,-1] #delete ID > names(biopsy) = c("thick", "u.size", "u.shape", "adhsn", "s.size", "nucl", "chrom", "n.nuc", "mit", "class") #change the feature names > biopsy.v2 = na.omit(biopsy) #delete the observations with missing values > set.seed(123) #random number generator > ind = sample(2, nrow(biopsy.v2), replace=TRUE, prob=c(0.7, 0.3)) > biop.train = biopsy.v2[ind==1,] #the training data set > biop.test = biopsy.v2[ind==2,] #the test data set With the data set up appropriately, we will use the same syntax style for a classification problem as we did previously for a regression problem, but before creating a classification tree, we will need to ensure that the outcome is a factor, which can be done using the str() function. as follows: > str(biop.test[,10]) Factor w/ 2 levels "benign","malignant": 1 1 1 1 1 2 1 2 1 1 … First, we will create the tree: > set.seed(123) > tree.biop <- rpart(class~., data=biop.train) Then, examine the table for the optimal number of splits: > print(tree.biop$cptable) CP nsplit rel error xerror xstd 1 0.79651163 0 1.0000000 1.0000000 0.06086254 2 0.07558140 1 0.2034884 0.2674419 0.03746996 3 0.01162791 2 0.1279070 0.1453488 0.02829278 4 0.01000000 3 0.1162791 0.1744186 0.03082013 The cross-validation error is at a minimum with only two splits (row 3). We can now prune the tree, plot the full and pruned tree, and see how it performs on the test set, as follows: > cp = min(tree.biop$cptable[3,]) > prune.tree.biop <- prune(tree.biop, cp = cp) > plot(as.party(tree.biop)) > plot(as.party(prune.tree.biop)) An examination of the tree plots shows that the uniformity of the cell size is the first split, then bare nuclei. The full tree had an additional split at the cell thickness. We can predict the test observations using type="class" in the predict() function: > rparty.test <- predict(prune.tree.biop, newdata=biop.test, type="class") > table(rparty.test, biop.test$class) rparty.test benign malignant benign 136 3 malignant 6 64 > (136+64)/209 [1] 0.9569378 The basic tree with just two splits gets us almost 96 percent accuracy. This still falls short but should encourage us to believe that we can improve on it with the upcoming methods, starting with random forests. Summary In this article we learned both the power and limitations of tree-based learning methods for both classification and regression problems. To improve on predictive ability, we have the tools of the random forest and gradient boosted trees at our disposal. Resources for Article: Further resources on this subject: Big Data Analysis (R and Hadoop) [article] Using R for Statistics, Research, and Graphics [article] First steps with R [article]
Read more
  • 0
  • 0
  • 7948
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-getting-started-apache-spark-dataframes
Packt
22 Sep 2015
5 min read
Save for later

Getting Started with Apache Spark DataFrames

Packt
22 Sep 2015
5 min read
 In this article article about Arun Manivannan’s book Scala Data Analysis Cookbook, we will cover the following recipes: Getting Apache Spark ML – a framework for large-scale machine learning Creating a data frame from CSV (For more resources related to this topic, see here.) Getting started with Apache Spark Breeze is the building block of Spark MLLib, the machine learning library for Apache Spark. In this recipe, we'll see how to bring Spark into our project (using SBT) and look at how it works internally. The code for this recipe could be found at https://github.com/arunma/ScalaDataAnalysisCookbook/blob/master/chapter1-spark-csv/build.sbt. How to do it... Pulling Spark ML into our project is just a matter of adding a few dependencies on our build.sbt file: spark-core, spark-sql, and spark-mllib: Under a brand new folder (which will be our project root), we create a new file called build.sbt. Next, let's add to the project dependencies the Spark libraries: organization := "com.packt" name := "chapter1-spark-csv" scalaVersion := "2.10.4" val sparkVersion="1.3.0" libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % sparkVersion, "org.apache.spark" %% "spark-sql" % sparkVersion, "org.apache.spark" %% "spark-mllib" % sparkVersion ) resolvers ++= Seq( "Apache HBase" at "https://repository.apache.org/content/repositories/releases", "Typesafe repository" at "http://repo.typesafe.com/typesafe/releases/" ) How it works... Spark has four major higher level tools built on top of the Spark Core: Spark Streaming, Spark ML Lib (Machine Learning), Spark SQL (An SQL interface for accessing data), and GraphX (for graph processing). The Spark Core is the heart of Spark, providing higher level abstractions in various languages for data representation, serialization, scheduling, metrics, and so on. For this recipe, we skipped streaming and GraphX and added the remaining three libraries. There’s more… Apache Spark is a cluster computing platform that claims to run about 100 times faster than Hadoop (that's a mouthful). In our terms, we could consider that as a means to run our complex logic over a massive amount of data at a blazingly high speed. The other good thing about Spark is that the programs we write are much smaller than the typical Map Reduce classes that we write for Hadoop. So, not only do our programs run faster, but it also takes lesser time to write them in the first place. Creating a data frame from CSV In this recipe, we'll look at how to create a new data frame from a Delimiter Separated Values (DSV) file. The code for this recipe could be found athttps://github.com/arunma/ScalaDataAnalysisCookbook/tree/master/chapter1-spark-csv in the DataFrameCSV class. How to do it... CSV support isn't first-class in Spark but is available through an external library from databricks. So, let's go ahead and add that up in build.sbt: After adding the spark-csv dependency, our complete build.sbt looks as follows: organization := "com.packt" name := "chapter1-spark-csv" scalaVersion := "2.10.4" val sparkVersion="1.3.0" libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % sparkVersion, "org.apache.spark" %% "spark-sql" % sparkVersion, "org.apache.spark" %% "spark-mllib" % sparkVersion, "com.databricks" %% "spark-csv" % "1.0.3" ) resolvers ++= Seq( "Apache HBase" at"https://repository.apache.org/content/repositories/releases", "Typesafe repository" at "http://repo.typesafe.com/typesafe/releases/" ) fork := true Before we create the actual data frame, there are three steps that we ought to do: create the Spark configuration, create the Spark context, and create the SQL context. SparkConf holds all of the information for running this Spark cluster. For this recipe, we are running locally, and we intend to use only two cores in the machine—local[2]: val conf = new SparkConf().setAppName("csvDataFrame").setMaster("local[2]") For this recipe, we'll be running Spark on standalone mode. Now let's load our pipe-separated file: org.apache.spark.sql.DataFrame val students=sqlContext.csvFile(filePath="StudentData.csv", useHeader=true, delimiter='|') How it works... The csvFile function of sqlContext accepts the full filePath of the file to be loaded. If the CSV has a header, then the useHeader flag will read the first row as column names. The delimiter flag, as expected, defaults to a comma, but you can override the character as needed. Instead of using the csvFile function, you can also use the load function available in the SQL context. The load function accepts the format of the file (in our case, it is CSV) and options as a map. We can specify the same parameters that we specified earlier using Map, like this: val options=Map("header"->"true", "path"->"ModifiedStudent.csv") val newStudents=sqlContext.load("com.databricks.spark.csv",options) Summary In this article, you learned in detail Apache Spark ML, a framework for large-scale machine learning. Then we saw the creation of a data frame from CSV with the help of example code. Resources for Article: Further resources on this subject: Integrating Scala, Groovy, and Flex Development with Apache Maven[article] Ridge Regression[article] Reactive Data Streams [article]
Read more
  • 0
  • 0
  • 9371

article-image-prototyping-levels-prototype
Packt
22 Sep 2015
13 min read
Save for later

Prototyping Levels with Prototype

Packt
22 Sep 2015
13 min read
Level design 101 – planning Now, just because we are going to be diving straight into Unity, I feel it's important to talk a little more about how level design is done in the game industry. While you may think a level designer will just jump into the editor and start playing, the truth is you would normally need to do a ton of planning ahead of time before you even open up your tool. Generally, a level begins with an idea. This can come from anything; maybe you saw a really cool building or a photo on the Internet gave you a certain feeling; maybe you want to teach the player a new mechanic. Turning this idea into a level is what a level designer does. Taking all of these ideas, the level designer will create a level design document, which will outline exactly what you're trying to achieve with the entire level from start to end. In this article by John Doran, author of Building FPS Games with Unity, a level design document will describe everything inside the level; listing all of the possible encounters, puzzles, so on and so forth, which the player will need to complete as well as any side quests that the player will be able to achieve. To prepare for this, you should include as many references as you can with maps, images, and movies similar to what you're trying to achieve. If you're working with a team, making this document available on a website or wiki will be a great asset so that you know exactly what is being done in the level, what the team can use in their levels, and how difficult their encounters can be. Generally, you'll also want a top-down layout of your level done either on a computer or with a graph paper, with a line showing a player's general route for the level with the encounters and missions planned out. (For more resources related to this topic, see here.) Of course, you don't want to be too tied down to your design document. It will change as you playtest and work on the level, but the documentation process will help solidify your ideas and give you a firm basis to work from. For those of you interested in seeing some level design documents, feel free to check out Adam Reynolds' Level Designer on Homefront and Call of Duty: World at War at http://wiki.modsrepository.com/index.php?title=Level_Design:_Level_Design_Document_Example. If you want to learn more about level design, I'm a big fan of Beginning Game Level Design by John Feil (previously, my teacher) and Marc Scattergood, Cengage Learning PTR. For more of an introduction to all of game design from scratch, check out Level Up!: The Guide to Great Video Game Design by Scott Rogers and Wiley and The Art of Game Design by Jesse Schel. For some online resources, Scott has a neat GDC talk called Everything I Learned About Level Design I Learned from Disneyland, which can be found at http://mrbossdesign.blogspot.com/2009/03/everything-i-learned-about-game-design.html, and World of Level Design (http://worldofleveldesign.com/) is a good source to learn about level design, though it does not talk about Unity specifically. In addition to a level design document, you can also create a game design document (GDD) that goes beyond the scope of just the level and includes story, characters, objectives, dialogue, concept art, level layouts, and notes about the game's content. However, it is something to do on your own. Creating architecture overview As a level designer, one of the most time-consuming parts of your job will be creating environments. There are many different ways out there to create levels. By default, Unity gives us some default meshes such as a Box, Sphere, and Cylinder. While it's technically possible to build a level in this way, it could get really tedious very quickly. Next, I'm going to quickly go through the most popular options to build levels for the games made in Unity before we jump into building a level of our own. 3D modelling software A lot of times, opening up a 3D modeling software package and building an architecture that way is what professional game studios will often do. This gives you maximum freedom to create your environment and allows you to do exactly what it is you'd like to do; but it requires you to be proficient in that tool, be it Maya, 3ds Max, Blender (which can be downloaded for free at blender.org), or some other tool. Then, you just need to export your models and import them into Unity. Unity supports a lot of different formats for 3D models (most commonly used are .obj and .fbx), but there are a lot of issues to consider. For some best practices when it comes to creating art assets, please visit http://blogs.unity3d.com/2011/09/02/art-assets-best-practice-guide/. Constructing geometry with brushes Constructive Solid Geometry (CSG), commonly referred to as brushes, is a tool artists/designers use to quickly block out pieces of a level from scratch. Using brushes inside the in-game level editor has been a common approach for artists/designers to create levels. Unreal Engine 4, Hammer, Radiant, and other professional game engines make use of this building structure, making it quite easy for people to create and iterate through levels quickly through a process called white-boxing, as it's very easy to make changes to the simple shapes. However; just like learning a modeling software tool, there can be a higher barrier for entry in creating complex geometry using a 3D application, but using CSG brushes will provide a quick solution to create shapes with ease. Unity does not support building things like this by default, but there are several tools in the Unity Asset Store, which allow you to do something like this. For example, sixbyseven studio has an extension called ProBuilder that can add this functionality to Unity, making it very easy to build out levels. The only possible downside is the fact that it does cost money, though it is worth every penny. However, sixbyseven has kindly released a free version of their tools called Prototype, which we installed earlier. It contains everything we will need for this chapter, but it does not allow us to add custom textures and some of the more advanced tools. We will be using ProBuilder later on in the book to polish the entire product. You can find out more information about ProBuilder at http://www.protoolsforunity3d.com/probuilder/. Modular tilesets Another way to generate architecture is through the use of "tiles" that are created by an artist. Similar to using Lego pieces, we can use these tiles to snap together walls and other objects to create a building. With creative uses of the tiles, you can create a large amount of content with just a minimal amount of assets. This is probably the easiest way to create a level at the expense of not being able to create unique looking buildings, since you only have a few pieces to work with. Titles such as Skyrim use this to a great extent to create their large world environments. Mix and match Of course, it's also possible to use a mixture of the preceding tools in order to use the advantages of certain ways of doing things. For example, you could use brushes to block out an area and then use a group of tiles called a tileset to replace the boxes with the highly detailed models, which is what a lot of AAA studios do. In addition, we could initially place brushes to test our gameplay and then add in props to break up the repetitiveness of the levels, which is what we are going to be doing. Creating geometry The first thing we are going to do is to learn how we can create geometry as described in the following steps: From the top menu, go to File | New Scene. This will give us a fresh start to build our project. Next, because we already have Prototype installed, let's create a cube by hitting Ctrl + K. Right now, our Cube (with a name of pb-Cube-1562 or something similar) is placed on a Position of 2, -7, -2. However, for simplicity's sake, I'm going to place it in the middle of the world. We can do this by typing in 0,0,0 by left-clicking in the X position field, typing 0, and then pressing Tab. Notice the cursor is now automatically at the Y part. Type in 0, press Tab again, and then, from the Z slot, press 0 again. Alternatively you can right-click on the Transform component and select Reset Position. Next, we have to center the camera back onto our Cube object. We can do this by going over to the Hierarchy tab and double-clicking on the Cube object (or selecting it and then pressing F). Now, to actually modify this cube, we are going to open up Prototype. We can do this by first selecting our Cube object, going to the Pb_Object component, and then clicking on the green Open Prototype button. Alternatively, you can also go to Tools | Prototype | Prototype Window. This is going to bring up a window much like the one I have displayed here. This new Prototype tab can be detached from the main Unity window or, if you drag from the tab over into Unity, it can be "hooked" into place elsewhere, like the following screenshot shows by my dragging and dropping it to the right of the Hierarchy tab. Next, select the Scene tab in the middle of the screen and press the G key to toggle us into the Object/Geometry mode. Alternatively, you can also click on the Element button in the Scene tab. Unlike the default Object/Top level mode, this will allow us to modify the cube directly to build upon it. For more information on the different modes, check out the Modes & Elements section from http://www.protoolsforunity3d.com/docs/probuilder/#buildingAndEditingGeometry. You'll notice the top of the Prototype tab has three buttons. These stand for what selection type you are currently wanting to use. The default is Vertex or the Point mode, which will allow us to select individual parts to modify. The next is Edge and the last is Face. Face is a good standard to use at this stage, because we only want to extend things out. Select the Face mode by either clicking on the button or pressing the H key twice until it says Editing Faces on the screen. Afterwards, select the box's right side. For a list of keyword shortcuts included with Prototype/ProBuilder, check out http://www.protoolsforunity3d.com/docs/probuilder/#keyboardShortcuts. Now, pull on the red handle to extend our brush outward. Easy enough. Note that, by default, while pulling things out, it is being done in 1 increment. This is nice when we are polishing our levels and trying to make things exactly where we want them, but right now, we are just prototyping. So, getting it out as quickly as possible is paramount to test if it's enjoyable. To help with this, we can use a feature of Unity called Unit Snapping. Undo the previous change we made by pressing Ctrl+Z. Then, move the camera over to the other side and select our longer face. Drag it 9 units out by holding down the Control key (Command on Mac). ProCore3D also has another tool out called ProGrids, which has some advanced unit snapping functionality, but we are not going to be using it. For more information on it, check out http://www.protoolsforunity3d.com/progrids/ If you'd like to change the distance traveled while using unit snapping, set it using the Edit | Snap Settings… menu. Next, drag both the sides out until they are 9 x 9 wide. To make things easier to see, select the Directional Light object in our scene via the Hierarchy tab and reduce the Light component's Intensity to . 5. So, at this point, we have a nice looking floor. However, to create our room, we are first going to need to create our ceiling. Select the floor we have created and press Ctrl + D to duplicate the brush. Once completed, change back into the Object/Top Level editing mode and move the brush so that its Position is at 0, 4, 0. Alternatively, you can click on the duplicated object and, from the Inspector tab, change the Position's Y value to 4. Go back into the sub-selection mode by hitting H to go back to the Faces mode. Then, hold down Ctrl and select all of the edges of our floor. Click on the Extrude button from the Prototype panel. This creates a new part on each of the four edges, which is by default .5 wide (change by clicking on the + button on the edge). This adds additional edges and/or faces to our object. Next, we are going to extrude again; but, rather than doing it from the menu, let's do it manually by selecting the tops of our newly created edges and holding down the Shift button and dragging it up along the Y (green) axis. We then hold down Ctrl after starting the extrusion to have it snap appropriately to fit around our ceiling. Note that the box may not look like this as soon as you let go, as Prototype needs time to compute lighting and materials, which it will mention from the bottom right part of Unity. Next, select Main Camera in the Hierarchy, hit W to switch to the Translate mode, and F to center the selection. Then, move our camera into the room. You'll notice it's completely dark due to the ceiling, but we can add light to the world to fix that! Let's add a point light by going to GameObject | Light | Point Light and position it in the center of the room towards the ceiling (In my case, it was at 4.5, 2.5. 3.5). Then, up the Range to 25 so that it hits the entire room. Finally, add a player to see how he interacts. First, delete the Main Camera object from Hierarchy, as we won't need it. Then, go into the Project tab and open up the AssetsUFPSBaseContentPrefabsPlayers folder. Drag and drop the AdvancedPlayer prefab, moving it so that it doesn't collide with the walls, floors, or ceiling, a little higher than the ground as shown in the following screenshot: Next, save our level (Chapter 3_1_CreatingGeometry) and hit the Play button. It may be a good idea for you to save your levels in such a way that you are able to go back and see what was covered in each section for each chapter, thus making things easier to find in the future. Again, remember that we can pull a weapon out by pressing the 1-5 keys. With this, we now have a simple room that we can interact with! Summary In this article, we take on the role of a level designer, who has been asked to create a level prototype to prove that our gameplay is solid. We will use the free Prototype tool to help in this endeavor. In addition, we will also learn some beginning level designs. Resources for Article: Further resources on this subject: Unity Networking – The Pong Game [article] Unity 3.x Scripting-Character Controller versus Rigidbody [article] Animations in Cocos2d-x [article]
Read more
  • 0
  • 0
  • 10546

article-image-embedded-linux-and-its-elements
Packt
22 Sep 2015
6 min read
Save for later

Embedded Linux and Its Elements

Packt
22 Sep 2015
6 min read
 In this article by Chris Simmonds, author of the book, Mastering Embedded Linux Programming, we'll cover the introduction of embedded Linux and its elements. (For more resources related to this topic, see here.) Why is embedded Linux popular? Linux first became a viable choice for embedded devices around 1999. That was when Axis (www.axis.com) released their first Linux-powered network camera and Tivo (www.tivo.com) their first DVR (Digital video recorder). Since 1999, Linux has become ever more popular, to the point that today it is the operating system of choice for many classes of product. As of this writing, in 2015, there are about 2 billion devices running Linux. That includes a large number of smart phones running Android, set top boxes and smart TVs and WiFi routers. Not to mention a very diverse range of devices such as vehicle diagnostics, weighing scales, industrial devices and medical monitoring units that ship in smaller volumes. So, why does your TV run Linux? At first glance, the function of a TV is simple: it has to display a stream of video on a screen. Why is a complex Unix-based operating system like Linux necessary? The simple answer is Moore's Law: Gordon Moore, co-founder of Intel stated in 1965 that the density of components on a chip will double every 2 years. That applies to the devices that we design and use in our everyday lives just as much as it does to desktops, laptops and servers. A typical SoC (System on Chip) at the heart of current devices contains many function block and has a technical reference manual that stretches to thousands of pages. Your TV is not simply displaying a video stream as the old analog sets used to. The stream is digital, possibly encrypted, and it needs processing to create an image. Your TV is (or soon will be) connected to the Internet. It can receive content from smart phones, tablets and home media servers. It can be (or soon will) used to play games. And so on and so on. You need a full operating system to manage all that hardware. Here are some points that drive the adoption of Linux: Linux has the functionality required. It has a good scheduler, a good network stack, support for many kinds of storage media, good support for multimedia devices, and so on. It ticks all the boxes. Linux has been ported to a wide range of processor architectures, including those important for embedded use: ARM, MIPS, x86 and PowerPC. Linux is open source. So you have the freedom to get the source code and modify it to meet your needs. You, or someone in the community, can create a board support package for your particular SoC, board or device. You can add protocols, features, technologies that may be missing from the mainline source code. Or, you can remove features that you don't need in order to reduce memory and storage requirements. Linux is flexible. Linux has an active community. In the case of the Linux kernel, very active. There is a new release of the kernel every 10 to 12 weeks, and each release contains code from around 1000 developers. An active community means that Linux is up to date and supports current hardware, protocols and standards. Open source licenses guarantee that you have access to the source code. There is no vendor tie-in. There is no vendor, no license fees, no restrictive NDAs, EULAs, and so on. Open source software is free in both senses: it gives you the freedom to adapt it for our own use and there is nothing to pay. For these reasons, Linux is an ideal choice for complex devices. But there are a few caveats I should mention here. Complexity makes it harder to understand. Coupled with the fast moving development process and the decentralized structures of open source, you have to put some effort into learning how to use it and to keep on re-learning as it changes. I hope that this article will help in the process. Elements of embedded Linux Every project begins by obtaining, customizing and deploying these four elements: Toolchain, Bootloader, Kernel, and Root filesystem. Toolchain The toolchain is the first element of embedded Linux and the starting point of your project. It should be constant throughout the project, in other words, once you have chosen your toolchain it is important to stick with it. Changing compilers and development libraries in an inconsistent way during a project will lead to subtle bugs. Obtaining a toolchain can be as simple as downloading and installing a package. But, the toolchain itself is a complex thing. Linux toolchains are almost always based on components from the GNU project (http://www.gnu.org). It is becoming possible to create toolchains based on LLVM/Clang (http://llvm.org). Bootloader The bootloader is the second element of Embedded Linux. It is the part that starts the system up and loads the operating system kernel. When considering which bootloader to focus on, there is one that stands out: U-Boot. In an embedded Linux system the bootloader has two main jobs: to start the system running and to load a kernel. In fact the first job is in somewhat subsidiary to the second in that it is only necessary to get as much of the system working as is necessary to load the kernel. Kernel The kernel is the third element of Embedded Linux. It is the component that is responsible for managing resources and interfacing with hardware, and so affects almost every aspect of your final software build. Usually it is tailored to your particular hardware configuration. The kernel has three main jobs to do: to manage resources, to interface to hardware, and to provide an API that offers a useful level of abstraction to user space programs, as summarized in the following diagram: Root filesystem The root filesystem is the fourth and final element of embedded Linux. The first objective is to create a minimal root filesystem that can give us a shell prompt. Then using that as a base we will add scripts to start other programs up, and to configure a network interface and user permissions. Knowing how to build the root filesystem from scratch is a useful skill. Summary In this article we briefly saw the introduction for embedded Linux and its elements. Resources for Article: Further resources on this subject: Virtualization[article] An Introduction to WEP [article] Raspberry Pi LED Blueprints [article]
Read more
  • 0
  • 0
  • 3613

article-image-how-to-simplify-your-development-workflow-with-gulp
Brian Hough
21 Sep 2015
10 min read
Save for later

How to Simplify Your Development Workflow with Gulp

Brian Hough
21 Sep 2015
10 min read
The use of task runners is a fairly recent addition to the Front-End developers toolbox. If you are even using a solution like Gulp, you are already ahead of the game. CSS compiling, JavaScript linting, Image optimization, are powerful tools. However, once you start leveraging a task runner to enhance your workflow, your Gulp file can quickly get out of control. It is very common to end up with a Gulp file that looks something like this: var gulp = require('gulp'); var compass = require('gulp-compass'); var autoprefixer = require('gulp-autoprefixer'); var uglify = require('gulp-uglify'); var imagemin = require('gulp-imagemin'); var plumber = require('gulp-plumber'); var notify = require('gulp-notify'); var watch = require('gulp-watch'); // JS Minification gulp.task('js-uglify', function() { returngulp.src('./src/js/**/*.js') .pipe(plumber({ errorHandler: notify.onError("ERROR: JS Compilation Failed") })) .pipe(uglify()) .pipe(gulp.dest('./dist/js')) }); }); // SASS Compliation gulp.task('sass-compile', function() { returngulp.src('./src/scss/main.scss') .pipe(plumber({ errorHandler: notify.onError("ERROR: CSS Compilation Failed") })) .pipe(compass({ style: 'compressed', css: './dist/css', sass: './src/scss', image: './src/img' })) .pipe(autoprefixer('> 1%', 'last 2 versions', 'Firefox ESR', 'Opera 12.1')) .pipe(gulp.dest('./dist/css')) }); }); // Image Optimization gulp.task('image-minification', function(){ returngulp.src('./src/img/**/*') .pipe(plumber({ errorHandler: notify.onError("ERROR: Image Minification Failed") })) .pipe(imagemin({ optimizationLevel: 3, progressive: true, interlaced: true })) .pipe(gulp.dest('./dist/img')); }); // Watch Task gulp.task('watch', function () { // Builds JavaScript watch('./src/js/**/*.js', function () { gulp.start('js-uglify'); }); // Builds CSS watch('./src/scss/**/*.scss', function () { gulp.start('css-compile'); }); // Optimizes Images watch(['./src/img/**/*.jpg', './src/img/**/*.png', './src/img/**/*.svg'], function () { gulp.start('image-minification'); }); }); // Default Task Triggers Watch gulp.task('default', function() { gulp.start('watch'); }); While this works, it is not very maintainable, especially as you add more and more tasks. The goal of our workflow tools are to be as easy and unobtrusive as possible. Let's look at some ways we can make our tasks easier to maintain as our workflow needs scale. Gulp Load Plugins Like most node-based projects, there are a lot of dependencies to maintain when using Gulp. Every new task often requires several new plugins to get up and running, making the giant list at the top of gulp file a maintenance nightmare. Luckily, there is an easy way to address thanks to gulp-load-plugins. gulp-load-plugins loads any Gulp plugins from your package.json automatically without you needing to manually require them. Each plugin can then be used as before without having to add each new plugin to your list at the top. To get started let's first add gulp-load-plugins to our package.json file. npm install --save-dev gulp-load-plugins Once we've done this, we can remove that giant list of dependencies from the top of our gulpfile.js. Instead we replace it with just two dependencies: var gulp = require('gulp'); var plugins = require('gulp-load-plugins')(); We now have a single object plugins that will contain all the plugins our project depends on. We just need to update our code to reflect that our plugins are part of this new object: var gulp = require('gulp'); var plugins = require('gulp-load-plugins')(); // JS Minification gulp.task('js-uglify', function() { returngulp.src('./src/js/**/*.js') .pipe(plugins.plumber({ errorHandler: plugins.notify.onError("ERROR: JS Compilation Failed") })) .pipe(plugins.uglify()) .pipe(gulp.dest('./dist/js')) }); }); // SASS Compliation gulp.task('sass-compile', function() { returngulp.src('./src/scss/main.scss') .pipe(plugins.plumber({ errorHandler: plugins.notify.onError("ERROR: CSS Compilation Failed") })) .pipe(plugins.compass({ style: 'compressed', css: './dist/css', sass: './src/scss', image: './src/img' })) .pipe(plugins.autoprefixer('> 1%', 'last 2 versions', 'Firefox ESR', 'Opera 12.1')) .pipe(gulp.dest('./dist/css')) }); }); // Image Optimization gulp.task('image-minification', function(){ returngulp.src('./src/img/**/*') .pipe(plugins.plumber({ errorHandler: plugins.notify.onError("ERROR: Image Minification Failed") })) .pipe(plugins.imagemin({ optimizationLevel: 3, progressive: true, interlaced: true })) .pipe(gulp.dest('./dist/img')); }); // Watch Task gulp.task('watch', function () { // Builds JavaScript plugins.watch('./src/js/**/*.js', function () { gulp.start('js-uglify'); }); // Builds CSS plugins.watch('./src/scss/**/*.scss', function () { gulp.start('css-compile'); }); // Optimizes Images plugins.watch(['./src/img/**/*.jpg', './src/img/**/*.png', './src/img/**/*.svg'], function () { gulp.start('image-minification'); }); }); // Default Task Triggers Watch gulp.task('default', function() { gulp.start('watch'); }); Now, each time we add a new plugin, this object will be automatically updated with it, making plugin maintenance a breeze. Centralized Configuration Going over our gulpfile.js you probably notice we repeat a lot of references, specifically items like source and destination folders, as well as plugin configuration objects. As our task list grows, and changes to these can be troublesome to maintain. Moving these items to a centralized configuration object, can be a life saver if you ever need to update one of these values. To get started let's create a new file called config.json: { "scssSrcPath":"./src/scss", "jsSrcPath":"./src/js", "imgSrcPath":"./src/img", "cssDistPath":"./dist/css", "jsDistPath":"./dist/js", "imgDistPath":"./dist/img", "browserList":"> 1%', 'last 2 versions', 'Firefox ESR', 'Opera 12.1" } What we have here is a basic JSON file that contains the most common, repeating configuration values. We have a source and destination path for Sass, JavaScript, and Image files, as well as a list of support browsers for Autoprefixer. Now let's add this configuration file to our gulpfile.js: var gulp = require('gulp'); var config = require('./config.json'); var plugins = require('gulp-load-plugins')(); // JS Minification gulp.task('js-uglify', function() { returngulp.src(config.jsSrcPath + '/**/*.js') .pipe(plugins.plumber({ errorHandler: plugins.notify.onError("ERROR: JS Compilation Failed") })) .pipe(plugins.uglify()) .pipe(gulp.dest(config.jsDistPath)) }); }); // SASS Compliation gulp.task('sass-compile', function() { returngulp.src(config.scssSrcPath + '/main.scss') .pipe(plugins.plumber({ errorHandler: plugins.notify.onError("ERROR: CSS Compilation Failed") })) .pipe(plugins.compass({ style: 'compressed', css: config.cssDistPath, sass: config.scssSrcPath, image: config.imgSrcPath })) .pipe(plugins.autoprefixer(config.browserList)) .pipe(gulp.dest(config.cssDistPath)) }); }); // Image Optimization gulp.task('image-minification', function(){ returngulp.src(config.imgSrcPath'/**/*') .pipe(plugins.plumber({ errorHandler: plugins.notify.onError("ERROR: Image Minification Failed") })) .pipe(plugins.imagemin({ optimizationLevel: 3, progressive: true, interlaced: true })) .pipe(gulp.dest(config.jsDistPath)); }); // Watch Task gulp.task('watch', function () { // Builds JavaScript plugins.watch(config.jsSrcPath + '/**/*.js', function () { gulp.start('js-uglify'); }); // Builds CSS plugins.watch(config.scssSrcPath + '/**/*.scss', function () { gulp.start('css-compile'); }); // Optimizes Images plugins.watch([config.imgSrcPath + '/**/*.jpg', config.imgSrcPath + '/**/*.png', config.imgSrcPath + '/**/*.svg'], function () { gulp.start('image-minification'); }); }); // Default Task Triggers Watch gulp.task('default', function() { gulp.start('watch'); }); First, we required our config file so that all our tasks have access to the object. Then we update each task using our configuration values including all our file paths and our browser support list. Now anytime these values are updated, we only have to do it one place. This approach is going to come in especially handy with our next step, which is modularizing our tasks. Modular Tasks You've probably noticed that we have leveraged node's module loading capabilities to achieve our results so far. However, we can take this one step further, by modularizing our tasks themselves. Placing each task in its own file allows us to give our workflow code structure and making it easier to maintain. The same benefits we gain from having modularized code in our projects can be extended to our workflow as well. Our first step is to pull our tasks into individual files. Create a folder named tasks and create the following four files: tasks/js-uglify.js: module.exports = function(gulp, plugins, config) { gulp.task('js-uglify', function() { returngulp.src(config.jsSrcPath + '/**/*.js') .pipe(plugins.plumber({ errorHandler: plugins.notify.onError("ERROR: JS Compilation Failed") })) .pipe(plugins.uglify()) .pipe(gulp.dest(config.jsDistPath)) }); }); }; tasks/sass-compile.js: module.exports = function(gulp, plugins, config) { gulp.task('sass-compile', function() { returngulp.src(config.scssSrcPath + '/main.scss') .pipe(plugins.plumber({ errorHandler: plugins.notify.onError("ERROR: CSS Compilation Failed") })) .pipe(plugins.compass({ style: 'compressed', css: config.cssDistPath, sass: config.scssSrcPath, image: config.imgSrcPath })) .pipe(plugins.autoprefixer(config.browserList)) .pipe(gulp.dest(config.cssDistPath)) }); }); }; tasks/image-minification.js: module.exports = function(gulp, plugins, config) { gulp.task('image-minification', function(){ returngulp.src(config.imgSrcPath'/**/*') .pipe(plugins.plumber({ errorHandler: plugins.notify.onError("ERROR: Image Minification Failed") })) .pipe(plugins.imagemin({ optimizationLevel: 3, progressive: true, interlaced: true })) .pipe(gulp.dest(config.jsDistPath)); }); }; tasks/watch.js: module.exports = function(gulp, plugins, config) { gulp.task('watch', function () { // Builds JavaScript plugins.watch(config.jsSrcPath + '/**/*.js', function () { gulp.start('js-uglify'); }); // Builds CSS plugins.watch(config.scssSrcPath + '/**/*.scss', function () { gulp.start('css-compile'); }); // Optimizes Images plugins.watch([config.imgSrcPath + '/**/*.jpg', config.imgSrcPath + '/**/*.png', config.imgSrcPath + '/**/*.svg'], function () { gulp.start('image-minification'); }); }); }; Here we are wrapping each individual task as a module and preparing to pass it three parameters. gulp will, of course, contain the Gulp code base, plugins will pass our task the full plugins object, and config will contain all our configuration values. Beyond this, our tasks remain unchanged. Next, we need to pull our tasks back into our gulpfile.js. Let's start by adding a line at the end of our config.json. "tasksPath":"./tasks" This will help us to keep our code a bit cleaner, and if we ever move our tasks we can simply update this reference. Now we just need our individual tasks: var gulp = require('gulp'); var config = require('./config.json'); var plugins = require('gulp-load-plugins')(); // JS Minification require(config.tasksPath + '/js-uglify')(gulp, plugins, config); // SASS Compliation require(config.tasksPath + '/sass-compile')(gulp, plugins, config); // Image Optimization require(config.tasksPath + '/image-minification')(gulp, plugins, config); // Watch Task require(config.tasksPath + '/watch')(gulp, plugins, config); // Default Task Triggers Watch gulp.task('default', function() { gulp.start('watch'); }); We have now required our four individual tasks from our gulpfile.js passing each the previously discussed parameters (gulp, plugins, config). Nothing changes about how we use these tasks, they simply now are self-contained within our code base. You will notice that our watch task is even able to access other tasks required in the same way. Conclusion As our front-end toolbox gets larger and larger, how we maintain that side of our code is increasingly important. It is possible to apply the same best practices we use on our project code to our workflow code as well. This further helps our tools get out of the way and lets us focus on coding. JavaScript developers of the world, unite! For more JavaScript tutorials and extra content, visit our dedicated page here. About The Author Brian Hough is a Front-End Architect, Designer, and Product Manager at Piqora. By day, he is working to prove that the days of bad Enterprise User Experiences are a thing of the past. By night, he obsesses about ways to bring designers and developers together using technology. He blogs about his early stage startup experience at lostinpixelation.com, or you can read his general musings on twitter @b_hough.
Read more
  • 0
  • 0
  • 6493
article-image-modeling-complex-functions-artificial-neural-networks
Packt
21 Sep 2015
13 min read
Save for later

Modeling complex functions with artificial neural networks

Packt
21 Sep 2015
13 min read
 In this article by Sebastian Raschka, the author of Python Machine Learning, we will take a look at the concept of multilayer artificial neural networks, which was inspired by hypotheses and models of how the human brain works to solve complex problem tasks. (For more resources related to this topic, see here.) Although artificial neural networks gained a lot of popularity in the recent years, early studies of neural networks goes back to the 1940s, when Warren McCulloch and Walter Pitt first described the concept of how neurons may work. However, the decades that followed saw the first implementation of the McCulloch-Pitt neuron model, Rosenblatt's perceptron in the 1950s. Many researchers and machine learning practitioners slowly began to lose interest in neural networks, since no one had a good solution for the training of a neural network with multiple layers. Eventually, interest in neural networks was rekindled in 1986 when D.E. Rumelhart, G.E. Hinton, and R.J. Williams were involved in the discovery and popularization of the backpropagation algorithm to train neural networks more efficiently (Rumelhart, David E.; Hinton, Geoffrey E.; Williams, Ronald J. (1986). Learning representations by back-propagating errors. Nature 323 (6088): 533–536). During the last decade, many more major breakthroughs have been made, known as deep learning algorithms. These can be used to create so-called feature detectors from unlabeled data to pre-train deep neural networks—neural networks that are composed of many layers. Neural networks are a hot topic not only in academic research but also in big technology companies such as Facebook, Microsoft, and Google. They invest heavily in artificial neural networks and deep learning research. Today, complex neural networks powered by deep learning algorithms are considered state of the art when it comes to solving complex problems, such as image and voice recognition. Introducing the multilayer neural network architecture In this section, we will connect multiple single neurons to a multilayer feed-forward neural network; this type of network is also called multilayer perceptron (MLP). The following figure illustrates the concept of an MLP consisting of three layers: one input layer, one hidden layer, and one output layer. The units in the hidden layer are fully connected to the input layer, and the output layer is fully connected to the hidden layer, respectively. As shown in the preceding diagram, we denote the ith activation unit in the jth layer as , and the activation units  and  are the bias units, which we set equal to 1. The activation of the units in the input layer is just its input plus the bias unit: Each unit in layer j is connected to all units in layer j + 1 via a weight coefficient; for example, the connection between unit a in layer j and unit b in layer j + 1 would be written as  . Note that the superscript i in  stands for the ith sample, not the ith layer; in the following paragraphs, we will often omit the superscript i for clarity. Activating a neural network via forward propagation In this section, we will describe the process of forward propagation to calculate the output of an MLP model. To understand how it fits into the context of learning an MLP model, let's summarize the MLP learning procedure in three simple steps: Starting at the input layer, we forward propagate the patterns of the training data through the network to generate an output. Based on the network's output, we calculate the error we want to minimize using a cost function, which we will describe later. We then backpropagate the error, find its derivative with respect to each weight in the network, and update the model. Finally, after we have repeated steps 1-3 for many epochs and learned the weights of the MLP, we use forward propagation to calculate the network output, and apply a threshold function to obtain the predicted class labels in the one-hot representation, which we described in the previous section. Now, let's walk through the individual steps of forward propagation to generate an output from the patterns in the training data. Since each unit in the hidden unit is connected to all units in the input layers, we first calculate the activation  as follows: Here, is the net input and  is the activation function, which has to be differentiable so as to learn the weights that connect the neurons using a gradient-based approach. To be able to solve complex problems such as image classification, we need non-linear activation functions in our MLP model, for example, the sigmoid (logistic) activation function: The sigmoid function is an "S"-shaped curve that maps the net input z onto a logistic distribution in the range 0 to 1, which passes the origin at z = 0.5 as shown in the following graph: Intuitively, we can think of the neurons in the MLP as logistic regression units that return values in the continuous range between 0 and 1. For purposes of code efficiency and readability, we will now write the activation in a more compact form using the concepts of basic linear algebra, which will allow us to vectorize our code implantation via NumPy rather than writing multiple nested and expensive Python for-loops: Here,  is our [m +1] x 1 dimensional feature vector for a sample  plus bias unit, and  is [m + 1] x h dimensional weight matrix where h is the number of hidden units in our neural network. After matrix-vector multiplication, we obtain the [m + 1] x 1 dimensional net input vector  . Furthermore, we can generalize this computation to all n samples in the training set: is now an n x [m + 1] matrix, and the matrix-matrix multiplication will result in an h x n dimensional net input matrix  . Finally, we apply the activation function g to each value in the net input matrix to get the h x n activation matrix  for the next layer (here, the output layer): Similarly, we can rewrite the activation of the output layer in the vectorized form: Here, we multiply the t x n matrix  (t is the number of output class labels) by the h x n dimensional matrix  to obtain the t x n dimensional matrix  (the columns in this matrix represent the outputs for each sample). Lastly, we apply the sigmoid activation function to obtain the continuous-valued output of our network: Classifying handwritten digits In this section, we will train our first multilayer neural network to classify handwritten digits from the popular MNIST dataset (Mixed National Institute of Standards and Technology database), which has been constructed by Yann LeCun and others (Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, November 1998) and serves as a popular benchmark dataset for machine learning algorithms. Obtaining the MNIST dataset The MNIST dataset is publicly available at http://yann.lecun.com/exdb/mnist/ and consists of these four parts: Training set images: train-images-idx3-ubyte.gz (9.9 MB, 47 MB unzipped, 60,000 samples) Training set labels: train-labels-idx1-ubyte.gz (29 KB, 60 KB unzipped, 60,000 labels) Test set images: t10k-images-idx3-ubyte.gz (1.6 MB, 7.8 MB, 10,000 samples) Test set labels: t10k-labels-idx1-ubyte.gz (5 KB, 10 KB unzipped, 10,000 labels) In this section, we will only be working with a subset of MNIST. Thus, we only need to download the training set images and training set labels. After downloading the files, I recommend that you unzip the files using the Unix/Linux GZip tool from the terminal for efficiency, for example, using the following command in your local MNIST download directory or, alternatively, your favorite unarchiver tool if you are working with a Microsoft Windows machine: gzip *ubyte.gz -d The images are stored in byte form, and using the following function, we will read them into NumPy arrays, which we will use to train our MLP: >>> import os >>> import struct >>> import numpy as np >>> def load_mnist(path): ... labels_path = os.path.join(path, 'train-labels-idx1-ubyte') ... images_path = os.path.join(path, 'train-images-idx3-ubyte') ... with open(labels_path, 'rb') as lbpath: ... magic, n = struct.unpack('>II', lbpath.read(8)) ... labels = np.fromfile(lbpath, dtype=np.uint8) ... with open(images_path, 'rb') as imgpath: ... magic, num, rows, cols = struct.unpack( ... ">IIII", imgpath.read(16)) ... images = np.fromfile(imgpath, ... dtype=np.uint8).reshape(len(labels), 784) ... return images, labels The load_mnist function returns an n x m dimensional NumPy array (images), where n is the number of samples (60,000), and m is the number of features. The images in the MNIST dataset consist of 28 x 28 pixels, and each pixel is represented by a grayscale intensity value. Here, we unroll the 28 x 28 pixels into 1D row vectors, which represent the rows in our images array (784 per row or image). The load_mnist function returns a second array, labels, which contains the 60,000 class labels (integers 0-9) of the handwritten digits. The way we read in the image might seem a little strange at first: magic, n = struct.unpack('>II', lbpath.read(8)) labels = np.fromfile(lbpath, dtype=np.int8) To understand how these two lines of code work, let's take a look at the dataset description from the MNIST website: [offset] [type] [value] [description] 0000 32 bit integer 0x00000801(2049) magic number (MSB first) 0004 32 bit integer 60000 number of items 0008 unsigned byte ?? label 0009 unsigned byte ?? label ........ xxxx unsigned byte ?? label Using the two lines of the preceding code, we first read in the "magic number," which is a description of the file protocol as well as the "number of items" (n) from the file buffer, before we read the following bytes into a NumPy array using the fromfile method. The fmt parameter value >II that we passed as an argument to struct.unpack can be composed of two parts: >: Big-endian (defines the order in which a sequence of bytes is stored) I: Unsigned integer After executing the following code, we should have a label vector of 60,000 instances, that is, a 60,000 × 784 dimensional image matrix: >>> X, y = load_mnist('mnist') >>> print('Rows: %d, columns: %d' % (X.shape[0], X.shape[1])) Rows: 60000, columns: 784 To get a idea of what those images in MNIST look like, let's define a function that reshapes a 784-pixel sample from our feature matrix into the original 28 × 28 image that we can plot via matplotlib's imshow function: >>> import matplotlib.pyplot as plt >>> def plot_digit(X, y, idx): ... img = X[idx].reshape(28,28) ... plt.imshow(img, cmap='Greys', interpolation='nearest') ... plt.title('true label: %d' % y[idx]) ... plt.show() Now let's use the plot_digit function to display an arbitrary digit (here, the fifth digit) from the dataset: >>> plot_digit(X, y, 4) Implementing a multilayer perceptron In this section, we will implement the code of an MLP with one input, one hidden, and one output layer to classify the images in the MNIST dataset. I tried to keep the code as simple as possible. However, it may seem a little complicated at first. If you are not running the code from the IPython notebook, I recommend that you copy it to a Python script file in your current working directory, for example, neuralnet.py, which you can then import into your current Python session via this: from neuralnet import NeuralNetMLP Now, let's initialize a new 784-50-10 MLP, a neural network with 784 input units (n_features), 50 hidden units (n_hidden), and 10 output units (n_output): >>> nn = NeuralNetMLP(n_output=10, ... n_features=X.shape[1], ... n_hidden=50, ... l2=0.1, ... l1=0.0, ... epochs=800, ... eta=0.001, ... alpha=0.001, ... decrease_const=0.00001, ... shuffle=True, ... minibatches=50, ... random_state=1) l2: The  parameter for L2 regularization. This is used to decrease the degree of overfitting; equivalently, l1 is the  for L1 regularization. epochs: The number of passes over the training set. eta: The learning rate . alpha: A parameter for momentum learning used to add a factor of the previous gradient to the weight update for faster learning: (where t is the current time step or epoch). decrease_const: The decrease constant d for an adaptive learning rate  that decreases over time for better convergence . shuffle: Shuffle the training set prior to every epoch to prevent the algorithm from getting stuck in circles. minibatches: Splitting of the training data into k mini-batches in each epoch. The gradient is computed for each mini-batch separately instead of the entire training data for faster learning. Next, we train the MLP using 10,000 samples from the already shuffled MNIST dataset. Note that we only use 10,000 samples to keep the time for training reasonable (up to 5 minutes on standard desktop computer hardware). However, you are encouraged to use more training data for model fitting to increase the predictive accuracy: >>> nn.fit(X[:10000], y[:10000], print_progress=True) Epoch: 800/800 Similar to our earlier Adaline implementation, we save the cost for each epoch in a cost_ list, which we can now visualize, making sure that the optimization algorithm has reached convergence. Here, we plot only every 50th step to account for the 50 mini-batches (50 minibatches × 800 epochs): >>> import matplotlib.pyplot as plt >>> plt.plot(range(len(nn.cost_)//50), nn.cost_[::50], color='red') >>> plt.ylim([0, 2000]) >>> plt.ylabel('Cost') >>> plt.xlabel('Epochs') >>> plt.show() As we can see, the optimization algorithm converged after approximately 700 epochs. Now let's evaluate the performance of the model by calculating the prediction accuracy: >>> y_pred = nn.predict(X[:10000]) >>> acc = np.sum(y[:10000] == y_pred, axis=0) / 10000 >>> print('Training accuracy: %.2f%%' % (acc * 100)) Training accuracy: 97.60% As you can see, the model gets most of the training data right. But how does it generalize to data that it hasn't seen before during training? Let's calculate the test accuracy on 5,000 images that were not included in the training set: >>> y_pred = nn.predict(X[10000:15000]) >>> acc = np.sum(y[10000:15000] == y_pred, axis=0) / 5000 >>> print('Test accuracy: %.2f%%' % (acc * 100)) Test accuracy: 92.40% Summary Based on the discrepancy between the training and test accuracy, we can conclude that the model slightly overfits the training data. To decrease the degree of overfitting, we can change the number of hidden units or the values of the regularization parameters, or fit the model on more training data. Resources for Article: Further resources on this subject: Asynchronous Programming with Python[article] The Essentials of Working with Python Collections[article] Python functions – Avoid repeating code [article]
Read more
  • 0
  • 0
  • 9813

article-image-scraping-data
Packt
21 Sep 2015
18 min read
Save for later

Scraping the Data

Packt
21 Sep 2015
18 min read
In this article by Richard Lawson, author of the book Web Scraping with Python, we will first cover a browser extension called Firebug Lite to examine a web page, which you may already be familiar with if you have a web development background. Then, we will walk through three approaches to extract data from a web page using regular expressions, Beautiful Soup and lxml. Finally, the article will conclude with a comparison of these three scraping alternatives. (For more resources related to this topic, see here.) Analyzing a web page To understand how a web page is structured, we can try examining the source code. In most web browsers, the source code of a web page can be viewed by right-clicking on the page and selecting the View page source option: The data we are interested in is found in this part of the HTML: <table> <tr id="places_national_flag__row"><td class="w2p_fl"><label for="places_national_flag" id="places_national_flag__label">National Flag: </label></td><td class="w2p_fw"><img src="/places/static/images/flags/gb.png" /></td><td class="w2p_fc"></td></tr> … <tr id="places_neighbours__row"><td class="w2p_fl"><label for="places_neighbours" id="places_neighbours__label">Neighbours: </label></td><td class="w2p_fw"><div><a href="/iso/IE">IE </a></div></td><td class="w2p_fc"></td></tr></table> This lack of whitespace and formatting is not an issue for a web browser to interpret, but it is difficult for us. To help us interpret this table, we will use the Firebug Lite extension, which is available for all web browsers at https://getfirebug.com/firebuglite. Firefox users can install the full Firebug extension if preferred, but the features we will use here are included in the Lite version. Now, with Firebug Lite installed, we can right-click on the part of the web page we are interested in scraping and select Inspect with Firebug Lite from the context menu, as shown here: This will open a panel showing the surrounding HTML hierarchy of the selected element: In the preceding screenshot, the country attribute was clicked on and the Firebug panel makes it clear that the country area figure is included within a <td> element of class w2p_fw, which is the child of a <tr> element of ID places_area__row. We now have all the information needed to scrape the area data. Three approaches to scrape a web page Now that we understand the structure of this web page we will investigate three different approaches to scraping its data, firstly with regular expressions, then with the popular BeautifulSoup module, and finally with the powerful lxml module. Regular expressions If you are unfamiliar with regular expressions or need a reminder, there is a thorough overview available at https://docs.python.org/2/howto/regex.html. To scrape the area using regular expressions, we will first try matching the contents of the <td> element, as follows: >>> import re >>> url = 'http://example.webscraping.com/view/United Kingdom-239' >>> html = download(url) >>> re.findall('<td class="w2p_fw">(.*?)</td>', html) ['<img src="/places/static/images/flags/gb.png" />', '244,820 square kilometres', '62,348,447', 'GB', 'United Kingdom', 'London', '<a href="/continent/EU">EU</a>', '.uk', 'GBP', 'Pound', '44', '@# #@@|@## #@@|@@# #@@|@@## #@@|@#@ #@@|@@#@ #@@|GIR0AA', '^(([A-Z]\d{2}[A-Z]{2})|([A-Z]\d{3}[A-Z]{2})|([A-Z]{2}\d{2} [A-Z]{2})|([A-Z]{2}\d{3}[A-Z]{2})|([A-Z]\d[A-Z]\d[A-Z]{2}) |([A-Z]{2}\d[A-Z]\d[A-Z]{2})|(GIR0AA))$', 'en-GB,cy-GB,gd', '<div><a href="/iso/IE">IE </a></div>'] This result shows that the <td class="w2p_fw"> tag is used for multiple country attributes. To isolate the area, we can select the second element, as follows: >>> re.findall('<td class="w2p_fw">(.*?)</td>', html)[1] '244,820 square kilometres' This solution works but could easily fail if the web page is updated. Consider if the website is updated and the population data is no longer available in the second table row. If we just need to scrape the data now, future changes can be ignored. However, if we want to rescrape this data in future, we want our solution to be as robust against layout changes as possible. To make this regular expression more robust, we can include the parent <tr> element, which has an ID, so it ought to be unique: >>> re.findall('<tr id="places_area__row"><td class="w2p_fl"><label for="places_area" id="places_area__label">Area: </label></td><td class="w2p_fw">(.*?)</td>', html) ['244,820 square kilometres'] This iteration is better; however, there are many other ways the web page could be updated in a way that still breaks the regular expression. For example, double quotation marks might be changed to single, extra space could be added between the <td> tags, or the area_label could be changed. Here is an improved version to try and support these various possiblilities: >>> re.findall('<tr id="places_area__row">.*?<tds*class=["']w2p_fw["']>(.*?) </td>', html)[0] '244,820 square kilometres' This regular expression is more future-proof but is difficult to construct, becoming unreadable. Also, there are still other minor layout changes that would break it, such as if a title attribute was added to the <td> tag. From this example, it is clear that regular expressions provide a simple way to scrape data but are too brittle and will easily break when a web page is updated. Fortunately, there are better solutions. Beautiful Soup Beautiful Soup is a popular library that parses a web page and provides a convenient interface to navigate content. If you do not already have it installed, the latest version can be installed using this command: pip install beautifulsoup4 The first step with Beautiful Soup is to parse the downloaded HTML into a soup document. Most web pages do not contain perfectly valid HTML and Beautiful Soup needs to decide what is intended. For example, consider this simple web page of a list with missing attribute quotes and closing tags:       <ul class=country> <li>Area <li>Population </ul> If the Population item is interpreted as a child of the Area item instead of the list, we could get unexpected results when scraping. Let us see how Beautiful Soup handles this: >>> from bs4 import BeautifulSoup >>> broken_html = '<ul class=country><li>Area<li>Population</ul>' >>> # parse the HTML >>> soup = BeautifulSoup(broken_html, 'html.parser') >>> fixed_html = soup.prettify() >>> print fixed_html <html> <body> <ul class="country"> <li>Area</li> <li>Population</li> </ul> </body> </html> Here, BeautifulSoup was able to correctly interpret the missing attribute quotes and closing tags, as well as add the <html> and <body> tags to form a complete HTML document. Now, we can navigate to the elements we want using the find() and find_all() methods: >>> ul = soup.find('ul', attrs={'class':'country'}) >>> ul.find('li') # returns just the first match <li>Area</li> >>> ul.find_all('li') # returns all matches [<li>Area</li>, <li>Population</li>] Beautiful Soup overview Here are the common methods and parameters you will use when scraping web pages with Beautiful Soup: BeautifulSoup(markup, builder): This method creates the soup object. The markup parameter can be a string or file object, and builder is the library that parses the markup parameter. find_all(name, attrs, text, **kwargs): This method returns a list of elements matching the given tag name, dictionary of attributes, and text. The contents of kwargs are used to match attributes. find(name, attrs, text, **kwargs): This method is the same as find_all(), except that it returns only the first match. If no element matches, it returns None. prettify(): This method returns the parsed HTML in an easy-to-read format with indentation and line breaks. For a full list of available methods and parameters, the official documentation is available at http://www.crummy.com/software/BeautifulSoup/bs4/doc/. Now, using these techniques, here is a full example to extract the area from our example country: >>> from bs4 import BeautifulSoup >>> url = 'http://example.webscraping.com/places/view/ United-Kingdom-239' >>> html = download(url) >>> soup = BeautifulSoup(html) >>> # locate the area row >>> tr = soup.find(attrs={'id':'places_area__row'}) >>> td = tr.find(attrs={'class':'w2p_fw'}) # locate the area tag >>> area = td.text # extract the text from this tag >>> print area 244,820 square kilometres This code is more verbose than regular expressions but easier to construct and understand. Also, we no longer need to worry about problems in minor layout changes, such as extra whitespace or tag attributes. Lxml Lxml is a Python wrapper on top of the libxml2 XML parsing library written in C, which makes it faster than Beautiful Soup but also harder to install on some computers. The latest installation instructions are available at http://lxml.de/installation.html. As with Beautiful Soup, the first step is parsing the potentially invalid HTML into a consistent format. Here is an example of parsing the same broken HTML: >>> import lxml.html >>> broken_html = '<ul class=country><li>Area<li>Population</ul>' >>> tree = lxml.html.fromstring(broken_html) # parse the HTML >>> fixed_html = lxml.html.tostring(tree, pretty_print=True) >>> print fixed_html <ul class="country"> <li>Area</li> <li>Population</li> </ul> As with BeautifulSoup, lxml was able to correctly parse the missing attribute quotes and closing tags, although it did not add the <html> and <body> tags. After parsing the input, lxml has a number of different options to select elements, such as XPath selectors and a find() method similar to Beautiful Soup. Instead, we will use CSS selectors here and in future examples, because they are more compact. Also, some readers will already be familiar with them from their experience with jQuery selectors. Here is an example using the lxml CSS selectors to extract the area data: >>> tree = lxml.html.fromstring(html) >>> td = tree.cssselect('tr#places_area__row > td.w2p_fw')[0] >>> area = td.text_content() >>> print area 244,820 square kilometres The key line with the CSS selector is highlighted. This line finds a table row element with the places_area__row ID, and then selects the child table data tag with the w2p_fw class. CSS selectors CSS selectors are patterns used for selecting elements. Here are some examples of common selectors you will need: Select any tag: * Select by tag <a>: a Select by class of "link": .link Select by tag <a> with class "link": a.link Select by tag <a> with ID "home": a#home Select by child <span> of tag <a>: a > span Select by descendant <span> of tag <a>: a span Select by tag <a> with attribute title of "Home": a[title=Home] The CSS3 specification was produced by the W3C and is available for viewing at http://www.w3.org/TR/2011/REC-css3-selectors-20110929/. Lxml implements most of CSS3, and details on unsupported features are available at https://pythonhosted.org/cssselect/#supported-selectors. Note that, internally, lxml converts the CSS selectors into an equivalent XPath. Comparing performance To help evaluate the trade-offs of the three scraping approaches described in this article, it would help to compare their relative efficiency. Typically, a scraper would extract multiple fields from a web page. So, for a more realistic comparison, we will implement extended versions of each scraper that extract all the available data from a country's web page. To get started, we need to return to Firebug to check the format of the other country features, as shown here: Firebug shows that each table row has an ID starting with places_ and ending with __row. Then, the country data is contained within these rows in the same format as the earlier area example. Here are implementations that use this information to extract all of the available country data: FIELDS = ('area', 'population', 'iso', 'country', 'capital', 'continent', 'tld', 'currency_code', 'currency_name', 'phone', 'postal_code_format', 'postal_code_regex', 'languages', 'neighbours') import re def re_scraper(html): results = {} for field in FIELDS: results[field] = re.search('<tr id="places_%s__row">.*?<td class="w2p_fw">(.*?)</td>' % field, html).groups()[0] return results from bs4 import BeautifulSoup def bs_scraper(html): soup = BeautifulSoup(html, 'html.parser') results = {} for field in FIELDS: results[field] = soup.find('table').find('tr', id='places_%s__row' % field).find('td', class_='w2p_fw').text return results import lxml.html def lxml_scraper(html): tree = lxml.html.fromstring(html) results = {} for field in FIELDS: results[field] = tree.cssselect('table > tr#places_%s__row > td.w2p_fw' % field)[0].text_content() return results Scraping results Now that we have complete implementations for each scraper, we will test their relative performance with this snippet: import time NUM_ITERATIONS = 1000 # number of times to test each scraper html = download('http://example.webscraping.com/places/view/ United-Kingdom-239') for name, scraper in [('Regular expressions', re_scraper), ('BeautifulSoup', bs_scraper), ('Lxml', lxml_scraper)]: # record start time of scrape start = time.time() for i in range(NUM_ITERATIONS): if scraper == re_scraper: re.purge() result = scraper(html) # check scraped result is as expected assert(result['area'] == '244,820 square kilometres') # record end time of scrape and output the total end = time.time() print '%s: %.2f seconds' % (name, end – start) This example will run each scraper 1000 times, check whether the scraped results are as expected, and then print the total time taken. Note the highlighted line calling re.purge(); by default, the regular expression module will cache searches and this cache needs to be cleared to make a fair comparison with the other scraping approaches. Here are the results from this script on my computer: $ python performance.py Regular expressions: 5.50 seconds BeautifulSoup: 42.84 seconds Lxml: 7.06 seconds The results on your computer will quite likely be different because of the different hardware used. However, the relative difference between each approach should be equivalent. The results show that Beautiful Soup is over six times slower than the other two approaches when used to scrape our example web page. This result could be anticipated because lxml and the regular expression module were written in C, while BeautifulSoup is pure Python. An interesting fact is that lxml performed comparatively well with regular expressions, since lxml has the additional overhead of having to parse the input into its internal format before searching for elements. When scraping many features from a web page, this initial parsing overhead is reduced and lxml becomes even more competitive. It really is an amazing module! Overview The following table summarizes the advantages and disadvantages of each approach to scraping: Scraping approach Performance Ease of use Ease to install Regular expressions Fast Hard Easy (built-in module) Beautiful Soup Slow Easy Easy (pure Python) Lxml Fast Easy Moderately difficult If the bottleneck to your scraper is downloading web pages rather than extracting data, it would not be a problem to use a slower approach, such as Beautiful Soup. Or, if you just need to scrape a small amount of data and want to avoid additional dependencies, regular expressions might be an appropriate choice. However, in general, lxml is the best choice for scraping, because it is fast and robust, while regular expressions and Beautiful Soup are only useful in certain niches. Adding a scrape callback to the link crawler Now that we know how to scrape the country data, we can integrate this into the link crawler. To allow reusing the same crawling code to scrape multiple websites, we will add a callback parameter to handle the scraping. A callback is a function that will be called after certain events (in this case, after a web page has been downloaded). This scrape callback will take a url and html as parameters and optionally return a list of further URLs to crawl. Here is the implementation, which is simple in Python: def link_crawler(..., scrape_callback=None): … links = [] if scrape_callback: links.extend(scrape_callback(url, html) or []) … The new code for the scraping callback function are highlighted in the preceding snippet. Now, this crawler can be used to scrape multiple websites by customizing the function passed to scrape_callback. Here is a modified version of the lxml example scraper that can be used for the callback function: def scrape_callback(url, html): if re.search('/view/', url): tree = lxml.html.fromstring(html) row = [tree.cssselect('table > tr#places_%s__row > td.w2p_fw' % field)[0].text_content() for field in FIELDS] print url, row This callback function would scrape the country data and print it out. Usually, when scraping a website, we want to reuse the data, so we will extend this example to save results to a CSV spreadsheet, as follows: import csv class ScrapeCallback: def __init__(self): self.writer = csv.writer(open('countries.csv', 'w')) self.fields = ('area', 'population', 'iso', 'country', 'capital', 'continent', 'tld', 'currency_code', 'currency_name', 'phone', 'postal_code_format', 'postal_code_regex', 'languages', 'neighbours') self.writer.writerow(self.fields) def __call__(self, url, html): if re.search('/view/', url): tree = lxml.html.fromstring(html) row = [] for field in self.fields: row.append(tree.cssselect('table > tr#places_{}__row > td.w2p_fw'.format(field)) [0].text_content()) self.writer.writerow(row) To build this callback, a class was used instead of a function so that the state of the csv writer could be maintained. This csv writer is instantiated in the constructor, and then written to multiple times in the __call__ method. Note that __call__ is a special method that is invoked when an object is "called" as a function, which is how the cache_callback is used in the link crawler. This means that scrape_callback(url, html) is equivalent to calling scrape_callback.__call__(url, html). For further details on Python's special class methods, refer to https://docs.python.org/2/reference/datamodel.html#special-method-names. This code shows how to pass this callback to the link crawler: link_crawler('http://example.webscraping.com/', '/(index|view)', max_depth=-1, scrape_callback=ScrapeCallback()) Now, when the crawler is run with this callback, it will save results to a CSV file that can be viewed in an application such as Excel or LibreOffice: Success! We have completed our first working scraper. Summary In this article, we walked through a variety of ways to scrape data from a web page. Regular expressions can be useful for a one-off scrape or to avoid the overhead of parsing the entire web page, and BeautifulSoup provides a high-level interface while avoiding any difficult dependencies. However, in general, lxml will be the best choice because of its speed and extensive functionality, and we will use it in future examples. Resources for Article: Further resources on this subject: Scientific Computing APIs for Python [article] Bizarre Python [article] Optimization in Python [article]
Read more
  • 0
  • 0
  • 6792

article-image-replacing-2d-sprites-3d-models
Packt
21 Sep 2015
21 min read
Save for later

Replacing 2D Sprites with 3D Models

Packt
21 Sep 2015
21 min read
In this article by Maya Posch author of the book Mastering AndEngine Game Development, when using a game engine that limits itself to handling scenes in two dimensions, it seems obvious that you would use two-dimensional images here, better known as sprites. After all, you won't need that third dimension, right? It is when you get into more advanced games and scenes that you notice that with animations, and also with the usage of existing assets, there are many advantages of using a three-dimensional model in a two-dimensional scene. In this article we will cover these topics: Using 3D models directly with AndEngine Loading of 3D models with an AndEngine game (For more resources related to this topic, see here.) Why 3D in a 2D game makes sense The reasons we want to use 3D models in our 2D scene include the following: Recycling of assets: You can use the same models as used for a 3D engine project, as well as countless others. Broader base of talent: You'll be able to use a 3D modeler for your 2D game, as good sprite artists are so rare. Ease of animation: Good animation with sprites is hard. With 3D models, you can use various existing utilities to get smooth animations with ease. As for the final impact it has on the game's looks, it's no silver bullet but should ease the development somewhat. The quality of the used models and produced animations as well as the way they are integrated into a scene will determine the final look. 2D and 3D compared In short: 2D sprite 3D model Defined using a 2D grid of pixels Defined using vertices in a 3D grid Only a single front view Rotatable to observe any desired side Resource-efficient Resource-intensive A sprite is an image, or—if it's animated—a series of images. Within the boundaries of its resolution (for example 64, x 64 pixels), the individual pixels make up the resulting image. This is a proven low-tech method, and it has been in use since the earliest video games. Even the first 3D games, such as Wolfenstein 3D and Doom, used sprites instead of models, as the former are easy to implement and require very few resources to render. With the available memory and processing capabilities of video consoles and personal computers until the later part of the 1990s, sprites were everywhere. It wasn't until the appearance of dedicated vertex graphics processors for consumer systems from companies such as 3dfx, Nvidia, and ATI that sprites would be largely replaced by vertex (3D) models. This is not to say that 3D models were totally new by then, of course. The technology had been in commercial use since the 1970s, when it was used for movie CGI and engineering in particular. In essence, both sprites and models are a representation of the same object; it's just that one contains more information than the other. Once rendered on the screen, the resulting image contains roughly the same amount of data. The biggest difference between sprites and models is the total amount of information that they can contain. For a sprite, there is no side or back. A model, on the other hand, has information about every part of its surface. It can be rotated in front of a camera to obtain a rendering of each of those orientations. A sprite is thus equivalent to a single orientation of a model. Dealing with the third dimension The first question that is likely to come to mind when it is suggested to use 3D models in what is advertised as a 2D engine is whether or not this will make the game engine into a 3D engine. The brief answer here is "No." The longer answer is that despite the presence of these models, the engine's camera and other features are not aware of this third dimension, and so they will not be able to deal with it. It's not unlike the ray-casting engine employed by titles such as Wolfenstein 3D, which always operated in a horizontal plane and, by default, was not capable of tilting the camera to look up or down. This does imply that AndEngine can be turned into a 3D engine if all of its classes are adapted to deal with another dimension. We're not going that far here, however. All that we are interested in right now is integrating 3D model support into the existing framework. For this, we need a number of things. The most important one is to be able to load these models. The second is to render them in such a way that we can use them within the AndEngine framework. As we explored earlier, the way of integrating 3D models into a 2D scene is by realizing that a model is just a very large collection of possible sprites. What we need is a camera so that we can orient it relatively to the model, similar to how the camera in a 3D engine works. We can then display the model from the orientation. Any further manipulations, such as scaling and scene-wide transformations, are performed on the model's camera configuration. The model is only manipulated to obtain a new orientation or frame of an animation. Setting up the environment We first need to load the model from our resources into the memory. For this, we require logic that fetches the file, parses it, and produces the output, which we can use in the following step of rendering an orientation of the model. To load the model, we can either write the logic for it ourselves or use an existing library. The latter approach is generally preferred, unless you have special needs that are not yet covered by an existing library. As we have no such special needs, we will use an existing library. Our choice here is the open Asset Import Library, or assimp for short. It can import numerous 3D model files in addition to other kinds of resource files, which we'll find useful later on. Assimp is written in C++, which means that we will be using it as a native library (.a or .so). To accomplish this, we first need to obtain its source code and compile it for Android. The main Assimp site can be found at http://assimp.sf.net/, and the Git repository is at https://github.com/assimp/assimp. From the latter, we obtain the current source for Assimp and put it into a folder called assimp. We can easily obtain the Assimp source by either downloading an archive file containing the full repository or by using the Git client (from http://git-scm.com/) and cloning the repository using the following command in an empty folder (the assimp folder mentioned): git clone https://github.com/assimp/assimp.git This will create a local copy of the remote Git repository. An advantage of this method is that we can easily keep our local copy up to date with the Assimp project's version simply by pulling any changes. As Assimp uses CMake for its build system, we will also need to obtain the CMake version for Android from http://code.google.com/p/android-cmake/. Android-Cmake contains the toolchain file that we will need to set up the cross-compilation from our host system to Android/ARM. Assuming that we put Android-cmake into the android-cmake folder, we can then find this toolchain file under android-cmake/toolchain/android.toolchain.cmake. We now need to either set the following environmental variable or make sure we have properly set it: ANDROID_NDK: This points to the root folder where the Android NDK is placed At this point, we can use either the command-line-based CMake tool or the cross-platform CMake GUI. We choose the latter for sheer convenience. Unless you are quite familiar with the working of CMake, the use of the GUI tool can make the experience significantly more intuitive, not to mention faster and more automated. Any commands we use in the GUI tool will, however, easily translate to the command-line tool. The first thing we do after opening the CMake GUI utility is specify the location of the source—the assimp source folder—and the output for the CMake-generated files. For this path to the latter, we will create a new folder called buildandroid inside the Assimp source folder and specify it as the build folder. We now need to set a variable inside the CMake GUI: CMAKE_MAKE_PROGRAM: This variable specifies the path to the Make executable. For Linux/BSD, use GNU Make or similar; for Windows, use MinGW Make. Next, we will want to click on the Configure button where we can set the type of Make files generated as well as specify the location of the toolchain file. For the Make file type, you will generally want to pick Unix makefiles on Linux or similar and MinGW makefiles on Windows. Next, pick the option that allows you to specify the cross-compile toolchain file and select this file inside the Android-cmake folder as detailed earlier. After this, the CMake GUI should output Configuring done. What has happened now is that the toolchain file that we linked to has configured CMake to use the NDK's compiler, which targets ARM as well as sets other configuration options. If we want, we can change some options here, such as the following: CMAKE_BUILD_TYPE: We can specify the type of build we want here, which includes the Debug and Release strings. ASSIMP_BUILD_STATIC_LIB: This is a boolean value. Setting it to true (or checking the box in the GUI) will generate only a library file for static linking and no .so file. Whether we want to build statically or not depends on our ultimate goals and distribution details. As static linking of external libraries is quite convenient and also reduces the total file size on the platform, which is generally already strapped for space, it seems obvious to link statically. The resulting .a library for a release build should be in the order of 16 megabytes, while a debug build is about 68 megabytes. When linking the final application, only those parts of the library that we'll use will be included in our application, shrinking the total file size once more. We are now ready to click on the Generate button, which should generate a Generating done output. If you get an error along the lines of Could not uniquely determine machine name for compiler, you should look at the paths used by CMake and check whether they exist. For the NDK toolchain on Windows, for example, the path may contain the windows part, whereas the NDK only has a folder called windows-x86_64. If we look into the buildandroid folder after this, we can see that CMake has generated a makefile and additional relevant files. We only need the central Make file in the buildandroid folder, however. In a terminal window, we navigate to this folder and execute the following command: make This should start the execution of the Make files that CMake generated and result in a proper build. At the end of this compilation sequence, we should have a library file in assimp/libs/armeabi-v7a/ called libassimp.a. For our project, we need this library and the Assimp include files. We can find them under assimp/include/assimp. We copy the folder with the include files to our project's /jni folder. The .a library is placed in the /jni folder as well. As this is a relatively simple NDK project, a simple file structure is fine. For a more complex project, we would want to have a separate /jni/libs folder, or something similar. Importing a model The Assimp library provides conversion tools for reading resource files, such as those for 3D mesh models, and provides a generic format on the application's side. For a 3D mesh file, Assimp provides us with an aiScene object that contains all the meshes and related data as described by the imported file. After importing a model, we need to read the sets of data that we require for rendering. These are the types of data: Vertices (positions) Normals Texture mapping (UV) Indices Vertices might be obvious; they are the positions of points between which lines of basic geometric shapes are drawn. Usually, three vertices are used to form a triangular face, which forms the basic shape unit for a model. Normals indicate the orientation of the vertex. We have one normal per vertex. Texture mapping is provided using so-called UV coordinates. Each vertex has a UV coordinate if texture mapping information is provided with the model. Finally, indices are values provided per face, indicating which vertices should be used. This is essentially a compression technique, allowing the faces to define the vertices that they will use so that shared vertices have to be defined only once. During the drawing process, these indices are used by OpenGL to find the vertices to draw. We start off our importer code by first creating a new file called assimpImporter.cpp in the /jni folder. We require the following include: #include "assimp/Importer.hpp" // C++ importer interface #include "assimp/scene.h" // output data structure #include "assimp/postprocess.h" // post processing flags // for native asset manager #include <sys/types.h> #include <android/asset_manager.h> #include <android/asset_manager_jni.h> The Assimp include give us access to the central Importer object, which we'll use for the actual import process, and the scene object for its output. The postprocess include contains various flags and presets for post-processing information to be used with Importer, such as triangulation. The remaining includes are meant to give us access to the Android Asset Manager API. The model file is stored inside the /assets folder, which once packaged as an APK is only accessible during runtime via this API, whether in Java or in native code. Moving on, we will be using a single function in our native code to perform the importing and processing. As usual, we have to first declare a C-style interface so that when our native library gets compiled, our Java code can find the function in the library: extern "C" { JNIEXPORT jboolean JNICALL Java_com_nyanko_andengineontour_MainActivity_getModelData(JNIEnv* env, jobject obj, jobject model, jobject assetManager, jstring filename); }; The JNIEnv* parameter and the first jobject parameter are standard in an NDK/JNI function, with the former being a handy pointer to the current JVM environment, offering a variety of utility functions. Our own parameters are the following: model assetManager filename The model is a basic Java class with getters/setters for the arrays of vertex, normal, UV and index data of which we create an instance and pass a reference via the JNI. The next parameter is the Asset Manager instance that we created in the Java code. Finally, we obtain the name of the file that we are supposed to load from the assets containing our mesh. One possible gotcha in the naming of the function we're exporting is that of underscores. Within the function name, no underscores are allowed, as underscores are used to indicate to the NDK what the package name and class names are. Our getModelData function gets parsed as being in the MainActivity class of the package com.nyanko.andengineontour. If we had tried to use, for example, get_model_data as the function name, it would have tried to find function data in the model class of the com.nyanko.andengineontour.get package. Next, we can begin the actual importing process. First, we define the aiScene instance, that will contain the imported scene, and the arrays for the imported data, as well as the Assimp Importer instance: const aiScene* scene = 0; jfloat* vertexArray; jfloat* normalArray; jfloat* uvArray; jshort* indexArray; Assimp::Importer importer; In order to use a Java string in native code, we have to use the provided method to obtain a reference via the env parameter: const char* utf8 = env->GetStringUTFChars(filename, 0); if (!utf8) { return JNI_FALSE; } We then create a reference to the Asset Manager instance that we created in Java: AAssetManager* mgr = AAssetManager_fromJava(env, assetManager); if (!mgr) { return JNI_FALSE; } We use this to obtain a reference to the asset we're looking for, being the model file: AAsset* asset = AAssetManager_open(mgr, utf8, AASSET_MODE_UNKNOWN); if (!asset) { return JNI_FALSE; } Finally, we release our reference to the filename string before moving on to the next stage: env->ReleaseStringUTFChars(filename, utf8); With access to the asset, we can now read it from the memory. While it is, in theory, possible to directly read a file from the assets, you will have to write a new I/O manager to allow Assimp to do this. This is because asset files, unfortunately, cannot be passed as a standard file handle reference on Android. For smaller models, however, we can read the entire file from the memory and pass this data to the Assimp importer. First, we get the size of the asset, create an array to store its contents, and read the file in it: int count = (int) AAsset_getLength(asset); char buf[count + 1]; if (AAsset_read(asset, buf, count) != count) { return JNI_FALSE; } Finally, we close the asset reference: AAsset_close(asset); We are now done with the asset manager and can move on to the importing of this model data: const aiScene* scene = importer.ReadFileFromMemory(buf, count, aiProcessPreset_TargetRealtime_Fast); if (!scene) { return JNI_FALSE; } The importer has a number of possible ways to read in the file data, as mentioned earlier. Here, we read from a memory buffer (buf) that we filled in earlier with the count parameter, indicating the size in bytes. The last parameter of the import function is the post-processing parameters. Here, we use the aiProcessPreset_TargetRealtime_Fast preset, which performs triangulation (converting non-triangle faces to triangles), and other sensible presets. The resulting aiScene object can contain multiple meshes. In a complete importer, you'd want to import all of them into a loop. We'll just look at importing the first mesh into the scene here. First, we get the mesh: aiMesh* mesh = scene->mMeshes[0]; This aiMesh object contains all of the information on the data we're interested in. First, however, we need to create our arrays: int vertexArraySize = mesh->mNumVertices * 3; int normalArraySize = mesh->mNumVertices * 3; int uvArraySize = mesh->mNumVertices * 2; int indexArraySize = mesh->mNumFaces * 3; vertexArray = new float[vertexArraySize]; normalArray = new float[normalArraySize]; uvArray = new float[uvArraySize]; indexArray = new jshort[indexArraySize]; For the vertex, normal, and texture mapping (UV) arrays, we use the number of vertices as defined in the aiMesh object as normal, and the UVs are defined per vertex. The former two have three components (x, y, z) and the UVs have two (x, y). Finally, indices are defined per vertex of the face, so we use the face count from the mesh multiplied by the number of vertices. All things but indices use floats for their components. The jshort type is a short integer type defined by the NDK. It's generally a good idea to use the NDK types for values that are sent to and from the Java side. Reading the data from the aiMesh object to the arrays is fairly straightforward: for (unsigned int i = 0; i < mesh->mNumVertices; i++) { aiVector3D pos = mesh->mVertices[i]; vertexArray[3 * i + 0] = pos.x; vertexArray[3 * i + 1] = pos.y; vertexArray[3 * i + 2] = pos.z; aiVector3D normal = mesh->mNormals[i]; normalArray[3 * i + 0] = normal.x; normalArray[3 * i + 1] = normal.y; normalArray[3 * i + 2] = normal.z; aiVector3D uv = mesh->mTextureCoords[0][i]; uvArray[2 * i * 0] = uv.x; uvArray[2 * i * 1] = uv.y; } for (unsigned int i = 0; i < mesh->mNumFaces; i++) { const aiFace& face = mesh->mFaces[i]; indexArray[3 * i * 0] = face.mIndices[0]; indexArray[3 * i * 1] = face.mIndices[1]; indexArray[3 * i * 2] = face.mIndices[2]; } To access the correct part of the array to write to, we use an index that uses the number of elements (floats or shorts) times the current iteration plus an offset to ensure that we reach the next available index. Doing things this way instead of pointing incrementation has the benefit that we do not have to reset the array pointer after we're done writing. There! We have now read in all of the data that we want from the model. Next is arguably the hardest part of using the NDK—passing data via the JNI. This involves quite a lot of reference magic and type-matching, which can be rather annoying and lead to confusing errors. To make things as easy as possible, we used the generic Java class instance so that we already had an object to put our data into from the native side. We still have to find the methods in this class instance, however, using what is essentially a Java reflection: jclass cls = env->GetObjectClass(model); if (!cls) { return JNI_FALSE; } The first goal is to get a jclass reference. For this, we use the jobject model variable, as it already contains our instantiated class instance: jmethodID setVA = env->GetMethodID(cls, "setVertexArray", "([F)V"); jmethodID setNA = env->GetMethodID(cls, "setNormalArray", "([F)V"); jmethodID setUA = env->GetMethodID(cls, "setUvArray", "([F)V"); jmethodID setIA = env->GetMethodID(cls, "setIndexArray", "([S)V"); We then obtain the method references for the setters in the class as jmethodID variables. The parameters in this class are the class reference we created, the name of the method, and its signature, being a float array ([F) parameter and a void (V) return type. Finally, we create our native Java arrays to pass back via the JNI: jfloatArray jvertexArray = env->NewFloatArray(vertexArraySize); env->SetFloatArrayRegion(jvertexArray, 0, vertexArraySize, vertexArray); jfloatArray jnormalArray = env->NewFloatArray(normalArraySize); env->SetFloatArrayRegion(jnormalArray, 0, normalArraySize, normalArray); jfloatArray juvArray = env->NewFloatArray(uvArraySize); env->SetFloatArrayRegion(juvArray, 0, uvArraySize, uvArray); jshortArray jindexArray = env->NewShortArray(indexArraySize); env->SetShortArrayRegion(jindexArray, 0, indexArraySize, indexArray); This code uses the env JNIEnv* reference to create the Java array and allocate memory for it in the JVM. Finally, we call the setter functions in the class to set our data. These essentially calls the methods on the Java class inside the JVM, providing the parameter data as Java types: env->CallVoidMethod(model, setVA, jvertexArray); env->CallVoidMethod(model, setNA, jnormalArray); env->CallVoidMethod(model, setUA, juvArray); env->CallVoidMethod(model, setIA, jindexArray); We only have to return JNI_TRUE now, and we're done. Building our library To build our code, we write the Android.mk and Application.mk files. Next, we go to the top level of our project in a terminal window and execute the ndk-build command. This will compile the code and place a library in the /libs folder of our project, inside a folder that indicates the CPU architecture it was compiled for. For further details on the ndk-build tool, you can refer to the official documentation at https://developer.android.com/ndk/guides/ndk-build.html. Our Android.mk file looks as follows: LOCAL_PATH := $(call my-dir) include $(CLEAR_VARS) LOCAL_MODULE := libassimp LOCAL_SRC_FILES := libassimp.a include $(PREBUILT_STATIC_LIBRARY) include $(CLEAR_VARS) LOCAL_MODULE := assimpImporter #LOCAL_MODULE_FILENAME := assimpImporter LOCAL_SRC_FILES := assimpImporter.cpp LOCAL_LDLIBS := -landroid -lz -llog LOCAL_STATIC_LIBRARIES := libassimp libgnustl_static include $(BUILD_SHARED_LIBRARY) The only things worthy of notice here are the inclusion of the Assimp library we compiled earlier and the use of the gnustl_static library. Since we only have a single native library in the project, we don't have to share the STL library. So, we link it with our library. Finally, we have the Application.mk file: APP_PLATFORM := android-9 APP_STL := gnustl_static There's not much to see here beyond the required specification of the STL runtime that we wish to use and the Android revision we are aiming for. After executing the build command, we are ready to build the actual application that performs the rendering of our model data. Summary With our code added, we can now load 3D models from a variety of formats, import it into our application, and create objects out of them, which we can use together with AndEngine. As implemented now, we essentially have an embedded rendering pipeline for 3D assets that extends the basic AndEngine 2D rendering pipeline. This provides a solid platform for the next stages in extending these basics even further to provide the texturing, lighting, and physics effects that we need to create an actual game. Resources for Article: Further resources on this subject: Cross-platform Building[article] Getting to Know LibGDX [article] Nodes [article]
Read more
  • 0
  • 0
  • 11785
article-image-introducing-jax-rs-api
Packt
21 Sep 2015
25 min read
Save for later

Introducing JAX-RS API

Packt
21 Sep 2015
25 min read
 In this article by Jobinesh Purushothaman, author of the book, RESTful Java Web Services, Second Edition, we will see that there are many tools and frameworks available in the market today for building RESTful web services. There are some recent developments with respect to the standardization of various framework APIs by providing unified interfaces for a variety of implementations. Let's take a quick look at this effort. (For more resources related to this topic, see here.) As you may know, Java EE is the industry standard for developing portable, robust, scalable, and secure server-side Java applications. The Java EE 6 release took the first step towards standardizing RESTful web service APIs by introducing a Java API for RESTful web services (JAX-RS). JAX-RS is an integral part of the Java EE platform, which ensures portability of your REST API code across all Java EE-compliant application servers. The first release of JAX-RS was based on JSR 311. The latest version is JAX-RS 2 (based on JSR 339), which was released as part of the Java EE 7 platform. There are multiple JAX-RS implementations available today by various vendors. Some of the popular JAX-RS implementations are as follows: Jersey RESTful web service framework: This framework is an open source framework for developing RESTful web services in Java. It serves as a JAX-RS reference implementation. You can learn more about this project at https://jersey.java.net. Apache CXF: This framework is an open source web services framework. CXF supports both JAX-WS and JAX-RS web services. To learn more about CXF, refer to http://cxf.apache.org. RESTEasy: This framework is an open source project from JBoss, which provides various modules to help you build a RESTful web service. To learn more about RESTEasy, refer to http://resteasy.jboss.org. Restlet: This framework is a lightweight, open source RESTful web service framework. It has good support for building both scalable RESTful web service APIs and lightweight REST clients, which suits mobile platforms well. You can learn more about Restlet at http://restlet.com. Remember that you are not locked down to any specific vendor here, the RESTful web service APIs that you build using JAX-RS will run on any JAX-RS implementation as long as you do not use any vendor-specific APIs in the code. JAX-RS annotations                                      The main goal of the JAX-RS specification is to make the RESTful web service development easier than it has been in the past. As JAX-RS is a part of the Java EE platform, your code becomes portable across all Java EE-compliant servers. Specifying the dependency of the JAX-RS API To use JAX-RS APIs in your project, you need to add the javax.ws.rs-api JAR file to the class path. If the consuming project uses Maven for building the source, the dependency entry for the javax.ws.rs-api JAR file in the Project Object Model (POM) file may look like the following: <dependency> <groupId>javax.ws.rs</groupId> <artifactId>javax.ws.rs-api</artifactId> <version>2.0.1</version><!-- set the tight version --> <scope>provided</scope><!-- compile time dependency --> </dependency> Using JAX-RS annotations to build RESTful web services Java annotations provide the metadata for your Java class, which can be used during compilation, during deployment, or at runtime in order to perform designated tasks. The use of annotations allows us to create RESTful web services as easily as we develop a POJO class. Here, we leave the interception of the HTTP requests and representation negotiations to the framework and concentrate on the business rules necessary to solve the problem at hand. If you are not familiar with Java annotations, go through the tutorial available at http://docs.oracle.com/javase/tutorial/java/annotations/. Annotations for defining a RESTful resource REST resources are the fundamental elements of any RESTful web service. A REST resource can be defined as an object that is of a specific type with the associated data and is optionally associated to other resources. It also exposes a set of standard operations corresponding to the HTTP method types such as the HEAD, GET, POST, PUT, and DELETE methods. @Path The @javax.ws.rs.Path annotation indicates the URI path to which a resource class or a class method will respond. The value that you specify for the @Path annotation is relative to the URI of the server where the REST resource is hosted. This annotation can be applied at both the class and the method levels. A @Path annotation value is not required to have leading or trailing slashes (/), as you may see in some examples. The JAX-RS runtime will parse the URI path templates in the same way even if they have leading or trailing slashes. Specifying the @Path annotation on a resource class The following code snippet illustrates how you can make a POJO class respond to a URI path template containing the /departments path fragment: import javax.ws.rs.Path; @Path("departments") public class DepartmentService { //Rest of the code goes here } The /department path fragment that you see in this example is relative to the base path in the URI. The base path typically takes the following URI pattern: http://host:port/<context-root>/<application-path>. Specifying the @Path annotation on a resource class method The following code snippet shows how you can specify @Path on a method in a REST resource class. Note that for an annotated method, the base URI is the effective URI of the containing class. For instance, you will use the URI of the following form to invoke the getTotalDepartments() method defined in the DepartmentService class: /departments/count, where departments is the @Path annotation set on the class. import javax.ws.rs.GET; import javax.ws.rs.Path; import javax.ws.rs.Produces; @Path("departments") public class DepartmentService { @GET @Path("count") @Produces("text/plain") public Integer getTotalDepartments() { return findTotalRecordCount(); } //Rest of the code goes here } Specifying variables in the URI path template It is very common that a client wants to retrieve data for a specific object by passing the desired parameter to the server. JAX-RS allows you to do this via the URI path variables as discussed here. The URI path template allows you to define variables that appear as placeholders in the URI. These variables would be replaced at runtime with the values set by the client. The following example illustrates the use of the path variable to request for a specific department resource. The URI path template looks like /departments/{id}. At runtime, the client can pass an appropriate value for the id parameter to get the desired resource from the server. For instance, the URI path of the /departments/10 format returns the IT department details to the caller. The following code snippet illustrates how you can pass the department ID as a path variable for deleting a specific department record. The path URI looks like /departments/10. import javax.ws.rs.Path; import javax.ws.rs.DELETE; @Path("departments") public class DepartmentService { @DELETE @Path("{id}") public void removeDepartment(@PathParam("id") short id) { removeDepartmentEntity(id); } //Other methods removed for brevity } In the preceding code snippet, the @PathParam annotation is used for copying the value of the path variable to the method parameter. Restricting values for path variables with regular expressions JAX-RS lets you use regular expressions in the URI path template for restricting the values set for the path variables at runtime by the client. By default, the JAX-RS runtime ensures that all the URI variables match the following regular expression: [^/]+?. The default regular expression allows the path variable to take any character except the forward slash (/). What if you want to override this default regular expression imposed on the path variable values? Good news is that JAX-RS lets you specify your own regular expression for the path variables. For example, you can set the regular expression as given in the following code snippet in order to ensure that the department name variable present in the URI path consists only of lowercase and uppercase alphanumeric characters: @DELETE @Path("{name: [a-zA-Z][a-zA-Z_0-9]}") public void removeDepartmentByName(@PathParam("name") String deptName) { //Method implementation goes here } If the path variable does not match the regular expression set of the resource class or method, the system reports the status back to the caller with an appropriate HTTP status code, such as 404 Not Found, which tells the caller that the requested resource could not be found at this moment. Annotations for specifying request-response media types The Content-Type header field in HTTP describes the body's content type present in the request and response messages. The content types are represented using the standard Internet media types. A RESTful web service makes use of this header field to indicate the type of content in the request or response message body. JAX-RS allows you to specify which Internet media types of representations a resource can produce or consume by using the @javax.ws.rs.Produces and @javax.ws.rs.Consumes annotations, respectively. @Produces The @javax.ws.rs.Produces annotation is used for defining the Internet media type(s) that a REST resource class method can return to the client. You can define this either at the class level (which will get defaulted for all methods) or the method level. The method-level annotations override the class-level annotations. The possible Internet media types that a REST API can produce are as follows: application/atom+xml application/json application/octet-stream application/svg+xml application/xhtml+xml application/xml text/html text/plain text/xml The following example uses the @Produces annotation at the class level in order to set the default response media type as JSON for all resource methods in this class. At runtime, the binding provider will convert the Java representation of the return value to the JSON format. import javax.ws.rs.Path; import javax.ws.rs.Produces; import javax.ws.rs.core.MediaType; @Path("departments") @Produces(MediaType.APPLICATION_JSON) public class DepartmentService{ //Class implementation goes here... } @Consumes The @javax.ws.rs.Consumes annotation defines the Internet media type(s) that the resource class methods can accept. You can define the @Consumes annotation either at the class level (which will get defaulted for all methods) or the method level. The method-level annotations override the class-level annotations. The possible Internet media types that a REST API can consume are as follows: application/atom+xml application/json application/octet-stream application/svg+xml application/xhtml+xml application/xml text/html text/plain text/xml multipart/form-data application/x-www-form-urlencoded The following example illustrates how you can use the @Consumes attribute to designate a method in a class to consume a payload presented in the JSON media type. The binding provider will copy the JSON representation of an input message to the Department parameter of the createDepartment() method. import javax.ws.rs.Consumes; import javax.ws.rs.core.MediaType; import javax.ws.rs.POST; @POST @Consumes(MediaType.APPLICATION_JSON) public void createDepartment(Department entity) { //Method implementation goes here… } The javax.ws.rs.core.MediaType class defines constants for all media types supported in JAX-RS. To learn more about the MediaType class, visit the API documentation available at http://docs.oracle.com/javaee/7/api/javax/ws/rs/core/MediaType.html. Annotations for processing HTTP request methods In general, RESTful web services communicate over HTTP with the standard HTTP verbs (also known as method types) such as GET, PUT, POST, DELETE, HEAD, and OPTIONS. @GET A RESTful system uses the HTTP GET method type for retrieving the resources referenced in the URI path. The @javax.ws.rs.GET annotation designates a method of a resource class to respond to the HTTP GET requests. The following code snippet illustrates the use of the @GET annotation to make a method respond to the HTTP GET request type. In this example, the REST URI for accessing the findAllDepartments() method may look like /departments. The complete URI path may take the following URI pattern: http://host:port/<context-root>/<application-path>/departments. //imports removed for brevity @Path("departments") public class DepartmentService { @GET @Produces(MediaType.APPLICATION_JSON) public List<Department> findAllDepartments() { //Find all departments from the data store List<Department> departments = findAllDepartmentsFromDB(); return departments; } //Other methods removed for brevity } @PUT The HTTP PUT method is used for updating or creating the resource pointed by the URI. The @javax.ws.rs.PUT annotation designates a method of a resource class to respond to the HTTP PUT requests. The PUT request generally has a message body carrying the payload. The value of the payload could be any valid Internet media type such as the JSON object, XML structure, plain text, HTML content, or binary stream. When a request reaches a server, the framework intercepts the request and directs it to the appropriate method that matches the URI path and the HTTP method type. The request payload will be mapped to the method parameter as appropriate by the framework. The following code snippet shows how you can use the @PUT annotation to designate the editDepartment() method to respond to the HTTP PUT request. The payload present in the message body will be converted and copied to the department parameter by the framework: @PUT @Path("{id}") @Consumes(MediaType.APPLICATION_JSON) public void editDepartment(@PathParam("id") Short id, Department department) { //Updates department entity to data store updateDepartmentEntity(id, department); } @POST The HTTP POST method posts data to the server. Typically, this method type is used for creating a resource. The @javax.ws.rs.POST annotation designates a method of a resource class to respond to the HTTP POST requests. The following code snippet shows how you can use the @POST annotation to designate the createDepartment() method to respond to the HTTP POST request. The payload present in the message body will be converted and copied to the department parameter by the framework: @POST public void createDepartment(Department department) { //Create department entity in data store createDepartmentEntity(department); } @DELETE The HTTP DELETE method deletes the resource pointed by the URI. The @javax.ws.rs.DELETE annotation designates a method of a resource class to respond to the HTTP DELETE requests. The following code snippet shows how you can use the @DELETE annotation to designate the removeDepartment() method to respond to the HTTP DELETE request. The department ID is passed as the path variable in this example. @DELETE @Path("{id}") public void removeDepartment(@PathParam("id") Short id) { //remove department entity from data store removeDepartmentEntity(id); } @HEAD The @javax.ws.rs.HEAD annotation designates a method to respond to the HTTP HEAD requests. This method is useful for retrieving the metadata present in the response headers, without having to retrieve the message body from the server. You can use this method to check whether a URI pointing to a resource is active or to check the content size by using the Content-Length response header field, and so on. The JAX-RS runtime will offer the default implementations for the HEAD method type if the REST resource is missing explicit implementation. The default implementation provided by runtime for the HEAD method will call the method designated for the GET request type, ignoring the response entity retuned by the method. @OPTIONS The @javax.ws.rs.OPTIONS annotation designates a method to respond to the HTTP OPTIONS requests. This method is useful for obtaining a list of HTTP methods allowed on a resource. The JAX-RS runtime will offer a default implementation for the OPTIONS method type, if the REST resource is missing an explicit implementation. The default implementation offered by the runtime sets the Allow response header to all the HTTP method types supported by the resource. Annotations for accessing request parameters You can use this offering to extract the following parameters from a request: a query, URI path, form, cookie, header, and matrix. Mostly, these parameters are used in conjunction with the GET, POST, PUT, and DELETE methods. @PathParam A URI path template, in general, has a URI part pointing to the resource. It can also take the path variables embedded in the syntax; this facility is used by clients to pass parameters to the REST APIs as appropriate. The @javax.ws.rs.PathParam annotation injects (or binds) the value of the matching path parameter present in the URI path template into a class field, a resource class bean property (the getter method for accessing the attribute), or a method parameter. Typically, this annotation is used in conjunction with the HTTP method type annotations such as @GET, @POST, @PUT, and @DELETE. The following example illustrates the use of the @PathParam annotation to read the value of the path parameter, id, into the deptId method parameter. The URI path template for this example looks like /departments/{id}: //Other imports removed for brevity javax.ws.rs.PathParam @Path("departments") public class DepartmentService { @DELETE @Path("{id}") public void removeDepartment(@PathParam("id") Short deptId) { removeDepartmentEntity(deptId); } //Other methods removed for brevity } The REST API call to remove the department resource identified by id=10 looks like DELETE /departments/10 HTTP/1.1. We can also use multiple variables in a URI path template. For example, we can have the URI path template embedding the path variables to query a list of departments from a specific city and country, which may look like /departments/{country}/{city}. The following code snippet illustrates the use of @PathParam to extract variable values from the preceding URI path template: @Produces(MediaType.APPLICATION_JSON) @Path("{country}/{city} ") public List<Department> findAllDepartments( @PathParam("country") String countyCode, @PathParam("city") String cityCode) { //Find all departments from the data store for a country //and city List<Department> departments = findAllMatchingDepartmentEntities(countyCode, cityCode ); return departments; } @QueryParam The @javax.ws.rs.QueryParam annotation injects the value(s) of a HTTP query parameter into a class field, a resource class bean property (the getter method for accessing the attribute), or a method parameter. The following example illustrates the use of @QueryParam to extract the value of the desired query parameter present in the URI. This example extracts the value of the query parameter, name, from the request URI and copies the value into the deptName method parameter. The URI that accesses the IT department resource looks like /departments?name=IT: @GET @Produces(MediaType.APPLICATION_JSON) public List<Department> findAllDepartmentsByName(@QueryParam("name") String deptName) { List<Department> depts= findAllMatchingDepartmentEntities (deptName); return depts; } @MatrixParam Matrix parameters are another way of defining parameters in the URI path template. The matrix parameters take the form of name-value pairs in the URI path, where each pair is preceded by semicolon (;). For instance, the URI path that uses a matrix parameter to list all departments in Bangalore city looks like /departments;city=Bangalore. The @javax.ws.rs.MatrixParam annotation injects the matrix parameter value into a class field, a resource class bean property (the getter method for accessing the attribute), or a method parameter. The following code snippet demonstrates the use of the @MatrixParam annotation to extract the matrix parameters present in the request. The URI path used in this example looks like /departments;name=IT;city=Bangalore. @GET @Produces(MediaType.APPLICATION_JSON) @Path("matrix") public List<Department> findAllDepartmentsByNameWithMatrix(@MatrixParam("name") String deptName, @MatrixParam("city") String locationCode) { List<Department> depts=findAllDepartmentsFromDB(deptName, city); return depts; } You can use PathParam, QueryParam, and MatrixParam to pass the desired search parameters to the REST APIs. Now, you may ask when to use what? Although there are no strict rules here, a very common practice followed by many is to use PathParam to drill down to the entity class hierarchy. For example, you may use the URI of the following form to identify an employee working in a specific department: /departments/{dept}/employees/{id}. QueryParam can be used for specifying attributes to locate the instance of a class. For example, you may use URI with QueryParam to identify employees who have joined on January 1, 2015, which may look like /employees?doj=2015-01-01. The MatrixParam annotation is not used frequently. This is useful when you need to make a complex REST style query to multiple levels of resources and subresources. MatrixParam is applicable to a particular path element, while the query parameter is applicable to the entire request. @HeaderParam The HTTP header fields provide necessary information about the request and response contents in HTTP. For example, the header field, Content-Length: 348, for an HTTP request says that the size of the request body content is 348 octets (8-bit bytes). The @javax.ws.rs.HeaderParam annotation injects the header values present in the request into a class field, a resource class bean property (the getter method for accessing the attribute), or a method parameter. The following example extracts the referrer header parameter and logs it for audit purposes. The referrer header field in HTTP contains the address of the previous web page from which a request to the currently processed page originated: @POST public void createDepartment(@HeaderParam("Referer") String referer, Department entity) { logSource(referer); createDepartmentInDB(department); } Remember that HTTP provides a very wide selection of headers that cover most of the header parameters that you are looking for. Although you can use custom HTTP headers to pass some application-specific data to the server, try using standard headers whenever possible. Further, avoid using a custom header for holding properties specific to a resource, or the state of the resource, or parameters directly affecting the resource. @CookieParam The @javax.ws.rs.CookieParam annotation injects the matching cookie parameters present in the HTTP headers into a class field, a resource class bean property (the getter method for accessing the attribute), or a method parameter. The following code snippet uses the Default-Dept cookie parameter present in the request to return the default department details: @GET @Path("cook") @Produces(MediaType.APPLICATION_JSON) public Department getDefaultDepartment(@CookieParam("Default-Dept") short departmentId) { Department dept=findDepartmentById(departmentId); return dept; } @FormParam The @javax.ws.rs.FormParam annotation injects the matching HTML form parameters present in the request body into a class field, a resource class bean property (the getter method for accessing the attribute), or a method parameter. The request body carrying the form elements must have the content type specified as application/x-www-form-urlencoded. Consider the following HTML form that contains the data capture form for a department entity. This form allows the user to enter the department entity details: <!DOCTYPE html> <html> <head> <title>Create Department</title> </head> <body> <form method="POST" action="/resources/departments"> Department Id: <input type="text" name="departmentId"> <br> Department Name: <input type="text" name="departmentName"> <br> <input type="submit" value="Add Department" /> </form> </body> </html> Upon clicking on the submit button on the HTML form, the department details that you entered will be posted to the REST URI, /resources/departments. The following code snippet shows the use of the @FormParam annotation for extracting the HTML form fields and copying them to the resource class method parameter: @Path("departments") public class DepartmentService { @POST //Specifies content type as //"application/x-www-form-urlencoded" @Consumes(MediaType.APPLICATION_FORM_URLENCODED) public void createDepartment(@FormParam("departmentId") short departmentId, @FormParam("departmentName") String departmentName) { createDepartmentEntity(departmentId, departmentName); } } @DefaultValue The @javax.ws.rs.DefaultValue annotation specifies a default value for the request parameters accessed using one of the following annotations: PathParam, QueryParam, MatrixParam, CookieParam, FormParam, or HeaderParam. The default value is used if no matching parameter value is found for the variables annotated using one of the preceding annotations. The following REST resource method will make use of the default value set for the from and to method parameters if the corresponding query parameters are found missing in the URI path: @GET @Produces(MediaType.APPLICATION_JSON) public List<Department> findAllDepartmentsInRange (@DefaultValue("0") @QueryParam("from") Integer from, @DefaultValue("100") @QueryParam("to") Integer to) { findAllDepartmentEntitiesInRange(from, to); } @Context The JAX-RS runtime offers different context objects, which can be used for accessing information associated with the resource class, operating environment, and so on. You may find various context objects that hold information associated with the URI path, request, HTTP header, security, and so on. Some of these context objects also provide the utility methods for dealing with the request and response content. JAX-RS allows you to reference the desired context objects in the code via dependency injection. JAX-RS provides the @javax.ws.rs.Context annotation that injects the matching context object into the target field. You can specify the @Context annotation on a class field, a resource class bean property (the getter method for accessing the attribute), or a method parameter. The following example illustrates the use of the @Context annotation to inject the javax.ws.rs.core.UriInfo context object into a method variable. The UriInfo instance provides access to the application and request URI information. This example uses UriInfo to read the query parameter present in the request URI path template, /departments/IT: @GET @Produces(MediaType.APPLICATION_JSON) public List<Department> findAllDepartmentsByName( @Context UriInfo uriInfo){ String deptName = uriInfo.getPathParameters().getFirst("name"); List<Department> depts= findAllMatchingDepartmentEntities (deptName); return depts; } Here is a list of the commonly used classes and interfaces, which can be injected using the @Context annotation: javax.ws.rs.core.Application: This class defines the components of a JAX-RS application and supplies additional metadata javax.ws.rs.core.UriInfo: This interface provides access to the application and request URI information javax.ws.rs.core.Request: This interface provides a method for request processing such as reading the method type and precondition evaluation. javax.ws.rs.core.HttpHeaders: This interface provides access to the HTTP header information javax.ws.rs.core.SecurityContext: This interface provides access to security-related information javax.ws.rs.ext.Providers: This interface offers the runtime lookup of a provider instance such as MessageBodyReader, MessageBodyWriter, ExceptionMapper, and ContextResolver javax.ws.rs.ext.ContextResolver<T>: This interface supplies the requested context to the resource classes and other providers javax.servlet.http.HttpServletRequest: This interface provides the client request information for a servlet javax.servlet.http.HttpServletResponse: This interface is used for sending a response to a client javax.servlet.ServletContext: This interface provides methods for a servlet to communicate with its servlet container javax.servlet.ServletConfig: This interface carries the servlet configuration parameters @BeanParam The @javax.ws.rs.BeanParam annotation allows you to inject all matching request parameters into a single bean object. The @BeanParam annotation can be set on a class field, a resource class bean property (the getter method for accessing the attribute), or a method parameter. The bean class can have fields or properties annotated with one of the request parameter annotations, namely @PathParam, @QueryParam, @MatrixParam, @HeaderParam, @CookieParam, or @FormParam. Apart from the request parameter annotations, the bean can have the @Context annotation if there is a need. Consider the example that we discussed for @FormParam. The createDepartment() method that we used in that example has two parameters annotated with @FormParam: public void createDepartment( @FormParam("departmentId") short departmentId, @FormParam("departmentName") String departmentName) Let's see how we can use @BeanParam for the preceding method to give a more logical, meaningful signature by grouping all the related fields into an aggregator class, thereby avoiding too many parameters in the method signature. The DepartmentBean class that we use for this example is as follows: public class DepartmentBean { @FormParam("departmentId") private short departmentId; @FormParam("departmentName") private String departmentName; //getter and setter for the above fields //are not shown here to save space } The following code snippet demonstrates the use of the @BeanParam annotation to inject the DepartmentBean instance that contains all the FormParam values extracted from the request message body: @POST public void createDepartment(@BeanParam DepartmentBean deptBean) { createDepartmentEntity(deptBean.getDepartmentId(), deptBean.getDepartmentName()); } @Encoded By default, the JAX-RS runtime decodes all request parameters before injecting the extracted values into the target variables annotated with one of the following annotations: @FormParam, @PathParam, @MatrixParam, or @QueryParam. You can use @javax.ws.rs.Encoded to disable the automatic decoding of the parameter values. With the @Encoded annotation, the value of parameters will be provided in the encoded form itself. This annotation can be used on a class, method, or parameters. If you set this annotation on a method, it will disable decoding for all parameters defined for this method. You can use this annotation on a class to disable decoding for all parameters of all methods. In the following example, the value of the path parameter called name is injected into the method parameter in the URL encoded form (without decoding). The method implementation should take care of the decoding of the values in such cases: @GET @Produces(MediaType.APPLICATION_JSON) public List<Department> findAllDepartmentsByName(@QueryParam("name") String deptName) { //Method body is removed for brevity } URL encoding converts a string into a valid URL format, which may contain alphabetic characters, numerals, and some special characters supported in the URL string. To learn about the URL specification, visit http://www.w3.org/Addressing/URL/url-spec.html. Summary With the use of annotations, the JAX-RS API provides a simple development model for RESTful web service programming. In case you are interested in knowing other Java RESTful Web Services books that Packt has in store for you, here is the link: RESTful Java Web Services, Jose Sandoval RESTful Java Web Services Security, René Enríquez, Andrés Salazar C Resources for Article: Further resources on this subject: The Importance of Securing Web Services[article] Understanding WebSockets and Server-sent Events in Detail[article] Adding health checks [article]
Read more
  • 0
  • 0
  • 9512

article-image-building-games-html5-and-dart
Packt
21 Sep 2015
19 min read
Save for later

Building Games with HTML5 and Dart

Packt
21 Sep 2015
19 min read
In this article written by Ivo Balbaert, author of the book Learning Dart - Second Edition, you will learn to create a well-known memory game. Also, you will design a model first and work up your way from a modest beginning to a completely functional game, step by step. You will also learn how to enhance the attractiveness of web games with audio and video techniques. The following topics will be covered in this article: The model for the memory game Spiral 1—drawing the board Spiral 2—drawing cells Spiral 3—coloring the cells Spiral 4—implementing the rules Spiral 5—game logic (bringing in the time element) Spiral 6—some finishing touches Spiral 7—using images (For more resources related to this topic, see here.) The model for the memory game When started, the game presents a board with square cells. Every cell hides an image that can be seen by clicking on the cell, but this disappears quickly. You must remember where the images are, because they come in pairs. If you quickly click on two cells that hide the same picture, the cells will "flip over" and the pictures will stay visible. The objective of the game is to turn over all the pairs of matching images in a very short time. After some thinking we came up with the following model, which describes the data handled by the application. In our game, we have a number of pictures, which could belong to a Catalog. For example, a travel catalog with a collection of photos from our trips or something similar. Furthermore, we have a collection of cells and each cell is hiding a picture. Also, we have a structure that we will call memory, and this contains the cells in a grid of rows and columns. We could draw it up as shown in the following figure. You can import the model from the game_memory_json.txt file that contains its JSON representation: A conceptual model of the memory game The Catalog ID is its name, which is mandatory, but the description is optional. The Picture ID consists of the sequence number within the Catalog. The imageUri field stores the location of the image file. width and height are optional properties, since they may be derived from the image file. The size may be small, medium, or large to help select an image. The ID of a Memory is its name within the Catalog, the collection of cells is determined by the memory length, for example, 4 cells per side. Each cell is of the same length cellLength, which is a property of the memory. A memory is recalled when all the image pairs are discovered. Some statistics must be kept, such as recall count, the best recall time in seconds, and the number of cell clicks to recover the whole image (minTryCount). The Cell has the row and column coordinates and also the coordinates of its twin with the same image. Once the model is discussed and improved, model views may be created: a Board would be a view of the Memory concept and a Box would be a view of the Cell concept. The application would be based on the Catalog concept. If there is no need to browse photos of a catalog and display them within a page, there would not be a corresponding view. Now, we can start developing this game from scratch. Spiral 1 – drawing the board The app starts with main() in educ_memory_game.dart: library memory; import 'dart:html'; part 'board.dart'; void main() { // Get a reference to the canvas. CanvasElement canvas = querySelector('#canvas'); (1) new Board(canvas); (2) } We'll draw a board on a canvas element. So, we need a reference that is given in line (1). The Board view is represented in code as its own Board class in the board.dart file. Since everything happens on this board, we construct its object with canvas as an argument (line (2)). Our game board will be periodically drawn as a rectangle in line (4) by using the animationFrame method from the Window class in line (3): part of memory; class Board { CanvasElement canvas; CanvasRenderingContext2D context; num width, height; Board(this.canvas) { context = canvas.getContext('2d'); width = canvas.width; height = canvas.height; window.animationFrame.then(gameLoop); (3) } void gameLoop(num delta) { draw(); window.animationFrame.then(gameLoop); } void draw() { clear(); border(); } void clear() { context.clearRect(0, 0, width, height); } void border() { context..rect(0, 0, width, height)..stroke(); (4) } } This is our first result: The game board Spiral 2 – drawing cells In this spiral, we will give our app code some structure: Board is a view, so board.dart is moved to the view folder. We will also introduce here the Memory class from our model in its own code memory.dart file in the model folder. So, we will have to change the part statements to the following: part 'model/memory.dart'; part 'view/board.dart'; The Board view needs to know about Memory. So, we will include it in the Board class and make its object in the Board constructor: new Board(canvas, new Memory(4)); The Memory class is still very rudimentary with only its length property: class Memory { num length; Memory(this.length); } Our Board class now also needs a method to draw the lines, which we decided to make private because it is specific to Board, as well as the clear() and border()methods: void draw() { _clear(); _border(); _lines(); } The lines method is quite straightforward; first draw it on a piece of paper and translate it to code using moveTo and lineTo. Remember that x goes from top-left to right and y goes from top-left to bottom: void _lines() { var gap = height / memory.length; var x, y; for (var i = 1; i < memory.length; i++) { x = gap * i; y = x; context ..moveTo(x, 0) ..lineTo(x, height) ..moveTo(0, y) ..lineTo(width, y); } } The result is a nice grid: Board with cells Spiral 3 – coloring the cells To simplify, we will start using colors instead of pictures to be shown in the grid. Up until now, we didn't implement the cell from the model. Let's do that in modelcell.dart. We start simple by saying that the Cell class has the row, column, and color properties, and it belongs to a Memory object passed in its constructor: class Cell { int row, column; String color; Memory memory; Cell(this.memory, this.row, this.column); } Because we need a collection of cells, it is a good idea to make a Cells class, which contains List. We give it an add method and also an iterator so that we are able to use a for…in statement to loop over the collection: class Cells { List _list; Cells() { _list = new List(); } void add(Cell cell) { _list.add(cell); } Iterator get iterator => _list.iterator; } We will need colors that are randomly assigned to the cells. We will also need some utility variables and methods that do not specifically belong to the model and don't need a class. Hence, we will code them in a folder called util. To specify the colors for the cells, we will use two utility variables: a List variable of colors (colorList), which has the name colors, and a colorMap variable that maps the names to their RGB values. Refer to utilcolor.dart; later on, we can choose some fancier colors: var colorList = ['black', 'blue', //other colors ]; var colorMap = {'black': '#000000', 'blue': '#0000ff', //... }; To generate (pseudo) random values (ints, doubles, or Booleans), Dart has the Random class from dart:math. We will use the nextInt method, which takes an integer (the maximum value) and returns a positive random integer in the range from 0 (inclusive) to max (exclusive). We will build upon this in utilrandom.dart to make methods that give us a random color: int randomInt(int max) => new Random().nextInt(max); randomListElement(List list) => list[randomInt(list.length - 1)]; String randomColor() => randomListElement(colorList); String randomColorCode() => colorMap[randomColor()]; Our Memory class now contains an instance of the Cells class: Cells cells; We build this in the Memory constructor in a nested for loop, where each cell is successively instantiated with a row and column, given a random color, and added to cells: Memory(this.length) { cells = new Cells(); var cell; for (var x = 0; x < length; x++) { for (var y = 0; y < length; y++) { cell = new Cell(this, x, y); cell.color = randomColor(); cells.add(cell); } } } We can draw a rectangle and fill it with a color at the same time. So, we realize that we don't need to draw lines as we did in the previous spiral! The _boxes method is called from the draw animation: with a for…in statement, we loop over the collection of cells and call the _colorBox method that will draw and color the cell for each cell: void _boxes() { for (Cell cell in memory.cells) { _colorBox(cell); } } void _colorBox(Cell cell) { var gap = height / memory.length; var x = cell.row * gap; var y = cell.column * gap; context ..beginPath() ..fillStyle = colorMap[cell.color] ..rect(x, y, gap, gap) ..fill() ..stroke() ..closePath(); } Spiral 4 – implementing the rules However, wait! Our game can only work if the same color appears in only two cells: a cell and its twin cell. Moreover, a cell can be hidden or not: the color can be seen or not? To take care of this, the Cell class gets two new attributes: Cell twin; bool hidden = true; The _colorBox method in the Board class can now show the color of the cell when hidden is false (line (2)); when hidden = true (the default state), a neutral gray color will be used for the cell (line (1)): static const String COLOR_CODE = '#f0f0f0'; We also gave the gap variable a better name, boxSize: void _colorBox(Cell cell) { var x = cell.column * boxSize; var y = cell.row * boxSize; context.beginPath(); if (cell.hidden) { context.fillStyle = COLOR_CODE; (1) } else { context.fillStyle = colorMap[cell.color]; (2) } // same code as in Spiral 3 } The lines (1) and (2) can also be stated more succinctly with the ? ternary operator. Remember that the drawing changes because the _colorBox method is called via draw at 60 frames per second and the board can react to a mouse click. In this spiral, we will show a cell when it is clicked together with its twin cell and then they will stay visible. Attaching an event handler for this is easy. We add the following line to the Board constructor: querySelector('#canvas').onMouseDown.listen(onMouseDown); The onMouseDown event handler has to know on which cell the click occurred. The mouse event e contains the coordinates of the click in its e.offset.x and e.offset.y properties (lines (3) and (4)). We will obtain the cell's row and column by using a truncating division ~/ operator dividing the x (which gives the column) and y (which gives the row) values by boxSize: void onMouseDown(MouseEvent e) { int row = e.offset.y ~/ boxSize; (3) int column = e.offset.x ~/ boxSize; (4) Cell cell = memory.getCell(row, column); (5) cell.hidden = false; (6) cell.twin.hidden = false; (7) } Memory has a collection of cells. To get the cell with a specified row and column value, we will add a getCell method to memory and call it in line (5). When we have the cell, we will set its hidden property and that of its twin cell to false (lines (6) to (7)). The getCell method must return the cell at the given row and column. It loops through all the cells in line (8) and checks each cell, whether it is positioned at that row and column (line (9)). If yes, it will return that cell: Cell getCell(int row, int column) { for (Cell cell in cells) { (8) if (cell.intersects(row, column)) { (9) return cell; } } } For this purpose, we will add an intersects method to the Cell class. This checks whether its row and column match the given row and column for the current cell (see line (10)): bool intersects(int row, int column) { if (this.row == row && this.column == column) { (10) return true; } return false; } Now, we have already added a lot of functionality, but the drawing of the board will need some more thinking: How to give a cell (and its twin cell) a random color that is not yet used? How to attach a cell randomly to a twin cell that is not yet used? To end this, we will have to make the constructor of Memory a lot more intelligent: Memory(this.length) { if (length.isOdd) { (1) throw new Exception( 'Memory length must be an even integer: $length.'); } cells = new Cells(); var cell, twinCell; for (var x = 0; x < length; x++) { for (var y = 0; y < length; y++) { cell = getCell(y, x); (2) if (cell == null) { (3) cell = new Cell(this, y, x); cell.color = _getFreeRandomColor(); (4) cells.add(cell); twinCell = _getFreeRandomCell(); (5) cell.twin = twinCell; (6) twinCell.twin = cell; twinCell.color = cell.color; cells.add(twinCell); } } } } The number of pairs given by ((length * length) / 2) must be even. This is only true if the length parameter of Memory itself is even, so we checked it in line (1). Again, we coded a nested loop and got the cell at that row and column. However, as the cell at that position has not yet been made (line (3)), we continued to construct it and assign its color and twin. In line (4), we called _getFreeRandomColor to get a color that is not yet used: String _getFreeRandomColor() { var color; do { color = randomColor(); } while (usedColors.any((c) => c == color)); (7) usedColors.add(color); (8) return color; } The do…while loop continues as long as the color is already in a list of usedColors. On exiting from the loop, we found an unused color, which is added to usedColors in line (8) and also returned. We then had to set everything for the twin cell. We searched for a free one with the _getFreeRandomCell method in line (5). Here, the do…while loop continues until a (row, column) position is found where cell == null is, meaning that we haven't yet created a cell there (line (9)). We will promptly do this in line (10): Cell _getFreeRandomCell() { var row, column; Cell cell; do { row = randomInt(length); column = randomInt(length); cell = getCell(row, column); } while (cell != null); (9) return new Cell(this, row, column); (10) } From line (6) onwards, the properties of the twin cell are set and added to the list. This is all we need to produce the following result: Paired colored cells Spiral 5 – game logic (bringing in the time element) Our app isn't playable yet: When a cell is clicked, its color must only show for a short period of time (say one second) When a cell and its twin cell are clicked within a certain time interval, they must remain visible All of this is coded in the mouseDown event handler and we also need a lastCellClicked variable of the Cell type in the Board class. Of course, this is exactly the cell we get in the mouseDown event handler. So, we will set it in line (5) in the following code snippet: void onMouseDown(MouseEvent e) { // same code as in Spiral 4 - if (cell.twin == lastCellClicked && lastCellClicked.shown) { (1) lastCellClicked.hidden = false; (2) if (memory.recalled) memory.hide(); (3) } else { new Timer(const Duration(milliseconds: 1000), () => cell.hidden = true); (4) } lastCellClicked = cell; (5) } In line (1), we checked whether the last clicked cell was the twin cell and whether this is still shown. Then, we made sure in (2) that it stays visible. shown is a new getter in the Cell class to make the code more readable: bool get shown => !hidden;. If at that moment all the cells were shown (the memory is recalled), we again hid them in line (3). If the last clicked cell was not the twin cell, we hid the current cell after one second in line (4). recalled is a simple getter (read-only property) in the Memory class and it makes use of a Boolean variable in Memory that is initialized to false (_recalled = false;): bool get recalled { if (!_recalled) { if (cells.every((c) => c.shown)) { (6) _recalled = true; } } return _recalled; } In line (6), we tested that if every cell is shown, then this variable is set to true (the game is over). every is a new method in the Cells List and a nice functional way to write this is given as follows: bool every(Function f) => list.every(f); The hide method is straightforward: hide every cell and reset the _recalled variable to false: hide() { for (final cell in cells) cell.hidden = true; _recalled = false; } This is it, our game works! Spiral 6 – some finishing touches A working program always gives its developer a sense of joy, and rightfully so. However, this doesn't that mean you can leave the code as it is. On the contrary, carefully review your code for some time to see whether there is room for improvement or optimization. For example, are the names you used clear enough? The color of a hidden cell is now named simply COLOR_CODE in board.dart, renaming it to HIDDEN_CELL_COLOR_CODE makes its meaning explicit. The List object used in the Cells class can indicate that it is List<Cell>, by applying the fact that Dart lists are generic. The parameter of the every method in the Cell class is more precise—it is a function that accepts a cell and returns bool. Our onMouseDown event handler contains our game logic, so it is very important to tune it if possible. After some thought, we see that the code from the previous spiral can be improved; in the following line, the second condition after && is, in fact, unnecessary: if (cell.twin == lastCellClicked && lastCellClicked.shown) {...} When the player has guessed everything correctly, showing the completed screen for a few seconds will be more satisfactory (line (2)). So, this portion of our event handler code will change to: if (cell.twin == lastCellClicked) { (1) lastCellClicked.hidden = false; if (memory.recalled) { // game over new Timer(const Duration(milliseconds: 5000), () => memory.hide()); (2) } } else if (cell.twin.hidden) { new Timer(const Duration(milliseconds: 800), () => cell.hidden = true); } Why don’t we show a "YOU HAVE WON!" banner. We will do this by drawing the text on the canvas (line (3)), so we must do it in the draw() method (otherwise, it would disappear after INTERVAL milliseconds): void draw() { _clear(); _boxes(); if (memory.recalled) { // game over context.font = "bold 25px sans-serif"; context.fillStyle = "red"; context.fillText("YOU HAVE WON !", boxSize, boxSize * 2); (3) } } Then, the same game with the same configuration can be played again. We could make it more obvious that a cell is hidden by decorating it with a small circle in the _colorBox method (line (4)): if (cell.hidden) { context.fillStyle = HIDDEN_CELL_COLOR_CODE; var centerX = cell.column * boxSize + boxSize / 2; var centerY = cell.row * boxSize + boxSize / 2; var radius = 4; context.arc(centerX, centerY, radius, 0, 2 * PI, false); (4) } We do want to give our player a chance to start over by supplying a Play again button. The easiest way will be to simply refresh the screen (line (5)) by adding this code to the startup script: void main() { canvas = querySelector('#canvas'); ButtonElement play = querySelector('#play'); play.onClick.listen(playAgain); new Board(canvas, new Memory(4)); } playAgain(Event e) { window.location.reload(); (5) } Spiral 7 – using images One improvement that certainly comes to mind is the use of pictures instead of colors as shown in the Using images screenshot. How difficult would that be? It turns out that this is surprisingly easy, because we already have the game logic firmly in place! In the images folder, we supply a number of game pictures. Instead of the color property, we give the cell a String property (image), which will contain the name of the picture file. We then replace utilcolor.dart with utilimages.dart, which contains a imageList variable with the image filenames. In utilrandom.dart, we will replace the color methods with the following code: String randomImage() => randomListElement(imageList); The changes to memory.dart are also straightforward: replace the usedColor list with List usedImages = []; and the _getFreeRandomColor method with _getFreeRandomImage, which will use the new list and method: List usedImages = []; String _getFreeRandomImage() { var image; do { image = randomImage(); } while (usedImages.any((i) => i == image)); usedImages.add(image); return image; } In board.dart, we replace _colorBox(cell) with _imageBox(cell). The only new thing is how to draw the image on canvas. For this, we need ImageElement objects. Here, we have to be careful to create these objects only once and not over and over again in every draw cycle, because this produces a flickering screen. We will store the ImageElements object in a Map: var imageMap = new Map<String, ImageElement>(); Then, we populate this in the Board constructor with a for…in loop over memory.cells: for (var cell in memory.cells) { ImageElement image = new Element.tag('img'); (1) image.src = 'images/${cell.image}'; (2) imageMap[cell.image] = image; (3) } We create a new ImageElement object in line (1), giving it the complete file path to the image file as a src property in line (2) and store it in imageMap in line (3). The image file will then be loaded into memory only once. We don't do any unnecessary network access to effectively cache the images. In the draw cycle, we will load the image from imageMap and draw it in the current cell with the drawImage method in line (4): if (cell.hidden) { // see previous code } else { ImageElement image = imageMap[cell.image]; context.drawImage(image, x, y); // resize to cell size (4) } Perhaps, you can think of other improvements? Why not let the player specify the game difficulty by asking the number of boxes. It is 16 now. Check whether the input is a square of an even number. Do you have enough colors to choose from? Perhaps, dynamically building a list with enough random colors would be a better idea. Calculating and storing the statistics discussed in the model would also make the game more attractive. Another enhancement from the model is to support different catalogs of pictures. Go ahead and exercise your Dart skills! Summary By thoroughly investigating two games applying all of Dart we have already covered, your Dart star begins to shine. For other Dart games, visit http://www.builtwithdart.com/projects/games/. You can find more information at http://www.dartgamedevs.org/ on building games. Resources for Article: Further resources on this subject: Slideshow Presentations [article] Dart with JavaScript [article] Practical Dart [article]
Read more
  • 0
  • 0
  • 19049
Modal Close icon
Modal Close icon