How-To Tutorials

12 Oct 2016

15 min read

Create a User Profile System and use the Null Coalesce Operator

12 Oct 2016

In this article by Jose Palala and Martin Helmich, author of PHP 7 Programming Blueprints, will show you how to build a simple profiles page with listed users which you can click on, and create a simple CRUD-like system which will enable us to register new users to the system, and delete users for banning purposes. (For more resources related to this topic, see here.) You will learn to use the PHP 7 null coalesce operator so that you can show data if there is any, or just display a simple message if there isn’t any. Let's create a simple UserProfile class. The ability to create classes has been available since PHP 5. A class in PHP starts with the word class, and the name of the class: class UserProfile { private $table = 'user_profiles'; } } We've made the table private and added a private variable, where we define which table it will be related to. Let's add two functions, also known as a method, inside the class to simply fetch the data from the database: function fetch_one($id) { $link = mysqli_connect(''); $query = "SELECT * from ". $this->table . " WHERE `id` =' " . $id "'"; $results = mysqli_query($link, $query); } function fetch_all() { $link = mysqli_connect('127.0.0.1', 'root','apassword','my_dataabase' ); $query = "SELECT * from ". $this->table . "; $results = mysqli_query($link, $query); } The null coalesce operator We can use PHP 7's null coalesce operator to allow us to check whether our results contain anything, or return a defined text which we can check on the views—this will be responsible for displaying any data. Lets put this in a file which will contain all the define statements, and call it: //definitions.php define('NO_RESULTS_MESSAGE', 'No results found'); require('definitions.php'); function fetch_all() { …same lines ... $results = $results ?? NO_RESULTS_MESSAGE; return $message; } On the client side, we'll need to come up with a template to show the list of user profiles. Let’s create a basic HTML block to show that each profile can be a div element with several list item elements to output each table. In the following function, we need to make sure that all values have been filled in with at least the name and the age. Then we simply return the entire string when the function is called: function profile_template( $name, $age, $country ) { $name = $name ?? null; $age = $age ?? null; if($name == null || $age === null) { return 'Name or Age need to be set'; } else { return '<div> <li>Name: ' . $name . ' </li> <li>Age: ' . $age . '</li> <li>Country: ' . $country . ' </li> </div>'; } } Separation of concerns In a proper MVC architecture, we need to separate the view from the models that get our data, and the controllers will be responsible for handling business logic. In our simple app, we will skip the controller layer since we just want to display the user profiles in one public facing page. The preceding function is also known as the template render part in an MVC architecture. While there are frameworks available for PHP that use the MVC architecture out of the box, for now we can stick to what we have and make it work. PHP frameworks can benefit a lot from the null coalesce operator. In some codes that I've worked with, we used to use the ternary operator a lot, but still had to add more checks to ensure a value was not falsy. Furthermore, the ternary operator can get confusing, and takes some getting used to. The other alternative is to use the isSet function. However, due to the nature of the isSet function, some falsy values will be interpreted by PHP as being a set. Creating views Now that we have our model complete, a template render function, we just need to create the view with which we can look at each profile. Our view will be put inside a foreach block, and we'll use the template we wrote to render the right values: //listprofiles.php <html> <!doctype html> <head> <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css"> </head> <body> <?php foreach($results as $item) { echo profile_template($item->name, $item->age, $item->country; } ?> </body> </html> Let's put the code above into index.php. While we may install the Apache server, configure it to run PHP, install new virtual hosts and the other necessary featuress, and put our PHP code into an Apache folder, this will take time, so, for the purposes of testing this out, we can just run PHP's server for development. To run the built-in PHP server (read more at http://php.net/manual/en/features.commandline.webserver.php) we will use the folder we are running, inside a terminal: php -S localhost:8000 If we open up our browser, we should see nothing yet—No results found. This means we need to populate our database. If you have an error with your database connection, be sure to replace the correct database credentials we supplied into each of the mysql_connect calls that we made. To supply data into our database, we can create a simple SQL script like this: INSERT INTO user_profiles ('Chin Wu', 30, 'Mongolia'); INSERT INTO user_profiles ('Erik Schmidt', 22, 'Germany'); INSERT INTO user_profiles ('Rashma Naru', 33, 'India'); Let's save it in a file such as insert_profiles.sql. In the same directory as the SQL file, log on to the MySQL client by using the following command: mysql -u root -p Then type use <name of database>: mysql> use <database>; Import the script by running the source command: mysql> source insert_profiles.sql Now our user profiles page should show the following: Create a profile input form Now let's create the HTML form for users to enter their profile data. Our profiles app would be no use if we didn't have a simple way for a user to enter their user profile details. We'll create the profile input form like this: //create_profile.php <html> <body> <form action="post_profile.php" method="POST"> <label>Name</label><input name="name"> <label>Age</label><input name="age"> <label>Country</label><input name="country"> </form> </body> </html> In this profile post, we'll need to create a PHP script to take care of anything the user posts. It will create an SQL statement from the input values and output whether or not they were inserted. We can use the null coalesce operator again to verify that the user has inputted all values and left nothing undefined or null: $name = $_POST['name'] ?? ""; $age = $_POST['country'] ?? ""; $country = $_POST['country'] ?? ""; This prevents us from accumulating errors while inserting data into our database. First, let's create a variable to hold each of the inputs in one array: $input_values = [ 'name' => $name, 'age' => $age, 'country' => $country ]; The preceding code is a new PHP 5.4+ way to write arrays. In PHP 5.4+, it is no longer necessary to put an actual array(); the author personally likes the new syntax better. We should create a new method in our UserProfile class to accept these values: Class UserProfile { public function insert_profile($values) { $link = mysqli_connect('127.0.0.1', 'username','password', 'databasename'); $q = " INSERT INTO " . $this->table . " VALUES ( '". $values['name']."', '".$values['age'] . "' ,'". $values['country']. "')"; return mysqli_query($q); } } Instead of creating a parameter in our function to hold each argument as we did with our profile template render function, we can simply use an array to hold our values. This way, if a new field needs to be inserted into our database, we can just add another field to the SQL insert statement. While we are at it, let's create the edit profile section. For now, we'll assume that whoever is using this edit profile is the administrator of the site. We'll need to create a page where, provided the $_GET['id'] or has been set, that the user that we will be fetching from the database and displaying on the form. <?php require('class/userprofile.php');//contains the class UserProfile into $id = $_GET['id'] ?? 'No ID'; //if id was a string, i.e. "No ID", this would go into the if block if(is_numeric($id)) { $profile = new UserProfile(); //get data from our database $results = $user->fetch_id($id); if($results && $results->num_rows > 0 ) { while($obj = $results->fetch_object()) { $name = $obj->name; $age = $obj->age; $country = $obj->country; } //display form with a hidden field containing the value of the ID ?> <form action="post_update_profile.php" method="post"> <label>Name</label><input name="name" value="<?=$name?>"> <label>Age</label><input name="age" value="<?=$age?>"> <label>Country</label><input name="country" value="<?=country?>"> </form> <?php } else { exit('No such user'); } } else { echo $id; //this should be No ID'; exit; } Notice that we're using what is known as the shortcut echo statement in the form. It makes our code simpler and easier to read. Since we're using PHP 7, this feature should come out of the box. Once someone submits the form, it goes into our $_POST variable and we'll create a new Update function in our UserProfile class. Admin system Let's finish off by creating a simple grid for an admin dashboard portal that will be used with our user profiles database. Our requirement for this is simple: We can just set up a table-based layout that displays each user profile in rows. From the grid, we will add the links to be able to edit the profile, or delete it, if we want to. The code to display a table in our HTML view would look like this: <table> <tr> <td>John Doe</td> <td>21</td> <td>USA</td> <td><a href="edit_profile.php?id=1">Edit</a></td> <td><a href="profileview.php?id=1">View</a> <td><a href="delete_profile.php?id=1">Delete</a> </tr> </table> This script to this is the following: //listprofiles.php $sql = "SELECT * FROM userprofiles LIMIT $start, $limit "; $rs_result = mysqli_query ($sql); //run the query while($row = mysqli_fetch_assoc($rs_result) { ?> <tr> <td><?=$row['name'];?></td> <td><?=$row['age'];?></td> <td><?=$row['country'];?></td> <td><a href="edit_profile.php?id=<?=$id?>">Edit</a></td> <td><a href="profileview.php?id=<?=$id?>">View</a> <td><a href="delete_profile.php?id=<?=$id?>">Delete</a> </tr> <?php } There's one thing that we haven't yet created: A delete_profile.php page. The view and edit pages - have been discussed already. Here's how the delete_profile.php page would look: <?php //delete_profile.php $connection = mysqli_connect('localhost','<username>','<password>', '<databasename>'); $id = $_GET['id'] ?? 'No ID'; if(is_numeric($id)) { mysqli_query( $connection, "DELETE FROM userprofiles WHERE id = '" .$id . "'"); } else { echo $id; } i(!is_numeric($id)) { exit('Error: non numeric $id'); } else { echo "Profile #" . $id . " has been deleted"; ?> Of course, since we might have a lot of user profiles in our database, we have to create a simple pagination. In any pagination system, you just need to figure out the total number of rows, and how many rows you want displayed per page. We can create a function that will be able to return a URL that contains the page number and how many to view per page. From our queries database, we first create a new function for us to select only up to the total number of items in our database: class UserProfile{ // …. Etc … function count_rows($table) { $dbconn = new mysqli('localhost', 'root', 'somepass', 'databasename'); $query = $dbconn->query("select COUNT(*) as num from '". $table . "'"); $total_pages = mysqli_fetch_array($query); return $total_pages['num']; //fetching by array, so element 'num' = count } For our pagination, we can create a simple paginate function which accepts the base_url of the page where we have pagination, the rows per page — also known as the number of records we want each page to have — and the total number of records found: require('definitions.php'); require('db.php'); //our database class Function paginate ($baseurl, $rows_per_page, $total_rows) { $pagination_links = array(); //instantiate an array to hold our html page links //we can use null coalesce to check if the inputs are null ( $total_rows || $rows_per_page) ?? exit('Error: no rows per page and total rows); //we exit with an error message if this function is called incorrectly $pages = $total_rows % $rows_per_page; $i= 0; $pagination_links[$i] = "<a href="http://". $base_url . "?pagenum=". $pagenum."&rpp=".$rows_per_page. ">" . $pagenum . "</a>"; } return $pagination_links; } This function will help display the above page links in a table: function display_pagination($links) { $display = ' <div class="pagination">'; <table><tr>'; foreach ($links as $link) { echo "<td>" . $link . "</td>"; } $display .= '</tr></table></div>'; return $display; } Notice that we're following the principle that there should rarely be any echo statements inside a function. This is because we want to make sure that other users of these functions are not confused when they debug some mysterious output on their page. By requiring the programmer to echo out whatever the functions return, it becomes easier to debug our program. Also, we're following the separation of concerns—our code doesn't output the display, it just formats the display. So any future programmer can just update the function's internal code and return something else. It also makes our function reusable; imagine that in the future someone uses our function—this way, they won't have to double check that there's some misplaced echo statement within our functions. A note on alternative short tags As you know, another way to echo is to use the <?= tag. You can use it like so: <?="helloworld"?>.These are known as short tags. In PHP 7, alternative PHP tags have been removed. The RFC states that <%, <%=, %> and <script language=php> have been deprecated. The RFC at https://wiki.php.net/rfc/remove_alternative_php_tags says that the RFC does not remove short opening tags (<?) or short opening tags with echo (<?=). Since we have laid out the groundwork of creating paginate links, we now just have to invoke our functions. The following script is all that is needed to create a paginated page using the preceding function: $mysqli = mysqli_connect('localhost','<username>','<password>', '<dbname>'); $limit = $_GET['rpp'] ?? 10; //how many items to show per page default 10; $pagenum = $_GET['pagenum']; //what page we are on if($pagenum) $start = ($pagenum - 1) * $limit; //first item to display on this page else $start = 0; //if no page var is given, set start to 0 /*Display records here*/ $sql = "SELECT * FROM userprofiles LIMIT $start, $limit "; $rs_result = mysqli_query ($sql); //run the query while($row = mysqli_fetch_assoc($rs_result) { ?> <tr> <td><?php echo $row['name']; ?></td> <td><?php echo $row['age']; ?></td> <td><?php echo $row['country']; ?></td> </tr> <?php } /* Let's show our page */ /* get number of records through */ $record_count = $db->count_rows('userprofiles'); $pagination_links = paginate('listprofiles.php' , $limit, $rec_count); echo display_pagination($paginaiton_links); The HTML output of our page links in listprofiles.php will look something like this: <div class="pagination"><table> <tr> <td> <a href="listprofiles.php?pagenum=1&rpp=10">1</a> </td> <td><a href="listprofiles.php?pagenum=2&rpp=10">2</a> </td> <td><a href="listprofiles.php?pagenum=3&rpp=10">2</a> </td> </tr> </table></div> Summary As you can see, we have a lot of use cases for the null coalesce. We learned how to make a simple user profile system, and how to use PHP 7's null coalesce feature when fetching data from the database, which returns null if there are no records. We also learned that the null coalesce operator is similar to a ternary operator, except this returns null by default if there is no data. Resources for Article: Further resources on this subject: Running Simpletest and PHPUnit [article] Mapping Requirements for a Modular Web Shop App [article] HTML5: Generic Containers [article]

0
0
8131

article-image-server-side-swift-building-slack-bot-part-1

Peter Zignego

12 Oct 2016

5 min read

Server-side Swift: Building a Slack Bot, Part 1

Peter Zignego

12 Oct 2016

5 min read

As a remote iOS developer, I love Slack. It’s my meeting room and my water cooler over the course of a work day. If you’re not familiar with Slack, it is a group communication tool popular in Silicon Valley and beyond. What makes Slack valuable beyond replacing email as the go-to communication method for buisnesses is that it is more than chat; it is a platform. Thanks to Slack’s open attitude toward developers with its API, hundreds of developers have been building what have become known as Slack bots. There are many different libraries available to help you start writing your Slack bot, covering a wide range of programming languages. I wrote a library in Apple’s new programming language (Swift) for this very purpose, called SlackKit. SlackKit wasn’t very practical initially—it only ran on iOS and OS X. On the modern web, you need to support Linux to deploy on Amazon Web Servies, Heroku, or hosted server companies such as Linode and Digital Ocean. But last June, Apple open sourced Swift, including official support for Linux (Ubuntu 14 and 15 specifically). This made it possible to deploy Swift code on Linux servers, and developers hit the ground running to build out the infrastructure needed to make Swift a viable language for server applications. Even with this huge developer effort, it is still early days for server-side Swift. Apple’s Linux Foundation port is a huge undertaking, as is the work to get libdispatch, a concurrency framework that provides much of the underpinning for Foundation. In addition to rough official tooling, writing code for server-side Swift can be a bit like hitting a moving target, with biweekly snapshot releases and multiple, ABI-incompatible versions to target. Zewo to Sixty on Linux Fortunately, there are some good options for deploying Swift code on servers right now, even with Apple’s libraries in flux. I’m going to focus in on one in particular: Zewo. Zewo is modular by design, allowing us to use the Swift Package Manager to pull in only what we need instead of a monolithic framework. It’s open source and is a great community of developers that spans the globe. If you’re interested in the world of server-side Swift, you should get involved! Oh, and of course they have a Slack. Using Zewo and a few other open source libraries, I was able to build a version of SlackKit that runs on Linux. A Swift Tutorial In this two-part post series I have detailed a step-by-step guide to writing a Slack bot in Swift and deploying it to Heroku. I’m going to be using OS X but this is also achievable on Linux using the editor of your choice. Prerequisites Install Homebrew: /usr/bin/ruby -e “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" Install swiftenv: brew install kylef/formulae/swiftenv Configure your shell: echo ‘if which swiftenv > /dev/null; then eval “$(swiftenv init -)”; fi’ >> ~/.bash_profile Download and install the latest Zewo-compatible snapshot: swiftenv install DEVELOPMENT-SNAPSHOT-2016-05-09-a swiftenv local DEVELOPMENT-SNAPSHOT-2016-05-09-a Install and Link OpenSSL: brew install openssl brew link openssl --force Let’s Keep Score The sample application we’ll be building is a leaderboard for Slack, like PlusPlus++ by Betaworks. It works like this: add a point for every @thing++, subtract a point for every @thing--, and show a leaderboard when asked @botname leaderboard. First, we need to create the directory for our application and initialize the basic project structure. mkdir leaderbot && cd leaderbot swift build --init Next, we need to edit Package.swift to add our dependency, SlackKit: importPackageDescription let package = Package( name: "Leaderbot", targets: [], dependencies: [ .Package(url: "https://github.com/pvzig/SlackKit.git", majorVersion: 0, minor: 0), ] ) SlackKit is dependent on several Zewo libraries, but thanks to the Swift Package Manager, we don’t have to worry about importing them explicitly. Then we need to build our dependencies: swift build And our development environment (we need to pass in some linker flags so that swift build knows where to find the version of OpenSSL we installed via Homebrew and the C modules that some of our Zewo libraries depend on): swift build -Xlinker -L$(pwd)/.build/debug/ -Xswiftc -I/usr/local/include -Xlinker -L/usr/local/lib -X In Part 2, I will show all of the Swift code, how to get an API token, how to test the app and deploy it on Heroku, and finally how to launch it. Disclaimer The linux version of SlackKit should be considered an alpha release. It’s a fun tech demo to show what’s possible with Swift on the server, not something to be relied upon. Feel free to report issues you come across. About the author Peter Zignego is an iOS developer in Durham, North Carolina. He writes at bytesized.co, tweets @pvzig, and freelances at Launch Software.fto help you start writing your Slack bot, covering a wide range of programming languages. I wrote a library in Apple’s new programming language (Swift) for this very purpose, called SlackKit. SlackKit wasn’t very practical initially—it only ran on iOS and OS X. On the modern web, you need to support Linux to deploy on Amazon Web Servies, Heroku, or hosted server

0
0
5013

article-image-basics-image-histograms-opencv

Packt

12 Oct 2016

11 min read

Basics of Image Histograms in OpenCV

Packt

12 Oct 2016

11 min read

In this article by Samyak Datta, author of the book Learning OpenCV 3 Application Development we are going to focus our attention on a different style of processing pixel values. The output of the techniques, which would comprise our study in the current article, will not be images, but other forms of representation for images, namely image histograms. We have seen that a two-dimensional grid of intensity values is one of the default forms of representing images in digital systems for processing as well as storage. However, such representations are not at all easy to scale. So, for an image with a reasonably low spatial resolution, say 512 x 512 pixels, working with a two-dimensional grid might not pose any serious issues. However, as the dimensions increase, the corresponding increase in the size of the grid may start to adversely affect the performance of the algorithms that work with the images. A primary advantage that an image histogram has to offer is that the size of a histogram is a constant that is independent of the dimensions of the image. As a consequence of this, we are guaranteed that irrespective of the spatial resolution of the images that we are dealing with, the algorithms that power our solutions will have to deal with a constant amount of data if they are working with image histograms. (For more resources related to this topic, see here.) Each descriptor captures some particular aspects or features of the image to construct its own form of representation. One of the common pitfalls of using histograms as a form of image representation as compared to its native form of using the entire two-dimensional grid of values is loss of information. A full-fledged image representation using pixel intensity values for all pixel locations naturally consists of all the information that you would need to reconstruct a digital image. However, the same cannot be said about histograms. When we study about image histograms in detail, we'll get to see exactly what information do we stand to lose. And this loss in information is prevalent across all forms of image descriptors. The basics of histograms At the outset, we will briefly explain the concept of a histogram. Most of you might already know this from your lessons on basic statistics. However, we will reiterate this for the sake of completeness. Histogram is a form of data representation technique that relies on an aggregation of data points. The data is aggregated into a set of predefined bins that are represented along the x axis, and the number of data points that fall within each of the bins make up the corresponding counts on the y axis. For example, let's assume that our data looks something like the following: D={2,7,1,5,6,9,14,11,8,10,13} If we define three bins, namely Bin_1 (1 - 5), Bin_2 (6 - 10), and Bin_3 (11 - 15), then the histogram corresponding to our data would look something like this: Bins Frequency Bin_1 (1 - 5) 3 Bin_2 (6 - 10) 5 Bin_3 (11 - 15) 3 What this histogram data tells us is that we have three values between 1 and 5, five between 6 and 10, and three again between 11 and 15. Note that it doesn't tell us what the values are, just that some n values exist in a given bin. A more familiar visual representation of the histogram in discussion is shown as follows: As you can see, the bins have been plotted along the x axis and their corresponding frequencies along the y axis. Now, in the context of images, how is a histogram computed? Well, it's not that difficult to deduce. Since the data that we have comprise pixel intensity values, an image histogram is computed by plotting a histogram using the intensity values of all its constituent pixels. What this essentially means is that the sequence of pixel intensity values in our image becomes the data. Well, this is in fact the simplest kind of histogram that you can compute using the information available to you from the image. Now, coming back to image histograms, there are some basic terminologies (pertaining to histograms in general) that you need to be aware of before you can dip your hands into code. We have explained them in detail here: Histogram size: The histogram size refers to the number of bins in the histogram. Range: The range of a histogram is the range of data that we are dealing with. The range of data as well as the histogram size are both important parameters that define a histogram. Dimensions: Simply put, dimensions refer to the number of the type of items whose values we aggregate in the histogram bins. For example, consider a grayscale image. We might want to construct a histogram using the pixel intensity values for such an image. This would be an example of a single-dimensional histogram because we are just interested in aggregating the pixel intensity values and nothing else. The data, in this case, is spread over a range of 0 to 255. On account of being one-dimensional, such histograms can be represented graphically as 2D plots—one-dimensional data (pixel intensity values) being plotted on the x axis (in the form of bins) along with the corresponding frequency counts along the y axis. We have already seen an example of this before. Now, imagine a color image with three channels: red, green, and blue. Let's say that we want to plot a histogram for the intensities in the red and green channels combined. This means that our data now becomes a pair of values (r, g). A histogram that is plotted for such data will have a dimensionality of 2. The plot for such a histogram will be a 3D plot with the data bins covering the x and y axes and the frequency counts plotted along the z axis. Now that we have discussed the theoretical aspects of image histograms in detail, let's start thinking along the lines of code. We will start with the simplest (and in fact the most ubiquitous) design of image histograms. The range of our data will be from 0 to 255 (both inclusive), which means that all our data points will be integers that fall within the specified range. Also, the number of data points will equal the number of pixels that make up our input image. The simplicity in design comes from the fact that we fix the size of the histogram (the number of bins) as 256. Now, take a moment to think about what this means. There are 256 different possible values that our data points can take and we have a separate bin corresponding to each one of those values. So such an image histogram will essentially depict the 256 possible intensity values along with the counts of the number of pixels in the image that are colored with each of the different intensities. Before taking a peek at what OpenCV has to offer, let's try to implement such a histogram on our own! We define a function named computeHistogram() that takes the grayscale image as an input argument and returns the image histogram. From our earlier discussions, it is evident that the histogram must contain 256 entries (for the 256 bins): one for each integer between 0 and 255. The value stored in the histogram corresponding to each of the 256 entries will be the count of the image pixels that have a particular intensity value. So, conceptually, we can use an array for our implementation such that the value stored in the histogram [ i ] (for 0≤i≤255) will be the count of the number of pixels in the image having the intensity of i. However, instead of using a C++ array, we will comply with the rules and standards followed by OpenCV and represent the histogram as a Mat object. We have already seen that a Mat object is nothing but a multidimensional array store. The implementation is outlined in the following code snippet: Mat computeHistogram(Mat input_image) { Mat histogram = Mat::zeros(256, 1, CV_32S); for (int i = 0; i < input_image.rows; ++i) { for (int j = 0; j < input_image.cols; ++j) { int binIdx = (int) input_image.at<uchar>(i, j); histogram.at<int>(binIdx, 0) += 1; } } return histogram; } As you can see, we have chosen to represent the histogram as a 256-element-column-vector Mat object. We iterate over all the pixels in the input image and keep on incrementing the corresponding counts in the histogram (which had been initialized to 0). As per our description of the image histogram properties, it is easy to see that the intensity value of any pixel is the same as the bin index that is used to index into the appropriate histogram bin to increment the count. Having such an implementation ready, let's test it out with the help of an actual image. The following code demonstrates a main() function that reads an input image, calls the computeHistogram() function that we have defined just now, and displays the contents of the histogram that is returned as a result: int main() { Mat input_image = imread("/home/samyak/Pictures/lena.jpg", IMREAD_GRAYSCALE); Mat histogram = computeHistogram(input_image); cout << "Histogram...n"; for (int i = 0; i < histogram.rows; ++i) cout << i << " : " << histogram.at<int>(i, 0) << "n"; return 0; } We have used the fact that the histogram that is returned from the function will be a single column Mat object. This makes the code that displays the contents of the histogram much cleaner. Histograms in OpenCV We have just seen the implementation of a very basic and minimalistic histogram using the first principles in OpenCV. The image histogram was basic in the sense that all the bins were uniform in size and comprised only a single pixel intensity. This made our lives simple when we designed our code for the implementation; there wasn't any need to explicitly check the membership of a data point (the intensity value of a pixel) with all the bins of our histograms. However, we know that a histogram can have bins whose sizes span more than one. Can you think of the changes that we might need to make in the code that we had written just now to accommodate for bin sizes larger than 1? If this change seems doable to you, try to figure out how to incorporate the possibility of non-uniform bin sizes or multidimensional histograms. By now, things might have started to get a little overwhelming to you. No need to worry. As always, OpenCV has you covered! The developers at OpenCV have provided you with a calcHist() function whose sole purpose is to calculate the histograms for a given set of arrays. By arrays, we refer to the images represented as Mat objects, and we use the term set because the function has the capability to compute multidimensional histograms from the given data: Mat computeHistogram(Mat input_image) { Mat histogram; int channels[] = { 0 }; int histSize[] = { 256 }; float range[] = { 0, 256 }; const float* ranges[] = { range }; calcHist(&input_image, 1, channels, Mat(), histogram, 1, histSize, ranges, true, false); return histogram; } Before we move on to an explanation of the different parameters involved in the calcHist() function call, I want to bring your attention to the abundant use of arrays in the preceding code snippet. Even arguments as simple as histogram sizes are passed to the function in the form of arrays rather than integer values, which at first glance seem quite unnecessary and counter-intuitive. The usage of arrays is due to the fact that the implementation of calcHist() is equipped to handle multidimensional histograms as well, and when we are dealing with such multidimensional histogram data, we require multiple parameters to be passed, one for each dimension. This would become clearer once we demonstrate an example of calculating multidimensional histograms using the calcHist() function. For the time being, we just wanted to clear the immediate confusion that might have popped up in your minds upon seeing the array parameters. Here is a detailed list of the arguments in the calcHist() function call: Source images Number of source images Channel indices Mask Dimensions (dims) Histogram size Ranges Uniform flag Accumulate flag The last couple of arguments (the uniform and accumulate flags) have default values of true and false, respectively. Hence, the function call that you have seen just now can very well be written as follows: calcHist(&input_image, 1, channels, Mat(), histogram, 1, histSize, ranges); Summary Thus in this article we have successfully studied fundamentals of using histograms in OpenCV for image processing. Resources for Article: Further resources on this subject: Remote Sensing and Histogram [article] OpenCV: Image Processing using Morphological Filters [article] Learn computer vision applications in Open CV [article]

0
0
22139

Packt

12 Oct 2016

7 min read

Moving from Windows to Appliance

Packt

12 Oct 2016

7 min read

In this article by Daniel Langenhan, author of VMware vRealize Orchestrator Cookbook, Second Edition, will show you how to move an existing Windows Orchestrator installation to the appliance. With vRO 7 the Windows install of Orchestrator doesn't exist anymore. (For more resources related to this topic, see here.) Getting ready We need an Orchestrator installed on windows. Download the same version of the Orchestrator appliance as you have installed in the Windows version. If needed upgrade the Windows version to the latest possible one. How to do it... There are three ways, using the migration tool, repointing to an external database or export/import the packages. Migration tool There is a migration tool that comes with vRO7 that allows you to pack up your vRO5.5 or 6.x install and deploy it into a vRO7. The migration tool works on Windows and Linux. It collects the configuration, the plug-ins as well as their configuration certificates, and licensing into a file. Follow these steps to use the migration tool: Deploy a new vRO7 Appliance. Log in to your Windows Orchestrator OS. Stop the VMware vCenter Orchestrator Service (Windows services). Open a Web browser and log in to your new vRO7 - control center and then go to Export/Import Configuration. Select Migrate Configuration and click on the here link. The link points to: https://[vRO7]:8283/vco-controlcenter/api/server/migration-tool . Stop the vRO7 Orchestrator service. Unzip the migration-tool.zip and copy the subfolder called migration‑cli into the Orchestrator director, for example, C:Program FilesVMwareInfrastructureOrchestratormigration-clibin. Open a command prompt. If you have a Java install, make sure your path points to it. Try java ‑version. If that works continue, if not, do the following: Set the PATH environment variable to the Java install that comes with Orchestrator, set PATH=%PATH%;C:Program FilesVMwareInfrastructureOrchestratorUninstall_vCenter Orchestratoruninstall-jrebin CD to the directory ..Orchestratormigration-clibin. Execute the command vro-migrate.bat export. There may be errors showing about SLF4J, you can ignore those. In the main directory (..Orchestrator) you should now find an orchestrator‑config‑export‑VC55‑[date].zip file. Go back to the Web browser and upload the ZIP file into Migration Configuration by clicking on Browse and select the file. Click on Import. You now see what can be imported. You can unselect item you wish not to migrate. Click Finish Migration. Restart the Orchestrator service. Check the settings. External database If you have an external database things are pretty easy. For using the initial internal database please see the additional steps in the There's more section of this recipe. Backup the external database. Connect to the Windows Orchestrator Configurator. Write down all the plugins you have installed as well as their version. Shutdown the Windows version and deploy the Appliance, this way you can use the same IP and hostname if you want. Login to the Appliance version's Configurator. Stop the Orchestrator service Install all plugins you had in the Windows version. Attach the external database. Make sure that all trusted SSL certificates are still there, such as vCenter and SSO. Check the authentication if it is still working. Use the test login. Check your licensing. Force a plugin reinstall (Troubleshooting | Reinstall the plug-ins when the server starts). Start the Orchestrator service and try to log in. Make a complete sanity check. Package transfer This is the method that will only pull your packages across. This the only easy method to use when you are transitioning between different databases, such as between MS SQL and PostgreSQL. Connect to your Windows version Create a package of all the workflows, action, and other items you need. Shutdown Windows and deploy the Appliance. Configure the Appliance with DB, Authentication and all the plugins you previously had. Import the package. How it works... Moving from the Windows version of Orchestrator to the Appliance version isn't such a big thing. Worst case scenario is using the packaging transfer. The only really important thing is to use the same version of the Windows Orchestrator as the Appliance version. You can download a lot of old versions including 5.5 from http://www.vmware.com/in.html. If you can't find the same version, upgrade your existing vCenter Orchestrator to one you can download. After you transferred the data to the appliance you need to make sure that everything works correctly and then you can upgrade to vRO7. There's more... When you just run Orchestrator from your Windows vCenter installation and didn't configure an external database then Orchestrator uses the vCenter database and mixes the Orchestrator tables with the vCenter tables. In order to only export the Orchestrator ones, we will use the MS SQL Server Management Studio (free download from www.microsoft.com called Microsoft SQL Server 2008 R2 RTM). To transfer the only the Orchestrator database tables from the vCenter MS-SQL to an external SQL do the following: Stop the VMware vCenter Orchestrator Service (Windows Services) on your Windows Orchestrator. Start the SQL Server Management Studio on your external SQL server. Connect to the vCenter DB. For SQL Express use: [vcenter]VIM_SQLEXP with Windows Authentication. Right-click on your vCenter Database (SQL Express: VIM_VCDB) and select Tasks | Export Data. In the wizard, select your source, which should be the correct one already and click Next. Choose SQL Server Native Client 10.0 and enter the name of your new SQL server. Click on New to create a new database on that SQL server (or use an empty one you created already. Click Next. Select Copy data from one or more tables or views and click Next. Now select every database which starts with VMO_ and then click Next. Select Run immediately and click Finish. Now you have the Orchestrator database extracted as an external database. You still need to configure a user and rights. Then proceed with the section External database in this recipe. Orchestrator client and 4K display scaling This recipe shows a hack how to make the Orchestrator client scale on 4K displays. Getting ready We need to download the program Resource Tuner (http://www.restuner.com/) the trial version will work, however, consider buying it if it works for you. You need to know the path to your Java install, this should be something like: C:Program Files (x86)Javajre1.x.xxbin How to do it... Before you start…. Please be careful as this impacts your whole Java environment. This worked for me very well with Java 1.8.0_91-b14. Download and install Resource Tuner. Run Resource Tuner as administrator. Open the file javaws.exe in your Java directory. Expand manifest and the click on the first entry (the name can change due to localization). Look for the line <dpiAware>true</dpiAware>. Exchange the true for a false Save end exit. Repeat the same for all the other java*.exe in the same directory as well as j2launcher.exe. Start the Client.jnlp (the file that downloads when you start the web application). How it works... In Windows 10 you can set the scaling of applications when you are using high definition monitors (4K displays). What you are doing is telling Java that it is not DPI aware, meaning it the will use the Windows 10 default scaler, instead of an internal scaler. There's more... For any other application such as Snagit or Photoshop I found that this solution works quite well: http://www.danantonielli.com/adobe-app-scaling-on-high-dpi-displays-fix/. Summary In this article we discussed about moving an existing Windows Orchestrator installation to the appliance and a hack on how to make the Orchestrator client scale on 4K displays. Resources for Article: Further resources on this subject: Working with VMware Infrastructure [article] vRealize Automation and the Deconstruction of Components [article] FAQ on Virtualization and Microsoft App-V [article]

0
0
7598

article-image-asynchronous-programming-f

Packt

12 Oct 2016

15 min read

Asynchronous Programming in F#

Packt

12 Oct 2016

15 min read

0
0
13428

article-image-solving-nlp-problem-keras-part-1

Sasank Chilamkurthy

12 Oct 2016

5 min read

Solving an NLP Problem with Keras, Part 1

Sasank Chilamkurthy

12 Oct 2016

5 min read

In a previous two-part post series on Keras, I introduced Convolutional Neural Networks(CNNs) and the Keras deep learning framework. We used them to solve a Computer Vision (CV) problem involving traffic sign recognition. Now, in this two-part post series, we will solve a Natural Language Processing (NLP) problem with Keras. Let’s begin. The Problem and the Dataset The problem we are going to tackle is Natural Language Understanding. The aim is to extract the meaning of speech utterances. This is still an unsolved problem. Therefore, we can break this problem into a solvable practical problem of understanding the speaker in a limited context. In particular, we want to identify the intent of a speaker asking for information about flights. The dataset we are going to use is Airline Travel Information System (ATIS). This dataset was collected by DARPA in the early 90s. ATIS consists of spoken queries on flight related information. An example utterance is I want to go from Boston to Atlanta on Monday. Understanding this is then reduced to identifying arguments like Destination and Departure Day. This task is called slot-filling. Here is an example sentence and its labels. You will observe that labels are encoded in an Inside Outside Beginning (IOB) representation. Let’s look at the dataset: |Words | Show | flights | from | Boston | to | New | York| today| |Labels| O | O | O |B-dept | O|B-arr|I-arr|B-date| The ATIS official split contains 4,978/893 sentences for a total of 56,590/9,198 words (average sentence length is 15) in the train/test set. The number of classes (different slots) is 128, including the O label (NULL). Unseen words in the test set are encoded by the <UNK> token, and each digit is replaced with string DIGIT;that is,20 is converted to DIGITDIGIT. Our approach to the problem is to use: Word embeddings Recurrent neural networks I'll talk about these briefly in the following sections. Word Embeddings Word embeddings map words to a vector in a high-dimensional space. These word embeddings can actually learn the semantic and syntactic information of words. For instance, they can understand that similar words are close to each other in this space and dissimilar words are far apart. This can be learned either using large amounts of text like Wikipedia, or specifically for a given problem. We will take the second approach for this problem. As an illustation, I have shown here the nearest neighbors in the word embedding space for some of the words. This embedding space was learned by the model that we’ll define later in the post: sunday delta california boston august time car wednesday continental colorado nashville september schedule rental saturday united florida toronto july times limousine friday american ohio chicago june schedules rentals monday eastern georgia phoenix december dinnertime cars tuesday northwest pennsylvania cleveland november ord taxi thursday us north atlanta april f28 train wednesdays nationair tennessee milwaukee october limo limo saturdays lufthansa minnesota columbus january departure ap sundays midwest michigan minneapolis may sfo later Recurrent Neural Networks Convolutional layers can be a great way to pool local information, but they do not really capture the sequentiality of data. Recurrent Neural Networks (RNNs) help us tackle sequential information like natural language. If we are going to predict properties of the current word, we better remember the words before it too. An RNN has such an internal state/memory that stores the summary of the sequence it has seen so far. This allows us to use RNNs to solve complicated word tagging problems such as Part Of Speech (POS) tagging or slot filling, as in our case. The following diagram illustrates the internals of RNN: Source: Nature RNN Let's briefly go through the diagram: Is the input to the RNN. x_1,x_2,...,x_(t-1),x_t,x_(t+1)... Is the hidden state of the RNN at the step. st This is computed based on the state at the step. t-1 As st=f(Uxt+Ws(t-1)) Here f is a nonlinearity such astanh or ReLU. ot Is the output at the step. t Computed as:ot=f(Vst)U,V,W Are the learnable parameters of RNN. For our problem, we will pass a word embeddings’ sequence as the input to the RNN. Putting it all together Now that we've setup the problem and have an understanding of the basic blocks, let's code it up. Since we are using the IOB representation for labels, it's not simpleto calculate the scores of our model. We therefore use the conlleval perl script to compute the F1 Scores. I've adapted the code from here for the data preprocessing and score calculation. The complete code is available at GitHub: $ git clone https://github.com/chsasank/ATIS.keras.git $ cd ATIS.keras I recommend using jupyter notebook to run and experiment with the snippets from the tutorial. $ jupyter notebook Conclusion In part 2, we will load the data using data.load.atisfull(). We will also define the Keras model, and then we will train the model. To measure the accuracy of the model, we’ll use model.predict_on_batch() and metrics.accuracy.conlleval(). And finally, we will improve our model to achieve better results. About the author Sasank Chilamkurthy works at Fractal Analytics. His work involves deep learning on medical images obtained from radiology and pathology. He is mainly interested in computer vision.

0
0
4638

article-image-reactive-python-asynchronous-programming-rescue-part-2

Xavier Bruhiere

10 Oct 2016

5 min read

Reactive Python - Asynchronous programming to the rescue, Part 2

Xavier Bruhiere

10 Oct 2016

5 min read

This two-part series explores asynchronous programming with Python using Asyncio. In Part 1 of this series, we started by building a project that shows how you can use Reactive Python in asynchronous programming. Let’s pick it back up here by exploring peer-to-peer communication and then just touching on service discovery before examining the streaming machine-to-machine concept. Peer-to-peer communication So far we’ve established a websocket connection to process clock events asynchronously. Now that one pin swings between 1's and 0's, let's wire a buzzer and pretend it buzzes on high states (1) and remains silent on low ones (0). We can rephrase that in Python, like so: # filename: sketches.py import factory class Buzzer(factory.FactoryLoop): """Buzz on light changes.""" def setup(self, sound): # customize buzz sound self.sound = sound @factory.reactive async def loop(self, channel, signal): """Buzzing.""" behavior = self.sound if signal == '1' else '...' self.out('signal {} received -> {}'.format(signal, behavior)) return behavior So how do we make them to communicate? Since they share a common parent class, we implement a stream method to send arbitrary data and acknowledge reception with, also, arbitrary data. To sum up, we want IOPin to use this API: class IOPin(factory.FactoryLoop): # [ ... ] @protocol.reactive async def loop(self, channel, msg): # [ ... ] await self.stream('buzzer', bits_stream) return 'acknowledged' Service discovery The first challenge to solve is service discovery. We need to target specific nodes within a fleet of reactive workers. This topic, however, goes past the scope of this post series. The shortcut below will do the job (that is, hardcode the nodes we will start), while keeping us focused on reactive messaging. # -*- coding: utf-8 -*- # vim_fenc=utf-8 # # filename: mesh.py """Provide nodes network knowledge.""" import websockets class Node(object): def __init__(self, name, socket, port): print('[ mesh ] registering new node: {}'.format(name)) self.name = name self._socket = socket self._port = port def uri(self, path): return 'ws://{socket}:{port}/{path}'.format(socket=self._socket, port=self._port, path=path) def connection(self, path=''): # instanciate the same connection as `clock` method return websockets.connect(self.uri(path)) # TODO service discovery def grid(): """Discover and build nodes network.""" # of course a proper service discovery should be used here # see consul or zookkeeper for example # note: clock is not a server so it doesn't need a port return [ Node('clock', 'localhost', None), Node('blink', 'localhost', 8765), Node('buzzer', 'localhost', 8765 + 1) ] Streaming machine-to-machine chat Let's provide FactoryLoop with the knowledge of the grid and implement an asynchronous communication channel. # filename: factory.py (continued) import mesh class FactoryLoop(object): def __init__(self, *args, **kwargs): # now every instance will know about the other ones self.grid = mesh.grid() # ... def node(self, name): """Search for the given node in the grid.""" return next(filter(lambda x: x.name == name, self.grid)) async def stream(self, target, data, channel): self.out('starting to stream message to {}'.format(target)) # use the node webscoket connection defined in mesh.py # the method is exactly the same as the clock async with self.node(target).connection(channel) as ws: for partial in data: self.out('> sending payload: {}'.format(partial)) # websockets requires bytes or strings await ws.send(str(partial)) self.out('< {}'.format(await ws.recv())) We added a bit of debugging lines to better understand how the data flows through the network. Every implementation of the FactoryLoop can both react to events and communicate with other nodes it is aware of. Wrapping up Time to update arduino.py and run our cluster of three reactive workers in three @click.command()# [ ... ]def main(sketch, **flags): # [ ... ] elif sketch == 'buzzer': sketchs.Buzzer(sound='buzz buzz buzz').run(flags['socket'], flags['port']) Launch three terminals or use a tool such as foreman to spawn multiple processes. Either way, keep in mind that you will need to track the scripts output. way, keep in mind that you will need to track the scripts output. $ # start IOPin and Buzzer on the same ports we hardcoded in mesh.py $ ./arduino.py buzzer --port 8766 $ ./arduino.py iopin --port 8765 $ # now that they listen, trigger actions with the clock (targetting IOPin port) $ ./arduino.py clock --port 8765 [ ... ] $ # Profit ! We just saw one worker reacting to a clock and another reacting to randomly generated events. The websocket protocol allowed us to exchange streaming data and receive arbitrary responses, unlocking sophisticated fleet orchestration. While we limited this example to two nodes, a powerful service discovery mechanism could bring to life a distributed network of microservices. By completing this post series, you should now have a better understanding of how to use Python with Asyncio for asynchronous programming. About the author Xavier Bruhiere is a lead developer at AppTurbo in Paris, where he develops innovative prototypes to support company growth. He is addicted to learning, hacking on intriguing hot techs (both soft and hard), and practicing high-intensity sports.

0
0
5890

article-image-modern-natural-language-processing-part-3

Brian McMahan

07 Oct 2016

8 min read

Modern Natural Language Processing – Part 3

Brian McMahan

07 Oct 2016

8 min read

In the previous two posts, I walked through how to preprocess raw data to a cleaner version and then turn that into a form which can be used in a machine learning experiment. I also discussed how you can set up a modular infrastructure so changing components isn't a hassle and your workflow is streamlined. In this final post in the series, I will outline a language model and discuss the modeling choices. I will outline the algorithms needed to both decode from the language model and to sample from it. Note that if you want to do a sequence labeling task instead of a language modeling task, the outputs must become your sequence labels, but the inputs are your sentences. The Language Model A language model has one goal: do not be surprised by the next token given all previous tokens. This translates into trying to maximize the probability of the next word given the previously seen words. It is useful to think of the 'shape' of our data's matrices at each step in our model. Specifically, though, our model will go through the following steps: Take as input the data being served from our server 2-dimensional matrices: (batch, sequence_length) Embed it using the matrix we constructed 3-dimensional tensors: (batch, sequence_length, embedding_size) Use any RNN variant to sequentially go through our data This will assume each example is on the batch dimension and time on the sequence_length dimension It will take vectors of size embedding_size and perform operations on them. Using the RNN output, we will apply a Dense layer, which will perform a classification back to our vocabulary space. This is our target. 0. Imports from keras.layers import Input, Embedding, LSTM, Dropout, Dense, TimeDistributed from keras.engine import Model from keras.optimizers import Adam from keras.callbacks import ModelCheckpoint class TrainingModel(object): def__init__(self, igor): self.igor = igor def make(self): ### code below goes here 1. Input Defining an entry point into Keras is very similar to the other layers. The only difference is that you have to give it information about the shape of the input data. I tend to give it more information than it needs—the batch size in addition to the sequence length—because omitting the batch size is useful when you watch variable batch sizes, but it serves no purpose otherwise. It also quells any paranoid worries that the model will break because it got the shape wrong at some point. words_in = Input(batch_shape=(igor.batch_size, igor.sequence_length), dtype='int32') 2. Embed This is where we can use the embeddings we had previously calculated. Note the mask_zero flag. This is set to True so that the Layer will calculate the mask—where each position in the input tensor is equal to 0. The Layer, in accordance to Keras' underlying functionality, is then pushed through the network to be used in final calculations. emb_W = self.igor.embeddings.astype(K.floatx()) words_embedded = Embedding(igor.vocab_size, igor.embedding_size, mask_zero=True, weights=[emb_W])(words_in) 3. Recurrence word_seq = LSTM(igor.rnn_size, return_sequences=True)(words_embedded) 4. Classification predictions = TimeDistributed(Dense(igor.vocab_size, activation='softmax'))(word_seq) 5. Compile Model Now, we can compile the model. Keras makes this simple: specify the inputs, outputs, loss, optimizer and metrics. I have omitted the custom metrics for now. I will bring them back up in evaluations below. optimizer = Adam(igor.learning_rate) model = Model(input=words_in, output=predictions) model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=custom_metrics) All together ### model.py from keras.layers import Input, Embedding, LSTM, Dropout, Dense, TimeDistributed from keras.engine import Model from keras.optimizers import Adam from keras.callbacks import ModelCheckpoint class TrainingModel(object): def__init__(self, igor): self.igor = igor def make(self): words_in = Input(batch_shape=(igor.batch_size, igor.sequence_length), dtype='int32') words_embedded = Embedding(igor.vocab_size, igor.embedding_size, mask_zero=True)(words_in) word_seq = LSTM(igor.rnn_size, return_sequences=True)(words_embedded) predictions = TimeDistributed(Dense(igor.vocab_size, activation='softmax'))(word_seq) optimizer = Adam(igor.learning_rate) self.model = Model(input=words_in, output=predictions) self.model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=custom_metrics) Training The driver is a useful part of the pipeline. Not only does it give a convenient entry point to the training, but it also allows you to easily switch between training, debugging, and testing. ### driver.py if__name__ == "__main__": import sys if__name__ == "__main__": import sys igor = Igor.from_file(sys.argv[2]) if sys.argv[1] == "train": igor.prep() next(igor.train_gen(forever=True)) model = TrainingModel(igor) model.make() try: model.train() exceptKeyboardInterruptas e: # safe exitting stuff here. # perhaps, model save. print("death by keyboard") Train Function ### model.py class TrainingModel(object): # ... def train(self): igor = self.igor train_data = igor.train_gen(forever=True) dev_data = igor.dev_gen(forever=True) callbacks = [ModelCheckpoint(filepath=igor.checkpoint_filepath, verbose=1, save_best_only=True)] self.model.fit_generator(generator=train_data, samples_per_epoch=igor.num_train_samples, nb_epoch=igor.num_epochs, callbacks=callbacks, verbose=1, validation_data=dev_data, nb_val_samples=igor.num_dev_samples) Failure to learn There are many ways that learning can fail. Stanford's CS231N course has a few things on this. Additionally, here are many Quora and Stack Overflow posts on debugging the learning process. Evaluating Language model evaluations aim to quantify how well the model captures the signal and anticipates the noise. For this, there are two standard metrics. The first is an aggregate of the probabilities of the model: Log Likelihood or Negative Log Likelihood. I will use Negative Log Likelihood (NLL) because it is more interpretable. The other is Perplexity. This is very related to NLL and originates from information theory as a way to quantify the information gain of the model's learned distribution to the empirical distribution of the test dataset. It is usually interpreted as the uniform uncertainty left in the data. At the time of writing this blog, masks in Keras currently do not get used in the accuracy calculations. But this will soon be implemented. Until then, there is a Keras fork that has these implemented. It can be found here. The custom_metrics from above would then simply be ['accuracy', 'perplexity']. Decoding Decoding is the process by which you infer a sequence of labels from a sequence of inputs. The idea and algorithms for it come from the signal processing research in which a noisy channel is emitting a signal and the task is to recover the signal. Typically, there is an encoder at one end that provides information so that a decoder at the other end can decode it. This means sequentially deciding which discrete token each part of the signal represents. In NLP, decoding is essentially the same task. In a sequence, the history of tokens can influence the likelihood of future tokens, so naive decoding by selecting the most likely token at each time step may not be the optimal sequence. The alternative solution, enumerating all possible sequences, is prohibitively expensive because of the combinatoric explosion of paths. Luckily, there are dynamic programming algorithms, such as the Viterbi algorithm, which solve such issues. The idea behind Viterbi is simple: Obtain the maximum likelihood classification at the last time step. At each time t, there is a set of k hypotheses that can be True. By looking at the previous time steps k hypotheses scores and the cost of transition from each to the current k possible, you can compute the best path so far for each of the k hypotheses. Thus, at every time step, you can do a linear update and ensure the optimal set of paths. At every time step, the backpointers were cached and used at the final time step to decode the path through the k states. Viterbi has its own limitations, such as also becoming expensive when the discrete hypothesis space is large. There are additional approximations, such as Beam Search, which uses a subset of the viterbi paths (selected by score at every time step). Several tasks are accomplished with this decoding. Sampling to produce a sentence (or caption an image) is typically done with a Beam search. Additionally, labeling each word in a sentence (such as part of speech tagging or entity tagging) is done with a sequential decoding procedure. Conclusion Now that you have completed this three part series, you can start to run your own NLP experiments! About the author Brian McMahan is in his final year of graduate school at Rutgers University, completing a PhD in computer science and an MS in cognitive psychology. He holds a BS in cognitive science from Minnesota State University, Mankato. At Rutgers, Brian investigates how natural language and computer vision can be brought closer together with the aim of developing interactive machines that can coordinate in the real world. His research uses machine learning models to derive flexible semantic representations of open-ended perceptual language.

0
0
1379

How-To Tutorials

Masayuki Takagi

07 Oct 2016

11 min read

How to start Chainer

Masayuki Takagi

07 Oct 2016

11 min read

0
0
4495

How-To Tutorials

article-image-introduction-neural-networks-chainer-part-3

Hiroyuki Vincent

06 Oct 2016

7 min read

Introduction to Neural Networks with Chainer – Part 3

Hiroyuki Vincent

06 Oct 2016

7 min read

In this final post of this three part series we cover the optimizer, batches, complex and networks and we also discuss running the code of our example on the GPU. Let’s continue from where we left off in Part 2 with training the model to go over optimizers. Optimizers The optimizer module chainer.optimizer is responsible for orchestrating the parameter updates minimizing the loss. It is instantiated with a class that inherits the base class chainer.optimizer.Optimizer. Once instantiated, it needs to be set up by calling its setup() method, passing it a Link to optimize, that in turn contains all the Chainer variables that are to be trained. Remember that we give it the whole model, an instance of the chainer.link.Chain class which is a subclass of the Chainer chainer.link.Link. We can then call the update method every time we want to optimize the parameters in the training loop. The update method can be invoked by passing a loss function and the arguments to it, in this case the input value and the target value. One motivation for defining the __call__ method as the loss function is precisely so that the model instance can be passed to the optimzer here. Note that the loss is both stored in the class instance and returned. It is returned so that the model instance can be directly passed to the optimizer, which expects a loss function. But, the loss is also stored in the model itself so that it can be read by the training loop to compute the average loss over the course of an epoch. The chainer.optimizers.SGD used in this example inherits from chainer.optimizer.GradientMethod which in turn inherits from the base optimizer class. The SGD, or Stochastic Gradient Descent optimizer performs a parameter optimization in each update much similar to the one we did in the previous article when performing a gradient descent. You have probably noted that it takes the learning rate as an argument in its constructor. What the update method actually does is that it first resets the gradients for the variables in the model, computes the loss, runs the backward() method on the parameters and then updates the parameters using the algorithm defined by the optimizer instance, a simple SGD in our case. Other optimization algorithms in the framework include Momentum SGD, AdaGrad, AdaDelta, Adam and RMSProp. To train the model in this example using AdaGrad instead of SGD, you only need to change the instantiation of the optimizer from optimizer = optimizers.SGD(lr=learning_rate) to optimizer = optimizers.AdaGrad(lr=learning_rate, eps=1e-08). The arguments differ from optimizer to optimizer. AdaGrad also takes the epsilon smoothing term for instance, usually a small number that in Chainer defaults to 1e-08. Always in Batches Each time you feed the model with training data (invoke the __call__ method directly on the model instance or use optimizer.update()), the data being passed needs to be in a mini-batch format. This means that you cannot feed the model above with data such as x = Variable(np.array([1, 2, 3], dtype=np.float32)) because you want to do online training. This would result in the following error. ... Invalid operation is performed in: LinearFunction (Forward) Expect: in_types[0].ndim >= 2 Actual: 1 < 2 However, wrapping the array in another list x = Variable(np.array([[1, 2, 3]], dtype=np.float32)) would work since you simply made a mini-batch of size one, which would yield the same results as online training. You can of course train the model using regular batch training by passing the whole dataset to the model to perform one update per epoch. You might have noticed that the loss being returned is multiplied by the size of the batch. This is because the Chainer loss functions such as chainer.functions.mean_squared_error returns the average loss over the given mini-batch. In order to compute the average loss over the complete dataset, we keep track of an accumulated loss over each individual sample. If you don't want to do that, you could simply pass the whole dataset as a single batch into the model at the end of an epoch to compute the loss. Extending to More Complex Networks Adding activation functions or noise such as dropout can be done by adding a single function calls in the model definition. They are all part of the chainer.functions module. If you'd like to add 50% dropout to the hidden layer in our autoencoder, you'd change the first forward pass line of code from h = self.l1(x) to h = F.dropout(self.l1(x), ratio=0.5). Since this is such a small network, you will see that the loss increases quite significantly. Adding a ReLU activation function would look like this, F.relu(self.l1(x)). These methods can be applied to other Links as well and not just the linear connection that we've used in the autoencoder. Creating other types of networks such as convolutional neural networks or recurrent ones are done by changing the layer definitions in the Chain constructor. What you need to be careful of when training is mainly to make sure that the dimensions of the Chainer variables that are passed between the layers and especially in the input layer match. If the first layer of a network is a convolutional layer, that is chainer.links.Convolution2D, the input dimensions is slightly different from this autoencoder example since there is an additional channel, width and height dimensionality to the data. Remember the data is still passed in batches. Running the Same Code on The GPU Assuming that CUDA is installed, you only need to add a few lines of code in order to train the model on the GPU. What you only need to do is to copy the Chainer Links and the training data to the GPU and you can run the same training code. Since the CuPy interface in Chainer implements a subset of the NumPy interface we could write it nicely in the following way. import numpy as np from chainer import cuda from chainer.cuda import cupy as cp xp = None model = Autoencoder() if device_id > 0: cuda.check_cuda_available() # A CUDA installation was found, set the default device cuda.get_device(device_id).use() # Copy the Links to the default device model.to_gpu() xp = cp else: xp = np # Replace all old occurrences of np with xp The device_id in this case identifies a GPU. If you have installed CUDA correctly, you should be able to list all devices with the nvidia-smi command in the CLI to see exactly which devices are available. The id can of course be hardcoded in the code itself but could for example be passed as an argument to the Python script. Depending on the specified id and the availability, the variable xp is set to NumPy or CuPy accordingly. What you need to change in the rest of the code is simply replacing all previous occurrences of np with xp. Saving and Loading Training Data The trained parameters can be written to files for persistence. This is natively supported by the framework using any of the modules in the chainer.serializers. It is also possible to load parameters into existing models and their layers in the same manner. This is useful when you need to stop the training or want to take snapshots during the training process. Summary Defining and training neural networks with Chainer is intuitive and requires little code. It is easy to maintain and experiment with various hyper parameters because of its design. In this second part of the series with Chainer, we implemented a neural network and trained it with randomly generated data and common patterns were introduced such as how to design the model and the loss function to demonstrate this fact. The network covered in this article along with its data is more or less a toy problem. But, hopefully you will try Chainer out on your own and experiment with what it's capable of. About the Author Hiroyuki Vincent Yamazaki is a graduate student at KTH, Royal Institute of Technology in Sweden, currently conducting research in convolutional neural networks at Keio University in Tokyo, partially using Chainer as a part of a double-degree programme. GitHub LinkedIn

0
0
1406

How-To Tutorials

article-image-getting-organized-npm-and-bower

Packt

06 Oct 2016

13 min read

Getting Organized with NPM and Bower

Packt

06 Oct 2016

13 min read

In this article by Philip Klauzinski and John Moore, the authors of the book Mastering JavaScript Single Page Application Development, we will learn about the basics of NMP and Bower. JavaScript was the bane of the web development industry during the early days of the browser-rendered Internet. Now, powers hugely impactful libraries such as jQuery, and JavaScript-rendered content (as opposed to server-side-rendered content) is even indexed by many search engines. What was once largely considered an annoying language used primarily to generate popup windows and alert boxes has now become, arguably, the most popular programming language in the world. (For more resources related to this topic, see here.) Not only is JavaScript now more prevalent than ever in frontend architecture, but it has become a server-side language as well, thanks to the Node.js runtime. We have also now seen the proliferation of document-oriented databases, such as MongoDB, which store and return JSON data. With JavaScript present throughout the development stack, the door is now open for JavaScript developers to become full-stack developers without the need to learn a traditional server-side language. Given the right tools and know-how, any JavaScript developer can create single page applications (SPAs) comprising entirely the language they know best, and they can do so using an architecture such as MEAN (MongoDB, Express, AngularJS, and Node.js). Organization is key to the development of any complex single page application. If you don't get organized from the beginning, you are sure to introduce an inordinate number of regressions to your app. The Node.js ecosystem will help you do this with a full suite of indispensable and open source tools, three of which we will discuss here. In this article, you will learn about: Node Package Manager The Bower front-end package manager What is Node Package Manager? Within any full-stack JavaScript environment, Node Package Manager (NPM) will be your go-to tool for setting up your development environment and managing server-side libraries. NPM can be used within both global and isolated environment contexts. We will first explore the use of NPM globally. Installing Node.js and NPM NPM is a component of Node.js, so before you can use it, you must install Node.js. You can find installers for both Mac and Windows at nodejs.org. Once you have Node.js installed, using NPM is incredibly easy and is done from the command-line interface (CLI). Start by ensuring you have the latest version of NPM installed, as it is updated more often than Node.js itself: $ npm install -g npm When using NPM, the -g option will apply your changes to your global environment. In this case, you want your version of NPM to apply globally. As stated previously, NPM can be used to manage packages both globally and within isolated environments. Therefore, we want essential development tools to be applied globally so that you can use them in multiple projects on the same system. On Mac and some Unix-based systems, you may have to run the npm command as the superuser (prefix the command with sudo) in order to install packages globally, depending on how NPM was installed. If you run into this issue and wish to remove the need to prefix npm with sudo, see docs.npmjs.com/getting-started/fixing-npm-permissions. Configuring your package.json file For any project you develop, you will keep a local package.json file to manage your Node.js dependencies. This file should be stored at the root of your project directory, and it will only pertain to that isolated environment. This allows you to have multiple Node.js projects with different dependency chains on the same system. When beginning a new project, you can automate the creation of the package.json file from the command line: $ npm init Running npm init will take you through a series of JSON property names to define through command-line prompts, including your app's name, version number, description, and more. The name and version properties are required, and your Node.js package will not install without them being defined. Several of the properties will have a default value given within parentheses in the prompt so that you may simply hit Enter to continue. Other properties will simply allow you to hit Enter with a blank entry and will not be saved to the package.json file or be saved with a blank value: name: (my-app) version: (1.0.0) description: entry point: (index.js) The entry point prompt will be defined as the main property in package.json and is not necessary unless you are developing a Node.js application. In our case, we can forgo this field. The npm init command may in fact force you to save the main property, so you will have to edit package.json afterward to remove it; however, that field will have no effect on your web app. You may also choose to create the package.json file manually using a text editor if you know the appropriate structure to employ. Whichever method you choose, your initial version of the package.json file should look similar to the following example: { "name": "my-app", "version": "1.0.0", "author": "Philip Klauzinski", "license": "MIT", "description": "My JavaScript single page application." } If you want your project to be private and want to ensure that it does not accidently get published to the NPM registry, you may want to add the private property to your package.json file and set it to true. Additionally, you may remove some properties that only apply to a registered package: { "name": "my-app", "author": "Philip Klauzinski", "description": "My JavaScript single page application.", "private": true } Once you have your package.json file set up the way you like it, you can begin installing Node.js packages locally for your app. This is where the importance of dependencies begins to surface. NPM dependencies There are three types of dependencies that can be defined for any Node.js project in your package.json file: dependencies, devDependencies, and peerDependencies. For the purpose of building a web-based SPA, you will only need to use the devDependencies declaration. The devDependencies ones are those that are required for developing your application, but not required for its production environment or for simply running it. If other developers want to contribute to your Node.js application, they will need to run npm install from the command line to set up the proper development environment. For information on the other types of dependencies, see docs.npmjs.com. When adding devDependencies to your package.json file, the command line again comes to the rescue. Let's use the installation of Browserify as an example: $ npm install browserify --save-dev This will install Browserify locally and save it along with its version range to the devDependencies object in your package.json file. Once installed, your package.json file should look similar to the following example: { "name": "my-app", "version": "1.0.0", "author": "Philip Klauzinski", "license": "MIT", "devDependencies": { "browserify": "^12.0.1" } } The devDependencies object will store each package as key-value pairs, in which the key is the package name and the value is the version number or version range. Node.js uses semantic versioning, where the three digits of the version number represent MAJOR.MINOR.PATCH. For more information on semantic version formatting, see semver.org. Updating your development dependencies You will notice that the version number of the installed package is preceded by a caret (^) symbol by default. This means that package updates will only allow patch and minor updates for versions above 1.0.0. This is meant to prevent major version changes from breaking your dependency chain when updating your packages to the latest versions. To update your devDependencies and save the new version numbers, you will enter the following from the command line. $ npm update --save-dev Alternatively, you can use the -D option as a shortcut for --save-dev: $ npm update -D To update all globally installed NPM packages to their latest versions, run npm update with the -g option: $ npm update -g For more information on semantic versioning within NPM, see docs.npmjs.com/misc/semver. Now that you have NPM set up and you know how to install your development dependencies, you can move on to installing Bower. Bower Bower is a package manager for frontend web assets and libraries. You will use it to maintain your frontend stack and control version chains for libraries such as jQuery, AngularJS, and any other components necessary to your app's web interface. Installing Bower Bower is also a Node.js package, so you will install it using NPM, much like you did with the Browserify example installation in the previous section, but this time you will be installing the package globally. This will allow you to run bower from the command line anywhere on your system without having to install it locally for each project. $ npm install -g bower You can alternatively install Bower locally as a development dependency so that you may maintain different versions of it for different projects on the same system, but this is generally not necessary. $ npm install bower --save-dev Next, check that Bower is properly installed by querying the version from the command line. $ bower -v Bower also requires the Git version control system (VCS) to be installed on your system in order to work with packages. This is because Bower communicates directly with GitHub for package management data. If you do not have Git installed on your system, you can find instructions for Linux, Mac, and Windows at git-scm.com. Configuring your bower.json file The process of setting up your bower.json file is comparable to that of the package.json file for NPM. It uses the same JSON format, has both dependencies and devDependencies, and can also be automatically created. $ bower init Once you type bower init from the command line, you will be prompted to define several properties with some defaults given within parentheses: ? name: my-app ? version: 0.0.0 ? description: My app description. ? main file: index.html ? what types of modules does this package expose? (Press <space> to? what types of modules does this package expose? globals ? keywords: my, app, keywords ? authors: Philip Klauzinski ? license: MIT ? homepage: http://gui.ninja ? set currently installed components as dependencies? No ? add commonly ignored files to ignore list? Yes ? would you like to mark this package as private which prevents it from being accidentally published to the registry? Yes These questions may vary depending on the version of Bower you install. Most properties in the bower.json file are not necessary unless you are publishing your project to the Bower registry, indicated in the final prompt. You will most likely want to mark your package as private unless you plan to register it and allow others to download it as a Bower package. Once you have created the bower.json file, you can open it in a text editor and change or remove any properties you wish. It should look something like the following example: { "name": "my-app", "version": "0.0.0", "authors": [ "Philip Klauzinski" ], "description": "My app description.", "main": "index.html", "moduleType": [ "globals" ], "keywords": [ "my", "app", "keywords" ], "license": "MIT", "homepage": "http://gui.ninja", "ignore": [ "**/.*", "node_modules", "bower_components", "test", "tests" ], "private": true } If you wish to keep your project private, you can reduce your bower.json file to two properties before continuing: { "name": "my-app", "private": true } Once you have the initial version of your bower.json file set up the way you like it, you can begin installing components for your app. Bower components location and the .bowerrc file Bower will install components into a directory named bower_components by default. This directory will be located directly under the root of your project. If you wish to install your Bower components under a different directory name, you must create a local system file named .bowerrc and define the custom directory name there: { "directory": "path/to/my_components" } An object with only a single directory property name is all that is necessary to define a custom location for your Bower components. There are many other properties that can be configured within a .bowerrc file. For more information on configuring Bower, see bower.io/docs/config/. Bower dependencies Bower also allows you to define both the dependencies and devDependencies objects like NPM. The distinction with Bower, however, is that the dependencies object will contain the components necessary for running your app, while the devDependencies object is reserved for components that you might use for testing, transpiling, or anything that does not need to be included in your frontend stack. Bower packages are managed using the bower command from the CLI. This is a user command, so it does not require super user (sudo) permissions. Let's begin by installing jQuery as a frontend dependency for your app: $ bower install jquery --save The --save option on the command line will save the package and version number to the dependencies object in bower.json. Alternatively, you can use the -S option as a shortcut for --save: $ bower install jquery -S Next, let's install the Mocha JavaScript testing framework as a development dependency: $ bower install mocha --save-dev In this case, we will use --save-dev on the command line to save the package to the devDependencies object instead. Your bower.json file should now look similar to the following example: { "name": "my-app", "private": true, "dependencies": { "jquery": "~2.1.4" }, "devDependencies": { "mocha": "~2.3.4" } } Alternatively, you can use the -D option as a shortcut for --save-dev: $ bower install mocha –D You will notice that the package version numbers are preceded by the tilde (~) symbol by default, in contrast to the caret (^) symbol, as is the case with NPM. The tilde serves as a more stringent guard against package version updates. With a MAJOR.MINOR.PATCH version number, running bower update will only update to the latest patch version. If a version number is composed of only the major and minor versions, bower update will update the package to the latest minor version. Searching the Bower registry All registered Bower components are indexed and searchable through the command line. If you don't know the exact package name of a component you wish to install, you can perform a search to retrieve a list of matching names. Most components will have a list of keywords within their bower.json file so that you can more easily find the package without knowing the exact name. For example, you may want to install PhantomJS for headless browser testing: $ bower search phantomjs The list returned will include any package with phantomjs in the package name or within its keywords list: phantom git://github.com/ariya/phantomjs.git dt-phantomjs git://github.com/keesey/dt-phantomjs qunit-phantomjs-runner git://github.com/jonkemp/... parse-cookie-phantomjs git://github.com/sindresorhus/... highcharts-phantomjs git://github.com/pesla/highcharts-phantomjs.git mocha-phantomjs git://github.com/metaskills/mocha-phantomjs.git purescript-phantomjs git://github.com/cxfreeio/purescript-phantomjs.git You can see from the returned list that the correct package name for PhantomJS is in fact phantom and not phantomjs. You can then proceed to install the package now that you know the correct name: $ bower install phantom --save-dev Now, you have Bower installed and know how to manage your frontend web components and development tools, but how do you integrate them into your SPA? This is where Grunt comes in. Summary Now that you have learned to set up an optimal development environment with NPM and supply it with frontend dependencies using Bower, it's time to start learning more about building a real app. Resources for Article: Further resources on this subject: API with MongoDB and Node.js [article] Tips & Tricks for Ext JS 3.x [article] Responsive Visualizations Using D3.js and Bootstrap [article]

0
0
12818

article-image-basics-classes-and-objects

Packt

06 Oct 2016

11 min read

Basics of Classes and Objects

Packt

06 Oct 2016

11 min read

In this article by Steven Lott, the author of the book Modern Python Cookbook, we will see how to use a class to encapsulate data plus processing. (For more resources related to this topic, see here.) Introduction The point of computing is to process data. Even when building something like an interactive game, the game state and the player's actions are the data, the processing computes the next game state and the display update. The data plus processing is ubiquitous. Some games can have a relatively complex internal state. When we think of console games with multiple players and complex graphics, there are complex, real-time state changes. On the other hand, when we think of a very simple casino game like Craps, the game state is very simple. There may be no point established, or one of the numbers 4, 5, 6, 8, 9, 10 may be the established point. The transitions are relatively simple, and are often denoted by moving markers and chips around on the casino table. The data includes the current state, player actions, and rolls of the dice. The processing is the rules of the game. A game like Blackjack has a somewhat more complex internal state change as each card is accepted. In games where the hands can be split, the state of play can become quite complex. The data includes the current game state, the player's commands, and the cards drawn from the deck. Processing is defined by the rules of the game as modified by any house rules. In the case of Craps, the player may place bets. Interestingly, the player's input, has no effect on the game state. The internal state of the game object is determined entirely by the next throw of the dice. This leads to a class design that's relatively easy to visualize. Using a class to encapsulate data plus processing The essential idea of computing is to process data. This is exemplified when we write functions that process data. Often, we'd like to have a number of closely related functions that work with a common data structure. This concept is the heart of object-oriented programming. A class definition will contain a number of methods that will control the internal state of an object. The unifying concept behind a class definition is often captured as a summary of the responsibilities allocated to the class. How can we do this effectively? What's a good way to design a class? Getting Ready Let's look at a simple, stateful object—a pair of dice. The context for this would be an application which simulates the casino game of Craps. The goal is to use simulation of results to help invent a better playing strategy. This will save us from losing real money while we try to beat the house edge. There's an important distinction between the class definition and an instance of the class, called an object. We call this idea – as a whole – Object-Oriented Programming. Our focus is on writing class definitions. Our overall application will create instances of the classes. The behavior that emerges from the collaboration of the instances is the overall goal of the design process. Most of the design effort is on class definitions. Because of this, the name object-oriented programming can be misleading. The idea of emergent behavior is an essential ingredient in object-oriented programming. We don't specify every behavior of a program. Instead, we decompose the program into objects, define the object's state and behavior via the object's classes. The programming decomposes into class definitions based on their responsibilities and collaborations. An object should be viewed as a thing—a noun. The behavior of the class should be viewed as verbs. This gives us a hint as to how we can proceed with design classes that work effectively. Object-oriented design is often easiest to understand when it relates to tangible real-world things. It's often easier to write a software to simulate a playing card than to create a software that implements an Abstract Data Type (ADT). For this example, we'll simulate the rolling of die. For some games – like the casino game of Craps – two dice are used. We'll define a class which models the pair of dice. To be sure that the example is tangible, we'll model the pair of dice in the context of simulating a casino game. How to do it... Write down simple sentences that describe what an instance of the class does. We can call these as the problem statements. It's essential to focus on short sentences, and emphasize the nouns and verbs. The game of Craps has two standard dice. Each die has six faces with point values from 1 to 6. Dice are rolled by a player. The total of the dice changes the state of the craps game. However, those rules are separate from the dice. If the two dice match, the number was rolled the hard way. If the two dice do not match, the number was easy. Some bets depend on this hard vs easy distinction. Identify all of the nouns in the sentences. Nouns may identify different classes of objects. These are collaborators. Examples include player and game. Nouns may also identify attributes of objects in questions. Examples include face and point value. Identify all the verbs in the sentences. Verbs are generally methods of the class in question. Examples include rolled and match. Sometimes, they are methods of other classes. Examples include change the state, which applies to the Craps game. Identify any adjectives. Adjectives are words or phrases which clarify a noun. In many cases, some adjectives will clearly be properties of an object. In other cases, the adjectives will describe relationships among objects. In our example, a phrase like the total of the dice is an example of a prepositional phrase taking the role of an adjective. The the total of phrase modifies the noun the dice. The total is a property of the pair of dice. Start writing the class with the class statement. class Dice: Initialize the object's attributes in the __init__ method. def __init__(self): self.faces = None We'll model the internal state of the dice with the self.faces attribute. The self variable is required to be sure that we're referencing an attribute of a given instance of a class. The object is identified by the value of the instance variable, self We could put some other properties here as well. The alternative is to implement the properties as separate methods. These details of the design decision is the subject for using properties for lazy attributes. Define the object's methods based on the various verbs. In our case, we have several methods that must be defined. Here's how we can implement dice are rolled by a player. def roll(self): self.faces = (random.randint(1,6), random.randint(1,6)) We've updated the internal state of the dice by setting the self.faces attribute. Again, the self variable is essential for identifying the object to be updated. Note that this method mutates the internal state of the object. We've elected to not return a value. This makes our approach somewhat like the approach of Python's built-in collection classes. Any method which mutates the object does not return a value. This method helps implement the total of the dice changes the state of the Craps game. The game is a separate object, but this method provides a total that fits the sentence. def total(self): return sum(self.faces) These two methods help answer the hard way and easy way questions. def hardway(self): return self.faces[0] == self.faces[1] def easyway(self): return self.faces[0] != self.faces[1] It's rare in a casino game to have a rule that has a simple logical inverse. It's more common to have a rare third alternative that has a remarkably bad payoff rule. In this case, we could have defined easy way as return not self.hardway(). Here's an example of using the class. First, we'll seed the random number generator with a fixed value, so that we can get a fixed sequence of results. This is a way to create a unit test for this class. >>> import random >>> random.seed(1) We'll create a Dice object, d1. We can then set its state with the roll() method. We'll then look at the total() method to see what was rolled. We'll examine the state by looking at the faces attribute. >>> from ch06_r01 import Dice >>> d1 = Dice() >>> d1.roll() >>> d1.total() 7 >>> d1.faces (2, 5) We'll create a second Dice object, d2. We can then set its state with the roll() method. We'll look at the result of the total() method, as well as the hardway() method. We'll examine the state by looking at the faces attribute. >>> d2 = Dice() >>> d2.roll() >>> d2.total() 4 >>> d2.hardway() False >>> d2.faces (1, 3) Since the two objects are independent instances of the Dice class, a change to d2 has no effect on d1. >>> d1.total() 7 How it works... The core idea here is to use ordinary rules of grammar – nouns, verbs, and adjectives – as a way to identify basic features of a class. Noun represents things. A good descriptive sentence should focus on tangible, real-world things more than ideas or abstractions. In our example, dice are real things. We try to avoid using abstract terms like randomizers or event generators. It's easier to describe the tangible features of real things, and then locate an abstract implementation that offers some of the tangible features. The idea of rolling the dice is an example of physical action that we can model with a method definition. Clearly, this action changes the state of the object. In rare cases – one time in 36 – the next state will happen to match the previous state. Adjectives often hold the potential for confusion. There are several cases such as: Some adjectives like first, last, least, most, next, previous, and so on will have a simple interpretation. These can have a lazy implementation as a method or an eager implementation as an attribute value. Some adjectives are more complex phrase like "the total of the dice". This is an adjective phrase built from a noun (total) and a preposition (of). This, too, can be seen as a method or an attribute. Some adjectives involve nouns that appear elsewhere in our software. We might have had a phrase like "the state of the Craps game" is a phrase where "state of" modifies another object, the "Craps game". This is clearly only tangentially related to the dice themselves. This may reflect a relationship between "dice" and "game". We might add a sentence to the problem statement like "The dice are part of the game". This can help clarify the presence of a relationship between game and dice. Prepositional phrases like "are part of" can always be reversed to create the a statement from the other object's point of view—"The game contains dice". This can help clarify the relationships among objects. In Python, the attributes of an object are – by default – dynamic. We don't specific a fixed list of attributes. We can initialize some (or all) of the attributes in the __init__() method of a class definition. Since attributes aren't static, we have considerable flexibility in our design. There's more... Capturing the essential internal state, and methods that cause state change is the first step in good class design. We can summarize some helpful design principles using the acronym SOLID. Single Responsibility Principle: A class should have one clearly-defined responsibility. Open/Closed Principle: A class should be open to extension – generally via inheritance – but closed to modification. We should design our classes so that we don't need to tweak the code to add or change features. Liskov Substitution Principle: We need to design inheritance so that a subclass can be used in place of the superclass. Interface Segregation Principle: When writing a problem statement, we want to be sure that collaborating classes have as few dependencies as possible. In many cases, this principle will lead us to decompose large problems into many small class definitions. Dependency Inversion Principle: It's less than ideal for a class to depend directly on other classes. It's better if a class depends on an abstraction, and a concrete implementation class is substituted for the abstract class. The goal is to create classes that have the proper behavior and also adhere to the design principles. Resources for Article: Further resources on this subject: Python Data Structures [article] Web scraping with Python (Part 2) [article] How is Python code organized [article]

0
0
4277

Packt

06 Oct 2016

7 min read

Python for Driving Hardware

Packt

06 Oct 2016

7 min read

In this article by Tim Cox, author of the book Raspberry Pi Cookbook for Python Programmers - Second Edition, we will see how to control Raspberry Pi with the help of your own buttons and switches. (For more resources related to this topic, see here.) Responding to a button Many applications using the Raspberry Pi require that actions are activated without a keyboard and screen attached to it. The GPIO pins provide an excellent way for the Raspberry Pi to be controlled by your own buttons and switches without a mouse/keyboard and screen. Getting ready You will need the following equipment: 2 x DuPont female to male patch wires Mini breadboard (170 tie points) or a larger one Push button switch (momentary close) or a wire connection to make/break the circuit Breadboarding wire (solid core) 1k ohm resistor The switches are as seen in the following diagram: The push button switch and other types of switches The switches used in the following examples are single pole single throw (SPST) momentary close push button switches. Single pole (SP) means that there is one set of contacts that makes a connection. In the case of the push switch used here, the legs on each side are connected together with a single pole switch in the middle. A double pole (DP) switch acts just like a single pole switch, except that the two sides are separated electrically, allowing you to switch two separate components on/off at the same time. Single throw (ST) means the switch will make a connection with just one position; the other side will be left open. Double throw (DT) means both positions of the switch will connect to different parts. Momentary close means that the button will close the switch when pressed and automatically open it when released. A latched push button switch will remain closed until it is pressed again. The layout of the button circuit We will use sound in this example, so you will also need speakers or headphones attached to audio socket of the Raspberry Pi. You will need to install a program called flite using the following command, which will let us make the Raspberry Pi talk: sudo apt-get install flite After it has been installed, you can test it with the following command: sudo flite -t "hello I can talk" If it is a little too quiet (or too loud), you can adjust the volume (0-100 percent) using the following command: amixer set PCM 100% How to do it… Create the btntest.py script as follows: #!/usr/bin/python3 #btntest.py import time import os import RPi.GPIO as GPIO #HARDWARE SETUP # GPIO # 2[==X==1=======]26[=======]40 # 1[=============]25[=======]39 #Button Config BTN = 12 def gpio_setup(): #Setup the wiring GPIO.setmode(GPIO.BOARD) #Setup Ports GPIO.setup(BTN,GPIO.IN,pull_up_down=GPIO.PUD_UP) def main(): gpio_setup() count=0 btn_closed = True while True: btn_val = GPIO.input(BTN) if btn_val and btn_closed: print("OPEN") btn_closed=False elif btn_val==False and btn_closed==False: count+=1 print("CLOSE %s" % count) os.system("flite -t '%s'" % count) btn_closed=True time.sleep(0.1) try: main() finally: GPIO.cleanup() print("Closed Everything. END") #End How it works… We set up the GPIO pin as required, but this time as an input, and we also enable the internal pull-up resistor (refer to the Pull-up and pull-down resistor circuits subsection in the There's more… section of this recipe for more information) using the following code: GPIO.setup(BTN,GPIO.IN,pull_up_down=GPIO.PUD_UP) After the GPIO pin is set up, we create a loop that will continuously check the state of BTN using GPIO.input(). If the value returned is false, the pin has been connected to 0V (ground) through the switch, and we will use flite to count out loud for us each time the button is pressed. Since we have called the main function from within a try/finally condition, it will still call GPIO.cleanup() even if we close the program using Ctrl + Z. We use a short delay in the loop; this ensures that any noise from the contacts on the switch is ignored. This is because when we press the button, there isn't always perfect contact as we press or release it, and it may produce several triggers if we press it again too quickly. This is known as software debouncing; we ignore the bounce in the signal here. There's more… The Raspberry Pi GPIO pins must be used with care; voltages used for inputs should be within specific ranges, and any current drawn from them should be minimized using protective resistors. Safe voltages We must ensure that we only connect inputs that are between 0V (Ground) and 3.3V. Some processors use voltages between 0V and 5V, so extra components are required to interface safely with them. Never connect an input or component that uses 5V unless you are certain it is safe, or you will damage the GPIO ports of the Raspberry Pi. Pull-up and pull-down resistor circuits The previous code sets the GPIO pins to use an internal pull-up resistor. Without a pull-up resistor (or pull-down resistor) on the GPIO pin, the voltage is free to float somewhere between 3.3V and 0V, and the actual logical state remains undetermined (sometimes 1 and sometimes 0). Raspberry Pi's internal pull-up resistors are 50k ohm - 65k ohm and the pull-down resistors are 50k ohm - 65k ohm. External pull-up/pull-down resistors are often used in GPIO circuits (as shown in the following diagram), typically using 10k ohm or larger for similar reasons (giving a very small current draw when not active). A pull-up resistor allows a small amount of current to flow through the GPIO pin and will provide a high voltage when the switch isn't pressed. When the switch is pressed, the small current is replaced by the larger one flowing to 0V, so we get a low voltage on the GPIO pin instead. The switch is active low and logic 0 when pressed. It works as shown in the following diagram: A pull-up resistor circuit Pull-down resistors work in the same way, except the switch is active high (the GPIO pin is logic 1 when pressed). It works as shown in the following diagram: A pull-down resistor circuit Protection resistors In addition to the switch, the circuit includes a resistor in series with the switch to protect the GPIO pin as shown in the following diagram: A GPIO protective current-limiting resistor The purpose of the protection resistor is to protect the GPIO pin if it is accidentally set as an output rather than an input. Imagine, for instance, that we have our switch connected between the GPIO and ground. Now the GPIO pin is set as an output and switched on (driving it to 3.3V) as soon as we press the switch; without a resistor present, the GPIO pin will directly be connected to 0V. The GPIO will still try to drive it to 3.3V; this would cause the GPIO pin to burn out (since it would use too much current to drive the pin to the high state). If we use a 1k ohm resistor here, the pin is able to be driven high using an acceptable amount of current (I = V/R = 3.3/1k = 3.3mA). Resources for Article: Further resources on this subject: Raspberry Pi LED Blueprints [article] Raspberry Pi and 1-Wire [article] Learning BeagleBone Python Programming [article]

0
0
14300

article-image-frontend-development-bootstrap-4

Packt

06 Oct 2016

19 min read

Frontend development with Bootstrap 4

Packt

06 Oct 2016

19 min read

0
0
34510

Amit Kothari

06 Oct 2016

6 min read

How to Manage Legacy Code

Amit Kothari

06 Oct 2016

6 min read

In my 10+ years of experience working as a software developer, I have spent more time working on legacy systems than on greenfield projects. What is legacy code and why do companies and teams invest so much to rewrite the code which is already working? What is legacy code? When we talk about legacy code, we think of old code written in old technology. But in reality, any code that is difficult to work with is legacy. If you are working on a code base that is difficult to understand, and making a small change takes a long time, then the code is legacy code. “Code without tests is bad code. It doesn't matter how well written it is; it doesn't matter how pretty or object-oriented or well-encapsulated it is. With tests, we can change the behavior of our code quickly and verifiably. Without them, we really don't know if our code is getting better or worse.” - Michael C. Feathers, Working Effectively with Legacy Code In today’s competitive world, companies want the software to be able to change quickly. However, working with legacy code is always slow because of the fear of introducing bugs, since there is no way to verify that the system is working as expected. Software modernization In my experience, the most common strategy to deal with legacy code is to replace it with clean code. If we keep adding more code to a legacy system without cleaning it up, we are just increasing to the technical debt. The process of rewriting legacy code to a modern technology is known as software modernization. Rewriting an existing application is quite different from writing a new app. In a greenfield app, we can start with a minimal viable product and then evolve our system based on user feedback. This image nicely explains the development cycle of a new product, so that you solve a user problem and then iteratively make it better. Unfortunately, when rewriting an existing app, we cannot follow the same approach because the end users already have the full-fledged solution. It will be like asking a user to use a skateboard as a replacement for a car. Because of this reason, some teams decide to write a new system in parallel to the legacy system. This means the legacy system is still live and in operation while the team is working on a new system to replace it. I have seen this approach work in a few small projects where the legacy system was quite stable and not in active development. However, in case of a large system that was developed over many years and is still changing based on new requirements, there is a lot of code to rewrite, which in most cases is not well documented and the requirements are not clear. All of this results in projects getting delayed, and business is not getting anything in return of their investments, and eventually projects get cancelled after companies have already invested a lot of time and money into it. Strategy to manage legacy systems Just like a new application, it is not a great idea to follow a big bang approach to tackle a legacy system. Here is the strategy to convert large legacy codebase to clean code. Divide and rule - Divide the system into different domains or modules and start rewriting one module at the time. The smaller the modules, the easier and quicker they will be to replace. The business will see return for their investment, users will get a better system, and along the way the development team can learn how to make the whole process smoother and better for the next module rewrite. Integrate with the existing system - It is worth investigating on how to integrate a module written in the new technology with the existing system. Start with a walking skeleton, that is, a very small functionality written in the new technology, and integrate it with the existing system. It is better to link all of the main architectural components as soon as possible instead of leaving the integration part for later. Test covering - While dealing with legacy code, there is a very good chance that the requirements are not very clear or well documented. We want the new module to work the same way as the existing legacy system. The best way to achieve this is by writing tests. Start with writing tests to run against the existing code and verify its behavior. Then the same tests can be reused and run against the new module to make sure it works the same way as the existing code. Faster feedback cycle - Even if we break down the system into smaller modules, we want to keep the feedback loop as small as possible. Use a continuous delivery approach to release software faster and more frequently to the users. Automate the release and deployment process and keep the development and testing environment in production—try to avoid any last-minute issues on deployment. Build a better system not a replica - We are putting all of this effort in rewriting the system, in order to think about making it better. Improve user experience by making the interface intuitive and simplify workflow, save time and effort by removing features that are not required anymore, and add the features that the users always wanted. Work in feature/domain teams - One key difference I have noticed between successful and failed/delayed projects is the team structure. Projects with cross functional teams have higher chances of meeting their goals, because everyone required to deliver the project is part of the same team. On the other hand, dividing teams based on technology like frontends, backends, and operations team can slow down the delivery process due to the lack of communication, and because each team has different priorities. Hopefully this has given you a good starting point for approaching legacy code. It can certainly be a bit of a challenge, and it demands that you think about the specifics of a given situation and ask yourself what's at stake and what's really important. If you're interested in learning more about dealing with legacy code, check out this article. About the author Amit Kothari is a full-stack software developer based in Melbourne, Australia. He has 10+ years of experience in designing and implementing software mainly in Java/JEE. His recent experience is in building web applications using JavaScript frameworks like React and AngularJS and backend micro services/ REST API in Java. He is passionate about lean software development and continuous delivery.

0
0
4349

How-To Tutorials

Create a User Profile System and use the Null Coalesce Operator

Server-side Swift: Building a Slack Bot, Part 1

Basics of Image Histograms in OpenCV

Moving from Windows to Appliance

Asynchronous Programming in F#

Solving an NLP Problem with Keras, Part 1

Reactive Python - Asynchronous programming to the rescue, Part 2

Modern Natural Language Processing – Part 3

How to start Chainer

Introduction to Neural Networks with Chainer – Part 3

Trending Topics

Getting Organized with NPM and Bower

Basics of Classes and Objects

Python for Driving Hardware

Frontend development with Bootstrap 4

How to Manage Legacy Code

Create a Free Account To Continue Reading

Sign in to activate your 7-day free access