Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7018 Articles
article-image-less-css-preprocessor
Packt
21 May 2013
7 min read
Save for later

LESS CSS Preprocessor

Packt
21 May 2013
7 min read
(For more resources related to this topic, see here.) Installing LESS We're going to start the recipes in this book by looking at how we can get hold of LESS, and adding support for it to your website. LESS comes in two versions, depending on whether you want to use it client side or server side; for the purpose of this recipe, we're going to use it client side. The library is hosted on Google Code, and can be downloaded or included (as a CDN link) from http://cdnjs.cloudflare.com/ajax/libs/less.js/1.3.1/less. min.js. How to do it... The following steps will guide you in installing LESS: Let's get started by creating a new folder on your PC, let's call it test less projects. Crack open a normal text editor of your choice, save a copy of the code from the What you need for this book section in the preface of this book, and save it as test less include.html. Add the following in between the <body> tags in the code: <form action="">Name: <input type="text" class="input" />Password: <input type="password" class="input" /><input type="submit" value="This is a button" /></form> It shows a very plain, basic form, so let's fix that by starting to use LESS to provide some styling. Create a new document in your text editor, then add the following, and save it as include.less: @color-button: #d24444;#submitfrm {color:#fff;background:@color-button;border:1px solid @color-button - #222;padding:5px 12px;} Let's now add a link to this file to your main HTML file, so go ahead and alter your code accordingly: <link rel="stylesheet/less" type="text/css" href="include.less"> That's all that's required, so if you now open your browser, and view the file, you should see on screen the same as the following screenshot: How it works... This recipe was intended to serve as a very basic example of how you can use less.js. You will be already familiar with what most of the code does, with two exceptions (highlighted in the following code snippet): @color-button: #d24444;#submitfrm {color:#fff;background:@color-button;border:1px solid @color-button - #222;padding:5px 12px;} The first exception is simply setting a variable called color-button, which holds a value of #d24444; this is the red background you see on the button. There are a couple of points of interest here: All variables used in LESS must be preceded with an @ sign, to denote that they are variables Variables don't actually exist in the LESS library Huh? I hear you ask. That surely doesn't make sense! Well, let me explain: when using LESS, variables are actually classed as constants, as you can't reassign a new value to an existing predefined variable. There is nothing stopping you from using an existing variable to calculate a new value, but that value must be assigned to a new variable, or used to work out a value for a CSS style: background:@ color-button;border: 1px solid @color-button - #222;padding: 5px 12px; There's more... In order for the library to work properly, you need to first include links to your .less stylesheets, and set the rel tag to stylesheet/less, in order for them to work properly: <link rel="stylesheet/less" type="text/css" href="styles.less"> Note the use of the rel attribute on this link, you need to use the /less value, in order for LESS to work properly. If you are using HTML5 syntax, you don't need to include the type="text/less" and type="text/javascript" values. Next, you need to include LESS; you can either download it from the website and include it locally in the same way that you would include any JavaScript file, or use the following CDN link; in either case, you must include it after your .less stylesheet: <script src = "http://cdnjs.cloudflare.com/ajax/libs/less.js/1.3.1/less.min.js"></script> If you get a 406 error in your browser, you may need to set a MIME (or Internet Media Type, as it is now known), as the text/LESS tags may not work properly. We've seen how LESS can compile styles on the fly, but this may not be ideal if you have a very large site, or have a development process which doesn't allow the use of creating styles dynamically. This isn't an issue with LESS, as you can easily generate the stylesheet prior to including it within your site's pages. Precompiling LESS client side In the previous section, we looked at how you can use Less to dynamically generate your compiled stylesheet from within your site. This may not suit everyone's needs. Here, we will see some alternatives that allow us to precompile our CSS styles, so we can then include the finished results on our site. Getting ready Here we're going to use the open source application WinLESS to compile a LESS file into a normal CSS stylesheet. You can download a copy of the program from http://www.winless.org. You will also need your favorite text editor. We're going to create a typical .less file. How to do it... Open up the text editor of your choice, and add in the following lines; save it as testprecompile.less in the folder you created from the Installing LESS section: .border-radius(@radius: 3px) { -webkit-border-radius: @radius;-moz-border-radius: @radius; border-radius: @radius; }.box-shadow(@x : 2px, @y : 2px, @blur : 5px, @spread : 0, @color :rgba(0,0,0,.6)) {-webkit-box-shadow: @x @y @blur @spread @color;-moz-box-shadow: @x @y @blur @spread @color;box-shadow: @x @y @blur @spread @color;}div { @color: green; background: @color; width: 300px; height:300px; margin: 30px auto; .border-radius(10px); .box-shadow(); } Double-click on the WinLess_1.5.3.msi file you downloaded to install it, accept all defaults, and double-click on it to open the application. Click on the Add folder button, and select the folder you created in the first step, and click on OK to add it to the folder list of WinLess. Click on the Refresh folder button to update the list on the right-hand side as shown in the following screenshot: Click on Compile to generate the CSS file; if you open the resulting CSS file, you will see the generated code as follows: div {background: #008000;width: 300px;height: 300px;margin: 30px auto;-webkit-border-radius: 10px;-moz-border-radius: 10px;border-radius: 10px;-webkit-box-shadow: 2px 2px 5px 0 rgba(0, 0, 0, 0.6);-moz-box-shadow: 2px 2px 5px 0 rgba(0, 0, 0, 0.6);box-shadow: 2px 2px 5px 0 rgba(0, 0, 0, 0.6);} As you make further changes to the .less file, WinLess will automatically update the CSS file for you; it will remain in the same folder as the .less file, until you are ready to use it in a production environment. How it works... WinLess is GUI front-ends to the command-line version of LESS, lessc.cmd. The GUI takes the content of the .less file, parses it, and gives a .css file as the output with the compiled CSS styles. WinLess includes an option to maintain a list of files that it will automatically monitor, so that when any are changed, it will automatically update the contents of the equivalent CSS file with the appropriate changes. Summary In this article we saw how to install support for LESS and how to compile CSS styles before adding adding the code to your website. Resources for Article : Further resources on this subject: Drupal 6 Theming: Adding and Optimizing CSS Files [Article] Building our own Plone 3 Theme Add-on Product [Article] Creating Themes for a Report using BIRT [Article]
Read more
  • 0
  • 0
  • 2876

article-image-tour-around-gimp
Packt
21 May 2013
5 min read
Save for later

A Tour Around GIMP

Packt
21 May 2013
5 min read
(For more resources related to this topic, see here.)   We will go to produce the following image: Step 1 – Creating a new file Working with GIMP requires you to work on canvas, just like an artist would do. In GIMP, the canvas is a file on your system. You may have a clean canvas (a new file), or you may also work on an image obtained from various sources. The File menu presents you with a number of options. From here, you can create a clean canvas by selecting New. From the next screenshot, we can see that there is an extensive menu for Create. You may start to import images copied from somewhere else, scan them, or take a screenshot to begin with. To create a new canvas, follow these steps: Click on File. From the menu, click on New. Now the Create a New Image dialog box will appear. Adjust the width and height.Before clicking on OK, set the background to be filled with nothing by selecting Transparency. Now you have the canvas to work on your first art, as shown in the following screenshot: If you have hand drawn the outline of the picture and scanned it previously like I do, locate the scanned picture on your machine, and all you got to do is drag the file into the canvas. If you happen to have a stylus pen, you could easily draw on the canvas. Step 2 – Setting the picture for painting It is the perfect time to introduce the toolbox. The Toolbox option provides you with a set of tools to complete your work. The collections of tools here involve tools to select areas, painting, transform, and touch-up tools. If the Toolbox window does not show up, press Ctrl + B. Following is a screenshot of the Toolbox window. The previous screenshot consists of two parts, the tools and also tool options that further define the properties of the tool. The default installation of GIMP puts the Toolbox window and Tool option together, but they can be separated. It is one of the GIMP feature to be able to dock windows together or separate them according to user preference. Click on Windows and choose Dockable Dialogs to expose various dialog boxes that you might want to bring up front while working. If you're not sure how to go with each item of the menu, place the cursor on the icon, and a tooltip will appear together with a shortcut key in bold. At the bottom of the Toolbox window is the color chooser, where user can select an active color and alternative color, making painting easier when these colors are switchable. Click on the magic wand icon .Then, by clicking on the hair of the boy, the hair will be selected.   Now, duplicate the layer, so we can have a layer exclusively for his hair. This step involves working with layers. Step 3 – Create layers It is a good idea to have a layer for each part of the image. Hair, shirt, skin – all should have their own layer. This way you can paint each part without accidently filling other parts. The concept of layers is important when you are working with GIMP. Think of it as if you are drawing on different set of papers or transparencies that you may cut and rearrange before you paste them on the drawing board. The Layers dockable window can be accessed from Windows | Dockable Dialogs | Layers. The Layers window is useful to navigate between layers. Refer to the following screenshot: To navigate between layers, follow these steps: Drag a layer to the   icon to quickly create a duplicate of the layer. Press Ctrl + I (or click on Select | Invert) to select the opposite part and delete it. Now you have a layer specific for a part. Create a layer for each of the parts. The layers should be aligned and overlapped on top of the original outline. To rename the layers, double-click on the layer name or click on the layer and press F2. The following is a screenshot of how the layers should look: There are various controls that can be used from the Layers window alone. You may set certain layers' visibility by clicking on the eye icon and control how it blends with other layers by selecting the mode. Layers are best used when composing images. Each of the images can be on a layer, so it is easier to adjust them around images on other layers. These layers can be organized to let certain layer sit on top of others. To do so, drag the layer to the top to make it sit on top of the other image. Step 4 – Coloring Click the Paintbrush Tool , and pick a color of your choice and paint away.   To select another color, click on , and you'll be given a menu, which has a set of colors to choose from:   It is up to you to pick any color from the spectrum. Once done, click on OK, and you're back to the canvas. Repeat the select-and-paint process for each part. To illustrate the separate layers being colored, let's look at the following screenshot: Every fill on the drawing is separated by layers to enable each layer to be modified without affecting the others. You may also specify the opacity of individual layers. Summery This concludes a brief tour of GIMP. These few basic steps are just the tip of the iceberg. It is amazing what GIMP has in store for you. Resources for Article : Further resources on this subject: Creating a Quick Logo for a Company with GIMP 2.6 [Article] Blender 2.5: Creating a UV Texture [Article] How To Create Amazing Text and Font Effects in Gimp 2.6 [Article]
Read more
  • 0
  • 0
  • 15090

Packt
20 May 2013
15 min read
Save for later

Play! Framework 2 – Dealing with Content

Packt
20 May 2013
15 min read
(For more resources related to this topic, see here.) In order to keep the article short and to the point, we'll only see the Java part. Keep in mind that the Scala version is little different for this level of detail. Body parsing for better reactivity As noted earlier, the way to manage content in Play! 2 is to use instances of body parsers. In brief, a body parser is a component that is responsible for parsing the body of an HTTP request as a stream to be converted into a predefined structure. This has a common sense ring to it, however their strength is in their way of consuming the stream—in a reactive fashion. Reactivity, in this context, is meant to describe a process where an application won't block on a task that is actually idle. As a stream consumption task is idle when no bytes are incoming, a body parser should behave the same. It will read and construct an internal representation of the incoming bytes. But it can also decide at any time that it has read enough to terminate and return the representation. On the other hand, if no more bytes are coming into the stream, it can relax its thread in favor of another request; it pauses its work until new bytes are received. Thinking about an HTTP request that is sending a bunch of XML content, the underlying action can use the XML-related body parser to handle it correctly (read reactively); that is, by parsing it and providing a DOM representation. To understand what a body parser actually is, we'll first look at how they are used—in the actions. An action in Play! 2 represents the piece of software that is able to handle an HTTP request; therefore, they are the right place to use a body parser. In the Java API, an action is allowed to be annotated with the Of annotation available in the BodyParser class. This annotation declares the expected type of request routed to it, and it requires a parameter that is the class of the parser that will be instantiated to parse the incoming request's body. The following screenshot shows an example: Isn't this helpful? We've gone from a request to a W3C document, in a single line. Functionally speaking, this works because an action is semantically a higher-order function that takes a body parser and generates a function that takes a request (and so its body) and results in an HTTP response (result). This result will then be used to construct the HTTP response by Play! 2. In Java, it is not all that obvious how to create a higher-order function. A good way, however, to achieve this was to add an annotation. An annotation can be processed at runtime in order to execute the right body parser (in this case). To illustrate this, we'll have a quick look at the Scala version: With this Scala version, it is easy to see that an action is dealing with a function from a request to a response. There are a plenty of predefined body parsers that can be used to handle our requests, and they are all defined in the BodyParser class as static inner classes. One can have a specific behavior to be applied on its expected request body, and even though a body parser has to be implemented in Scala, a Java coder can simply extend these current implementations. Actually, they're already providing enough control to cover all custom use cases. So, we have in our hands tools to handle the following content types: JSON XML URL form encoded Multipart (for uploading files) Text Raw (fallback) As we can see from the previous list, there is, obviously, an implementation for the x-www-form-urlencoded content type. Indeed, this is the parser we've used so far to retrieve data from the client side. For example, using POST requests throughout HTML forms. But wait, we never had to add such annotations to our actions, and, moreover, we've never looked in the parsed result. That's true, Play! 2, as a great framework, is already doing a lot of stuff for us. And that's because it's a web framework; it takes advantage of HTTP; in this case, using the content-type header. Based on this hint, it seems obvious that Play! Framework 2 will look in this header to find the right parser to apply. So annotations are mandatory, but where did we use them previously? In the bindFromRequest method, of course. Let's see how. We have used form instances, and we fed them some data through the client. Those instances were applied on the request using the bindFromRequest method, and this method's job was to look for data according to the provided content type. And, of course, this content type was set in the header by the HTML forms themselves. Indeed, an HTTP GET will send data in the request URL (query string), where an HTTP POST will be sent with a body that contains all data encoded by default as URL parameters (that is, x-www-url-encoded). So, we can now give an overview of what the bindFromRequest method does. When we ask a form to be filled in with data, this method will: Gather data as URL-form encoded data, if any Gather data from parts (if the content type is multipart-data) Gather data as JSON-encoded, if any Gather data from the query string (that's why GET requests were working as well)/p> Fill in the form's data with all of them (and validate) You might be wondering the worth of such annotations; the quick answer to that is they allow new types of parsers, but they can also enforce certain actions' requests to match a given content type. Another advantage of such annotations is that they allow us to extend or narrow the length of the body that can be handled. By default, 100 K are accepted, and this can be either configured (parsers.text.maxLength=42K) or passed as an argument to the annotation. With all of this in mind, we are now ready to implement these concepts in our code, and what we're going to do is to update our code base to create a kind of forum. A forum where one can log in, initiate a chat, reply to non-closed ones (based on their date), or even attach files to them. Creating a forum In this section, we'll refactor our existing application in order to enable it to act as a forum. And, chances are high that it won't be necessary to learn anything new; we'll just re-use the skills gathered so far; but we'll also use the parsing commodities that Play! 2 offers us. Reorganizing and logging in The very first thing we have to do is to enable a user to log in; this ability was already created in the Data controller. However, for that, we'll update our Application controller a bit, to create a new index action that will check whether a user is logged in or not. So, index is now the new entry point of the application and can be routed from / in the routes file. And, it's solely meant to check if a user has logged in or not. This check is based on the session content, as we simply check whether a user's e-mail is present in the session. We never see what a session can be in Play! 2, but we saw that Play! 2 is completely stateless. So, a session in Play! 2 is only an encrypted map of the value stored in the cookie. Thus it cannot be that big, and definitely cannot contain full data. If the user is present, we redirect the request to the chatroom by calling redirect with the expected action. This will prevent the browser from posting the request again if the user reloads the page. This method is called POST-redirect-GET. Otherwise, we respond with an Unauthorized HTTP response (401) that contains the HTML login page. The two actions (shown in the next screenshot) are so simple that we won't cover them further, except for a single line: session().clear(). It is simply revoking the cookie's content, which will require the subsequent request to create a new one, which then doesn't contain the previously stored e-mail. And finally, enter, which shows how a request's body can easily be handled using the relevant method: asFormUrlEncoded. It should look like that shown in the following screenshot: Indeed, one would normally have to use a form to retrieve this information for us, which would do it for us (behind the scenes); but in this case we have only a single parameter to retrieve, so a form would be overkill. So far, so good; we are now able to create a user, log in with it, and use a login page. To target having cleaner code, it would be worth splitting the Data controller code into several pieces (matter of a good separation of subject). Hence, the Users controller is created, in which will be placed the user-related actions taken out of Data. Now, we'll move back to something we saw earlier but didn't cover—the routes. Chats.allChats() action call. Chatting In the previous section, we were introduced to the Chats controller and its allChats action. If the names are self-descriptive, the underlying code isn't that much. First of all, we're now dealing with Chat instances that must be persisted somewhere in a database, along with their underlying items. But we'll also prepare for the next section, which relates to multipart data (for instance, it's helpful for file upload). That's why we'll add a brand new type, Image, which is also linked to Chat. Having said that, it would be worth checking our new chat implementation: Before we cover the Item and Image types, we'll first go to the Chats controller to see what's going on. Finally, we can see our allChats action; it's simply rendering all existing instances within a template. Even the rest of the controller is simple; everything is done in templates, which are left as exercises (we're so good at them now!). However, there's still the loadChat action that contains something related to this article: Long chatId = Long.parseLong(queryString.get("chatid")[0]); This action handles requests asking to show a particular Chat instance, which is a resource and thus should be served using a GET request. This implies that the parameter value is stored in the query string (or in the URL itself) rather than in the request body. Regarding query string access, it's more interesting to analyze the following line: Map<String,String[]> queryString = request().queryString(); In fact, all actions contextually refer to a request object, which is accessible using the request() method. This request object declares a queryString() method that returns a map of string and an array of strings. What comes next is trivial; we just get chatid out of this map (ok... in a very unsafe way). Until now, we have been able to log in and access the chatroom, where we can create or show chat instances. But we're still unable to reply to a chat. That's what will be tackled now. For that, we need to create an action that will, based on a chat ID, post a new message linked to the logged in user, and then attach this message as an item of the underlying Chat instance. For this, we must update the Item class with persistence information. Afterwards, we'll be able to update the Chats controller in order to create instances. Ok, it's like a beefed-up POJO; let's jump into the action that will create Item instances. The workflow to post a message for a user starts by enabling him/her to participate in a chat. This is done by loading it (using the loadChat action) where the user will be able to post a new message (an overview of the UI will be presented at the end of this article for illustration only). The following screenshot shows how it can be done: Observe how the user was recovered using the session. Still, nothing cumbersome to review here, we've just re-used a lot of stuff we've already covered. The action receives a POST request in which information about the message is given, and then we can bind the request to itemForm and finally save to the database the item contained in the resulting form. At most, we should notice that we're still free to encode the body as we want, and also that the chat ID is not a part of the form but a part of the action signature—that's because it is a part of the URL (routing). We've almost finished our forum; the only thing needed is to enable users to post images. Handling multipart content types The HTTP protocol is ready to accept, from a client, a lot of data and/or large chunks of data, at once. A way to achieve this is to use a specific encoding type: multipart/form-data. Such requests will have a body that can hold several data pieces formatted differently and attributed with different names. So, Play! 2 is a web framework that fits into HTTP as much as possible; that's why it deals with such requests goods, and provides an API that hides almost all of the tricky parts. In this section, we'll see how one could upload an image along with some caption text that will be attached to a specific chat. Before diving into the workflow, let's first create the holding structure: Image. This newly introduced type is not hard to understand as well; only two things should be pointed out: The pic() method that relies on the filePath field to recover the file itself. It uses a File instance to memorize subsequent calls. The enum type that prepares the action logic to filter the incoming files based on the given MIME type. This logic could also be defined in the validate method. These instances are always locked in with the connected user who uploaded it and will be added to a Chat instance. This will allow a chatroom to display all attached images with their caption beside the messages themselves. Now we're ready to look at the file upload itself by paying some attention to the last action of the Chats controller, that is, receiveImage. As we are used to simplifying the code (Play! 2 is there to ease our work, after all) and to get straight to the point, we reflected this in our receiveImage action.. In a very few lines, we declared a new action that expects requests to be multipart encoded containing at least two parts, where the first is a map of data (no matter how this map is encoded) to fill in imageForm (essentially a caption). The second will be the image part. After binding the request with the form and verifying that no errors have occurred, we can move to the body content in order to recover the binary data that was sent along with its metadata: the file content, its content type, its length, and so on. That was quite an intuitive thing to do – asking the body to be parsed as a multipart/multidata and and get it as an Http.MultipartFormData object, which has a getFile method that returns an Http.MultipartFormData.FilePart value. To understand why we didn't specify a body parser, recall that Play! 2 is able, most of the time, to discover which method fits best by itself. The Http.MultipartFormData. FilePart type is not only allowing us to recover the content as a file, but also its key in the multipart body, its filename header, and (especially) its content type. Having all of these things in hand, we are now able to check the content-type validity against the image's enum, and to store the image by getting the file path of the provided file. This file path will target the Temp directory of your machine. In the real world, the file should be relocated in a dedicated folder or maybe on an S3 repository. Et voilà! We have now learned about some of the features that can provide a very simple forum. The following screenshot shows what it could look like (without any efforts on the design, of course). First, the forms to show and enter archived and active chats: On entering an active chat, let's say the one named Today, we reach a page similar to the one shown next: Using the Attach an image form, we can select an image on our filesystem to be sent to the server. The result obtained is shown as follows: Until now, we have spoken about handling various content types coming from the outside world, but what about our application having to render content other than HTML? That's what we're about to see next.
Read more
  • 0
  • 0
  • 2223

article-image-techniques-for-creating-a-multimedia-database
Packt
17 May 2013
37 min read
Save for later

Techniques for Creating a Multimedia Database

Packt
17 May 2013
37 min read
(For more resources related to this topic, see here.) Tier architecture The rules surrounding technology are constantly changing. Decisions and architectures based on current technology might easily become out of date with hardware changes. To best understand how multimedia and unstructured data fit and can adapt to the changing technology, it's important to understand how and why we arrived at our different current architectural positions. In some cases we have come full circle and reinvented concepts that were in use 20 years ago. Only by learning from the lessons of the past can we see how to move forward to deal with this complex environment. In the past 20 years a variety of architectures have come about in an attempt to satisfy some core requirements: Allow as many users as possible to access the system Ensure those users had good performance for accessing the data Enable those users to perform DML (insert/update/delete) safely and securely (safely implies ability to restore data in the event of failure) The goal of a database management system was to provide an environment where these points could be met. The first databases were not relational. They were heavily I/O focused as the computers did not have much memory and the idea of caching data was deemed to be too expensive. The servers had kilobytes and then eventually, megabytes of memory. This memory was required foremost by the programs to run in them. The most efficient architecture was to use pointers to link the data together. The architecture that emerged naturally was hierarchical and a program would navigate the hierarchy to find rows related to each other. Users connected in via a dumb terminal. This was a monitor with a keyboard that could process input and output from a basic protocol and display it on the screen. All the processing of information, including how the screen should display it (using simple escape sequence commands), was controlled in the server. Traditional no tier The mainframes used a block mode structure, where the user would enter a screen full of data and press the Enter key. After doing this the whole screen of information was sent to the server for processing. Other servers used asynchronous protocols, where each letter, as it was typed, was sent to the server for processing. This method was not as efficient as block mode because it required more server processing power to handle the data coming in. It did provide a friendlier interface for data entry as mistakes made could be relayed immediately back to the user. Block mode could only display errors once the screen of data was sent, processed, and returned. As more users started using these systems, the amount of data in them began to grow and the users wanted to get more intelligence out of the data entered. Requirements for reporting appeared as well as the ability to do ad hoc querying. The databases were also very hard to maintain and enhance as the pointer structure linked everything together tightly. It was very difficult to perform maintenance and changes to code. In the 1970s the relational database concept was formulated and it was based on sound mathematical principles. In the early 1980s the first conceptual relational databases appeared in the marketplace with Oracle leading the way. The relational databases were not received well. They performed poorly and used a huge amount of server resources. Though they achieved a stated goal of being flexible and adaptable, enabling more complex applications to be built quicker, the performance overheads of performing joins proved to be a major issue. Benefits could be seen in them, but they could never be seen as being able to be used in any environment that required tens to hundreds or thousands of concurrent users. The technology wasn't there to handle them. To initially achieve better performance the relational database vendors focused on using a changing hardware feature and that was memory. By the late 1980s the computer servers were starting to move from 16 bit to 32 bit. The memory was increasing and there was drop in the price. By adapting to this the vendors managed to take advantage of memory and improved join performance. The relational databases in effect achieved a balancing act between memory and disk I/O. Accessing a disk was about a thousand times slower than accessing memory. Memory was transient, meaning if there was a power failure and if there was data stored in memory, it would be lost. Memory was also measured in megabytes, but disk was measured in gigabytes. Disk was not transient and generally reliable, but still required safeguards to be put in place to protect from disk failure. So the balancing act the databases performed involved caching data in memory that was frequently accessed, while ensuring any modifications made to that data were always stored to disk. Additionally, the database had to ensure no data was lost if a disk failed. To improve join performance the database vendors came up with their own solutions involving indexing, optimization techniques, locking, and specialized data storage structures. Databases were judged on the speed at which they could perform joins. The flexibility and ease in which applications could be updated and modified compared to the older systems soon made the relational database become popular and must have. As all relational databases conformed to an international SQL standard, there was a perception that a customer was never locked into a propriety system and could move their data between different vendors. Though there were elements of truth to this, the reality has shown otherwise. The Oracle Database key strength was that you were not locked into the hardware and they offered the ability to move a database between a mainframe to Windows to Unix. This portability across hardware effectively broke the stranglehold a number of hardware vendors had, and opened up the competition enabling hardware vendors to focus on the physical architecture rather than the operating system within it. In the early 1990s with the rise in popularity of the Apple Macintosh, the rules changed dramatically and the concept of a user friendly graphical environment appeared. The Graphical User Interface (GUI) screen offered a powerful interface for the user to perform data entry. Though it can be argued that data entry was not (and is still not) as fast as data entry via a dumb terminal interface, the use of colors, varying fonts, widgets, comboboxes, and a whole repository of specialized frontend data entry features made the interface easier to use and more data could be entered with less typing. Arguably, the GUI opened up the computer to users who could not type well. The interface was easier to learn and less training was needed to use the interface. Two tier The GUI interface had one major drawback; it was expensive to run on the CPU. Some vendors experimented with running the GUI directly on the server (the Solaris operating system offered this capability), but it become obvious that this solution would not scale. To address this, the two-tier architecture was born. This involved using the GUI, which was running on an Apple Macintosh or Microsoft Windows or other Windows environment (Microsoft Windows wasn't the only GUI to run on Intel platforms) to handle the display processing. This was achieved by moving the application displayed to the computer that the user was using. Thus splitting the GUI presentation layer and application from the database. This seemed like an ideal solution as the database could now just focus on handling and processing SQL queries and DML. It did not have to be burdened with application processing as well. As there were no agreed network protocols, a number had to be used, including named pipes, LU6.2, DECNET, and TCP/IP. The database had to handle language conversion as the data was moved between the client and the server. The client might be running on a 16-bit platform using US7ASCII as the character set, but the server might be running on 32-bit using EBCDIC as the character set. The network suddenly became very complex to manage. What proved to be the ultimate show stopper with the architecture had nothing to do with the scalability of client or database performance, but rather something which is always neglected in any architecture, and that is the scalability of maintenance. Having an environment of a hundred users, each with their own computer accessing the server, requires a team of experts to manage those computers and ensure the software on it is correct. Application upgrades meant upgrading hundreds of computers at the same time. This was a time-consuming and manual task. Compounded by this is that if the client computer is running multiple applications, upgrading one might impact the other applications. Even applying an operating system patch could impact other applications. Users also might install their own software on their computer and impact the application running on it. A lot of time was spent supporting users and ensuring their computers were stable and could correctly communicate with the server. Three tier Specialized software vendors tried to come to the rescue by offering the ability to lock down a client computer from being modified and allowing remote access to the computer to perform remote updates. Even then, the maintenance side proved very difficult to deal with and when the idea of a three tier architecture was pushed by vendors, it was very quickly adopted as the ideal solution to move towards because it critically addressed the maintenance issue. In the mid 1990s the rules changed again. The Internet started to gain in popularity and the web browser was invented. The browser opened up the concept of a smart presentation layer that is very flexible and configured using a simple mark up language. The browser ran on top of the protocol called HTTP, which uses TCP/IP as the underlying network protocol. The idea of splitting the presentation layer from the application became a reality as more applications appeared in the browser. The web browser was not an ideal platform for data entry as the HTTP protocol was stateless making it very hard to perform transactions in it. The HTTP protocol could scale. The actual usage involved the exact same concepts as block mode data entry performed on mainframe computers. In a web browser all the data is entered on the screen, and then sent in one go to the application handling the data. The web browser also pushed the idea that the operating system the client is running on is immaterial. The web browsers were ported to Apple computers, Windows, Solaris, and Unix platforms. The web browser also introduced the idea of standard for the presentation layer. All vendors producing a web browser had to conform to the agreed HTML standard. This ensured that anyone building an application that confirmed to HTML would be able to run on any web browser. The web browser pushed the concept that the presentation layer had to run on any client computer (later on, any mobile device as well) irrespective of the operating system and what else was installed on it. The web browser was essentially immune from anything else running on the client computer. If all the client had to use was a browser, maintenance on the client machine would be simplified. HTML had severe limitations and it was not designed for data entry. To address this, the Java language came about and provided the concept of an applet which could run inside the browser, be safe, and provide an interface to the user for data entry. Different vendors came up with different architectures for splitting their two tier application into a three tier one. Oracle achieved this by taking their Oracle Forms product and moving it to the middle application tier, and providing a framework where the presentation layer would run as a Java applet inside the browser. The Java applet would communicate with a process on the application server and it would give it its own instructions for how to draw the display. When the Forms product was replaced with JDeveloper, the same concept was maintained and enhanced. The middle tier became more flexible and multiple middle application tiers could be configured enabling more concurrent users. The three tier architecture has proven to be an ideal environment for legacy systems, giving them a new life and enabling them be put in an environment where they can scale. The three tier environment has a major flaw preventing it from truly scaling. The flaw is the bottleneck between the application layer and the database. The three tier environment also is designed for relational databases. It is not designed for multimedia databases.In the architecture if the digital objects are stored in the database, then to be delivered to the customer they need to pass through the application-database network (exaggerating the bottleneck capacity issues), and from there passed to the presentation layer. Those building in this environment naturally lend themselves to the concept that the best location for the digital objects is the middle tier. This then leads to issues of security, backing up, management, and all the issues previously cited for why storing the digital objects in the database is ideal. The logical conclusion to this is to move the database to the middle tier to address this. In reality, the logical conclusion is to move the application tier back into the database tier. Virtualized architecture In the mid 2000s the idea of a virtualization began to appear in the marketplace. A virtualization was not really a new idea and the concept has existed on the IBM MVS environment since the late 1980s. What made this virtualization concept powerful was that it could run Windows, Linux, Solaris, and Mac environments within them. A virtualized environment was basically the ability to run a complete operating system within another operating system. If the computer server had sufficient power and memory, it could run multiple virtualizations (VMs). We can take the snapshot of a VM, which involves taking a view of the disk and memory and storing it. It then became possible to rollback to the snapshot. A VM could be easily cloned (copied) and backed up. VMs could also be easily transferred to different computer servers. The VM was not tied to a physical server and the same environment could be moved to new servers as their capacity increased. A VM environment became attractive to administrators simply because they were easy to manage. Rather than running five separate servers, an administrator could have the one server with five virtualizations in it. The VM environment entered at a critical moment in the evolution of computer servers. Prior to 2005 most computer servers had one or two CPUs in them. The advanced could have as many as 64 (for example, the Sun E10000), but generally, one or two was the simplest solution. The reason was that computer power was doubling every two years following Moore's law. By around 2005 the market began to realize that there was a limit to the speed of an individual CPU due to physical limitations in the size of the transistors in the chips. The solution was to grow the CPUs sideways and the concept of cores came about. A CPU could be broken down into multiple cores, where each one acted like a separate CPU but was contained in one chip. With the introduction of smart threading, the number of virtual cores increased. A single CPU could now simulate eight or more CPUs. This concept has changed the rules. A server can now run with a large number of cores whereas 10 years ago it was physically limited to one or two CPUs. If a process went wild and consumed all the resources of one CPU, it impacted all users. In the multicore CPU environment, a rogue process will not impact the others. In a VM the controlling operating system (which is also called a hypervisor, and can be hardware, firmware, or software centric) can enable VMs to be constrained to certain cores as well as CPU thresholds within that core. This allows a VM to be fenced in. This concept was taken by Amazon and the concept of the cloud environment formed. This architecture is now moving into a new path where users can now use remote desktop into their own VM on a server. The user now needs a simple laptop (resulting in the demise of the tower computer) to use remote desktop (or equivalent) into the virtualization. They then become responsible for managing their own laptop, and in the event of an issue, it can be replaced or wiped and reinstalled with a base operating system on it. This simplifies the management. As all the business data and application logic is in the VM, the administrator can now control it, easily back it up, and access it. Though this VM cloud environment seems like a good solution to resolving the maintenance scalability issue, a spanner has been thrown in the works at the same time as VMs are becoming popular, so was the evolution of the mobile into a portable hand held device with applications running on it. Mobile applications architecture The iPhone, iPad, Android, Samsung, and other devices have caused a disruption in the marketplace as to how the relationship between the user and the application is perceived and managed. These devices are simpler and on the face of it employ a variety of architectures including two tier and three tier. Quality control of the application is managed by having an independent and separate environment, where the user can obtain their application for the mobile device. The strict controls Apple employs for using iTunes are primarily to ensure that the Trojan code or viruses are not embedded in the application, resulting in a mobile device not requiring a complex and constantly updating anti-virus software. Though the interface is not ideal for heavy data entry, the applications are naturally designed to be very friendly and use touch screen controls. The low cost combined with their simple interface has made them an ideal product for most people and are replacing the need for a laptop in a number of cases. Application vendors that have applications that naturally lend themselves to this environment are taking full advantage of it to provide a powerful interface for clients to use. The result is that there are two architectures today that exist and are moving in different directions. Each one is popular and resolves certain issues. Each has different interfaces and when building and configuring a storage repository for digital objects, both these environments need to be taken into consideration. For a multimedia environment the ideal solution to implement the application is based on the Web. This is because the web environment over the last 15 years has evolved into one which is very flexible and adaptable for dealing with the display of those objects. From the display of digital images to streaming video, the web browser (with sometimes plugins to improve the display) is ideal. This includes the display of documents. The browser environment though is not strong for the editing of these digital objects. Adobe Photoshop, Gimp, Garage Band, Office, and a whole suite of other products are available that are designed to edit each type of digital object perfectly. This means that currently the editing of those digital objects requires a different solution to the loading, viewing and delivery of those digital objects. There is no right solution for the tier architecture to manage digital objects. The N-Tier model moves the application and database back into the database tier. An HTTP server can also be located in this tier or for higher availability it can be located externally. Optimal performance is achieved by locating the application as close to the database as possible. This reduces the network bottleneck. By locating the application within the database (in Oracle this is done by using PL/SQL or Java) an ideal environment is configured where there is no overhead between the application and database. The N-Tier model also supports the concept of having the digital objects stored outside the environment and delivered using other methods. This could include a streaming server. The N-Tier model also supports the concept of transformation servers. Scalability is achieved by adding more tiers and spreading the database between them. The model also deals with the issue of the connection to the Internet becoming a bottleneck. A database server in the tier is moved to another network to help balance the load. For Oracle this can be done using RAC to achieve a form of transparent scalability. In most situations, Tuning, scalability at the server is achieved using manual methods using a form of application partitioning. Basic database configuration concepts When a database administrator first creates a database that they know will contain digital objects, they will be confronted with some basic database configuration questions covering key sizing features of the database. When looking at the Oracle Database there are a number of physical and logical structures built inside the database. To avoid confusion with other database management systems, it's important to note that an Oracle Database is a collection of schemas, whereas in other database management the terminology for a database equates to exactly one schema. This confusion has caused a lot of issues in the past. An Oracle Database administrator will say it can take 30 minutes to an hour to create a database, whereas a SQL Server administrator will say it takes seconds to create a database. In Oracle to create a schema (the same as a SQL Server database) also takes seconds to perform. For the physical storage of tables, the Oracle Database is composed of logical structures called tablespaces. The tablespace is designed to provide a transparent layer between the developer creating a table and the physical disk system and to ensure the two are independent. Data in a table that resides in a tablespace can span multiple disks and disk subsystem or a network storage system. A subsystem equating to a Raid structure has been covered in greater detail at the end of this article. A tablespace is composed of many physical datafiles. Each datafile equates to one physical file on the disk. The goal when creating a datafile is to ensure its allocation of storage is contiguous in that the operating system and doesn't split its location into different areas on the disk (Raid and NAS structures store the data in different locations based on their core structure so this rule does not apply to them). A contiguous file will result in less disk activity being performed when full tablespace scans are performed. In some cases, especially, when reading in very large images, this can improve performance. A datafile is fragmented (when using locally managed tablespaces, the default in Oracle) into fixed size extents. Access to the extents is controlled via a bitmap which is managed in the header of the tablespace (which will reside on a datafile). An extent is based on the core Oracle block size. So if the extent is 128 KB and the database block size is 8 KB, 16 Oracle blocks will exist within the extent. An Oracle block is the smallest unit of storage within the database. Blocks are read into memory for caching, updated, and changes stored in the redo logs. Even though the Oracle block is the smallest unit of storage, as a datafile is an operating system file, based on the type of server filesystem (UNIX can be UFS and Windows can be NTFS), the unit of storage at this level can change. The default in Windows was once 512 bytes, but with NTFS can be as high as 64 KB. This means every time a request is made to the disk to retrieve data from the filesystem it does a read to return this amount of data. So if the Oracle block's size was 8 KB in size and the filesystem block size was 64 KB, when Oracle requests a block to be read in, the filesystem will read in 64 KB, return the 8 KB requested, and reject the rest. Most filesystems cache this data to improve performance, but this example highlights how in some cases not balancing the database block size with the filesystem block size can result in wasted I/O. The actual answer to this is operating system and filesystem dependent, and it also depends on whether Oracle is doing read aheads (using the init.ora parameter db_file_multiblock_read_count). When Oracle introduced the Exadata they put forward the idea of putting smarts into the disk layer. Rather than the database working out how best to retrieve the physical blocks of data, the database passes a request for information to the disk system. As the Exadata knows about its own disk performance, channel speed, and I/O throughput, it is in a much better position for working out the optimal method for extracting the data. It then works out the best way of retrieving it based on the request (which can be a query). In some cases it might do a full table scan because it can process the blocks faster than if it used an index. It now becomes a smart disk system rather than a dumb/blind one. This capability has changed the rules for how a database works with the underlying storage system. ASM—Automated Storage Management In Oracle 10G, Oracle introduced ASM primarily to improve the performance of Oracle RAC (clustered systems, where multiple separate servers share the same database on the same disk). It replaces the server filesystem and can handle mirroring and load balancing of datafiles. ASM takes the filesystem and operating system out of the equation and enables the database administrator to have a different degree of control over the management of the disk system. Block size The database block size is the fundamental unit of storage within an Oracle Database. Though the database can support different block sizes, a tablespace is restricted to one fixed block size. The block sizes available are 4 KB, 8 KB, 16 KB, and 32 KB (a 32 KB block size is valid only on 64-bit platforms). The current tuning mentality says it's best to have one block size for the whole database. This is based on the idea that the one block size makes it easier to manage the SGA and ensure that memory isn't wasted. If multiple block sizes are used, the database administrator has to partition the SGA into multiple areas and assign each a block size. So if the administrator decided to have the database at 8 KB and 16 KB, they would have to set up a database startup parameter indicating the size of each: DB_8K_CACHE_SIZE = 2GDB_16K_CACHE_SIZE = 1G The problem that an administrator faces is that it can be hard to judge memory usage with table usage. In the above scenario the tables residing in the 8 KB block might be accessed a lot more than 16 KB ones, meaning the memory needs to be adjusted to deal with that. This balancing act of tuning invariably results in the decision that unless exceptional situations warrant its use, it's best to keep to the same database blocks size across the whole database. This makes the job of tuning simpler. As is always the case when dealing with unstructured data, the rules change. The current thinking is that it's more efficient to store the data in a large block size. This ensures there is less wasted overhead and fewer block reads to read in a row of data. The challenge is that the size of the unstructured data can vary dramatically. It's realistic for an image thumbnail to be under 4 KB in size. This makes it an ideal candidate to be stored in the row with the other relational data. Even if an 8 KB block size is used, the thumbnail and other relational data might happily exist in the one block. A photo might be 10 MB in size requiring a large number of blocks to be used to store it. If a 16 KB block size is used, it requires about 64 blocks to store 1 MB (assuming there is some overhead that requires overall extra storage for the block header). An 8 KB block size requires about 130 blocks. If you have to store 10 MB, the number of blocks increases 10 times. For an 8 KB block that is over 1300 reads is sufficient for one small-sized 10 MB image. With images now coming close to 100 MB in size, this figure again increases by a factor of 10. It soon becomes obvious that a very large block size is needed. When storing video at over 4 GB in size, even a 32 KB block size seems too small. As is covered later in the article, unstructured data stored in an Oracle blob does not have to be cached in the SGA. In fact, it's discouraged because in most situations the data is not likely to be accessed on a frequent basis. This generally holds true but there are cases, especially with video, where this does not hold true and this situation is covered later. Under the assumption that the thumbnails are accessed frequently and should be cached and the originals are accessed infrequently and should not be cached, the conclusion is that it now becomes practical to split the SGA in two. The unstructured, uncached data is stored in a tablespace using a large block size (32 KB) and the remaining data is stored in a more acceptable and reasonable 8 KB block. The SGA for the 32 KB is kept to a bare minimum as it will not be used, thus bypassing the issue of perceived wasted memory by splitting the SGA in two. In the following table a simple test was done using three tablespace block sizes. The aim was to see if the block size would impact load and read times. The load involved reading in 67 TIF images totaling 3 GB in size. The result was that the tablespace block size made no statistical significant difference. The test was done using a 50-MB extent size and as shown shown in the next segment, this size will impact performance. So to correctly understand how important block size can be, one has to look at not only the block size but also the extent size. Details of the environment used to perform these tests CREATE TABLESPACE tbls_name BLOCKSIZE 4096/8192/16384 EXTENTMANAGEMENT LOCAL UNIFORM SIZE 50M segment space management autodatafile 'directory/datafile' size 5G reuse; The following table compares the various block sizes: Tablespace block size Blocks Extents Load time Read time 4 KB 819200 64 3.49 minutes 1.02 minutes 8 KB 403200 63 3.46 minutes 0.59 minutes 16 KB 201600 63 3.55 minutes 0.59 minutes UNIFORM extent size and AUTOALLOCATE When creating a tablespace to store the unstructured data, the next step after the block size is determined is to work out what the most efficient extent size will be. As a table might contain data ranging from hundreds of gigabytes to terabytes determining the extent size is important. The larger the extent, the potential to possible waste space if the table doesn't use it all is greater. The smaller the extent size the risk is that the table will grow into tens or hundreds of thousands of extents. As a locally managed tablespace uses a bitmap to manage the access to the extents and is generally quite fast, having it manage tens of thousands of extents might be pushing its performance capabilities. There are two methods available to the administrator when creating a tablespace. They can manually specify the fragment size using the UNIFORM extent size clause or they can let the Oracle Database calculate it using the AUTOALLOCATE clause. Tests were done to determine what the optimal fragment size was when AUTOALLOCATE was not used. The AUTOALLOCATE is a more set-and-forget method and one goal was to see if this clause was as efficient as manually setting it. Locally managed tablespace UNIFORM extent size Covers testing performed to try to find an optimal extent and block size. The results showed that a block size of 16384 (16 KB) is ideal, though 8192 (8 KB) is acceptable. The block size of 32 KB was not tested. The administrator, who might be tempted to think the larger the extent size, the better the performance, would be surprised that the results show that this is not always the case and an extent size between 50 MB-200 MB is optimal. For reads with SECUREFILES the number of extents was not a major performance factor but it was for writes. When compared to the AUTOALLOCATE clause, it was shown there was no real performance improvement or loss when used. The testing showed that an administrator can use this clause knowing they will get a good all round result when it comes to performance. The syntax for configuration is as follows: EXTENT MANAGEMENT LOCAL AUTOALLOCATE segment space management auto Repeated tests showed that this configuration produced optimal read/write times without the database administrator having to worry about what the extent size should be. For a 300 GB tablespace it produced a similar number of extents as when a 50M extent size was used. As has been covered, once an image is loaded it is rare that it is updated. A relational database fragmentation within a tablespace is caused by repeated creation/dropping of schema objects and extents of different sizes, resulting in physical storage gaps, which are not easily reused. Storage is lost. This is analogous to the Microsoft Windows environment with its disk storage. After a period of time, the disk becomes fragmented making it hard to find contiguous storage and locate similar items together. Locating all the pieces in a file as close together as possible can dramatically reduce the number of disk reads required to read it in. With NTFS (a Microsoft disk filesystem format) the system administrator can on creation determine whether extents are autoallocated or fragmented. This is similar in concept to the Oracle tablespace creation. Testing was not done to check if the fragmentation scenario is avoided with the AUTOALLOCATE clause. The database administrator should therefore be aware of the tablespace usage and whether it is likely going to be stable once rows are added (in which case AUTOALLOCATE can be used simplifying storage management). If it is volatile, the UNIFORM clause might be considered as a better option. Temporary tablespace For working with unstructured data, the primary uses of the TEMPORARY tablespace is to hold the contents of temporary tables and temporary lobs. A temporary lob is used for processing a temporary multimedia object. In the following example, a temporary blob is created. It is not cached in memory. A multimedia image type is created and loaded into it. Information is extracted and the blob is freed. This is useful if images are stored temporarily outside the database. This is not the same case as using a bfile which Oracle Multimedia supports. The bfile is a permanent pointer to an image stored outside the database. SQL>declareimage ORDSYS.ORDImage;ctx raw(4000);beginimage := ordsys.ordimage.init();dbms_lob.createtemporary(image.source.localdata,FALSE);image.importfrom(ctx, 'file', 'LOADING_DIR', 'myimg.tif');image.setProperties;dbms_output.put_line( 'width x height = ' || image.width ||'x' || image.height);dbms_lob.freetemporary(image.source.localdata);end;/width x height = 2809x4176 It's important when using this tablespace to ensure that all code, especially on failure, performs a dbms_lob.freetemporary function, to ensure that storage leakage doesn't occur. This will result in the tablespace continuing to grow until it runs out of room. In this case the only way to clean it up is to either stop all database processes referencing, then resize the datafile (or drop and recreate the temporary tablespace after creating another interim one), or to restart the database and mount it. The tablespace can then be resized or dropped and recreated. UNDO tablespace The UNDO tablespace is used by the database to store sufficient information to rollback a transaction. In a database containing a lot of digital objects, the size of the database just for storage of the objects can exceed terabytes. In this situation the UNDO tablespace can be sized larger giving added opportunity for the database administrator to perform flashback recovery from user error. It's reasonable to size the UNDO tablespace at 50 GB even growing it to 100 GB in size. The larger the UNDO tablespace the further back in time the administrator can go and the greater the breathing space between user failure, user failure detected and reported, and the database administrator doing the flash back recovery. The following is an example flashback SQL statement. The as of timestamp clause tells Oracle to find rows that match the timestamp from the current time going back so that we can have a look at a table an hour ago: select t.vimg.source.srcname || '=' ||dbms_lob.getlength(t.vimg.source.localdata)from test_load as of timestamp systimestamp - (1/24) t; SYSTEM tablespace The SYSTEM tablespace contains the data dictionary. In Oracle 11g R2 it also contains any compiled PL/SQL code (where PLSQL_CODE_TYPE=NATIVE). The recommended initial starting size of the tablespace should be 1500 MB. Redo logs The following test results highlight how important it is to get the size and placement of the redo logs correct. The goal was to determine what combination of database parameters and redo/undo size were optimal. In addition, an SSD was used as a comparison. Based on the result of each test, the parameters and/or storage was modified to see whether it would improve the results. When it appeared an optimal parameter/storage setting was found, it was locked in while the other parameters were tested further. This enabled multiple concurrent configurations to be tested and an optimal result to be calculated. The test involved loading 67 images into the database. Each image varied in size between 40 to 80 MB resulting in 2.87 GB of data being loaded. As the test involved only image loading, no processing such as setting properties or extraction of metadata was performed. Archiving on the database was not enabled. All database files resided on hard disk unless specified. In between each test a full database reboot was done. The test was run at least three times with the range of results shown as follows: Database parameter descriptions used:Redo Buffer Size = LOG_BUFFERMultiblock Read Count = db_file_multiblock_read_count Source disk Redo logs Database parameters Fastest time Slowest time Hard disk Hard disk 3 x 50 MB Redo buffer size = 4 MB Multiblock read count = 64 UNDO tablespace on HD (10 GB) Table datafile on HD 3 minutes and 22 sec 3 minutes and 53 sec Hard disk Hard disk 3 x 1 GB Redo buffer size = 4 MB Multiblock read count = 64 UNDO tablespace on HD (10 GB) Table datafile on HD 2 minutes and 49 sec 2 minutes and 57 sec Hard disk SSD 3 x 1 GB Redo buffer size = 4 MB Multiblock read count = 64 UNDO tablespace on HD (10 GB) Table datafile on HD 1 minute and 30 sec 1 minute and 41 sec Hard disk SSD 3 x 1 GB Redo buffer size = 64 MB Multiblock read count = 64 UNDO tablespace on HD (10 GB) Table datafile on HD 1 minute and 23 sec 1 minute and 48 sec Hard disk SSD 3 x 1 GB Redo buffer size = 8 MB Multiblock read count = 64 UNDO tablespace on HD (10 GB) Table datafile on HD 1 minute and 18 sec 1 minute and 29 sec Hard disk SSD 3 x 1 GB Redo buffer size = 16 MB Multiblock read count = 64 UNDO tablespace on HD (10 GB) Table datafile on HD 1 minute and 19 sec 1 minute and 27 sec Hard disk SSD 3 x 1 GB Redo buffer size = 16 MB Multiblock read count = 256 UNDO tablespace on HD (10 GB) Table datafile on HD 1 minute and 27 sec 1 minute and 41 sec Hard disk SSD 3 x 1 GB Redo buffer size = 8 MB Multiblock read count = 64 UNDO tablespace = 1 GB on SSD Table datafile on HD 1 minute and 21 sec 1 minute and 49 sec SSD SSD 3 x 1 GB Redo buffer size = 8 MB Multiblock read count = 64 UNDO tablespace = 1 GB on SSD Table datafile on HD 53 sec 54 sec SSD SSD 3 x 1 GB Redo buffer size = 8 MB Multiblock read count = 64 UNDO tablespace = 1 GB on SSD Table datafile on SSD 1 minute and 20 sec 1 minute and 20 sec Analysis The tests show a huge improvement when the redo logs were moved to a Solid State Drive (SSD). Though the conclusion that can be drawn is this: the optimal step to perform it might be self defeating. A number of manufacturers of SSD acknowledge there are limitations with the SSD when it comes to repeated writes. The Mean Time to Failure (MTF) might be 2 million hours for reads; for writes the failure rate can be very high. Modern SSD and flash cards offer much improved wear leveling algorithms to reduce failures and make performance more consistent. No doubt improvements will continue in the future. A redo log by its nature is constant and has heavy writes. So, moving the redo logs to the SSD might quickly result in it becoming damaged and failing. For an organization that on configuration performs one very large load of multimedia, the solution might be to initially keep the redo logs on SSD, and once the load is finished, to move the redo logs to a hard drive. Increasing the size of the redo logs from 50 MB to 1 GB improves performance and all database containing unstructured data should have a redo log size of at least 1 GB. The number of logs should be at least 10; preferred is from 50 to 100. As is covered later, disk is cheaper today than it once was, and 100 GB of redo logs is not that large a volume of data as it once was. The redo logs should always be mirrored. The placement or size of the UNDO tablespace makes no difference with performance. The redo buffer size (LOG_BUFFER) showed a minor improvement when it was increased in size, but the results were inconclusive as the figures varied. A figure of LOG_BUFFER=8691712, showed the best results and database administrators might use this figure as a starting point for tuning. The changing of multiblock read count (DB_FILE_MULTIBLOCK_READ_COUNT) from the default value of 64 to 256 showed no improvement. As the default value (in this case 64) is set by the database as optimal for the platform, the conclusion that can be drawn is that the database has set this figure to be a good size. By moving the original images to an SSD showed another huge improvement in performance. This highlighted how the I/O bottleneck of reading from disk and the writing to disk (redo logs) is so critical for digital object loading. The final test involved moving the datafile containing the table to the SSD. It highlighted a realistic issue that DBAs face in dealing with I/O. The disk speed and seek time might not be critical in tuning if the bottleneck is the actual time it takes to transfer the data to and from the disk to the server. In the test case the datafile was moved to the same SSD as the redo logs resulting in I/O competition. In the previous tests the datafile was on the hard disk and the database could write to the disk (separate I/O channel) and to the redo logs (separate I/O channel) without one impacting the other. Even though the SSD is a magnitude faster in performance than the disk, it quickly became swamped with calls for reads and writes. The lesson is that it's better to have multiple smaller SSDs on different I/O channels into the server than one larger channel. Sites using a SAN will soon realize that even though SAN might offer speed, unless it offers multiple I/O channels into the server, its channel to the server will quickly become the bottleneck, especially if the datafiles and the images for loading are all located on the server. The original tuning notion of separating data fi les onto separate disks that was performed more than 15 years ago still makes sense when it comes to image loading into a multimedia database. It's important to stress that this is a tuning issue while dealing with image loading not when running the database in general. Tuning the database in general is a completely different story and might result in a completely different architecture.
Read more
  • 0
  • 0
  • 9514

article-image-article-making-money-your-game
Packt
17 May 2013
23 min read
Save for later

Making Money with Your Game

Packt
17 May 2013
23 min read
(For more resources related to this topic, see here.) Your game development strategy If you want to build a game to make some money, it is imperative you take a few things into consideration before starting off building one. The first question you will need to ask yourself is probably this: who am I going to build a game for? Are you aiming at everyone capable of playing games or do you want to target a very specific segment of people and meet their gaming needs? This is the difference between broad and niche targeting. An example of very broadly targeted games are most tower defense games, in which you need to build towers with diverse properties to repel an army. Games such as Tetris, Bejeweled, Minesweeper, and most light puzzle games in general. Angry Birds is another example of a game that is popular with a broad audience because of its simplicity, likeable graphics, and incredible amount of clever marketing. Casual games in general seem to appeal to the masses because of the following few factors: Simplicity prevails: most gamers get used to the game in mere minutes. There are little or no knowledge prerequisites: you are not expected to already know some of the background story or have experience in these types of games. Casual gamers tend to do well even though they put in less time to practice. Even if you do well from the start, you can still become better at it. A game at which you cannot become better by replaying doesn't hold up for long. Notable exceptions are games of chance like roulette and the slots, which do prove to be addictive; but that is for other reasons, such as the chance to win money. The main advantage to building casual games is that practically everyone is a potential user of your game. As such, the achievable success can be enormous. World of Warcraft is a game which has moved from rather hardcore and niche to more casual over the years. They did this because they had already reached most regular gamers out there, and decided to convince the masses that you can play World of Warcraft even if you don't play a lot in general. The downside of trying to please everyone is the amount of competition. Sticking out as a unique game among numerous games out there is tremendously difficult. This is especially true if you don't have an impressive marketing machine to back it up. A good example of a niche game is any game built after a movie. Games on Star Trek, Star Wars, Lord of the Rings, and so on, are mostly aimed at people who have seen and liked those movies. Niche games can also be niche because they are targeted solely at a specific group of gamers. For example, people who prefer playing FPS (First Person Shooters) games, and do so every single day. In essence, niche games have the following properties (note that they oppose casual or broadly targeted games): Steep learning curve: mastery requires many hours of dedicated gaming. Some knowledge or experience with games is required. An online shooter game such as PlanetSide 2 requires you to have at least some experience with shooter games in the past, since you are pitted against people who know what they are doing. The more you play the game, the more are the useful rewards you get. Playing a lot is often rewarded with items that make you even stronger in the game, thus reinforcing the fact that you already became better by playing more. StarCraft is a game released by Blizzard in 1998 and is still being played in tournaments today, even though there is a follow up: StarCraft 2. Games such as the original StarCraft are perfectly feasible to be built in HTML5 and run on a browser or smartphone. When StarCraft was released, the average desktop computer had less power than many smartphones have today. Technically, the road is open; replicating the same level of success is another matter though. The advantage of aiming at a niche of gamers is the unique position you can take in their lives. Maybe there are not many people in your target group, but it will be easier to get and keep their attention with your game since it is specifically built for them. In addition, it doesn't mean that because you have a well-defined aim, gamers can't drop in from unexpected corners. People you would have never thought would play your game can still like what you did. It is for this reason that knowing your gamers is so important, and why tools such as Playtomic exist. The disadvantage of the niche marketing is obvious: your game is very unlikely to grow beyond a certain point; it will probably never rank among the most played games in the world. The type of games you are going to develop is one choice, the amount of detail you put in each one of them is another. You can put as much effort into building a single game as you want. In essence, a game is never finished. A game can always have an extra level, Easter egg, or another nice little detail. When scoping your game, you must have decided whether you will use a shotgun or a sniper development strategy. In a shotgun strategy, you develop and release games quickly. Each game still has a unique element that should distinguish it from other games: the UGP (Unique Gaming Proposition). But games released under a shotgun strategy don't deliver a lot of details; they are not polished. The advantages of adopting a shotgun strategy are plenty: Low development cost; thus every game represents a low risk The short time to market allows for using world events as game settings You have several games on the market at once, but often you only need a single one to succeed in order to cover the expenses incurred for the others Short games can be given to the public for free but monetized by selling add-ons, such as levels However, it's not just rainbows and sunshine when you adopt this strategy. There are several reasons why you wouldn't choose shotgun strategy: A game that doesn't feel finished has less chance of being successful than a perfected one. Throwing your game on the market not only tests whether a certain concept works, but also exposes it to the competitors, who can now start building a copy. Of course, you have the first mover advantage, but it's not as big as it could have been. You must always be careful not to throw garbage on the market either, or you might ruin your name as a developer. However, don't get confused. The shotgun strategy is not an excuse for building mediocre games. Every game you release should have an original touch to it— something that no other game has. If a game doesn't have that new flavor, why would anyone prefer it over all the others? Then, of course, there is the sniper strategy, which involves building a decent and well-thought-out game and releasing it to the market with the utmost care and support. This is what distributors such as Apple urge developers to do, and for good reason—you wouldn't want your app store full of crappy games, would you? Some other game distributers, such as Steam, are even pickier in the games they allow to be distributed, making the shotgun strategy nearly impossible. But it is also the strategy most successful game developers use. Take a look at developers such as Rockstar (developer of the GTA series), Besthesda (developer of the Elder Scroll series), Bioware (developer of the Mass Effect series), Blizzard (developer of the Warcraft series), and many others. These are no small fries, yet they don't have that many games on the market. This tactic of developing high quality games and hoping they will pay off is obviously not without risk. In order to develop a truly amazing game, you also need the time and money to do so. If your game fails to sell, this can be a real problem for you or your company. Even for HTML5 games, this can be the case, especially since devices and browsers keep getting more and more powerful. When the machines running the games get more powerful, the games themselves often become more complex and take longer to develop. We have taken a look at two important choices one needs to make when going into the game-developing business. Let's now look at the distribution channels that allow you to make money with your game, but before that let's summarize the topic we have just covered: Deciding who you want to target is extremely important even before you start developing your game. Broad targeting is barely targeting at all. It is about making a game accessible and likeable by as many people as possible. Niche targeting is taking a deeper look and interest in a certain group of people and building a game to suit their specific gaming needs. In developing and releasing games, there are two big strategies: shotgun and sniper. In the shotgun strategy you release games rapidly. Each game still has unique elements that no other games possess, but they are not as elaborate or polished as they could be. With the sniper strategy you build only a few games, but each one is already perfected at the time of release and only needs slight polishing when patches are released for it. Making money with game apps If you built your game into an app, you have several distribution channels you can turn to, such as Firefox marketplace, the IntelAppUp center, Windows Phone Store, Amazon Appstore, SlideMe, Mobango, Getjar, and Apple Appsfire. But the most popular players on the market currently are Google Play and the iOS App Store. The iOS App Store is not to be confused with the Mac app store. iPad and Mac have two different operating systems, iOS and Mac OS, so they have separate stores. Games can be released both in the iOS and the Mac store. There could also be some confusion between Google Play and Chrome Web Store. Google Play contains all the apps available for smartphones that have Google's Android operating system. The Chrome Web Store lets you add apps to your Google Chrome browser. So there are quite a few distribution channels to pick from and here we will have a quick look at Google Play, the iOS App store, and the Chrome Web Store. Google Play Google Play is the Android's default app shop and the biggest competitor of the iOS App Store. If you wish to become an Android app developer, there is a $25 fee and a developer distribution agreement that you must read. In return for the entrance fee and signing this agreement, they allow you to make use of their virtual shelves and have all its benefits. You can set your price as you please, but for every game you sell, Google will cash in about 30 percent. It is possible to do some geological price discrimination. So you could set your price at, let's say €1 in Belgium, while charging €2 in Germany. You can change your price at any time; however, if you release a game for free, there is no turning back. The only way to monetize this app afterwards is by allowing in game advertising, selling add-ons, or creating an in-game currency that can be bought with real money. Introducing an in-game currency that is bought with real money can be a very appealing format. A really successful example of this monetizing scheme can be found in The Smurfs games. In this game you build your own Smurf village, complete with Big Smurf, Smurfette, and a whole lot of mushrooms. Your city gets bigger as you plant more crops and build new houses, but it is a slow process. To speed it up a bit you can buy special berries in exchange for real money, which in turn allows you to build exclusive mushrooms and other things. This monetization scheme becomes very popular as has been shown in games such as League Of Legends, PlanetSide 2, World of Tanks, and many others. For Google Play apps, this in-app payment system is supported by Android's Google Checkout. In addition, Google allows you access to some basic statistics for your game, such as the number of players and the devices on which they play, as shown in the following diagram: Image Information such as this allows you to redesign your game to boost your success. For example, you could notice if a certain device doesn't have that many unique users, even though it is a very popular device and is bought by many people. If this is the case, maybe your game doesn't looks nice on this particular smartphone or tablet and you should optimize for it. The biggest competitor and initiator of all apps is the iOS App Store, so let's have a look at this. iOS App Store The iOS App Store was the first of its kind and at the time of writing this book, it still has the biggest revenue. In order to publish apps in the iOS App Store, you need to subscribe to the iOS Developer Program, which costs $99 annually—almost four times the subscription fee of Google Play. In effect, they offer about the same thing as Google Play does; as you can see in this short list: You pick your own prices and get 70 percent of sales revenue You receive monthly payments without credit card, hosting, or marketing fees There is support and adequate documentation to get you started More importantly, here are the following differences between Google Play and the iOS App Store: As mentioned earlier, signing up for Google Play is cheaper. The screening process of Apple seems to be more strict than that of Google Play, which results in a longer time to reach the market and a higher chance of never even reaching the market. Google Play incorporates a refund option that allows the buyer of your app to be refunded if he or she uninstalls the app or game within 24 hours. If you want your game to make use of some Android core functionalities, this is possible since the platform is open source. Apple, on the other hand, is very protective of its iOS platform and doesn't allow the same level of flexibility for apps. This element might not seem that important for games just yet, but it might be for very innovative games that do want to make use of this freedom. iOS reaches more people than Android does, though the current trend indicates that this might change in the near future. There seem to be significant differences in the kind of people buying Apple devices and the users of smartphones or tablets with Android OS. Apple fans tend to have a lower barrier towards spending money on their apps than Android users do. In general, iPads and iPhones are more expensive than other tablets and smartphones, attracting people who have no problem with spending even more money on the device. This difference in target group seems to make it more difficult for Android game developers to make money from their games. The last option for selling apps that we will discuss here is the Chrome Web Store. The Chrome Web Store The Chrome Web Store differs from Google Play and the iOS App Store in that it provides apps specifically for the Chrome browser and not for mobile devices. The Chrome Store offers web apps. Web apps are like applications you would install on your PC, except web apps are installed in your browser and are mostly written using HTML, CSS, and JavaScript, like our ImpactJS games. The first thing worth noticing about the Chrome Store is the one-time $5 entrance fee for posting apps. If this in itself is not good enough, the transaction fee for selling an app is only 5 percent. This is a remarkable difference to both Google Play and the iOS App Store. If you are already developing a game for your own website and packaged it as an app for Android and/or Apple, you can just as well launch it on the Chrome Web Store. Turning your ImpactJS game into a web app for the Chrome Store can be done using AppMobi, yet Google itself provides detailed documentation on how to do this manually. One of the biggest benefits of the web app is the facilitation of the permission process. Let's say your web app needs the location of the user in order to work. While iPad apps ask for permission every time they need location data, the web app asks permission only once: at installation time. Furthermore, you have the same functionalities and payment modalities as in Google Play, give or take. For instance, there is also the option to incorporate a free trial version, also known as a freemium. A freemium model is when you allow downloading a demo version for free with the option of upgrading it to a full version for a price. The Smurfs game also uses the freemium model, albeit with a difference. The entire game is free, but players can opt to pay real money to buy things that would otherwise cost them a lot of time to acquire. In this freemium model, you pay for convenience and unique items. For instance, in PlanetSide 2, acquiring a certain sniper rifle might take you several days or $10, depending on how you choose to play the freemium game. If you plan on releasing an ImpactJS game for Android, there is no real reason why you wouldn't do so for the Chrome Web Store. That being said, let's have a quick recap: The time when the iOS app store was the only app store out there is long gone; there is an impressive repertoire of app stores to choose from, which includes Firefox Marketplace, Intel AppUp Center, Windows Phone Store, Amazon Appstore, SlideMe, Mobango, Getjar, Appsfire, Google Play, among others. The biggest app stores are currently Google Play and the iOS App Store. They differ greatly on several fronts, of which the most important ones are: Subscription fee Screening process Type of audience they attract The Chrome Web Store sells web apps that act like normal apps but are available in the Chrome browser. The Chrome Web Store is cheap and easy to subscribe to. You must definitely have a go at releasing your game on this platform. In-game advertising In-game advertising is another way to make money with your game. In-game advertising is a growing market and is currently already being used by major companies; it was also used by Barack Obama in both his 2008 and 2012 campaigns, as shown in the following in-game screenshot: Image There is a trend towards more dynamic in-game advertising. The game manufacturers make sure there is space for advertising in the game, but the actual ads themselves are decided upon later. Depending on what is known about you, these can then change to become relevant to you as a player and a real-life consumer. When just starting out to build games, in-game adverting doesn't get that spectacular though. Most of the best known in-game advertisers for online games don't even want their ads in startup games. The requirements for Google AdSense are the following: Game plays: Minimum 500,000 per day Game types: Web-based Flash only Integration: Must be technically capable of SDK integration Traffic source: Eighty percent of traffic must be from the US and the UK Content: Family-safe and targeted at users aged 13 and over Distribution: Must be able to report embedded destinations and have control over where the games are distributed The requirements for another big competitor, Ad4Game, aren't mild either: At least 10,000 daily unique visitors Sub-domains and blogs are not accepted The Alexa rank should be less than 400,000 Adult/violent/racist content is not allowed If you are just starting out, these prerequisites are not good news. Not only because you need such a large number of players before even starting the advertising, but also because currently all support goes to Flash games. HTML5 games are not fully supported yet, though that will probably change. Luckily, there are companies out there that do allow you to start using advertising even though you don't have 10,000 visitors a day. Tictacti is one of those companies. Once again, almost all support goes to the Flash games, but they do have one option available for an HTML5 game:pre-roll. Pre-roll simply means that a screen with an ad appears before you can start the game. Integration of the pre-roll advertising is rather straightforward and does not require a change to your game, but to your index.html file, as in the following example from Tictacti: code While adding this to your game's index.html file, you fill out your own publisher ID and you are basically ready to go. Tictacti is similar to Google Analytics, and it also provides you with some relevant information about the ads on your game's website, as shown in the following diagram: Image Be careful, however, pre-roll advertising is one of the most intruding and annoying kinds of advertising. Technically, it is not even in-game advertising at all, since it runs before you play the game. If your game is not yet well established enough to convince the player to endure the advertising before being able to play, don't choose this option. Give your game some time to build a reputation before putting your gamers through this. As the last option, we will have a look at selling your actual distribution rights with MarketJS. But let's first briefly recap on in-game advertising: In-game advertising is a growing market. Even Barack Obama made use of in-game billboards to support his campaign. There is a trend towards more dynamic in-game advertising—using your location and demographic information to adapt the ads in a game. Currently, even the most accessible companies that offer online in-game advertising are focused on Flash games and require many unique visitors before even allowing you to show their ads. Tictacti is a notable exception, as it has low prerequisites and an easy implementation; though advertising is currently limited to pre-roll ads. Always take care to first build a positive reputation for your game and allow advertisements later. Selling distribution rights with MarketJS The last option we will investigate in this chapter is selling your game's distribution rights. You can still make money by just posting your game to all the app stores and on your own website, but it becomes increasingly difficult to be noticed. Quality can only prevail if people know it is out there, thus making a good game is sometimes not enough—you need marketing. If you are a beginner game builder with great game ideas and the skills to back it up, that's great, but marketing may not be your cup of tea. This is where MarketJS comes into play. MarketJS acts as an intermediate between you as the game developer and the game publishers. The procedure is simple once you have a game: You sign up on their website, http://www.marketjs.com. Upload the game on your own website or directly to the MarketJS server. Post your game for publishers to see. You set several options such as the price and contract type that would suit you the best. You have five contract options: Complete distribtution contract: Sell all your distribution rights to the game. Exclusive distribution partner contract: Here you restrict yourself to work with one distributor but still retain the rights to the game. Non-exclusive contract: Here, any distributor can buy the rights to use your game, but you can go on selling rights as long as you want. Revenue share: Here you negotiate on how to split the revenues derived from the game. Customized contract: This can basically have any terms. You can choose this option if you are not sure yet what you want out of your game. A part of the webpage on which you fill out your contracting preferences is shown in the following screenshot: Image After you have posted a demo, it is a matter of waiting for a publisher to spot it, get stunned by its magnificence, and offer to work with you. The big contribution of MarketJS to the gaming field is this ability to let the game developer focus on developing games. Someone else takes care of the marketing aspect, which is a totally different ballgame. MarketJS also offers a few interesting statistics such as the average price of a game on their website, as shown in the following diagram. It grants you some insight on whether you should take up game developing as a living or keep doing it as a hobby. Image According to MarketJS, prices for non-exclusive rights average between $500 and $1000, while selling exclusive rights to a game range somewhere between $1500 and $2000. If you can build a decent game within this price range, you are more than ready to go: MarketJS is a company that brings game distributers and developers closer together. Their focus is on HTML5 games, so they are great if you are a startup ImpactJS game developer. They require no subscription fee and have a straightforward process to turn your game into a showcase with a price tag. Summary In this article we have taken a look at some important elements while considering your game development strategy. Do you wish to adapt a shotgun approach and develop a lot of games in a short time span? Or will you use the sniper strategy and only build a few, but very polished games? You also need to decide upon the audience you wish to reach out to with your game. You have the option of building a game that is liked by everyone, but the competition is steep. Making money on the application stores is possible, but for Android and Apple there are registration fees. If you decide to develop apps, it is worth giving the Chrome Web Store a try (it runs web apps). In-game advertising is another way to fund your efforts, though most companies offering this service for online games have high prerequisites and support Flash games more than they do the newer HTML5 games. One of the most promising of monetization schemes is the freemium model. Players are allowed to freely play your game but they pay real money for extras. This is an easily tolerated model since the game is essentially free to play for anyone not willing to spend money, and no annoying advertising is present either. A combination of in-game advertising and freemium is possible as well: people annoyed by the advertising pay a fee and in return they are not bothered by it anymore. A final option is leaving the marketing aspect to someone else by selling your distribution rights with the help of MarketJS. They aim for HTML5 games and this option is especially useful for the beginner game developer who has difficulty in marketing his or her game. Resources for Article : Further resources on this subject: HTML5: Developing Rich Media Applications using Canvas [Article] Flash 10 Multiplayer Game: The Lobby and New Game Screen Implementation [Article] Building HTML5 Pages from Scratch [Article]
Read more
  • 0
  • 0
  • 20528

article-image-analyzing-network-forensic-data-become-expert
Packt
17 May 2013
6 min read
Save for later

Analyzing network forensic data (Become an expert)

Packt
17 May 2013
6 min read
(For more resources related to this topic, see here.) Getting ready It's not new to use tunneling techniques to attempt to circumvent firewalls or other countermeasures that do not make use of deep packet inspection. Tools such as iodine, Hping3, Netcross, and so on, can be used to encapsulate certain type of traffic within the payload of another protocol and thus get its contents hidden. The use of these tools can be tedious for network administrators who try to control the information entering and leaving their network. Tshark can be an effective tool to detect this type of traffic if it's done wisely. How to do it... So suppose we are performing a periodic analysis of our network. We decide to start looking at the protocols used to make connections with the outside. Note that with the condition ip.addr! = 192.168.1.0/24 we will ensure that Tshark displays packets involving communication between our LAN and any other network (in this case the outside because we do not have more subnets). This way, we are also filtering connections between hosts on our LAN. bmerino@Mordor:/$ tshark -r icmp.pcap -o column.format:'"Protocol","%p"' -R "ip.addr != 192.168.1.0/24" | sort -uDNSHTTPICMPSSLTCPTLSv1 According to our policies (only allowed HTTP/HTTPS, DNS, and ICMP), we can see that all protocols seem normal. Since ICMP can result from a variety of network issues we decided to run the following: We know now that a user of our network is communicating with an external host via ICMP. What is striking about this communication is the size of those packets. To see exactly what kind of ICMP packets they are sending, we run: bmerino@Mordor:/$ tshark -r icmp.pcap -T fields -e icmp.code -eicmp.type -R 'frame.number==41628'0 8 The user is sending ICMP echo request (type = 8/code = 0) of a size well above a standard ping. To know for sure the reason for that size we do a dump of one packet to see its payload: With the dump, we verified that the user was leaking information to the outside via ICMP packets. In the example the user is sending the RSA private key of a certain user/service. We realized that all payloads started with "BOF" to indicate the beginning of the file content. Likewise the "EOF" string was being used to indicate the end. With this data we could find out that the user was using the ICMP Exfiltration module from Metasploit to get all kinds of files from the internal host. We could take advantage of the BOF string to list all files sent by that user (the external IP has been omitted). bmerino@Mordor:/$ tshark -r icmp.pcap -R "icmp contains BOF &&ip.src == 192.168.1.42" This would generate the following output: 41591 764.064922000 192.168.1.42 -> *.*.*.*ICMP 938 Echo (ping)request id=0x1b3d, seq=1/256, ttl=6441593 765.065198000 192.168.1.42 -> *.*.*.*ICMP 938 Echo (ping)request id=0x1b3d, seq=3/768, ttl=64 Although most files were plaintext, one of them looked as follows: bmerino@Mordor:/$ tshark -r icmp.pcap -R "frame.number==268" -V |grep ^0 | cut -d" " -f21 | tr -d 'nBOF'TVqQAAMAAAAEAAAA//8AALgAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAAAAA4fug4AtAnNIbgTM0hVGhpcywcm9ncmtIGNhbm5vdCiZSydW4gaW4gRE9TIG1vZGUuDQ0KJAAAAAAAAAQRQAATAEEALmrjQAAAAAAAAAAAADwELAQUAAJgAAAiAAAAAAAAAEwAAAAQAAAAs This seems to indicate that he was sending a file encoded in base 64, possibly to deal with nonprintable characters. Leveraging the output and using a little bash we get the following: bmerino@Mordor:/$ tshark -r icmp.pcap -R "frame.number==268" -V |grep ^0 | cut -d" " -f21 | tr -d 'n' | sed 's/^...//' | base64 -dMZ ? ? ? ? @ ? ? ? ? ! ? L ? !This program cannot be run in DOS mode.$PEL ? 4 ?? bL ? @ 0 < ? ! d .textp ? ? `.rdata ? Effectively the output indicated part of the PE header of an executable file. Now we can dump it to a file and analyze it with our favorite debugger. There's more... Let's see another case. The internal server (192.168.1.130/24) of a certain organization has been compromised several times. The fact that the server is not accessible from the outside led us to think that an internal user had also been compromised, and the attacker was doing "pivoting" from that machine to the server. Another option was that an infected USB had been the root of the problem. However, after checking that the firewall did not filter the traffic DMZ ---> Internal Network, we could confirm that this was the origin of the intrusion. The web server (192.168.20.1/24) in the DMZ was compromised, and from here the attacker could access the internal LAN using the Administrator account. To do this the attacker used Mimikatz to get the credentials of that machine and then ran Psexec against the internal server with the Administrator account. We realized this when we checked the traffic from DMZ to the Internal LAN as follows (note the use of the flag SYN = 1 and ACK = 0 to show only connections initiated from the DMZ): bmerino@Mordor:~$ tshark -r pivo.pcap -T fields -e ip.src -R "ip.src ==192.168.20.0/24 && ip.dst==192.168.1.130 && tcp.flags.syn==1 && tcp.flags.ack==0" | sort -u192.168.20.1 According to this information the machine 192.168.20.1 (the web server) started some kind of connection with the internal server. The following output shows an excerpt of the type of traffic generated between both machines: bmerino@Mordor:~$ tshark -r pivo.pcap -o column.format:'"Info","%i","Protocol","%p"' -R "ip.src == 192.168.20.1 && ip.dst==192.168.1.130" | head-4 This would generate the following output: Tree Connect AndX Request, Path: 192.168.1.130IPC$ SMBRedirect (Redirect for host) ICMP[TCP Retransmission] Tree Connect AndX Request, Path: 192.168.1.130IPC$ SMBTrans2 Request, QUERY_PATH_INFO, Query File Basic Info, Path: PSEXESVC.EXE SMB As shown in the output, the web server initiated a NETBIOS connection with the internal server (something totally suspicious). From there, he ran Psexec to authenticate to the server machine. You can also see the plaintext password and the user used to login the server (surely he uses Psexec with -u option, which sends the password in clear text): bmerino@Mordor:~$ tshark -r pivo.pcap -x "ip.src == 192.168.20.1 &&ip.dst==192.168.1.130" | grep "A.D.M.l" -m 1 -A 1 | awk -F " " '{print$3}' This would generate the following output: A.D.M.l.o.c.1.0.1............... Note that there will be more complex cases that will require much deeper research. Take a look, for example, at this article at http://www.securityartwork.es/2012/11/27/ covert-channels-2/?lang=en, which explains some covert-channel techniques with a few TCP and IP fields. In such cases, the joint use of debuggers, traffic analysis tools, and event correlation will be our best ally. Summary This article explains how to obtain evidence from suspicious network traffic. We will look at tunneling techniques to attempt to circumvent security mechanisms (ICMP exfiltration, UDP tunnels, and so on) in addition to other post-exploitation attacks Resources for Article : Further resources on this subject: Linux4afrika: An Interview with the Founder [Article] HP Network Node Manager 9: Understanding Smart Plug-Ins [Article] Oracle Business Intelligence : Getting Business Information from Data [Article]
Read more
  • 0
  • 0
  • 4013
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at €18.99/month. Cancel anytime
article-image-integration-system-center-operations-manager-2012-sp1
Packt
17 May 2013
9 min read
Save for later

Integration with System Center Operations Manager 2012 SP1

Packt
17 May 2013
9 min read
(For more resources related to this topic, see here.) This article provides tips and techniques to allow administrators to integrate Operations Manager 2012 with Virtual Machine Manager 2012 to monitor the health and performance of virtual machine hosts and their virtual machines, as well as to use the Operations Manager reporting functionality. In a hybrid hypervisor environment (for example, Hyper-V, VMware), using Operations Manager management packs ( MPs ) (for example, Veeam MP), you can monitor the Hyper-V hosts and the VMware hosts, which allow you to use only the System Center Console to manage and monitor the hybrid hypervisor environment. You can also monitor the health and availability of the VMM infrastructure, management, database, and library servers. The following screenshot will show you the diagram views of the virtualized environment through the Operations Manager: Installing System Center Operations Manager 2012 SP1 This recipe will guide you through the process of installing a System Center Operations Manager for the integration with VMM. Operations Manager has an integrated product and company knowledge for proactive tuning. It also allows the user to compute the OS, applications, services, and out-of-the-box network monitoring, reporting, and many more features extensibility through management packs, thus providing a cross-platform visibility. The deployment used on this recipe assumes a small environment with all components being installed on the same server. For datacenters and enterprise deployments, it is recommended to distribute the features and services across multiple servers to allow for scalability. For a complete design reference and complex implementation of SCOM 2012, follow up the Microsoft Operations Manager deployment guide available at http://go.microsoft.com/fwlink/?LinkId=246682. When planning, use Operations Guide for System Center 2012—Operations Manager (http://go.microsoft.com/fwlink/p/?LinkID=207751) to determine the hardware requirements. Getting ready Before starting, check out the system requirements and design planning for System Center Operations Manager 2012 SP1 at http://technet.microsoft.com/en-us/library/jj656654.aspx My recommendation is to deploy on a Windows Server 2012 and the SQL Server 2012 SP1 version. How to do it... Carry out the following steps to install Operations Manager 2012 SP1: Browse to the SCOM installation folder and click on Setup. Click on Install. On the Select the features to install page, select the components that apply to your environment, and then click on Next as shown in the following screenshot: The recommendation is to have a dedicated server, but it all depends on the size of the deployment. You can select all of the components to be installed on the same server for a small deployment. Type in the location where you'd install Operations Manager 2012 SP1, or accept the default location and click on Next. The installation will check if your system has passed all of the requirements. A screen showing the issues will be displayed if any of the requirements are not met, and you will be asked to fix and verify it again before continuing with the installation, as shown in the following screenshot: If all of the prerequisites are met, click on Next to proceed with the setup. On the Specify an installation option page, if this is the first Operations Manager, select the Create the first Management Server in a new management group option and provide a value in the Management group name field. Otherwise, select the Add a management server to an existing management group option as shown in the following screenshot: Click on Next to continue, accept the EULA, and click on Next. On the Configure the operational database page, type the server and instance name of the server and the SQL Server port number. It is recommended to keep the default values in the Database name, Database size (MB), Data file folder, and Log file folder boxes. Click on Next. The installation account needs DB owner rights on the database. On the SQL Server instance for Reporting Services page, select the instance where you want to host the Reporting Services (SSRS). Make sure the SQL Server has the SQL Server Full-Text Search and Analysis server component installed. On the Configure Operations Manager accounts page, provide the domain account credentials (for example, labsvc-scom) for the Operations Manager services. You can use a single domain account. For account requirements, see the Microsoft Operations Manager deployment guide at http://go.microsoft.com/fwlink/?LinkId=246682. On the Help improve System Center 2012 – Operations Manager page, select the desired options and click on Next. On the Installation Summary page, review the options and click on Install, and then on click on Close. The Operations Manager console will open. How it works... When deploying SCOM 2012, it is important to consider the placement of the components. Work on the SCOM design before implementing it. See the OpsMgr 2012 Design Guide available at http://blogs.technet.com/b/momteam/archive/2012/04/13/ opsmgr-2012-design-guide.aspx. On the Configure Operational Database page, if you are installing the first management server, a new operational database will be created. If you are installing additional management servers, an existing database will be used. On the SQL Server instance for Reporting Services page, make sure you have previously configured the Reporting Services at SQL setup using the Reporting Services Configuration Manager tool, and that the SQL Server Agent is running. During the OpsMgr setup, you will be required to provide the Management Server Action Account credentials and the System Center Configuration service and System Center Data Access service account credentials too. The recommendation is to use a domain account so that you can use the same account for both the services. The setup will automatically assign the local computer Administrators group to the Operations Manager administrator's role. The single-server scenario combines all roles onto a single instance and supports the following services: monitoring and alerting, reporting, audit collection, agentless-exception management, and data. If you are planning to monitor the network, it is recommended to move the SQL Server tempdb database to a separate disk that has multiple spindles. There's more... To confirm the health of the management server, carry out the following steps: In the OpsMgr console, click on the Administration workspace. In Device Management, select Management Servers to confirm that the installed server has a green check mark in the Health State column. See also The Deploying System Center 2012 – Operations Manager article available at http://technet.microsoft.com/en-us/library/hh278852.aspx Installing management packs After installing Operations Manager, you need to install some management packs and agents on the Hyper-V servers and on the VMM server. This recipe will guide you through the installation, but first make sure you have installed the Operations Manager Operations console on the VMM management server. You need to import the following management packs for the VMM 2012 SP1 integration: Windows Server operating system Windows Server 2008 operating system (Discovery) Internet Information Services 2003 Internet Information Services 7 Internet Information Services library SQL Server Core Library Getting ready Before you begin, make sure the correct version of PowerShell is installed, that is, PowerShell v2 for SC 2012 and PowerShell v3 for SC2012 SP1. How to do it... Carry out the following steps to install the required MPs in order to integrate with VMM 2012 SP1: In the OpsMgr console, click on the Administration workspace on the bottom-left pane. On the left pane, right-click on Management Packs and click on Import Management Packs. In the Import Management Packs wizard, click on Add, and then click on Add from catalog. In the Select Management Packs from Catalog dialog box, for each of the following management packs, repeat the steps 5 to 7: Windows Server operating system Windows Server 2008 operating system (Discovery) Internet Information Services 2003 Internet Information Services 7 Internet Information Services library SQL Server Core Library There are numerous management packs for Operations Manager. You can use this recipe to install other OpsMgr MPs from the catalog web service. You can also download the MPs from the Microsoft System Center Marketplace, which contains the MPs and documentation from Microsoft and some non-Microsoft companies. Save them to a shared folder and then import. See http://systemcenter.pinpoint. microsoft.com/en-US/home. In the Find field, type in the management pack to search in the online catalog and click on Search. The Management packs in the catalog list will show all of the packs that match the search criterion. To import, select the management pack, click on Select, and then click on Add as shown in the following screenshot: In the View section, you can refine the search by selecting, for example, to show only those management packs released within the last three months. The default view lists all of the management packs in the catalog. Click on OK after adding the required management packs. On the Select Management Packs page, the MPs will be listed with either a green icon, a yellow icon, or a red icon. The green icon indicates that the MP can be imported. The yellow information icon means that it is dependent on other MPs that are available in the catalog, and you can fix the dependency by clicking on Resolve. The red error icon indicates that it is dependent on other MPs, but the dependent MPs are not available in the catalog. Click on Import if all management packs have their icon statuses as green. On the Import Management Packs page, the progress for each management pack will be displayed. Click on Close when the process is finished. How it works... You can import the management packs available for Operations Manager using the following: The OpsMgr console: You can perform the following actions in the Management Packs menu of the Administration workspace: Import directly from Microsoft's online catalog Import from disk/share Download the management pack from the online catalog to import at a later time The Internet browser: You can download the management pack from the online catalog to import at a later time, or to install on an OpsMgr that is not connected to the Internet While using the OpsMgr console, verify whether all management packs show a green status. Any MP displaying the yellow information icon or the red error icon in the import list will not be imported. If there is no Internet connection on the OpsMgr, use an Internet browser to locate and download the management pack to a folder/share. Then copy the management pack to the OpsMgr server and use the option to import from disk/share. See also The Installing System Center Operations Manager 2012 SP1 recipe Visit Microsoft System Center Marketplace available at http://go.microsoft.com/fwlink/?LinkId=82105
Read more
  • 0
  • 0
  • 3388

article-image-building-bar-graph-cityscape
Packt
16 May 2013
4 min read
Save for later

Building a bar graph cityscape

Packt
16 May 2013
4 min read
(For more resources related to this topic, see here.) Building a bar graph cityscape (Intermediate) In this article, we will take the standard bar graph and explore how we can manipulate it into something completely different. By default, the bar graph is a horizontal bar as follows: In this article, we are going to paint it black, turn it on its side, and make it into a building. Getting ready To start, I've created a simple wallpaper that looks a little bit like a cityscape at sunset. How to do it... Create a new skin folder and name it sunset. This time, create another folder inside it called memory, and create a memory.ini skin file within it. Write the following code into the memory.ini file: [Rainmeter]Name: Memory PlazaVersion: 1Update=1000Author=You![Variables]width=80height=200[Measure]Measure=PhysicalMemory[Meter]MeasureName=MeasureMeter=BAR X=0 Y=0 BarColor=0,0,0,255 BarOrientation=Vertical W=#width# H=#height# Save the file, then load up the new skin. This is what I got: Click-and-drag the black graph and move it to that it sit directly on the horizon. This is what you should end up with: So, now you have successfully built a vertical bar graph that measures your memory use and added it to your cityscape! I will call this building Memory Plaza. Memory Plaza will grow as most of your memory is used up by programs running in Windows. Make sure it doesn't get too tall! How it works... There are several interesting things that happened in this recipe: We created a memory measurement to see how much memory is being used We created a bar meter and connected it to the memory measurement We turned the bar meter on its side and customized it to look just like the buildings We created a memory measure in the following code block: [Measure]Measure=PhysicalMemory In the block, we created a new measurement named Measure. There are other types of memory measurements we can use, but we have just gone with PhysicalMemory. The full documentation on memory measurement is available at http://docs.rainmeter.net/manual/measures/memory. We then created the bar meter in the following code block: [Meter]MeasureName=MeasureMeter=BARX=0Y=0BarColor=0,0,0,255BarOrientation=VerticalW=#width#H=#height# The full documentation on the bar meter is available at http://docs.rainmeter.net/manual/meters/bar You should be able to guess what is going on with the MeasureName, Meter, X, and Y fields. The BarColor field provides the color of the bar. We've made it black to match the color of the horizon. As we want the bar to grow upwards as a building, we have set BarOrientation to be Vertical. The last two fields are new: W=#width#H=#height# If you were to guess that the W field represented width and the H field represented height, then you would be correct. The values in the hashes like #width# are variables. Variables are like containers for values that we can change the contents of. When we want to take the values out to use, we use the variable names within Rainmeter by wrapping them in hashes. If you look higher up the code block, you will find that we declared the variables like so: [Variables]width=80height=200 The obvious benefit to this is that you only have to write down the values for width and height once. There's more... Right now, it is not obvious where the maximum readings are. Why not draw a crane, or building scaffolding, to mark the highest position to which the buildings can grow? Summary This article helped you to create a live cityscape with buildings that grow or shrink depending on the resources that your Windows operating system is consuming, such as RAM usage, memory usage, and so on. Resources for Article : Further resources on this subject: User Interface Design in ICEfaces 1.8: Part 1 [Article] Enlighten your desktop with Elive [Article] User Interface Design in ICEfaces 1.8: Part 2 [Article]
Read more
  • 0
  • 0
  • 2030

article-image-quick-start-selenium-tests
Packt
16 May 2013
4 min read
Save for later

Quick Start into Selenium Tests

Packt
16 May 2013
4 min read
(For more resources related to this topic, see here.) Step 1 – Recording and adding commands in a test In this section we will show you how to record a test on a demo e-commerce application. We will test the product search feature of the application using the following steps: Launch the Firefox browser. Open the website for testing in the Firefox browser. For this example we will use http://demo.magentocommerce.com/. Open Selenium IDE from the Tools menu. Selenium IDE by default sets the recording mode on. If it's not pressed, you can start recording by pressing the (record) button in the top-right corner. Now switch back to the Firefox browser window and type Nokia in the search textbox and click on the Search button as shown: Check if the link Nokia 2610 Phone is present in the search results. We can do that by selecting the link and opening the context menu (right-click) and selecting Show All Available Commands | assertElementPresent link=Nokia 2610 Phone. Next, we will click on the Nokia 2610 Phone link to open the product page and check if the Nokia 2610 Phone text is displayed on the product page. To do this, select the Nokia 2610 Phone text and open the context menu (right-click) and select Show All Available Commands | assertTextPresent link=Nokia 2610 Phone: Go back to Selenium IDE. All the previous steps are recorded by Selenium IDE in the Command-Target-Value format as shown in the following screenshot. Stop the recording session by clicking on the Recording button: Step 2 – Saving the recorded test Before we play back the recorded test, let's save it in Selenium IDE: Select File | Save Test Case from the Selenium IDE main menu: In the Save As dialog box, enter the test case name as SearchTest.html and click on the Save button. The test will be saved with the name SearchTest. Step 3 – Saving the test suite In Selenium IDE, we can group multiple tests in a test suite. Let's create a test suite and Selenium IDE will automatically add SearchTest to this suite: Select File | Save Test Suite from the Selenium IDE main menu. In the Save As dialog box, enter the test case name as SearchFeatureTests.html and click on the Save button. You can create and record more than one test case in a test suite. Step 4 – Running the recorded test Selenium IDE provides multiple ways to execute the tests: Option 1 – running a single test case Select the test which you want to execute from the test suite pane and click on the (play current test case) button . Selenium IDE will start the playback of the test and you can see the steps that we recorded earlier are being played automatically in the browser window. At end of execution, Selenium IDE will display results as per the following screenshot: Option 2 – running all tests from a test suite If you have multiple tests in a test suite, you can use the (play the entire test suite) button to play all the test cases. After the test is executed in Selenium IDE, you can see the results in Log tab. All the steps which are successfully completed will be heighted in green and checks in dark green. If there are any failures in the test, those will be highlighted in red. This is how Selenium IDE helps you testing your web application. Summary We learned how to record a test, save a test case, enhance a test by adding commands, and run a test with Selenium IDE. This article also got you started on with programming with Selenium WebDriver. Resources for Article : Further resources on this subject: Python Testing: Installing the Robot Framework [Article] First Steps with Selenium RC [Article] User Extensions and Add-ons in Selenium 1.0 Testing Tools [Article]
Read more
  • 0
  • 0
  • 2918

article-image-drilling-back-source-data-dynamics-gp-2013-using-dashboards
Packt
15 May 2013
13 min read
Save for later

Drilling Back to Source Data in Dynamics GP 2013 using Dashboards

Packt
15 May 2013
13 min read
(For more resources related to this topic, see here.) Recap Throughout the book we've been building a dashboard that looks like the following screenshot: We're done with the hard parts, and now we're connecting the pieces and cleaning things up. Learning about hyperlinks Hyperlinks are a feature of Excel 2013 that have been in the product for a while. Links can be built via the interface or with a formula. They provide a great way to link sheets together for the user. In the previous chapters, we added additional information to our Revenue and Net Income tabs. We'll start by linking these tabs to our dashboard. To build our hyperlinks: Open the Dashboard.xlsx file of GP 2013 that we've been working with. On the Dashboard tab, select cell D7. This should be the Revenue label. Click on Insert | Hyperlink on the Excel ribbon. In the Link to: section, on the left-hand side, select Place in This Document. In the center section, under Or select a place in this document:, pick Revenue, as shown in the following screenshot: Click on OK. The Revenue label will turn blue and be underlined indicating a hyperlink, as shown in the following screenshot: Click the new Revenue link to drill down to the Revenue tab. We We need to do the same thing for the net Income line. To link to additional net income information, follow these steps: Click on the Dashboard tab. On the Dashboard tab, select cell E7. This should be the Net Income label. Click on Insert | Hyperlink on the Excel ribbon. In the Link to: section, on the left-hand side, select Place in This Document. In the center section under Or select a place in this Document, pick 'Net Income' and click on OK. Save the file. Hyperlinks don't have to link back to another Excel sheet. They can also link to more information on the Web or to a location in SharePoint, for example. Finally, we can link them back to a transaction in Dynamics GP 2013. That's up next. Using drill downs in GP 2013 At its simplest, a drill down is a hyperlink that links back into Dynamics GP. When the user clicks the hyperlink, the focus changes to Microsoft Dynamics GP 2013, and the linked window opens in GP with the appropriate data. In the real world, a dashboard might display cash balances for each bank account, or checkbook in GP terms. The operating checkbook would have a hyperlink attached on the dashboard. Clicking on the link would cause the checkbook register inquiry window to open in Dynamics GP and display information from the operating checkbook. Drill down background There are limited training resources available around drill downs. Drill Down Builder gets only a few pages in the SmartList Builder User Guide. Other books on the market that cover SmartList Builder skip Drill Down Builder altogether. When I pushed Microsoft for a list of pre-built drill downs, it couldn't supply one. Also, Microsoft inconsistently uses the terms drill down and drillback interchangeably. For our purposes, they are the same thing. A drill down link can work for inquiries and transactions throughout GP. Since these drill downs are both poorly documented and numerous, you would think that they would be hard to use, except that Microsoft gave us a huge shortcut. The Office Data Connector files that we've been using for our dashboard contain drill down links. Each ODC file has one or more columns that link back into Dynamics GP. If you can't find the link you are looking for, you can even build your own with the optional Drill Down Builder module from Microsoft. In this article, we're going to build some links, explain how they work, and add some to the dashboard. Before we get rolling, there are few things that you need to know: The user must have Dynamics GP 2013 open and be logged in to the company they are drilling into for the drill down to work. The hyperlink will not open Dynamics GP 2013 for you. This arrangement also makes licensing and security straightforward, since it's controlled by the GP 2013 interface. The user must have permission in Dynamics GP 2013 to open the window that they are trying to drill back into. For example, if a user doesn't have access to payroll inquiry via GP, we certainly don't want them to be able to drill down into that data via Excel. Drill Down Builder is not required to drill down into Dynamics GP 2013. Drill Down Builder is a part of the optional SmartList Builder product available from Microsoft at an additional cost. Drill Down Builder is used to create drill downs and is covered later in the article. It is possible to drill down from a local instance of Excel to Microsoft Dynamics GP 2013 on a Citrix server. I didn't say it was easy, but it can be done. We'll look at options at the end of the article. At the release of Microsoft Dynamics GP 2013, drilling down from Excel to GP via the new web client was not available. It may be made available later via a service pack. Now that we have all the background out of the way, let's drill down! Using drill downs Drill downs are simplest to explain when we bring the data into Microsoft Excel 2013, so we'll go down that route with a common example. To build your first drill down, follow these steps: Open the sample company in Microsoft Dynamics GP 2013. Select Financial in the navigation list on the left-hand side. In the pane above, select Excel Reports. Double-click the selection marked TWO AccountTransactions. The type should be Data Connection. Scroll all the way to the right-hand side of the resulting Excel file. You should see two columns labeled Account Index for Drillback and Journal Entry for Drillback. These are the two default drill downs URLs for journal entry transactions: Account Index for Drillback: This entry will open the Account Maintenance window for this account. That's not terribly helpful in most cases since it just lists the account setup. Account Index for Journal: This entry will open the Journal Entry Inquiry window for posted transactions and the Transaction Entry window for unposted entries. Both of these windows then allow drill back into additional detail. We have the link details, but it's not yet a link in Dynamics GP. To build a formula-based link in Excel 2013, follow these steps: In the Excel sheet, insert a column between columns A and B to create a blank column B. In cell B1, type JE Link. In cell B2, type =HYPERLINK(DR2,A2). Cell DR2 should be the first cell under Journal Entry for Drillback. Here, we're building a hyperlink using a formula instead of the interface. Unlike the interface-based link we used for revenue, a formula-based link is dynamic, making it easy to build a link per line. Column B now contains the Journal Entry number with a link. Scroll down to journal entry 27 and click on the link. Click on Yes when the security notice appears. There is a way to disable this box using a registry entry, but there are variations based on your version of Windows and Office. You can find out more at http://www.msoutlook. info/question/245. Make sure to back up the registry before making changes. The Journal Entry Inquiry window will open for journal entry 27. A user can then click on Source Document to continue drilling back into the source of this journal entry. The reason that we selected journal entry 27 to drill back into is that this is a posted journal entry. If we had selected an unposted journal entry, the Transaction Entry window would have opened. In the Dynamics GP interface, you can't use an inquiry window to inquire on an unposted journal entry. You get an error message that says that entry hasn't been posted. Because of this, the drill down created is different for posted and unposted transactions. Fixing the journal entry drill down problem In the release to manufacturing (RTM) version of Dynamics GP 2013, drilling back to an unposted journal entry generates the error message, The URL was missing required Dynamics GP Drill Back parameters. There was a change to the way that the URL was structured in GP 2013, and it broke this functionality. There is an "e" in the constant for the action type that shouldn't be there. A fix is due in an upcoming service pack, but if you don't want to wait, there is another option. Executing this SQL code for each GP 2013 company will fix the issue: alter FUNCTION dgppJournalEntry (@action int,@JRNENTRY int,@RCTRXSEQ numeric(19, 5),@DCSTATUS int,@DOCTYPE int)RETURNS varchar(2000)ASBEGINDECLARE @ActionType varchar(15),@FunctionName varchar(50),@URIstring varchar(255)select @FunctionName = 'OpenJournal'if @action = 1select @ActionType = 'OPEN'elseselect @ActionType = 'OPEN'select @URIstring = '&Act=' + @ActionType + '&Func=' + @FunctionName+ '&JRNENTRY=' + ltrim(str(@JRNENTRY))+ '&RCTRXSEQ=' + ltrim(str(@RCTRXSEQ))+ '&DCSTATUS=' + ltrim(str(@DCSTATUS))+ '&DOCTYPE=' + ltrim(str(@DOCTYPE))RETURN( @URIstring )END The code is also available at https://www.box.com/s/xutg9wbeb9f531cvuevk (Short link: http://bit.ly/13VEIr8). Drill down link structure Since we have so much flexibility with drill downs, it's worth understanding what the structure of a drill down looks like. Here is my drill down link for journal entry 27: dgpp://DGPB/?Db=GP2013&Srv=MPOLINO2011&Cmp=TWO&Prod=0&Act=OPEN&Func=O penJournalInq&JRNENTRY=27&RCTRXSEQ=1&YEAR1=2014&TRXDATE=01/01/2014 That thing is huge! The good news is that it breaks down pretty easily. All the elements are connected by the ampersand (&) symbol. The description of other elements is given in the following table: Drill Down elements Description dgpp://DGPB/? This is the drill back URL that indicates that the program to work with is Dynamics GP. Db=GP2013 This is the database instance. You won't see a database instance if your GP installation uses the base SQL Server instance. The base instance is more common. In this case, the instance is named GP 2013. Srv=MPOLINO2011 This is the server name. In our example here, the server name is MPOLINO2011. Cmp=TWO Cmp represents the database name for the company to drill back to. Our example uses TWO, the sample company. Prod=0 This is the product. Product 0 equates to Dynamics GP. Other product numbers might refer to Fixed Assets, Project Accounting, or an ISV solution. Product numbers are listed in the Dynamics.set file. Act=OPEN This is the action, where we are going to open a window. Func=OpenJournalInq Func represents the function. The function we are performing is opening the Journal Inquiry window. JRNENTRY=27 This is the first parameter; we want to return Journal Entry 27. RCTRXSEQ=1 Recurring Transaction Sequence is the second parameter and it is set to 1. Since recurring transactions can have the same journal entry, this specifies which instance of a recurring transaction to use. TRXDATE=01/01/2014 The final parameter is the transaction date, January 1, 2014. Drill down links for inventory, sales, or other transactions will be similar. In our case, the links are already built for us and the link elements are static. Since we know the structure, we can also make the link dynamic and let it get values from a cell. To illustrate this: Clear column B. Copy and paste the value from cell DR2 into cell B2. It should look like dgpp://DGPB/?Db=GP2013&Srv=MPOLINO2011&Cmp=TWO&Prod=0&Act=OPE N&Func=OpenJournalInq&JRNENTRY=27&RCTRXSEQ=1&YEAR1=2014&TRXDA TE=01/01/2014. Enclose the entry in quotes. Put an equal sign (=) in front of the first quotation mark to make it a formula. In the &JRNENTRY=27& section, change this to be &JRNENTRY="&A169&"&. In the &YEAR1=2014& section, change this to be &YEAR1="&BH169&"&. In the &TRXDATE=01/01/2014 section, change this to be &TRXDATE="&TEXT(D169,"mm/dd/yyyy")&". The final formula should look like ="dgpp://DGPB/?Db=GP2013&Srv=MPOL INO2011&Cmp=TWO&Prod=0&Act=OPEN&Func=OpenJournalInq&JRNENTRY=" &A169&"&RCTRXSEQ=1&YEAR1="&BH169&"&TRXDATE="&TEXT(D169,"mm/dd/ yyyy")&"". Now we have a dynamic formula that gets the appropriate values from the various cells. Note the double quotes at the end to make it all work. It's time to see how we can apply this practically to our dashboard. We'll take our Top 10 Customers tab and enhance it with a drill back to customer information. To do this: Make sure that your Dashboard.xlsx file of GP 2013 is open. Select the Top 10 Customers tab. Click inside the pivot table. If Field List doesn't open on the right-hand side, click on the Analyze tab under PivotTable Tools and pick Field List. In Field List, check the box next to Customer Number Uncheck the box next to Customer Name. Ensure that the pivot table still shows the top 10 customers sorted by Document Amount. In cell C3, next to the pivot table header, type Link. Save the file. Okay, everything is prepared. Now, we need to go find the link. To do that, follow these steps: Open Microsoft Dynamics GP 2013. Select Sales from the navigation pane on the left-hand side. Pick Excel Reports from the navigation list above. In the center, find Data Connector, not report, labeled TWO Customers and double-click on it. When Excel opens, click on OK to put the data in a table. Scroll to the right-hand side in the resulting Excel file to find the column labeled Customer Number For Drillback. It should be near column FR. Select the first row below Customer Number For Drillback. Right-click and select Copy. Return to the Top 10 Customers tab in the Dashboard.xlsx file of GP 2013. Select cell D4. Right-click and pick Paste. Click on the link pasted into cell D4. Put an equal sign (=) at the front. Place quotation marks (" ") on the front and back of the link, after the equal sign. It should look similar to ="dgpp://DGPB/?Db=GP2013&Srv=MPOLINO2 011&Cmp=TWO&Prod=0&Act=OPEN&Func=OpenCustNmbr&CUSTNMBR=AARONF IT0001". At the end of the formula, replace the customer number between the equal sign and the final quote with "&A4&". The final formula should look similar to ="dgpp://DGPB/?Db=GP2013&Srv=MPOLINO2011&Cmp=TWO&Prod=0&Act =OPEN&Func=OpenCustNmbr&CUSTNMBR="&A4&"". Note that there are two sets of quotes at the end. In cell C4, type the formula, =Hyperlink(D4,"Drillback"). Copy cells C4 and D4 down through all 10 customers. Click one of the drill back links. The Customer Maintenance window should open for the customer selected, as shown in the following screenshot: With a different drill back, we could link to the Customer Inquiry window. Save the file.
Read more
  • 0
  • 0
  • 5230
article-image-article-insight-hyper-v-storage
Packt
14 May 2013
14 min read
Save for later

Insight into Hyper-V Storage

Packt
14 May 2013
14 min read
Types of Hyper-V virtual storage In the previous section, we discovered what virtual storage is and how it contributes to the server virtualization architecture. With all the information we got from the previous section about virtual storage, let's now move ahead and see the virtual machine storage options that Hyper-V offers us. In this section, we will go through different types of virtual machine storage options, such as VHD, VHDX, fixed disk, dynamic disk, differencing disk, and pass-through disk. We will discuss each of them in detail so that you understand all their ins and outs for better planning and sizing. Each of these virtual storage options has a different set of properties than the others, and the administrator must choose the correct virtual storage that best fits the server operating systems and application needs. Let's now start by discussing each of these virtual storage options for virtual machines based on Hyper-V. Virtual disk formats First we must select the virtual disk format before we go ahead and create the virtual hard disk file. There are two possible virtual disk formats available with Windows Server 2012 Hyper-V; they are VHD and VHDX. Let's first make a short comparison between the two virtual hard disk formats, which will provide us with some quick guidance on making the appropriate selection according to the usage. table Virtual hard disk (VHD) After the release of Hyper-V, Microsoft provided its customers with a native hypervisor for their server virtualization needs. But this newly released product had to face challenges for being called an enterprise virtualization platform. And virtual hard disk (VHD) limitation was one of the caveats that customers faced. There are a few limitations to the VHD format; one is the limited size of the virtual hard disk (which we faced before) and another is the possibility of data inconsistency due to power failures. Virtual hard disk (VHD) is a file-based storage for your virtual machine that is based on Hyper-V; this is a default and basic level of storage functionality for a virtual machine. An administrator can create a virtual hard disk (VHD) file for a virtual machine within Hyper-V for a specific size, where defining the size is mandatory. This VHD file can have a different set of properties based on its type. A virtual hard disk (VHD) file or file extension is similar to the VMKD file format, which is a VMware virtual machine hard disk extension. VHD files also existed in the earlier versions of server and desktop virtualization software from Microsoft, for example, Virtual PC and Virtual Server. Virtual hard disk (VHDX) As we saw, the two main limitations of VHD format based virtual hard disk are size and data inconsistency due to power failure; Microsoft addressed these two main limitations of the VHD file format and introduced a new virtual hard disk format called VHDX. This virtual hard disk format allowed customers to create virtual hard disks of up to 64 TB, where earlier the virtual hard disk format (VHD) only allowed virtual hard disks up to the size of 2 TB. Also, as this new format has a resilient architecture, the possibility of data corruption due to power failure also reduced. Virtual disk types The virtual hard disk format decides the maximum size of a virtual hard disk, while the virtual hard disk type decides the functionality and features a virtual hard disk will provide. Microsoft Windows Server 2012 Hyper-V provides four types of virtual hard disks for virtual machines based on Hyper-V. These four virtual hard disk types are as follows: Dynamic disk Fixed disk Differencing disk Pass-through disk You should choose the virtual hard disk type based on your server and application requirements. Each type of virtual hard disk provides different set of disk performance and functionalities, so proper planning is highly important to ensure that you select the right virtual storage for your workload. Let's now discover each of these storage types in detail. Dynamic disk When you create a new virtual machine, and create a new virtual hard disk from New Virtual Machine Wizard, the wizard chooses the dynamic disk as the virtual hard disk type for you. Dynamic disks, as they sound, are dynamic; this means they get changed over time or due to the occurrence of certain events. Dynamic disks are the best choice for economic usage of the server's storage. With whatever size of dynamic disk you create, it won't immediately deduct the same amount of disk space from the physical storage of the Hyper-V server but instead will get created with a very small size, and over a period of time, keep growing as you put data and content on this disk. This dynamic growth of a disk is the actual concept behind this type of virtual hard disk. Since dynamic disks are not of a fixed size and are actually small in size, they cannot deliver a good disk I/O for storage-intensive applications. Real-world example Over the years, I have seen many cases where a production workload (VM) has had performance bottlenecks, especially for the disk subsystem of a virtual machine. And among these, in the case of performance problems related to the virtual machine disk subsystem, the majority of times I saw people using dynamic virtual hard disks for their production workloads. And since dynamic disks do not have a fixed size storage for virtual machines, they are not a good choice for disk-intensive applications and server roles. A dynamic disk has another problem of not being able to provide good results for disk fragmentation or other similar activities due to its design. Dynamic disks are good for the testing and research and development types of virtual machines where the performance factor is not very important. Now let's see how we can create a dynamic disk for a virtual machine. As we have said previously, when you create a virtual machine via New Virtual Machine Wizard, it also gives you the option to create a virtual hard disk for the virtual machine by default. This wizard for creating a new virtual machine, along with the creation of the virtual hard disk, provides a dynamic disk by default, and so the disk type option is not provided as a selection option. In the following brief steps, we will see how to create a dynamic disk: Open Hyper-V Manager from Administrative Tools. From the Hyper-V Manager snap-in, find the New button on the right action pane and click on Hard Disk. New Hard Disk Wizard will open; it will first ask you to select the hard disk format, which could be either VHD or VHDX, depending on the size of your hard disk. Then you will be prompted to select the disk type; here we will select Dynamic Disk. The next section of New Hard Disk Wizard will ask you the name of the virtual hard disk and the location where you want this virtual hard disk to be created and stored. Now the last section of this wizard will ask you the size of the disk you want to create. This section also gives you the functionality to copy content from a physical disk of the server or any other virtual disk that has already been created. Fixed disk A fixed disk is like a static disk that has a fixed size and doesn't grow over time if we go on adding content to it. Fixed disks provide better performance as compared to dynamic disks, because when we create a fixed disk of 100 GB, Hyper-V creates a VHD or VHDX file of 100 GB. It should be noted here that creating this 100 GB fixed disk will take a long time as it has to create a VHD/VHDX file of 100 GB, and the larger disk you create, the longer time it will take. A fixed disk allocates a fixed size from the physical storage of the Hyper-V server, and so this big chunk of allocated disk space allows the virtual machine to receive better I/O performance from this type of virtual hard disk. Fixed disks are always recommended for production workloads because their better performance allows administrators to perform faster read/write operations on virtual disks. Fixed disks are mainly created for virtual machines that run disk-intensive applications, where a high disk I/O is required for virtual machine storage, for example, the virtual hard disk you will create if you are going to virtualize a file sever. Here you will store all the files to the hard disk and so it should be a fixed disk, but at the same time the operating system disk of the file server can be kept as a dynamic disk because there will not be much disk activity on it. To create a fixed disk, you need to perform the following steps: Open Hyper-V Manager from Administrative Tools. From the Hyper-V Manager snap-in, find the New button on the right action pane and click on Hard Disk. New Hard Disk Wizard will open; it will first ask you to select the hard disk format, which could be either VHD or VHDX, depending on the size of your hard disk. You will then be prompted to select the disk type; here we will select Fixed Disk. The next section of New Hard Disk Wizard will ask you for the name of your virtual hard disk and the location where you want this virtual hard disk to be created and stored. Now the last section of this wizard will ask you the size of the disk you want to create. This section also gives you the functionality to copy content from a physical disk of the server or any other virtual disk that has already been created. Differencing disk A differencing disk has a parent-child model associated with its architecture. Mainly, it comes into use when an administrator takes a snapshot of a virtual machine, where after creating the snapshot, Hyper-V leaves the first parent VHD intact and creates a new child disk that gets linked to the parent virtual hard disk. Both parent and child disks always have the same disk format; this means that if the parent disk is created as VHD, the child disk cannot be VHDX. A differencing disk is usually never recommended for production workloads because if you to create a snapshot of a production workload, you will stop writing to the production virtual hard disk. Differencing disks are the same in nature as dynamic disks, where the disk size grows over a period of time as we go on adding more data to the disk; this nature of the disk may not give you good performance for the disk subsystem of the production workload. Another problem with the differencing disk is that when we create a snapshot of the virtual machine, from that point in time, all the data gets written on to the differencing disk and your parent VHD/VHDX becomes idle and isolated from the new data changes. And in the case of multiple snapshots you will have data written on multiple differencing disks. So if any of the differencing disks (snapshots) get misplaced or deleted, you will lose all the data that was written on it at that particular period of time. So in a nutshell, it is highly recommended not to create a snapshot of a production virtual machine; but if you have taken it already, make sure that you restore it to its parent VHD/VHDX as early as possible. To create a differencing disk, you may perform the following steps: Previously, we saw the steps for creating other disk types; follow those same steps until we reach the step where we need to select the disk type. When we are prompted to select the disk type, select Differencing Disk. The next section of New Hard Disk Wizard will ask you the name of the virtual hard disk and the location where you want this virtual hard disk to be created and stored. Now the last section of this wizard will ask you the size of the disk you want to create. This section also gives you the functionality to copy content from a physical disk of the server or any other virtual disk that has already been created. Pass-through disk A pass-through disk is a storage type in which an administrator presents a physical hard disk, which is associated or attached to the Hyper-V host server, to the virtual machine as a raw disk. This type of virtual machine storage is called a pass-through or raw disk; in this type of storage, the physical disk or LUN passes through the hypervisor and later to the virtual machine guest system. This physical disk that is associated or attached to the Hyper-V server could be a SAN LUN storage bound to the Hyper-V server, or it could be a locally installed physical hard disk. To mitigate all these aforementioned risks, administrators prefer to use a pass-through disk as a local disk for highly critical virtual workloads and keep the virtual machine storage on this pass-through disk. As a first step, the pass-through disk is attached or made available to the Hyper-V server; once the disk is available to the server, you have to bring the disk offline before you pass the disk through to a virtual machine available on the same Hyper-V box. We cannot make a partition on the disk available to a virtual machine; it is only the local disk of the Hyper-V server that can be made available as a pass-through disk. In many cases, admins prefer to use pass-through disks instead of using VHDs, especially if they want their virtual machine to boot from a SAN LUN, or in the case of databases, where an Exchange Server mailbox database can be placed on pass-through SAN LUN disks. Pass-through disks are also suitable for environments where the application's high availability methodology is placed on a build blocks level. In these types of high availability requirements for the application, an administrator needs to bind the same data disks (pass-through LUNs) to another virtual machine, and you are all good to go. It should also be noted here that pass-through disks don't get included in a backup based on a VSS snapshot. This means that if you have taken a backup of a virtual machine using a VSS-based backup solution, all the pass-through disks of the virtual machine will not be backed up and only the VHD-based/VHDX-based disks will be included in the VSS snapshot backup. The following steps need to be carried out to provide a pass-through disk to a virtual machine within the Hyper-V server: Open Hyper-V Manager from Administrative Tools. Take the virtual machine settings that you want to configure and add a pass-through disk. Add SCSI Controller from the Add Hardware section of the virtual machine settings, or if you wish, you can also add a physical disk to an IDE controller. Then select the hard drive and click on the Add button. Once you click on the Add button at the controller level, you will be prompted to select the physical hard drive that you want to connect to the virtual machine. After selecting the appropriate disk, you need to connect it to the virtual machine. First click on Apply and then click on the OK button to commit the changes. Image Virtual Fibre Channel SAN With the release of Windows Server 2012, Hyper-V now offers virtual FC SAN connectivity to virtual machines, to allow virtual machines to connect to a virtual SAN. The administrator first connects a virtual SAN network setup on the Hyper-V server, just like what we would do to create a virtual switch for different network segments. Once the virtual fibre SAN network gets set up, the Fibre Channel adapter can be added to the virtual machine that needs to be associated with the Fibre Channel SAN network. Hyper-V allows the Hyper-V administrator to configure WWNs and other settings related to Fibre Channel SAN; all these settings can be customized from the virtual machine Fibre Channel adapter or as global settings from the Fibre Channel network of the virtual SAN manager. Image The preceding screenshot describes the connectivity of a standalone Hyper-V server to the FC SAN. In the first phase of connectivity, our Hyper-V server gets connected to the FC SAN switch through a Fibre Channel medium. Then in the second phase of connectivity, we configure the virtual Fibre Channel switch on the Hyper-V server. Once the virtual fabric switch is configured, we can simply add the virtual Fibre Channel adapter to the virtual machine and connect the adapter to the virtual Fibre Channel switch.
Read more
  • 0
  • 0
  • 13356

article-image-net-45-extension-methods-iqueryable
Packt
14 May 2013
4 min read
Save for later

.NET 4.5 Extension Methods on IQueryable

Packt
14 May 2013
4 min read
(For more resources related to this topic, see here.) Getting ready Refer to the IQueryableExtensions.cs file in the ExtensionMethods.Libraryproject for the extension methods. The models are located in Models/PagedList.cs and Models/IPagedList.cs. These methods are used in the IQueryableExtensionTests.cs file in the ExtensionMethods.Tests project. How to do it... The following code snippet shows a general use of extension methods on IQueryables: public static User ByUserId(this IQueryable<User> query, int userId) { return query.First(u => u.UserId == userId); } The following code snippet is a paged list class for pagination of data: public class PagedList<T> : List<T>, IPagedList { public PagedList(IQueryable<T> source, int index, int pageSize) { this.TotalCount = source.Count(); this.PageSize = pageSize; this.PageIndex = index; this.AddRange(source.Skip(index * pageSize).Take(pageSize). ToList()); } public PagedList(List<T> source, int index, int pageSize) { this.TotalCount = source.Count(); this.PageSize = pageSize; this.PageIndex = index; this.AddRange(source.Skip(index * pageSize).Take(pageSize). ToList()); } public int TotalCount { get; set; } public int PageIndex { get; set; } public int PageSize { get; set; } public bool IsPreviousPage { get { return (PageIndex > 0); } } public bool IsNextPage { get { return (PageIndex * PageSize) <=TotalCount; } } } The following code snippet is the extension method that executes and converts the query to the PagedList object: public static PagedList<T> ToPagedList<T>(this IQueryable<T> source, int index, int pageSize) { return new PagedList<T>(source, index, pageSize); } The following code snippet shows how we use these extension methods: [TestMethod] public void UserByIdReturnsCorrectUser() { var query = new List<User> { new User {UserId = 1}, new User {UserId = 2} }.AsQueryable(); var user = query.ByUserId(1); Assert.AreEqual(1, user.UserId); } [TestMethod] public void PagedList_Contains_Correct_Number_Of_Elements() { var query = new List<int>{1,2,3,4,5,6,7,8,9,10}.AsQueryable(); var pagedList = query.ToPagedList(0, 5); Assert.AreEqual(5, pagedList.Count); Assert.AreEqual(10, pagedList.TotalCount); } How it works... The first code snippet ByUserId is the most commonly used type of extension method for IQueryable types. An alternative to this method is to use the repository pattern and add a method of getting a user by the Id. But sometimes, we will expose the query to lower levels of the app such as the service layer where we might need to use this feature at multiple places, hence refactoring that logic into an extension method makes perfect sense. This extension method evaluates and executes the query immediately due to requesting a single value using the First() method: query.First(u => u.UserId == userId); The second code snippet gives us a PagedList model which becomes a valuable class when working with grids or pagination. The constructor accepts an IQueryable or IList and converts that data into a paged list. Take note of the line in which we evaluate the source by calling ToList(). This line executes the query on the provider: this.AddRange(source.Skip(index * pageSize).Take(pageSize).ToList()); In the code snippets using these extension methods, we have created a list and cast it to an IQueryable type. This is purely for the purpose of demonstration. In a real application, the query would be coming from a LINQ to SQL or entities context, which is in charge of executing the query against a database. We need to be careful of how extension methods on IQueryable are written. A poorly written query will result in unexpected behavior, such as premature query execution. If the extension method is simply building up the query (using method chaining), ensure that the query is not evaluated inside the method. If the query is evaluated and executed before the method finishes, any other use of the query outside of the extension method will result in operating on the data in memory. Summary In this article, you have learned a few tricks and caveats when using extending IQueryable. Resources for Article : Further resources on this subject: Working With ASP.NET DataList Control [Article] NHibernate 3.0: Using LINQ Specifications in the data access layer [Article] LINQ to Objects [Article]
Read more
  • 0
  • 0
  • 17478

article-image-tracking-faces-haar-cascades
Packt
13 May 2013
4 min read
Save for later

OpenCV: Tracking Faces with Haar Cascades

Packt
13 May 2013
4 min read
Conceptualizing Haar cascades When we talk about classifying objects and tracking their location, what exactly are we hoping to pinpoint? What constitutes a recognizable part of an object? Photographic images, even from a webcam, may contain a lot of detail for our (human) viewing pleasure. However, image detail tends to be unstable with respect to variations in lighting, viewing angle, viewing distance, camera shake, and digital noise. Moreover, even real differences in physical detail might not interest us for the purpose of classification. I was taught in school, that no two snowflakes look alike under a microscope. Fortunately, as a Canadian child, I had already learned how to recognize snowflakes without a microscope, as the similarities are more obvious in bulk. Thus, some means of abstracting image detail is useful in producing stable classification and tracking results. The abstractions are called features, which are said to be extracted from the image data. There should be far fewer features than pixels, though any pixel might influence multiple features. The level of similarity between two images can be evaluated based on distances between the images' corresponding features. For example, distance might be defined in terms of spatial coordinates or color coordinates. Haar-like features are one type of feature that is often applied to real-time face tracking. They were first used f or this purpose by Paul Viola and Michael Jones in 2001. Each Haar-like feature describes the pattern of contrast among adjacent image regions. For example, edges, vertices, and thin lines each generate distinctive features. For any given image, the features may vary depending on the regions' size, which may be called the window size. Two images that differ only in scale should be capable of yielding similar features, albeit for different window sizes. Thus, it is useful to generate features for multiple window sizes. Such a collection of features is called a cascade. We may say a Haar cascade is scale-invariant or, in other words, robust to changes in scale. OpenCV provides a classifier and tracker for scale-invariant Haar cascades, whic h it expects to be in a certain file format. Haar cascades, as implemented in OpenCV, are not robust to changes in rotation. For example, an upside-down face is not considered similar to an upright face and a face viewed in profile is not considered similar to a face viewed from the front. A more complex and more resource-intensive implementation could improve Haar cascades' robustness to rotation by considering multiple transformations of images as well as multiple window sizes. However, we will confine ourselves to the implementation in OpenCV. Getting Haar cascade data As part of your OpenCV setup, you probably have a directory called haarcascades. It contains cascades that are trained for certain subjects using tools that come with OpenCV. The directory's full path depends on your system and method of setting up OpenCV, as follows: Build from source archive:: <unzip_destination>/data/haarcascades Windows with self-extracting ZIP:<unzip_destination>/data/haarcascades Mac with MacPorts:MacPorts: /opt/local/share/OpenCV/haarcascades Mac with Homebrew:The haarcascades file is not included; to get it, download the source archive Ubuntu with apt or Software Center: The haarcascades file is not included; to get it, download the source archive If you cannot find haarcascades, then download the source archive from http://sourceforge.net/projects/opencvlibrary/files/opencv-unix/2.4.3/OpenCV-2.4.3.tar.bz2/download (or the Windows self-extracting ZIP from http://sourceforge.net/projects/opencvlibrary/files/opencvwin/ 2.4.3/OpenCV-2.4.3.exe/download), unzip it, and look for <unzip_destination>/data/haarcascades. Once you find haarcascades, create a directory called cascades in the same folder as cameo.py and copy the following files from haarcascades into cascades: haarcascade_frontalface_alt.xmlhaarcascade_eye.xmlhaarcascade_mcs_nose.xmlhaarcascade_mcs_mouth.xml As their names suggest, these cascades are for tracking faces, eyes, noses, and mouths. They require a frontal, upright view of the subject. With a lot of patience and a powerful computer, you can make your own cascades, trained for various types of objects. Creating modules We should continue to maintain good separation between application-specific code and reusable code. Let's make new modules for tracking classes and their helpers. A file called trackers.py should be created in the same directory as cameo.py (and, equivalently, in the parent directory of cascades ). Let's put the following import statements at the start of trackers.py: import cv2import rectsimport utils Alongside trackers.py and cameo.py, let's make another file called rects.py containing the following import statement: import cv2 Our face tracker and a definition of a face will go in trackers.py, while various helpers will go in rects.py and our preexisting utils.py file.
Read more
  • 0
  • 0
  • 22603
article-image-android-native-application-api
Packt
13 May 2013
21 min read
Save for later

Android Native Application API

Packt
13 May 2013
21 min read
(For more resources related to this topic, see here.) Based on the features provided by the functions defined in these header files, the APIs can be grouped as follows: Activity lifecycle management: native_activity.h looper.h Windows management: rect.h window.h native_window.h native_window_jni.h Input (including key and motion events) and sensor events: input.h keycodes.h sensor.h Assets, configuration, and storage management: configuration.h asset_manager.h asset_manager_jni.h storage_manager.h obb.h In addition, Android NDK also provides a static library named native app glue to help create and manage native activities. The source code of this library can be found under the sources/android/native_app_glue/ directory. In this article, we will first introduce the creation of a native activity with the simple callback model provided by native_acitivity.h, and the more complicated but flexible two-threaded model enabled by the native app glue library. We will then discuss window management at Android NDK, where we will draw something on the screen from the native code. Input events handling and sensor accessing are introduced next. Lastly, we will introduce asset management, which manages the files under the assets folder of our project. Note that the APIs covered in this article can be used to get rid of the Java code completely, but we don't have to do so. The Managing assets at Android NDK recipe provides an example of using the asset management API in a mixed-code Android project. Before we start, it is important to keep in mind that although no Java code is needed in a native activity, the Android application still runs on Dalvik VM, and a lot of Android platform features are accessed through JNI. The Android native application API just hides the Java world for us. Creating a native activity with the native_activity.h interface The Android native application API allows us to create a native activity, which makes writing Android apps in pure native code possible. This recipe introduces how to write a simple Android application with pure C/C++ code. Getting ready Readers are expected to have basic understanding of how to invoke JNI functions. How to do it… The following steps to create a simple Android NDK application without a single line of Java code: Create an Android application named NativeActivityOne. Set the package name as cookbook.chapter5.nativeactivityone. Right-click on the NativeActivityOne project, select Android Tools | Add Native Support. Change the AndroidManifest.xml file as follows: <manifest package="cookbook.chapter5.nativeactivityone"android:versionCode="1"android:versionName="1.0"><uses-sdk android_minSdkVersion="9"/><application android_label="@string/app_name"android:icon="@drawable/ic_launcher"android:hasCode="true"><activity android_name="android.app.NativeActivity"android:label="@string/app_name"android:configChanges="orientation|keyboardHidden"><meta-data android_name="android.app.lib_name"android:value="NativeActivityOne" /><intent-filter><action android_name="android.intent.action.MAIN" /><category android_name="android.intent.category.LAUNCHER" /></intent-filter></activity></application></manifest> We should ensure that the following are set correctly in the preceding file: The activity name must be set to android.app.NativeActivity. The value of the android.app.lib_name metadata must be set to the native module name without the lib prefix and .so suffix. android:hasCode needs to be set to true, which indicates that the application contains code. Note that the documentation in <NDK root>/docs/NATIVE-ACTIVITY.HTML gives an example of the AndroidManifest.xml file with android:hasCode set to false, which will not allow the application to start. Add two files named NativeActivityOne.cpp and mylog.h under the jni folder. The ANativeActivity_onCreate method should be implemented in NativeActivityOne.cpp. The following is an example of the implementation: void ANativeActivity_onCreate(ANativeActivity* activity,void* savedState, size_t savedStateSize) {printInfo(activity);activity->callbacks->onStart = onStart;activity->callbacks->onResume = onResume;activity->callbacks->onSaveInstanceState = onSaveInstanceState;activity->callbacks->onPause = onPause;activity->callbacks->onStop = onStop;activity->callbacks->onDestroy = onDestroy;activity->callbacks->onWindowFocusChanged =onWindowFocusChanged;activity->callbacks->onNativeWindowCreated =onNativeWindowCreated;activity->callbacks->onNativeWindowResized =onNativeWindowResized;activity->callbacks->onNativeWindowRedrawNeeded =onNativeWindowRedrawNeeded;activity->callbacks->onNativeWindowDestroyed =onNativeWindowDestroyed;activity->callbacks->onInputQueueCreated = onInputQueueCreated;activity->callbacks->onInputQueueDestroyed =onInputQueueDestroyed;activity->callbacks->onContentRectChanged =onContentRectChanged;activity->callbacks->onConfigurationChanged =onConfigurationChanged;activity->callbacks->onLowMemory = onLowMemory;activity->instance = NULL;} Add the Android.mk file under the jni folder: LOCAL_PATH := $(call my-dir)include $(CLEAR_VARS)LOCAL_MODULE := NativeActivityOneLOCAL_SRC_FILES := NativeActivityOne.cppLOCAL_LDLIBS := -landroid -lloginclude $(BUILD_SHARED_LIBRARY) Build the Android application and run it on an emulator or a device. Start a terminal and display the logcat output using the following: $ adb logcat -v time NativeActivityOne:I *:S Alternatively, you can use the logcat view at Eclipse to see the logcat output. When the application starts, you should be able to see the following logcat output: As shown in the screenshot, a few Android activity lifecycle callback functions are executed. We can manipulate the phone to cause other callbacks being executed. For example, long pressing the home button and then pressing the back button will cause the onWindowFocusChanged callback to be executed. How it works… In our example, we created a simple, "pure" native application to output logs when the Android framework calls into the callback functions defined by us. The "pure" native application is not really pure native. Although we did not write a single line of Java code, the Android framework still runs some Java code on Dalvik VM. Android framework provides an android.app.NativeActivity.java class to help us create a "native" activity. In a typical Java activity, we extend android.app.Activity and overwrite the activity lifecycle methods. NativeActivity is also a subclass of android. app.Activity and does similar things. At the start of a native activity, NativeActivity. java will call ANativeActivity_onCreate, which is declared in native_activity.h and implemented by us. In the ANativeActivity_onCreate method, we can register our callback methods to handle activity lifecycle events and user inputs. At runtime, NativeActivity will invoke these native callback methods when the corresponding events occurred. In a word, NativeActivity is a wrapper that hides the managed Android Java world for our native code, and exposes the native interfaces defined in native_activity.h. The ANativeActivity data structure: Every callback method in the native code accepts an instance of the ANativeActivity structure. Android NDK defines the ANativeActivity data structure in native_acitivity.h as follows: typedef struct ANativeActivity {struct ANativeActivityCallbacks* callbacks;JavaVM* vm;JNIEnv* env;jobject clazz;const char* internalDataPath;const char* externalDataPath;int32_t sdkVersion;void* instance;AAssetManager* assetManager;} ANativeActivity; The various attributes of the preceding code are explained as follows: callbacks: It is a data structure that defines all the callbacks that the Android framework will invoke with the main UI thread. vm: It is the application process' global Java VM handle. It is used in some JNI functions. env: It is a JNIEnv interface pointer. JNIEnv is used through local storage data , so this field is only accessible through the main UI thread. clazz: It is a reference to the android.app.NativeActivity object created by the Android framework. It can be used to access fields and methods in the android. app.NativeActivity Java class. In our code, we accessed the toString method of android.app.NativeActivity. internalDataPath: It is the internal data directory path for the application. externalDataPath: It is the external data directory path for the application. internalDataPath and externalDataPath are NULL at Android 2.3.x. This is a known bug and has been fixed since Android 3.0. If we are targeting devices lower than Android 3.0, then we need to find other ways to get the internal and external data directories. sdkVersion: It is the Android platform's SDK version code. Note that this refers to the version of the device/emulator that runs the app, not the SDK version used in our development. instance: It is not used by the framework. We can use it to store user-defined data and pass it around. assetManager: It is the a pointer to the app's instance of the asset manager. We will need it to access assets data. We will discuss it in more detail in the Managing assets at Android NDK recipe of this article There's more… The native_activity.h interface provides a simple single thread callback mechanism, which allows us to write an activity without Java code. However, this single thread approach infers that we must quickly return from our native callback methods. Otherwise, the application will become unresponsive to user actions (for example, when we touch the screen or press the Menu button, the app does not respond because the GUI thread is busy executing the callback function). A way to solve this issue is to use multiple threads. For example, many games take a few seconds to load. We will need to offload the loading to a background thread, so that the UI can display the loading progress and be responsive to user inputs. Android NDK comes with a static library named android_native_app_glue to help us in handling such cases. The details of this library are covered in the Creating a native activity with the Android native app glue recipe. A similar problem exists at Java activity. For example, if we write a Java activity that searches the entire device for pictures at onCreate, the application will become unresponsive. We can use AsyncTask to search and load pictures in the background, and let the main UI thread display a progress bar and respond to user inputs. Creating a native activity with the Android native app glue The previous recipe described how the interface defined in native_activity.h allows us to create native activity. However, all the callbacks defined are invoked with the main UI thread, which means we cannot do heavy processing in the callbacks. Android SDK provides AsyncTask, Handler, Runnable, Thread, and so on, to help us handle things in the background and communicate with the main UI thread. Android NDK provides a static library named android_native_app_glue to help us execute callback functions and handle user inputs in a separate thread. This recipe will discuss the android_native_app_glue library in detail. Getting ready The android_native_app_glue library is built on top of the native_activity.h interface. Therefore, readers are recommended to read the Creating a native activity with the native_activity.h interface recipe before going through this one. How to do it… The following steps create a simple Android NDK application based on the android_native_app_glue library: Create an Android application named NativeActivityTwo. Set the package name as cookbook.chapter5.nativeactivitytwo. Right-click on the NativeActivityTwo project, select Android Tools | Add Native Support. Change the AndroidManifest.xml file as follows: <manifest package="cookbook.chapter5.nativeactivitytwo"android:versionCode="1"android:versionName="1.0"><uses-sdk android_minSdkVersion="9"/><application android_label="@string/app_name"android:icon="@drawable/ic_launcher"android:hasCode="true"><activity android_name="android.app.NativeActivity"android:label="@string/app_name"android:configChanges="orientation|keyboardHidden"><meta-data android_name="android.app.lib_name"android:value="NativeActivityTwo" /><intent-filter><action android_name="android.intent.action.MAIN" /><category android_name="android.intent.category.LAUNCHER" /></intent-filter></activity></application></manifest> Add two files named NativeActivityTwo.cpp and mylog.h under the jni folder. NativeActivityTwo.cpp is shown as follows: #include <jni.h>#include <android_native_app_glue.h>#include "mylog.h"void handle_activity_lifecycle_events(struct android_app* app,int32_t cmd) {LOGI(2, "%d: dummy data %d", cmd, *((int*)(app->userData)));}void android_main(struct android_app* app) {app_dummy(); // Make sure glue isn't stripped.int dummyData = 111;app->userData = &dummyData;app->onAppCmd = handle_activity_lifecycle_events;while (1) {int ident, events;struct android_poll_source* source;if ((ident=ALooper_pollAll(-1, NULL, &events, (void**)&source)) >=0) {source->process(app, source);}}} Add the Android.mk file under the jni folder: LOCAL_PATH := $(call my-dir)include $(CLEAR_VARS)LOCAL_MODULE := NativeActivityTwoLOCAL_SRC_FILES := NativeActivityTwo.cppLOCAL_LDLIBS := -llog -landroidLOCAL_STATIC_LIBRARIES := android_native_app_glueinclude $(BUILD_SHARED_LIBRARY)$(call import-module,android/native_app_glue) Build the Android application and run it on an emulator or device. Start a terminal and display the logcat output by using the following command: adb logcat -v time NativeActivityTwo:I *:S When the application starts, you should be able to see the following logcat output and the device screen will shows a black screen: On pressing the back button, the following output will be shown: How it works… This recipe demonstrates how the android_native_app_glue library is used to create a native activity. The following steps should be followed to use the android_native_app_glue library: Implement a function named android_main. This function should implement an event loop, which will poll for events continuously. This method will run in the background thread created by the library. Two event queues are attached to the background thread by default, including the activity lifecycle event queue and the input event queue. When polling events using the looper created by the library, you can identify where the event is coming from, by checking the returned identifier (either LOOPER_ID_MAIN or LOOPER_ID_INPUT). It is also possible to attach additional event queues to the background thread. When an event is returned, the data pointer will point to an android_poll_source data structure. We can call the process function of this structure. The process is a function pointer, which points to android_app->onAppCmd for activity lifecycle events, and android_app->onInputEvent for input events. We can provide our own processing functions and direct the corresponding function pointers to these functions. In our example, we implement a simple function named handle_activity_lifecycle_ events and point the android_app->onAppCmd function pointer to it. This function simply prints the cmd value and the user data passed along with the android_app data structure. cmd is defined in android_native_app_glue.h as an enum. For example, when the app starts, the cmd values are 10, 11, 0, 1, and 6, which correspond to APP_CMD_START, APP_CMD_RESUME, APP_CMD_INPUT_CHANGED, APP_CMD_INIT_WINDOW, and APP_CMD_ GAINED_FOCUS respectively. android_native_app_glue Library Internals: The source code of the android_native_ app_glue library can be found under the sources/android/native_app_glue folder of Android NDK. It only consists of two files, namely android_native_app_glue.c and android_native_app_glue.h. Let's first describe the flow of the code and then discuss some important aspects in detail. Since the source code for native_app_glue is provided, we can modify it if necessary, although in most cases it won't be necessary. android_native_app_glue is built on top of the native_activity.h interface. As shown in the following code (extracted from sources/android/native_app_glue/ android_native_app_glue.c). It implements the ANativeActivity_onCreate function, where it registers the callback functions and calls the android_app_create function. Note that the returned android_app instance is pointed by the instance field of the native activity, which can be passed to various callback functions: void ANativeActivity_onCreate(ANativeActivity* activity,void* savedState, size_t savedStateSize) {LOGV("Creating: %pn", activity);activity->callbacks->onDestroy = onDestroy;activity->callbacks->onStart = onStart;activity->callbacks->onResume = onResume;… …activity->callbacks->onNativeWindowCreated =onNativeWindowCreated;activity->callbacks->onNativeWindowDestroyed =onNativeWindowDestroyed;activity->callbacks->onInputQueueCreated = onInputQueueCreated;activity->callbacks->onInputQueueDestroyed =onInputQueueDestroyed;activity->instance = android_app_create(activity, savedState,savedStateSize);} The android_app_create function (shown in the following code snippet) initializes an instance of the android_app data structure, which is defined in android_native_app_ glue.h. This function creates a unidirectional pipe for inter-thread communication. After that, it spawns a new thread (let's call it background thread thereafter) to run the android_ app_entry function with the initialized android_app data as the input argument. The main thread will wait for the background thread to start and then return: static struct android_app* android_app_create(ANativeActivity*activity, void* savedState, size_t savedStateSize) {struct android_app* android_app = (struct android_app*)malloc(sizeof(struct android_app));memset(android_app, 0, sizeof(struct android_app));android_app->activity = activity;pthread_mutex_init(&android_app->mutex, NULL);pthread_cond_init(&android_app->cond, NULL);……int msgpipe[2];if (pipe(msgpipe)) {LOGE("could not create pipe: %s", strerror(errno));return NULL;}android_app->msgread = msgpipe[0];android_app->msgwrite = msgpipe[1];pthread_attr_t attr;pthread_attr_init(&attr);pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);pthread_create(&android_app->thread, &attr, android_app_entry,android_app);// Wait for thread to start.pthread_mutex_lock(&android_app->mutex);while (!android_app->running) {pthread_cond_wait(&android_app->cond, &android_app->mutex);}pthread_mutex_unlock(&android_app->mutex);return android_app;} The background thread starts with the android_app_entry function (as shown in the following code snippet), where a looper is created. Two event queues will be attached to the looper. The activity lifecycle events queue is attached to the android_app_entry function. When the activity's input queue is created, the input queue is attached (to the android_ app_pre_exec_cmd function of android_native_app_glue.c). After attaching the activity lifecycle event queue, the background thread signals the main thread it is already running. It then calls a function named android_main with the android_app data. android_main is the function we need to implement, as shown in our sample code. It must run in a loop until the activity exits: static void* android_app_entry(void* param) {struct android_app* android_app = (struct android_app*)param;… …//Attach life cycle event queue with identifier LOOPER_ID_MAINandroid_app->cmdPollSource.id = LOOPER_ID_MAIN;android_app->cmdPollSource.app = android_app;android_app->cmdPollSource.process = process_cmd;android_app->inputPollSource.id = LOOPER_ID_INPUT;android_app->inputPollSource.app = android_app;android_app->inputPollSource.process = process_input;ALooper* looper = ALooper_prepare(ALOOPER_PREPARE_ALLOW_NON_CALLBACKS);ALooper_addFd(looper, android_app->msgread, LOOPER_ID_MAIN,ALOOPER_EVENT_INPUT, NULL, &android_app->cmdPollSource);android_app->looper = looper;pthread_mutex_lock(&android_app->mutex);android_app->running = 1;pthread_cond_broadcast(&android_app->cond);pthread_mutex_unlock(&android_app->mutex);android_main(android_app);android_app_destroy(android_app);return NULL;} The following diagram indicates how the main and background thread work together to create the multi-threaded native activity: We use the activity lifecycle event queue as an example. The main thread invokes the callback functions, which simply writes to the write end of the pipe, while true loop implemented in the android_main function will poll for events. Once an event is detected, the function calls the event handler, which reads the exact command from the read end of the pipe and handles it. The android_native_app_glue library implements all the main thread stuff and part of the background thread stuff for us. We only need to supply the polling loop and the event handler as illustrated in our sample code. Pipe: The main thread creates a unidirectional pipe in the android_app_create function by calling the pipe method. This method accepts an array of two integers. After the function is returned, the first integer will be set as the file descriptor referring to the read end of the pipe, while the second integer will be set as the file descriptor referring to the write end of the pipe. A pipe is usually used for Inter-process Communication (IPC), but here it is used for communication between the main UI thread and the background thread created at android_ app_entry. When an activity lifecycle event occurs, the main thread will execute the corresponding callback function registered at ANativeActivity_onCreate. The callback function simply writes a command to the write end of the pipe and then waits for a signal from the background thread. The background thread is supposed to poll for events continuously and once it detects a lifecycle event, it will read the exact event from the read end of the pipe, signal the main thread to unblock and handle the events. Because the signal is sent right after receiving the command and before actual processing of the events, the main thread can return from the callback function quickly without worrying about the possible long processing of the events. Different operating systems have different implementations for the pipe. The pipe implemented by Android system is "half-duplex", where communication is unidirectional. That is, one file descriptor can only write, and the other file descriptor can only read. Pipes in some operating system is "full-duplex", where the two file descriptors can both read and write. Looper is an event tracking facility, which allows us to attach one or more event queues for an event loop of a thread. Each event queue has an associated file descriptor. An event is data available on a file descriptor. In order to use a looper, we need to include the android/ looper.h header file. The library attaches two event queues for the event loop to be created by us in the background thread, including the activity lifecycle event queue and the input event queue. The following steps should be performed in order to use a looper: Create or obtain a looper associated with the current thread: This is done by the ALooper_prepare function: ALooper* ALooper_prepare(int opts); This function prepares a looper associated with the calling thread and returns it. If the looper doesn't exist, it creates one, associates it with the thread, and returns it Attach an event queue: This is done by ALooper_addFd. The function has the following prototype: int ALooper_addFd(ALooper* looper, int fd, int ident, int events,ALooper_callbackFunc callback, void* data); The function can be used in two ways. Firstly, if callback is set to NULL, the ident set will be returned by ALooper_pollOnce and ALooper_pollAll. Secondly, if callback is non-NULL, then the callback function will be executed and ident is ignored. The android_native_app_glue library uses the first approach to attach a new event queue to the looper. The input argument fd indicates the file descriptor associated with the event queue. ident is the identifier for the events from the event queue, which can be used to classify the event. The identifier must be bigger than zero when callback is set to NULL. callback is set to NULL in the library source code, and data points to the private data that will be returned along with the identifier at polling. In the library, this function is called to attach the activity lifecycle event queue to the background thread. The input event queue is attached using the input queue specific function AInputQueue_attachLooper, which we will discuss in the Detecting and handling input events at NDK recipe. Poll for events: This can be done by either one of the following two functions: int ALooper_pollOnce(int timeoutMillis, int* outFd, int*outEvents, void** outData);int ALooper_pollAll(int timeoutMillis, int* outFd, int* outEvents,void** outData); These two methods are equivalent when callback is set to NULL in ALooper_addFd. They have the same input arguments. timeoutMillis specifies the timeout for polling. If it is set to zero, then the functions return immediately; if it is set to negative, they will wait indefinitely until an event occurs. The functions return the identifier (greater than zero) when an event occurs from any input queues attached to the looper. In this case, outFd, outEvents, and outData will be set to the file descriptor, poll events, and data associated with the event. Otherwise, they will be set to NULL. Detach event queues: This is done by the following function: int ALooper_removeFd(ALooper* looper, int fd); It accepts the looper and file descriptor associated with the event queue, and detaches the queue from the looper.
Read more
  • 0
  • 0
  • 16359

article-image-move-further-numpy-modules
Packt
13 May 2013
7 min read
Save for later

Move Further with NumPy Modules

Packt
13 May 2013
7 min read
(For more resources related to this topic, see here.) Linear algebra Linear algebra is an important branch of mathematics. The numpy.linalg package contains linear algebra functions. With this module, you can invert matrices, calculate eigenvalues, solve linear equations, and determine determinants, among other things. Time for action – inverting matrices The inverse of a matrix A in linear algebra is the matrix A-1, which when multiplied with the original matrix, is equal to the identity matrix I. This can be written, as A* A-1 = I. The inv function in the numpy.linalg package can do this for us. Let's invert an example matrix. To invert matrices, perform the following steps: We will create the example matrix with the mat. A = np.mat("0 1 2;1 0 3;4 -3 8") print "An", A The A matrix is printed as follows: A [[ 0 1 2] [ 1 0 3] [ 4 -3 8]] Now, we can see the inv function in action, using which we will invert the matrix. inverse = np.linalg.inv(A) print "inverse of An", inverse The inverse matrix is shown as follows: inverse of A [[-4.5 7. -1.5] [-2. 4. -1. ] [ 1.5 -2. 0.5]] If the matrix is singular or not square, a LinAlgError exception is raised. If you want, you can check the result manually. This is left as an exercise for the reader. Let's check what we get when we multiply the original matrix with the result of the inv function: print "Checkn", A * inverse The result is the identity matrix, as expected. Check[[ 1. 0. 0.][ 0. 1. 0.][ 0. 0. 1.]] What just happened? We calculated the inverse of a matrix with the inv function of the numpy.linalg package. We checked, with matrix multiplication, whether this is indeed the inverse matrix. import numpy as npA = np.mat("0 1 2;1 0 3;4 -3 8")print "An", Ainverse = np.linalg.inv(A)print "inverse of An", inverseprint "Checkn", A * inverse Solving linear systems A matrix transforms a vector into another vector in a linear way. This transformation mathematically corresponds to a system of linear equations. The numpy.linalg function, solve, solves systems of linear equations of the form Ax = b; here A is a matrix, b can be 1D or 2D array, and x is an unknown variable. We will see the dot function in action. This function returns the dot product of two floating-point arrays. Time for action – solving a linear system Let's solve an example of a linear system. To solve a linear system, perform the following steps: Let's create the matrices A and b. iA = np.mat("1 -2 1;0 2 -8;-4 5 9")print "An", Ab = np.array([0, 8, -9])print "bn", b The matrices A and b are shown as follows: Solve this linear system by calling the solve function. x = np.linalg.solve(A, b)print "Solution", x The following is the solution of the linear system: Solution [ 29. 16. 3.] Check whether the solution is correct with the dot function. print "Checkn", np.dot(A , x) The result is as expected: Check[[ 0. 8. -9.]] What just happened? We solved a linear system using the solve function from the NumPy linalg module and checked the solution with the dot function. import numpy as npA = np.mat("1 -2 1;0 2 -8;-4 5 9")print "An", Ab = np.array([0, 8, -9])print "bn", bx = np.linalg.solve(A, b)print "Solution", xprint "Checkn", np.dot(A , x) Finding eigenvalues and eigenvectors Eigenvalues are scalar solutions to the equation Ax = ax, where A is a two-dimensional matrix and x is a one-dimensional vector. Eigenvectors are vectors corresponding to eigenvalues. The eigvals function in the numpy.linalg package calculates eigenvalues. The eig function returns a tuple containing eigenvalues and eigenvectors. Time for action – determining eigenvalues and eigenvectors Let's calculate the eigenvalues of a matrix. Perform the following steps to do so: Create a matrix as follows: A = np.mat("3 -2;1 0")print "An", A The matrix we created looks like the following: A[[ 3 -2][ 1 0]] Calculate eigenvalues by calling the eig function. print "Eigenvalues", np.linalg.eigvals(A) The eigenvalues of the matrix are as follows: Eigenvalues [ 2. 1.] Determine eigenvalues and eigenvectors with the eig function. This function returns a tuple, where the first element contains eigenvalues and the second element contains corresponding Eigenvectors, arranged column-wise. eigenvalues, eigenvectors = np.linalg.eig(A)print "First tuple of eig", eigenvaluesprint "Second tuple of eign", eigenvectors The eigenvalues and eigenvectors will be shown as follows: First tuple of eig [ 2. 1.]Second tuple of eig[[ 0.89442719 0.70710678][ 0.4472136 0.70710678]] Check the result with the dot function by calculating the right- and left-hand sides of the eigenvalues equation Ax = ax. for i in range(len(eigenvalues)):print "Left", np.dot(A, eigenvectors[:,i])print "Right", eigenvalues[i] * eigenvectors[:,i]print The output is as follows: Left [[ 1.78885438][ 0.89442719]]Right [[ 1.78885438][ 0.89442719]]Left [[ 0.70710678][ 0.70710678]]Right [[ 0.70710678][ 0.70710678]] What just happened? We found the eigenvalues and eigenvectors of a matrix with the eigvals and eig functions of the numpy.linalg module. We checked the result using the dot function . import numpy as npA = np.mat("3 -2;1 0")print "An", Aprint "Eigenvalues", np.linalg.eigvals(A)eigenvalues, eigenvectors = np.linalg.eig(A)print "First tuple of eig", eigenvaluesprint "Second tuple of eign", eigenvectorsfor i in range(len(eigenvalues)):print "Left", np.dot(A, eigenvectors[:,i])print "Right", eigenvalues[i] * eigenvectors[:,i]print Singular value decomposition Singular value decomposition is a type of factorization that decomposes a matrix into a product of three matrices. The singular value decomposition is a generalization of the previously discussed eigenvalue decomposition. The svd function in the numpy.linalg package can perform this decomposition. This function returns three matrices – U, Sigma, and V – such that U and V are orthogonal and Sigma contains the singular values of the input matrix. The asterisk denotes the Hermitian conjugate or the conjugate transpose. Time for action – decomposing a matrix It's time to decompose a matrix with the singular value decomposition. In order to decompose a matrix, perform the following steps: First, create a matrix as follows: A = np.mat("4 11 14;8 7 -2")print "An", A The matrix we created looks like the following: A[[ 4 11 14][ 8 7 -2]] Decompose the matrix with the svd function. U, Sigma, V = np.linalg.svd(A, full_matrices=False)print "U"print Uprint "Sigma"print Sigmaprint "V"print V The result is a tuple containing the two orthogonal matrices U and V on the left- and right-hand sides and the singular values of the middle matrix. [-0.31622777 0.9486833 ]]Sigma[ 18.97366596 9.48683298]V[[-0.33333333 -0.66666667 -0.66666667][ 0.66666667 0.33333333 -0.66666667]]U[[-0.9486833 -0.31622777] We do not actually have the middle matrix—we only have the diagonal values. The other values are all 0. We can form the middle matrix with the diag function. Multiply the three matrices. This is shown, as follows: print "Productn", U * np.diag(Sigma) * V The product of the three matrices looks like the following: Product[[ 4. 11. 14.][ 8. 7. -2.]] What just happened? We decomposed a matrix and checked the result by matrix multiplication. We used the svd function from the NumPy linalg module. import numpy as npA = np.mat("4 11 14;8 7 -2")print "An", AU, Sigma, V = np.linalg.svd(A, full_matrices=False)print "U"print Uprint "Sigma"print Sigmaprint "V"print Vprint "Productn", U * np.diag(Sigma) * V Pseudoinverse The Moore-Penrose pseudoinverse of a matrix can be computed with the pinv function of the numpy.linalg module (visit http://en.wikipedia.org/wiki/Moore%E2%80%93Penrose_pseudoinverse). The pseudoinverse is calculated using the singular value decomposition. The inv function only accepts square matrices; the pinv function does not have this restriction.
Read more
  • 0
  • 0
  • 2067
Modal Close icon
Modal Close icon