Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7019 Articles
article-image-so-what-openelec
Packt
04 Sep 2013
3 min read
Save for later

So, what is OpenELEC?

Packt
04 Sep 2013
3 min read
(For more resources related to this topic, see here.) Open Embedded Linux Entertainment Center (OpenELEC) is an open source embedded operating system developed specifically for the purpose of running complete and easy-to-use media center solutions on a wide variety of hardware. Built around the great open source media player and organizer, XBMC, OpenELEC is optimized to deliver a smooth, intuitive, and efficient user experience. It is developed with the goal of making the task of setting up and maintaining a Home Theater PC (HTPC) easy and straightforward for all users, regardless of their technical skills. This makes OpenELEC the obvious choice for anyone looking to enhance their home media experience with a fully functional media center. The OpenELEC project is open source, so anyone can download and use the operating system completely free of charge. This is a great enticement to get started with setting up a media center of your own, capable of providing many hours of digital entertainment for yourself and your friends and family to enjoy. Media center features Once installed and configured, OpenELEC incorporates XBMC to provide a wide variety of advanced features in a streamlined and straightforward interface. The following are the media types that can be indexed and/or browsed: Movies TV shows Music Pictures The experience of browsing indexed media is enhanced by automatic inclusion of media information and relevant images available from online databases. This provides easy access to ratings, resumes, artwork, and trailers, because everything is incorporated directly in the interface. Hardware requirements Because OpenELEC is very versatile in hardware compatibility, you will most likely be able to turn "that old PC, which has just been lying around" into a fully functional media center, just by installing OpenELEC on it. This approach is great for a fun hobby project, giving you the opportunity to experiment with OpenELEC at no cost. Alternatively, you can get brand new PCs small enough to fit behind a TV, making them the obvious choice for HTPC use. With a very small footprint and compatibility with small atom- or fusion-based platforms, this is where OpenELEC shows its full potential as a lightweight embedded operating system. Summary So, what is OpenELEC? finds out what OpenELEC actually is, what you can do with it, and why it’s so great. Resources for Article : Further resources on this subject: Creating a file server (Samba) [Article] Webcam and Video Wizardry [Article] Our First Project – A Basic Thermometer [Article]
Read more
  • 0
  • 0
  • 2931

article-image-creating-sample-application-simple
Packt
04 Sep 2013
8 min read
Save for later

Creating a sample application (Simple)

Packt
04 Sep 2013
8 min read
(For more resources related to this topic, see here.) How to do it... To create an application, include the JavaScript and CSS files in your page. Perform the following steps: Create an HTML document, index.html, under your project directory. Please note that this directory should be placed in the web root of your web server. Create the directories styles and scripts under your project directory. Copy the CSS file kendo.mobile.all.min.css, from <downloaded directory>/styles to the styles directory created in step 2. Then add a reference to the CSS file in the head section of the document. Download the jQuery library from jQuery.com. Place this file in the scripts directory and add a reference to this file in the document before closing the body tag. You can also specify the CDN location of the file in the document. Copy the JavaScript file kendo.mobile.min.js, from the <downloaded directory>/js tag to the scripts directory created in step 2. Then add a reference to this JavaScript file in the document (after jQuery). Add the text "Hello Kendo!!" in the body tag of the index.html file as follows: <!DOCTYPE HTML><html><head><title>My first Kendo Mobile Application</title><link rel="stylesheet"type="text/css"href="styles/kendo.mobile.all.min.css"></head><body>Hello Kendo!!<script type="text/javascript"src = "scripts/jquery.min.js"></script><script type="text/javascript"src = "scripts/kendo.mobile.min.js"></script></body></html> The preceding code snippet is a simple HTML page with references to Kendo Mobile CSS and JavaScript files. These files are minified and contain all the features, themes, and widgets. In production, you would like to include only those that are required. The downloaded ZIP file includes CSS and JavaScript files for specific features. However, in development you can use the minified files that contain all features. Another thing to note is that apart from the reference to the script kendo.mobile.min.js, the page also includes a reference to jQuery. It is the only external dependency for Kendo UI. When you view this page on a mobile device, you will see the text Hello Kendo!! shown. This page does not include any of the widgets that come as a part of the library. Now let's build on top of our Hello World application and add some visual elements; that is, UI widgets to the page. This can be done with the following steps: Add a layout first. A mobile application generally has a header, a footer, and multiple views. It is also observed that while navigating through different views in an application, the header and footer remain constant. The framework allows you to define a global layout that may contain a header and a footer for all the views in the application. Also, the framework allows you to define multiple views that can share the same layout. The following is the same page that now includes a header and footer defined in the layout: <body><div data-role="layout" data-id="defaultLayout"> <header data-role="header"> <div data-role="navbar"> My first application </div> </header> <footer data-role="footer"> <div data-role="tabstrip"> <a data-icon="about">About</a> <a data-icon="settings">Settings</a> </div> </footer> </div></body> The body contains a few div tags with data attributes. Let's look into one of these tags in detail. <div data-role="layout" data-id="defaultLayout"> Here, the div tag contains two data attributes, role and id. The role data attribute is used to initialize and configure a widget. The data-role attribute has a value, layout, identifying the target element as a layout widget. All the widgets are expected to have a role data attribute that helps in marking the target element for a specific purpose. It instructs the library which widget needs to be added to the page. The id data attribute is used to identify the widget (the layout widget) in the page. A page may define several layout widgets and each one of these must be identified by a unique ID. Here, the data-id attribute has defaultLayout as its value. Now there can be many views referring to this layout by its id. Similarly, there are other elements in the page with the data-role attribute, defining them as one of widgets in the page. Let's take a look at the header and footer widgets defined inside the layout. <header data-role="header">... </header><footer data-role="footer">...</footer> The header and footer tags have the role data attribute set to header and footer respectively. This aligns them to the top and bottom of the page, giving the rest of the available space for different views to render. Also, note that there is a navbar widget in the header and a tabstrip widget defined in the footer. As mentioned earlier, the framework comes with several widgets that can help you build the application rapidly. Now add views to the page. The index.html page now has a layout defined and when you run the page in the browser, you will see an error message in the console which says: Uncaught Error: Your kendo mobile application element does not contain any direct child elements with data-role="view" attribute set. Make sure that you instantiate the mobile application using the correct container. Views represent the actual content that has to be displayed between the header and the footer that we defined while creating a layout. A layout cannot exist without a view and hence you see that error message in the console. To fix this error, you need to define a view for your mobile application. Add the following to your index.html page: <div data-role="view" data-layout="defaultLayout"> Hello Kendo!!</div> As mentioned earlier, every widget needs to have a role data attribute to identify itself as a particular widget in the page. Here, the target element is defined as a view widget and tied to the layout by defining the data-layout attribute. The data-layout attribute has a value defaultLayout that is the same as the value for the data-id attribute of the layout that we defined earlier. This attaches the view to the layout and you will not see the error message anymore. Similarly, you can have multiple Views defined in the page that can make use of the same layout. Now, there's only one pending task for the application to start working: initializing the application. A Kendo Mobile application can be initialized using the Application object. To do that, add the following code to the page: <script> var app = new kendo.mobile.Application();</script> Include the previous script block right after references to jQuery and Kendo Mobile and before closing the body tag. This single line of JavaScript code will initialize your Kendo Mobile application and all the widgets with the data-role attribute. The Application object is used for many other purposes . How it works... When you run the index.html page in a browser, you will see a navbar and a tabstrip in the header and footer of the page. Also, the message Hello Kendo!! being shown in the body of the page. The following screenshot shows how it will look like when you view the page on an iPhone: If you have noticed, this looks like a native iOS application. The framework has the capability to render the application that looks like a native application on a device. When you view the same page on an Android device, it will look like an native Android application, as shown in the following screenshot: The framework identifies the platform on which the mobile application is being run and then provides native look and feel to the application. There are ways in which you can customize this behavior. Summary Creating a sample application (Simple)got us started with the Kendo UI Mobile framework and showed us how to build a sample application using the same. We also saw some of the Mobile UI widgets, such as layouts, views, navbar, and tabstrip in brief. Resources for Article : Further resources on this subject: Working with remote data [Article] The Decider: External APIs [Article] Constructing and Evaluating Your Design Solution [Article]
Read more
  • 0
  • 0
  • 4682

article-image-setting-scans
Packt
04 Sep 2013
10 min read
Save for later

Setting up scans

Packt
04 Sep 2013
10 min read
(For more resources related to this topic, see here.) Setting up a scan in Spiceworks The first thing Spiceworks tries to do to scan a network is contact Active Directory(AD); it also uses AD to populate the People portion of your Inventory. Let's set up AD first, as everything else we will be configuring is on the same page. We are all about saving your time and not going back and forth between pages. If you do not have AD in your environment, you can just skip to the Configuring IP range scans section. Scanning and Active Directory There is a wealth of information within AD that Spiceworks uses. We are going to need to configure Spiceworks to log into AD and get that information. OK, we need to get to the Active Directory Configuration screen in Spiceworks in order to do that. As with most things within the app, it is just a couple of clicks. From anywhere in the app, mouse over the Inventory link at the top of the page; a menu will open up. Click on Settings. This will take us to the Settings screen. You will be spending a lot of time here so you can either get very used to these clicks or just have a separate tab open with these settings already set up. The top section is called Getting Started and the first link is Active Directory Configuration. That is our destination for this section so click away. It will take you to the Active Directory Configuration page: There are three sections that are highlighted. Let's go over each and what they do: The area highlighted as 1 is where you are going to enter the credentials that allow Spiceworks to log into your AD and get information. You specify the Active Directory Server (Domain Controller), username and password. Usernames must be in either domain/username or username@domain.com. If you have SSL enabled for AD inquiries, check the Use SSL box. The area highlighted as 2 shows the frequency at which Spiceworks retrieves information from your AD environment. When Spiceworks queries AD, it does not cause a huge amount of traffic or load. Shortening these times should not cause undue stress on your AD servers. This is useful because when you add a user in AD, it will automatically get loaded into Spiceworks at the next scan. If you want any changes you make to users in Spiceworks to be uploaded into your AD environment, the section highlighted as 3 is for you. Just click on the box and any modifications you make in Spiceworks will automatically be synchronized with your AD. There is one more section that is not in the screenshot. This deals with your user portal and help desk. Setting up AD in your Spiceworks really makes a lot of difference with scans and filling in information. It is recommended that if you are running AD, hook this up. If you are wary about Spiceworks writing data into your AD environment, just set up the user that Spiceworks uses to connect as read-only and don't check the box that writes changes back to AD. Easy enough. Since you are convinced that you should connect your AD to Spiceworks, just fill in the ActiveDirectory server, User, and Password fields and click on Save. Spiceworks will automatically test the credentials and let you know immediately if it can connect. If you have some challenges with Spiceworks connecting to your Domain Controller with just the server name, another method is to put the IP address directly into that field. Let's move on to setting up an IP range scan and get some devices into your Spiceworks install. Configuring IP range scans Remember the Settings page that we have been to a couple of times? We are going back! In case you have forgotten, just mouse over to either Inventory or Help Desk and click on the Settings link at the bottom of the left column. Once on the Settings page, we are going to click on the Network Scan link. It is in the first section of links titled Getting Started. This takes us to the main Network Scan page. The first section is where we are going to set up our IP ranges. Since you will not have any ranges in here as you just installed Spiceworks, let's get one configured so you can get some information into the app. To do this, just click on the Add IP Range button and this window will pop up. There is a lot of flexibility that Spiceworks gives you regarding how it scans IP ranges. You can put a fill range (192.168.1.1-254) with or without exclusions, or just a single IP if you so wish. The next box is for exclusions, if you so choose. If you decide you want to scan a range that has both servers and desktops, you can exclude server IP addresses. This is handy. The last options are for scheduling this IP range scan. If you choose the Daily at… option as we have seen in the screenshot, you can also select the time of the day to run this scan. Other options in this drop-down list are every 4, 6, 8, or 12 hours. If you do decide that you want to scan on an hourly basis, the time of the day magically disappears. The bottom of the window lets you select what days of the week you want to run the scan. When Spiceworks runs an initial scan, it can take a bit of time as there is a ton of data that it is collecting. Spiceworks tries a multitude of credentials and reads all information from devices, which it then writes to the database. Once Spiceworks has scanned and written the data to the database, any subsequent scans just write delta data into it. Enter what range you want to scan, any exclusions you choose, and the scan frequency, and click on the Add button. Congratulations! You have just added an IP range scan! Scanning credentials As we have covered, Spiceworks uses a multitude of credentials to try and figure out what is on your network and put those devices into the inventory. This has been completely overhauled in Spiceworks. In this easy-to-use interface, you can enter all the credentials that you are going to need to have a successful scan. Here you can configure multiple usernames/passwords for the following protocols: WMI SSH SNMP Enable ESX/vSphere HTTP iLo SNMP v2c/v3 Telnet Intel vPro As you can see, if you need to put device-specific usernames and passwords into Spiceworks, you can do so using the format, Domainusername. So if you have a server that uses a unique username/password combination, it is easy to set all that up through this interface. The preceding screenshot shows an example of this. Something new in Spiceworks is the section where it shows devices that the credentials were successfully used on. This is really helpful for troubleshooting any scan errors! To add your own username/password combinations, just use these easy-to-follow directions: Click on the protocol you want to add credentials to on the left column (WMI, SNMP, and so on). Click on +Add Account in the middle column labeled Existing Accounts. Enter all the pertinent information on the left pane labeled Edit Account.For usernames that have passwords, there is a Show Password button as well, so you can make sure that you didn't fat finger it! That's it. Just fill in any credentials that will let Spiceworks access your devices on your network, and as far as permissions are concerned you should be good to go! Best practices and kicking off your first Spiceworks scan You have everything you need to start your first Spiceworks scan. It might be best to read the following best practices before you kick it off, though. They will guide you through some potential pitfalls. Scanning best practices For initial scans, be aware of the number of IP addresses you are scanning and the amount of information that Spiceworks is going to pull out of those devices. If you put in a full IP range on your first scan, do not expect Spiceworks to be completed in 10-15 minutes. The initial scan is the most network traffic intensive and will take the longest duration of time. Do full initial scans during nonbusiness hours. Though running an initial full scan shouldn't flood your network, depending on your network configuration, it is always best to run full initial scans during nonbusiness hours just in case. If you are running a 24 x 7 business, break up your IP ranges into smaller chunks and scan that way. Expect some unknown devices. Unless you are a super administrator with a team of hundreds behind you to make sure that every aspect of your network is 100 percent buttoned down, there will most likely be a few devices that Spiceworks cannot connect to. One of the biggest culprits is that WMI has been disabled, or that there is a firewall of some sort blocking Spiceworks from connecting to the machine. Don't get down on yourself if the scan doesn't work 100 percent the first time. If you are really worried about traffic that Spiceworks might cause, what information it collects, or how it will affect workstation performance, just set up a test environment and run a scan there. Whether it be 5 machines or 500, Spiceworks does the same to each one; so test away. Spiceworks is not designed to scan 10,000 devices at one time without a performance hit. If you have a very large network, break it up into smaller chunks for best performance. Spiceworks could get through a 10,000 device scan, but it would hurt performance until the scan is complete. If you have multiple sites linked either by WAN or VPN connections, drop a remote collector at these to run local scans and then send the data back to your main Spiceworks installation. You can find more information at http://community.spiceworks.com/help/Remote_Collectors OK, now that you have read the required best practices, you can set up your IP range on the Network Scan settings page, check the box associated with that range and click on Start Scan. Away you go! Depending on the IP range you set and the time of the day, your scan could take just a few minutes or several hours. If you are having some serious issues trying to get a successful scan, open a browser and hit this site: http://community.spiceworks.com/support. There are in-depth articles and even real-live support folks that can dive into the specifics of your environment, and they won't give up until you are successful. Let's assume that even if you did have an issue, it is resolved and you have got your first scan under your belt. Summary We were provided with details on how to set up a scan in Spiceworks. Also, we got to know how to run the scan we set up and the best practices. Resources for Article : Further resources on this subject: Using SpriteFonts in a Board-based Game with XNA 4.0 [Article] Why CoffeeScript?HTML5 Games Development: Using Local Storage to Store Game Data [Article] Making Money with Your Game [Article]
Read more
  • 0
  • 0
  • 11583

article-image-introduction-xenconvert
Packt
04 Sep 2013
3 min read
Save for later

Introduction to XenConvert

Packt
04 Sep 2013
3 min read
(For more resources related to this topic, see here.) System requirements Since XenConvert can only convert Windows-based hosts and installs on the same host, the requirements are pretty much the same, as follows: Operating system: Windows XP, Windows Vista, Windows 7, Windows Server 2003 (SP1 or later), Windows Server 2008 (R2) .Net Framework 4.0 Disk Space: 40 MB free disk space XenServer version 6.0 or 6.1 Converting a physical machine to a virtual machine Let's take a quick look at how to convert a physical machine to a virtual machine. First we need to install XenConvert on the source physical machine. We can download XenConvert from this link: http://www.citrix.com/downloads/xenserver/tools/conversion.html. Once the standard Windows installation process is complete, launch the XenConvert tool; but before that we need to prepare the host machine for the conversion. To know more about XenConvert, refer to the XenConvert guide at http://support.citrix.com/article/CTX135017. Preparing the host machine For best results, prepare the host machine as follows: Enable Windows Automount on Windows Server operating systems. Disable Windows Autoplay. Remove any virtualization software before performing a conversion. Ensure that adequate free space exists at the destination, which is approximately 101 percent of used space of all source volumes. Remove any network interface teams; they are not applicable to a virtual machine. We need to run the XenConvert tool on the host machine to start the physical-to-virtual conversion. We can convert the physical machine directly to our XenServer if this host machine is accessible. The other options are to convert to VHD, OVF, or vDisk, which can be imported later on to XenServer using some methods. These options are more useful if we don't have enough disk space or connectivity with XenServer. I chose XenServer and clicked on Next . We can select multiple partitions to be included in the conversion, or select none from the drop-down menu in Source Volume and those disks won't be included in the conversion. We can also increase or decrease the size of the new virtual partition to be allocated for this virtual machine. Click on Next . We'll be asked to provide the details of the XenServer host. The hostname needs either an IP address or a FQDN of the XenServer; a username and password are standard login requirements. In the Workspace field, enter the path to the folder to store the intermediate OVF package that XenConvert will use during the conversion process. XenConvert will store the OVF package in the path we give. Click on Next to select the storage repositories found with XenServer and continue to the last step, in which we'll be provided with the summary of the conversion. Soon after the conversion is completed, we'll be able to have this new machine in our XenCenter. We'll need to have XenServer Tools installed on this new virtual machine. Summary In this article we covered an advanced topic that explained the process of converting a physical Windows server to a virtual machine using XenConvert. Resources for Article : Further resources on this subject: Citrix XenApp Performance Essentials [Article] Defining alerts [Article] Publishing applications [Article]
Read more
  • 0
  • 0
  • 8933

article-image-using-virtual-destinations-advanced
Packt
04 Sep 2013
5 min read
Save for later

Using Virtual Destinations (Advanced)

Packt
04 Sep 2013
5 min read
(For more resources related to this topic, see here.) Getting ready For this article we will use the sample application virtual-destinations to demonstrate the use of Virtual Destinations. How to do it... To run the sample for this article, open a terminal, change the path to point to the directory where virtual-destinations is located, and run it by typing mvn compile exec:java. In the terminal where you started the example, you will see output similar to the following snippet, indicating that the application is running: Starting Virtual Destination example now... Queue A Consumer 1 processed 500 Messages Queue A Consumer 2 processed 500 Messages Queue B Consumer processed 1000 Messages Finished running the Virtual Destination example. How it works... Our example takes advantage of ActiveMQ's Virtual Destination feature to allow a JMS Producer to send messages to a topic and have those messages be received by a number of queue consumers. Let's take a look at the sent code first: private void sendMessages() throws Exception { Connection connection = connectionFactory.createConnection(); Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE); Destination destination = session.createTopic("VirtualTopic.Foo"); MessageProducer producer = session.createProducer(destination); for (int i = 0; i < 1000; ++i) { producer.send(session.createMessage()); } connection.close(); } As we can see, there's not much new here; we are just using standard JMS API code to send our messages. The important part is in the name of the topic destination VirtualTopic.Foo, which informs ActiveMQ Broker we want this topic to be one of those special Virtual Destinations. Now let's take a look at the consumer code, and then we'll try and make sense of how this works: Queue queueA = receiverSession.createQueue("Consumer.A. VirtualTopic.Foo"); VirtualMessageListener listenerA1 = new VirtualMessageListener(done); MessageConsumer consumerA1 = receiverSession.createConsumer(queueA); consumerA1.setMessageListener(listenerA1); VirtualMessageListener listenerA2 = new VirtualMessageListener(done); MessageConsumer consumerA2 = receiverSession.createConsumer(queueA); consumerA2.setMessageListener(listenerA2);  Queue queueB = receiverSession.createQueue("Consumer.B. VirtualTopic.Foo"); VirtualMessageListener listenerB = new VirtualMessageListener(done); MessageConsumer consumerB = receiverSession.createConsumer(queueB); consumerB.setMessageListener(listenerB); As we can see, the consumer code is also not very mysterious; we create two consumers on the Consumer.A.VirtualTopic.Foo queue and one on the Consumer.B.VirtualTopic.Foo queue, and yet when we run the example, the two consumers on queue A split 500 of the topic message and the consumer on queue B gets its own copy of the 1,000 messages. So what's happening on the broker then to make this happen? By default, whenever a topic is created with a name matching the pattern VirtualTopic.<TopicName>, we gain access to the feature known as Virtual Destinations. These Virtual Destinations allow us to send a message to a topic but create consumers that have all the benefits of queue consumers, namely, those of load balancing and message persistence. Our queue consumers just need to use their own naming convention to access the Virtual Destination functionality. Each queue that we want to create must be named using the pattern Consumer.<Name>.VirtualTopic.<TopicName>. In the following figure we can see a visual representation of the message flow in our example: As messages are sent to our example's topic, they are distributed to both the queues we created. Two of our consumers split the work of processing messages from queue A, while the other consumer gets all the messages from queue B. In our example we didn't attach a listener to the topic itself, but we could have done so and it would also have received all the messages we sent. It's easy to imagine using the topic as a way to monitor the messages that are being distributed to the queue consumers in this scenario; this pattern is often referred to as a wiretap. If it isn't already apparent, Virtual Destinations are a very powerful feature for your messaging applications. Topics are great for broadcasting events but when we need to be able to consume messages that were sent while a client was offline, the only recourse is to use a durable topic subscription. Unfortunately, durable subscriptions have a number of limitations, the least of which is that only one JMS connection can be actively subscribed to a logical topic subscription. This means that we can't load balance messages and we can't have fast failover if a subscriber goes down. Virtual Destinations solve these problems since all of our consumers subscribe to a queue so their messages are persistent, and the work of processing the messages on the queue can be shared by more than one active subscription. There's more... The naming pattern for Virtual Destinations is not set in stone. By default, ActiveMQ is configured to use the pattern we used in our sample application, but you can easily change this. In the following XML snippet, we configured our broker to all topics on our broker into virtual topics. We use the wildcard syntax here, which matches every topic name a client sends messages to: <broker> <destinationInterceptors> <virtualDestinationInterceptor> <virtualDestinations><virtualTopic name=">" prefix="VirtualTopic Consumers.*."/> </virtualDestinations></virtualDestinationInterceptor> </destinationInterceptors> </broker> More information on Virtual Destinations can be found on the ActiveMQ website, http://activemq.apache.org/virtual-destinations.html. Summary This article thus explained what Virtual Destinations are and how to use them to avoid the limitations of JMS's durable topic subscriptions. Resources for Article: Further resources on this subject: Routing to an external ActiveMQ broker [Article] So, what is Apache Wicket? [Article] Apache Geronimo Logging [Article]
Read more
  • 0
  • 0
  • 5999

article-image-so-what-apache-wicket
Packt
04 Sep 2013
7 min read
Save for later

So, what is Apache Wicket?

Packt
04 Sep 2013
7 min read
(For more resources related to this topic, see here.) Wicket is a component-based Java web framework that uses just Java and HTML. Here, you will see the main advantages of using Apache Wicket in your projects. Using Wicket, you will not have mutant HTML pages. Most of the Java web frameworks require the insertion of special syntax to the HTML code, making it more difficult for Web designers. On the other hand, Wicket adopts HTML templates by using a namespace that follows the XHTML standard. It consists of an id attribute in the Wicket namespace (wicket:id). You won't need scripts to generate messy HTML code. Using Wicket, the code will be clearer, and refactoring and navigating within the code will be easier. Moreover, you can utilize any HTML editor to edit the HTML files, and web designers can work with little knowledge of Wicket in the presentation layer without worrying about business rules and other developer concerns. The advantages for developers are as follows: All code is written in Java No XML configuration files POJO-centric programming No Back-button problems (that is, unexpected and undesirable results on clicking on the browser's Back button) Ease of creating bookmarkable pages Great compile-time and runtime problem diagnosis Easy testability of components Another interesting thing is that concepts such as generics and anonymous subclasses are widely used in Wicket, leveraging the Java programming language to the max. Wicket is based on components. A component is an object that interacts with other components and encapsulates a set of functionalities. Each component should be reusable, replaceable, extensible, encapsulated, and independent, and it does not have a specific context. Wicket provides all these principles to developers because it has been designed taking into account all of them. In particular, the most remarkable principle is reusability. Developers can create custom reusable components in a straightforward way. For instance, you could create a custom component called SearchPanel (by extending the Panel class, which is also a component) and use it in all your other Wicket projects. Wicket has many other interesting features. Wicket also aims to make the interaction of the stateful server-side Java programming language with the stateless HTTP protocol more natural. Wicket's code is safe by default. For instance, it does not encode state in URLs. Wicket is also efficient (for example, it is possible to do a tuning of page-state replication) and scalable (Wicket applications can easily work on a cluster). Last, but not least, Wicket has support for frameworks like EJB and Spring. Installation In seven easy steps, you can build a Wicket "Hello World" application. Step 1 – what do I need? Before you start to use Apache Wicket 6, you will need to check if you have all of the required elements, listed as follows: Wicket is a Java framework, so you need to have Java virtual machine (at least Version 6) installed on your machine. Apache Maven is required. Maven is a tool that can be used for building and managing Java projects. Its main purpose is to make the development process easier and more structured. More information on how to install and configure Maven can be found at http://maven.apache.org. The examples of this book use the Eclipse IDE Juno version, but you can also use other versions or other IDEs, such as NetBeans. In case you are using other versions, check the link for installing the plugins to the version you have; the remaining steps will be the same. In case of other IDEs, you will need to follow some tutorial to install other equivalent plugins or not use them at all. Step 2 – installing the m2eclipse plugin The steps for installing the m2eclipse plugin are as follows: Go to Help | Install New Software. Click on Add and type in m2eclipse in the Name field; copy and paste the link https://repository.sonatype.org/content/repositories/forge-sites/m2e/1.3.0/N/LATEST onto the Location field. Check all options and click on Next. Conclude the installation of the m2eclipse plugin by accepting all agreements and clicking on Finish. Step 3 – creating a new Maven application The steps for creating a new Maven application are as follows: Go to File | New | Project. Then go to Maven | Maven Project. Click on Next and type wicket in the next form. Choose the wicket-archetype-quickstart maven Archetype and click on Next. Fill the next form according to the following screenshot and click on Finish: Step 4 – coding the "Hello World" program In this step, we will build the famous "Hello World" program. The separation of concerns will be clear between HTML and Java code. In this example, and in most cases, each HTML file has a corresponding Java class (with the same name). First, we will analyse the HTML template code. The content of the HomePage.html file must be replaced by the following code: <!DOCTYPE html> <html > <body> <span wicket_id="helloWorldMessage">Test</span> </body> </html> It is simple HTML code with the Wicket template wicket:id="helloWorldMessage". It indicates that in the Java code related to this page, a method will replace the message Test by another message. Now, let's edit the corresponding Java class; that is, HomePage. package com.packtpub.wicket.hello_world; import org.apache.wicket.markup.html.WebPage; import org.apache.wicket.markup.html.basic.Label; public class HomePage extends WebPage { public HomePage() { add(new Label("helloWorldMessage", "Hello world!!!")); } } The class HomePage extends WebPage; that is, it inherits some of the WebPage class's methods and attributes, and it becomes a WebPage subtype. One of these inherited methods is the method add(), where a Label object can be passed as a parameter. A Label object can be built by passing two parameters: an identifier and a string. The method add() is called in the HomePage class's constructor and will change the message in wicket:id="helloWorldMessage" with Hello world!!!. The resulting HTML code will be as shown in the following code snippet: <!DOCTYPE html> <html > <body> <span>Hello world!!!</span> </body> </html> Step 5 – compile and run! The steps to compile and run the project are as follows: To compile, right-click on the project and go to Run As | Maven install. Verify if the compilation was successful. If not, Wicket provides good error messages, so you can try to fix what is wrong. To run the project, right-click on the class Start and go to Run As | Java application. The class Start will run an embedded Jetty instance that will run the application. Verify if the server has started without any problems. Open a web browser and enter this in the address field: http://localhost:8080. In case you have changed the port, enter http://localhost:<port>. The browser should show Hello world!!!. The most common problem that can occur is that port 8080 is already in use. In this case, you can go into the Java Start class (found at src/test/java) and set another port by replacing 8080 in connector. setPort(8080) (line 21) by another number (for example, 9999). To stop the server, you can either click on Console and press any key or click on the red square on the console, which indicates termination. And that's it! By this point, you should have a working Wicket "Hello World" application and are free to play around and discover more about it. Summary This article describes how to create a simple "Hello World" application using Apache Wicket 6. Resources for Article : Further resources on this subject: Tips for Deploying Sakai [Article] OSGi life cycle [Article] Apache Wicket: Displaying Data Using DataTable [Article]
Read more
  • 0
  • 0
  • 5478
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-understanding-point-time-recovery
Packt
04 Sep 2013
28 min read
Save for later

Understanding Point-In-Time-Recovery

Packt
04 Sep 2013
28 min read
(For more resources related to this topic, see here.) Understanding the purpose of PITR PostgreSQL offers a tool called pg_dump to backup a database. Basically, pg_dump will connect to the database, read all the data in transaction isolation level "serializable" and return the data as text. As we are using "serializable", the dump is always consistent. So, if your pg_dump starts at midnight and finishes at 6 A.M, you will have created a backup, which contains all the data as of midnight but no further data. This kind of snapshot creation is highly convenient and perfectly feasible for small to medium amounts of data. A dump is always consistent. This means that all foreign keys are intact; new data added after starting the dump will be missing. It is most likely the most common way to perform standard backups. But, what if your data is so valuable and maybe so large in size that you want to backup it incrementally? Taking a snapshot from time to time might be enough for some applications; for highly critical data, it is clearly not. In addition to that, replaying 20 TB of data in textual form is not efficient either. Point-In-Time-Recovery has been designed to address this problem. How does it work? Based on a snapshot of the database, the XLOG will be replayed later on. This can happen indefinitely or up to a point chosen by you. This way, you can reach any point in time. This method opens the door to many different approaches and features: Restoring a database instance up to a given point in time Creating a standby database, which holds a copy of the original data Creating a history of all changes In this article, we will specifically feature on the incremental backup functionality and describe how you can make your data more secure by incrementally archiving changes to a medium of choice. Moving to the bigger picture The following picture provides an overview of the general architecture in use for Point-In-Time-Recovery: PostgreSQL produces 16 MB segments of transaction log. Every time one of those segments is filled up and ready, PostgreSQL will call the so called archive_command. The goal of archive_command is to transport the XLOG file from the database instance to an archive. In our image, the archive is represented as the pot on the bottom-right side of the image. The beauty of the design is that you can basically use an arbitrary shell script to archive the transaction log. Here are some ideas: Use some simple copy to transport data to an NFS share Run rsync to move a file Use a custom made script to checksum the XLOG file and move it to an FTP server Copy the XLOG file to a tape The possible options to manage XLOG are only limited by imagination. The restore_command is the exact counterpart of the archive_command. Its purpose is to fetch data from the archive and provide it to the instance, which is supposed to replay it (in our image, this is labeled as Restored Backup). As you have seen, replay might be used for replication or simply to restore a database to a given point in time as outlined in this article. Again, the restore_command is simply a shell script doing whatever you wish, file by file. It is important to mention that you, the almighty administrator, are in charge of the archive. You have to decide how much XLOG to keep and when to delete it. The importance of this task cannot be underestimated. Keep in mind, when then archive_command fails for some reason, PostgreSQL will keep the XLOG file and retry after a couple of seconds. If archiving fails constantly from a certain point on, it might happen that the master fills up. The sequence of XLOG files must not be interrupted; if a single file is missing, you cannot continue to replay XLOG. All XLOG files must be present because PostgreSQL needs an uninterrupted sequence of XLOG files; if a single file is missing, the recovery process will stop there at the very latest. Archiving the transaction log After taking a look at the big picture, we can take a look and see how things can be put to work. The first thing you have to do when it comes to Point-In-Time-Recovery is to archive the XLOG. PostgreSQL offers all the configuration options related to archiving through postgresql.conf. Let us see step by step what has to be done in postgresql.conf to start archiving: First of all, you should turn archive_mode on. In the second step, you should configure your archive command. The archive command is a simple shell command taking two parameters: %p: This is a placeholder representing the XLOG file that should be archived, including its full path (source). %f: This variable holds the name of the XLOG without the path pointing to it. Let us set up archiving now. To do so, we should create a place to put the XLOG. Ideally, the XLOG is not stored on the same hardware as the database instance you want to archive. For the sake of this example, we assume that we want to apply an archive to /archive. The following changes have to be made to postgresql.conf: wal_level = archive # minimal, archive, or hot_standby # (change requires restart) archive_mode = on # allows archiving to be done # (change requires restart) archive_command = 'cp %p /archive/%f' # command to use to archive a logfile segment # placeholders: %p = path of file to archive # %f = file name only Once those changes have been made, archiving is ready for action and you can simply restart the database to activate things. Before we restart the database instance, we want to focus your attention on wal_level. Currently three different wal_level settings are available: minimal archive hot_standby The amount of transaction log produced in the case of just a single node is by far not enough to synchronize an entire second instance. There are some optimizations in PostgreSQL, which allow XLOG-writing to be skipped in the case of single-node mode. The following instructions can benefit from wal_level being set to minimal: CREATE TABLE AS, CREATE INDEX, CLUSTER, and COPY (if the table was created or truncated within the same transaction). To replay the transaction log, at least archive is needed. The difference between archive and hot_standby is that archive does not have to know about currently running transactions. For streaming replication, however, this information is vital. Restarting can either be done through pg_ctl –D /data_directory –m fast restart directly or through a standard init script. The easiest way to check if our archiving works is to create some useless data inside the database. The following code snippets shows a million rows can be made easily: test=# CREATE TABLE t_test AS SELECT * FROM generate_series(1,1000000);SELECT 1000000test=# SELECT * FROM t_test LIMIT 3;generate_series----------------- 1 2 3(3 rows) We have simply created a list of numbers. The important thing is that 1 million rows will trigger a fair amount of XLOG traffic. You will see that a handful of files have made it to the archive: iMac:archivehs$ ls -ltotal 131072-rw------- 1 hs wheel 16777216 Mar 5 22:31000000010000000000000001-rw------- 1 hs wheel 16777216 Mar 5 22:31000000010000000000000002-rw------- 1 hs wheel 16777216 Mar 5 22:31000000010000000000000003-rw------- 1 hs wheel 16777216 Mar 5 22:31000000010000000000000004 Those files can be easily used for future replay operations. If you want to save storage, you can also compress those XLOG files. Just add gzip to your archive_command. Taking base backups In the previous section, you have seen that enabling archiving takes just a handful of lines and offers a great deal of flexibility. In this section, we will see how to create a so called base backup, which can be used to apply XLOG later on. A base backup is an initial copy of the data. Keep in mind that the XLOG itself is more or less worthless. It is only useful in combination with the initial base backup. In PostgreSQL, there are two main options to create an initial base backup: Using pg_basebackup Traditional copy/rsync based methods The following two sections will explain in detail how a base backup can be created: Using pg_basebackup The first and most common method to create a backup of an existing server is to run a command called pg_basebackup, which was introduced in PostgreSQL 9.1.0. Basically pg_basebackup is able to fetch a database base backup directly over a database connection. When executed on the slave, pg_basebackup will connect to the database server of your choice and copy all the data files in the data directory over to your machine. There is no need to log into the box anymore, and all it takes is one line of code to run it; pg_basebackup will do all the rest for you. In this example, we will assume that we want to take a base backup of a host called sample.postgresql-support.de. The following steps must be performed: Modify pg_hba.conf to allow replication Signal the master to take pg_hba.conf changes into account Call pg_basebackup Modifying pg_hba.conf To allow remote boxes to log into a PostgreSQL server and to stream XLOG, you have to explicitly allow replication. In PostgreSQL, there is a file called pg_hba.conf, which tells the server which boxes are allowed to connect using which type of credentials. Entire IP ranges can be allowed or simply discarded through pg_hba.conf. To enable replication, we have to add one line for each IP range we want to allow. The following listing contains an example of a valid configuration: # TYPE DATABASE USER ADDRESS METHODhost replication all 192.168.0.34/32 md5 In this case we allow replication connections from 192.168.0.34. The IP range is identified by 32 (which simply represents a single server in our case). We have decided to use MD5 as our authentication method. It means that the pg_basebackup has to supply a password to the server. If you are doing this in a non-security critical environment, using trust as authentication method might also be an option. What happens if you actually have a database called replication in your system? Basically, setting the database to replication will just configure your streaming behavior, if you want to put in rules dealing with the database called replication, you have to quote the database name as follows: "replication". However, we strongly advise not to do this kind of trickery to avoid confusion. Signaling the master server Once pg_hba.conf has been changed, we can tell PostgreSQL to reload the configuration. There is no need to restart the database completely. We have three options to make PostgreSQL reload pg_hba.conf: By running an SQL command: SELECT pg_reload_conf(); By sending a signal to the master: kill –HUP 4711 (with 4711 being the process ID of the master) By calling pg_ctl: pg_ctl –D $PGDATA reload (with $PGDATA being the home directory of your database instance) Once we have told the server acting as data source to accept streaming connections, we can move forward and run pg_basebackup. pg_basebackup – basic features pg_basebackup is a very simple-to-use command-line tool for PostgreSQL. It has to be called on the target system and will provide you with a ready-to-use base backup, which is ready to consume the transaction log for Point-In-Time-Recovery. The syntax of pg_basebackup is as follows: iMac:dbhs$ pg_basebackup --help pg_basebackup takes a base backup of a running PostgreSQL server. Usage: pg_basebackup [OPTION]... Options controlling the output: -D, --pgdata=DIRECTORY receive base backup into directory -F, --format=p|t output format (plain (default), tar) -x, --xlog include required WAL files in backup (fetch mode) -X, --xlog-method=fetch|stream include required WAL files with specified method -z, --gzip compress tar output -Z, --compress=0-9 compress tar output with given compression level General options: -c, --checkpoint=fast|spread set fast or spread checkpointing -l, --label=LABEL set backup label -P, --progress show progress information -v, --verbose output verbose messages -V, --version output version information, then exit -?, --help show this help, then exit Connection options: -h, --host=HOSTNAME database server host or socket directory -p, --port=PORT database server port number -s, --status-interval=INTERVAL time between status packets sent to server (in seconds) -U, --username=NAME connect as specified database user -w, --no-password never prompt for password -W, --password force password prompt (should happen automatically) A basic call to pg_basebackup would look like that: iMac:dbhs$ pg_basebackup -D /target_directory -h sample.postgresql-support.de In this example, we will fetch the base backup from sample.postgresql-support.de and put it into our local directory called /target_directory. It just takes this simple line to copy an entire database instance to the target system. When we create a base backup as shown in this section, pg_basebackup will connect to the server and wait for a checkpoint to happen before the actual copy process is started. In this mode, this is necessary because the replay process will start exactly at this point in the XLOG. The problem is that it might take a while until a checkpoint kicks in; pg_basebackup does not enforce a checkpoint on the source server straight away to make sure that normal operations are not disturbed. If you don't want to wait on a checkpoint, consider using --checkpoint=fast. It will enforce an instant checkpoint and pg_basebackup will start copying instantly. By default, a plain base backup will be created. It will consist of all the files in directories found on the source server. If the base backup should be stored on tape, we suggest to give –-format=t a try. It will automatically create a TAR archive (maybe on a tape). If you want to move data to a tape, you can save an intermediate step easily this way. When using TAR, it is usually quite beneficial to use it in combination with --gzip to reduce the size of the base backup on disk. There is also a way to see a progress bar while doing the base backup but we don't recommend to use this option (--progress) because it requires pg_basebackup to determine the size of the source instance first, which can be costly. pg_basebackup – self-sufficient backups Usually a base backup without XLOG is pretty worthless. This is because the base backup is taken while the master is fully operational. While the backup is taken, those storage files in the database instance might have been modified heavily. The purpose of the XLOG is to fix those potential issues in the data files reliably. But, what if we want to create a base backup, which can live without (explicitly archived) XLOG? In this case, we can use the --xlog-method=stream option. If this option has been chosen, pg_basebackup will not just copy the data as it is but it will also stream the XLOG being created during the base backup to our destination server. This will provide us with just enough XLOG to allow us to start a base backup made that way directly. It is self-sufficient and does not need additional XLOG files. This is not Point-In-Time-Recovery but it can come in handy in case of trouble. Having a base backup, which can be started right away, is usually a good thing and it comes at fairly low cost. Please note that --xlog-method=stream will require two database connections to the source server, not just one. You have to keep that in mind when adjusting max_wal_senders on the source server. If you are planning to use Point-In-Time-Recovery and if there is absolutely no need to start the backup as it is, you can safely skip the XLOG and save some space this way (default mode). Making use of traditional methods to create base backups These days pg_basebackup is the most common way to get an initial copy of a database server. This has not always been the case. Traditionally, a different method has been used which works as follows: Call SELECT pg_start_backup('some label'); Copy all data files to the remote box through rsync or any other means. Run SELECT pg_stop_backup(); The main advantage of this old method is that there is no need to open a database connection and no need to configure XLOG-streaming infrastructure on the source server. A main advantage is also that you can make use of features such as ZFS-snapshots or similar means, which can help to dramatically reduce the amount of I/O needed to create an initial backup. Once you have started pg_start_backup, there is no need to hurry. It is not necessary and not even especially desirable to leave the backup mode soon. Nothing will happen if you are in backup mode for days. PostgreSQL will archive the transaction log as usual and the user won't face any kind of downside. Of course, it is bad habit not to close backups soon and properly. However, the way PostgreSQL works internally does not change when a base backup is running. There is nothing filling up, no disk I/O delayed, or anything of this sort. Tablespace issues If you happen to use more than one tablespace, pg_basebackup will handle this just fine if the filesystem layout on the target box is identical to the filesystem layout on the master. However, if your target system does not use the same filesystem layout there is a bit more work to do. Using the traditional way of doing the base backup might be beneficial in this case. In case you are using --format=t (for TAR), you will be provided with one TAR file per tablespace. Keeping an eye on network bandwidth Let us imagine a simple scenario involving two servers. Each server might have just one disk (no SSDs). Our two boxes might be interconnected through a 1 Gbit link. What will happen to your applications if the second server starts to run a pg_basebackup? The second box will connect, start to stream data at full speed and easily kill your hard drive by using the full bandwidth of your network. An application running on the master might instantly face disk wait and offer bad response times. Therefore it is highly recommended to control the bandwidth used up by rsync to make sure that your business applications have enough spare capacity (mainly disk, CPU is usually not an issue). If you want to limit rsync to, say, 20 MB/sec, you can simply use rsync --bwlimit=20000. This will definitely make the creation of the base backup take longer but it will make sure that your client apps will not face problems. In general we recommend a dedicated network interconnect between master and slave to make sure that a base backup does not affect normal operations. Limiting bandwidth cannot be done with pg_basebackup onboard functionality.Of course, you can use any other tool to copy data and achieve similar results. If you are using gzip compression with –-gzip, it can work as an implicit speed brake. However, this is mainly a workaround. Replaying the transaction log Once we have created ourselves a shiny initial base backup, we can collect the XLOG files created by the database. When the time has come, we can take all those XLOG files and perform our desired recovery process. This works as described in this section. Performing a basic recovery In PostgreSQL, the entire recovery process is governed by a file named recovery.conf, which has to reside in the main directory of the base backup. It is read during startup and tells the database server where to find the XLOG archive, when to end replay, and so forth. To get you started, we have decided to include a simple recovery.conf sample file for performing a basic recovery process: restore_command = 'cp /archive/%f %p'recovery_target_time = '2013-10-10 13:43:12' The restore_command is essentially the exact counterpart of the archive_command you have seen before. While the archive_command is supposed to put data into the archive, the restore_command is supposed to provide the recovering instance with the data file by file. Again, it is a simple shell command or a simple shell script providing one chunk of XLOG after the other. The options you have here are only limited by imagination; all PostgreSQL does is to check for the return code of the code you have written, and replay the data provided by your script. Just like in postgresql.conf, we have used %p and %f as placeholders; the meaning of those two placeholders is exactly the same. To tell the system when to stop recovery, we can set the recovery_target_time. The variable is actually optional. If it has not been specified, PostgreSQL will recover until it runs out of XLOG. In many cases, simply consuming the entire XLOG is a highly desirable process; if something crashes, you want to restore as much data as possible. But, it is not always so. If you want to make PostgreSQL stop recovery at a specific point in time, you simply have to put the proper date in. The crucial part here is actually to know how far you want to replay XLOG; in a real work scenario this has proven to be the trickiest question to answer. If you happen to a recovery_target_time, which is in the future, don't worry, PostgreSQL will start at the very last transaction available in your XLOG and simply stop recovery. The database instance will still be consistent and ready for action. You cannot break PostgreSQL, but, you might break your applications in case data is lost because of missing XLOG. Before starting PostgreSQL, you have to run chmod 700 on the directory containing the base backup, otherwise, PostgreSQL will error out: iMac:target_directoryhs$ pg_ctl -D /target_directorystartserver startingFATAL: data directory "/target_directory" has group or world accessDETAIL: Permissions should be u=rwx (0700). This additional security check is supposed to make sure that your data directory cannot be read by some user accidentally. Therefore an explicit permission change is definitely an advantage from a security point of view (better safe than sorry). Now that we have all the pieces in place, we can start the replay process by starting PostgreSQL: iMac:target_directoryhs$ pg_ctl –D /target_directory startserver startingLOG: database system was interrupted; last known up at 2013-03-1018:04:29 CETLOG: creating missing WAL directory "pg_xlog/archive_status"LOG: starting point-in-time recovery to 2013-10-10 13:43:12+02LOG: restored log file "000000010000000000000006" from archiveLOG: redo starts at 0/6000020LOG: consistent recovery state reached at 0/60000B8LOG: restored log file "000000010000000000000007" from archiveLOG: restored log file "000000010000000000000008" from archiveLOG: restored log file "000000010000000000000009" from archiveLOG: restored log file "00000001000000000000000A" from archivecp: /tmp/archive/00000001000000000000000B: No such file ordirectoryLOG: could not open file "pg_xlog/00000001000000000000000B" (logfile 0, segment 11): No such file or directoryLOG: redo done at 0/AD5CE40LOG: last completed transaction was at log time 2013-03-1018:05:33.852992+01LOG: restored log file "00000001000000000000000A" from archivecp: /tmp/archive/00000002.history: No such file or directoryLOG: selected new timeline ID: 2cp: /tmp/archive/00000001.history: No such file or directoryLOG: archive recovery completeLOG: database system is ready to accept connectionsLOG: autovacuum launcher started The amount of log produced by the database tells us everything we need to know about the restore process and it is definitely worth investigating this information in detail. The first line indicates that PostgreSQL has found out that it has been interrupted and that it has to start recovery. From the database instance point of view, a base backup looks more or less like a crash needing some instant care by replaying XLOG; this is precisely what we want. The next couple of lines (restored log file ...) indicate that we are replaying one XLOG file after the other that have been created since the base backup. It is worth mentioning that the replay process starts at the sixth file. The base backup knows where to start, so PostgreSQL will automatically look for the right XLOG file. The message displayed after PostgreSQL reaches the sixth file (consistent recovery state reached at 0/60000B8) is of importance. PostgreSQL states that it has reached a consistent state. This is important. The reason is that the data files inside a base backup are actually broken by definition, but, the data files are not broken beyond repair. As long as we have enough XLOG to recover, we are very well off. If you cannot reach a consistent state, your database instance will not be usable and your recovery cannot work without providing additional XLOG. Practically speaking, not being able to reach a consistent state usually indicates a problem somewhere in your archiving process and your system setup. If everything up to now has been working properly, there is no reason not to reach a consistent state. Once we have reached a consistent state, one file after the other will be replayed successfully until the system finally looks for the 00000001000000000000000B file. The problem is that this file has not been created by the source database instance. Logically, an error pops up. Not finding the last file is absolutely normal; this type of error is expected if the recovery_target_time does not ask PostgreSQL to stop recovery before it reaches the end of the XLOG stream. Don't worry, your system is actually fine. You have successfully replayed everything to the file showing up exactly before the error message. As soon as all the XLOG has been consumed and the error message discussed earlier has been issued, PostgreSQL reports the last transaction it was able or supposed to replay, and starts up. You have a fully recovered database instance now and you can connect to the database instantly. As soon as the recovery has ended, recovery.conf will be renamed by PostgreSQL to recovery.done to make sure that it does not do any harm when the new instance is restarted later on at some point. More sophisticated positioning in the XLOG Up to now, we have recovered a database up to the very latest moment available in our 16 MB chunks of transaction log. We have also seen that you can define the desired recovery timestamp. But the question now is: How do you know which point in time to perform the recovery to? Just imagine somebody has deleted a table during the day. What if you cannot easily determine the recovery timestamp right away? What if you want to recover to a certain transaction? recovery.conf has all you need. If you want to replay until a certain transaction, you can refer to recovery_target_xid. Just specify the transaction you need and configure recovery_target_inclusive to include this very specific transaction or not. Using this setting is technically easy but as mentioned before, it is not easy by far to find the right position to replay to. In a typical setup, the best way to find a reasonable point to stop recovery is to use pause_at_recovery_target. If this is set to true, PostgreSQL will not automatically turn into a productive instance if the recovery point has been reached. Instead, it will wait for further instructions from the database administrator. This is especially useful if you don't know exactly how far to replay. You can replay, log in, see how far the database is, change to the next target time, and continue replaying in small steps. You have to set hot_standby = on in postgresql.conf to allow reading during recovery. Resuming recovery after PostgreSQL has paused can be done by calling a simple SQL statement: SELECT pg_xlog_replay_resume(). It will make the instance move to the next position you have set in recovery.conf. Once you have found the right place, you can set the pause_at_recovery_target back to false and call pg_xlog_replay_resume. Alternatively, you can simply utilize pg_ctl –D ... promote to stop recovery and make the instance operational. Was this explanation too complicated? Let us boil it down to a simple list: Add a restore_command to the recovery.conf file. Add recovery_target_time to the recovery.conf file. Set pause_at_recovery_target to true in the recovery.conf file. Set hot_standby to on in postgresql.conf file. Start the instance to be recovered. Connect to the instance once it has reached a consistent state and as soon as it stops recovering. Check if you are already where you want to be. If you are not: Change recovery_target_time. Run SELECT pg_xlog_replay_resume(). Check again and repeat this section if it is necessary. Keep in mind that once recovery has finished and once PostgreSQL has started up as a normal database instance, there is (as of 9.2) no way to replay XLOG later on. Instead of going through this process, you can of course always use filesystem snapshots. A filesystem snapshot will always work with PostgreSQL because when you restart a snapshotted database instance, it will simply believe that it had crashed before and recover normally. Cleaning up the XLOG on the way Once you have configured archiving, you have to store the XLOG being created by the source server. Logically, this cannot happen forever. At some point, you really have to get rid of this XLOG; it is essential to have a sane and sustainable cleanup policy for your files. Keep in mind, however, that you must keep enough XLOG so that you can always perform recovery from the latest base backup. But if you are certain that a specific base backup is not needed anymore, you can safely clean out all the XLOG that is older than the base backup you want to keep. How can an administrator figure out what to delete? The best method is to simply take a look at your archive directory: 000000010000000000000005000000010000000000000006000000010000000000000006.00000020.backup000000010000000000000007000000010000000000000008 Check out the filename in the middle of the listing. The .backup file has been created by the base backup. It contains some information about the way the base backup has been made and tells the system where to continue replaying the XLOG. If the backup file belongs to the oldest base backup you need to keep around, you can safely erase all the XLOG lower than file number 6; in this case, file number 5 could be safely deleted. In our case, 000000010000000000000006.00000020.backup contains the following information: START WAL LOCATION: 0/6000020 (file 000000010000000000000006)STOP WAL LOCATION: 0/60000E0 (file 000000010000000000000006)CHECKPOINT LOCATION: 0/6000058BACKUP METHOD: streamedBACKUP FROM: masterSTART TIME: 2013-03-10 18:04:29 CETLABEL: pg_basebackup base backupSTOP TIME: 2013-03-10 18:04:30 CET The .backup file will also provide you with relevant information such as the time the base backup has been made. It is plain there and so it should be easy for ordinary users to read this information. As an alternative to deleting all the XLOG files at one point, it is also possible to clean them up during replay. One way is to hide an rm command inside your restore_command. While this is technically possible, it is not necessarily wise to do so (what if you want to recover again?). Also, you can add the recovery_end_command command to your recovery.conf file. The goal of recovery_end_command is to allow you to automatically trigger some action as soon as the recovery ends. Again, PostgreSQL will call a script doing precisely what you want. You can easily abuse this setting to clean up the old XLOG when the database declares itself active. Switching the XLOG files If you are going for an XLOG file-based recovery, you have seen that one XLOG will be archived every 16 MB. What would happen if you never manage to create 16 MB of changes? What if you are a small supermarket, which just makes 50 sales a day? Your system will never manage to fill up 16 MB in time. However, if your system crashes, the potential data loss can be seen as the amount of data in your last unfinished XLOG file. Maybe this is not good enough for you. A postgresql.conf setting on the source database might help. The archive_timeout tells PostgreSQL to create a new XLOG file at least every x seconds. So, if you are this little supermarket, you can ask the database to create a new XLOG file every day shortly before you are heading for home. In this case, you can be sure that the data of the day will safely be on your backup device already. It is also possible to make PostgreSQL switch to the next XLOG file by hand. A procedure named pg_switch_xlog() is provided by the server to do the job: test=# SELECT pg_switch_xlog();pg_switch_xlog----------------0/17C0EF8(1 row) You might want to call this procedure when some important patch job has finished or if you want to make sure that a certain chunk of data is safely in your XLOG archive. Summary In this article, you have learned about Point-In-Time-Recovery, which is a safe and easy way to restore your PostgreSQL database to any desired point in time. PITR will help you to implement better backup policies and make your setups more robust. Resources for Article: Further resources on this subject: Introduction to PostgreSQL 9 [Article] PostgreSQL: Tips and Tricks [Article] PostgreSQL 9: Reliable Controller and Disk Setup [Article]
Read more
  • 0
  • 0
  • 5261

article-image-understanding-big-picture
Packt
04 Sep 2013
7 min read
Save for later

Understanding the big picture

Packt
04 Sep 2013
7 min read
(For more resources related to this topic, see here.) So we've got this thing for authentication and authorization. Let's see who is responsible and what for. There is an AccessDecisionManager, which, as the name suggests, is responsible for deciding whether we can access something or not; if not, an AccessDeniedException or InsufficientAuthenticationException is thrown. AuthenticationManager is another crucial interface. It is responsible for confirming who we are. Both are just interfaces, so we can swap our own implementations if we like. In a web application, the job of talking with these two components and the user is handled by a web filter called DelegatingFilterProxy, which is decomposed into several small filters. Each one is responsible for a different thing, so we can turn them on, off, or put our own filters in between and mess with them anyway we like. These are quite important, and we will dig into them later. For the big picture, all we need to know is that these filters take care of all the talking, redirect the user to the login page (or an access-denied page), and save the current user details in an HTTPSession. Well, the last part, while true, is a bit misleading. User details are kept in a SecurityContext object, which we can get a hold of by calling SecurityContextHolder.getContext(), and which in the end is stored in HTTPSession by our filters. But we had promised a big picture, not the gory details, so here it is: Quite simple, right? If we have an authentication protocol without login and password, it works in a similar way. We just switch one of the filters, or the authentication manager, to a different implementation. If we don't have a web application, we just need to do the talking ourselves. But this is all for web resources (URLs). What is much more interesting and useful is securing calls to methods. It looks, for example, like this: @PreAuthorize(["isAuthenticated() and hasRole('ROLE_ADMIN')"])public void somethingOnlyAdminCanDo() {} Here, we decided that somethingOnlyAdminCanDo will be protected by our AccessDecisionManager and that the user must be authenticated (not anonymous) and has to have an admin role. Can a user be anonymous and have an admin role at the same time? In theory, yes, but it would not make any sense. Because it's much cheaper to check if he is authenticated and stop right there. We see a bit of optimization in here. We could drop the isAuthenticated() method and the behavior wouldn't change. We can put this kind of annotation on any Java method, but our configuration and mechanism to fire up the security will depend on the type of objects we are trying to protect. For objects declared as Spring beans (which is a short name for anything defined in our Inversion of Control (IoC) configuration, either via XML or annotations), we don't need to do much. Spring will just create proxies (dynamic classes) that take over calls to our secured methods and fire up AccessDecisionManager before passing the call to the object we really wanted to call. For objects outside of the IoC container (anything created with new or just code not defined in Spring context), we can use the power of Aspect Oriented Programming (AOP) to get the same effect. If you don't know what AOP is, don't worry. It's just a bit of magic at the classloader and bytecode level. For now, the only important thing is that it works basically in the same way. This is depicted as follows: We can do much more than this, as we'll see next, but these are the basics. So, how does the AccessDecisionManager decide whether we can access something or not? Imagine a council of very old Jedi masters sitting around a fire. They decide whether or not you are permitted to call a secured method or access a web resource. Each of these masters makes a decision or abstains. Each of them can consult additional information (not only who you are and what you want to do, but every aspect of the situation). In Spring Security, those smart people are called AccessDecisionVoters, and each of them has one vote. The council can be organized in many different ways. It has one voice, and so it may make the decision based on a majority of votes. It may be veto-based, where everything is allowed unless someone disagrees. Or it may need everyone to agree to grant access, otherwise access is denied. The council is the AccessDecisionManager, and we have three implementations previously mentioned out of the box. We can also decide who's in the council and who is not. This is probably the most important decision we can make, because this will decide the security model that we will use in our application. Let's talk about the most popular counselors (implementations of AccessDecisionVoter). Model based on roles (RoleVoter): This guy makes his decision based on the role of the user and the required role for the resource/method. So if we write @PreAuthorize("hasRole('ROLE_ADMIN')"), you better be a damn admin or you'll get a no-no from this guy. Model based on entity access control permissions (AclEntryVoter): This guy doesn't worry about roles. He is much more than that. Acl stands for Access Control List, which represents a list of permissions. Every user has a list of permissions, possibly for every domain object (usually an object in the database), that you want to secure. So, for example, if we have a bank application, the supervisor can give Frank access to a single specific customer (say, ACME—A Company that Makes Everything), which is represented as an entity in the database and as an object in our system. No other employee will be able to do anything to that customer unless the supervisor grants that person the same permission as Frank. This is probably the most scrutinous voter we would ever use. Our customer can have a very detailed configuration with him/her. On the other hand, this is also the most cumbersome, as we need to create a usable graphical interface to set permissions for every user and every domain object. While we have done this a few times, most of our customers wanted a simpler approach, and even those who started with a graphical user interface to configure everything asked for a simplified version based on business rules, at the end of the project. If your customer describes his security needs in terms of rules such as "Frank can edit every customer he has created but he cannot do anything other than view other customers", it means it's time for PreInvocationAuthorizationAdviceVoter. Business rules model (PreInvocationAuthorizationAdviceVoter): This is usually used when you want to implement static business rules in the application. This goes like "if I've written a blog post, I can change it later, but others can only comment" and "if a friend asked me to help him write the blog post, I can do that, because I'm his friend". Most of these things are also possible to implement with ACLs, but would be very cumbersome. This is our favorite voter. With it, it's very easy to write, test, and change the security restrictions, because instead of writing every possible relation in the database (as with ACL voter) or having only dumb roles, we write our security logic in plain old Java classes. Great stuff and most useful, once you see how it works. Did we mention that this is a council? Yes we did. The result of this is that we can mix any voters we want and choose any council organization we like. We can have all three voters previously mentioned and allow access if any of them says "yes". There are even more voters. And we can write new ones ourselves. Do you feel the power of the Jedi council already? Do you feel the power of the Jedi council already? Summary This section provides an overview of authentication and authorization, which are the principles of Spring security. Resources for Article : Further resources on this subject: Migration to Spring Security 3 [Article] Getting Started with Spring Security [Article] So, what is Spring for Android? [Article]
Read more
  • 0
  • 0
  • 8546

article-image-using-gerrit-github
Packt
04 Sep 2013
14 min read
Save for later

Using Gerrit with GitHub

Packt
04 Sep 2013
14 min read
In this article by Luca Milanesio, author of the book Learning Gerrit Code review, we will learn about Gerrit Code revew. GitHub is the world's largest platform for the free hosting of Git Projects, with over 4.5 million registered developers. We will now provide a step-by-step example of how to connect Gerrit to an external GitHub server so as to share the same set of repositories. Additionally, we will provide guidance on how to use the Gerrit Code Review workflow and GitHub concurrently. By the end of this article we will have our Gerrit installation fully integrated and ready to be used for both open source public projects and private projects on GitHub. (For more resources related to this topic, see here.) GitHub workflow GitHub has become the most popular website for open source projects, thanks to the migration of some major projects to Git (for example, Eclipse) and new projects adopting it, along with the introduction of the social aspect of software projects that piggybacks on the Facebook hype. The following diagram shows the GitHub collaboration model: The key aspects of the GitHub workflow are as follows: Each developer pushes to their own repository and pulls from others Developers who want to make a change to another repository, create a fork on GitHub and work on their own clone When forked repositories are ready to be merged, pull requests are sent to the original repository maintainer The pull requests include all of the proposed changes and their associated discussion threads Whenever a pull request is accepted, the change is merged by the maintainer and pushed to their repository on GitHub   GitHub controversy The preceding workflow works very effectively for most open source projects; however, when the projects gets bigger and more complex, the tools provided by GitHub are too unstructured, and a more defined review process with proper tools, additional security, and governance is needed. In May 2012 Linus Torvalds , the inventor of Git version control, openly criticized GitHub as a commit editing tool directly on the pull request discussion thread: " I consider GitHub useless for these kinds of things. It's fine for hosting, but the pull requests and the online commit editing, are just pure garbage " and additionally, " the way you can clone a (code repository), make changes on the web, and write total crap commit messages, without GitHub in any way making sure that the end result looks good. " See https://github.com/torvalds/linux/pull/17#issuecomment-5654674. Gerrit provides the additional value that Linus Torvalds claimed was missing in the GitHub workflow: Gerrit and GitHub together allows the open source development community to reuse the extended hosting reach and social integration of GitHub with the power of governance of the Gerrit review engine. GitHub authentication The list of authentication backends supported by Gerrit does not include GitHub and it cannot be used out of the box, as it does not support OpenID authentication. However, a GitHub plugin for Gerrit has been recently released in order to fill the gaps and allow a seamless integration. GitHub implements OAuth 2.0 for allowing external applications, such as Gerrit, to integrate using a three-step browser-based authentication. Using this scheme, a user can leverage their existing GitHub account without the need to provision and manage a separate one in Gerrit. Additionally, the Gerrit instance will be able to self-provision the SSH public keys needed for pushing changes for review. In order for us to use GitHub OAuth authentication with Gerrit, we need to do the following: Build the Gerrit GitHub plugin Install the GitHub OAuth filter into the Gerrit libraries (/lib under the Gerrit site directory) Reconfigure Gerrit to use the HTTP authentication type   Building the GitHub plugin The Gerrit GitHub plugin can be found under the Gerrit plugins/github repository on https://gerrit-review.googlesource.com/#/admin/projects/plugins/github. It is open source under the Apache 2.0 license and can be cloned and built using the Java 6 JDK and Maven. Refer to the following example: $ git clone https://gerrit.googlesource.com/plugins/github $ cd github $ mvn install […] [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------- [INFO] Total time: 9.591s [INFO] Finished at: Wed Jun 19 18:38:44 BST 2013 [INFO] Final Memory: 12M/145M [INFO] ------------------------------------------------------- The Maven build should generate the following artifacts: github-oauth/target/github-oauth*.jar, the GitHub OAuth library for authenticating Gerrit users github-plugin/target/github-plugin*.jar, the Gerrit plugin for integrating with GitHub repositories and pull requests Installing GitHub OAuth library The GitHub OAuth JAR file needs to copied to the Gerrit /lib directory; this is required to allow Gerrit to use it for filtering all HTTP requests and enforcing the GitHub three-step authentication process: $ cp github-oauth/target/github-oauth-*.jar /opt/gerrit/lib/ Installing GitHub plugin The GitHub plugin includes the additional support for the overall configuration, the advanced GitHub repositories replication, and the integration of pull requests into the Code Review process. We now need to install the plugin before running the Gerrit init again so that we can benefit from the simplified automatic configuration steps: $ cp github-plugin/target/github-plugin-*.jar /opt/gerrit/plugins/github.jar Register Gerrit as a GitHub OAuth application Before going through the Gerrit init, we need to tell GitHub to trust Gerrit as a partner application. This is done through the generation of a ClientId/ClientSecret pair associated to the exact Gerrit URLs that will be used for initiating the 3-step OAuth authentication. We can register a new application in GitHub through the URL https://github.com/settings/applications/new, where the following three fields are requested: Application name : It is the logical name of the application authorized to access GitHub, for example, Gerrit. Main URL : The Gerrit canonical web URL used for redirecting to GitHub OAuth authentication, for example, https://myhost.mydomain:8443. Callback URL : The URL that GitHub should redirect to when the OAuth authentication is successfully completed, for example, https://myhost.mydomain:8443/oauth. GitHub will automatically generate a unique pair ClientId/ClientSecret that has to be provided to Gerrit identifying them as a trusted authentication partner. ClientId/ClientSecret are not GitHub credentials and cannot be used by an interactive user to access any GitHub data or information. They are only used for authorizing the integration between a Gerrit instance and GitHub. Running Gerrit init to configure GitHub OAuth We now need to stop Gerrit and go through the init steps again in order to reconfigure the Gerrit authentication. We need to enable HTTP authentication by choosing an HTTP header to be used to verify the user's credentials, and to go through the GitHub settings wizard to configure the OAuth authentication. $ /opt/gerrit/bin/gerrit.sh stop Stopping Gerrit Code Review: OK $ cd /opt/gerrit $ java -jar gerrit.war init [...] *** User Authentication *** Authentication method []: HTTP RETURN Get username from custom HTTP header [Y/n]? Y RETURN Username HTTP header []: GITHUB_USER RETURN SSO logout URL : /oauth/reset RETURN *** GitHub Integration *** GitHub URL [https://github.com]: RETURN Use GitHub for Gerrit login ? [Y/n]? Y RETURN ClientId []: 384cbe2e8d98192f9799 RETURN ClientSecret []: f82c3f9b3802666f2adcc4 RETURN Initialized /opt/gerrit $ /opt/gerrit/bin/gerrit.sh start Starting Gerrit Code Review: OK   Using GitHub login for Gerrit Gerrit is now fully configured to register and authenticate users through GitHub OAuth. When opening the browser to access any Gerrit web pages, we are automatically redirected to the GitHub for login. If we have already visited and authenticated with GitHub previously, the browser cookie will be automatically recognized and used for the authentication, instead of presenting the GitHub login page. Alternatively, if we do not yet have a GitHub account, we create a new GitHub profile by clicking on the SignUp button. Once the authentication process is successfully completed, GitHub requests the user's authorization to grant access to their public profile information. The following screenshot shows GitHub OAuth authorization for Gerrit: The authorization status is then stored under the user's GitHub applications preferences on https://github.com/settings/applications. Finally, GitHub redirects back to Gerrit propagating the user's profile securely using a one-time code which is used to retrieve the full data profile including username, full name, e-mail, and associated SSH public keys. Replication to GitHub The next steps in the Gerrit to GitHub integration is to share the same Git repositories and then keep them up-to-date; this can easily be achieved by using the Gerrit replication plugin. The standard Gerrit replication is a master-slave, where Gerrit always plays the role of the master node and pushes to remote slaves. We will refer to this scheme as push replication because the actual control of the action is given to Gerrit through a git push operation of new commits and branches. Configure Gerrit replication plugin In order to configure push replication we need to enable the Gerrit replication plugin through Gerrit init: $ /opt/gerrit/bin/gerrit.sh stop Stopping Gerrit Code Review: OK $ cd /opt/gerrit $ java -jar gerrit.war init [...] *** Plugins *** Prompt to install core plugins [y/N]? y RETURN Install plugin reviewnotes version 2.7-rc4 [y/N]? RETURN Install plugin commit-message-length-validator version 2.7-rc4 [y/N]? RETURN Install plugin replication version 2.6-rc3 [y/N]? y RETURN Initialized /opt/gerrit $ /opt/gerrit/bin/gerrit.sh start Starting Gerrit Code Review: OK The Gerrit replication plugin relies on the replication.config file under the /opt/gerrit/etc directory to identify the list of target Git repositories to push to. The configuration syntax is a standard .ini format where each group section represents a target replica slave. See the following simplest replication.config script for replicating to GitHub: [remote "github"] url = git@github.com:myorganisation/${name}.git The preceding configuration enables all of the repositories in Gerrit to be replicated to GitHub under the myorganisa tion GitHub Team account. Authorizing Gerrit to push to GitHub Now, that Gerrit knows where to push, we need GitHub to authorize the write operations to its repositories. To do so, we need to upload the SSH public key of the underlying OS user where Gerrit is running to one of the accounts in the GitHub myorganisation team, with the permissions to push to any of the GitHub repositories. Assuming that Gerrit runs under the OS user gerrit, we can copy and paste the SSH public key values from the ~gerrit/.ssh/id_rsa.pub (or ~gerrit/.ssh/id_dsa.pub) to the Add an SSH Key section of the GitHub account under target URL to be set to: https://github.com/settings/ssh Start working with Gerrit replication Everything is now ready to start playing with Gerrit to GitHub replication. Whenever a change to a repository is made on Gerrit, it will be automatically replicated to the corresponding GitHub repository. In reality there is one additional operation that is needed on the GitHub side: the actual creation of the empty repositories using https://github.com/new associated to the ones created in Gerrit. We need to make sure that we select the organization name and repository name, consistent with the ones defined in Gerrit and in the replication.config file. Never initialize the repository from GitHub with an empty commit or readme file; otherwise the first replication attempt from Gerrit will result in a conflict and will then fail. Now GitHub and Gerrit are fully connected and whenever a repository in GitHub matches one of the repositories in Gerrit, it will be linked and synchronized with the latest set of commits pushed in Gerrit. Thanks to the Gerrit-GitHub authentication previously configured, Gerrit and GitHub share the same set of users and the commits authors will be automatically recognized and formatted by GitHub. The following screenshot shows Gerrit commits replicated to GitHub: Reviewing and merging to GitHub branches The final goal of the Code Review process is to agree and merge changes to their branches. The merging strategies need to be aligned with real-life scenarios that may arise when using Gerrit and GitHub concurrently. During the Code Review process the alignment between Gerrit and GitHub was at the change level, not influenced by the evolution of their target branches. Gerrit changes and GitHub pull requests are isolated branches managed by their review lifecycle. When a change is merged, it needs to align with the latest status of its target branch using a fast-forward, merge, rebase, or cherry-pick strategy. Using the standard Gerrit merge functionality, we can apply the configured project merge strategy to the current status of the target branch on Gerrit. The situation on GitHub may have changed as well, so even if the Gerrit merge has succeeded there is no guarantee that the actual subsequent synchronization to GitHub will do the same! The GitHub plugin mitigates this risk by implementing a two-phase submit + merge operation for merging opened changes as follows: Phase-1 : The change target branch is checked against its remote peer on GitHub and fast forwarded if needed. If two branches diverge, the submit + merge is aborted and manual merge intervention is requested. Phase-2 : The change is merged on its target branch in Gerrit and an additional ad hoc replication is triggered. If the merge succeeds then the GitHub pull request is marked as completed. At the end of Phase-2 the Gerrit and GitHub statuses will be completely aligned. The pull request author will then receive the notification that his/her commit has been merged. Using Gerrit and GitHub on http://gerrithub.io When using Gerrit and GitHub on the web with public or private repositories, all of the commits are replicated from Gerrit to GitHub, and each one of them has a complete copy of the data. If we are using a Git and collaboration server on GitHub over the Internet, why can't we do the same for its Gerrit counterpart? Can we avoid installing a standalone instance of Gerrit just for the purpose of going through a formal Code Review? One hassle-free solution is to use the GerritHub service (http://gerrithub.io), which offers a free Gerrit instance on the cloud already configured and connected with GitHub through the github-plugin and github-oauth authentication library. All of the flows that we have covered in this article are completely automated, including the replication and automatic pull request to change automation. As accounts are shared with GitHub, we do not need to register or create another account to use GerritHub; we can just visit http://gerrithub.io and start using Gerrit Code Review with our existing GitHub projects without having to teach our existing community about a new tool. GerritHub also includes an initial setup Wizard for the configuration and automation of the Gerrit projects and the option to configure the Gerrit groups using the existing GitHub. Once Gerrit is configured, the Code Review and GitHub can be used seamlessly for achieving maximum control and social reach within your developer community. Summary We have now integrated our Gerrit installation with GitHub authentication for a seamless Single-Sign-On experience. Using an existing GitHub account we started using Gerrit replication to automatically mirror all the commits to GitHub repositories, allowing our projects to have an extended reach to external users, free to fork our repositories, and to contribute changes as pull requests. Finally, we have completed our Code Review in Gerrit and managed the merge to GitHub with a two-phase change submit + merge process to ensure that the target branches on both Gerrit and GitHub have been merged and aligned accordingly. Similarly to GitHub, this Gerrit setup can be leveraged for free on the web without having to manage a separate private instance, thanks to the free set target URL to http://gerrithub.io service available on the cloud. Resources for Article : Further resources on this subject: Getting Dynamics NAV 2013 on Your Computer – For (Almost) Free [Article] Building Your First Zend Framework Application [Article] Quick start - your first Sinatra application [Article]
Read more
  • 0
  • 1
  • 51232

article-image-audio-playback
Packt
04 Sep 2013
17 min read
Save for later

Audio Playback

Packt
04 Sep 2013
17 min read
(For more resources related to this topic, see here.) Understanding FMOD One of the main reasons why I chose FMOD for this book is that it contains two separate APIs—the FMOD Ex Programmer's API, for low-level audio playback, and FMOD Designer, for high-level data-driven audio. This will allow us to cover game audio programming at different levels of abstraction without having to use entirely different technologies. Besides that reason, FMOD is also an excellent piece of software, with several advantages to game developers: License: It is free for non-commercial use, and has reasonable licenses for commercial projects. Cross-platform: It works across an impressive number of platforms. You can run it on Windows, Mac, Linux, Android, iOS, and on most of the modern video game consoles by Sony, Microsoft, and Nintendo. Supported formats: It has native support for a huge range of audio file formats, which saves you the trouble of having to include other external libraries and decoders. Programming languages: Not only can you use FMOD with C and C++, there are also bindings available for other programming languages, such as C# and Python. Popularity: It is extremely popular, being widely considered as the industry standard nowadays. It was used in games such as BioShock, Crysis, Diablo 3, Guitar Hero, Start Craft II, and World of Warcraft. It is also used to power several popular game engines, such as Unity3D and CryEngine. Features: It is packed with features, covering everything from simple audio playback, streaming and 3D sound, to interactive music, DSP effects and low-level audio programming. Installing FMOD Ex Programmer's API Installing a C++ library can be a bit daunting at first. The good side is that once you have done it for the first time, the process is usually the same for every other library. Here are the steps that you should follow if you are using Microsoft Visual Studio: Download the FMOD Ex Programmer's API from http://www.fmod.org and install it to a folder that you can remember, such as C:FMOD. Create a new empty project, and add at least one .cpp file to it. Then, right-click on the project node on the Solution Explorer , and select Properties from the list. For all the steps that follow, make sure that the Configuration option is set to All Configurations . Navigate to C/C++ | General , and add C:FMODapiinc to the list of Additional Include Directories (entries are separated by semicolons). Navigate to Linker | General , and add C:FMODapilib to the list of Additional Library Directories . Navigate to Linker | Input , and add fmodex_vc.lib to the list of Additional Dependencies . Navigate to Build Events | Post-Build Event , and add xcopy /y "C:FMODapifmodex.dll" "$(OutDir)" to the Command Lin e list. Include the <fmod.hpp> header file from your code. Creating and managing the audio system Everything that happens inside FMOD is managed by a class named FMOD::System, which we must start by instantiating with the FMOD::Syste m_Create() function: FMOD::System* system; FMOD::System_Create(&system); Notice that the function returns the system object through a parameter. You will see this pattern every time one of the FMOD functions needs to return a value, because they all reserve the regular return value for an error code. We will discuss error checking in a bit, but for now let us get the audio engine up and running. Now that we have a system object instantiated, we also need to initialize it by calling the init() method: system->init(100, FMOD_INIT_NORMAL, 0); The first parameter specifies the maximum number of channels to allocate. This controls how many sounds you are able to play simultaneously. You can choose any number for this parameter because the system performs some clever priority management behind the scenes and distributes the channels using the available resources. The second and third parameters customize the initialization process, and you can usually leave them as shown in the example. Many features that we will use work properly only if we update the system object every frame. This is done by calling the update() method from inside your game loop: system->update(); You should also remember to shutdown the system object before your game ends, so that it can dispose of all resources. This is done by calling the release() method: system->release(); Loading and streaming audio files One of the greatest things about FMOD is that you can load virtually any audio file format with a single method call. To load an audio file into memory, use the createSound() method: FMOD::Sound* sound; system->createSound("sfx.wav", FMOD_DEFAULT, 0, &sound); To stream an audio file from disk without having to store it in memory, use the createStream() method: FMOD::Sound* stream; system->createStream("song.ogg", FMOD_DEFAULT, 0, &stream); Both methods take the path of the audio file as the first parameter, and return a pointer to an FMOD::Sound object through the fourth parameter, which you can use to play the sound. The paths in the previous examples are relative to the application path. If you are running these examples in Visual Studio, make sure that you copy the audio files into the output folder (for example, using a post-build event such as xcopy /y "$(ProjectDir)*.ogg" "$(OutDir)"). The choice between loading and streaming is mostly a tradeoff between memory and processing power. When you load an audio file, all of its data is uncompressed and stored in memory, which can take up a lot of space, but the computer can play it without much effort. Streaming, on the other hand, barely uses any memory, but the computer has to access the disk constantly, and decode the audio data on the fly. Another difference (in FMOD at least) is that when you stream a sound, you can only have one instance of it playing at any time. This limitation exists because there is only one decode buffer per stream. Therefore, for sound effects that have to be played multiple times simultaneously, you have to either load them into memory, or open multiple concurrent streams. As a rule of thumb, streaming is great for music tracks, voice cues, and ambient tracks, while most sound effects should be loaded into memory. The second and third parameters allow us to customize the behavior of the sound. There are many different options available, but the following list summarizes the ones we will be using the most. Using FMOD_DEFAULT is equivalent to combining the first option of each of these categories: FMOD_LOOP_OFF and FMOD_LOOP_NORMAL: These modes control whether the sound should only play once, or loop once it reaches the end FMOD_HARDWARE and FMOD_SOFTWARE: These modes control whether the sound should be mixed in hardware (better performance) or software (more features) FMOD_2D and FMOD_3D: These modes control whether to use 3D sound We can combine multiple modes using the bitwise OR operator (for instance, FMOD_DEFAULT | FMOD_LOOP_NORMAL | FMOD_SOFTWARE). We can also tell the system to stream a sound even when we are using the createSound() method, by setting the FMOD_CREATESTREAM flag. In fact, the createStream() method is simply a shortcut for this. When we do not need a sound anymore (or at the end of the game) we should dispose of it by calling the release() method of the sound object. We should always release the sounds we create, regardless of the audio system also being released. sound->release(); Playing sounds With the sounds loaded into memory or prepared for streaming, all that is left is telling the system to play them using the playSound() method: FMOD::Channel* channel; system->playSound(FMOD_CHANNEL_FREE, sound, false, &channel); The first parameter selects in which channel the sound will play. You should usually let FMOD handle it automatically, by passing FMOD_CHANNEL_FREE as the parameter. The second parameter is a pointer to the FMOD::Sound object that you want to play. The third parameter controls whether the sound should start in a paused state, giving you a chance to modify some of its properties without the changes being audible. If you set this to true, you will also need to use the next parameter so that you can unpause it later. The fourth parameter is an output parameter that returns a pointer to the FMOD::Channel object in which the sound will play. You can use this handle to control the sound in multiple ways, which will be the main topic of the next chapter. You can ignore this last parameter if you do not need any control over the sound, and simply pass in 0 in its place. This can be useful for non-lopping one-shot sounds. system->playSound(FMOD_CHANNEL_FREE, sound, false, 0); Checking for errors So far, we have assumed that every operation will always work without errors. However, in a real scenario, there is room for a lot to go wrong. For example, we could try to load an audio file that does not exist. In order to report errors, every function and method in FMOD has a return value of type FMOD_RESULT, which will only be equal to FMOD_OK if everything went right. It is up to the user to check this value and react accordingly: FMOD_RESULT result = system->init(100, FMOD_INIT_NORMAL, 0); if (result != FMOD_OK) { // There was an error, do something about it } For starters, it would be useful to know what the error was. However, since FMOD_RESULT is an enumeration, you will only see a number if you try to print it. Fortunately, there is a function called FMOD_ErrorString() inside the fmod_errors.h header file which will give you a complete description of the error. You might also want to create a helper function to simplify the error checking process. For instance, the following function will check for errors, print a description of the error to the standard output, and exit the application: #include <iostream> #include <fmod_errors.h> void ExitOnError(FMOD_RESULT result) { if (result != FMOD_OK) { std::cout << FMOD_ErrorString(result) << std::endl; exit(-1); } } You could then use that function to check for any critical errors that should cause the application to abort: ExitOnError(system->init(100, FMOD_INIT_NORMAL, 0)); The initialization process described earlier also assumes that everything will go as planned, but a real game should be prepared to deal with any errors. Fortunately, there is a template provided in the FMOD documentation which shows you how to write a robust initialization sequence. It is a bit long to cover here, so I urge you to refer to the file named Getting started with FMOD for Windows.pdf inside the documentation folder for more information. For clarity, all of the code examples will continue to be presented without error checking, but you should always check for errors in a real project. Project 1 building a simple audio manager In this project, we will be creating a SimpleAudioManager class that combines everything that was covered in this chapter. Creating a wrapper for an underlying system that only exposes the operations that we need is known as the façade design pattern , and is very useful in order to keep things nice and simple. Since we have not seen how to manipulate sound yet, do not expect this class to be powerful enough to be used in a complex game. Its main purpose will be to let you load and play one-shot sound effects with very little code (which could in fact be enough for very simple games). It will also free you from the responsibility of dealing with sound objects directly (and having to release them) by allowing you to refer to any loaded sound by its filename. The following is an example of how to use the class: SimpleAudioManager audio; audio.Load("explosion.wav"); audio.Play("explosion.wav"); From an educational point of view, what is perhaps even more important is that you use this exercise as a way to get some ideas on how to adapt the technology to your needs. It will also form the basis of the next chapters in the book, where we will build systems that are more complex. Class definition Let us start by examining the class definition: #include <string> #include <map> #include <fmod.hpp> typedef std::map<std::string, FMOD::Sound*> SoundMap; class SimpleAudioManager { public: SimpleAudioManager(); ~SimpleAudioManager(); void Update(float elapsed); void Load(const std::string& path); void Stream(const std::string& path); void Play(const std::string& path); private: void LoadOrStream(const std::string& path, bool stream); FMOD::System* system; SoundMap sounds; }; From browsing through the list of public class members, it should be easy to deduce what it is capable of doing: The class can load audio files (given a path) using the Load() method The class can stream audio files (given a path) using the Stream() method The class can play audio files (given a path) using the Play() method (granted that they have been previously loaded or streamed) There is also an Update() method and a constructor/destructor pair to manage the sound system The private class members, on the other hand, can tell us a lot about the inner workings of the class: At the core of the class is an instance of FMOD::System responsible for driving the entire sound engine. The class initializes the sound system on the constructor, and releases it on the destructor. Sounds are stored inside an associative container, which allows us to search for a sound given its file path. For this purpose, we will be relying on one of the C++ Standard Template Library (STL ) associative containers, the std::map class, as well as the std::string class for storing the keys. Looking up a string key is a bit inefficient (compared to an integer, for example), but it should be fast enough for our needs. An advantage of having all the sounds stored on a single container is that we can easily iterate over them and release them from the class destructor. Since the code for loading and streaming audio file is almost the same, the common functionality has been extracted into a private method called LoadOrStream(), to which Load() and Stream() delegate all of the work. This prevents us from repeating the code needlessly. Initialization and destruction Now, let us walk through the implementation of each of these methods. First we have the class constructor, which is extremely simple, as the only thing that it needs to do is initialize the system object. SimpleAudioManager::SimpleAudioManager() { FMOD::System_Create(&system); system->init(100, FMOD_INIT_NORMAL, 0); } Updating is even simpler, consisting of a single method call: void SimpleAudioManager::Update(float elapsed) { system->update(); } The destructor, on the other hand, needs to take care of releasing the system object, as well as all the sound objects that were created. This process is not that complicated though. First, we iterate over the map of sounds, releasing each one in turn, and clearing the map at the end. The syntax might seem a bit strange if you have never used an STL iterator before, but all that it means is to start at the beginning of the container, and keep advancing until we reach its end. Then we finish off by releasing the system object as usual. SimpleAudioManager::~SimpleAudioManager() { // Release every sound object and clear the map SoundMap::iterator iter; for (iter = sounds.begin(); iter != sounds.end(); ++iter) iter->second->release(); sounds.clear(); // Release the system object system->release(); system = 0; } Loading or streaming sounds Next in line are the Load() and Stream() methods, but let us examine the private LoadOrStream() method first. This method takes the path of the audio file as a parameter, and checks if it has already been loaded (by querying the sound map). If the sound has already been loaded there is no need to do it again, so the method returns. Otherwise, the file is loaded (or streamed, depending on the value of the second parameter) and stored in the sound map under the appropriate key. void SimpleAudioManager::LoadOrStream(const std::string& path, bool stream) { // Ignore call if sound is already loaded if (sounds.find(path) != sounds.end()) return; // Load (or stream) file into a sound object FMOD::Sound* sound; if (stream) system->createStream(path.c_str(), FMOD_DEFAULT, 0, &sound); else system->createSound(path.c_str(), FMOD_DEFAULT, 0, &sound); // Store the sound object in the map using the path as key sounds.insert(std::make_pair(path, sound)); } With the previous method in place, both the Load() and the Stream() methods can be trivially implemented as follows: void SimpleAudioManager::Load(const std::string& path) { LoadOrStream(path, false); } void SimpleAudioManager::Stream(const std::string& path) { LoadOrStream(path, true); } Playing sounds Finally, there is the Play() method, which works the other way around. It starts by checking if the sound has already been loaded, and does nothing if the sound is not found on the map. Otherwise, the sound is played using the default parameters. void SimpleAudioManager::Play(const std::string& path) { // Search for a matching sound in the map SoundMap::iterator sound = sounds.find(path); // Ignore call if no sound was found if (sound == sounds.end()) return; // Otherwise play the sound system->playSound(FMOD_CHANNEL_FREE, sound->second, false, 0); } We could have tried to automatically load the sound in the case when it was not found. In general, this is not a good idea, because loading a sound is a costly operation, and we do not want that happening during a critical gameplay section where it could slow the game down. Instead, we should stick to having separate load and play operations. A note about the code samples Although this is a book about audio, all the samples need an environment to run on. In order to keep the audio portion of the samples as clear as possible, we will also be using the Simple and Fast Multimedia Library 2.0 (SFML ) (http://www.sfml-dev.org). This library can very easily take care of all the miscellaneous tasks, such as window creation, timing, graphics, and user input, which you will find in any game. For example, here is a complete sample using SFML and the SimpleAudioManager class. It creates a new window, loads a sound, runs a game loop at 60 frames per second, and plays the sound whenever the user presses the space key. #include <SFML/Window.hpp> #include "SimpleAudioManager.h" int main() { sf::Window window(sf::VideoMode(320, 240), "AudioPlayback"); sf::Clock clock; // Place your initialization logic here SimpleAudioManager audio; audio.Load("explosion.wav"); // Start the game loop while (window.isOpen()) { // Only run approx 60 times per second float elapsed = clock.getElapsedTime().asSeconds(); if (elapsed < 1.0f / 60.0f) continue; clock.restart(); sf::Event event; while (window.pollEvent(event)) { // Handle window events if (event.type == sf::Event::Closed) window.close(); // Handle user input if (event.type == sf::Event::KeyPressed && event.key.code == sf::Keyboard::Space) audio.Play("explosion.wav"); } // Place your update and draw logic here audio.Update(elapsed); } // Place your shutdown logic here return 0; } Summary In this article, we have seen some of the advantages of using the FMOD audio engine. We saw how to install the FMOD Ex Programmer's API in Visual Studio, how to initialize, manage, and release the FMOD sound system, how to load or stream an audio file of any type from disk, how to play a sound that has been previously loaded by FMOD, how to check for errors in every FMOD function, and how to create a simple audio manager that encapsulates the act of loading and playing audio files behind a simple interface. Resources for Article : Further resources on this subject: Using SpriteFonts in a Board-based Game with XNA 4.0 [Article] HTML5 Games Development: Using Local Storage to Store Game Data [Article] Making Money with Your Game [Article]
Read more
  • 0
  • 0
  • 6656
article-image-lets-make-particles
Packt
04 Sep 2013
15 min read
Save for later

Let's Make Particles

Packt
04 Sep 2013
15 min read
(For more resources related to this topic, see here.) The particle systems in Motion 5 are a powerful engine by which we can take nearly any object, image, layer, or group and animate it using the parameters available to us in the HUD and Inspector. A particle system consists of two items—the emitter and the cell. The cell is referenced by the emitter and the emitter creates the animation over a lifespan specified by you. Let's say we had a PNG layer of an orange. If we created particles out of it, that orange would be put into a cell that is referenced by the emitter. You could use the emitter's parameters to duplicate that orange multiple times per second and have it animate in a particular direction until you decide it should end or die. On top of the ability to turn nearly anything your heart desires into particles, in Motion's library there are pre-animated particle emitters available to incorporate in all of your animations. Making particles and changing values in the HUD Let's take a look at how we can create a particle system using a shape from Motion's library and tweak a few of its parameters using the HUD. Getting ready From the exercise files of this article, double-click the 07_01 project. There is the Shape layer in the Layers tab, whose scale has been animated to repeat for the duration of the project. Our goal is to place the heart in a particle system so that we can have hundreds rain down onto the Canvas. How to do it... The following steps will take you through creating your first particle system: Make sure your playhead is at the beginning of your project. Select the Pink Heart layer and press E , or from the toolbar choose Create a particle emitter (the icon with the three bubbles rising up, as shown in the following screenshot): Play back the project. Several things just happened after you pressed that button. In the Layers tab, notice that the Pink Heart layer has been turned off. A cell and emitter have been created just above it. The cell holds information about the heart while the emitter is creating all the duplicate copies of the heart that are shooting out in a 360-degree circle across the screen at the same speed. You can see a screenshot of this next. Press F7 to bring up the HUD and see some of these parameters in more detail: Right now, thirty hearts are being born every second and live for a duration of five seconds where they pop off the screen. All the duplicate hearts also hold the original scale of the heart being referenced. Change the Birth Rate value to 5 and Life to 10 . Bring down the Scale parameter to 50 , as shown in the following screenshot: Instead of having the hearts come from the center of the screen, let's have them rain down from the top of the Canvas. Decrease the size of your Canvas by clicking on it and pressing Command + - a few times. Select the particle emitter and drag it up and offscreen. Use the Shift key to constrain the movement. Change the Emission Range slider from 360 to 180 and make sure the arrows point down. Play back the animation and tweak the Birth Rate and Scale sliders as desired. See the following screenshot for reference: There's more… If you're finding the HUD limited in terms of the options available to you for the emitter, don't worry, the Inspector has several additional parameters, including the option to add random values to your emitter that will add more realism to it! Take a peak at the following screenshot: Tweaking particle parameters in the Inspector In the previous recipe, we switched a few parameters in the HUD for our particle system of hearts, but if we're looking to fine-tune our animation, we need to go to the Inspector. Let's take a look at the several additional parameters available. Getting ready From the exercise files of this article, double-click the 07_02 project. Play back the project. There is a particle system in the center of the Canvas being emitted in the form of a rectangle. Currently, thirty hearts are being born every second and live for five seconds. We're going to tweak our particle system to change the flow of the animation so that our hearts emit out in a circle and away from the screen. How to do it... By following these steps, we'll gain a better understanding of how we can control the particle system: Select the Emitter in the Layers tab. Press F4 to open the Emitter tab of the Inspector. Change the Shape menu's option from rectangle to circle and the Arrangement option to Random Fill . Set the radius to 500. Increase Birth Rate to 200 and set Speed in Cell Controls to 0. If you play back the project, you'll see the hearts trying to form the shape of a circle. Since there is no speed value, the hearts stay put in the shape they originate from, as shown in this screenshot: Set Angle Value to 15 and Spin to 30. Rather than having the particles pop onscreen when they're born and pop offscreen when they die, we're going to add some tags to the Opacity Over Life parameter. The trick is to click the white line where we want to add a tag above it. Click the line once close to the beginning and twice close to the end. Use the following screenshot for reference: Click on the first tag and under the line, drag the Opacity slider to 0 (indicated in this screenshot). Repeat this step for the last slider. Now the hearts fade in at birth and fade out at death: As a final step, we're going to keyframe the speed of the particle system to start at 100 percent and eventually go down to 0 to reveal the shape of the circle we created. Go to the beginning of the Timeline and click the diamond icon next to Speed to add a keyframe. Change its value to 100. Move to three seconds and add another keyframe by clicking the diamond icon next to Speed again. Move to three seconds and set Speed to 0. Play back the animation and compare it to the following screenshot: There's more... You can create some amazing animations by changing the type of shape used to emit particles. For instance, if you change the shape of your particles to Image and use text as the source, the particles will spell out the word as long there is a substantial number of hearts being born and the Arrangement option is set to Random Fill . We'll see an example of this in a later exercise, but the following screenshot gives you a sneak peak on the end results you can get: Adding randomness values To give your particle systems just a little more, we're going to add a little randomness to some of the parameters. This randomness will help give your animations a more organic feel. Getting ready From the exercise files of this article, double-click the 07_03 project. Play back the project. The particle system consists of little demograms that rain down from a line. The Angle and Spin values cause the demograms to rotate while moving, and the Opacity Over Life parameter allows each particle to fade in and out during its lifespan. The particles also change color over their life as reflected in the Color Mode menu and the gradient outlining the cycle. We're going to add randomness values to tweak the animation. Whenever we add a randomness value, it's going to look at the original value above it and add and subtract to it randomly every second based on the number you specify. For example, say the Scale value is set to 50. If you add a Scale randomness of 10, every second the demograms will be somewhere between a scale of 40 and 60. How to do it... Let's start to randomize different values in this recipe: Select the emitter in the Layers tab and press F4 . In the Emitter tab of the Inspector, change the Birth Rate Randomness slider's value to 10. Now, every second, somewhere between 10 and 30 objects are born. Now, set Life Randomness to 3, Angle Randomness to 40, Spin Randomness to 50, and Scale Randomness to 80. Play back the project. Now we have our demograms growing at different sizes, being born at different angles and spinning at different rates! The following screenshot shows the Inspector with all of the mentioned changes and also a Color Over Life change (see the previous recipe to learn how to do it). The following is a screenshot showing a frame of the particle animation in the Canvas with random values applied: While playing back, click the Generate button next to Random Seed . You'll notice that your animation changes and a new number is created next to Generate . Random Seed takes all the random parameters and creates the animation based on this number. Change this number and change the randomness of the animation, as shown here: Working with particle behaviors While there are already a ton of parameters you can animate by keyframing their values in the particle and cell emitters, Motion offers you a few particle behaviors worth taking a look at from the library as well. Getting ready From the exercise files of this article, double-click the 07_04 project. Play back the project. The animation consists of a bouncing alarm clock from Motion's library in the foreground and a particle system referencing that alarm clock in the background. Unlike the other particle systems we've worked with in the previous recipes, the source of the particle system has been turned back on to preserve it. Our goal is to add two particle behaviors to have the clocks scale down and spin over its life cycle. By adjusting the speed and direction of our emitter, we will also have the clocks look like they're being pulled off into the distance. Also note, the anchor point of the clock was adjusted in advance in order to have the particles spin around the center of the clock. How to do it... Open the Particle group and select Emitter . Press Command + 2 to go to the Library tab. Under Behaviors , choose Particles | Spin Over Life . Click Apply . Press F7 to open the HUD. Set Increment Type to Birth and Death Values . Set Birth to 360 and Death to 0. If you're having trouble, try changing the values in the Behaviors tab of the Inspector. Now, play back the animation. Now, let's have the clocks in the particle system scale over life. Select the Emitter again. Press Command + 2 to go to the Library . Under Behaviors , choose Particles | Scale Over Life .Click Apply. In the HUD, set Increment Type to Birth and Death Values . Set Scale at Birth to 100 and Scale at Death to 0. If you're having trouble, try changing the values in the Behaviors tab of the Inspector. Play back the animation. The clocks now scale down gradually until they die, as shown in the following screenshot: Press F4 to go to the Emitter tab of the Inspector. Change the Emission Longitude value to 180 and Speed to 2000 , as shown in the following screenshot. Change the other parameters as desired. There's more... When you start building complex particle animations, keep in mind that you can save them to the Library as well as any of the behaviors you tweak. All particle presets can be found in the Library under Particle Emitters in a dedicated theme folder you create. The more time you spend automating your work, the more time you can dedicate to the creative process! Working with particle presets So far, we've created our own particle systems by selecting objects to place into cells referenced by the emitter. Motion ships with over 200 presets! One of the best ways to learn Motion is to dissect how some of the particle presets were created. Let's take a look at what it has to offer! Getting ready From the exercise files of this article, double-click on the 07_05 project. There are two particle presets in this project but one has been turned off. Play back the project and familiarize yourself with the Magic Dust preset. Stop the playback. Turn off the Magic Dust group and turn on the Buskerit group. Play back the project again. In this recipe, we'll take a look at each of these presets and change a few parameters to look at how the animations were created. How to do it... Let's now tweak these parameters of our presets in more detail: Reveal the content of the Buskerit group by clicking the disclosure triangle next to it. It contains a group called busker and seven still images that have been turned off. The images are being referenced by the emitter. Click the disclosure triangle next to the Busker group. It contains the content of the Busker emitter and the seven cells that hold information from those layers that have been shot off. The Emitter also has a Scale Over Life behavior in it. While the object continually changes scale over the animation, the Scale Over Life behavior was used to animate the "pop-up" intro of the instruments, whereas keyframes were used to change the scale over the project. Select the Scale Over Life behavior and press F2 to go to the Behaviors tab of the Inspector. Notice the Increment Type parameter set to Custom . Click the disclosure triangle next to Custom Scale to see the graph, as shown in the following screenshot: Select the Emitter in the Layers tab and press Command + 8 to open the Keyframe Editor . Notice that both the scale and rotation of the emitter are keyframed periodically throughout the entire animation, as shown in the following screenshot. In the Keyframe Editor , press Shift + K to move forward between keyframes. Change the Rotation and Scale values by double-clicking the values while on an existing keyframe. You'll know you're on one because the diamond shape will appear highlighted! Click the disclosure triangle to close the Buskerit group, turn it off, and turn on the Magic Dust group. Click the disclosure triangle to open it. A still image of a spark was used as the source for the Emitter. Several behaviors were used to animate the sparks. With the emitter selected, press F4 to go to the Emitter tab of the Inspector. Set Emission Range to 45 , Birth Rate to 200 , and Speed to 1000 . Play back the project to see the effect it has and compare it to the following: Select the Magic Dust group and go to the gear icon under the mini-Timeline. Choose Basic Motion | Motion Path . Tweak the path as desired. Play back the animation. There's more... Learning how particle presets work is a great way to get used to Motion. Particle presets are the best way to learn Motion! Looking at the library and seeing how something was created is the best way to get under the hood of the application and start creating your own animations. Don't be afraid to explore. Working with particle presets in 3D Particle presets are already powerful on their own but turn them into 3D and you'll find that a few of them will actually look like they have been extruded. Let's take a look! Getting ready From the exercise files of this article, double-click on the 07_06 project. Play back the project. There is a particle system of thin water being projected across the screen. Go to the Properties tab in the Inspector and click the rotation disclosure triangle to see the object rotate 90 degrees on its y axis. The water becomes invisible because it's flat the minute you change perspective. If we promote a particle system to 3D, we can get rid of this problem and have a few additional options available to us in terms of the way we emit the water. How to do it... Let's get rid of the flatness of our particle system by making it 3D: Select the Particle Emitter category and press F4 to go to the Inspector. Click on the 3D checkbox. As soon as you press the button, more options become available to you in the Emitter Controls section, including Emission Latitude and Emission Longitude , as seen here: Two additional selections—Box and Sphere —also became available under the Shape drop-down. Click on the word Rectangle and choose Box . Press F1 to go to the Properties tab of the Inspector. Click the disclosure triangle next to Scale and increase Scale Z to 200 . Rotate the water 90 degrees on the Y axis and notice it's no longer flat. Use the following screenshot of the Properties tab for reference: Go to the beginning of the Timeline and bring back Rotation Y to 0. Click the diamond shape next to Rotation Y to add a keyframe. Move to the end of the Timeline; and change the parameter to 720. Play back the project and see that, no matter what angle it is, the particle system never goes flat, as shown here:
Read more
  • 0
  • 0
  • 6722

article-image-integrating-storm-and-hadoop
Packt
04 Sep 2013
17 min read
Save for later

Integrating Storm and Hadoop

Packt
04 Sep 2013
17 min read
(For more resources related to this topic, see here.) In this article, we will implement the Batch and Service layers to complete the architecture. There are some key concepts underlying this big data architecture: Immutable state Abstraction and composition Constrain complexity Immutable state is the key, in that it provides true fault-tolerance for the architecture. If a failure is experienced at any level, we can always rebuild the data from the original immutable data. This is in contrast to many existing data systems, where the paradigm is to act on mutable data. This approach may seem simple and logical; however, it exposes the system to a particular kind of risk in which the state is lost or corrupted. It also constrains the system, in that you can only work with the current view of the data; it isn't possible to derive new views of the data. When the architecture is based on a fundamentally immutable state, it becomes both flexible and fault-tolerant. Abstractions allow us to remove complexity in some cases, and in others they can introduce complexity. It is important to achieve an appropriate set of abstractions that increase our productivity and remove complexity, but at an appropriate cost. It must be noted that all abstractions leak, meaning that when failures occur at a lower abstraction, they will affect the higher-level abstractions. It is therefore often important to be able to make changes within the various layers and understand more than one layer of abstraction. The designs we choose to implement our abstractions must therefore not prevent us from reasoning about or working at the lower levels of abstraction when required. Open source projects are often good at this, because of the obvious access to the code of the lower level abstractions, but even with source code available, it is easy to convolute the abstraction to the extent that it becomes a risk. In a big data solution, we have to work at higher levels of abstraction in order to be productive and deal with the massive complexity, so we need to choose our abstractions carefully. In the case of Storm, Trident represents an appropriate abstraction for dealing with the data-processing complexity, but the lower level Storm API on which Trident is based isn't hidden from us. We are therefore able to easily reason about Trident based on an understanding of lower-level abstractions within Storm. Another key issue to consider when dealing with complexity and productivity is composition. Composition within a given layer of abstraction allows us to quickly build out a solution that is well tested and easy to reason about. Composition is fundamentally decoupled, while abstraction contains some inherent coupling to the lower-level abstractions—something that we need to be aware of. Finally, a big data solution needs to constrain complexity. Complexity always equates to risk and cost in the long run, both from a development perspective and from an operational perspective. Real-time solutions will always be more complex than batch-based systems; they also lack some of the qualities we require in terms of performance. Nathan Marz's Lambda architecture attempts to address this by combining the qualities of each type of system to constrain complexity and deliver a truly fault-tolerant architecture. We divided this flow into preprocessing and "at time" phases, using streams and DRPC streams respectively. We also introduced time windows that allowed us to segment the preprocessed data. In this article, we complete the entire architecture by implementing the Batch and Service layers. The Service layer is simply a store of a view of the data. In this case, we will store this view in Cassandra, as it is a convenient place to access the state alongside Trident's state. The preprocessed view is identical to the preprocessed view created by Trident, counted elements of the TF-IDF formula (D, DF, and TF), but in the batch case, the dataset is much larger, as it includes the entire history. The Batch layer is implemented in Hadoop using MapReduce to calculate the preprocessed view of the data. MapReduce is extremely powerful, but like the lower-level Storm API, is potentially too low-level for the problem at hand for the following reasons: We need to describe the problem as a data pipeline; MapReduce isn't congruent with such a way of thinking Productivity We would like to think of a data pipeline in terms of streams of data, tuples within the stream and predicates acting on those tuples. This allows us to easily describe a solution to a data processing problem, but it also promotes composability, in that predicates are fundamentally composable, but pipelines themselves can also be composed to form larger, more complex pipelines. Cascading provides such an abstraction for MapReduce in the same way as Trident does for Storm. With these tools, approaches, and considerations in place, we can now complete our real-time big data architecture. There are a number of elements, that we will update, and a number of elements that we will add. The following figure illustrates the final architecture, where the elements in light grey will be updated from the existing recipe, and the elements in dark grey will be added in this article: Implementing TF-IDF in Hadoop TF-IDF is a well-known problem in the MapReduce communities; it is well-documented and implemented, and it is interesting in that it is sufficiently complex to be useful and instructive at the same time. Cascading has a series of tutorials on TF-IDF at http://www.cascading.org/2012/07/31/cascading-for-the-impatient-part-5/, which documents this implementation well. For this recipe, we shall use a Clojure Domain Specific Language (DSL) called Cascalog that is implemented on top of Cascading. Cascalog has been chosen because it provides a set of abstractions that are very semantically similar to the Trident API and are very terse while still remaining very readable and easy to understand. Getting ready Before you begin, please ensure that you have installed Hadoop by following the instructions at http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/. How to do it… Start by creating the project using the lein command: lein new tfidf-cascalog Next, you need to edit the project.clj file to include the dependencies: (defproject tfidf-cascalog "0.1.0-SNAPSHOT" :dependencies [[org.clojure/clojure "1.4.0"] [cascalog "1.10.1"] [org.apache.cassandra/cassandra-all "1.1.5"] [clojurewerkz/cassaforte "1.0.0-beta11-SNAPSHOT"] [quintona/cascading-cassandra "0.0.7-SNAPSHOT"] [clj-time "0.5.0"] [cascading.avro/avro-scheme "2.2-SNAPSHOT"] [cascalog-more-taps "0.3.0"] [org.apache.httpcomponents/httpclient "4.2.3"]] :profiles{:dev{:dependencies[[org.apache.hadoop/hadoop-core "0.20.2-dev"] [lein-midje "3.0.1"] [cascalog/midje-cascalog "1.10.1"]]}}) It is always a good idea to validate your dependencies; to do this, execute lein deps and review any errors. In this particular case, cascading-cassandra has not been deployed to clojars, and so you will receive an error message. Simply download the source from https://github.com/quintona/cascading-cassandra and install it into your local repository using Maven. It is also good practice to understand your dependency tree. This is important to not only prevent duplicate classpath issues, but also to understand what licenses you are subject to. To do this, simply run lein pom, followed by mvn dependency:tree. You can then review the tree for conflicts. In this particular case, you will notice that there are two conflicting versions of Avro. You can fix this by adding the appropriate exclusions: [org.apache.cassandra/cassandra-all "1.1.5" :exclusions [org.apache.cassandra.deps/avro]] We then need to create the Clojure-based Cascade queries that will process the document data. We first need to create the query that will create the "D" view of the data; that is, the D portion of the TF-IDF function. This is achieved by defining a Cascalog function that will output a key and a value, which is composed of a set of predicates: (defn D [src] (let [src (select-fields src ["?doc-id"])] (<- [?key ?d-str] (src ?doc-id) (c/distinct-count ?doc-id :> ?n-docs) (str "twitter" :> ?key) (str ?n-docs :> ?d-str)))) You can define this and any of the following functions in the REPL, or add them to core.clj in your project. If you want to use the REPL, simply use lein repl from within the project folder. The required namespace (the use statement), require, and import definitions can be found in the source code bundle. We then need to add similar functions to calculate the TF and DF values: (defn DF [src] (<- [?key ?df-count-str] (src ?doc-id ?time ?df-word) (c/distinct-count ?doc-id ?df-word :> ?df-count) (str ?df-word :> ?key) (str ?df-count :> ?df-count-str))) (defn TF [src] (<- [?key ?tf-count-str] (src ?doc-id ?time ?tf-word) (c/count ?tf-count) (str ?doc-id ?tf-word :> ?key) (str ?tf-count :> ?tf-count-str))) This Batch layer is only interested in calculating views for all the data leading up to, but not including, the current hour. This is because the data for the current hour will be provided by Trident when it merges this batch view with the view it has calculated. In order to achieve this, we need to filter out all the records that are within the current hour. The following function makes that possible: (deffilterop timing-correct? [doc-time] (let [now (local-now) interval (in-minutes (interval (from-long doc-time) now))] (if (< interval 60) false true)) Each of the preceding query definitions require a clean stream of words. The text contained in the source documents isn't clean. It still contains stop words. In order to filter these and emit a clean set of words for these queries, we can compose a function that splits the text into words and filters them based on a list of stop words and the time function defined previously: (defn etl-docs-gen [rain stop] (<- [?doc-id ?time ?word] (rain ?doc-id ?time ?line) (split ?line :> ?word-dirty) ((c/comp s/trim s/lower-case) ?word-dirty :> ?word) (stop ?word :> false) (timing-correct? ?time))) We will be storing the outputs from our queries to Cassandra, which requires us to define a set of taps for these views: (defn create-tap [rowkey cassandra-ip] (let [keyspace storm_keyspace column-family "tfidfbatch" scheme (CassandraScheme. cassandra-ip "9160" keyspace column-family rowkey {"cassandra.inputPartitioner""org.apache.cassandra.dht.RandomPartitioner" "cassandra.outputPartitioner" "org.apache.cassandra.dht.RandomPartitioner"}) tap (CassandraTap. scheme)] tap)) (defn create-d-tap [cassandra-ip] (create-tap "d"cassandra-ip)) (defn create-df-tap [cassandra-ip] (create-tap "df" cassandra-ip)) (defn create-tf-tap [cassandra-ip] (create-tap "tf" cassandra-ip)) The way this schema is created means that it will use a static row key and persist name-value pairs from the tuples as column:value within that row. This is congruent with the approach used by the Trident Cassandra adaptor. This is a convenient approach, as it will make our lives easier later. We can complete the implementation by a providing a function that ties everything together and executes the queries: (defn execute [in stop cassandra-ip] (cc/connect! cassandra-ip) (sch/set-keyspace storm_keyspace) (let [input (tap/hfs-tap (AvroScheme. (load-schema)) in) stop (hfs-delimited stop :skip-header? true) src (etl-docs-gen input stop)] (?- (create-d-tap cassandra-ip) (D src)) (?- (create-df-tap cassandra-ip) (DF src)) (?- (create-tf-tap cassandra-ip) (TF src)))) Next, we need to get some data to test with. I have created some test data, which is available at https://bitbucket.org/qanderson/tfidf-cascalog. Simply download the project and copy the contents of src/data to the data folder in your project structure. We can now test this entire implementation. To do this, we need to insert the data into Hadoop: hadoop fs -copyFromLocal ./data/document.avro data/document.avro hadoop fs -copyFromLocal ./data/en.stop data/en.stop Then launch the execution from the REPL: => (execute "data/document" "data/en.stop" "127.0.0.1") How it works… There are many excellent guides on the Cascalog wiki (https://github.com/nathanmarz/cascalog/wiki), but for completeness's sake, the nature of a Cascalog query will be explained here. Before that, however, a revision of Cascading pipelines is required. The following is quoted from the Cascading documentation (http://docs.cascading.org/cascading/2.1/userguide/htmlsingle/): Pipe assemblies define what work should be done against tuple streams, which are read from tap sources and written to tap sinks. The work performed on the data stream may include actions such as filtering, transforming, organizing, and calculating. Pipe assemblies may use multiple sources and multiple sinks, and may define splits, merges, and joins to manipulate the tuple streams. This concept is embodied in Cascalog through the definition of queries. A query takes a set of inputs and applies a list of predicates across the fields in each tuple of the input stream. Queries are composed through the application of many predicates. Queries can also be composed to form larger, more complex queries. In either event, these queries are reduced down into a Cascading pipeline. Cascalog therefore provides an extremely terse and powerful abstraction on top of Cascading; moreover, it enables an excellent development workflow through the REPL. Queries can be easily composed and executed against smaller representative datasets within the REPL, providing the idiomatic API and development workflow that makes Clojure beautiful. If we unpack the query we defined for TF, we will find the following code: (defn DF [src] (<- [?key ?df-count-str] (src ?doc-id ?time ?df-word) (c/distinct-count ?doc-id ?df-word :> ?df-count) (str ?df-word :> ?key) (str ?df-count :> ?df-count-str))) The <- macro defines a query, but does not execute it. The initial vector, [?key ?df-count-str], defines the output fields, which is followed by a list of predicate functions. Each predicate can be one of the following three types: Generators: A source of data where the underlying source is either a tap or another query. Operations: Implicit relations that take in input variables defined elsewhere and either act as a function that binds new variables or a filter. Operations typically act within the scope of a single tuple. Aggregators: Functions that act across tuples to create aggregate representations of data. For example, count and sum. The :> keyword is used to separate input variables from output variables. If no :> keyword is specified, the variables are considered as input variables for operations and output variables for generators and aggregators. The (src ?doc-id ?time ?df-word) predicate function names the first three values within the input tuple, whose names are applicable within the query scope. Therefore, if the tuple ("doc1" 123324 "This") arrives in this query, the variables would effectively bind as follows: ?doc-id: "doc1" ?time: 123324 ?df-word: "This" Each predicate within the scope of the query can use any bound value or add new bound variables to the scope of the query. The final set of bound values that are emitted is defined by the output vector. We defined three queries, each calculating a portion of the value required for the TF-IDF algorithm. These are fed from two single taps, which are files stored in the Hadoop filesystem. The document file is stored using Apache Avro, which provides a high-performance and dynamic serialization layer. Avro takes a record definition and enables serialization/deserialization based on it. The record structure, in this case, is for a document and is defined as follows: {"namespace": "storm.cookbook", "type": "record", "name": "Document", "fields": [ {"name": "docid", "type": "string"}, {"name": "time", "type": "long"}, {"name": "line", "type": "string"} ] } Both the stop words and documents are fed through an ETL function that emits a clean set of words that have been filtered. The words are derived by splitting the line field using a regular expression: (defmapcatop split [line] (s/split line #"[[](),.)s]+")) The ETL function is also a query, which serves as a source for our downstream queries, and defines the [?doc-id ?time ?word] output fields. The output tap, or sink, is based on the Cassandra scheme. A query defines predicate logic, not the source and destination of data. The sink ensures that the outputs of our queries are sent to Cassandra. The ?- macro executes a query, and it is only at execution time that a query is bound to its source and destination, again allowing for extreme levels of composition. The following, therefore, executes the TF query and outputs to Cassandra: (?- (create-tf-tap cassandra-ip) (TF src)) There's more… The Avro test data was created using the test data from the Cascading tutorial at http://www.cascading.org/2012/07/31/cascading-for-the-impatient-part-5/. Within this tutorial is the rain.txt tab-separated data file. A new column was created called time that holds the Unix epoc time in milliseconds. The updated text file was then processed using some basic Java code that leverages Avro: Schema schema = Schema.parse(SandboxMain.class.getResourceAsStream("/document.avsc")); File file = new File("document.avro"); DatumWriter<GenericRecord> datumWriter = new GenericDatumWriter<GenericRecord>(schema); DataFileWriter<GenericRecord> dataFileWriter = new DataFileWriter<GenericRecord>(datumWriter); dataFileWriter.create(schema, file); BufferedReader reader = new BufferedReader(new InputStreamReader(SandboxMain.class.getResourceAsStream("/rain.txt"))); String line = null; try { while ((line = reader.readLine()) != null) { String[] tokens = line.split("t"); GenericRecord docEntry = new GenericData.Record(schema); docEntry.put("docid", tokens[0]); docEntry.put("time", Long.parseLong(tokens[1])); docEntry.put("line", tokens[2]); dataFileWriter.append(docEntry); } } catch (IOException e) { e.printStackTrace(); } dataFileWriter.close(); Persisting documents from Storm In the previous recipe, we looked at deriving precomputed views of our data taking some immutable data as the source. In that recipe, we used statically created data. In an operational system, we need Storm to store the immutable data into Hadoop so that it can be used in any preprocessing that is required. How to do it… As each tuple is processed in Storm, we must generate an Avro record based on the document record definition and append it to the data file within the Hadoop filesystem. We must create a Trident function that takes each document tuple and stores the associated Avro record. Within the tfidf-topology project created in, inside the storm.cookbook.tfidf.function package, create a new class named PersistDocumentFunction that extends BaseFunction. Within the prepare function, initialize the Avro schema and document writer: public void prepare(Map conf, TridentOperationContext context) { try { String path = (String) conf.get("DOCUMENT_PATH"); schema = Schema.parse(PersistDocumentFunction.class .getResourceAsStream("/document.avsc")); File file = new File(path); DatumWriter<GenericRecord> datumWriter = new GenericDatumWriter<GenericRecord>(schema); dataFileWriter = new DataFileWriter<GenericRecord>(datumWriter); if(file.exists()) dataFileWriter.appendTo(file); else dataFileWriter.create(schema, file); } catch (IOException e) { throw new RuntimeException(e); } } As each tuple is received, coerce it into an Avro record and add it to the file: public void execute(TridentTuple tuple, TridentCollector collector) { GenericRecord docEntry = new GenericData.Record(schema); docEntry.put("docid", tuple.getStringByField("documentId")); docEntry.put("time", Time.currentTimeMillis()); docEntry.put("line", tuple.getStringByField("document")); try { dataFileWriter.append(docEntry); dataFileWriter.flush(); } catch (IOException e) { LOG.error("Error writing to document record: " + e); throw new RuntimeException(e); } } Next, edit the TermTopology.build topology and add the function to the document stream: documentStream.each(new Fields("documentId","document"), new PersistDocumentFunction(), new Fields()); Finally, include the document path into the topology configuration: conf.put("DOCUMENT_PATH", "document.avro"); How it works… There are various logical streams within the topology, and certainly the input for the topology is not in the appropriate state for the recipes in this article containing only URLs. We therefore need to select the correct stream from which to consume tuples, coerce these into Avro records, and serialize them into a file. The previous recipe will then periodically consume this file. Within the context of the topology definition, include the following code: Stream documentStream = getUrlStream(topology, spout) .each(new Fields("url"), new DocumentFetchFunction(mimeTypes), new Fields("document", "documentId", "source")); documentStream.each(new Fields("documentId","document"), new PersistDocumentFunction(), new Fields()); The function should consume tuples from the document stream whose tuples are populated with already fetched documents.
Read more
  • 0
  • 0
  • 2862

article-image-installation
Packt
03 Sep 2013
7 min read
Save for later

Installation

Packt
03 Sep 2013
7 min read
(For more resources related to this topic, see here.) Step 1 – preparing for the deployment OpenNMS is a large and complex piece of software and its deployment can be intimidating. Making sure we are prepared will save us a lot of trouble along the road. Here are some of the things needed before getting started: You should have a good internet connection. There are a lot of software packages to download, install and configure. You should have a clean (newly installed) and up-to-date system. OpenNMS can be deployed on any platform where the JVM runs such as Linux, Unix, Solaris, Mac, and Windows. You should read the official documentation or at least skim through the quick start guide at http://www.opennms.org/wiki/QuickStart, the tutorial at http://www.opennms.org/wiki/Tutorial, and the install guide at http://www.opennms.org/documentation/installguide.html. Step 2 – setting up OpenNMS software repositories OpenNMS conveniently makes available its software to several operating systems through their native software repository applications (for example, APT, YUM, and fink). The procedure for this will depend on your platform; instructions can be found at http://www.opennms.org/documentation/installguide.html#before-you-begin. Users of Windows can safely ignore this step as there is a standalone package providing a more Windows-like installation procedure. Step 3 – installing the Java Development Kit (JDK) Normally, Java software only needs the Java Runtime Environment (JRE) to be installed to run. But this will not be sufficient, you must install the JDK, either the OpenJDK implementation or Oracle's. The installation is straight forward and detailed at http://www.opennms.org/documentation/installguide.html#java. The default options are fine to get started. If installing on Windows, both the 3 2-bit and 64-bit JDK are available on Oracle's website. Bear in mind that the JDK must match the OpenNMS standalone setup executable that you will run. Step 4 – installing and configure PostgreSQL PostgreSQL is usually available in Linux with excellent support from the operating systems' repositories. Otherwise, there are available binaries for other common operating systems. Here, instructions will diverge a little from OpenNMS's available documentation at http://www.opennms.org/documentation/installguide.html#postgresql, which are meant to get you started with as little fuss as possible at the expense of security. Access controls will be configured using encrypted passwords (not using trust authentication as in the online tutorials). This can be achieved by editing the pg_hba.conf file; its location will vary depending on your platform. On CentOS 6 it is located in /var/lib/pgsql/data/ and on Debian 6 in /etc/postgresql/8.4/main/ (refer to your OS documentation). Locate the following configuration lines and edit the authentication method to use md5 encrypted passwords as shown in the following code. You will need to reload or restart the service for changes to take effect: # "local" is for Unix domain socket connections onlylocal all postgres md5local all all md5# IPv4 local connections:host all all 127.0.0.1/32 md5# IPv6 local connections:host all all ::1/128 md5 The PostgreSQL defaults in postgresql.conf should be fine, but to take advantage of OpenNMS' ability to scale you will need to tune PostgreSQL further. A good starting point is going through http://www.opennms.org/documentation/installguide.html#postgresql-configure and http://www.opennms.org/documentation/installguide.html#performance-tuning. To complete the database setup we will execute some SQL commands from the command line to do a number of initial tasks. The following code shows how to set a strong password for the PostgreSQL user postgres , create a database user called opennms with restricted privileges and a password of your choice, and create a database called opennms owned by our new opennms user (creating the database manually is not necessarily and would also be taken care of by the installer): # su - postgres$ psql -c "ALTER USER postgres WITH PASSWORD 'newpassword'" -d template1$ psql -c "CREATE USER opennms WITH LOGIN ENCRYPTED PASSWORD 'opennmspassword';"$ psql -c "CREATE DATABASE opennms WITH OWNER=opennms ENCODING 'UNICODE';" With a new database at our disposal we can now start importing functions and data into it. The next step is to install the iplike package that contains an optimized function to do lookups based on IP addresses. It is usually available in your OS software repository. If you are on Windows, you do not have to do this explicitly. It is taken care automatically by the standalone OpenNMS package. Once installed, you may have to run a script as shown in the following code to install the function in the opennms database: # install_iplike.sh If you are not using opennms (the default) as the database name you will have to edit the iplike script to change the database name manually. If you are using PostgreSQL 9.0 or later the procedural language is already installed by default, otherwise it needs to be installed in the opennms database with the following command: # createlang -U postgres plpgsql opennms If you are on Windows, you will be installing the OpenNMS standalone package. You do not need to worry about installing iplike or the plpgsql procedural language as both of them will be installed automatically. Step 5 – installing OpenNMS Before we go ahead with the OpenNMS installation, it is a good time to install the remaining optional dependencies jicmp and jrrd. On Windows those dependencies come with the standalone package and you do not need to do anything. On other OS it should be pulled in as a dependency when installing using APT, YUM, or fink. We are now ready to install OpenNMS as detailed at http://www.opennms.org/documentation/installguide.html#installing; instructions are included for various platforms. Once installation of OpenNMS software is complete, either through the software repositories or using the standalone package for Windows, you must take care to properly configure it for database access. Locate the file $OPENNMS_HOME/etc/opennms-datasources.xml and edit the data sources as shown in the following code: <jdbc-data-source name="opennms" database-name="opennms" class-name="org.postgresql.Driver" url="jdbc:postgresql://localhost:5432/opennms" user-name="opennms" password="opennmspassword" /><jdbc-data-source name="opennms-admin" database-name="template1" class-name="org.postgresql.Driver" url="jdbc:postgresql://localhost:5432/template1" user-name="postgres" password="newpassword" /> If everything was done correctly up to this point the OpenNMS installation can now be completed. First, optionally tell OpenNMS to find a suitable JRE (from your JDK) and then finalize the OpenNMS deployment using the install tool that comes with it. Essentially, the following two commands should be executed: # $OPENNMS_HOME/bin/runjava -s# $OPENNMS_HOME/bin/install -dis If the installation was successful you should be able to start the OpenNMS service. The command for doing this will depend on your platform and method of installation: On debian it is as simple as the following command: # service opennms start Red Hat systems like Fedora moved to the systemctl service manager, it would be something more like the following command: # systemctl start opennms.service On Windows you can do the following: # cd C:Program FilesOpenNMSbin# opennms.bat start And that's it Once started we can log in to OpenNMS with the credentials admin/admin using a browser pointed at http : //localhost:8980/opennms. If this is your first time installing a large Java system, there is a good chance it will not work the first time around. Don't give up, the scary errors are simply likely the result of one or two minor mistakes along the way. Repeat and verify each step all over again. Summary Installation helps you learn how to download and install OpenNMS with the minimum fuss and then set it up so that you can use it as soon as possible. Resources for Article: Further resources on this subject: Geronimo Architecture: Part 2 [Article] Moodle 2.0 Multimedia: Working with 2D and 3D Maps [Article] Network Monitoring Essentials [Article]
Read more
  • 0
  • 0
  • 1466
article-image-scratching-tip-iceberg
Packt
03 Sep 2013
15 min read
Save for later

Scratching the Tip of the Iceberg

Packt
03 Sep 2013
15 min read
Boost is a huge collection of libraries. Some of those libraries are small and meant for everyday use and others require a separate article to describe all of their features. This article is devoted to some of those big libraries and to give you some basics to start with. The first two recipes will explain the usage of Boost.Graph. It is a big library with an insane number of algorithms. We'll see some basics and probably the most important part of it visualization of graphs. We'll also see a very useful recipe for generating true random numbers. This is a very important requirement for writing secure cryptography systems. Some C++ standard libraries lack math functions. We'll see how that can be fixed using Boost. But the format of this article leaves no space to describe all of the functions. Writing test cases is described in the Writing test cases and Combining multiple test cases in one test module recipes. This is important for any production-quality system. The last recipe is about a library that helped me in many courses during my university days. Images can be created and modified using it. I personally used it to visualize different algorithms, hide data in images, sign images, and generate textures. Unfortunately, even this article cannot tell you about all of the Boost libraries. Maybe someday I'll write another book... and then a few more. Working with graphs Some tasks require a graphical representation of data. Boost.Graph is a library that was designed to provide a flexible way of constructing and representing graphs in memory. It also contains a lot of algorithms to work with graphs, such as topological sort, breadth first search, depth first search, and Dijkstra shortest paths. Well, let's perform some basic tasks with Boost.Graph! Getting ready Only basic knowledge of C++ and templates is required for this recipe. How to do it... In this recipe, we'll describe a graph type, create a graph of that type, add some vertexes and edges to the graph, and search for a specific vertex. That should be enough to start using Boost.Graph. We start with describing the graph type: #include <boost/graph/adjacency_list.hpp> #include <string> typedef std::string vertex_t; typedef boost::adjacency_list< boost::vecS , boost::vecS , boost::bidirectionalS , vertex_t > graph_type; Now we construct it: graph_type graph; Let's use a non portable trick that speeds up graph construction: static const std::size_t vertex_count = 5; graph.m_vertices.reserve(vertex_count); Now we are ready to add vertexes to the graph: typedef boost::graph_traits<graph_type> ::vertex_descriptor descriptor_t; descriptor_t cpp = boost::add_vertex(vertex_t("C++"), graph); descriptor_t stl = boost::add_vertex(vertex_t("STL"), graph); descriptor_t boost = boost::add_vertex(vertex_t("Boost"), graph); descriptor_t guru = boost::add_vertex(vertex_t("C++ guru"), graph); descriptor_t ansic = boost::add_vertex(vertex_t("C"), graph); It is time to connect vertexes with edges: boost::add_edge(cpp, stl, graph); boost::add_edge(stl, boost, graph); boost::add_edge(boost, guru, graph); boost::add_edge(ansic, guru, graph); We make a function that searches for a vertex: template <class GraphT> void find_and_print(const GraphT& g, boost::string_ref name) { Now we will write code that gets iterators to all vertexes: typedef typename boost::graph_traits<graph_type> ::vertex_iterator vert_it_t; vert_it_t it, end; boost::tie(it, end) = boost::vertices(g); It's time to run a search for the required vertex: typedef boost::graph_traits<graph_type>::vertex_descriptor desc_t; for (; it != end; ++ it) { desc_t desc = *it; if (boost::get(boost::vertex_bundle, g)[desc] == name.data()) { break; } } assert(it != end); std::cout << name << 'n'; } /* find_and_print */ How it works... In step 1, we are describing what our graph must look like and upon what types it must be based. boost::adjacency_list is a class that represents graphs as a two-dimensional structure, where the first dimension contains vertexes and the second dimension contains edges for that vertex. boost::adjacency_list must be the default choice for representing a graph; it suits most cases. The first template parameter, boost::adjacency_list, describes the structure used to represent the edge list for each of the vertexes; the second one describes a structure to store vertexes. We can choose different STL containers for those structures using specific selectors, as listed in the following table: Selector STL container boost::vecS std::vector boost::listS std::list boost::slistS std::slist boost::setS std::set boost::multisetS std::multiset boost::hash_setS std::hash_set The third template parameter is used to make an undirected, directed, or bidirectional graph. Use the boost::undirectedS, boost::directedS, and boost::bidirectionalS selectors respectively. The fifth template parameter describes the datatype that will be used as the vertex. In our example, we chose std::string. We can also support a datatype for edges and provide it as a template parameter. Steps 2 and 3 are trivial, but at step 4 you will see a non portable way to speed up graph construction. In our example, we use std::vector as a container for storing vertexes, so we can force it to reserve memory for the required amount of vertexes. This leads to less memory allocations/deallocations and copy operations during insertion of vertexes into the graph. This step is non-portable because it is highly dependent on the current implementation of boost::adjacency_list and on the chosen container type for storing vertexes. At step 4, we see how vertexes can be added to the graph. Note how boost::graph_traits<graph_type> has been used. The boost::graph_traits class is used to get types that are specific for a graph type. We'll see its usage and the description of some graph-specific types later in this article. Step 5 shows what we need do to connect vertexes with edges. If we had provided a datatype for the edges, adding an edge would look as follows: boost::add_edge(ansic, guru, edge_t(initialization_parameters), graph) Note that at step 6 the graph type is a template parameter. This is recommended to achieve better code reusability and make this function work with other graph types. At step 7, we see how to iterate over all of the vertexes of the graph. The type of vertex iterator is received from boost::graph_traits. The function boost::tie is a part of Boost.Tuple and is used for getting values from tuples to the variables. So calling boost::tie(it, end) = boost::vertices(g) will put the begin iterator into the it variable and the end iterator into the end variable. It may come as a surprise to you, but dereferencing a vertex iterator does not return vertex data. Instead, it returns the vertex descriptor desc, which can be used in boost::get(boost::vertex_bundle, g)[desc] to get vertex data, just as we have done in step 8. The vertex descriptor type is used in many of the Boost.Graph functions; we saw its use in the edge construction function in step 5. As already mentioned, the Boost.Graph library contains the implementation of many algorithms. You will find many search policies implemented, but we won't discuss them in this article. We will limit this recipe to only the basics of the graph library. There's more... The Boost.Graph library is not a part of C++11 and it won't be a part of C++1y. The current implementation does not support C++11 features. If we are using vertexes that are heavy to copy, we may gain speed using the following trick: vertex_descriptor desc = boost::add_vertex(graph); boost::get(boost::vertex_bundle, g_)[desc] = std::move(vertex_data); It avoids copy constructions of boost::add_vertex(vertex_data, graph) and uses the default construction with move assignment instead. The efficiency of Boost.Graph depends on multiple factors, such as the underlying containers types, graph representation, edge, and vertex datatypes. Visualizing graphs Making programs that manipulate graphs was never easy because of issues with visualization. When we work with STL containers such as std::map and std::vector, we can always print the container's contents and see what is going on inside. But when we work with complex graphs, it is hard to visualize the content in a clear way: too many vertexes and too many edges. In this recipe, we'll take a look at the visualization of Boost.Graph using the Graphviz tool. Getting ready To visualize graphs, you will need a Graphviz visualization tool. Knowledge of the preceding recipe is also required. How to do it... Visualization is done in two phases. In the first phase, we make our program output the graph's description in a text format; in the second phase, we import the output from the first step to some visualization tool. The numbered steps in this recipe are all about the first phase. Let's write the std::ostream operator for graph_type as done in the preceding recipe: #include <boost/graph/graphviz.hpp> std::ostream& operator<<(std::ostream& out, const graph_type& g) { detail::vertex_writer<graph_type> vw(g); boost::write_graphviz(out, g, vw); return out; } The detail::vertex_writer structure, used in the preceding step, must be defined as follows: namespace detail { template <class GraphT> class vertex_writer { const GraphT& g_; public: explicit vertex_writer(const GraphT& g) : g_(g) {} template <class VertexDescriptorT> void operator()(std::ostream& out, const VertexDescriptorT& d) const { out << " [label="" << boost::get(boost::vertex_bundle, g_)[d] << ""]"; } }; // vertex_writer } // namespace detail That's all. Now, if we visualize the graph from the previous recipe using the std::cout << graph; command, the output can be used to create graphical pictures using the dot command-line utility: $ dot -Tpng -o dot.png digraph G { 0 [label="C++"]; 1 [label="STL"]; 2 [label="Boost"]; 3 [label="C++ guru"]; 4 [label="C"]; 0->1 ; 1->2 ; 2->3 ; 4->3 ; }   The output of the preceding command is depicted in the following figure: We can also use the Gvedit or XDot programs for visualization if the command line frightens you. How it works... The Boost.Graph library contains function to output graphs in Graphviz (DOT) format. If we write boost::write_graphviz(out, g) with two parameters in step 1, the function will output a graph picture with vertexes numbered from 0. That's not very useful, so we provide an instance of the vertex_writer class that outputs vertex names. As we can see in step 2, the format of output must be DOT, which is understood by the Graphviz tool. You may need to read the Graphviz documentation for more info about the DOT format. If you wish to add some data to the edges during visualization, we need to provide an instance of the edge visualizer as a fourth parameter to boost::write_graphviz. There's more... C++11 does not contain Boost.Graph or the tools for graph visualization. But you do not need to worry—there are a lot of other graph formats and visualization tools and Boost. Graph can work with plenty of them. Using a true random number generator I know of many examples of commercial products that use incorrect methods for getting random numbers. It's a shame that some companies still use rand() in cryptography and banking software. Let's see how to get a fully random uniform distribution using Boost.Random that is suitable for banking software. Getting ready Basic knowledge of C++ is required for this recipe. Knowledge of different types of distributions will also be helpful. The code in this recipe requires linking against the boost_random library. How to do it... To create a true random number, we need some help from the operating system or processor. This is how it can be done using Boost: We'll need to include the following headers: #include <boost/config.hpp> #include <boost/random/random_device.hpp> #include <boost/random/uniform_int_distribution.hpp> Advanced random number providers have different names under different platforms: static const std::string provider = #ifdef BOOST_WINDOWS "Microsoft Strong Cryptographic Provider" #else "/dev/urandom" #endif ; Now we are ready to initialize the generator with Boost.Random: boost::random_device device(provider); Let's get a uniform distribution that returns a value between 1000 and 65535: boost::random::uniform_int_distribution<unsigned short> random(1000); That's it. Now we can get true random numbers using the random(device) call. How it works... Why does the rand() function not suit banking? Because it generates pseudo-random numbers, which means that the hacker could predict the next generated number. This is an issue with all pseudo-random number algorithms. Some algorithms are easier to predict and some harder, but it's still possible. That's why we are using boost::random_device in this example (see step 3). That device gathers information about random events from all around the operating system to construct an unpredictable hardware-generated number. The examples of such events are delays between pressed keys, delays between some of the hardware interruptions, and the internal CPU random number generator. Operating systems may have more than one such type of random number generators. In our example for POSIX systems, we used /dev/urandom instead of the more secure /dev/random because the latter remains in a blocked state until enough random events have been captured by the OS. Waiting for entropy could take seconds, which is usually unsuitable for applications. Use /dev/random to create long-lifetime GPG/SSL/SSH keys. Now that we are done with generators, it's time to move to step 4 and talk about distribution classes. If the generator just generates numbers (usually uniformly distributed), the distribution class maps one distribution to another. In step 4, we made a uniform distribution that returns a random number of unsigned short type. The parameter 1000 means that distribution must return numbers greater or equal to 1000. We can also provide the maximum number as a second parameter, which is by default equal to the maximum value storable in the return type. There's more... Boost.Random has a huge number of true/pseudo random generators and distributions for different needs. Avoid copying distributions and generators; this could turn out to be an expensive operation. C++11 has support for different distribution classes and generators. You will find all of the classes from this example in the <random> header in the std:: namespace. The Boost.Random libraries do not use C++11 features, and they are not really required for that library either. Should you use Boost implementation or STL? Boost provides better portability across systems; however, some STL implementations may have assembly-optimized implementations and might provide some useful extensions. Using portable math functions Some projects require specific trigonometric functions, a library for numerically solving ordinary differential equations, and working with distributions and constants. All of those parts of Boost.Math would be hard to fit into even a separate book. A single recipe definitely won't be enough. So let's focus on very basic everyday-use functions to work with float types. We'll write a portable function that checks an input value for infinity and not-a-number (NaN) values and changes the sign if the value is negative. Getting ready Basic knowledge of C++ is required for this recipe. Those who know C99 standard will find a lot in common in this recipe. How to do it... Perform the following steps to check the input value for infinity and NaN values and change the sign if the value is negative: We'll need the following headers: #include <boost/math/special_functions.hpp> #include <cassert> Asserting for infinity and NaN can be done like this: template <class T> void check_float_inputs(T value) { assert(!boost::math::isinf(value)); assert(!boost::math::isnan(value)); Use the following code to change the sign: if (boost::math::signbit(value)) { value = boost::math::changesign(value); } // ... } // check_float_inputs That's it! Now we can check that check_float_inputs(std::sqrt(-1.0)) and check_float_inputs(std::numeric_limits<double>::max() * 2.0) will cause asserts. How it works... Real types have specific values that cannot be checked using equality operators. For example, if the variable v contains NaN, assert(v!=v) may or may not pass depending on the compiler. For such cases, Boost.Math provides functions that can reliably check for infinity and NaN values. Step 3 contains the boost::math::signbit function, which requires clarification. This function returns a signed bit, which is 1 when the number is negative and 0 when the number is positive. In other words, it returns true if the value is negative. Looking at step 3 some readers might ask, "Why can't we just multiply by -1 instead of calling boost::math::changesign?". We can. But multiplication may work slower than boost::math::changesign and won't work for special values. For example, if your code can work with nan, the code in step 3 will be able to change the sign of -nan and write nan to the variable. The Boost.Math library maintainers recommend wrapping math functions from this example in round parenthesis to avoid collisions with C macros. It is better to write (boost::math::isinf)(value) instead of boost::math::isinf(value). There's more... C99 contains all of the functions described in this recipe. Why do we need them in Boost? Well, some compiler vendors think that programmers do not need them, so you won't find them in one very popular compiler. Another reason is that the Boost.Math functions can be used for classes that behave like numbers. Boost.Math is a very fast, portable, reliable library.
Read more
  • 0
  • 0
  • 1823

article-image-resource-manager
Packt
03 Sep 2013
11 min read
Save for later

Resource Manager

Packt
03 Sep 2013
11 min read
(For more resources related to this topic, see here.) Resource definitions In order to be able to define resources, we need to create a module that will be in charge of handling this. The main idea is that before calling a certain asset through ResourceManager, it has to be defined inResourceDefinitions. In this way, ResourceManager will always have access to some metadata it needs to create the asset (filenames, sizes, volumes, and so on). In order to identify the asset types (sounds, images, tiles, and fonts), we will define some constants (note that the number values of these constants are arbitrary; you could use whatever you want here). Let’s call them RESOURCE_TYPE_[type] (feel free to use another convention if you want to). To make things easier, just follow this convention for now since it’s the one we’ll use in the rest of the book. You should enter them in main.lua as follows: RESOURCE_TYPE_IMAGE = 0RESOURCE_TYPE_TILED_IMAGE = 1RESOURCE_TYPE_FONT = 2RESOURCE_TYPE_SOUND = 3 If you want to understand the actual reason behind these resource type constants, take a look at the load function of our ResourceManager entity in the next section. We need to create a file named resource_definitions.lua and add some simple methods that will handle it. Add the following line to it: module ( “ResourceDefinitions”, package.seeall ) The preceding line indicates that all of the code in the file should be treated as a module function, being accessed through ResourceDefinitions in the code. This is one of many Lua patterns used to create modules. If you’re not used to the Lua’s module function, you can read about it in the modules tutorial at http://lua-users.org/wiki/ModulesTutorial. Next, we will create a table that contains these definitions: local definitions = {} This will be used internally and is not accessible through the module API, so we create it using the keyword local. Now, we need to create the setter, getter, and unload methods for the definitions. The setter method (called set) stores the definition parameter (a table) in the definitions table, using the name parameter (a string) as the key, as follows: function ResourceDefinitions:set(name, definition) definitions[name] = definitionend The getter method (called get, duh!) retrieves the definition that was previously stored (by use of ResourceDefinitions:set ()) using the name parameter as the key of the definitions table, as follows: function ResourceDefinitions:get(name) return definitions[name]end The final method that we’re creating is remove. We use it to clear the memory space used by the definition. In order to achieve this we assign nil to an entry in the definitions table indexed by the name parameter as follows: function ResourceDefinitions:remove (name) definitions[name] = nilend In this way, we remove the reference to the object, allowing the memory to be released by the garbage collector. This may seem useless here, but it’s a good example of how you should manage your objects to be removed from memory by the garbage collector. And besides this, we don’t know information comes in a resource definition; it may be huge, we just don’t know. This is all we need for the resource definitions. We’re making use of the dynamism that Lua provides. See how easy it was to create a repository for definitions that is abstracted from the content of each definition. We’ll define different fields for each asset type, and we don’t need to define them beforehand as we probably would have needed to do in C++. Resource manager We will now create our resource manager. This module will be in charge of creating and storing our decks and assets in general. We’ll retrieve the assets with one single command, and they’ll come from the cache or get created using the definition. We need to create a file named resource_manager.lua and add the following line to it: module ( “ResourceManager”, package.seeall ) This is the same as in the resource definitions; we’re creating a module that will be accessed using ResourceManager. ASSETS_PATH = ‘assets/’ We now create the ASSETS_PATH constant. This is the path where we will store our assets. You could have many paths for different kinds of assets, but in order to keep things simple, we’ll keep all of them in one single directory in this example. Using this constant will allow us to use just the filename instead of having to write the whole path when creating the actual resource definitions, saving us some phalanx injuries! local cache = {} Again, we’re creating a cache table as a local variable. This will be the variable that will store our initialized assets. Now we should take care of implementing the important functionality. In order to make this more readable, I’ll be using methods that we define in the following pages. So, I recommend that you read the whole section before trying to run what we code now. The full source code can be downloaded from the book’s website, featuring inline comments. In the book, we removed the comments for brevity’s sake. Getter The first thing we will implement is our getter method since it’s simple enough: function ResourceManager:get ( name ) if (not self:loaded ( name )) then self:load ( name ) end return cache[name] end This method receives a name parameter that is the identifier of the resource we’re working with. On the first line, we call loaded (a method that we will define soon) to see if the resource identified by name was already loaded. If it was, we just need to return the cached value, but if it was not we need to load it, and that’s what we do in the if statement. We use the internalload method (which we will define later as well) to take care of the loading. We will make this load method store the loaded object in the cache table. So after loading it, the only thing we have to do is return the object contained in the cache table indexed by name. One of the auxiliary functions that we use here is loaded. Let’s implement it since it’s really easy to do so: function ResourceManager:loaded ( name ) return cache[name] ~= nil end What we do here is check whether the cache table indexed by the name parameter is not equal to nil. If cache has an object under that key, this will return true, and that’s what we were looking for to decide whether the object represented by the name parameter was already loaded. Loader load and its auxiliary functions are the most important methods of this module. They’ll be slightly more complex than what we’ve done so far since they make the magic happen. Pay special attention to this section. It’s not particularly hard, but it might get confusing. Like the previous methods, this one receives just the name parameter that represents the asset we’re loading as follows: function ResourceManager:load ( name ) First of all, we retrieve the definition for the resource associated to name. We make a call to the get method from ResourceDefinitions, which we defined earlier as follows: local resourceDefinition = ResourceDefinitions:get( name ) If the resource definition does not exist (because we forgot to define it before), we print an error to the screen, as follows: if not resourceDefinition then print(“ERROR: Missing resource definition for “ .. name ) If the resource definition was retrieved successfully, we create a variable that will hold the resource and (pay attention) we call the correct load auxiliary function, depending on the asset type. else local resource Remember the RESOURCE_TYPE_[type] constants that we created in the ResourceDefinitions module? This is the reason for their existence. Thanks to the creation of the RESOURCE_TYPE_[type] constants, we now know how to load the resources correctly. When we define a resource, we must include a type key with one of the resource types. We’ll insist on this soon. What we do now is call the correct load method for images, tiled images, fonts, and sounds, using the value stored in resourceDefinition.type as follows: if (resourceDefinition.type == RESOURCE_TYPE_IMAGE) then resource = self:loadImage ( resourceDefinition ) elseif (resourceDefinition.type == RESOURCE_TYPE_TILED_IMAGE) then resource = self:loadTiledImage ( resourceDefinition ) elseif (resourceDefinition.type == RESOURCE_TYPE_FONT) then resource = self:loadFont ( resourceDefinition ) elseif (resourceDefinition.type == RESOURCE_TYPE_SOUND) then resource = self:loadSound ( resourceDefinition ) end After loading the current resource, we store it in the cache table, in an entry specified by the name parameter, as follows: -- store the resource under the name on cache cache[name] = resource endend Now, let’s take a look at all of the different load methods. The expected definitions are explained before the actual functions so you have a reference when reading them. Images Loading images is something that we’ve already done, so this is going to look somewhat familiar. In this book, we’ll have two ways of defining images. Let’s take a look at them: {type = RESOURCE_TYPE_IMAGEfileName = “tile_back.png”,width = 62,height = 62,} As you may have guessed, the type key is the one used in the load function. In this case, we need to make it of type RESOURCE_TYPE_IMAGE. Here we are defining an image that has specific width and height values, and that is located at assets/title_back.png. Remember that we will use ASSET_PATH in order to avoid writing assets/ a zillion times. That’s why we’re not writing it on the definition. Another useful definition is: {type = RESOURCE_TYPE_IMAGEfileName = “tile_back.png”,coords = { -10, -10, 10, 10 }} This is handy when you want a specific rectangle inside a bigger image. You can use the cords attribute to define this rectangle. For example, we get a square with 20 pixel long sides centered in the image by specifying coords = { -10, -10, 10, 10 }. Now, let’s take a look at the actual loadImage method to see how this all falls into place: function ResourceManager:loadImage ( definition ) local image First of all, we use the same technique of defining an empty variable that will hold our image: local filePath = ASSETS_PATH .. definition.fileName We create the actual full path by appending the value of fileName in the definition to the value of the ASSETS_PATH constant. if checks whether the coords attribute is defined: if definition.coords thenimage = self:loadGfxQuad2D ( filePath, definition.coords ) Then, we use another auxiliary function called loadGfxQuad2D. This will be in charge of creating the actual image. The reason why we’re using another auxiliary function is that the code used to create the image is the same for both definition styles, but the data in the definition needs to be processed differently. In this case, we just pass the coordinates of the rectangle. else local halfWidth = definition.width / 2 local halfHeight = definition.height / 2 image = self:loadGfxQuad2D(filePath, {-halfWidth, -halfHeight, halfWidth, halfHeight} ) If there were no coords attribute, we’d assume the image is defined using width and height. So what we do is to define a rectangle that covers the whole width and height for the image. We do this by calculating halfWidth and halfHeight and then passing these values to theloadGfxQuad2D method. Remember the discussion about the texture coordinates in Moai SDK; this is the reason why we need to divide the dimensions by 2 and pass them as negative and positive parameters for the rectangle. This allows it to be centered on (0, 0). After loading the image, we return it so it can be stored in the cache by the load method: end return imageend Now the last method we need to write is loadGfxQuad2D. This method is basically to display an image as follows: function ResourceManager:loadGfxQuad2D ( filePath, coords ) local image = MOAIGfxQuad2D.new () image:setTexture ( filePath ) image:setRect ( unpack(cords) ) return imageend Lua’s unpack method is a nice tool that allows you to pass a table as separate parameters. You can use it to split a table into multiple variables as well: x, y = unpack ( position_table )   What we do here is instantiate the MOAIGfxQuad2D class, set the texture we defined in the previous function, and use the coordinates we constructed to set the rectangle this image will use from the original texture. Then we return it so loadImage can use it. Well! That was it for images. It may look complicated at first, but it’s not that complex. The rest of the assets will be simpler than this, so if you understood this one, the rest will be a piece of cake.
Read more
  • 0
  • 0
  • 2046
Modal Close icon
Modal Close icon