How-To Tutorials

(For more resources related to this topic, see here.) Posting on your blog The central activity you'll be doing with your blog is adding posts. A post is like an article in a magazine; it's got a title, content, and an author (in this case, you, though WordPress allows multiple authors to contribute to a blog). If a blog is like an online diary, every post is an entry in that diary. A blog post also has a lot of other information attached to it, such as a date, excerpt, tags, and categories. In this section, you will learn how to create a new post and what kind of information to attach to it. Adding a simple post Let's review the process of adding a simple post to your blog. Whenever you want to add content or carry out a maintenance process on your WordPress website, you have to start by logging in to the WP Admin (WordPress Administration panel) of your site. To get to the admin panel, just point your web browser to http://yoursite.com/wp-admin. Remember that if you have installed WordPress in a subfolder (for example, blog), your URL has to include the subfolder (that is, http://yoursite.com/blog/wp-admin). When you first log in to the WP Admin, you'll be at the Dashboard. The Dashboard has a lot of information on it so don't worry about that right now. The quickest way to get to the Add New Post page at any time is to click on + New and then the Post link at the top of the page in the top bar. This is the Add New Post page: To add a new post to your site quickly, all you have to do is: Type in a title into the text field under Add New Post (for example, Making Lasagne). Type the text of your post in the content box. Note that the default view is Visual, but you actually have a choice of the Text view as well. Click on the Publish button, which is at the far right. Note that you can choose to save a draft or preview your post as well. Once you click on the Publish button, you have to wait while WordPress performs its magic. You'll see yourself still on the Edit Post screen, but now the following message would have appeared telling you that your post was published, and giving you a link View post: If you view the front page of your site, you'll see that your new post has been added at the top (newest posts are always at the top). Common post options Now that we've reviewed the basics of adding a post, let's investigate some of the other options on the Add New Post and Edit Post pages. In this section we'll look at the most commonly used options, and in the next section we'll look at the more advanced options. Categories and tags Categories and tags are two types of information that you can add to a blog post. We use them to organize the information in your blog by topic and content (rather than just by, say, date), and to help visitors find what they are looking for on your blog. Categories are primarily used for structural organizing. They can be hierarchical, meaning a category can be a parent of another category. A relatively busy blog will probably have at least 10 categories, but probably not more than 15 or 20. Each post in such a blog is likely to have from one up to, maybe four categories assigned to it. For example, a blog about food and cooking might have these categories: Cooking Adventures, In The Media, Ingredients, Opinion, Recipes Found, Recipes Invented, and Restaurants. Of course, the numbers mentioned are just suggestions; you can create and assign as many categories as you like. The way you structure your categories is entirely up to you as well. There are no true rules regarding this in the WordPress world, just guidelines like these. Tags are primarily used as shorthand for describing the topics covered in a particular blog post. A relatively busy blog will have anywhere from 15 to even 100 tags in use. Each post in this blog is likely to have 3 to 10 tags assigned to it. For example, a post on the food blog about a recipe for butternut squash soup may have these tags: soup, vegetarian, autumn, hot, and easy. Again, you can create and assign as many tags as you like. Let's add a new post to the blog. After you give it a title and content, let's add tags and categories. While adding tags, just type your list of tags into the Tags box on the right, separated by commas: Then click on the Add button. The tags you just typed in will appear below the text field with little x buttons next to them. You can click on an x button to delete a tag. Once you've used some tags in your blog, you'll be able to click on the Choose from the most used tags link in this box so that you can easily re-use tags. Categories work a bit differently than tags. Once you get your blog going, you'll usually just check the boxes next to existing categories in the Categories box. In this case, as we don't have any existing categories, we'll have to add one or two. In the Categories box on the right, click on the + Add New Category link. Type your category into the text field, and click on the Add New Category button. Your new category will show up in the list, already checked. Look at the following screenshot: If in the future you want to add a category that needs a parent category, select — Parent Category — from the pull-down menu before clicking on the Add New Category button. If you want to manage more details about your categories, move them around, rename them, assign parent categories, and assign descriptive text. You can do so on the Categories page. Click on the Publish button, and you're done (you can instead choose to schedule a post; we'll explore that in detail in a few pages). When you look at the front page of your site, you'll see your new post on the top, your new category in the sidebar, and the tags and category (that you chose for your post) listed under the post itself. Images in your posts Almost every good blog post needs an image! An image will give the reader an instant idea of what the post is about, and the image will draw people's attention as well. WordPress makes it easy to add an image to your post, control default image sizes, make minor edits to that image, and designate a featured image for your post. Adding an image to a post Luckily, WordPress makes adding images to your content very easy. Let's add an image to the post we just created. You can click on Edit underneath your post on the front page of your site to get there quickly. Alternatively, go back to the WP Admin, open Posts in the main menu, and then click on the post's title. To add an image to a post, first you'll need to have that image on your computer, or know the exact URL pointing to the image if it's already online. Before you get ready to upload an image, make sure that your image is optimized for the Web. Huge files will be uploaded slowly and slow down the process of viewing your site. Just to give you a good example here, I'm using a photo of my own so I don't have to worry about any copyright issues (always make sure to use only the images that you have the right to use, copyright infringement online is a serious problem, to say the least). I know it's on the desktop of my computer. Once you have a picture on your computer and know where it is, carry out the following steps to add the photo to your blog post: Click on the Add Media button, which is right above the content box and below the title box: The box that appears allows you to do a number of different things regarding the media you want to include in your post. The most user-friendly feature here, however, is the drag-and-drop support. Just drag the image from your desktop and drop it into the center area of the page labeled as Drop files anywhere to upload. Immediately after dropping the image, the uploader bar will show the progress of the operation, and when it's done, you'll be able to do some final tuning up. The fields that are important right now are Title, Alt Text, Alignment, Link To, and Size. Title is a description for the image, Alt Text is a phrase that's going to appear instead of the image in case the file goes missing or any other problems present themselves, Alignment will tell the image whether to have text wrap around it and whether it should be right, left, or center, Link To instructs WordPress whether or not to link the image to anything (a common solution is to select None), and Size is the size of the image. Once you have all of the above filled out click on Insert into post. This box will disappear, and your image will show up in the post—right where your cursor was prior to clicking on the Add Media button—on the edit page itself (in the visual editor, that is. If you're using the text editor, the HTML code of the image will be displayed instead). Now, click on the Update button, and go and look at the front page of your site again. There's your image! Controlling default image sizes You may be wondering about those image sizes. What if you want bigger or smaller thumbnails? Whenever you upload an image, WordPress creates three versions of that image for you. You can set the pixel dimensions of those three versions by opening Settings in the main menu, and then clicking on Media. This takes you to the Media Settings page. Here you can specify the size of the uploaded images for: Thumbnail size Medium size Large size If you change the dimensions on this page, and click on the Save Changes button, only images you upload in the future will be affected. Images you've already uploaded to the site will have had their thumbnail, medium, and large versions created already using the old settings. It's a good idea to decide what you want your three media sizes to be early on in your site, so you can set them and have them applied to all images, right from the start. Another thing about uploading images is the whole craze with HiDPI displays, also called Retina displays. Currently, WordPress is in a kind of a transitional phase with images and being in tune with the modern display technology; the Retina Ready functionality was introduced quite recently in WordPress 3.5. In short, if you want to make your images Retina-compatible (meaning that they look good on iPads and other devices with HiDPI screens), you should upload the images at twice the dimensions you plan to display them in. For example, if you want your image to be presented as 800 pixel wide and 600 pixel high, upload it as 1,600 pixel wide and 1,200 pixel high. WordPress will manage to display it properly anyway, and whoever visits your site from a modern device will see a high-definition version of the image. In future versions, WordPress will surely provide a more managed way of handling Retina-compatible images. Editing an uploaded image As of WordPress 2.9, you can now make minor edits on images you've uploaded. In fact, every image that has been previously uploaded to WordPress can be edited. In order to do this, go to Media Library by clicking on the Media button in the main sidebar. What you'll see is a standard WordPress listing (similar to the one we saw while working with posts) presenting all media files and allowing you to edit each one. When you click on the Edit link and then the Edit Image button on the subsequent screen, you'll enter the Edit Media section. Here, you can perform a number of operations to make your image just perfect. As it turns out, WordPress does a good enough job with simple image tuning so you don't really need expensive software such as Photoshop for this. Among the possibilities you'll find cropping, rotating, and flipping vertically and horizontally. For example, you can use your mouse to draw a box as I have done in the preceding image. On the right, in the box marked Image Crop, you'll see the pixel dimensions of your selection. Click on the Crop icon (top left), then the Thumbnail radio button (on the right), and then Save (just below your photo). You now have a new thumbnail! Of course, you can adjust any other version of your image just by making a different selection prior to hitting the Save button. Play around a little and you can become familiar with the details. Designating a featured image As of WordPress 2.9, you can designate a single image that represents your post. This is referred to as the featured image. Some themes will make use of this, and some will not. The default theme, the one we've been using, is named Twenty Thirteen, and it uses the featured image right above the post on the front page. Depending on the theme you're using, its behavior with featured images can vary, but in general, every modern theme supports them in one way or the other. In order to set a featured image, go to the Edit Post screen. In the sidebar you'll see a box labeled Featured Image. Just click on the Set featured image link. After doing so, you'll see a pop-up window, very similar to the one we used while uploading images. Here, you can either upload a completely new image or select an existing image by clicking on it. All you have to do now is click on the Set featured image button in the bottom right corner. After completing the operation, you can finally see what your new image looks like on the front page. Also, keep in mind that WordPress uses featured images in multiple places not only the front page. And as mentioned above, much of this behavior depends on your current theme. Using the visual editor versus text editor WordPress comes with a visual editor, otherwise known as a WYSIWYG editor (pronounced wissy-wig, and stands for What You See Is What You Get). This is the default editor for typing and editing your posts. If you're comfortable with HTML, you may prefer to write and edit your posts using the text editor—particularly useful if you want to add special content or styling. To switch from the rich text editor to the text editor, click on the Text tab next to the Visual tab at the top of the content box: You'll see your post in all its raw HTML glory, and you'll get a new set of buttons that lets you quickly bold and italicize text, as well as add link code, image code, and so on. You can make changes and swap back and forth between the tabs to see the result. Even though the text editor allows you to use some HTML elements, it's not a fully fledged HTML support. For instance, using the <p> tags is not necessary in the text editor, as they will be stripped by default. In order to create a new paragraph in the text editor, all you have to do is press the Enter key twice. That being said, at the same time, the text editor is currently the only way to use HTML tables in WordPress (within posts and pages). You can easily place your table content inside the <table><tr><td> tags and WordPress won't alter it in any way, effectively allowing you to create the exact table you want. Another thing the text editor is most commonly used for is introducing custom HTML parameters in the <img /> and <a> tags and also custom CSS classes in other popular tags. Some content creators actually prefer working with the text editor rather than the visual editor because it gives them much more control and more certainty regarding the way their content is going to be presented on the frontend. Lead and body One of many interesting publishing features WordPress has to offer is the concept of the lead and the body of the post. This may sound like a strange thing, but it's actually quite simple. When you're publishing a new post, you don't necessarily want to display its whole contents right away on the front page. A much more user-friendly approach is to display only the lead, and then display the complete post under its individual URL. Achieving this in WordPress is very simple. All you have to do is use the Insert More Tag button available in the visual editor (or the more button in the text editor). Simply place your cursor exactly where you want to break your post (the text before the cursor will become the lead) and then click on the Insert More Tag button: An alternative way of using this tag is to switch to the text editor and input the tag manually, which is <!--more-->. Both approaches produce the same result. Clicking on the main Update button will save the changes. On the front page, most WordPress themes display such posts by presenting the lead along with a Continue reading link, and then the whole post (both the lead and the rest of the post) is displayed under the post's individual URL. Drafts, pending articles, timestamps, and managing posts There are four additional, simple but common, items I'd like to cover in this section: drafts, pending articles, timestamps, and managing posts. Drafts WordPress gives you the option to save a draft of your post so that you don't have to publish it right away but can still save your work. If you've started writing a post and want to save a draft, just click on the Save Draft button at the right (in the Publish box), instead of the Publish button. Even if you don't click on the Save Draft button, WordPress will attempt to save a draft of your post for you, about once a minute. You'll see this in the area just below the content box. The text will say Saving Draft... and then show the time of the last draft saved: At this point, after a manual save or an autosave, you can leave the Edit Post page and do other things. You'll be able to access all of your draft posts from Dashboard or from the Edit Posts page. In essence, drafts are meant to hold your "work in progress" which means all the articles that haven't been finished yet, or haven't even been started yet, and obviously everything in between. Pending articles Pending articles is a functionality that's going to be a lot more helpful to people working with multi-author blogs, rather than single-author blogs. The thing is that in a bigger publishing structure, there are individuals responsible for different areas of the publishing process. WordPress, being a quality tool, supports such a structure by providing a way to save articles as Pending Review. In an editor-author relationship, if an editor sees a post marked as Pending Review, they know that they should have a look at it and prepare it for the final publication. That's it for the theory, and now how to do it. While creating a new post, click on the Edit link right next to the Status: Draft label: Right after doing so, you'll be presented with a new drop-down menu from which you can select Pending Review and then click on the OK button. Now just click on the Save as Pending button that will appear in place of the old Save Draft button, and you have a shiny new article that's pending review. Timestamps WordPress will also let you alter the timestamp of your post. This is useful if you are writing a post today that you wish you'd published yesterday, or if you're writing a post in advance and don't want it to show up until the right day. By default, the timestamp will be set to the moment you publish your post. To change it, just find the Publish box, and click on the Edit link (next to the calendar icon and Publish immediately), and fields will show up with the current date and time for you to change: Change the details, click on the OK button, and then click on Publish to publish your post (or save a draft). Managing posts If you want to see a list of your posts so that you can easily skim and manage them, you just need to go to the Edit Post page in the WP Admin and navigate to Posts in the main menu. Once you do so, there are many things you can do on this page as with every management page in the WP Admin.

(For more resources related to this topic, see here.) BusyBox Compiling BusyBox, the Swiss Army Knife of Embedded Linux, it can be compiled into a single binary for different architectures. Before compiling software, we must get a compiler and the corresponding libraries, a build host and a target platform, the build host is the one running the compiler, the target platform is the one running the target binary. Herein, the desktop development system is a 64 bit X86 Ubuntu systems, it will be used as our build host, and an ARM Android system will be used as our target platform. To compile BusyBox on X86-64 Ubuntu system for ARM, we need a cross compiler. The gcc-arm-linux-gnueabi cross compiler can be installed directly on Ubuntu: $ sudo apt-get install gcc-arm-linux-gnueabi On the other Linux distributions, Google's official NDK is a good choice if want to share Android's Bionic C library, but since Bionic C library lacks lots of POSIX C header files, if want to get most of BusyBox applets building, the prebuilt version of Linaro GCC with Glibc is preferable, we can download it from http://www.linaro.org/downloads/, for example: http://releases.linaro.org/13.04/components/toolchain/binaries/gcc-linaro-aarch64-none-elf-4.7-2013.04-20130415_linux.tar.bz2. Before compiling, to simplify the configuration, enable the largest generic configuration with make defconfig and configure cross compiler: arm-linux-gnueabi-gcc with: make menuconfig. $ make defconfig $ make menuconfig Busybox Settings ---> Build Options ---> (arm-linux-gnueabi-) Cross Compiler prefix After configuration, we can simply compile it with: $ make Then, a BusyBox binary will be compiled for ARM: $ file busybox busybox: ELF 32-bit LSB executable, ARM, version 1(SYSV), dynamically linked (uses shared libs), stripped To list the shared libraries required by BusyBox binary for ARM, a command arm-linux-gnueabi-readelf should be used: $ arm-linux-gnueabi-readelf -d ./busybox grep "Shared library:" | cut -d'[' -f2 | tr -d ']'| libm.so.6 libc.so.6 ld-linux.so.3 To get the full path, we can get the library search path at first: $ arm-linux-gnueabi-ld --verbose grep SEARCH | tr ';' 'n' | cut -d'"' -f2 | tr -d '"'| /lib/arm-linux-gnueabi /usr/lib/arm-linux-gnueabi /usr/arm-linux-gnueabi/lib Then, we can find out that /usr/arm-linux-gnueabi/lib is the real search path in our platform and we can get the full path of the libraries as below: $ ls /usr/arm-linux-gnueabi/lib/{libm.so.6,libc.so.6,ld-linux.so.3} /usr/arm-linux-gnueabi/lib/ld-linux.so.3 /usr/arm-linux-gnueabi/lib/libc.so.6/usr/arm-linux-gnueabi/lib/libm.so.6 By default, the binary is dynamically linked, to enable static linking, configure BusyBox as following: Busybox Settings ---> Build Options ---> [*] Build BusyBox as a static binary (no shared libs) If using a new Glibc to compile BusyBox with static linking, to avoid such error: inetd.c:(.text.prepare_socket_fd+0x7e): undefined reference to `bindresvport' We need to disable CONFIG_FEATURE_INETD_RPC: Networking Utilities ---> [*] inetd [ ] Support RPC services Then, recompile it with make. BusyBox Installation This section shows how to install the above compiled BusyBox binaries on an ARM Android system. The installation of BusyBox means to create soft links for all of its built-in applets, use wc applet as an example: $ ln -s busybox wc $ echo "Hello, Busybox." ./wc -w| 2 BusyBox can be installed at the earlier compiling stage or at run-time. To build a minimal embedded file system with BusyBox, we'd better install it at the compiling stage with 'make install' for it helps to create the basic directory architecture of a standard Linux root file system and create soft links to BusyBox under the corresponding directories. With this installation method, we need to configure the target installation directory as following, use ~/busybox-ramdisk/ directory as an example: Busybox Settings ---> Installation Options ("make install" behavior) ---> (~/busybox-ramdisk/) BusyBox installation prefix After installation, we may get such a list of the file and directories: $ ls ~/busybox-ramdisk/ bin linuxrc sbin usr But to install it on an existing ARM Android system, it may be easier to install BusyBox at run-time with its --install option. With --install, by default, hard links will be created, to create soft links (symbolic links), -s option should be appended. If want to create links across different file systems (E.g. in Android system, to install BusyBox to /system/bin but BusyBox itself is put in the /data directory), -s must be used. To use the -s option, BusyBox should be configured as below: Busybox Settings ---> General Configuration ---> [*] Support --install [-s] to install applet links at runtime Now, let’s introduce how to install the above compiled BusyBox binaries to an existing ARM Android system. To do so, the Android system must be rooted to make sure the /data and / directories are writable. We will not show how to root an Android device, please get help from your product maker. Or if no real rooted Android device available, the Android emulator (emulator) provided by ADT (Android Development Toolkit, http://developer.android.com/sdk/index.html) can be used to start a rooted Android system on a virtual ARM Android device. To create a virtual ARM Android device and to use the Android emulator, please read the online documents provided by Google on http://developer.android.com/tools/help/android.html and http://developer.android.com/tools/help/emulator.html. Now, let's assume a rooted Android system is already running there with the USB debugging option enabled for Android Debug Bridge (adb, http://developer.android.com/tools/help/adb.html) support, for example, to check if such a device is started, we can run: $ adb devices List of devices attached emulator-5554 device As we can see, a virtual Android device running on Android emulator: emulator-5554 is there. Now, we are able to show how to install the BusyBox binaries on the existing Android system. Since the dynamically linked and statically linked BusyBox binaries are different, we will introduce how to install them respectively. Install the statically linked Busybox binary To install the statically installed Busybox binary, we only need to upload the BusyBox binary itself: $ adb push busybox /data/ Afterwards, install it with the --install option, for example, install it to the /bin directory of the Andriod system: $ adb shell root@android:/ # mount -o remount,rw / root@android:/ # mkdir /bin/ root@android:/ # /data/busybox --install -s /bin To be able to create the /bin directory for installation, the / directory is remounted to be writable. After installation, soft links are created under /bin for all of the built-in applets. If the -s option is not used, it will fail to create the hard links across the /bin and /data directories, that's why we must use -s option, the failure log looks like: busybox: /bin/[: Invalid cross-device link busybox: /bin/[[: Invalid cross-device link busybox: /bin/acpid: Invalid cross-device link busybox: /bin/add-shell: Invalid cross-device link (...truncated...) To execute the just installed applets, use md5sum as an example: $ /bin/md5sum /bin/ls 19994347b06d5ef7dbcbce0932960395 /bin/ls To run the applets without the full path, the /bin directory should be appended to the PATH variable: $ export PATH=$PATH:/bin Then, all of the BusyBox applets can be executed directly, that means we have installed Busybox successfully. To make the settings permanently, the above commands can be added to a script and such a script can be run as an Android service. Install the statically linked Busybox binary To install the statically installed Busybox binary, we only need to upload the BusyBox binary itself: $ adb push busybox /data/ Afterwards, install it with the --install option, for example, install it to the /bin directory of the Andriod system: $ adb shell root@android:/ # mount -o remount,rw / root@android:/ # mkdir /bin/ root@android:/ # /data/busybox --install -s /bin To be able to create the /bin directory for installation, the / directory is remounted to be writable. After installation, soft links are created under /bin for all of the built-in applets. If the -s option is not used, it will fail to create the hard links across the /bin and /data directories, that's why we must use -s option, the failure log looks like: busybox: /bin/[: Invalid cross-device link busybox: /bin/[[: Invalid cross-device link busybox: /bin/acpid: Invalid cross-device link busybox: /bin/add-shell: Invalid cross-device link (...truncated...) To execute the just installed applets, use md5sum as an example: $ /bin/md5sum /bin/ls 19994347b06d5ef7dbcbce0932960395 /bin/ls To run the applets without the full path, the /bin directory should be appended to the PATH variable: $ export PATH=$PATH:/bin Then, all of the BusyBox applets can be executed directly, that means we have installed Busybox successfully. To make the settings permanently, the above commands can be added to a script and such a script can be run as an Android service. Install the dynamically linked BusyBox binary For a dynamically linked BusyBox, to install it, besides the installation of the BusyBox binary itself, the required dynamic linker/loader (ld-linux.so) and the dependent shared libraries (libc.so and libm.so) should be installed too. For the basic installation procedure are the same as the one for statically linked BusyBox, herein, we only introduce how to install the required ld-linux.so.3, libc.so.6 and libm.so.6. Without the above dynamic linker/loader and libraries, we may get such error while running the dynamically linked BusyBox: $ /data/busybox --install -s /bin /system/bin/sh: /data/busybox: No such file or directory Before installation, create another /lib directory on target Android system and then upload the above files to it: $ adb shell mkdir /lib $ adb push /usr/arm-linux-gnueabi/lib/ld-linux.so.3 /lib/ $ adb push /usr/arm-linux-gnueabi/lib/libc.so.6 /lib/ $ adb push /usr/arm-linux-gnueabi/lib/libm.so.6 /lib/ With the above installation, the dynamically linked BusyBox binary should also be able to run and we are able to install and execute its applets as before: $ /data/busybox --install -s /bin As we can see, the installation of the dynamically linked BusyBox binary requires extra installation of the dependent linker/loader and libraries. For real embedded system development, if only the BusyBox binary itself uses the shared libraries, the static linking should be used instead at compiling to avoid the extra installation and the time-cost run-time linking, otherwise, if many binaries share the same libraries, to reduce the total size cost for the size critical embedded systems, dynamic linking may be preferable. Summary In this article we learned about BusyBox compiling and its installation. BusyBox is a free GPL-licensed toolbox aimed at the embedded world. It is a collection of the tiny versions of many common Unix utilities. Resources for Article : Further resources on this subject: Embedding Doctests in Python Docstrings [Article] The Business Layer (Java EE 7 First Look) [Article] Joomla! 1.6: Organizing and Managing Content [Article]

(For more resources related to this topic, see here.) Files are the building blocks of any computer system. This article deals with portable handling of read-only application resources, and provides recipes to store the application data. Let us briefly consider the problems covered in this article. The first one is the access to application data files. Often, application data for desktop operating systems resides in the same folder as the executable file. With Android, things get a little more complicated. The application files are packaged in the .apk file, and we simply cannot use the standard fopen()-like functions, or the std::ifstream and std::ofstream classes. The second problem results from the different rules for the filenames and paths. Windows and Linux-based systems use different path separator characters, and provide different low-level file access APIs. The third problem comes from the fact that file I/O operations can easily become the slowest part in the whole application. User experience can become problematic if interaction lags are involved. To avoid delays, we should perform the I/O on a separate thread and handle the results of the Read() operation on yet another thread. To implement this, we have all the tools required. We start with abstract I/O interfaces, implement a portable .zip archives handling approach, and proceed to asynchronous resources loading. Abstracting file streams File I/O APIs differ slightly between Windows and Android (POSIX) operating systems, and we have to hide these differences behind a consistent set of C++ interfaces. Getting ready Please make sure you are familiar with the UNIX concept of the file and memory mapping. Wikipedia may be a good start (http://en.wikipedia.org/wiki/Memory-mapped_file). How to do it... From now on, our programs will read input data using the following simple interface. The base class iObject is used to add an intrusive reference counter to instances of this class: class iIStream: public iObject { public: virtual void Seek( const uint64 Position ) = 0; virtual uint64 Read( void* Buf, const uint64 Size ) = 0; virtual bool Eof() const = 0; virtual uint64 GetSize() const = 0; virtual uint64 GetPos() const = 0; The following are a few methods that take advantage of memory-mapped files: virtual const ubyte* MapStream() const = 0; virtual const ubyte* MapStreamFromCurrentPos() const = 0; }; This interface supports both memory-mapped access using the MapStream() and MapStreamFromCurrentPos() member functions, and sequential access with the BlockRead() and Seek() methods. To write some data to the storage, we use an output stream interface, as follows (again, the base class iObject is used to add a reference counter): class iOStream: public iObject { public: virtual void Seek( const uint64 Position ) = 0; virtual uint64 GetFilePos() const = 0; virtual uint64 Write( const void* B, const uint64 Size ) = 0; }; The Seek(), GetFileSize(), GetFilePos(), and filename-related methods of the iIStream interface can be implemented in a single class called FileMapper: class FileMapper: public iIStream { public: explicit FileMapper( clPtr<iRawFile> File ); virtual ~FileMapper(); virtual std::string GetVirtualFileName() const{return FFile->GetVirtualFileName(); } virtual std::string GetFileName() const{ return FFile->GetFileName(); } Read a continuous block of data from this stream and return the number of bytes actually read: virtual uint64 BlockRead( void* Buf, const uint64 Size ) { uint64 RealSize =( Size > GetBytesLeft() ) ? GetBytesLeft() : Size; Return zero if we have already read everything: if ( RealSize < 0 ) { return 0; } memcpy( Buf, ( FFile->GetFileData() + FPosition ),static_cast<size_t>( RealSize ) ); Advance the current position and return the number of copied bytes: FPosition += RealSize; return RealSize; } virtual void Seek( const uint64 Position ) { FPosition = Position; } virtual uint64 GetFileSize() const { return FFile->GetFileSize(); } virtual uint64 GetFilePos() const { return FPosition; } virtual bool Eof() const { return ( FPosition >= GetFileSize() ); } virtual const ubyte* MapStream() const { return FFile->GetFileData(); } virtual const ubyte* MapStreamFromCurrentPos() const { return ( FFile->GetFileData() + FPosition ); } private: clPtr<iRawFile> FFile; uint64 FPosition; }; The FileMapper uses the following iRawFile interface to abstract the data access: class iRawFile: public iObject { public: iRawFile() {}; virtual ~iRawFile() {}; void SetVirtualFileName( const std::string& VFName );voidSetFileName( const std::string& FName );std::string GetVirtualFileName() const; std::string GetFileName(); virtual const ubyte* GetFileData() const = 0; virtual uint64 GetFileSize() const = 0; protected: std::string FFileName; std::string FVirtualFileName; }; Along with the trivial GetFileName() and SetFileName() methods implemented here, in the following recipes we implement the GetFileData() and GetFileSize() methods. How it works... The iIStream::BlockRead() method is useful when handling non-seekable streams. For the fastest access possible, we use memory-mapped files implemented in the following recipe. The MapStream() and MapStreamFromCurrentPos() methods are there to provide access to memory-mapped files in a convenient way. These methods return a pointer to the memory where your file, or a part of it, is mapped to. The iOStream::Write() method works similar to the standard ofstream::write() function. Refer to the project 1_AbstractStreams for the full source code of this and the following recipe. There's more... The important problem while programming for multiple platforms, in our case for Windows and Linux-based Android, is the conversion of filenames. We define the following PATH_SEPARATOR constant, using OS-specific macros, to determine the path separator character in the following way: #if defined( _WIN32 ) const char PATH_SEPARATOR = '\'; #else const char PATH_SEPARATOR = '/'; #endif The following simple function helps us to make sure we use valid filenames for our operating system: inline std::string Arch_FixFileName(const std::string& VName) { std::string s( VName ); std::replace( s.begin(), s.end(), '\', PATH_SEPARATOR ); std::replace( s.begin(), s.end(), '/', PATH_SEPARATOR ); return s; } See also Implementing portable memory-mapped files Working with in-memory files Implementing portable memory-mapped files Modern operating systems provide a powerful mechanism called the memory-mapped files. In short, it allows us to map the contents of the file into the application address space. In practice, this means we can treat files as usual arrays and access them using C pointers. Getting ready To understand the implementation of the interfaces from the previous recipe we recommend to read about memory mapping. The overview of this mechanism implementation in Windows can be found on the MSDN page at http://msdn.microsoft.com/en-us/library/ms810613.aspx. To find out more about memory mapping, the reader may refer to the mmap() function documentation. How to do it... In Windows, memory-mapped files are created using the CreateFileMapping() and MapViewOfFile() API calls. Android uses the mmap() function, which works pretty much the same way. Here we declare the RawFile class implementing the iRawFile interface. RawFile holds a pointer to a memory-mapped file and its size: ubyte* FFileData; uint64 FSize; For the Windows version, we use two handles for the file and memory-mapping object, and for the Android, we use only the file handle: #ifdef _WIN32 HANDLE FMapFile; HANDLE FMapHandle; #else int FFileHandle; #endif We use the following function to open the file and create the memory mapping: bool RawFile::Open( const string& FileName,const string& VirtualFileName ) { At first, we need to obtain a valid file descriptor associated with the file: #ifdef OS_WINDOWS FMapFile = (void*)CreateFileA( FFileName.c_str(),GENERIC_READ,FILE_SHARE_READ,NULL, OPEN_EXISTING,FILE_ATTRIBUTE_NORMAL |FILE_FLAG_RANDOM_ACCESS,NULL ); #else FFileHandle = open( FileName.c_str(), O_RDONLY ); if ( FFileHandle == -1 ) { FFileData = NULL; FSize = 0; } #endif Using the file descriptor, we can create a file mapping. Here we omit error checks for the sake of clarity. However, the example in the supplementary materials contains more error checks: #ifdef OS_WINDOWS FMapHandle = (void*)CreateFileMapping( ( HANDLE )FMapFile,NULL, PAGE_READONLY,0, 0, NULL ); FFileData = (Lubyte*)MapViewOfFile((HANDLE)FMapHandle,FILE_MAP_READ, 0,0,0 ); DWORD dwSizeLow = 0, dwSizeHigh = 0; dwSizeLow = ::GetFileSize( FMapFile, &dwSizeHigh ); FSize = ((uint64)dwSizeHigh << 32) | (uint64)dwSizeLow; #else struct stat FileInfo; fstat( FFileHandle, &FileInfo ); FSize = static_cast<uint64>( FileInfo.st_size ); FFileData = (Lubyte*) mmap(NULL, FSize, PROT_READ,MAP_PRIVATE, FFileHandle, 0); close( FFileHandle ); #endif return true; } The correct deinitialization function closes all the handles: bool RawFile::Close() { #ifdef OS_WINDOWS if ( FFileData ) UnmapViewOfFile( FFileData ); if ( FMapHandle ) CloseHandle( (HANDLE)FMapHandle ); CloseHandle( (HANDLE)FMapFile ); #else if ( FFileData ) munmap( (void*)FFileData, FSize ); #endif return true; } The main functions of the iRawFile interface, GetFileData and GetFileSize, have trivial implementation here: virtual const ubyte* GetFileData() { return FFileData; } virtual uint64 GetFileSize() { return FSize; } How it works... To use the RawFile class we create an instance and wrap it into a FileMapper class instance: clPtr<RawFile> F = new RawFile(); F->Open("SomeFileName"); clPtr<FileMapper> FM = new FileMapper(F); The FM object can be used with any function supporting the iIStream interface. The hierarchy of all our iRawFile implementations looks like what is shown in the following figure: Implementing file writers Quite frequently, our application might want to store some of its data on the disk. Another typical use case we have already encountered is the downloading of some file from the network into a memory buffer. Here, we implement two variations of the iOStream interface for the ordinary and in-memory files. How to do it... Let us derive the FileWriter class from the iOStream interface. We add the Open() and Close() member functions on top of the iOStream interface and carefully implement the Write() operation. Our output stream implementation does not use memory-mapped files and uses ordinary file descriptors, as shown in the following code: class FileWriter: public iOStream { public: FileWriter(): FPosition( 0 ) {} virtual ~FileWriter() { Close(); } bool Open( const std::string& FileName ) { FFileName = FileName; We split Android and Windows-specific code paths using defines: #ifdef _WIN32 FMapFile = CreateFile( FFileName.c_str(),GENERIC_WRITE, FILE_SHARE_READ,NULL, CREATE_ALWAYS,FILE_ATTRIBUTE_NORMAL, NULL ); return !( FMapFile == ( void* )INVALID_HANDLE_VALUE ); #else FMapFile = open( FFileName.c_str(), O_WRONLY|O_CREAT ); FPosition = 0; return !( FMapFile == -1 ); #endif } The same technique is used in the other methods. The difference between both OS systems is is trivial, so we decided to keep everything inside a single class and separate the code using defines: void Close() { #ifdef _WIN32 CloseHandle( FMapFile ); #else if ( FMapFile != -1 ) { close( FMapFile ); } #endif } virtual std::string GetFileName() const { return FFileName; } virtual uint64 GetFilePos() const { return FPosition; } virtual void Seek( const uint64 Position ) { #ifdef _WIN32 SetFilePointerEx( FMapFile,*reinterpret_cast<const LARGE_INTEGER*>(&Position ),NULL, FILE_BEGIN ); #else if ( FMapFile != -1 ) { lseek( FMapFile, Position, SEEK_SET ); } #endif FPosition = Position; } However, things may get more complex if you decide to support more operating systems. It can be a good refactoring exercise. virtual uint64 Write( const void* Buf, const uint64 Size ) { #ifdef _WIN32 DWORD written; WriteFile( FMapFile, Buf, DWORD( Size ),&written, NULL ); #else if ( FMapFile != -1 ) { write( FMapFile, Buf, Size ); } #endif FPosition += Size; return Size; } private: std::string FFileName; #ifdef _WIN32 HANDLE FMapFile; #else int FMapFile; #endif uint64 FPosition; }; How it works… Now we can also present an implementation of the iOStream that stores everything in a memory block. To store arbitrary data in a memory block, we declare the Blob class, as shown in the following code: class Blob: public iObject { public: Blob(); virtual ~Blob(); Set the blob data pointer to some external memory block: void SetExternalData( void* Ptr, size_t Sz ); Direct access to data inside this blob: void* GetData(); … Get the current size of the blob: size_t GetSize() const; Check if this blob is responsible for managing the dynamic memory it uses: bool OwnsData() const; … Increase the size of the blob and add more data to it. This method is very useful in a network downloader: bool AppendBytes( void* Data, size_t Size ); … }; There are lots of other methods in this class. You can find the full source code in the Blob.h file. We use this Blob class, and declare the MemFileWriter class, which implements our iOStream interface, in the following way: class MemFileWriter: public iOStream { public: MemFileWriter(clPtr<Blob> Container); Change the absolute position inside a file, where new data will be written to: virtual void Seek( const uint64 Position ) { if ( Position > FContainer->GetSize() ) { Check if we are allowed to resize the blob: if ( Position > FMaxSize - 1 ) { return; } And try to resize it: if ( !FContainer->SafeResize(static_cast<size_t>( Position ) + 1 )) { return; } } FPosition = Position; } Write data to the current position of this file: virtual uint64 Write( const void* Buf, const uint64 Size ) { uint64 ThisPos = FPosition; Ensure there is enough space: Seek( ThisPos + Size ); if ( FPosition + Size > FMaxSize ) { return 0; } void* DestPtr = ( void* )( &( ( ( ubyte* )(FContainer->GetData() ))[ThisPos] ) ); Write the actual data: memcpy( DestPtr, Buf, static_cast<size_t>( Size ) ); return Size; } } private: … }; We omit the trivial implementations of GetFileName(), GetFilePos(), GetMaxSize(), SetContainer(), GetContainer(), GetMaxSize(), and SetMaxSize() member functions, along with fields declarations. You will find the full source code of them in the code bundle of the book. See also Working with in-memory files Working with in-memory files Sometimes it is very convenient to be able to treat some arbitrary in-memory runtime generated data as if it were in a file. As an example, let's consider using a JPEG image downloaded from a photo hosting, as an OpenGL texture. We do not need to save it into the internal storage, as it is a waste of CPU time. We also do not want to write separate code for loading images from memory. Since we have our abstract iIStream and iRawFile interfaces, we just implement the latter to support memory blocks as the data source. Getting ready In the previous recipes, we already used the Blob class, which is a simple wrapper around a void* buffer. How to do it... Our iRawFile interface consists of two methods: GetFileData() and GetFileSize(). We just delegate these calls to an instance of Blob: class ManagedMemRawFile: public iRawFile { public: ManagedMemRawFile(): FBlob( NULL ) {} virtual const ubyte* GetFileData() const { return ( const ubyte* )FBlob->GetData(); } virtual uint64 GetFileSize() const { return FBlob->GetSize(); } void SetBlob( const clPtr<Blob>& Ptr ) { FBlob = Ptr; } private: clPtr<Blob> FBlob; }; Sometimes it is useful to avoid the overhead of using a Blob object, and for such cases we provide another class, MemRawFile, that holds a raw pointer to a memory block and optionally takes care of the memory allocation: class MemRawFile: public iRawFile { public: virtual const ubyte* GetFileData() const { return (const ubyte*) FBuffer; } virtual uint64 GetFileSize() const { return FBufferSize; } void CreateFromString( const std::string& InString ); void CreateFromBuffer( const void* Buf, uint64 Size ); void CreateFromManagedBuffer( const void* Buf, uint64 Size ); private: bool FOwnsBuffer; const void* FBuffer; uint64 FBufferSize; }; How it works... We use the MemRawFile as an adapter for the memory block extracted from a .zip file and ManagedMemRawFile as the container for data downloaded from photo sites.

Machine listening We can acquire data from various sources. In a visualization context, however, we may encounter situations wherein we will need to control some elements of an animation with respect to some particular characteristic of a signal, for example, their amplitude or their frequency. Yet, these kinds of information are attributes of the signal, rather than parts of it. In other words, we need something to happen not with respect to some existent data (that is, our signal in this context) but with respect to certain characteristics of a data flow. Consider that an audio signal is completely unaware of how loud it is or of what its frequency is. Remember that audio signals are merely streams of numbers and that sounds are merely fluctuations of air pressure. The reason we understand sounds as having loudness or pitch, is because our auditory apparatus analyzes them and provides the brain with information on certain sonic qualities. Further, more sophisticated perceptual and cognitive processes perform additional kinds of analyses to extract as well as attribute information and meanings, so that we perceptually decipher what we hear. Likewise, we can say that a signal is periodic and has a certain frequency, only if we somehow analyze it. Remember that the output of a sinusoidal wave at a frequency of 200 Hz is just a flow of numbers between ±1. The datum 200 is not part of this signal, so the only way to make something happen with respect to this number is to actually generate it by means of analyzing the audio signal against its frequency. The task of retrieving statistical and other kinds of information from audio signals is generally referred to as machine listening. Machine listening is, in essence, to analyze signals in order to generate information that represent certain qualities of these signals. To properly understand and evaluate the kind of information we may get from some machine listening algorithm, it is worth distinguishing briefly the different kinds of properties a signal may have. Acoustic properties refer to physical properties of sound, and consequently of audio signals, particularly qualities such as amplitude, frequency, and spectrum. Psychoacoustic properties refer to low-level perceptional properties of audio signals, such as loudness, pitch, and timbre. Psychoacoustic properties are fundamentally different than their acoustic equivalents, the latter being intrinsically linked to perception. For instance, loudness refers to how loud something sounds, while the amplitude stands for the actual amount of the displacement of the air particles that occurs in the physical space. It has to be stressed that the various psychoacoustic qualities do relate and depend upon the acoustic properties of sound; nonetheless, the relationships are very complex and not that straightforward as they may appear to be. For example, loudness does not depend exclusively upon amplitude, but it also depends upon frequency, spectral content, and even upon a series of psychological and other factors. We can also speak of several families of higher-level perceptional properties, such as musical ones (scale, tonality, rhythm, genre, expressivity, and so on), cognitive ones (semantics, symbolical signification, and so on), and psychological ones (irritability, entertainability, ability to cause relaxation, and so on). Again, such properties may depend or relate to some extent to the acoustic or psychoacoustic qualities of sound; yet the inter-relationships may be extremely complex and even not fully understood in certain cases. Machine listening algorithms are not limited only to simple acoustic properties of a signal; sophisticated algorithms have been proposed for more complex problems as well such as musical style recognition and rhythm extraction. As far as musical qualities are concerned, the more specialized term musical information retrieval is sometimes encountered too. In SuperCollider, we can easily perform basic audio analyses to retrieve information on both physical as well as certain perceptional properties of audio signals using the available machine listening UGens, the most important of which will be discussed immediately. Music Information Retrieval (MIR) is an interdisciplinary field of science dealing with how to retrieve and classify information from music Tracking amplitude and loudness We can also use the Peak UGen, which will return the maximum peak amplitude every time it receives a trigger or the PeakFollower UGen which smoothly decays from the maximum value by some specified decay time. To track the minimum or the maximum value of a signal, we can use the RunningMin or RunningMax UGens. To track Root Mean Square (RMS) amplitude, we can use the RunningSum UGen. The following example shows how to use these UGens: (// tracking amplitude { var sound = SinOsc.ar(mul:LFNoise2.kr(1).range(0,1)); // source RunningSum.rms(sound,100).poll(label:'rms'); // rms Amplitude.kr(sound).poll(label:'peak'); // peak Peak.kr(sound, Impulse.kr(1)).poll(label:'peak_trig'); // peak when triggered PeakFollower.kr(sound).poll(label:'peak_dec'); // peak with decay RunningMin.kr(sound).poll(label:'min'); // minimum RunningMax.kr(sound).poll(label:'max'); // maximum Out.ar(0,sound); // write to output }.play; ) Sometimes we may want something to happen when a signal is silent or at least when it is below a certain level. In such cases, we can use DetecteSilence. There also exists a Loudness UGent which will estimate loudness in Sones (the measure of loudness). It is designed to analyze spectra and requires an FFT window of size 1024 for sampling rates of 44100 or 48000 and of the size 2048 for 88200 or 96000, respectively. For example: ( // track loudness { var sound, loudness; sound = SinOsc.ar(LFNoise2.ar(1).range(100,10000), mul:LFNoise0.ar(1).range(0,1)); // source loudness = FFT(LocalBuf(1024),sound); // sampling rates of 44.1/48K // loudness = FFT(LocalBuf(1024),sound); // sampling rates of 88.2/96K loudness = Loudness.kr(loudness).poll(label:loudness); Out.ar(0, sound); }.play; ) Tracking frequency As far as frequency is concerned, there are a number of relevant UGens, each of them implemented differently. The most simple one is ZeroCrossing, which will estimate the frequency by keeping track of how often an input signal crosses the horizontal axis, which represents 0 in terms of amplitude. Pitch is a more accurate frequency tracker, which also allows for some tweaking. Note that, regardless of its name, it performs frequency tracking rather than pitch tracking, the latter also depends on a series of other factors. More advanced frequency trackers are Tartini (which is based on the method used in the homonymous open source pitch tracker) and Qitch (which has to be used along with one of the special auxiliary WAV files it is distributed with). Tartini and Qitch are not included in the standard SuperCollider distribution but on the SC3Plugins extension bundle (available at http://sc3-plugins.sourceforge.net/). Pitch, Tartini, and Qitch will all return an array of instances of OutProxy obtaining both the estimated frequency as well as a flag of 1 or 0 to denote whether they successfully tracked some frequency or not. When attempting to track frequency we should always bear in mind that the former being a complicated process, not all trackers would work equally well for all kinds of signals. For example: ( // frequency tracking var qitchBuffer = Buffer.read (Server.default,"/Users/marinos/Library/Application Support/ SuperCollider/Extensions/SC3plugins/PitchDetection/extraqitchfiles/ QspeckernN2048SR44100.wav"); // path to auxiliary wav file for Qitch { // a complex signal var sound = Saw.ar(LFNoise2.ar(1).range(500,1000).poll(label: ActualFrequency)) + WhiteNoise.ar(0.4); ZeroCrossing.ar(sound).poll(label:ZeroCross); Pitch.kr(sound).poll(label:Pitch); Tartini.kr(sound).poll(label:Tartini); Qitch.kr(sound,qitchBuffer).poll(label:Qitch); Out.ar(0,sound!2); }.play; ) For this signal, Qitch is probably the most reasonable choice, judging by the output on my machine: ActualFrequency: 864.222 ZeroCross: 6368.27 Pitch: 171.704 Pitch: 1 Tartini: 95.0917 Tartini: 1 Qitch: 845.466 Qitch: 1 Timbre analysis and feature detection Timbre is a psycho-acoustic quality, and refers to what makes sounds distinct even if they have the same loudness and pitch. Of course this is a broad oversimplification of a very complex subject; in reality there isn't even a consensus on what exactly timbre stands for. While timbre has been proposed to depend on several qualities, in a machine listening context timbre recognition is almost exclusively based on analyzing spectra. Herein, we will focus on how to broadly detect several spectral features, rather than timbre per se, which is a rather indefinite quality. By the term feature we refer to anything that could be characteristic about a signal's spectral characteristics. In SuperCollider there is a plethora of relevant UGens, both in the standard distribution as well as in extension libraries. Of the most useful are SpecCentroid and ScpeFlatness used to calculate the spectral centroid and the spectral flatness, respectively. The former roughly stands for the most perceptually prominent frequency range in our signal while the latter is an indicator of how complicated our signal is (for example, for a sinusoid it would be 0 while for white noise close to 1). The SpecPcile UGen will calculate the cumulative distribution of a spectrum, and given a percentile of spectral energy as an argument, will return that frequency from which the given percentile of spectral energy lies below. In the SC3Plugins extensions bundle, we will also find the FFTCrest UGen, which will calculate the spectral crest of a signal, which, in short, indicates how flat or peaky a signal is, and the SensoryDissonance UGen, which will attempt to calculate how dissonant a signal is (with 1 being totally dissonant and 0 being totally consonant). The FFTSpread UGen measures the spectral spread of a signal, that is how wide or narrow its spectrum is and FFTSlope calculates the slope of the linear correlation line derived from the spectral magnitudes. Finally, the Goertzel UGen calculates the magnitude and phase at a single specified frequency. For example: ( // feature extraction { var sound = SinOsc.ar(240,mul:0.5) + Resonz.ar(ClipNoise.ar,2000,0.6,mul:SinOsc.kr(0.05).range(0,0.5)) + Saw.ar(2000,mul:SinOsc.kr(0.1).range(0,0.3)); var fft = FFT(LocalBuf(2048),sound); // a complex signal SpecCentroid.kr(fft).poll(label:Centroid); SpecFlatness.kr(fft).poll(label:Flatness); SpecPcile.kr(fft,0.8).poll(label:Percentile); FFTCrest.kr(fft,1800,2200).poll(label:Crest); SensoryDissonance.kr(fft).poll(label:Dissonance); Out.ar(0,sound!2); }.play; ) Summary In this article, we discussed about machine listening techniques and ways to retrieve information from audio signals. Resources for Article: Further resources on this subject: Data Profiling with IBM Information Analyzer [Article] Preparation Analysis of Data Source [Article] Spatial Data Services [Article]

(For more resources related to this topic, see here.) Multiplayer is everywhere. It's a staple of AAA games and small-budget indie offerings alike. Multiplayer games tap into our most basic human desires. Whether it be teaming up with strangers to survive a zombie apocalypse, or showing off your skills in a round of "Capture the Flag" on your favorite map, no artificial intelligence in the world comes close to the feeling of playing with a living, breathing, and thinking human being. Unity3D has a sizable number of third-party networking middleware aimed at developing multiplayer games, and is arguably one of the easiest platforms to prototype multiplayer games. The first networking system most people encounter in Unity is the built-in Unity Networking API . This API simplifies a great many tasks in writing networked code by providing a framework for networked objects rather than just sending messages. This works by providing a NetworkView component, which can serialize object state and call functions across the network. Additionally, Unity provides a Master server, which essentially lets players search among all public servers to find a game to join, and can also help players in connecting to each other from behind private networks. In this article, we will cover: Introducing multiplayer Introducing UDP communication Setting up your own Master server for testing What a NetworkView is Serializing object state Calling RPCs Starting servers and connecting to them Using the Master server API to register servers and browse available hosts Setting up a dedicated server model Loading networked levels Creating a Pong clone using Unity networking Introducing multiplayer games Before we get started on the details of communication over the Internet, what exactly does multiplayer entail in a game? As far as most players are concerned, in a multiplayer game they are sharing the same experience with other players. It looks and feels like they are playing the same game. In reality, they aren't. Each player is playing a separate game, each with its own game state. Trying to ensure that all players are playing the exact same game is prohibitively expensive. Instead, games attempt to synchronize just enough information to give the illusion of a shared experience. Games are almost ubiquitously built around a client-server architecture, where each client connects to a single server. The server is the main hub of the game, ideally the machine for processing the game state, although at the very least it can serve as a simple "middleman" for messages between clients. Each client represents an instance of the game running on a computer. In some cases the server might also have a client, for instance some games allow you to host a game without starting up an external server program. While an MMO ( Massively Multiplayer Online ) might directly connect to one of these servers, many games do not have prior knowledge of the server IPs. For example, FPS games often let players host their own servers. In order to show the user a list of servers they can connect to, games usually employ another server, known as the "Master Server" or alternatively the "Lobby server". This server's sole purpose is to keep track of game servers which are currently running, and report a list of these to clients. Game servers connect to the Master server in order to announce their presence publicly, and game clients query the Master server to get an updated list of game servers currently running. Alternatively, this Master server sometimes does not keep track of servers at all. Sometimes games employ "matchmaking", where players connect to the Lobby server and list their criteria for a game. The server places this player in a "bucket" based on their criteria, and whenever a bucket is full enough to start a game, a host is chosen from these players and that client starts up a server in the background, which the other players connect to. This way, the player does not have to browse servers manually and can instead simply tell the game what they want to play. Introducing UDP communication The built-in Unity networking is built upon RakNet . RakNet uses UDP communication for efficiency. UDP ( User Datagram Protocols ) is a simple way to send messages to another computer. These messages are largely unchecked, beyond a simple checksum to ensure that the message has not been corrupted. Because of this, messages are not guaranteed to arrive, nor are they guaranteed to only arrive once (occasionally a single message can be delivered twice or more), or even in any particular order. TCP, on the other hand, guarantees each message to be received just once, and in the exact order they were sent, although this can result in increased latency (messages must be resent several times if they fail to reach the target, and messages must be buffered when received, in order to be processed in the exact order they were sent). To solve this, a reliability layer must be built on top of UDP. This is known as rUDP ( reliable UDP ). Messages can be sent unreliably (they may not arrive, or may arrive more than once), or reliably (they are guaranteed to arrive, only once per message, and in the correct order). If a reliable message was not received or was corrupt, the original sender has to resend the message. Additionally, messages will be stored rather than immediately processed if they are not in order. For example, if you receive messages 1, 2, and 4, your program will not be able to handle those messages until message 3 arrives. Allowing unreliable or reliable switching on a per-message basis affords better overall performance. Messages, such as player position, are better suited to unreliable messages (if one fails to arrive, another one will arrive soon anyway), whereas damage messages must be reliable (you never want to accidentally drop a damage message, and having them arrive in the same order they were sent reduces race conditions). In Unity, you can serialize the state of an object (for example, you might serialize the position and health of a unit) either reliably or unreliably (unreliable is usually preferred). All other messages are sent reliably. Setting up the Master Server Although Unity provide their own default Master Server and Facilitator (which is connected automatically if you do not specify your own), it is not recommended to use this for production. We'll be using our own Master Server, so you know how to connect to one you've hosted yourself. Firstly, go to the following page: http://unity3d.com/master-server/ We're going to download two of the listed server components: the Master Server and the Facilitator as shown in the following screenshot: The servers are provided in full source, zipped. If you are on Windows using Visual Studio Express, open up the Visual Studio .sln solution and compile in the Release mode. Navigate to the Release folder and run the EXE (MasterServer.exe or Facilitator.exe). If you are on a Mac, you can either use the included XCode project, or simply run the Makefile (the Makefile works under both Linux and Mac OS X). The Master Server, as previously mentioned, enables our game to show a server lobby to players. The Facilitator is used to help clients connect to each other by performing an operation known as NAT punch-through . NAT is used when multiple computers are part of the same network, and all use the same public IP address. NAT will essentially translate public and private IPs, but in order for one machine to connect to another, NAT punch-through is necessary. You can read more about it here: http://www.raknet.net/raknet/manual/natpunchthrough.html The default port for the Master Server is 23466, and for the Facilitator is 50005. You'll need these later in order to configure Unity to connect to the local Master Server and Facilitator instead of the default Unity-hosted servers. Now that we've set up our own servers, let's take a look at the Unity Networking API itself. NetworkViews and state serialization In Unity, game objects that need to be networked have a NetworkView component. The NetworkView component handles communication over the network, and even helps make networked state serialization easier. It can automatically serialize the state of a Transform, Rigidbody, or Animation component, or in one of your own scripts you can write a custom serialization function. When attached to a game object, NetworkView will generate a NetworkViewID for NetworkView. This ID serves to uniquely identify a NetworkView across the network. An object can be saved as part of a scene with NetworkView attached (this can be used for game managers, chat boxes, and so on), or it can be saved in the project as a prefab and spawned later via Network.Instantiate (this is used to generate player objects, bullets, and so on). Network.Instantiate is the multiplayer equivalent to GameObject.Instantiate —it sends a message over the network to other clients so that all clients spawn the object. It also assigns a network ID to the object, which is used to identify the object across multiple clients (the same object will have the same network ID on every client). A prefab is a template for a game object (such as the player object). You can use the Instantiate methods to create a copy of the template in the scene. Spawned network game objects can also be destroyed via Network.Destroy. It is the multiplayer counterpart of GameObject.Destroy. It sends a message to all clients so that they all destroy the object. It also deletes any RPC messages associated with that object. NetworkView has a single component that it will serialize. This can be a Transform, a Rigidbody, an Animation, or one of your own components that has an OnSerializeNetworkView function. Serialized values can either be sent with the ReliableDeltaCompressed option, where values are always sent reliably and compressed to include only changes since the last update, or they can be sent with the Unreliable option, where values are not sent reliably and always include the full values (not the change since the last update, since that would be impossible to predict over UDP). Each method has its own advantages and disadvantages. If data is constantly changing, such as player position in a first person shooter, in general Unreliable is preferred to reduce latency. If data does not often change, use the ReliableDeltaCompressed option to reduce bandwidth (as only changes will be serialized). NetworkView can also call methods across the network via Remote Procedure Calls ( RPC ). RPCs are always completely reliable in Unity Networking, although some networking libraries allow you to send unreliable RPCs, such as uLink or TNet. Writing a custom state serializer While initially a game might simply serialize Transform or Rigidbody for testing, eventually it is often necessary to write a custom serialization function. This is a surprisingly easy task. Here is a script that sends an object's position over the network: using UnityEngine; using System.Collections; public class ExampleUnityNetworkSerializePosition : MonoBehaviour { public void OnSerializeNetworkView( BitStream stream, NetworkMessageInfo info ) { // we are currently writing information to the network if( stream.isWriting ) { // send the object's position Vector3 position = transform.position; stream.Serialize( ref position ); } // we are currently reading information from the network else { // read the first vector3 and store it in 'position' Vector3 position = Vector3.zero; stream.Serialize( ref position ); // set the object's position to the value we were sent transform.position = position; } } } Most of the work is done with BitStream. This is used to check if NetworkView is currently writing the state, or if it is reading the state from the network. Depending on whether it is reading or writing, stream.Serialize behaves differently. If NetworkView is writing, the value will be sent over the network. However, if NetworkView is reading, the value will be read from the network and saved in the referenced variable (thus the ref keyword, which passes Vector3 by reference rather than value). Using RPCs RPCs are useful for single, self-contained messages that need to be sent, such as a character firing a gun, or a player saying something in chat. In Unity, RPCs are methods marked with the [RPC] attribute. This can be called by name via networkView.RPC( "methodName", … ). For example, the following script prints to the console on all machines when the space key is pressed. using UnityEngine; using System.Collections; public class ExampleUnityNetworkCallRPC : MonoBehavior { void Update() { // important – make sure not to run if this networkView is notours if( !networkView.isMine ) return; // if space key is pressed, call RPC for everybody if( Input.GetKeyDown( KeyCode.Space ) ) networkView.RPC( "testRPC", RPCMode.All ); } [RPC] void testRPC( NetworkMessageInfo info ) { // log the IP address of the machine that called this RPC Debug.Log( "Test RPC called from " + info.sender.ipAddress ); } } Also note the use of NetworkView.isMine to determine ownership of an object. All scripts will run 100 percent of the time regardless of whether your machine owns the object or not, so you have to be careful to avoid letting some logic run on remote machines; for example, player input code should only run on the machine that owns the object. RPCs can either be sent to a number of players at once, or to a specific player. You can either pass an RPCMode to specify which group of players to receive the message, or a specific NetworkPlayer to send the message to. You can also specify any number of parameters to be passed to the RPC method. RPCMode includes the following entries: All (the RPC is called for everyone) AllBuffered (the RPC is called for everyone, and then buffered for when new players connect, until the object is destroyed) Others (the RPC is called for everyone except the sender) OthersBuffered (the RPC is called for everyone except the sender, and then buffered for when new players connect, until the object is destroyed) Server (the RPC is sent to the host machine) Initializing a server The first thing you will want to set up is hosting games and joining games. To initialize a server on the local machine, call Network.InitializeServer. This method takes three parameters: the number of allowed incoming connections, the port to listen on, and whether to use NAT punch-through. The following script initializes a server on port 25000 which allows 8 clients to connect: using UnityEngine; using System.Collections; public class ExampleUnityNetworkInitializeServer : MonoBehavior { void OnGUI() { if( GUILayout.Button( "Launch Server" ) ) { LaunchServer(); } } // launch the server void LaunchServer() { // Start a server that enables NAT punchthrough, // listens on port 25000, // and allows 8 clients to connect Network.InitializeServer( 8, 25005, true ); } // called when the server has been initialized void OnServerInitialized() { Debug.Log( "Server initialized" ); } } You can also optionally enable an incoming password (useful for private games) by setting Network.incomingPassword to a password string of the player's choice, and initializing a general-purpose security layer by calling Network.InitializeSecurity(). Both of these should be set up before actually initializing the server. Connecting to a server To connect to a server you know the IP address of, you can call Network.Connect. The following script allows the player to enter an IP, a port, and an optional password and attempts to connect to the server: using UnityEngine; using System.Collections; public class ExampleUnityNetworkingConnectToServer : MonoBehavior { private string ip = ""; private string port = ""; private string password = ""; void OnGUI() { GUILayout.Label( "IP Address" ); ip = GUILayout.TextField( ip, GUILayout.Width( 200f ) ); GUILayout.Label( "Port" ); port = GUILayout.TextField( port, GUILayout.Width( 50f ) ); GUILayout.Label( "Password (optional)" ); password = GUILayout.PasswordField( password, '*',GUILayout.Width( 200f ) ); if( GUILayout.Button( "Connect" ) ) { int portNum = 25005; // failed to parse port number – a more ideal solution is tolimit input to numbers only, a number of examples can befound on the Unity forums if( !int.TryParse( port, out portNum ) ) { Debug.LogWarning( "Given port is not a number" ); } // try to initiate a direct connection to the server else { Network.Connect( ip, portNum, password ); } } } void OnConnectedToServer() { Debug.Log( "Connected to server!" ); } void OnFailedToConnect( NetworkConnectionError error ) { Debug.Log( "Failed to connect to server: " +error.ToString() ); } } Connecting to the Master Server While we could just allow the player to enter IP addresses to connect to servers (and many games do, such as Minecraft), it's much more convenient to allow the player to browse a list of public servers. This is what the Master Server is for. Now that you can start up a server and connect to it, let's take a look at how to connect to the Master Server you downloaded earlier. First, make sure both the Master Server and Facilitator are running. I will assume you are running them on your local machine (IP is 127.0.0.1), but of course you can run these on a different computer and use that machine's IP address. Keep in mind, if you want the Master Server publicly accessible, it must be installed on a machine with a public IP address (it cannot be in a private network). Let's configure Unity to use our Master Server rather than the Unity-hosted test server. The following script configures the Master Server and Facilitator to connect to a given IP (by default 127.0.0.1): using UnityEngine; using System.Collections; public class ExampleUnityNetworkingConnectToMasterServer : MonoBehaviour { // Assuming Master Server and Facilitator are on the same machine public string MasterServerIP = "127.0.0.1"; void Awake() { // set the IP and port of the Master Server to connect to MasterServer.ipAddress = MasterServerIP; MasterServer.port = 23466; // set the IP and port of the Facilitator to connect to Network.natFacilitatorIP = MasterServerIP; Network.natFacilitatorPort = 50005; } }

(For more resources related to this topic, see here.) Styling components versus themes Before we get into this article, it's important to have a good understanding of the difference between styling an individual component and creating a theme. Almost every display component in Sencha Touch has the option to set its own style. For example, a panel component can use a style in this way: { xtype: 'panel', style: 'border: none; font: 12px Arial black', html: 'Hello World' } The style can also be set as an object using: { xtype: 'panel', style : { 'border' : 'none', 'font' : '12px Arial black', 'border-left': '1px solid black' } html: 'Hello World' } You will notice that inside the style block, we have quoted both sides of the configuration setting. This is still the correct syntax for JavaScript and a very good habit to get in to for using style blocks. This is because a number of standard CSS styles use a dash as part of their name. If we do not add quotes to border-left, JavaScript will read this as border minus left and promptly collapse in a pile of errors. We can also set a style class for a component and use an external CSS file to define the class as follows: { xtype: 'panel', cls: 'myStyle', html: 'Hello World' } Your external CSS file could then control the style of the component in the following manner: .myStyle { border: none; font: 12px Arial black; } This class-based control of display is considered a best practice as it separates the style logic from the display logic. This means that when you need to change a border color, it can be done in one file instead of hunting through multiple files for individual style settings. These styling options are very useful for controlling the display of individual components. There are also certain style elements, such as border, padding, and margin, that can be set directly in the components' configuration: { xtype: 'panel', bodyMargin: '10 5 5 5', bodyBorder: '1px solid black', bodyPadding: 5, html: 'Hello World' } These configurations can accept either a number to be applied to all sides or a CSS string value, such as 1px solid black or 10 5 5 5. The number should be entered without quotes but the CSS string values need to be within quotes. These kind of small changes can be helpful in styling your application, but what if you need to do something a bit bigger? What if you want to change the color or appearance of the entire application? What if you want to create your own default style for your buttons? This is where themes and UI styles come into play. UI styling for toolbars and buttons Let's do a quick review of the basic MVC application we created, Creating a Simple Application, and use it to start our exploration of styles with toolbars and buttons. To begin, we are going to add a few things to the first panel, which has our titlebar, toolbar, and Hello World text. Adding the toolbar In app/views, you'll find Main.js. Go ahead and open that in your editor and takea look at the first panel in our items list: items: [ { title: 'Hello', iconCls: 'home', xtype: 'panel', html: 'Hello World', items: [ { xtype: 'titlebar', docked: 'top', title: 'About TouchStart' } ] }... We're going to add a second toolbar on top of the existing one. Locate the items section, and after the curly braces for our first toolbar, add the second toolbar in the following manner: { xtype: 'titlebar', docked: 'top', title: 'About TouchStart' }, { docked: 'top', xtype: 'toolbar', items: [ {text: 'My Button'} ]} Don't forget to add a comma between the two toolbars. Extra or missing commas While working in Sencha Touch, one of the most common causes of parse errors is an extra or missing comma. When you are moving the code around, always make sure you have accounted for any stray or missing commas. Fortunately for us, the Safari Error Console will usually give us a pretty good idea about the line number to look at for these types of parse errors. A more detailed list of common errors can be found at: http://javascript.about.com/od/reference/a/error.htm Now when you take a look at the first tab, you should see our new toolbar with our button to the left. Since the toolbars both have the same background, they are a bit difficult to differentiate. So, we are going to change the appearance of the bottom bar using the ui configuration option: { docked: 'top', xtype: 'toolbar', ui: 'light', items: [ {text: 'My Button'} ] } The ui configuration is the shorthand for a particular set of styles in Sencha Touch. There are several ui styles included with Sencha Touch, and later on, we will show you how to make your own. Styling buttons Buttons can also use the ui configuration setting, for which they offer several different options: normal: This is the default button back: This is a button with the left side narrowed to a point round: This is a more drastically rounded button small: This is a smaller button action: This is a brighter version of the default button (the color varies according to the active color of the theme, which we will see later) forward: This is a button with the right side narrowed to a point Buttons also have some color options built into the ui option. These color options are confirm and decline. These options are combined with the previous shape options using a hyphen; for example, confirm-small or decline-round. Let's add some new buttons and see how this looks on our screen. Locate the items list with our button in the second toolbar: items: [ {text: 'My Button'} ] Replace that old items list with the following new items list: items: [ { text: 'Back', ui: 'back' }, { text: 'Round', ui: 'round' }, { text: 'Small', ui: 'small' }, { text: 'Normal', ui: 'normal' }, { text: 'Action', ui: 'action' }, { text: 'Forward', ui: 'forward' } ] This will produce a series of buttons across the top of our toolbar. As you may notice, all of our buttons are aligned to the left. You can move buttons to the right by adding a spacer xtype in front of the buttons you want pushed to the right. Try this by adding the following between our Forward and Action buttons: { xtype: 'spacer'}, This will make the Forward button move over to the right-hand side of the toolbar: Since buttons can actually be used anywhere, we can add some to our title bar and use the align property to control where they appear. Modify the titlebar for our first panel and add an items section, as shown in the following code: { xtype: 'titlebar', docked: 'top', title: 'About TouchStart', items: [ { xtype: 'button', text: 'Left', align: 'left' }, { xtype: 'button', text: 'Right', align: 'right' } ] } Now we should have two buttons in our title bar, one on either side of the title: Let's also add some buttons to the panel container so we can see what the ui options confirm and decline look like. Locate the end of the items section of our HelloPanel container and add the following after the second toolbar: { xtype: 'button', text: 'Confirm', ui: 'confirm', width: 100 }, { xtype: 'button', text: 'Decline', ui: 'decline', width: 100 } There are two things you may notice that differentiate our panel buttons from our toolbar buttons. The first is that we declare xtype:'button' in our panel but we don't in our toolbar. This is because the toolbar assumes it will contain buttons and xtype only has to be declared if you use something other than a button. The panel does not set a default xtype attribute, so every item in the panel must declare one. The second difference is that we declare width for the buttons. If we don't declare width when we use a button in a panel, it will expand to the full width of the panel. On the toolbar, the button auto-sizes itself to fit the text. You will also see that our two buttons in the panel are mashed together. You can separate them out by adding margin: 5 to each of the button configuration sections. These simple styling options can help make your application easier to navigate and provide the user with visual clues for important or potentially destructive actions. The tab bar The tab bar at the bottom also understands the ui configuration option. In this case, the available options are light and dark. The tab bar also changes the icon appearance based on the ui option; a light toolbar will have dark icons and a dark toolbar will have light icons. These icons are actually part of a special font called Pictos. Sencha Touch started using the Pictos font in Version 2.2 instead of images icons because of compatibility issues on some mobile devices. The icon mask from previous versions of Sencha Touch is available but has been discontinued as of Version 2.2. You can see some of the icons available in the documentation for the Ext.Button component: http://docs.sencha.com/touch/2.2.0/#!/api/Ext.Button If you're curious about the Pictos font, you can learn more about it at http://pictos.cc/

(For more resources related to this topic, see here.) List of data sources Here, the wide varieties of data sources that are supported in the PowerPivot interface are given in brief. The vital part is to install providers such as OLE DB and ODBC that support the existing data source, because when installing the PowerPivot add-in, it will not install the provider too and some providers might already be installed with other applications. For instance, if there is a SQL Server installed in the system, the OLE DB provider will be installed with the SQL Server, so that later on it wouldn't be necessary to install OLE DB while adding data sources by using the SQL Server as a data source. Hence, make sure to verify the provider before it is added as a data source. Perhaps the provider is required only for relational database data sources. Relational database By using RDBMS, you can import tables and views into the PowerPivot workbook. The following is a list of various data sources: Microsoft SQL Server Microsoft SQL Azure Microsoft SQL Server Parallel Data Warehouse Microsoft Access Oracle Teradata Sybase Informix IBM DB2 Other providers (OLE DB / ODBC) Multidimensional sources Multidimensional data sources can only be added from Microsoft Analysis Services (SSAS). Data feeds The three types of data feeds are as follows: SQL Server Reporting Service (SSRS) Azure DataMarket dataset Other feeds such as atom service documents and single data feed Text files The two types of text files are given as follows: Excel file Text file Purpose of importing data from a variety of sources In order to make a decision about a particular subject area, you should analyze all the required data that is relevant to the subject area. If the data is stored in a variety of data sources, importing the data from different data sources has to be done. If all the data is only in one data source, only the data needs to be imported from the required tables for the subject and then various types of analysis can be done. The reason why users need to import data from different data sources is that they would then have an ample amount of data when they need to make any analysis. Another generic reason would be to cross-analyze data from different business systems such as Customer Relationship Management (CRM) and Campaign Management System (CMS). Data sourcing from only one source wouldn't be as sophisticated as an analysis done from different data sources as the amount of data from which the analysis was done for multisourced data is more detailed than the data from only a single source. It also might reveal conflicts between multiple data sources. In real time, usually in the e-commerce industry, blogs, and forum websites wouldn't ask for more details about customers at the time of registration, because the time consumed for long registrations would discourage the user, leading to cancellation the of the order. For instance, the customer table that would be stored in the database of an e-commerce industry would contain the following attributes: Customer FirstName LastName E-mail BirthDate Zip Code Gender However, this kind of industry needs to know their customers more in order to increase their sales. Since the industry only saves a few attributes about the customer during registration, it is difficult to track down the customers and it is even more difficult to advertise according to individual customers. Therefore, in order to find some relevant data about the customers, the e-commerce industries try to make another data source using the Internet or other types of sources. For instance, by using the Postcode given by the customer during registration, the user can get Country | State | City from various websites and then use the information obtained to make a table format either in Excel or CSV, as follows: Location Postcode City State Country So, finally the user would have two sources—one source is from the user's RDBMS database and the other source is from the Excel file created later—both of these can be used in order to make a customer analysis based on their location. General overview of ETL The main goal is to facilitate the development of data migration applications by applying data transformations. Extraction Transform Load (ETL) comprises of the first step in a data warehouse process. It is the most important and the hardest part, because it determines the quality of the data warehouse and the scope of the analyses the users will be able to build upon it. Let us discuss in detail what ETL is. The first substep is extraction. As the users would want to regroup all the information a company has, they will have to collect the data from various heterogeneous sources (operational data such as databases and/or external data such as CSV files) and various models (relational, geographical, web pages, and so on). The difficulty starts here. As data sources are heterogeneous, formats are usually different, and the users would have different types of data models for similar information (different names and/or formats) and different keys for the same objects. These are some of the main problems. The aim of the transformation task is to correct such problems, as much as possible. Let us now see what a transformation task is. The users would need to transform heterogeneous data of different sources into homogeneous data. Here are some examples of what they can do: Extraction of the data sources Identification of relevant data sources Filtering of non-admissible data Modification of formats or values Summary This article shows the users how to prepare data for analysis. We also covered different types of data sources that can be imported into the PowerPivot interface. Some information about the provider and a general overview of the ETL process has also been given. Users now know how the ETL process works in the PowerPivot interface; also, users were shown an introduction and the advantages of DAX. Resources for Article: Further resources on this subject: Creating a pivot table [Article] What are SSAS 2012 dimensions and cube? [Article] An overview of Oracle Hyperion Interactive Reporting [Article]

There might be times rather than just plain text, so you may wish to include images, text formatting, and URLs in the body of your e-mail. HTML e-mails will allow you to do this and CodeIgniter Email library can easily be set to do just that. How to do it... HTML e-mails can be sent by executing the following steps: Create a file email.php at /path/to/codeigniter/application/controllers/. Add the following code to the controller file email.php: <?php if ( ! defined('BASEPATH')) exit('No direct script access allowed'); class Email extends CI_Controller { function __construct() { parent::__construct(); $this->load->helper('url'); $this->load->library('email'); } public function index() { redirect('email/send_email'); } public function send_email() { $config['protocol'] = 'sendmail'; $config['mailpath'] = '/usr/sbin/sendmail'; $config['charset'] = 'iso-8859-1'; $config['wordwrap'] = TRUE; $config['mailtype'] = 'html'; $this->email->initialize($config); $this->email->from('from@domain.com', 'Your Name'); $this->email->to('to@domain.com'); $this->email->subject('This is a html email'); $html = 'This is an <b>HTML</b> email'; $this->email->message($html); $this->email->send(); echo $this->email->print_debugger(); } } How it works... In the constructor controller we load the Email library (highlighted in the following code), which provides support for us to send e-mails: function __construct() { parent::__construct(); $this->load->helper('url'); $this->load->library('email'); } Next, public function index() redirects us to the function public function send_mail(), which sets some initial configuration variables for CodeIgniter Email library to work with, such as the system used to send the e-mail (in this case, sendmail), the path to send e-mail on your system, the mailtype variable (text or HTML), and so on. Take a look at the following line of code: $config['mailtype'] = 'html'; Here, we're telling CodeIgniter to send the e-mail as HTML rather than as text. These configuration settings are initialized (that is, passed to the Email library) and we begin to build the e-mail by setting the to, from, subject, and message attributes: $this->email->from('from@domain.com', 'Your Name'); $this->email->to('to@domain.com'); $this->email->subject('This is a text email'); $this->email->message('And this is some content for the text email.'); Then, send the e-mail using the following code: $this->email->send(); If all works out as planned, you should see an output similar to the following code: Your message has been successfully sent using the following protocol: sendmail User-Agent: CodeIgniter Date: Fri, 4 Oct 2013 08:56:59 +0200 From: "Your Name" <from@domain.com> Return-Path: <from@domain.com> To: to@domain.com Subject: =?iso-8859-1?Q?This_is_a_html_email?= Reply-To: "from@domain.com" <from@domain.com> X-Sender: from@domain.com X-Mailer: CodeIgniter X-Priority: 3 (Normal) Message-ID: <524e66bbf282f@domain.com> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="B_ALT_524e66bbf2868" This is a multi-part message in MIME format. Your email application may not support this format. --B_ALT_524e66bbf2868 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit This is an HTML email --B_ALT_524e66bbf2868 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable This is an <b>HTML</b> email --B_ALT_524e66bbf2868-- Sending attachments with CodeIgniter Email There might be times when you wish to send an attachment along with the e-mail, such as an invoice to a customer for a recent purchase or perhaps an image. The CodeIgniter Email library can easily be set to do just that. How to do it... You can send attachments with CodeIgniter Email by executing the following steps: Create a file email.php at /path/to/codeigniter/application/controllers/. Add the following code to the controller file, email.php: <?php if ( ! defined('BASEPATH')) exit('No direct script access allowed'); class Email extends CI_Controller { function __construct() { parent::__construct(); $this->load->helper('url'); $this->load->library('email'); } public function index() { redirect('email/send_email'); } public function send_email() { $config['protocol'] = 'sendmail'; $config['mailpath'] = '/usr/sbin/sendmail'; $config['charset'] = 'iso-8859-1'; $config['wordwrap'] = TRUE; $config['mailtype'] = 'html'; $this->email->initialize($config); $this->email->from('from@domain.com', 'Your Name'); $this->email->to('to@domain.com'); $this->email->subject('This is a html email'); $html = 'This is an <b>HTML</b> email with an attachment, <i>lovely!</i>'; $this->email->message($html); $this->email->attach('/path/to/attachment'); $this->email->send(); echo $this->email->print_debugger(); } } How it works... In the constructor controller we load the Email library (highlighted in the following code), which provides support to send e-mails: function __construct() { parent::__construct(); $this->load->helper('url'); $this->load->library('email'); } Next, public function index() redirects us to the function, public function send_mail(), which sets some initial configuration variables for the CodeIgniter Email library to work with, such as the system used to send the e-mail (in this case, sendmail), the path to send mail on your system, the mailtype variable (text or HTML), and so on. These configuration settings are initialized (that is, passed to the Email library) and we begin to build the e-mail; setting the to, from, subject, and message attributes, as well as the path to the attachment we're sending in the e-mail (highlighted in the following code): $this->email->from('from@domain.com', 'Your Name'); $this->email->to('to@domain.com'); $this->email->subject('This is a html email'); $html = 'This is an <b>HTML</b> email with an attachment, <i>lovely!</i>'; $this->email->message($html); $this->email->attach('/path/to/attachment'); Then, send the e-mail using the following code: $this->email->send(); Sending bulk e-mails with CodeIgniter Email There may be times when you wish to send out bulk e-mails; perhaps to all the people who have paid to go on a tour. You may wish to send them each a personalized e-mail, and also add an attachment. You may also want to pull their e-mail preference (plain text or HTML) from the account on your database and send them the correct format of e-mail. That's what we're going to do here. Getting ready We need to know each person's preferences such as whether they want HTML e-mails or text, and also their individual reference number (or booking ID) for their trip. As per this requirement, we are going to have a database to hold all the information; so copy the following code into your database: CREATE TABLE `bookers` ( `id` int(11) NOT NULL AUTO_INCREMENT, `firstname` varchar(50) NOT NULL, `lastname` varchar(50) NOT NULL, `email` varchar(255) NOT NULL, `email_pref` varchar(4) NOT NULL, `booking_ref` varchar(10) NOT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=3 ; INSERT INTO `bookers` (`id`, `firstname`, `lastname`, `email`, `email_pref`, `booking_ref`) VALUES (1, 'Robert', 'Foster', 'example1@domain1.com', 'html', 'ABC123'), (2, 'Lucy', 'Welsh', 'example2@domain2.com', 'html', 'DEF456'); How to do it... Create a file email.php at /path/to/codeigniter/application/controllers/. Add the following code to the controller file, email.php: <?php if ( ! defined('BASEPATH')) exit('No direct script access allowed'); class Email extends CI_Controller { function __construct() { parent::__construct(); $this->load->helper('url'); $this->load->library('email'); } public function index() { redirect('email/send_email'); } public function send_email() { $config['protocol'] = 'sendmail'; $config['mailpath'] = '/usr/sbin/sendmail'; $config['charset'] = 'iso-8859-1'; $config['wordwrap'] = TRUE; $query = "SELECT * FROM bookers "; $result = $this->db->query($query); foreach ($result->result() as $row) { $this->email->clear(); if ($row->email_pref == 'text') { $config['mailtype'] = 'text'; $body = 'Hi ' . $row->firstname . ', Thanks you for booking with us, please find attached the itinerary for your trip. This is your booking reference number: ' . $row->booking_ref . ' Thanks for booking with us, have a lovely trip.'; } else { $config['mailtype'] = 'html'; $body = 'Hi ' . $row->firstname . ',<br /><br />Thanks you for booking with us, please find attached the itinerary for your trip. </p>This is your booking reference number: <b>' . $row->booking_ref . '</b><br /><br />Thanks for booking with us, have a lovely trip.'; } $this->email->initialize($config); $this->email->to($row->email); $this->email->from('bookings@thecodeigniterholidaycompany.com'); $this->email->subject('Holiday booking details'); $this->email->message($body); $this->email->send(); } echo $this->email->print_debugger(); } } How it works... In the constructor controller we load the Email library (highlighted in the following code), which provides support for us to send e-mails: function __construct() { parent::__construct(); $this->load->helper('url'); $this->load->library('email'); } Next, public function index() redirects us to the function, public function send_mail(), which sets some initial configuration variables for CodeIgniter Email library to work with, such as the system used to send the e-mail (in this case, sendmail), the path to send mail from your system. We then query the database for each of the customer's booking details: $query = "SELECT * FROM bookers "; $result = $this->db->query($query); foreach ($result->result() as $row) { } The query will loop through each result and send a specific e-mail based on the values retrieved from the database in each loop. Firstly, we give ourselves a clean slate by clearing all the settings and variables from a previous loop iteration by using the CodeIgniter email function: $this->email->clear(); We then look at their e-mail preference and set the e-mail sending (mailtype) variable accordingly, along with the text for the body of the e-mails. So, if someone prefers HTML, we look for that preference and define the body of the HTML e-mail, otherwise for a text e-mail, we look for the text e-mail preference and define the body for the text e-mail: if ($row->email_pref == 'text') { $config['mailtype'] = 'text'; $body = 'Hi ' . $row->firstname . ', Thank you for booking with us, please find attached the itinerary for your trip. This is your booking reference number: ' . $row->booking_ref . ' Thanks for booking with us, have a lovely trip.'; } else { $config['mailtype'] = 'html'; $body = 'Hi ' . $row->firstname . ',<br /><br />Thank you for booking with us, please find attached the itinerary for your trip. </p>This is your booking reference number: <b>' . $row->booking_ref . '</b><br /><br />Thanks for booking with us, have a lovely trip.'; } After this, we initialize the configuration variables. Those of you who have looked at the previous few recipes will notice that the initialization takes place later in the code of this recipe than in others. This is because we cannot initialize the config variables earlier as some of the variables rely on the preferences of individual customers, which are fetched from the database. So, we have to wait until each user's details are fetched from a database to initialize each iteration of the configuration settings. And finally, we send the e-mail: $this->email->send(); If all goes well, you should see an output similar to the following: Your message has been successfully sent using the following protocol: sendmail User-Agent: CodeIgniter Date: Fri, 4 Oct 2013 20:06:13 +0200 To: to@domain.com From: <bookings@thecodeigniterholidaycompany.com> Return-Path: <bookings@thecodeigniterholidaycompany.com> Subject: =?iso-8859-1?Q?Holiday_booking_details?= Reply-To: "bookings@thecodeigniterholidaycompany.com" <bookings@thecodeigniterholidaycompany.com> X-Sender: bookings@thecodeigniterholidaycompany.com X-Mailer: CodeIgniter X-Priority: 3 (Normal) Message-ID: <524f0395942a2@thecodeigniterholidaycompany.com> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="B_ALT_524f0395942bb" This is a multi-part message in MIME format. Your email application may not support this format. --B_ALT_524f0395942bb Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Hi RobertThanks you booking with us, please find attached the itinerary for your trip. This is your booking reference number: ABC123 Thanks for booking with us, have a lovely trip. --B_ALT_524f0395942bb Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Hi Robert<br /><br />Thanks you booking with us,=20 please find attached the itinerary for your trip. </p>This is your booking reference number:=20 <b>ABC123</b><br /><br /> Thanks for booking with us, have a lovely trip. --B_ALT_524f0395942bb-- Using an HTML table with DataTable DataTable is a free to use library that turns your normal looking html table into an interactive marvel with sortable and searchable columns and a whole lot more; we're going to use it with CodeIgniter, merging DataTable and CodeIgniter table functionality. It's simple to use and is able to handle most of the things you will need it for. Here, in this recipe, we're going to use it with DataTable to create an interactive HTML table that is sortable and searchable. It has pagination too! If you want database results, move on to the next recipe, Using an HTML table with DataTable and a database, where we'll look at populating a table from a database query. Getting ready For this recipe, you will need to follow the given procedure: Ensure that you've downloaded DataTable from the following link: https://datatables.net/download/ Unzip the downloaded .zip file, and move the files to a location on your web server or localhost, which will be accessible by CodeIgniter. For this recipe, I have put the folder at application/views; but you can make your own choice if you wish. How to do it... Create four files as given: /path/to/codeigniter/application/controllers/table.php /path/to/codeigniter/application/views/table_header.php /path/to/codeigniter/application/views/table_body.php /path/to/codeigniter/application/views/table_footer.php Create the controller file, table.php, and add the following code to it: <?php if ( ! defined('BASEPATH')) exit('No direct script access allowed'); class Table extends CI_Controller { function __construct() { parent::__construct(); $this->load->library('table'); } public function index() { $tmpl = array ( 'table_open' => '<table border="0" cellpadding="4" cellspacing="0" id="example">', 'heading_row_start' => '<tr>', 'heading_row_end' => '</tr>', 'heading_cell_start' => '<th>', 'heading_cell_end' => '</th>', 'row_start' => '<tr>', 'row_end' => '</tr>', 'cell_start' => '<td>', 'cell_end' => '</td>', 'row_alt_start' => '<tr>', 'row_alt_end' => '</tr>', 'cell_alt_start' => '<td>', 'cell_alt_end' => '</td>', 'table_close' => '</table>' ); $this->table->set_template($tmpl); $this->table->set_heading(array('ID', 'First Name', 'Last Name')); $this->table->add_row(array('1', 'Rob', 'Foster')); $this->table->add_row(array('2', 'Lucy', 'Welsh')); $this->table->add_row(array('3', 'George', 'Foster')); $this->table->add_row(array('4', 'Jackie', 'Foster')); $this->table->add_row(array('5', 'Antony', 'Welsh')); $this->table->add_row(array('6', 'Rowena', 'Welsh')); $this->table->add_row(array('7', 'Peter', 'Foster')); $this->table->add_row(array('8', 'Jenny', 'Foster')); $this->table->add_row(array('9', 'Oliver', 'Welsh')); $this->table->add_row(array('10', 'Felicity', 'Foster')); $this->table->add_row(array('11', 'Harrison', 'Foster')); $this->table->add_row(array('12', 'Mia', 'The Cat')); $data['table'] = $this->table->generate(); $this->load->view('tables/table_header'); $this->load->view('tables/table_body',$data); $this->load->view('tables/table_footer'); } } Create the view file, table_header.php, and add the following code to it: <html> <head> <style type="text/css" title="currentStyle"> @import "<?php echo $this->config->item('base_url') ; ?>application/views/DataTables-1.9.4/media/css/demo_page.css"; @import "<?php echo $this->config->item('base_url') ; ?>application/views/DataTables-1.9.4/media/css/jquery.dataTables.css"; </style> <script type="text/javascript" language="javascript" src="<?php echo $this->config->item('base_url') ; ?>application/views/DataTables-1.9.4/media/js/jquery.js"></script> <script type="text/javascript" language="javascript" src="<?php echo $this->config->item('base_url') ; ?>application/views/DataTables-1.9.4/media/js/jquery.dataTables.js"></script> <script type="text/javascript" charset="utf-8"> $(document).ready(function() { $('#example').dataTable(); } ); </script> </head> <body> Take a look at the <script> tag: <script type="text/javascript" charset="utf-8"> $(document).ready(function() { $('#example').dataTable(); } ); </script> The #example parameter is the ID of the table (detailed in the following How it works... section). Ensure that the value example in <script> and table markup is the same. Create the view file, table_body.php, and add the following code to it: <?php echo $table ; ?> Create the controller file, table_footer.php, and add the following code to it:   </body> </html> How it works... The constructor in the table controller loads the CodeIgniter's Table library: function __construct() { parent::__construct(); $this->load->library('table'); } The public function index() function is called. We then define how we want our HTML table markup to appear. This is where you can place any markup for the specific CSS, using which you can style the elements of the table: $tmpl = array ( 'table_open' => '<table border="0" cellpadding="4" cellspacing="0" id="example">', 'heading_row_start' => '<tr>', 'heading_row_end' => '</tr>', 'heading_cell_start' => '<th>', 'heading_cell_end' => '</th>', 'row_start' => '<tr>', 'row_end' => '</tr>', 'cell_start' => '<td>', 'cell_end' => '</td>', 'row_alt_start' => '<tr>', 'row_alt_end' => '</tr>', 'cell_alt_start' => '<td>', 'cell_alt_end' => '</td>', 'table_close' => '</table>' ); Take a closer look at the table_open element of the $tmpl array. Look for the item highlighted in the preceding code. The id="example" item is used by DataTable (in the <script> tag of the file table_header.php) to apply its CSS and functionality. You can, of course, name it anything you like, but be sure to reflect that change in the JavaScript. We then call $this->table->set_template() to apply the HTML table markup: $this->table->set_template($tmpl); We then set the table headers and apply the data for our rows. Ensure that the number of items in the table headers is the same as the number of items in the table data: $this->table->set_heading(array('ID', 'First Name', 'Last Name')); $this->table->add_row(array('1', 'Rob', 'Foster')); $this->table->add_row(array('2', 'Lucy', 'Welsh')); $this->table->add_row(array('3', 'George', 'Foster')); $this->table->add_row(array('4', 'Jackie', 'Foster')); $this->table->add_row(array('5', 'Antony', 'Welsh')); $this->table->add_row(array('6', 'Rowena', 'Welsh')); $this->table->add_row(array('7', 'Peter', 'Foster')); $this->table->add_row(array('8', 'Jenny', 'Foster')); $this->table->add_row(array('9', 'Oliver', 'Welsh')); $this->table->add_row(array('10', 'Felicity', 'Foster')); $this->table->add_row(array('11', 'Harrison', 'Foster')); $this->table->add_row(array('12', 'Mia', 'The Cat')); We then generate the table. The $this->table->generate() function will return a string of HTML, which we save in $data['table']. $data['table'] = $this->table->generate(); The $data array our view files for rendering to the browser: $this->load->view('tables/table_header'); $this->load->view('tables/table_body',$data); $this->load->view('tables/table_footer');

(For more resources related to this topic, see here.) DSC approaches declarative syntax that describes what needs to be done rather than covering imperative syntax that specifies how a task can be performed. Getting ready To use DSC, first define a desired configuration. Like functions, configurations in DSC can be defined in the Windows PowerShell language by using the Configuration keyword and stored in script (.ps1) or module (.psm1) files. Also, similar to functions, configurations need to be defined and then run. How to do it... To use a configuration, invoke the Configuration block the same way you would invoke a Windows PowerShell function, passing in any expected parameters you have defined (two in the preceding sample). For example, in this case, the MyWebConfig configuration can be invoked as follows: PS C :>MyWebConf -MachineName $env:COMPUTERNAME –WebsitePath \PSShareMyWebSites This will create a folder with the same name as your configuration name and will contain our MOF output file. The following command creates an MOF file known as the configuration instance document. Path represents the target directory where your MOF files are located. Wait causes the execution of the DSC resources to run in the background, that is, an interactive process. PS C :>Start-DscConfiguration -Path . MyWebConf –Wait –Verbose How it works... Each Configuration block must have at least one Node block. Each Node block can have one or more resource provider blocks. You can use the same role provider more than once in the same Node block. In addition to new language keywords, DSC includes the following set of CMDLETs for managing configurations: Start-DscConfiguration: This CMDLET deploys a configuration to one or more target nodes and applies the configuration on those nodes by using the local configuration manager Get-DscConfiguration: This CMDLET returns the current configuration from one or more target machines: PS C :>$Sess = New-CimSession -ComputerName localhost PS C :>Get-DscConfiguration –CimSession $Sess Restore-DscConfiguration: This CMDLET restores the current configuration from one or more target machines: PS C :>$Sess = New-CimSession -ComputerName localhost PS C :>Restore-DscConfiguration –CimSession $Sess There's more… There is one more CMDLET that helps to detect the configuration drift: Test-DscConfiguration: This CMDLET checks for one or more target nodes and returns a Boolean value indicating whether the current desired state matches the actual state. Have a look at the following command: PS C :>$session = New-CimSession -ComputerName localhost PS C :>Test-DscConfiguration –CimSession $session This will either return True when the current and actual configuration matches or False if there's a mismatch Summary In this article, we learned about how to execute the new feature introduced with the release of Windows PowerShell v4.0—Desired State Configuration. Resources for Article: Further resources on this subject: Exchange Server 2010 Windows PowerShell: Working with Address Lists [Article] So, what is PowerShell 3.0 WMI? [Article] Inventorying Servers with PowerShell [Article]

(For more resources related to this topic, see here.) Now it's down to the real deal—the bits and bytes. In this article, we will go through different roles and services and how they can be deployed as highly available for the clients. We will also walk through some examples of how a simple site design might look like for different scenarios. First, let's take a look at roles within Configuration Manager that cannot be deployed as highly available and what options we have. It is important to note that when installing a site system role on another server Configuration Manager, the installation files are installed on the first available NTFS disk with maximum available disk space. If you wish to exclude a disk, create an empty file called no_sms_on_drive.sms on the root folder of the drive. Also, Configuration Manager uses the site server computer account to try to install the role on a remote server (so, this account needs to be added to local administrator on the server) unless we specify a domain account (which also needs to be a local admin) when we run the setup. Site Server The first and foremost important role that cannot be deployed as highly available is the Site Server role. It does not support network load balancing or other clustering features available in Windows Server. In order to make sure that you can restore the site in case of disaster, you have to use the built-in backup tasks. Endpoint Protection The Endpoint Protection role allows for management of endpoint protection agents. If this role is unavailable for the clients, they will still continue to run as before but you cannot update the policy centrally. This role is not seen as critical for Configuration Manager and cannot be deployed as highly available. If something happens to your endpoint server, just reinstall the role on another server. It is important to know that this role can only be deployed once at the top of your hierarchy; so, if we have a CAS, we would place this role there. Same is true if we have a standalone primary site with secondary sites. Asset Intelligence Synchronization Point The Asset Intelligence Synchronization Point role connects to System Center online (a cloud service from Microsoft) in order to update the asset catalog with the latest information. If this role becomes unavailable, the site will just store data locally until it becomes online and is able to synchronize again. This role cannot be deployed as highly available, and it is important to note that this role can only be installed on the CAS role (if present) or on a standalone primary site as well. Fallback Status Point The Fallback Status Point role is used to monitor client deployments and the clients that are unable to connect to their management point, or if there is something wrong with the management point. All communication between clients and this role happens over regular HTTP requests and therefore this role should be placed on a standalone server. All status messages that get sent to the Fallback Status Point role will be registered in the logfiles fspisapi.log and fspmgr.log. This role is not considered critical and cannot be deployed as highly available. It is important to note that clients can only be assigned to one fallback point and the assignment happens during client installation. Out of Band Service Point The Out of Band Service Point role is used to manage and provision clients which have Intel vPRO or Intel AMT functionality, which allows Configuration Manager to manage clients before the operating system is active. This role cannot be deployed as highly available. Secondary sites Secondary sites by default contain one distribution point and one management point, and as mentioned previously, should only be used when you need control on how data is moved up the hierarchy. A secondary site does not support more than one management point. Data for a secondary site is primarily located in the primary site database. You can recover a secondary site by reinstalling it from its primary site. Windows Intune Connector The Windows Intune Connector role is a new role with Service Pack 1. It allows for mobile management from Configuration Manager using Windows Intune. This role can only be installed once and it needs to be in the top-level site in the hierarchy. So, now that we have covered the roles that cannot be deployed as highly available, let us go forward with the roles that we can deploy as highly available. Management Point The Management Point role is used by clients for policy downloads, data reporting, and is the primary role for client-to-site communication. When a client is assigned to a site (depending on how you have defined the client installation parameters), it will download a list of all the available management points in the site and get assigned to an appropriate one. The clients will connect to another management point if one of the following happens: Client restarts Network changes 25 hours have passed The contact with its primary management point is lost It will also check if the management point is Intranet-only or is available from the Internet, and if it communicates using HTTP or HTTPS. (Remember that if the management point is set up on the Internet, only HTTPS traffic is allowed.) Now besides manual installation, clients will look for a management point within one of the following locations if something from one of the points mentioned above happens: When installing a client manually, you can append the parameters CCMSetup.exe /mp:MPFQDN SMSSITECODE=SITECODE FSP=FSPFQDN. to define which management point to contact, which site to register to, and which FSP to contact. Site information: After a client has successfully signed up with a site, it will download a list of available management points which it stores in its cache. If it does not have any other management point available, it will continue looking in other locations. Active Directory: This only applies if you have extended the schema. Configuration Manager can publish management points directly under System Management Container with the site code. (This can be disabled under Administration | Sites | Site Properties | Configure Site Components | Management Point.) Extending the schema expands the Active Directory schema with specific Configuration Manager attributes. This allows clients to locate specific Configuration Manager server roles such as Management Points, boundaries, and its site from Active Directory. This is an irreversible action, but it allows for clients to easily locate site resources. The site code and the management points that are available within the site will be stored under Active Directory Users and Computer | Domain | System | System Management. As you can see in the following screenshot, the clients will look here if there is another management point available within this container: DNS: Clients will write a DNS query for service locator point record within the current domain. All management points will appear within the DNS zone as a prefix _mssms_mp_SITECODE, with a port number, and which host it is pointing to. These DNS records can also be automatically be published by Configuration Manager and can be disabled from the same location as Active Directory publishing. The following screenshot shows the DNS zone showing management point: WINS: If your organization is using WINS, Configuration Manager can also publish site information there as well. If a client manages to find another management point within one of the locations mentioned in the preceding list, it will try to establish a connection with it. By installing multiple management points within a site, you will gain redundant communication points for the clients within a site. Since clients have multiple places to get the information, you have multiple fallback points. There is no extra configuration required for the management points to become highly available after they are installed. It is also important to note that by default the management point talks directly to the site database server, to query for data and to publish data. If we want to reduce the CPU processing against the site database, we can set up a management point against a SQL replica. This requires that we set up a site server database with the same SQL replication installed for the site replica database. When we set up this management point, we will always query the replica database for information, but since the replication happens one-way (site to replica) the management point will always need to contact the site database to write data to the site. But we can set up multiple management points to talk to the SQL replica server, this will help to offload the site-server database. Summary In this article, we went through different roles of Configuration Manager and how we can deploy them as highly available. Resources for Article: Further resources on this subject: Client-Side Endpoint Protection Tasks in Microsoft SCEP 2012 [Article] Testing Workflows for Microsoft Dynamics AX 2009 Administration [Article] Microsoft LightSwitch Application using SQL Azure Database [Article]

(For more resources related to this topic, see here.) The Spring MVC portlet The Spring MVC portlet follows the Model-View-Controller design pattern. The model refers to objects that imply business rules. Usually, each object has a corresponding table in the database. The view refers to JSP files that will be rendered into the HTML markup. The controller is a Java class that distributes user requests to different JSP files. A Spring MVC portlet usually has the following folder structure: In the previous screenshot, there are two Spring MVC portlets: leek-portlet and lettuce-portlet. You can see that the controller classes are clearly named as LeekController.java and LettuceController.java. The JSP files for the leek portlet are view/leek/leek.jsp, view/leek/edit.jsp, and view/leek/help.jsp. The definition of the leek portlet in the portlet.xml file is as follows: <portlet> <portlet-name>leek</portlet-name> <display-name>Leek</display-name> <portlet-class>org.springframework.web.portlet.DispatcherPortlet</portlet-class> <init-param> <name>contextConfigLocation</name> <value>/WEB-INF/context/leek-portlet.xml</value> </init-param> <supports> <mime-type>text/html</mime-type> <portlet-mode>view</portlet-mode> <portlet-mode>edit</portlet-mode> <portlet-mode>help</portlet-mode> </supports> ... <supported-publishing-event> <qname >x:ipc.share</qname> </supported-publishing-event> </portlet> You can see from the previous code that the portlet class for a Spring MVC portlet is the org.springframework.web.portlet.DispatcherPortlet.java class. When a Spring MVC portlet is called, this class runs. It also calls the WEB-INF/context/leek/portlet.xml file and initializes the singletons defined in that file when the leek portlet is deployed. The leek portlet supports the view, edit, and help mode. It can also fire a portlet event with ipc.share as its qualified name. Yo can use the method to import the leek and lettuce portlets (whose source code can be downloaded from the Packt site) to your Liferay IDE. Then, carry out the following steps: Deploy the leek-portlet package and wait until the leek portlet and lettuce portlet are registered by the Liferay Portal. Log in as the portal administrator and add the two Spring MVC portlets onto a portal page. Your portal page should look similar to the following screenshot: The default view of the leek portlet comes from the view/leek/leek.jsp file whose logic is defined through the following method in the com.uibook.leek.portlet.LeekController.java class: @RequestMapping public String render(Model model, SessionStatus status, RenderRequest req) { return "leek"; } This method calls the view/leek/leek.jsp file. In the default view of the leek portlet, when you click on the radio button for Snow water from last winter and then on the Get Water button, the following form will be submitted: <form action="http://localhost:8080/web/uibook/home?p_auth=wwMoBV4C&p_p_id=leek_WAR_leekportlet&p_p_lifecycle=1&p_p_state=normal&p_p_mode=view&p_p_col_id=column-2&p_p_col_pos=1&p_p_col_count=2&_leek_WAR_leekportlet_action=sprayWater" id="_leek_WAR_leekportlet_leekFm" method="post" name="_leek_WAR_leekportlet_leekFm"> This form will fire an action URL because p_p_lifecycle is equal to 1. As the action name is sprayWater in the URL, the DispatcherPortlet.java class (as specified in the portlet.xml file) calls the following method: @ActionMapping(params="action=sprayWater") public void sprayWater(ActionRequest request, ActionResponse response, SessionStatus sessionStatus) { String waterType = request.getParameter("waterSupply"); if(waterType != null){ request.setAttribute("theWaterIs", waterType); sessionStatus.setComplete(); } } This method simply gets the value for the waterSupply parameter as specified in the following code, which comes from the view/leek/leek.jsp file: <input type="radio" name="<portlet:namespace />waterSupply" value="snow water from last winter">Snow water from last winter The value is snow water from last winter, which is set as a request attribute. As the previous sprayWater(…) method does not specify a request parameter for a JSP file to be rendered, the logic goes to the default view of the leek portlet. So, the view/leek/leek.jsp file will be rendered. Here, as you can see, the two-phase logic is retained in the Spring MVC portlet, as has been explained in the Understanding a simple JSR-286 portlet section of this article. Now the theWaterIs request attribute has a value, which is snow water from last winter. So, the following code in the leek.jsp file runs and displays the Please enjoy some snow water from last winter. message, as shown in the previous screenshot: <c:if test="${not empty theWaterIs}"> <p>Please enjoy some ${theWaterIs}.</p> </c:if> In the previous screenshot, the Passing you a gift... link is rendered with the following code in the leek.jsp file: <a href="<portlet:actionURL name="shareGarden"></portlet:actionURL>">Passing you a gift ...</a> When this link is clicked, an action URL named shareGarden is fired. So, the DispatcherPortlet.java class will call the following method: @ActionMapping("shareGarden") public void pitchBallAction(SessionStatus status, ActionResponse response) { String elementType = null; Random random = new Random(System.currentTimeMillis()); int elementIndex = random.nextInt(3) + 1; switch(elementIndex) { case 1 : elementType = "sunshine"; break; ... } QName qname = new QName("http://uibook.com/events","ipc.share"); response.setEvent(qname, elementType); status.setComplete(); } This method gets a value for elementType (the type of water in our case) and sends out this elementType value to another portlet based on the ipc.share qualified name. The lettuce portlet has been defined in the portlet.xml file as follows to receive such a portlet event: <portlet> <portlet-name>lettuce</portlet-name> ... <supported-processing-event> <qname >x:ipc.share</qname> </supported-processing-event> </portlet> When the ipc.share portlet event is sent, the portal page refreshes. Because the lettuce portlet is on the same page as the leek portlet, the portlet event is received by the following method in the com.uibook.lettuce.portlet.LettuceController.java class: @EventMapping(value ="{http://uibook.com/events}ipc.share") public void receiveEvent(EventRequest request, EventResponse response, ModelMap map) { Event event = request.getEvent(); String element = (String)event.getValue(); map.put("element", element); response.setRenderParameter("element", element); } This receiveEvent(…) method receives the ipc.share portlet event, gets the value in the event (which can be sunshine, rain drops, wind, or space), and puts it in the ModelMap object with element as the key. Now, the following code in the view/lettuce/lettuce.jsp file runs: <c:choose> <c:when test="${empty element}"> <p> Please share the garden with me! </p> </c:when> <c:otherwise> <p>Thank you for the ${element}!</p> </c:otherwise> </c:choose> As the element parameter now has a value, a message similar to Thank you for the wind will show in the lettuce portlet. The wind is a gift from the leek to the lettuce portlet. In the default view of the leek portlet, there is a Some shade, please! button. This button is implemented with the following code in the view/leek/leek.jsp file: <button type="button" onclick="<portlet:namespace />loadContentThruAjax();">Some shade, please!</button> When this button is clicked, a _leek_WAR_leekportlet_loadContentThruAjax() JavaScript function will run: function <portlet:namespace />loadContentThruAjax() { ... document.getElementById("<portlet:namespace />content").innerHTML=xmlhttp.responseText; ... xmlhttp.open('GET','<portlet:resourceURL escapeXml="false" id="provideShade"/>',true); xmlhttp.send(); } This loadContentThruAjax() function is an Ajax call. It fires a resource URL whose ID is provideShade. It maps the following method in the com.uibook.leek.portlet.LeekController.java class: @ResourceMapping(value = "provideShade") public void provideShade(ResourceRequest resourceRequest, ResourceResponse resourceResponse) throws PortletException, IOException { resourceResponse.setContentType("text/html"); PrintWriter out = resourceResponse.getWriter(); StringBuilder strB = new StringBuilder(); strB.append("The banana tree will sway its leaf to cover you from the sun."); out.println(strB.toString()); out.close(); } This method simply sends the The banana tree will sway its leaf to cover you from the sun message back to the browser. The previous loadContentThruAjax() method receives this message, inserts it in the <div id="_leek_WAR_leekportlet_content"></div> element, and shows it. About the Vaadin portlet Vaadin is an open source web application development framework. It consists of a server-side API and a client-side API. Each API has a set of UI components and widgets. Vaadin has themes for controlling the appearance of a web page. Using Vaadin, you can write a web application purely in Java. A Vaadin application is like a servlet. However, unlike the servlet code, Vaadin has a large set of UI components, controls, and widgets. For example, in correspondence to the <table> HTML element, the Vaadin API has a com.vaadin.ui.Table.java class. The following is a comparison between servlet table implementation and Vaadin table implementation: Servlet Code Vaadin Code PrintWriter out = response.getWriter(); out.println("<table>n" +    "<tr>n" +    "<td>row 2, cell 1</td>n" +    "<td>row 2, cell 2</td>" +    "</tr>n" +    "</table>"); sample = new Table(); sample.setSizeFull(); sample.setSelectable(true); ... sample.setColumnHeaders(new String[] { "Country", "Code" });   Basically, if there is a label element in HTML, there is a corresponding Label.java class in Vaadin. In the sample Vaadin code, you will find the use of the com.vaadin.ui.Button.java and com.vaadin.ui.TextField.java classes. Vaadin supports portlet development based on JSR-286. Vaadin support in Liferay Portal Starting with Version 6.0, the Liferay Portal was bundled with the Vaadin Java API, themes, and a widget set described as follows: ${APP_SERVER_PORTAL_DIR}/html/VAADIN/themes/ ${APP_SERVER_PORTAL_DIR}/html/VAADIN/widgetsets/ ${APP_SERVER_PORTAL_DIR}/WEB-INF/lib/vaadin.jar A Vaadin control panel for the Liferay Portal is also available for download. It can be used to rebuild the widget set when you install new add-ons in the Liferay Portal. In the ${LPORTAL_SRC_DIR}/portal-impl/src/portal.properties file, we have the following Vaadin-related setting: vaadin.resources.path=/html vaadin.theme=liferay vaadin.widgetset=com.vaadin.portal.gwt.PortalDefaultWidgetSet In this section, we will discuss two Vaadin portlets. These two Vaadin portlets are run and tested in Liferay Portal 6.1.20 because, at the time of writing, the support for Vaadin is not available in the new Liferay Portal 6.2. It is expected that when the Generally Available (GA) version of Liferay Portal 6.2 is available, the support for Vaadin portlets in the new Liferay Portal 6.2 will be ready. Vaadin portlet for CRUD operations CRUD stands for create, read, update, and delete. We will use a peanut portlet to illustrate the organization of a Vaadin portlet. In this portlet, a user can create, read, update, and delete data. This portlet is adapted from a SimpleAddressBook portlet from a Vaadin demo. Its structure is as shown in the following screenshot: You can see that it does not have JSP files. The view, model, and controller are all incorporated in the PeanutApplication.java class. Its portlet.xml file has the following content: <portlet-class>com.vaadin.terminal.gwt.server.ApplicationPortlet2</portlet-class> <init-param> <name>application</name> <value>peanut.PeanutApplication</value> </init-param> This means that when the Liferay Portal calls the peanut portlet, the com.vaadin.terminal.gwt.server.ApplicationPortlet2.java class will run. This ApplicationPortlet2.java class will in turn call the peanut.PeanutApplication.java class, which will retrieve data from the database and generate the HTML markup. The default UI of the peanut portlet is as follows: This default UI is implemented with the following code: HorizontalSplitPanel splitPanel = new HorizontalSplitPanel(); setMainWindow(new Window("Address Book", splitPanel)); VerticalLayout left = new VerticalLayout(); left.setSizeFull(); left.addComponent(contactList); contactList.setSizeFull(); left.setExpandRatio(contactList, 1); splitPanel.addComponent(left); splitPanel.addComponent(contactEditor); splitPanel.setHeight("450"); contactEditor.setCaption("Contact details editor"); contactEditor.setSizeFull(); contactEditor.getLayout().setMargin(true); contactEditor.setImmediate(true); bottomLeftCorner.setWidth("100%"); left.addComponent(bottomLeftCorner); The previous code comes from the initLayout() method of the PeanutApplication.java class. This method is run when the portal page is first loaded. The new Window("Address Book", splitPanel) statement instantiates a window area, which is the whole portlet UI. This window is set as the main window of the portlet; every portlet has a main window. The splitPanel attribute splits the main window into two equal parts vertically; it is like the 2 Columns (50/50) page layout of Liferay. The splitPanel.addComponent(left) statement adds the contact information table to the left pane of the main window, while the splitPanel.addComponent(contactEditor) statement adds the contact details of the editor to the right pane of the main window. The left variable is a com.vaadin.ui.VerticalLayout.java object. In the left.addComponent(bottomLeftCorner) statement, the left object adds a bottomLeftCorner object to itself. The bottomLeftCorner object is a com.vaadin.ui.HorizontalLayout.java object. It takes the space across the left vertical layout under the contact information table. This bottomLeftCorner horizontal layout will house the contact-add button and the contact-remove button. The following screenshot gives you an idea of how the screen will look: When the + icon is clicked, a button click event will be fired which runs the following code: Object id = ((IndexedContainer) contactList.getContainerDataSource()).addItemAt(0); contactList.getItem(id).getItemProperty("First Name").setValue("John"); contactList.getItem(id).getItemProperty("Last Name").setValue("Doe"); This code adds an entry in the contactList object (contact information table) initializing the contact's first name to John and the last name to Doe. At the same time, the ValueChangeListener property of the contactList object is triggered and runs the following code: contactList.addListener(new Property.ValueChangeListener() { public void valueChange(ValueChangeEvent event) { Object id = contactList.getValue(); contactEditor.setItemDataSource(id == null ? null : contactList .getItem(id)); contactRemovalButton.setVisible(id != null); } }); This code populates the contactEditor variable, a com.vaadin.ui.Form.Form.java object, with John Doe's contact information and displays the Contact details editor section in the right pane of the main window. After that, an end user can enter John Doe's other contact details. The end user can also update John Doe's first and last names. If you have noticed, the last statement of the previous code snippet mentions contactRemovalButton. At this time, the John Doe entry in the contact information table is highlighted. If the end user clicks on the contact removal button, this information will be removed from both the contact information table and the contact details editor. Actually, the end user can highlight any entry in the contact information table and edit or delete it. You may have seen that during the whole process of creating, reading, updating, and deleting the contact, the portal page URL did not change and the portal page did not refresh. All the operations were performed through Ajax calls to the application server. This means that only a few database accesses happened during the whole process. This improves the site performance and reduces load on the application server. It also implies that if you develop Vaadin portlets in the Liferay Portal, you do not have to know the friendly URL configuration skill on a Liferay Portal project. In the peanut portlet, a developer cannot retrieve the logged-in user in the code, which is a weak point. In the following section, a potato portlet is implemented in such a way that a developer can retrieve the Liferay Portal information, including the logged-in user information. Summary In this article, we learned about portlets and their development. We learned ways todevelop simple JSR 286 portlets, SpringMVC portlets, and Vaadin portlets. We also learned to implement the view, edit, and help modes of a portlet. Resources for Article: Further resources on this subject: Setting up and Configuring a Liferay Portal [Article] Liferay, its Installation and setup [Article] Building your First Liferay Site [Article]

(For more resources related to this topic, see here.) According to Bjarne Stoustrup in his book The C++ Programming Language, Third Edition: Designing and implementing a general input/output facility for a programming language is notoriously difficult... An I/O facility should be easy, convenient, and safe to use; efficient and flexible; and, above all, complete. It shouldn't surprise anyone that a design team, focused on providing efficient and easy I/O, has delivered such a facility through Node. Through a symmetrical and simple interface, which handles data buffers and stream events so that the implementer does not have to, Node's Stream module is the preferred way to manage asynchronous data streams for both internal modules and, hopefully, for the modules developers will create. A stream in Node is simply a sequence of bytes. At any time, a stream contains a buffer of bytes, and this buffer has a zero or greater length: Because each character in a stream is well defined, and because every type of digital data can be expressed in bytes, any part of a stream can be redirected, or "piped", to any other stream, different chunks of the stream can be sent do different handlers, and so on. In this way stream input and output interfaces are both flexible and predictable and can be easily coupled. Digital streams are well described using the analogy of fluids, where individual bytes (drops of water) are being pushed through a pipe. In Node, streams are objects representing data flows that can be written to and read from asynchronously. The Node philosophy is a non-blocking flow, I/O is handled via streams, and so the design of the Stream API naturally duplicates this general philosophy. In fact, there is no other way of interacting with streams except in an asynchronous, evented manner—you are prevented, by design, from blocking I/O. Five distinct base classes are exposed via the abstract Stream interface: Readable, Writable, Duplex, Transform, and PassThrough. Each base class inherits from EventEmitter, which we know of as an interface to which event listeners and emitters can be bound. As we will learn, and here will emphasize, the Stream interface is an abstract interface. An abstract interface functions as a kind of blueprint or definition, describing the features that must be built into each constructed instance of a Stream object. For example, a readable stream implementation is required to implement a public read method which delegates to the interface's internal _read method. In general, all stream implementations should follow these guidelines: As long as data exists to send, write to a stream until that operation returns false, at which point the implementation should wait for a drain event, indicating that the buffered stream data has emptied Continue to call read until a null value is received, at which point wait for a readable event prior to resuming reads Several Node I/O modules are implemented as streams. Network sockets, file readers and writers, stdin and stdout, zlib, and so on. Similarly, when implementing a readable data source, or data reader, one should implement that interface as a Stream interface. It is important to note that as of Node 0.10.0 the Stream interface changed in some fundamental ways. The Node team has done its best to implement backwards-compatible interfaces, such that (most) older programs will continue to function without modification. In this article we will not spend any time discussing the specific features of this older API, focusing on the current (and future) design. The reader is encouraged to consult Node's online documentation for information on migrating older programs. Implementing readable streams Streams producing data that another process may have an interest in are normally implemented using a Readable stream. A Readable stream saves the implementer all the work of managing the read queue, handling the emitting of data events, and so on. To create a Readable stream: var stream = require('stream'); var readable = new stream.Readable({ encoding : "utf8", highWaterMark : 16000, objectMode: true }); As previously mentioned, Readable is exposed as a base class, which can be initialized through three options: encoding: Decode buffers into the specified encoding, defaulting to UTF-8. highWaterMark: Number of bytes to keep in the internal buffer before ceasing to read from the data source. The default is 16 KB. objectMode: Tell the stream to behave as a stream of objects instead of a stream of bytes, such as a stream of JSON objects instead of the bytes in a file. Default false. In the following example we create a mock Feed object whose instances will inherit the Readable stream interface. Our implementation need only implement the abstract _read method of Readable, which will push data to a consumer until there is nothing more to push, at which point it triggers the Readable stream to emit an "end" event by pushing a null value: var Feed = function(channel) { var readable = new stream.Readable({ encoding : "utf8" }); var news = [ "Big Win!", "Stocks Down!", "Actor Sad!" ]; readable._read = function() { if(news.length) { return readable.push(news.shift() + "n"); } readable.push(null); }; return readable; } Now that we have an implementation, a consumer might want to instantiate the stream and listen for stream events. Two key events are readable and end. The readable event is emitted as long as data is being pushed to the stream. It alerts the consumer to check for new data via the read method of Readable. Note again how the Readable implementation must provide a private _read method, which services the public read method exposed to the consumer API. The end event will be emitted whenever a null value is passed to the push method of our Readable implementation. Here we see a consumer using these methods to display new stream data, providing a notification when the stream has stopped sending data: var feed = new Feed(); feed.on("readable", function() { var data = feed.read(); data && process.stdout.write(data); }); feed.on("end", function() { console.log("No more news"); }); Similarly, we could implement a stream of objects through the use of the objectMode option: var readable = new stream.Readable({ objectMode : true }); var prices = [ { price : 1 }, { price : 2 } ]; ... readable.push(prices.shift()); // } { prices : 1 } // } { prices : 2 } Here we see that each read event is receiving an object, rather than a buffer or string. Finally, the read method of a Readable stream can be passed a single argument indicating the number of bytes to be read from the stream's internal buffer. For example, if it was desired that a file should be read one byte at a time, one might implement a consumer using a routine similar to: readable.push("Sequence of bytes"); ... feed.on("readable", function() { var character; while(character = feed.read(1)) { console.log(character); }; }); // } S // } e // } q // } ... Here it should be clear that the Readable stream's buffer was filled with a number of bytes all at once, but was read from discretely. Pushing and pulling We have seen how a Readable implementation will use push to populate the stream buffer for reading. When designing these implementations it is important to consider how volume is managed, at either end of the stream. Pushing more data into a stream than can be read can lead to complications around exceeding available space (memory). At the consumer end it is important to maintain awareness of termination events, and how to deal with pauses in the data stream. One might compare the behavior of data streams running through a network with that of water running through a hose. As with water through a hose, if a greater volume of data is being pushed into the read stream than can be efficiently drained out of the stream at the consumer end through read, a great deal of back pressure builds, causing a data backlog to begin accumulating in the stream object's buffer. Because we are dealing with strict mathematical limitations, read simply cannot be compelled to release this pressure by reading more quickly—there may be a hard limit on available memory space, or other limitation. As such, memory usage can grow dangerously high, buffers can overflow, and so forth. A stream implementation should therefore be aware of, and respond to, the response from a push operation. If the operation returns false this indicates that the implementation should cease reading from its source (and cease pushing) until the next _read request is made. In conjunction with the above, if there is no more data to push but more is expected in the future the implementation should push an empty string (""), which adds no data to the queue but does ensure a future readable event. While the most common treatment of a stream buffer is to push to it (queuing data in a line), there are occasions where one might want to place data on the front of the buffer (jumping the line). Node provides an unshift operation for these cases, which behavior is identical to push, outside of the aforementioned difference in buffer placement. Writable streams A Writable stream is responsible for accepting some value (a stream of bytes, a string) and writing that data to a destination. Streaming data into a file container is a common use case. To create a Writable stream: var stream = require('stream'); var readable = new stream.Writable({ highWaterMark : 16000, decodeStrings: true }); The Writable streams constructor can be instantiated with two options: highWaterMark: The maximum number of bytes the stream's buffer will accept prior to returning false on writes. Default is 16 KB decodeStrings: Whether to convert strings into buffers before writing. Default is true. As with Readable streams, custom Writable stream implementations must implement a _write handler, which will be passed the arguments sent to the write method of instances. One should think of a Writable stream as a data target, such as for a file you are uploading. Conceptually this is not unlike the implementation of push in a Readable stream, where one pushes data until the data source is exhausted, passing null to terminate reading. For example, here we write 100 bytes to stdout: var stream = require('stream'); var writable = new stream.Writable({ decodeStrings: false }); writable._write = function(chunk, encoding, callback) { console.log(chunk); callback(); } var w = writable.write(new Buffer(100)); writable.end(); console.log(w); // Will be `true` There are two key things to note here. First, our _write implementation fires the callback function immediately after writing, a callback that is always present, regardless of whether the instance write method is passed a callback directly. This call is important for indicating the status of the write attempt, whether a failure (error) or a success. Second, the call to write returned true. This indicates that the internal buffer of the Writable implementation has been emptied after executing the requested write. What if we sent a very large amount of data, enough to exceed the default size of the internal buffer? Modifying the above example, the following would return false: var w = writable.write(new Buffer(16384)); console.log(w); // Will be 'false' The reason this write returns false is that it has reached the highWaterMark option—default value of 16 KB (16 * 1024). If we changed this value to 16383, write would again return true (or one could simply increase its value). What to do when write returns false? One should certainly not continue to send data! Returning to our metaphor of water in a hose: when the stream is full, one should wait for it to drain prior to sending more data. Node's Stream implementation will emit a drain event whenever it is safe to write again. When write returns false listen for the drain event before sending more data. Putting together what we have learned, let's create a Writable stream with a highWaterMark value of 10 bytes. We will send a buffer containing more than 10 bytes (composed of A characters) to this stream, triggering a drain event, at which point we write a single Z character. It should be clear from this example that Node's Stream implementation is managing the buffer overflow of our original payload, warning the original write method of this overflow, performing a controlled depletion of the internal buffer, and notifying us when it is safe to write again: var stream = require('stream'); var writable = new stream.Writable({ highWaterMark: 10 }); writable._write = function(chunk, encoding, callback) { process.stdout.write(chunk); callback(); } writable.on("drain", function() { writable.write("Zn"); }); var buf = new Buffer(20, "utf8"); buf.fill("A"); console.log(writable.write(buf.toString())); // false The result should be a string of 20 A characters, followed by false, then followed by the character Z. The fluid data in a Readable stream can be easily redirected to a Writable stream. For example, the following code will take any data sent by a terminal (stdin is a Readable stream) and pass it to the destination Writable stream, stdout: process.stdin.pipe(process.stdout); Whenever a Writable stream is passed to a Readable stream's pipe method, a pipe event will fire. Similarly, when a Writable stream is removed as a destination for a Readable stream, the unpipe event fires. To remove a pipe, use the following: unpipe(destination stream)   Duplex streams A duplex stream is both readable and writeable. For instance, a TCP server created in Node exposes a socket that can be both read from and written to: var stream = require("stream"); var net = require("net"); net .createServer(function(socket) { socket.write("Go ahead and type something!"); socket.on("readable", function() { process.stdout.write(this.read()) }); }) .listen(8080); When executed, this code will create a TCP server that can be connected to via Telnet: telnet 127.0.0.1 8080 Upon connection, the connecting terminal will print out Go ahead and type something! —writing to the socket. Any text entered in the connecting terminal will be echoed to the stdout of the terminal running the TCP server (reading from the socket). This implementation of a bi-directional (duplex) communication protocol demonstrates clearly how independent processes can form the nodes of a complex and responsive application, whether communicating across a network or within the scope of a single process. The options sent when constructing a Duplex instance merge those sent to Readable and Writable streams, with no additional parameters. Indeed, this stream type simply assumes both roles, and the rules for interacting with it follow the rules for the interactive mode being used. As a Duplex stream assumes both read and write roles, any implementation is required to implement both _write and _read methods, again following the standard implementation details given for the relevant stream type. Transforming streams On occasion stream data needs to be processed, often in cases where one is writing some sort of binary protocol or other "on the fly" data transformation. A Transform stream is designed for this purpose, functioning as a Duplex stream that sits between a Readable stream and a Writable stream. A Transform stream is initialized using the same options used to initialize a typical Duplex stream. Where Transform differs from a normal Duplex stream is in its requirement that the custom implementation merely provide a _transform method, excluding the _write and _read method requirement. The _transform method will receive three arguments, first the sent buffer, an optional encoding argument, and finally a callback which _transform is expected to call when the transformation is complete: _transform = function(buffer, encoding, cb) { var transformation = "..."; this.push(transformation) cb(); } Let's imagine a program that wishes to convert ASCII (American Standard Code for Information Interchange) codes into ASCII characters, receiving input from stdin. We would simply pipe our input to a Transform stream, then piping its output to stdout: var stream = require('stream'); var converter = new stream.Transform(); converter._transform = function(num, encoding, cb) { this.push(String.fromCharCode(new Number(num)) + "n") cb(); } process.stdin.pipe(converter).pipe(process.stdout); Interacting with this program might produce an output resembling the following: 65 A 66 B 256 A 257 a   Using PassThrough streams This sort of stream is a trivial implementation of a Transform stream, which simply passes received input bytes through to an output stream. This is useful if one doesn't require any transformation of the input data, and simply wants to easily pipe a Readable stream to a Writable stream. PassThrough streams have benefits similar to JavaScript's anonymous functions, making it easy to assert minimal functionality without too much fuss. For example, it is not necessary to implement an abstract base class, as one does with for the _read method of a Readable stream. Consider the following use of a PassThrough stream as an event spy: var fs = require('fs'); var stream = new require('stream').PassThrough(); spy.on('end', function() { console.log("All data has been sent"); }); fs.createReadStream("./passthrough.js").pipe(spy).pipe(process.std out); Summary As we have learned, Node's designers have succeeded in creating a simple, predictable, and convenient solution to the very difficult problem of enabling efficient I/O between disparate sources and targets. Its abstract Stream interface facilitates the instantiation of consistent readable and writable interfaces, and the extension of this interface into HTTP requests and responses, the filesystem, child processes, and other data channels makes stream programming with Node a pleasant experience. Resources for Article: Further resources on this subject: So, what is Node.js? [Article] Getting Started with Zombie.js [Article] So, what is KineticJS? [Article]

(For more resources related to this topic, see here.) QlikView error messages displayed during the running of the script, during reload, or just after the script is run are key to understanding what errors are contained in your code. After an error is detected and the error dialog appears, review the error, and click on OK or Cancel on the Script Error dialog box. If you have the debugger open, click on Close, then click on Cancel on the Sheet Properties dialog. Re-enter the Script Editor and examine your script to fix the error. Errors can come up as a result of syntax, formula or expression errors, join errors, circular logic, or any number of issues in your script. The following are a few common error messages you will encounter when developing your QlikView script. The first one, illustrated in the following screenshot, is the syntax error we received when running the code that missed a comma after Sales. This is a common syntax error. It's a little bit cryptic, but the error is contained in the code snippet that is displayed. The error dialog does not exactly tell you that it expected a comma in a certain place, but with practice, you will realize the error quickly. The next error is a circular reference error. This error will be handled automatically by QlikView. You can choose to accept QlikView's fix of loosening one of the tables in the circular reference (view the data model in Table Viewer for more information on which table is loosened, or view the Document Properties dialog, Tables tab to find out which table is marked Loosely Coupled). Alternatively, you can choose another table to be loosely coupled in the Document Properties, Tables tab, or you can go back into the script and fix the circular reference with one of the methods. The following screenshot is a warning/error dialog displayed when you have a circular reference in a script: Another common issue is an unknown statement error that can be caused by an error in writing your script—missed commas, colons, semicolons, brackets, quotation marks, or an improperly written formula. In the case illustrated in the following screenshot, the error has encountered an unknown statement—namely, the Customers line that QlikView is attempting to interpret as Customers Load *…. The fix for this error is to add a colon after Customers in the following way: Customers: There are instances when a load script will fail silently. Attempting to store a QVD or CSV to a file that is locked by another user viewing it is one such error. Another example is when you have two fields with the same name in your load statement. The debugger can help you find the script lines in which the silent error is present. Summary In this article we learned about QlikView error messages displayed during the script execution. Resources for Article: Further resources on this subject: Meet QlikView [Article] Introducing QlikView elements [Article] Linking Section Access to multiple dimensions [Article]

(For more resources related to this topic, see here.) A quick overview of AR concepts As AR has become increasingly popular in the media over the last few years, unfortunately, several distorted notions of Augmented Reality have evolved. Anything that is somehow related to the real world and involves some computing, such as standing in front of a shop and watching 3D models wear the latest fashions, has become AR. Augmented Reality emerged from research labs a few decades ago and different definitions of AR have been produced. As more and more research fields (for example, computer vision, computer graphics, human-computer interaction, medicine, humanities, and art) have investigated AR as a technology, application, or concept, multiple overlapping definitions now exist for AR. Rather than providing you with an exhaustive list of definitions, we will present some major concepts present in any AR application. Sensory augmentation The term Augmented Reality itself contains the notion of reality. Augmenting generally refers to the aspect of influencing one of your human sensory systems, such as vision or hearing, with additional information. This information is generally defined as digital or virtual and will be produced by a computer. The technology currently uses displays to overlay and merge the physical information with the digital information. To augment your hearing, modified headphones or earphones equipped with microphones are able to mix sound from your surroundings in realtime with sound generated by your computer. Displays The TV screen at home is the ideal device to perceive virtual content, streamed from broadcasts or played from your DVD. Unfortunately, most common TV screens are not able to capture the real world and augment it. An Augmented Reality display needs to simultaneously show the real and virtual worlds. One of the first display technologies for AR was produced by Ivan Sutherlandin 1964 (named "The Sword of Damocles"). The system was rigidly mounted on the ceiling and used some CRT screens and a transparent display to be able to create the sensation of visually merging the real and virtual. Since then, we have seen different trends in AR display, going from static to wearable and handheld displays. One of the major trends is the usage of optical see-through (OST) technology. The idea is to still see the real world through a semitransparent screen and project some virtual content on the screen. The merging of the real and virtual worlds does not happen on the computer screen, but directly on the retina of your eye, as depicted in the following figure: The other major trend in AR display is what we call video see-through (VST) technology. You can imagine perceiving the world not directly, but through a video on a monitor. The video image is mixed with some virtual content (as you will see in a movie) and sent back to some standard display, such as your desktop screen, your mobile phone, or the upcoming generation of head-mounted displays as shown in the following figure: In this book, we will work on Android-driven mobile phones and, therefore, discuss only VST systems; the video camera used will be the one on the back of your phone. Registration in 3D With a display (OST or VST) in your hands, you are already able to superimpose things from your real world, as you will see in TV advertisements with text banners at the bottom of the screen. However, any virtual content (such as text or images will remain fixed in its position on the screen. The superposition being really static, your AR display will act as a head-up display (HUD), but won't really be an AR as shown in the following figure: Google Glass is an example of an HUD. While it uses a semitransparent screen like an OST, the digital content remains in a static position. AR needs to know more about real and virtual content. It needs to know where things are in space (registration) and follow where they are moving (tracking). Registration is basically the idea of aligning virtual and real content in the same space. If you are into movies or sports, you will notice that 2D or 3D graphics are superimposed onto scenes of the physical world quite often. In ice hockey, the puck is often highlighted with a colored trail. In movies such as Walt Disney'sTRON (1982 version), the real and virtual elements are seamlessly blended. However, AR differs from those effects as it is based on all of the following aspects (proposed by Ronald T. Azumain 1997): It's in 3D: In the olden days, some of the movies were edited manually to merge virtual visual effects with real content. A well-known example is Star Wars, where all the lightsaber effects have been painted by hand by hundreds of artists and, thus, frame by frame. Nowadays, more complex techniques support merging digital 3D content (such as characters or cars) with the video image (and is called match moving). AR is inherently always doing that in a 3D space. The registration happens in real time: In a movie, everything is prerecorded and generated in a studio; you just play the media. In AR, everything is in real time, so your application needs to merge, at each instance, reality and virtuality. It's interactive: In a movie, you only look passively at the scene from where it has been shot. In AR, you can actively move around, forward, and backward and turn your AR display—you will still see an alignment between both worlds. Interaction with the environment Building a rich AR application needs interaction between environments; otherwise you end up with pretty, 3D graphics that can turn boring quite fast. AR interaction refers to selecting and manipulating digital and physical objects and navigating in the augmented scene. Rich AR applications allow you to use objects which can be on your table, to move some virtual characters, use your hands to select some floating virtual objects while walking on the street, or speak to a virtual agent appearing on your watch to arrange a meeting later in the day. We will look at how some of the standard mobile interaction techniques can also be applied to AR. We will also dig into specific techniques involving the manipulation of the real world. Summary Thus we have learned about the AR concepts through this article. Resources for Article: Further resources on this subject: Marker-based Augmented Reality on iPhone or iPad [Article] Creating Dynamic UI with Android Fragments [Article] Introducing an Android platform [Article]

(For more resources related to this topic, see here.) This article explores how to work with images and maps. Python has some well-known image libraries that allow us to process images in both aesthetic and scientific ways. We will touch on PIL's capabilities by demonstrating how to process images by applying filters and by resizing them. Furthermore, we will show how to use image files as annotation for our matplotlibs' charts. To deal with data visualization of geospatial datasets, we will cover the functionality of Python's available libraries and public APIs that we can use with map-based visual representations. The final recipe shows how Python can create CAPTCHA test images. Processing images with PIL Why use Python for image processing, if we could use WIMP (http://en.wikipedia.org/wiki/WIMP_(computing)) or WYSIWYG (http://en.wikipedia.org/wiki/WYSIWYG) to achieve the same goal? This is used because we want to create an automated system to process images in real time without human support, thus, optimizing the image pipeline. Getting ready Note that the PIL coordinate system assumes that the (0,0) coordinate is in the upper-left corner. The Image module has a useful class and instance methods to perform basic operations over a loaded image object (im): im = Image.open(filename): This opens a file and loads the image into im object. im.crop(box): This crops the image inside the coordinates defined by box. box defines left, upper, right, lower pixels coordinates (for example: box = (0, 100, 100,100)). im.filter(filter): This applies a filter on the image and returns a filtered image. im.histogram(): This returns a histogram list for this image, where each item represents the number of pixels. Number of items in the list is 256 for single channel images, but if the image is not a single channel image, there can be more items in the list. For an RGB image the list contains 768 items (one set of 256 values for each channel). im.resize(size, filter): This resizes the image and uses a filter for resampling. The possible filters are NEAREST, BILINEAR, BICUBIC, and ANTIALIAS. The default is NEAREST. im.rotate(angle, filter): This rotates an image in the counter clockwise direction. im.split(): This splits bands of image and returns a tuple of individual bands. Useful for splitting an RGB image into three single band images. im.transform(size, method, data, filter): This applies transformation on a given image using data and a filter. Transformation can be AFFINE, EXTENT, QUAD, and MESH. You can read more about transformation in the official documentation. Data defines the box in the original image where the transformation will be applied. The ImageDraw module allows us to draw over the image, where we can use functions such as arc, ellipse, line, pieslice, point, and polygon to modify the pixels of the loaded image. The ImageChops module contains a number of image channel operations (hence the name Chops) that can be used for image composition, painting, special effects, and other processing operations. Channel operations are allowed only for 8-bit images. Here are some interesting channel operations: ImageChops.duplicate(image): This copies current image into a new image object ImageChops.invert(image): This inverts an image and returns a copy ImageChops.difference(image1, image2): This is useful for verification that images are the same without visual inspection The ImageFilter module contains the implementation of the kernel class that allows the creation of custom convolution kernels. This module also contains a set of healthy common filters that allows the application of well-known filters (BLUR and MedianFilter) to our image. There are two types of filters provided by the ImageFilter module: fixed image enhancement filters and image filters that require certain arguments to be defined; for example, size of the kernel to be used. We can easily get the list of all fixed filter names in IPython: In [1]: import ImageFilter In [2]: [ f for f in dir(ImageFilter) if f.isupper()] Out[2]: ['BLUR', 'CONTOUR', 'DETAIL', 'EDGE_ENHANCE', 'EDGE_ENHANCE_MORE', 'EMBOSS', 'FIND_EDGES', 'SHARPEN', 'SMOOTH', 'SMOOTH_MORE'] The next example shows how we can apply all currently supported fixed filters on any supported image: import os import sys from PIL import Image, ImageChops, ImageFilter class DemoPIL(object): def __init__(self, image_file=None): self.fixed_filters = [ff for ff in dir(ImageFilter) if ff.isupper()] assert image_file is not None assert os.path.isfile(image_file) is True self.image_file = image_file self.image = Image.open(self.image_file) def _make_temp_dir(self): from tempfile import mkdtemp self.ff_tempdir = mkdtemp(prefix="ff_demo") def _get_temp_name(self, filter_name): name, ext = os.path.splitext(os.path.basename(self.image_file)) newimage_file = name + "-" + filter_name + ext path = os.path.join(self.ff_tempdir, newimage_file) return path def _get_filter(self, filter_name): # note the use Python's eval() builtin here to return function object real_filter = eval("ImageFilter." + filter_name) return real_filter def apply_filter(self, filter_name): print "Applying filter: " + filter_name filter_callable = self._get_filter(filter_name) # prevent calling non-fixed filters for now if filter_name in self.fixed_filters: temp_img = self.image.filter(filter_callable) else: print "Can't apply non-fixed filter now." return temp_img def run_fixed_filters_demo(self): self._make_temp_dir() for ffilter in self.fixed_filters: temp_img = self.apply_filter(ffilter) temp_img.save(self._get_temp_name(ffilter)) print "Images are in: {0}".format((self.ff_tempdir),) if __name__ == "__main__": assert len(sys.argv) == 2 demo_image = sys.argv[1] demo = DemoPIL(demo_image) # will create set of images in temporary folder demo.run_fixed_filters_demo() We can run this easily from the command prompt: $ pythonch06_rec01_01_pil_demo.py image.jpeg We packed our little demo in the DemoPIL class, so we can extend it easily while sharing the common code around the demo function run_fixed_filters_demo. Common code here includes opening the image file, testing if the file is really a file, creating temporary directory to hold our filtered images, building the filtered image filename, and printing useful information to user. This way the code is organized in a better manner and we can easily focus on our demo function, without touching other parts of the code. This demo will open our image file and apply every fixed filter available in ImageFilter to it and save that new filtered image in a unique temporary directory. The location of this temporary directory is retrieved, so we can open it with our OS's file explorer and view the created images. As an optional exercise, try extending this demo class to perform other filters available in ImageFilter on the given image. How to do it... The example in this section shows how we can process all the images in a certain folder. We specify a target path, and the program reads all the image files in that target path (images folder) and resizes them to a specified ratio (0.1 in this example), and saves each one in a target folder called thumbnail_folder: import os import sys from PIL import Image class Thumbnailer(object): def __init__(self, src_folder=None): self.src_folder = src_folder self.ratio = .3 self.thumbnail_folder = "thumbnails" def _create_thumbnails_folder(self): thumb_path = os.path.join(self.src_folder, self.thumbnail_folder) if not os.path.isdir(thumb_path): os.makedirs(thumb_path) def _build_thumb_path(self, image_path): root = os.path.dirname(image_path) name, ext = os.path.splitext(os.path.basename(image_path)) suffix = ".thumbnail" return os.path.join(root, self.thumbnail_folder, name + suffix + ext) def _load_files(self): files = set() for each in os.listdir(self.src_folder): each = os.path.abspath(self.src_folder + '/' + each) if os.path.isfile(each): files.add(each) return files def _thumb_size(self, size): return (int(size[0] * self.ratio), int(size[1] * self.ratio)) def create_thumbnails(self): self._create_thumbnails_folder() files = self._load_files() for each in files: print "Processing: " + each try: img = Image.open(each) thumb_size = self._thumb_size(img.size) resized = img.resize(thumb_size, Image.ANTIALIAS) savepath = self._build_thumb_path(each) resized.save(savepath) except IOError as ex: print "Error: " + str(ex) if __name__ == "__main__": # Usage: # ch06_rec01_02_pil_thumbnails.py my_images assert len(sys.argv) == 2 src_folder = sys.argv[1] if not os.path.isdir(src_folder): print "Error: Path '{0}' does not exits.".format((src_folder)) sys.exit(-1) thumbs = Thumbnailer(src_folder) # optionally set the name of theachumbnail folder relative to *src_folder*. thumbs.thumbnail_folder = "THUMBS" # define ratio to resize image to # 0.1 means the original image will be resized to 10% of its size thumbs.ratio = 0.1 # will create set of images in temporary folder thumbs.create_thumbnails() How it works... For the given src_folder folder, we load all the files in this folder and try to load each file using Image.open(); this is the logic of the create_thumbnails() function. If the file we try to load is not an image, IOError will be thrown, and it will print this error and skip to next file in the sequence. If we want to have more control over what files we load, we should change the _load_files() function to only include files with certain extension (file type): for each in os.listdir(self.src_folder): if os.path.isfile(each) and os.path.splitext(each) is in ('.jpg','.png'): self._files.add(each) This is not foolproof, as file extension does not define file type, it just helps the operating system to attach a default program to the file, but it works in majority of the cases and is simpler than reading a file header to determine the file content (which still does not guarantee that the file really is the first couple of bytes, say it is). There's more... With PIL, although not used very often, we can easily convert images from one format to the other. This is achievable with two simple operations: first open an image in a source format using open(), and then save that image in the other format using save(). Format is defined either implicitly via filename extension (.png or .jpeg), or explicitly via the format of the argument passed to the save() function.